DETAILED ACTION
This communication is in response to the application filed 12/20/23 in which claims 1-20 were presented for examination.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 8/27/24 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or non-obviousness.
Claims 1 and 4 are rejected under 35 U.S.C. 103 as being unpatentable over Chen (US 2019/0114543 A1; published Apr. 18, 2019) in view of Krishnan (US 2005/0049497 A1; published Mar. 3, 2005) (hereinafter “Krishnan-1”), Korchemny (US 11,176,293 B1; published Nov. 16, 2021), and Nurvitadhi (US 2019/0042529 A1; published Feb. 7, 2019).
Regarding claim 1, Chen discloses [a] device, comprising: an electronic processor having: (Chen ¶ 34 (“The present invention aims to realize local learning applied to local AI device(s), such as smartphone, tablet, smart-TV, telephone, computer, home entertainment, wearable device, and so on, instead of standalone or cloud computing server(s) with high level hardware.”))
[…]
a layout of circuit gates implementing an optimized neural network model, (Chen ¶ 43 (“The training and the inference can be performed at the same time if there is enough hardware resource, for example, in case where the inference only uses some groups of N groups of computing engines.”)) the optimized neural network model obtained by removing a first neuron of a plurality of neurons in a trained neural network model to decrease hardware computational resources utilized for the first neuron; and (Chen ¶ 59 (“The original neural network 4 includes a plurality of neurons 41 and a plurality of links 42 between the neurons 41, and it has a (relatively) complete neural network structure. In the training phase, large data source is used to train the original neural network 4, so as to enhance its model generality; which means that the model may be effective in general cases.”), ¶ 60 (“After the original neural network 4 obtains enough model generality in the training phase, it is pruned to become the pruned neural network 4′ for the application phase.”)).
Although Chen teaches that the pruned neural network (“optimized neural network”) is then used to perform inferencing in the application stage and output a result (“prediction indication”) at the local device, see FIG. 6, Chen does not expressly disclose:
wherein the layout causes the electronic processor to:
when receiving a runtime dataset for a patient via the set of input pins, extract a plurality of features from the runtime dataset; (but see Krishnan-1 ¶ 49 (“The extraction modules (22-4) can use relevant data in the domain knowledge base (27) to extract relevant parameters and produce probabilistic assertions (elements) about the patient that are relevant to an instant in time or time period. The domain knowledge required for extraction is generally specific to each source. For example, extraction from a text source may be carried out by phrase spotting, wherein a list of rules are provided that specify the phrases of interest and the inferences that can be drawn therefrom. For example, if there is a statement in a doctor's note with the words—“There is evidence of lesions in the left breast”—then, in order to infer from this sentence that the patient has or may have breast cancer, a rule can be specified that directs the system to look for the phrase “lesion,” and, if it is found, to assert that the patient may have breast cancer with a some degree of confidence.”))
apply the plurality of features to the optimized neural network model to obtain a confidence level; and (but see Krishnan-1 ¶ 51 (“The model builder (24-2) builds classification models implemented by the classification method (24-1), which are trained (and possibly dynamically optimized) to analyze various extracted features and provide diagnostic assistance and assessment on various levels, depending on the implementation. It is to be appreciated that the classification models may be “black boxes” that are unable to explain their prediction to a user (which is the case if classifiers are built using neural networks, example). The classification models may be “white boxes” that are in a human readable form (which is the case if classifiers are built using decision trees, for example). In other embodiments, the classification models may be “gray boxes” that can partially explain how solutions are derived (e.g., a combination of “white box” and “black box” type classifiers). The type of classification models that are implemented will depend on the domain knowledge data and model building process (24-2). The type of model building process will vary depending on the classification scheme implemented, which may include decision trees, support vector machines, Bayesian networks, probabilistic reasoning, etc., and other classification methods that are known to those of ordinary skill in the art.”), ¶ 53 (“The diagnostic/workflow assistance module (26) can provide one or more diagnostic and decision support functions as described above with reference to FIG. 1. For instance, the diagnostic/workflow assistance module (26) can command the classification module (24) to classify one or more breast lesions detected in ultrasound image data (4) as malignant or benign and provide a probability of such diagnosis and (optionally) a measure of confidence in the diagnosis, based on a set of features extracted from ultrasound image data (3) and/or non-image patient data records (4). The classification engine (25-1) could perform such classification using one or more classification models that are trained to analyze the combined features output from module (23).”))
output a prediction indication based on the confidence level via the output pins (but see Krishnan-1 ¶ 24 (“In another exemplary embodiment of the invention, the CAD system (10) can extract and analyze information from image data (1) and (optionally) non-image data (2) to automatically generate and output a probability of diagnosis and (optionally) a measure of confidence of the diagnosis (11) or alternatively output a suggested therapy with a probability and (optional) measure of confidence as to the impact of the suggested therapy, e.g., the probability that the suggested therapy will have the desired (beneficial) impact. Collectively, the output (11) can be referred to herein as “Probability and Confidence of Suggestion”.”)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Chen to incorporate the teachings of Krishnan-1 to use the pruned neural network model to extract features from diagnostic data and generate a predicted diagnosis, at least because doing so would provide automated decision support functions to assist a physician in various aspects of physician workflow including, but not limited to, diagnosing medical conditions (breast tumors) and determining efficacious healthcare or diagnostic or therapeutic paths for the subject patient. See Krishnan-1 ¶ 2.
Chen and Krishnan-1 do not expressly disclose that the electronic processor having a set of input pins to receive the input feature data and a set of output pins to output a result. However, Korchemny col. 24, ll. 15-27 (“The emulator includes multiple FPGAs (or other programmable devices), for example, elements 204.sub.1 to 204.sub.N in FIG. 10. Each FPGA can include one or more FPGA interfaces through which the FPGA is connected to other FPGAs of the emulator (and potentially other emulator hardware components), in order for the FPGAs to exchange signals. An FPGA interface may also be referred to as an input/output pin or an FPGA pad. While some embodiments disclosed herein make use of emulators comprising FPGAs, other embodiments can include other types of logic blocks instead of or along with, the FPGAs for emulating DUTs, for example, custom FPGAs, specialized ASICs for emulation or prototyping, memories, and input/output devices.”)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Chen and Krishnan-1 to incorporate the teachings of Korchemny to implement the hardware processor as an FPGA at least because the “highly flexible nature of programmable logic devices makes them an excellent fit for accelerating many computing tasks.” Nurvitadhi ¶ 4.
Regarding claim 4, Chen, in view of Krishnan-1, Korchemny, and Nurvitadhi, discloses the invention of claim 1 as discussed above. Chen et al. do not expressly disclose wherein the runtime dataset comprises a breast cancer dataset, and wherein the confidence level comprises a possibility indication of breast cancer in the patient (but see Krishnan-1 ¶ 25 (“More specifically, by way of example, for purposes of diagnosing breast cancer, the CAD system (10) may comprise methods for automatically detecting and diagnosing (or otherwise characterizing) suspect breast lesions in breast tissue and outputting, for example, a probability of malignancy of such lesions, together with an optional measure of confidence in such diagnosis. In this example, the CAD system (10) could extract and analyze relevant features from a screening X-ray mammogram (image data) and clinical history information (non-image data) of a patient and provide a current estimate and confidence of malignancy.”)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Chen to incorporate the teachings of Krishnan-1 to apply the pruned neural network to predict cardiac disease based on features extracted from cardiac health information, at least because doing so would provide automated assistance to a physician for various aspects of physician workflow including, for example, automated assessment of regional myocardial function through wall motion analysis, automated diagnosis of heart diseases and conditions such as cardiomyopathy, coronary artery disease and other heart-related medical conditions, and other automated decision support functions to assist physician workflow. Krishnan-1 ¶ 11.
Claim 2 is rejected under 35 U.S.C. 103 as being unpatentable over Chen, Krishnan-1, Korchemny, and Nurvitadhi as applied to claim 1 above, and further in view of Raut, Gopal, et al. "RECON: resource-efficient CORDIC-based neuron architecture." IEEE Open Journal of Circuits and Systems 2 (2021): 170-181 (“Raut”) and Ramadhan, Luthfi, Neural Network: The Dead Neuron (Nov. 16, 2021) (available at https://towardsdatascience.com/neural-network-the-dead-neuron-eaa92e575748/) ( “Ramadhan”).
Regarding claim 2, Chen, in view of Krishnan-1, Korchemny, and Nurvitadhi, discloses the invention of claim 1 as discussed above. Krishnan-1 et al. do not expressly disclose wherein the first neuron comprises a multiplier, a first adder, and a second adder, (but see Raut FIGURE 2 (reproduced below):
PNG
media_image1.png
536
640
media_image1.png
Greyscale
(the multiply-accumulate unit includes a first adder and a second adder for a bias element)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Chen to incorporate the teachings of Raut to include a multiply-accumulate unit with a second bias adder, at least because doing so would enable replacing the output of neurons with small output variances with simple bias units. See Chen ¶ 75.
Chen further discloses:
wherein the first neuron of the plurality of neurons in the trained neural network model has been removed based on a first range of the first neuron, (Chen ¶ 69 (“A neuron statistic engine 50 is designed to determine which neuron should be pruned. In particular, the neuron statistic engine 50 is designed to compute and store activity statistics for each neuron at the application phase. The neuron statistic engine 50 may be set in the local AI device to prune the original neural network 4 therein.”), ¶ 70 (“The activity statistics may include a histogram of neuron's input and/or output, a mean of neuron's input and/or output, a variance of neuron's input and/or output, and other kinds of statistical quantities. A histogram is shown in the top-right side of FIG. 5, with bins of output values in X-axis and count(s) in Y-axis.”))
wherein the first range comprises a plurality of first neuron outputs corresponding to a plurality of training datasets, (Chen ¶ 70 (“The activity statistics may include a histogram of neuron's input and/or output, a mean of neuron's input and/or output, a variance of neuron's input and/or output, and other kinds of statistical quantities. A histogram is shown in the top-right side of FIG. 5, with bins of output values in X-axis and count(s) in Y-axis.”)).
Chen does not expressly disclose:
wherein each of the plurality of first neuron outputs corresponds to a second adder output of the second adder, and (but see Raut FIGURE 2 above (illustrating the output of the MAC unit is the output of the second bias adder)). The rationale for combining Chen with Raut is the same as set forth above.
Although Chen teaches performing pruning according a range of the output values of neurons, Chen et al. does not expressly disclose wherein each of the plurality of first neuron outputs was a negative value (but see Ramadhan (“The drawback of ReLU is that they cannot learn on examples for which their activation is zero. It usually happens if you initialize the entire neural network with zero and place ReLU on the hidden layers. Another cause is when a large gradient flows through, a ReLU neuron will update its weight and might be ended up with a big negative weight and bias. If this happens, this neuron will always produce 0 during the forward propagation, and then the gradient flowing through this neuron will forever be zero irrespective of the input.
In other words, the weights of this neuron will never be updated again. Such a neuron can be considered as a dead neuron, which is considered a kind of permanent "brain damage" in biological terms. A dead neuron can be thought of as a natural Dropout.”) (the output of a neuron is always zero if the input to the ReLU activation function is negative)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Chen to incorporate the teachings of Ramadhan to remove dead neurons, i.e., neurons whose output is always zero, at least because it prevents cutting the gradient to the previous layer during backpropagation.
Claim 3 is rejected under 35 U.S.C. 103 as being unpatentable over Chen, Krishnan-1, Korchemny, and Nurvitadhi as applied to claim 1 above, and further in view of Krishnan (US 2005/0020903 A1; published Jan. 27, 2005) (“Krishnan-2”).
Regarding claim 3, Chen, in view of Krishnan-1, Korchemny, and Nurvitadhi, discloses the invention of claim 1 as discussed above. Chen et al. do not expressly disclose wherein the runtime dataset comprises a heart disease dataset, and wherein the confidence level comprises a possibility indication of heart disease in the patient (but see Krishnan-2 ¶ 11 (“Exemplary embodiments of the invention generally include systems and methods for providing automated diagnosis and decision support for cardiac imaging. More specifically, exemplary embodiments of the invention include CAD (computer-aided diagnosis) systems and applications for cardiac imaging, which implement automated methods for extracting and analyzing relevant features/parameters from a collection of patient information (including image data and/or non-image data) of a subject patient to provide automated assistance to a physician for various aspects of physician workflow including, for example, automated assessment of regional myocardial function through wall motion analysis, automated diagnosis of heart diseases and conditions such as cardiomyopathy, coronary artery disease and other heart-related medical conditions, and other automated decision support functions to assist physician workflow.”)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Chen to incorporate the teachings of Krishnan-2 to apply the pruned neural network to predict cardiac disease based on features extracted from cardiac health information, at least because doing so would provide automated assistance to a physician for various aspects of physician workflow including, for example, automated assessment of regional myocardial function through wall motion analysis, automated diagnosis of heart diseases and conditions such as cardiomyopathy, coronary artery disease and other heart-related medical conditions, and other automated decision support functions to assist physician workflow. Krishnan-2 ¶ 11.
Claims 5-7 and 15-16 are rejected under 35 U.S.C. 103 as being unpatentable over Chen in view of Barnard (US 2018/0101763 A1; published Apr. 12, 2018).
Regarding claim 5, Chen discloses [a] method for range-based hardware optimization, comprising:
obtaining a trained neural network model, the trained neural network model comprising a plurality of neurons; (Chen ¶ 58 (“In common cases, the original neural network 4 is a deep neural network constructed in a standalone or cloud computing server. However, according to the present invention, the original neural network 4 is a local neural network provided in a local learning system 2.”), ¶ 59 (“The original neural network 4 includes a plurality of neurons 41 and a plurality of links 42 between the neurons 41, and it has a (relatively) complete neural network structure. In the training phase, large data source is used to train the original neural network 4, so as to enhance its model generality; which means that the model may be effective in general cases.”))
determining a plurality of ranges for the plurality of neurons based on a plurality of training datasets, the plurality of ranges corresponding to the plurality of neurons; (Chen ¶ 69 (“A neuron statistic engine 50 is designed to determine which neuron should be pruned. In particular, the neuron statistic engine 50 is designed to compute and store activity statistics for each neuron at the application phase. The neuron statistic engine 50 may be set in the local AI device to prune the original neural network 4 therein.”), ¶ 70 (“The activity statistics may include a histogram of neuron's input and/or output, a mean of neuron's input and/or output, a variance of neuron's input and/or output, and other kinds of statistical quantities. A histogram is shown in the top-right side of FIG. 5, with bins of output values in X-axis and count(s) in Y-axis.”))
removing a first neuron from the trained neural network model based on a first range of the plurality of ranges to decrease hardware computational resources utilized for the first neuron; and (Chen ¶¶73-76 (“[0073] The neuron statistic engine 50 may perform the pruning or the merging according to any or all of the following pruning/merging criteria: [0074] For neurons with small output values, it deactivates them in the inference phase. That is, the neurons disappear in the pruned neural network 4′. [0075] For neurons with small output variances, it replaces them respectively with simple bias units, which means that the neurons only respectively have constants instead of variables. [0076] For neurons with same histogram or similar histograms, it merges them to remain only one neuron active. The links connected to the pruned neuron are instead connected to the remaining neuron. For example, neurons N11 and N12 have same histogram, so one of them can be merged into the other, as correspondingly shown in FIG. 4.”)).
Chen does not expressly disclose:
generating an optimized circuit layout for hardware that implements an optimized neural network model generated based on the plurality of neurons without the first neuron (but see Barnard ¶ 8 (“The convolutional neural network may be embodied in hardware on an integrated circuit. There may be provided a method of manufacturing, at an integrated circuit manufacturing system, hardware for implementing a convolutional neural network. There may be provided an integrated circuit definition dataset that, when processed in an integrated circuit manufacturing system, configures the system to manufacture hardware for implementing a convolutional neural network. There may be provided a non-transitory computer readable storage medium having stored thereon a computer readable description of an integrated circuit that, when processed, causes a layout processing system to generate a circuit layout description used in an integrated circuit manufacturing system to manufacture hardware for implementing a convolutional neural network.”)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Chen to incorporate the teachings of Barnard to generate a circuit layout description to manufacture hardware for implementing the pruned neural network, at least because doing so would enable implementing the pruned neural network in a smart home device. See Chen ¶ 94.
Regarding claim 6, Chen, in view of Barnard, discloses the invention of claim 5 as discussed above. Chen does not expressly disclose manufacturing a chip according to the optimized circuit layout, wherein the chip has a first number of gates less than a second number of gates for a second chip implementing the trained neural network model (but see Barnard ¶ 8 (“The convolutional neural network may be embodied in hardware on an integrated circuit. There may be provided a method of manufacturing, at an integrated circuit manufacturing system, hardware for implementing a convolutional neural network. There may be provided an integrated circuit definition dataset that, when processed in an integrated circuit manufacturing system, configures the system to manufacture hardware for implementing a convolutional neural network. There may be provided a non-transitory computer readable storage medium having stored thereon a computer readable description of an integrated circuit that, when processed, causes a layout processing system to generate a circuit layout description used in an integrated circuit manufacturing system to manufacture hardware for implementing a convolutional neural network.”)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Chen to incorporate the teachings of Barnard to generate a circuit layout description to manufacture hardware for implementing the pruned neural network, at least because doing so would enable implementing the pruned neural network in a smart home device. See Chen ¶ 94.
Regarding claim 7, Chen, in view of Barnard, discloses the invention of claim 5 as discussed above. Chen further discloses wherein the trained neural network model comprises a multilayer perceptron neural network model (Chen FIG. 4).
Regarding claim 15, Chen discloses [a] system for range-based hardware optimization, comprising:
an electronic processor, and (Chen ¶ 34 (“The present invention aims to realize local learning applied to local AI device(s), such as smartphone, tablet, smart-TV, telephone, computer, home entertainment, wearable device, and so on, instead of standalone or cloud computing server(s) with high level hardware.”)).
Chen does not expressly disclose:
a non-transitory computer-readable medium storing machine-executable instructions, which, when executed by the electronic processor, cause the electronic processor to: (but see Barnard ¶ 8 (“The convolutional neural network may be embodied in hardware on an integrated circuit. There may be provided a method of manufacturing, at an integrated circuit manufacturing system, hardware for implementing a convolutional neural network. There may be provided an integrated circuit definition dataset that, when processed in an integrated circuit manufacturing system, configures the system to manufacture hardware for implementing a convolutional neural network. There may be provided a non-transitory computer readable storage medium having stored thereon a computer readable description of an integrated circuit that, when processed, causes a layout processing system to generate a circuit layout description used in an integrated circuit manufacturing system to manufacture hardware for implementing a convolutional neural network.”)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Chen to incorporate the teachings of Barnard to generate a circuit layout description to manufacture hardware for implementing the pruned neural network, at least because doing so would enable implementing the pruned neural network in a smart home device. See Chen ¶ 94.
Chen further discloses:
obtain a trained neural network model, the trained neural network model comprising: a plurality of neurons; (Chen ¶ 58 (“In common cases, the original neural network 4 is a deep neural network constructed in a standalone or cloud computing server. However, according to the present invention, the original neural network 4 is a local neural network provided in a local learning system 2.”), ¶ 59 (“The original neural network 4 includes a plurality of neurons 41 and a plurality of links 42 between the neurons 41, and it has a (relatively) complete neural network structure. In the training phase, large data source is used to train the original neural network 4, so as to enhance its model generality; which means that the model may be effective in general cases.”))
determine a plurality of ranges for the plurality of neurons based on a plurality of training datasets, the plurality of ranges corresponding to the plurality of neurons; (Chen ¶ 69 (“A neuron statistic engine 50 is designed to determine which neuron should be pruned. In particular, the neuron statistic engine 50 is designed to compute and store activity statistics for each neuron at the application phase. The neuron statistic engine 50 may be set in the local AI device to prune the original neural network 4 therein.”), ¶ 70 (“The activity statistics may include a histogram of neuron's input and/or output, a mean of neuron's input and/or output, a variance of neuron's input and/or output, and other kinds of statistical quantities. A histogram is shown in the top-right side of FIG. 5, with bins of output values in X-axis and count(s) in Y-axis.”))
remove a first neuron based on a first range of the plurality of ranges to decrease hardware computational resources utilized for the first neuron; and (Chen ¶¶73-76 (“[0073] The neuron statistic engine 50 may perform the pruning or the merging according to any or all of the following pruning/merging criteria: [0074] For neurons with small output values, it deactivates them in the inference phase. That is, the neurons disappear in the pruned neural network 4′. [0075] For neurons with small output variances, it replaces them respectively with simple bias units, which means that the neurons only respectively have constants instead of variables. [0076] For neurons with same histogram or similar histograms, it merges them to remain only one neuron active. The links connected to the pruned neuron are instead connected to the remaining neuron. For example, neurons N11 and N12 have same histogram, so one of them can be merged into the other, as correspondingly shown in FIG. 4.”)).
Chen does not expressly disclose:
generate an optimized circuit layout for hardware that implements an optimized neural network model generated based on the plurality of neurons without the first neuron (but see Barnard ¶ 8 (“The convolutional neural network may be embodied in hardware on an integrated circuit. There may be provided a method of manufacturing, at an integrated circuit manufacturing system, hardware for implementing a convolutional neural network. There may be provided an integrated circuit definition dataset that, when processed in an integrated circuit manufacturing system, configures the system to manufacture hardware for implementing a convolutional neural network. There may be provided a non-transitory computer readable storage medium having stored thereon a computer readable description of an integrated circuit that, when processed, causes a layout processing system to generate a circuit layout description used in an integrated circuit manufacturing system to manufacture hardware for implementing a convolutional neural network.”)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Chen to incorporate the teachings of Barnard to generate a circuit layout description to manufacture hardware for implementing the pruned neural network, at least because doing so would enable implementing the pruned neural network in a smart home device. See Chen ¶ 94.
Regarding claim 16, Chen, in view of Barnard, discloses the invention of claim 15 as discussed above. Chen does not expressly disclose wherein the optimized circuit layout for a first chip that implements the optimized neural network model uses a first number of gates less than a second number of gates for a second circuit layout to implement the trained neural network model (but see Barnard ¶ 8 (“The convolutional neural network may be embodied in hardware on an integrated circuit. There may be provided a method of manufacturing, at an integrated circuit manufacturing system, hardware for implementing a convolutional neural network. There may be provided an integrated circuit definition dataset that, when processed in an integrated circuit manufacturing system, configures the system to manufacture hardware for implementing a convolutional neural network. There may be provided a non-transitory computer readable storage medium having stored thereon a computer readable description of an integrated circuit that, when processed, causes a layout processing system to generate a circuit layout description used in an integrated circuit manufacturing system to manufacture hardware for implementing a convolutional neural network.”)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Chen to incorporate the teachings of Barnard to generate a circuit layout description to manufacture hardware for implementing the pruned neural network, at least because doing so would enable implementing the pruned neural network in a smart home device. See Chen ¶ 94.
Claims 8, 9, 11-13, 17, and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Chen and Barnard as applied to claims 5 and 15 above, and further in view of Raut.
Regarding claim 8, Chen, in view of Barnard, discloses the invention of claim 5 as discussed above. Although Chen teaches a neural network composed of neurons, Chen does not expressly disclose wherein the first neuron comprises a multiplier, a first adder, and a second adder, wherein the multiplier is configured to produce a plurality of multiplier outputs based on a plurality of inputs and a plurality of corresponding weights, wherein the first adder is configured to produce a first adder output based on the plurality of multiplier outputs, and wherein the second adder is configured to produce a second adder output based on the first adder output and a bias (but see Raut FIGURE 2 (reproduced below):
PNG
media_image1.png
536
640
media_image1.png
Greyscale
(the multiply-accumulate unit includes a first adder and a second adder for a bias element)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Chen to incorporate the teachings of Raut to include a multiply-accumulate unit with a second bias adder, at least because doing so would enable replacing the output of neurons with small output variances with simple bias units. See Chen ¶ 75.
Regarding claim 9, Chen, in view of Barnard and Raut, discloses the invention of claim 8 as discussed above. Chen further discloses wherein the first range comprises a plurality of first neuron outputs corresponding to the plurality of training datasets, and (Chen ¶ 70 (“The activity statistics may include a histogram of neuron's input and/or output, a mean of neuron's input and/or output, a variance of neuron's input and/or output, and other kinds of statistical quantities. A histogram is shown in the top-right side of FIG. 5, with bins of output values in X-axis and count(s) in Y-axis.”)).
Chen does not expressly disclose wherein each of the plurality of first neuron outputs corresponds to the second adder output (but see Raut FIGURE 2 (reproduced below):
PNG
media_image1.png
536
640
media_image1.png
Greyscale
(the multiply-accumulate unit includes a first adder and a second adder for a bias element)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Chen to incorporate the teachings of Raut to include a multiply-accumulate unit with a second bias adder, at least because doing so would enable replacing the output of neurons with small output variances with simple bias units. See Chen ¶ 75.
Regarding claim 11, Chen, in view of Barnard and Raut, discloses the invention of claim 8 as discussed above. Chen does not expressly disclose wherein the second adder output corresponds to an input to a rectified linear unit (but see Raut FIGURE 2 (reproduced below):
PNG
media_image1.png
536
640
media_image1.png
Greyscale
(the multiply-accumulate unit includes a first adder and a second adder for a bias element)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Chen to incorporate the teachings of Raut to include a multiply-accumulate unit with a second bias adder, at least because doing so would enable replacing the output of neurons with small output variances with simple bias units. See Chen ¶ 75.
Regarding claim 12, Chen, in view of Barnard and Raut, discloses the invention of claim 11 as discussed above. Chen does not expressly disclose removing the rectified linear unit associated with the first neuron (but see Raut, Gopal, et al. "RECON: resource-efficient CORDIC-based neuron architecture." IEEE Open Journal of Circuits and Systems 2 (2021): 170-181 FIGURE 2 (reproduced below):
PNG
media_image1.png
536
640
media_image1.png
Greyscale
(the neuron includes the MAC and the activation unit and, therefore, pruning the neuron also prunes the activation function)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Chen to incorporate the teachings of Raut to include a multiply-accumulate unit with a second bias adder, at least because doing so would enable replacing the output of neurons with small output variances with simple bias units. See Chen ¶ 75.
Regarding claim 13, Chen, in view of Barnard and Raut, discloses the invention of claim 8 as discussed above. Chen further discloses wherein the trained neural network model comprises a first hidden layer of the plurality of neurons and a second layer, (Chen FIG. 4 (two hidden layers))
wherein the second layer comprises a plurality of second neurons, and (Chen FIG. 4 (second layer includes multiple neurons))
wherein removing the first neuron comprises:
removing a plurality of input mappings in the trained neural network model from the plurality of inputs to the first neuron; and (Chen ¶ 64 (“As shown in the right side of FIG. 4, the pruned neural network 4′ in the local learning system 2 is trained only by limited data source, collected in a specific environment, for example, home, office, classroom, and so on. However, even though the pruned neural network 4′ lacks some neurons or some links, it is still effective to learn and recognize objects or conditions in the specific environment, because the specific environment has less variety.”) (FIG. 4 is reproduced below:
PNG
media_image2.png
320
340
media_image2.png
Greyscale
(the pruned nodes 41’ have the input mapping removed))
removing a plurality of output mappings in the trained neural network model from the first neuron to the plurality of second neurons (Chen ¶ 64 (“As shown in the right side of FIG. 4, the pruned neural network 4′ in the local learning system 2 is trained only by limited data source, collected in a specific environment, for example, home, office, classroom, and so on. However, even though the pruned neural network 4′ lacks some neurons or some links, it is still effective to learn and recognize objects or conditions in the specific environment, because the specific environment has less variety.”) (FIG. 4 is reproduced below:
PNG
media_image2.png
320
340
media_image2.png
Greyscale
(the pruned nodes 41’ have the output mapping removed)).
Regarding claim 17, Chen, in view of Barnard, discloses the invention of claim 15 as discussed above. Chen does not expressly disclose: wherein the first neuron comprises a multiplier, a first adder, and a second adder, wherein the multiplier is configured to produce a plurality of multiplier outputs based on a plurality of inputs and a plurality of corresponding weights, wherein the first adder is configured to produce a first adder output based on the plurality of multiplier outputs, and wherein the second adder is configured to produce a second adder output based on the first adder output and a bias (but see Raut FIGURE 2 (reproduced below):
PNG
media_image1.png
536
640
media_image1.png
Greyscale
(the multiply-accumulate unit includes a first adder and a second adder for a bias element)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Chen to incorporate the teachings of Raut to include a multiply-accumulate unit with a second bias adder, at least because doing so would enable replacing the output of neurons with small output variances with simple bias units. See Chen ¶ 75.
Regarding claim 18, Chen, in view of Barnard and Raut, discloses the invention of claim 17 as discussed above. Chen does not expressly disclose: wherein the second adder output corresponds to an input to a rectified linear unit, and wherein the machine-executable instructions further cause the electronic processor to remove the rectified linear unit associated with the first neuron (but see Raut FIGURE 2 (reproduced below):
PNG
media_image1.png
536
640
media_image1.png
Greyscale
(the neuron includes the MAC and the activation unit and, therefore, pruning the neuron also prunes the activation function)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Chen to incorporate the teachings of Raut to include a multiply-accumulate unit with a second bias adder, at least because doing so would enable replacing the output of neurons with small output variances with simple bias units. See Chen ¶ 75.
Claims 10 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Chen, Barnard, and Raut as applied to claims 9 and 17 above, and further in view of Ramadhan.
Regarding claim 10, Chen, in view of Barnard and Raut, discloses the invention of claim 9 as discussed above. Chen et al. do not expressly disclose wherein each of the plurality of first neuron outputs is a negative value (but see Ramadhan (“The drawback of ReLU is that they cannot learn on examples for which their activation is zero. It usually happens if you initialize the entire neural network with zero and place ReLU on the hidden layers. Another cause is when a large gradient flows through, a ReLU neuron will update its weight and might be ended up with a big negative weight and bias. If this happens, this neuron will always produce 0 during the forward propagation, and then the gradient flowing through this neuron will forever be zero irrespective of the input.
In other words, the weights of this neuron will never be updated again. Such a neuron can be considered as a dead neuron, which is considered a kind of permanent "brain damage" in biological terms. A dead neuron can be thought of as a natural Dropout.”) (the output of a neuron is always zero if the input to the ReLU activation function is negative)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Chen to incorporate the teachings of Ramadhan to remove dead neurons, i.e., neurons whose output is always zero, at least because it prevents cutting the gradient to the previous layer during backpropagation.
Regarding claim 19, Chen, in view of Barnard and Raut, discloses the invention of claim 17 as discussed above. Chen further discloses wherein the first range comprises a plurality of first neuron outputs corresponding to the plurality of training datasets, and (Chen ¶ 70 (“The activity statistics may include a histogram of neuron's input and/or output, a mean of neuron's input and/or output, a variance of neuron's input and/or output, and other kinds of statistical quantities. A histogram is shown in the top-right side of FIG. 5, with bins of output values in X-axis and count(s) in Y-axis.”)).
Chen does not expressly disclose:
wherein each of the plurality of first neuron outputs corresponds to the second adder output, and (but see Raut FIGURE 2 above (illustrating the output of the MAC unit is the output of the second bias adder)). The rationale for combining Chen with Raut is the same as set forth above.
Although Chen teaches performing pruning according a range of the output values of neurons, Chen et al. does not expressly disclose wherein each of the plurality of first neuron outputs is a negative value (but see Ramadhan (“The drawback of ReLU is that they cannot learn on examples for which their activation is zero. It usually happens if you initialize the entire neural network with zero and place ReLU on the hidden layers. Another cause is when a large gradient flows through, a ReLU neuron will update its weight and might be ended up with a big negative weight and bias. If this happens, this neuron will always produce 0 during the forward propagation, and then the gradient flowing through this neuron will forever be zero irrespective of the input.
In other words, the weights of this neuron will never be updated again. Such a neuron can be considered as a dead neuron, which is considered a kind of permanent "brain damage" in biological terms. A dead neuron can be thought of as a natural Dropout.”) (the output of a neuron is always zero if the input to the ReLU activation function is negative)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Chen to incorporate the teachings of Ramadhan to remove dead neurons, i.e., neurons whose output is always zero, at least because it prevents cutting the gradient to the previous layer during backpropagation.
Claims 14 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Chen, Barnard, and Raut as applied to claims 13 and 17 above, and further in view of Shirahata (US 2020/0202222 A1; published Jun. 25, 2020).
Regarding claim 14, Chen, in view of Barnard and Raut, discloses the invention of claim 13 as discussed above. Chen does not expressly disclose wherein the second layer comprises a softmax layer to convert a plurality of second neuron outputs corresponding to the plurality of second neurons into a plurality of probabilities corresponding to the plurality of second neurons (but see Shirahata ¶ 49(“The softmax layer converts the variable generated by the fully-connected layer to probability. For example, the softmax layer models the firing by carrying out arithmetic operation of causing the neuron data for output to pass through the activation function σ for normalization.”)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Chen to incorporate the teachings of Shirahata to employ a softmax layer to convert the variable generated by the fully connected layer to probability, at least because doing so would enable generating a prediction.
Regarding claim 20, Chen, in view of Barnard and Raut, discloses the invention of claim 17 as discussed above. Chen further discloses wherein the trained neural network model comprises a first hidden layer of the plurality of neurons and a second layer, (Chen FIG. 4 (two hidden layers))
wherein the second layer comprises a plurality of second neurons, (Chen FIG. 4 (second layer includes multiple neurons))
wherein to remove the first neuron, the machine-executable instructions cause the electronic processor to:
remove a plurality of input mappings in the trained neural network model from a plurality of inputs to the first neuron; and (Chen ¶ 64 (“As shown in the right side of FIG. 4, the pruned neural network 4′ in the local learning system 2 is trained only by limited data source, collected in a specific environment, for example, home, office, classroom, and so on. However, even though the pruned neural network 4′ lacks some neurons or some links, it is still effective to learn and recognize objects or conditions in the specific environment, because the specific environment has less variety.”) (FIG. 4 is reproduced below:
PNG
media_image2.png
320
340
media_image2.png
Greyscale
(the pruned nodes 41’ have the input mapping removed))
remove a plurality of output mappings in the trained neural network model from the first neuron to the plurality of second neurons, and (Chen ¶ 64 (“As shown in the right side of FIG. 4, the pruned neural network 4′ in the local learning system 2 is trained only by limited data source, collected in a specific environment, for example, home, office, classroom, and so on. However, even though the pruned neural network 4′ lacks some neurons or some links, it is still effective to learn and recognize objects or conditions in the specific environment, because the specific environment has less variety.”) (FIG. 4 is reproduced below:
PNG
media_image2.png
320
340
media_image2.png
Greyscale
(the pruned nodes 41’ have the output mapping removed)).
Chen does not expressly disclose:
wherein the second layer comprises a softmax layer to convert a plurality of second neuron outputs corresponding to the plurality of second neurons into a plurality of probabilities corresponding to the plurality of second neurons (but see Shirahata ¶ 49(“The softmax layer converts the variable generated by the fully-connected layer to probability. For example, the softmax layer models the firing by carrying out arithmetic operation of causing the neuron data for output to pass through the activation function σ for normalization.”)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Chen to incorporate the teachings of Shirahata to employ a softmax layer to convert the variable generated by the fully connected layer to probability, at least because doing so would enable generating a prediction.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Crowley, Elliot J., et al. "A closer look at structured pruning for neural network compression." arXiv preprint arXiv:1810.04622 (2018).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SHAHID KHAN whose telephone number is (571)270-0419. The examiner can normally be reached M-F, 9-5 est.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Usmaan Saeed can be reached at (571)272-4046. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SHAHID K KHAN/Primary Examiner, Art Unit 2146
Prosecution Insights