DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Amendments
This Office Action is in response to the amendment filed on 05/04/2026.
Claims 1, 7, and 16-18 have been amended.
Claims 10-11 have been cancelled.
No new claims have been added.
The objections and rejections from the prior correspondence that are not restated herein are withdrawn.
Response to Arguments
Applicant's arguments filed on 05/04/2026 have been fully considered.
Applicant’s arguments regarding the 35 U.S.C. 101 rejections of the previous office action have been fully considered but are not persuasive. Applicant argues:
“The claims provide a specific improvement to computer functionality, specifically, to the training of neural networks. The Office Action has alleged that the batch normalization after dropout is merely " ... rearranging the order in which the mathematical calculations of dropout and batch normalization are performed." (Office Action, page 3.) However, the claims recite far more than a mere reordering. The claims recite a unified model architecture with parallel partial models trained with different dropout rates, where back propagation is performed as a whole to update parameters of the entire model, and where the hidden layer size is determined based on a positive correlation with the dropout rate. This specific combination of architectural and training features produces a concrete technical improvement, i.e., suppressing non-training targets from batch normalization during back propagation, thereby improving training efficiency. See, e.g., Ex parte Desjardins, Appeal 2024-000567 (ARP Sept. 26, 2025) (vacating § 101 rejection of machine learning training claims, finding that claims reciting specific improvements to how a machine learning model operates integrate any abstract idea into a practical application, and cautioning that examiners "should not evaluate claims at such a high level of generality" as to equate any machine learning with an unpatentable algorithm).
Under Step 2B, the specific combination of features recited in the amended claims – parallel partial models with different dropout rates, batch normalization after dropout, unified back propagation to update parameters of the model as a whole, and positive correlation-based hidden layer sizing - is not well-understood, routine, or conventional. The Examiner has not established that this specific combination was previously known.”
Examiner respectfully disagrees. The alleged technical improvement of “suppressing non-training targets from batch normalization during back propagation” simply describes the result of the mathematical concept of dropout, which is a technique performed to prevent the overfitting of the model during training, thus achieving improved training efficiency by improving the abstract idea itself. According to MPEP § 2106.05(a), the judicial exception alone cannot provide the improvement, and that improvement must be provided by one or more additional elements. A model structure having two partial models connected in parallel are mere instructions to apply the abstract idea using a particular model structure and is evaluated as an additional element under Step 2A Prong 2 and 2B. Further, Desjardins addressed catastrophic forgetting when training a model on sequential tasks by measuring each parameter's importance to a first task, then penalizing changes to the important parameters while training the model on a second task, thereby directly improving how machine learning models are trained when learning new tasks. The instant application does not address forgetting of the model during learning but rather uses dropout to suppress non-training targets from batch normalization, which is a technique to prevent overfitting. Moreover, for the independent claims, the examiner only rejected as well-understood, routine, or conventional the specific step of “acquiring/acquire learning data used for training of a model including a first partial model and a second partial model”, which is data gathering. Accordingly, the claims neither integrate the abstract idea into a practical application nor amount to significantly more than the judicial exception.
Applicant’s arguments regarding the 35 U.S.C. 103 rejections of the previous office action have been fully considered but are not persuasive. Applicant argues:
“First, Noguchi does not teach or suggest training partial models by dropout, let alone by dropout based on different dropout rates. Noguchi teaches a processing model M1 that includes partial models PM1, PM2, and PM3, each of which processes a different type of input information (e.g., images, titles, and body text). (Noguchi, par. [0118].) While Noguchi teaches correcting connection coefficients of the processing model M1 using backpropagation (Noguchi, par. [0105]), Noguchi is entirely silent on dropout, batch normalization, or any relationship between a dropout rate and hidden layer size. Noguchi therefore fails to teach or suggest the amended claim features requiring training partial models with different dropout rates, performing batch normalization after dropout, performing back propagation as a whole to update a parameter of a model in which parallel partial models are trained with different dropout rates, or a hidden layer having a size determined based on a positive correlation between the dropout rate and the hidden layer size.”
Examiner respectfully disagrees. Applicant’s arguments regarding NOGUCHI not teaching or suggesting training partial models by dropout, let alone by dropout based on different dropout rate, are moot because LI is relied on for teaching this limitation.
Applicant further argues:
“Second, Li does not teach or suggest applying different dropout rates to partial models within a single unified model. Li teaches training entirely separate, independent DNNs (DNN-0, DNN-1, ... DNN-N) with different dropout rates for a voting scheme in an autonomous driving system. (Li, par. [0019].) Li's DNNs are not partial models within a single model; rather, they are completely independent networks. The amended claims now explicitly require that "the first partial model and the second partial model are connected in parallel" within a single model, and that "back propagation is performed as a whole to update a parameter of the model." This unified training architecture is fundamentally different from Li's separate independent DNNs. There is no teaching or suggestion in Li to apply different dropout rates to parallel sub-components of a single model that are trained together via whole-model back propagation.”
Examiner respectfully disagrees. In response to applicant's arguments against the references individually, one cannot show nonobviousness by attacking references individually where the rejections are based on combinations of references (See In re Keller, 642 F.2d 413, 208 USPQ 871 (CCPA 1981); In re Merck & Co., 800 F.2d 1091, 231 USPQ 375 (Fed. Cir. 1986)). The combination of NOGUCHI and LI teaches applying different dropout rates to a plurality of neural networks. Further, LI [0019] teaches: “The plurality of DNNs 220 are then utilized with input data (camera images, LIDAR, etc.) to generate a plurality of outputs for an autonomous driving feature (e.g., object detection and tracking). The plurality of outputs could be collectively utilized by a voting scheme 224, which could determine a best output based on the plurality of outputs. This could be any suitable best output determination, such as selecting the most likely occurring output, an average of the outputs, or some combination thereof.” The different outputs resulting from using different dropout rates are collectively used by the voting scheme to determine the best output, which helps reach the desired level of accuracy and robustness when training the model.
Applicant further argues:
“Third, Xiang Li actually teaches away from combining dropout and batch normalization. Xiang Li's paper is titled "Understanding the Disharmony between Dropout and Batch Normalization by Variance Shift" and its central finding is that combining dropout and batch normalization causes "variance shift" that leads to "erroneous predictions" and "drops the performance." (Xiang Li, Abstract.) Xiang Li explicitly states that "the original Dropout and BN are not recommended to appear in the bottleneck part due to their variance shift conflict." (Xiang Li, Section 6, "Summary of Guidelines"). The stated motivation in the Office Action, "to achieve cooperation between dropout and batch normalization" (Office Action, p. 25), directly contradicts Xiang Li's actual teaching, which is that these techniques conflict and cause disharmony. A person of ordinary skill in the art reading Xiang Li would be discouraged from, not motivated to, combine dropout and batch normalization in the manner claimed.”
Examiner respectfully disagrees. Applicant cuts the quote short and mischaracterizes XIANG LI as teaching away from the combination. XIANG LI’s quote on page 2689, 6. Summary of Guidelines states: “In modern CNN architectures, the original Dropout and BN are not recommended to appear in the bottleneck part due to their variance shift conflict, except that we have a relatively large feature dimension.” XIANG LI [page 2688, 5. Strategy to Combine Them Better] teaches: “Since we get a clear knowledge about the disharmony between Dropout and BN, we can easily develop an approach to combine them together, to see whether an extra improvement can be obtained. In this section, we introduce one possible solution that slightly modifies the formula of Dropout and make it less sensitive to variance, which can alleviate the shift problem and stabilize the numerical behaviors. […] We name this form of Dropout as "Uout" for simplicity. […] We observe that "Uout" with larger ratios tends to perform favorably well, which indicates its superior stability.” XIANG LI [page 2684, 3. Theoretical Analyses] teaches: “In this design (i.e., BN directly follows Dropout), if we want to alleviate the variance shift risks, i.e.,
Δ
p
→
1
, the only thing we can do is to eliminate Dropout which means setting the Dropout retain ratio
p
→
1
. Fortunately, the architectures where Dropout brings benefits (e.g., in Wide ResNet) do not follow this type of arrangement. In fact, they adopt the case (b) in Fig. 2, which is more common in practice, and we will describe it in details as follows.” Accordingly, XIANG LI’s case (b) presents an arrangement in which using BN after dropout brings benefits using the Wide ResNet architecture.
Applicant further argues:
“Fourth, none of the cited references teaches or suggests "a hidden layer having a size determined based on a positive correlation between the second dropout rate and the size of the hidden layer" as now recited by amended claim 1. This feature is not taught by Noguchi, Li, or Xiang Li, and none of the other cited references cures this deficiency.
Claims 14-15 depend from claim 13, which depends from independent claim 1. As discussed above, the combination of Noguchi, Li, and Xiang Li fails to teach or suggest all of the features of claim 1 as amended. Additionally, as to the "positive correlation" feature now recited in claim 1, Srivastava teaches that dropout reduces the effective number of units (from n to pn), and suggests compensating by using n/p units. (Srivastava, Section A 1, "Network Size"). This is an inverse relationship; as the dropout rate increases (i.e., p decreases), the network needs more units to compensate for the capacity reduction. This is fundamentally different from the claimed positive correlation, where the hidden layer size is deliberately determined based on a positive correlation with the dropout rate. Accordingly, Applicant respectfully requests withdrawal of the rejection of claims 14-15.”
Examiner respectfully disagrees. SRIVASTAVA teaches this limitation and the relationship disclosed in SRIVASTAVA is not an inverse relationship. SRIVASTAVA [page 1950, section 9.1 Linear Regression] teaches: “For
p
close to 1, all the inputs are retained and the regularization constant is small. As more dropout is done (by decreasing
p
), […].” SRIVASTAVA [page 1952, section A.1 Network Size] teaches: "It is to be expected that dropping units will reduce the capacity of a neural network. If
n
is the number of hidden units in any layer and
p
is the probability of retaining a unit, then instead of
n
hidden units, only
p
n
units will be present after dropout, in expectation. Moreover, this set of
p
n
units will be different each time and the units are not allowed to build co-adaptations freely. Therefore, if an
n
-sized layer is optimal for a standard neural net on any given task, a good dropout net should have at least
n
p
units." Examiner’s note: SRIVASTAVA [page 1950, section 9.1 Linear Regression] teaches that more dropout is done by decreasing
p
. SRIVASTAVA [page 1952, section A.1 Network Size] teaches that a good dropout net should have at least
n
p
units, where
n
is the layer size. When
p
decreases (more dropout), the size of the layer increases, thus teaching a hidden layer having a size determined based on a positive correlation between the second dropout rate and the size of the hidden layer.
The dependent claims remain rejected based on rejections of their respective parent claims, and additionally based on the rejections shown in the 35 U.S.C. 103 rejections below.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(d):
(d) REFERENCE IN DEPENDENT FORMS.—Subject to subsection (e), a claim in dependent form shall contain a reference to a claim previously set forth and then specify a further limitation of the subject matter claimed. A claim in dependent form shall be construed to incorporate by reference all the limitations of the claim to which it refers.
The following is a quotation of pre-AIA 35 U.S.C. 112, fourth paragraph:
Subject to the following paragraph [i.e., the fifth paragraph of pre-AIA 35 U.S.C. 112], a claim in dependent form shall contain a reference to a claim previously set forth and then specify a further limitation of the subject matter claimed. A claim in dependent form shall be construed to incorporate by reference all the limitations of the claim to which it refers.
Claims 3 and 14-15 are rejected under 35 U.S.C. 112(d) or pre-AIA 35 U.S.C. 112, 4th paragraph, as being of improper dependent form for failing to further limit the subject matter of the claim upon which it depends, or for failing to include all the limitations of the claim upon which it depends.
Regarding Claim 3:
The claim depends on claim 1 and merely restates that the second partial model includes a hidden layer.
Regarding Claim 14:
The claim depends on claim 13, which further depends on claim 1. Claim 14 states “generating the model including the second partial model including a hidden layer based on the second dropout rate”. Claim 1 states “generating, by using the learning data, the model in a manner in which the first partial model is trained by first dropout based on a first dropout rate and the second partial model is trained by second dropout based on a second dropout rate different from the first dropout rate” and “wherein the second partial model includes a hidden layer having a size determined based on a positive correlation between the second dropout rate and the size of the hidden layer”. Claim 14 merely restates that the second partial model includes a hidden layer and is generated based on the second dropout rate.
Regarding Claim 15:
The claim depends on claim 14. Claim 14 depends on claim 13, which further depends on claim 1. Claim 15 states “generating the model including the second partial model including a hidden layer having a size determined based on the second dropout rate.” Claim 1 states “generating, by using the learning data, the model in a manner in which the first partial model is trained by first dropout based on a first dropout rate and the second partial model is trained by second dropout based on a second dropout rate different from the first dropout rate” and “wherein the second partial model includes a hidden layer having a size determined based on a positive correlation between the second dropout rate and the size of the hidden layer”. Claim 15 merely restates that the second partial model includes a hidden layer having a size determined based on the second dropout rate.
Applicant may cancel the claim(s), amend the claim(s) to place the claim(s) in proper dependent form, rewrite the claim(s) in independent form, or present a sufficient showing that the dependent claim(s) complies with the statutory requirements.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-9 and 12-18 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1: Claims 1-9 and 12-15 are directed to a process. Claims 16-18 are directed to a machine or an article of manufacture.
With respect to claims 1, 16, 17, and 18:
2A Prong 1: The claims recite an abstract idea. Specifically:
(Claims 1, 16, and 17) generating/generate […] the model […]
(Claim 18) the model being trained […]
[…] by using the learning data, the model in a manner in which the first partial model is trained by first dropout based on a first dropout rate and the second partial model is trained by second dropout based on a second dropout rate different from the first dropout rate, (Mathematical concepts – generating/training a model based on a dropout rate involves mathematical calculations (see [0162]) – see MPEP § 2106.04(a)(2)(I))
(Claims 1 and 17) wherein performing batch normalization after the first dropout and after the second dropout suppresses nodes that are not activated by the respective dropout from being subjected to batch normalization during back propagation, thereby suppressing non-training targets from being subjected to batch normalization, (Mathematical concepts – performing batch normalization after a dropout suppresses nodes involves mathematical calculations (see [0162], [0148-0153], and [FIG.10]) – see MPEP § 2106.04(a)(2)(I))
wherein back propagation is performed as a whole to update a parameter of the model, (Mathematical concepts – Performing backpropagation to update parameters involves mathematical calculations – see MPEP § 2106.04(a)(2)(I))
wherein the second partial model includes a hidden layer having a size determined based on a positive correlation between the second dropout rate and the size of the hidden layer. (Mathematical concepts – Determining a layer size based on a positive correlation involves the use of a mathematical relationship – see MPEP § 2106.04(a)(2)(I))
(Claim 16) wherein performing batch normalization after the first dropout and after the second dropout suppresses nodes that are not activated by the respective dropout from being subjected to batch normalization during back propagation, thereby improving training efficiency. (Mathematical concepts – performing batch normalization after a dropout suppresses nodes involves mathematical calculations (see [0162], [0148-0153], and [FIG.10]) – see MPEP § 2106.04(a)(2)(I))
(Claim 18) wherein performing batch normalization after the dropout for the first partial model and after the dropout for the second partial model suppresses nodes that are not activated by the respective dropout from being subjected to batch normalization during back propagation. (Mathematical concepts – performing batch normalization after a dropout suppresses nodes involves mathematical calculations (see [0162], [0148-0153], and [FIG.10]) – see MPEP § 2106.04(a)(2)(I))
If claim limitations, under their broadest reasonable interpretation, cover performance of the limitations as a mental process, but for the recitation of generic computer components, then the claim limitations fall within the mathematical or mental process grouping of abstract ideas. Accordingly, the claim “recites” an abstract idea.
2A Prong 2: The additional elements recited in the claims do not integrate the abstract idea into a practical application, individually or in combination.
Additional elements:
(Claim 1) An information processing method executed by a computer, the information processing method comprising (Mere instructions to apply an abstract idea on a computer or merely uses a computer as a tool to perform an abstract idea – see MPEP § 2106.05(f).)
(Claim 16) An information processing apparatus comprising: a processor; (Mere recitation of a generic computer component – see MPEP § 2106.05(b)(I))
(Claim 16) a memory storing instructions that, when executed by the processor, cause the processor to: (Mere instructions to apply an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea – see MPEP § 2106.05(f).)
(Claims 1, 16, and 17) acquiring/acquire learning data used for training of a model including a first partial model and a second partial model; (Mere data gathering – Adding insignificant extra-solution activity of mere data gathering to the judicial exception – see MPEP § 2106.05(g).)
wherein the first partial model and the second partial model are connected in parallel; (Mere instructions to apply an abstract idea on a computer or merely uses a computer as a tool to perform an abstract idea – see MPEP § 2106.05(f).)
(Claim 17) A non-transitory computer-readable storage medium having stored therein an information processing program for causing a computer to execute: (Mere instructions to apply an abstract idea on a computer or merely uses a computer as a tool to perform an abstract idea – see MPEP § 2106.05(f).)
(Claim 18) A non-transitory computer-readable storage medium having stored therein an information processing program for causing a computer to be operated as a model including a first partial model and a second partial model, (Mere instructions to apply an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea – see MPEP § 2106.05(f).)
Since the claim as a whole, looking at the additional elements individually and in combination, does not contain any other additional elements that are indicative of integration into a practical application, the claim is directed to an abstract idea.
2B: The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Additional elements:
(Claim 1) An information processing method executed by a computer, the information processing method comprising (Mere instructions to apply an abstract idea on a computer or merely uses a computer as a tool to perform an abstract idea – see MPEP § 2106.05(f).)
(Claim 16) An information processing apparatus comprising: a processor; (Mere recitation of a generic computer component – see MPEP § 2106.05(b)(I))
(Claim 16) and a memory storing instructions that, when executed by the processor, cause the processor to: (Mere instructions to apply an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea – see MPEP § 2106.05(f).)
(Claims 1, 16, and 17) acquiring/acquire learning data used for training of a model including a first partial model and a second partial model; (Simply appending well-understood, routine, conventional activities previously known to the industry, specified at a high level of generality, to the judicial exception (WURC) - see MPEP § 2106.05(d)(ll)(i) - Receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 (utilizing an intermediary computer to forward information).)
wherein the first partial model and the second partial model are connected in parallel; (Mere instructions to apply an abstract idea on a computer or merely uses a computer as a tool to perform an abstract idea – see MPEP § 2106.05(f).)
(Claim 17) A non-transitory computer-readable storage medium having stored therein an information processing program for causing a computer to execute: (Mere instructions to apply an abstract idea on a computer or merely uses a computer as a tool to perform an abstract idea – see MPEP § 2106.05(f).)
(Claim 18) A non-transitory computer-readable storage medium having stored therein an information processing program for causing a computer to be operated as a model including a first partial model and a second partial model, (Mere instructions to apply an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea – see MPEP § 2106.05(f).)
Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
With respect to claim 2:
2A Prong 2: The additional elements recited in the claim do not integrate the abstract idea into a practical application, individually or in combination.
Additional elements:
wherein the second partial model includes a larger number of layers than the first partial model. (Mere instructions to apply an abstract idea on a computer or merely uses a computer as a tool to perform an abstract idea – see MPEP § 2106.05(f).)
2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Additional elements:
wherein the second partial model includes a larger number of layers than the first partial model. (Mere instructions to apply an abstract idea on a computer or merely uses a computer as a tool to perform an abstract idea – see MPEP § 2106.05(f).)
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, or provide significantly more than the judicial exception, the claim is not patent eligible.
With respect to claim 3:
2A Prong 2: The additional elements recited in the claim do not integrate the abstract idea into a practical application, individually or in combination.
Additional elements:
wherein the second partial model includes a hidden layer. (Mere instructions to apply an abstract idea on a computer or merely uses a computer as a tool to perform an abstract idea – see MPEP § 2106.05(f).)
2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Additional elements:
wherein the second partial model includes a hidden layer. (Mere instructions to apply an abstract idea on a computer or merely uses a computer as a tool to perform an abstract idea – see MPEP § 2106.05(f).)
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, or provide significantly more than the judicial exception, the claim is not patent eligible.
With respect to claim 4:
2A Prong 2: The additional elements recited in the claim do not integrate the abstract idea into a practical application, individually or in combination.
Additional elements:
wherein the model includes an input layer to which the learning data is input, and an output from the input layer is input to each of the first partial model and the second partial model. (Mere instructions to apply an abstract idea on a computer or merely uses a computer as a tool to perform an abstract idea – see MPEP § 2106.05(f).)
2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Additional elements:
wherein the model includes an input layer to which the learning data is input, and an output from the input layer is input to each of the first partial model and the second partial model. (Mere instructions to apply an abstract idea on a computer or merely uses a computer as a tool to perform an abstract idea – see MPEP § 2106.05(f).)
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, or provide significantly more than the judicial exception, the claim is not patent eligible.
With respect to claim 5:
2A Prong 2: The additional elements recited in the claim do not integrate the abstract idea into a practical application, individually or in combination.
Additional elements:
wherein the first partial model includes a first embedding layer in which an input from the input layer is embedded, and the second partial model includes a second embedding layer in which an input from the input layer is embedded. (Mere instructions to apply an abstract idea on a computer or merely uses a computer as a tool to perform an abstract idea – see MPEP § 2106.05(f).)
2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Additional elements:
wherein the first partial model includes a first embedding layer in which an input from the input layer is embedded, and the second partial model includes a second embedding layer in which an input from the input layer is embedded. (Mere instructions to apply an abstract idea on a computer or merely uses a computer as a tool to perform an abstract idea – see MPEP § 2106.05(f).)
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, or provide significantly more than the judicial exception, the claim is not patent eligible.
With respect to claim 6:
2A Prong 1: The claim recites an abstract idea. Specifically:
wherein the model includes a combining layer that combines an output from the first partial model and an output from the second partial model. (Mathematical concepts – combining outputs from a first and second partial model in a combining layer involves mathematical calculations. The combining layer includes a processing layer EL31 that combines the outputs of the partial model PM1 and the outputs of the partial model PM2, and calculates an average of these outputs (see [0132]) – see MPEP § 2106.04(a)(2)(I))
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, or provide significantly more than the judicial exception, the claim is not patent eligible.
With respect to claim 7:
2A Prong 2: The additional elements recited in the claim do not integrate the abstract idea into a practical application, individually or in combination.
Additional elements:
wherein the first partial model includes a first output layer whose output is input to the combining layer, and the second partial model includes a second output layer whose output is input to the combining layer. (Mere instructions to apply an abstract idea on a computer or merely uses a computer as a tool to perform an abstract idea – see MPEP § 2106.05(f).)
2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Additional elements:
wherein the first partial model includes a first output layer whose output is input to the combining layer, and the second partial model includes a second output layer whose output is input to the combining layer. (Mere instructions to apply an abstract idea on a computer or merely uses a computer as a tool to perform an abstract idea – see MPEP § 2106.05(f).)
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, or provide significantly more than the judicial exception, the claim is not patent eligible.
With respect to claim 8:
2A Prong 2: The additional elements recited in the claim do not integrate the abstract idea into a practical application, individually or in combination.
Additional elements:
wherein the combining layer includes a softmax layer. (Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer or merely uses a computer as a tool to perform an abstract idea – see MPEP § 2106.05(f).)
2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Additional elements:
wherein the combining layer includes a softmax layer. (Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer or merely uses a computer as a tool to perform an abstract idea – see MPEP § 2106.05(f).)
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, or provide significantly more than the judicial exception, the claim is not patent eligible.
With respect to claim 9:
2A Prong 1: The claim recites an abstract idea. Specifically:
wherein the combining layer performs combining processing for the output of the first partial model and the output of the second partial model before the softmax layer. (Mathematical concepts – Combining processing involves mathematical calculations (see [0116]) – see MPEP § 2106.04(a)(2))
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, or provide significantly more than the judicial exception, the claim is not patent eligible.
With respect to claim 12:
2A Prong 2: The additional elements recited in the claim do not integrate the abstract idea into a practical application, individually or in combination.
Additional elements:
acquiring information indicating the first dropout rate, and (Mere data gathering – Adding insignificant extra-solution activity of mere data gathering to the judicial exception – see MPEP § 2106.05(g).)
generating the model including the first partial model having a size based on the first dropout rate. (Mere instructions to apply an abstract idea on a computer or merely uses a computer as a tool to perform an abstract idea – see MPEP § 2106.05(f).)
2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Additional elements:
acquiring information indicating the first dropout rate, and (Simply appending well-understood, routine, conventional activities previously known to the industry, specified at a high level of generality, to the judicial exception (WURC) - see MPEP § 2106.05(d)(ll)(i) - Receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 (utilizing an intermediary computer to forward information).)
generating the model including the first partial model having a size based on the first dropout rate. (Mere instructions to apply an abstract idea on a computer or merely uses a computer as a tool to perform an abstract idea – see MPEP § 2106.05(f).)
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, or provide significantly more than the judicial exception, the claim is not patent eligible.
With respect to claim 13:
2A Prong 2: The additional elements recited in the claim do not integrate the abstract idea into a practical application, individually or in combination.
Additional elements:
acquiring information indicating the second dropout rate, and (Mere data gathering – Adding insignificant extra-solution activity of mere data gathering to the judicial exception – see MPEP § 2106.05(g).)
generating the model including the second partial model having a size based on the second dropout rate. (Mere instructions to apply an abstract idea on a computer or merely uses a computer as a tool to perform an abstract idea – see MPEP § 2106.05(f).)
2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Additional elements:
acquiring information indicating the second dropout rate, and (Simply appending well-understood, routine, conventional activities previously known to the industry, specified at a high level of generality, to the judicial exception (WURC) - see MPEP § 2106.05(d)(ll)(i) - Receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 (utilizing an intermediary computer to forward information).)
generating the model including the second partial model having a size based on the second dropout rate. (Mere instructions to apply an abstract idea on a computer or merely uses a computer as a tool to perform an abstract idea – see MPEP § 2106.05(f).)
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, or provide significantly more than the judicial exception, the claim is not patent eligible.
With respect to claim 14:
2A Prong 2: The additional elements recited in the claim(s) do not integrate the abstract idea into a practical application, individually or in combination.
Additional elements:
generating the model including the second partial model including a hidden layer based on the second dropout rate. (Mere instructions to apply an abstract idea on a computer or merely uses a computer as a tool to perform an abstract idea – see MPEP § 2106.05(f).)
2B: The claim(s) does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Additional elements:
generating the model including the second partial model including a hidden layer based on the second dropout rate. (Mere instructions to apply an abstract idea on a computer or merely uses a computer as a tool to perform an abstract idea – see MPEP § 2106.05(f).)
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, or provide significantly more than the judicial exception, the claim is not patent eligible.
With respect to claim 15:
2A Prong 2: The additional elements recited in the claim(s) do not integrate the abstract idea into a practical application, individually or in combination.
Additional elements:
generating the model including the second partial model including a hidden layer having a size determined based on the second dropout rate. (Mere instructions to apply an abstract idea on a computer or merely uses a computer as a tool to perform an abstract idea – see MPEP § 2106.05(f).)
2B: The claim(s) does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Additional elements:
generating the model including the second partial model including a hidden layer having a size determined based on the second dropout rate. (Mere instructions to apply an abstract idea on a computer or merely uses a computer as a tool to perform an abstract idea – see MPEP § 2106.05(f).)
Since the claim does not recite additional elements that either integrate the judicial exception into a practical application, or provide significantly more than the judicial exception, the claim is not patent eligible.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 3-4, 6-7, and 12-18 are rejected under 35 U.S.C. 103 as being unpatentable over NOGUCHI (US 20190005399 A1) in view of LI (US 20220044033 A1), XIANG LI ("Understanding the Disharmony between Dropout and Batch Normalization by Variance Shift"), and SRIVASTAVA (“Dropout: A Simple Way to Prevent Neural Networks from Overfitting”), hereinafter NOGUCHI, LI, XIANG LI, and SRIVASTAVA, respectively.
Regarding Claim 1:
NOGUCHI teaches:
An information processing method executed by a computer, the information processing method comprising: (NOGUCHI [0020] teaches: "The information providing device 10 is an information processing device that executes learning processing described later".)
acquiring learning data used for training of a model including a first partial model and a second partial model, (NOGUCHI [0059] teaches: "Next, the information providing device 10 acquires learning data used for learning (i.e., for training) the processing model (i.e., of a model) from the data server 50 (Step S2)." NOGUCHI [0118] teaches: "[…] the processing model M1 includes a partial model PM1 (i.e., including a first partial model). […] The processing model M1 also includes a partial model PM2 (i.e., including a second partial model) [...].")
wherein the first partial model and the second partial model are connected in parallel; (NOGUCHI [FIG. 5] shows PM1 and PM2 (i.e., the first partial model and the second partial model) connected side by side (i.e., connected in parallel) to the synthesis model.)
wherein back propagation is performed as a whole to update a parameter of the model, (NOGUCHI [0118] teaches: "[…] the processing model M1 includes a partial model PM1 (i.e., including a first partial model). […] The processing model M1 also includes a partial model PM2 (i.e., including a second partial model) [...]." NOGUCHI [0105] teaches: “The learning unit 42 corrects the connection coefficient of the processing model M1 based on a difference between the output information that is actually output by the processing model M1 and the output information that is expected to be output from the input information. For example, the learning unit 42 may correct the connection coefficient using a method such as backpropagation.” Examiner’s note: Processing model M1 includes both partial model PM1 and partial model PM2, as shown in NOGUCHI’s FIG. 5. Further, learning unit 42 corrects the connection coefficient of M1 by using backpropagation (i.e., back propagation is performed as a whole to update a parameter of the model).)
[…] the second partial model […] (NOGUCHI [0118] teaches: "The processing model M1 also includes a partial model PM2 (i.e., the second partial model) [...].")
NOGUCHI is not relied upon for teaching:
generating, by using the learning data, the model in a manner in which the first partial model is trained by first dropout based on a first dropout rate and the second partial model is trained by second dropout based on a second dropout rate different from the first dropout rate,
wherein performing batch normalization after the first dropout and after the second dropout suppresses nodes that are not activated by the respective dropout from being subjected to batch normalization during back propagation, thereby suppressing non-training targets from being subjected to batch normalization,
wherein the second partial model includes a hidden layer having a size determined based on a positive correlation between the second dropout rate and the size of the hidden layer.
However, LI teaches: generating, by using the learning data, the model in a manner in which the first partial model is trained by first dropout based on a first dropout rate and the second partial model is trained by second dropout based on a second dropout rate different from the first dropout rate, […] (LI [0003] teaches: “obtain a training dataset for training (i.e., by using the learning data) a plurality of deep neural networks (DNNs) for the autonomous driving feature […].” LI [0009] teaches: "In some implementations, training the plurality of DNNs comprises using a different dropout rate for each of the plurality of DNNs." LI [0019] teaches: “The plurality of DNNs 220 are then utilized with input data (camera images, LIDAR, etc.) to generate a plurality of outputs for an autonomous driving feature (e.g., object detection and tracking). The plurality of outputs could be collectively utilized by a voting scheme 224, which could determine a best output based on the plurality of outputs. This could be any suitable best output determination, such as selecting the most likely occurring output, an average of the outputs, or some combination thereof.” Examiner's note: under BRI, the first dropout rate and second dropout rate can be interpreted as using different dropout rates for each of the plurality of DNNs. Additionally, generating, […] the model in a manner in which the first partial model is trained by can be reasonably interpreted as training the plurality of DNNs (i.e., first/second partial models) using different dropout rates.)
Accordingly, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention, having the teachings of NOGUCHI and LI before them, to apply LI’s different dropout rates in NOGUCHI’s partial models in the information providing device. One would have been motivated to make such a combination in order to reach the desired level of accuracy and robustness when training the neural networks (i.e., models) (LI [0002], [0005], and [0015]).
NOGUCHI in view of LI is not relied upon for teaching:
wherein performing batch normalization after the first dropout and after the second dropout suppresses nodes that are not activated by the respective dropout from being subjected to batch normalization during back propagation, thereby suppressing non-training targets from being subjected to batch normalization,
wherein the second partial model includes a hidden layer having a size determined based on a positive correlation between the second dropout rate and the size of the hidden layer.
However, XIANG LI teaches: wherein performing batch normalization after the first dropout and after the second dropout suppresses nodes that are not activated by the respective dropout from being subjected to batch normalization during back propagation, thereby suppressing non-training targets from being subjected to batch normalization, (XIANG LI [page 2682, Figure 1. Up] teaches performing batch normalization after dropout. Furthermore, XIANG LI [page 2684, 3. Theoretical Analyses] teaches: “Therefore, we only need to consider the cases where Dropout comes before BN.” Furthermore, XIANG LI [page 2683, 2. Related Work and Preliminaries] teaches that dropout can be interpreted as a way of regularizing a neural network by adding noise to its hidden units (i.e., nodes) by multiplying hidden activations of hidden units by Bernoulli distributed random variables which take the value 1 with probability
p
(
0
≤
p
≤
1
)
and 0 otherwise (i.e., suppresses nodes). Hidden units (i.e., nodes) that are multiplied by a Bernoulli random variable with a value of 0 will result in a 0 for that hidden unit. Under broadest reasonable interpretation, suppresses nodes can be interpreted as the hidden unit being multiplied by the Bernoulli random variable with a value of 0, which makes the activation for that hidden unit (i.e., node) equal to 0 by the dropout (i.e., nodes that are not activated by the respective dropout). Additionally, XIANG LI [page 2683, 2. Related Work and Preliminaries] also teaches in eq. (2) that
μ
and
σ
2
(see also Figure 1) participate in the backpropagation. When the value for the instance in the mini-batch is processed by a hidden unit multiplied by a Bernoulli random variable of 0, the instance
x
i
…
m
will not end up participating in the backpropagation (i.e., during back propagation), and thus suppressed (i.e., thereby suppressing non-training targets from being subjected to batch normalization).)
Accordingly, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention, having the teachings of NOGUCHI, LI, and XIANG LI before them, to include XIANG LI’s performing batch normalization after dropout in each partial model in NOGUCHI and LI’s information processing device. One would have been motivated to make such a combination in order to track the accuracy of the model as it trains and to achieve cooperation between dropout and batch normalization for regularizing neural networks with a very large feature dimension (XIANG LI [page 2682, Abstract], [page 2682, 1. Introduction], [page 2683, 2. Related Work and Preliminaries], [page 2684, 3. Theoretical Analyses], and [page 2689, 6. Summary of Guidelines]).
NOGUCHI in view of LI and XIANG LI is not relied upon for teaching, but SRIVASTAVA teaches: wherein the […] model includes a hidden layer having a size determined based on a positive correlation between the second dropout rate and the size of the hidden layer. (SRIVASTAVA [page 1950, section 9.1 Linear Regression] teaches: “For
p
close to 1, all the inputs are retained and the regularization constant is small. As more dropout is done (by decreasing
p
), […].” SRIVASTAVA [page 1952, section A.1 Network Size] teaches: "It is to be expected that dropping units will reduce the capacity of a neural network. If
n
is the number of hidden units in any layer and
p
is the probability of retaining a unit, then instead of
n
hidden units, only
p
n
units will be present after dropout, in expectation. Moreover, this set of
p
n
units will be different each time and the units are not allowed to build co-adaptations freely. Therefore, if an
n
-sized layer is optimal for a standard neural net on any given task, a good dropout net should have at least
n
p
units." Examiner’s note: SRIVASTAVA [page 1950, section 9.1 Linear Regression] teaches that more dropout is done by decreasing
p
. SRIVASTAVA [page 1952, section A.1 Network Size] teaches that a good dropout net should have at least
n
p
units, where
n
is the layer size. When
p
decreases (more dropout), the size of the layer increases, thus teaching a hidden layer having a size determined based on a positive correlation between the second dropout rate and the size of the hidden layer.)
Accordingly, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention, having the teachings of NOGUCHI, LI, XIANG LI, and SRIVASTAVA before them, to include SRIVASTAVA’s hidden layer generation based on dropout rates in NOGUCHI, LI, and XIANG LI’s information providing device. One would have been motivated to make such a combination because using “dropout improved generalization performance on all data sets compared to neural networks that did not use dropout”, hence leading to a better performing model (SRIVASTAVA [page 1935, section 6. Experimental Results]).
Regarding Claim 3:
NOGUCHI in view of LI, XIANG LI, and SRIVASTAVA teaches the elements of claim 1 as outlined above. NOGUCHI further teaches:
[…] the second partial model […] (NOGUCHI [0118] teaches: "The processing model M1 also includes a partial model PM2 (i.e., the second partial model) [...].")
SRIVASTAVA further teaches: wherein the […] model includes a hidden layer. (SRIVASTAVA [page 1950, section 9.1 Linear Regression] teaches: “For
p
close to 1, all the inputs are retained and the regularization constant is small. As more dropout is done (by decreasing
p
), […].” SRIVASTAVA [page 1952, section A.1 Network Size] teaches: "It is to be expected that dropping units will reduce the capacity of a neural network. If
n
is the number of hidden units in any layer (i.e., the […] model includes a hidden layer) and
p
is the probability of retaining a unit, then instead of
n
hidden units, only
p
n
units will be present after dropout, in expectation. Moreover, this set of
p
n
units will be different each time and the units are not allowed to build co-adaptations freely. Therefore, if an
n
-sized layer is optimal for a standard neural net on any given task, a good dropout net should have at least
n
p
units.")
Regarding Claim 4:
NOGUCHI in view of LI, XIANG LI, and SRIVASTAVA teaches the elements of claim 1 as outlined above. NOGUCHI further teaches:
wherein the model includes an input layer to which the learning data is input, and an output from the input layer is input to each of the first partial model and the second partial model. (NOGUCHI [Fig. 1] teaches pieces of input information, such as image and writing, that are input into the partial models. Under BRI, the input layer can be interpreted as the image layer and writing layer in Fig. 1, which receive data from the learning data, to input the data into each of the partial models (i.e., first/second partial model).)
Regarding Claim 6:
NOGUCHI in view of LI, XIANG LI, and SRIVASTAVA teaches the elements of claim 1 as outlined above. NOGUCHI further teaches:
wherein the model includes a combining layer that combines an output from the first partial model and an output from the second partial model. (NOGUCHI [Fig. 5] teaches a synthesis model SM1 that combines the outputs of each partial model (e.g., PM1, PM2, PM3) in the model M1. Therefore, under BRI, the combining layer can be interpreted as the synthesis model SM1.)
Regarding Claim 7:
NOGUCHI in view of LI, XIANG LI, and SRIVASTAVA teaches the elements of claim 6 as outlined above. NOGUCHI further teaches:
wherein the first partial model includes a first output layer whose output is input to the combining layer, and the second partial model includes a second output layer whose output is input to the combining layer. (NOGUCHI [Fig. 5] teaches a synthesis model SM1 that combines the outputs of each partial model (e.g., PM1, PM2, PM3) in the model M1. Under BRI, the output layer of each partial model can be interpreted as the layer containing the output characteristic information of each partial model.)
Regarding Claim 12:
NOGUCHI in view of LI, XIANG LI, and SRIVASTAVA teaches the elements of claim 1 as outlined above. LI further teaches:
acquiring information indicating the first dropout rate, (LI [0019] teaches: "For example only, the dropout rate for DNN-0 220-0 could be 0.2, the dropout rate for DNN 220-1 could be 0.4, and the dropout rate for DNN-N 220 -N (e.g., for N=3) could be 0.6.")
generating the model including the first partial model having a size based on the first dropout rate. (LI [0019] teaches: "In other words, the higher the dropout rate, the more connections are removed, and the model is smaller." LI [0009] teaches: "In some implementations, training the plurality of DNNs comprises using a different dropout rate for each of the plurality of DNNs." Examiner’s note: Under BRI, “generating the model” can be interpreted as training the plurality of DNNs using different dropout rates.)
Regarding Claim 13:
NOGUCHI in view of LI, XIANG LI, and SRIVASTAVA teaches the elements of claim 1 as outlined above. LI further teaches:
acquiring information indicating the second dropout rate, and (LI [0019] teaches: "For example only, the dropout rate for DNN-0 220-0 could be 0.2, the dropout rate for DNN 220-1 could be 0.4, and the dropout rate for DNN-N 220 -N (e.g., for N=3) could be 0.6.")
generating the model including the second partial model having a size based on the second dropout rate. (LI [0019] teaches: "In other words, the higher the dropout rate, the more connections are removed, and the model is smaller." LI [0009] teaches: "In some implementations, training the plurality of DNNs comprises using a different dropout rate for each of the plurality of DNNs." Examiner’s note: Under BRI, “generating the model” can be interpreted as training the plurality of DNNs using different dropout rates.)
Regarding Claim 14:
NOGUCHI in view of LI, XIANG LI, and SRIVASTAVA teaches the elements of claim 13 as outlined above. LI further teaches:
generating the model including the second partial model […] based on the second dropout rate (LI [0019] teaches: "For example only, the dropout rate for DNN-0 220-0 could be 0.2, the dropout rate for DNN 220-1 could be 0.4, and the dropout rate for DNN-N 220 -N (e.g., for N=3) could be 0.6." LI [0019] teaches: "In other words, the higher the dropout rate, the more connections are removed, and the model is smaller." LI [0009] teaches: "In some implementations, training the plurality of DNNs comprises using a different dropout rate for each of the plurality of DNNs." Examiner’s note: Under BRI, “generating the model” can be interpreted as training the plurality of DNNs using different dropout rates.)
SRIVASTAVA further teaches: generating the model […] including a hidden layer based on the […] dropout rate. (SRIVASTAVA [page 1952, section A.1 Network Size] teaches: "It is to be expected that dropping units will reduce the capacity of a neural network. If
n
is the number of hidden units in any layer and
p
is the probability of retaining a unit, then instead of
n
hidden units, only
p
n
units will be present after dropout, in expectation. Moreover, this set of
p
n
units will be different each time and the units are not allowed to build co-adaptations freely. Therefore, if an
n
-sized layer is optimal for a standard neural net on any given task, a good dropout net should have at least
n
/
p
units.")
Regarding Claim 15:
NOGUCHI in view of LI, XIANG LI, and SRIVASTAVA teaches the elements of claim 14 as outlined above. LI further teaches:
generating the model including the second partial model […] based on the second dropout rate (LI [0019] teaches: "For example only, the dropout rate for DNN-0 220-0 could be 0.2, the dropout rate for DNN 220-1 could be 0.4, and the dropout rate for DNN-N 220 -N (e.g., for N=3) could be 0.6." LI [0019] teaches: "In other words, the higher the dropout rate, the more connections are removed, and the model is smaller." LI [0009] teaches: "In some implementations, training the plurality of DNNs comprises using a different dropout rate for each of the plurality of DNNs." Examiner’s note: Under BRI, “generating the model” can be interpreted as training the plurality of DNNs using different dropout rates.)
SRIVASTAVA further teaches: generating the model […] including a hidden layer having a size determined based on the […] dropout rate. (SRIVASTAVA [page 1952, section A.1 Network Size] teaches: "It is to be expected that dropping units will reduce the capacity of a neural network. If
n
is the number of hidden units in any layer and
p
is the probability of retaining a unit, then instead of
n
hidden units, only
p
n
units will be present after dropout, in expectation. Moreover, this set of
p
n
units will be different each time and the units are not allowed to build co-adaptations freely. Therefore, if an
n
-sized layer is optimal for a standard neural net on any given task, a good dropout net should have at least
n
p
units." Examiner’s note: SRIVASTAVA [page 1950, section 9.1 Linear Regression] teaches that more dropout is done by decreasing
p
. SRIVASTAVA [page 1952, section A.1 Network Size] teaches that a good dropout net should have at least
n
p
units, where
n
is the layer size. Therefore, the size of any layer
n
is determined by the amount of dropout performed.)
Regarding Claim 16:
The claim recites similar limitations as corresponding claim 1 and is rejected for similar reasons as claim 1 using similar teachings and rationale. Additionally, NOGUCHI further teaches:
a processor; and a memory storing instructions that, when executed by the processor, cause the processor to: (NOGUCHI [0099] teaches: “The control unit 40 is a controller, and is implemented when various programs stored in a storage device inside the information providing device 10 are executed by a processor such as a central processing unit (CPU) and a micro processing unit (MPU) using a RAM and the like as a working area, for example.”)
However, NOGUCHI is not relied upon for teaching, but XIANG LI teaches: wherein performing batch normalization after the first dropout and after the second dropout suppresses nodes that are not activated by the respective dropout from being subjected to batch normalization during back propagation, thereby improving training efficiency. (XIANG LI [page 2682, Figure 1. Up] teaches performing batch normalization after […] dropout. Furthermore, XIANG LI [page 2684, 3. Theoretical Analyses] teaches: “Therefore, we only need to consider the cases where Dropout comes before BN.” Furthermore, XIANG LI [page 2683, 2. Related Work and Preliminaries] teaches that dropout can be interpreted as a way of regularizing a neural network by adding noise to its hidden units (i.e., nodes) by multiplying hidden activations of hidden units by Bernoulli distributed random variables which take the value 1 with probability
p
(
0
≤
p
≤
1
)
and 0 otherwise (i.e., suppresses nodes). Hidden units (i.e., nodes) that are multiplied by a Bernoulli random variable with a value of 0 will result in a 0 for that hidden unit. Under broadest reasonable interpretation, suppresses nodes can be interpreted as the hidden unit being multiplied by the Bernoulli random variable with a value of 0, which makes the activation for that hidden unit (i.e., node) equal to 0 by the dropout (i.e., nodes that are not activated by the respective dropout). Additionally, XIANG LI [page 2683, 2. Related Work and Preliminaries] also teaches in eq. (2) that
μ
and
σ
2
(see also Figure 1) participate in the backpropagation. When the value for the instance in the mini-batch is processed by a hidden unit multiplied by a Bernoulli random variable of 0, the instance
x
i
…
m
will not end up participating in the backpropagation (i.e., during back propagation), and thus suppressed. Moreover, XIANG LI [page 2683, 2. Related Work and Preliminaries] also teaches that the normalization of activations that depends on the mini-batch allows efficient training (i.e., performing batch normalization after dropout […] thereby improving training efficiency).)
Accordingly, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention, having the teachings of NOGUCHI, LI, and XIANG LI before them, to include XIANG LI’s performing batch normalization after dropout in each partial model in NOGUCHI and LI’s information processing device. One would have been motivated to make such a combination in order to track the accuracy of the model as it trains and to achieve cooperation between dropout and batch normalization for regularizing neural networks with a very large feature dimension (XIANG LI [page 2682, Abstract], [page 2682, I. Introduction], [page 2683, 2. Related Work and Preliminaries], [page 2684, 3. Theoretical Analyses], and [page 2689, 6. Summary of Guidelines]).
Regarding Claim 17:
The claim recites similar limitations as corresponding claim 1 and is rejected for similar reasons as claim 1 using similar teachings and rationale. NOGUCHI further teaches:
A non-transitory computer-readable storage medium having stored therein an information processing program for causing a computer to execute: (NOGUCHI [0002] teaches: "[0002] The present invention relates to a learning device, a generation device, a learning method, a generation method, and a non-transitory computer readable storage medium." NOGUCHI [Claim 15] teaches: "A non-transitory computer-readable storage medium having stored therein a learning program that causes a computer to execute a process comprising.")
Regarding Claim 18:
The claim recites similar limitations as corresponding claims 1 and 17 and is rejected for similar reasons as claims 1 and 17 using similar teachings and rationale.
Claims 2 and 8-9 are rejected under 35 U.S.C. 103 as being unpatentable over NOGUCHI in view of LI, XIANG LI, and SRIVASTAVA, as applied to claim 1 above, and further in view of POUDEL (US 20200134772 A1), hereinafter POUDEL.
Regarding Claim 2:
NOGUCHI in view of LI, XIANG LI, and SRIVASTAVA teaches the elements of claim 1 as outlined above. NOGUCHI in view of LI, XIANG LI, and SRIVASTAVA is not relied upon for teaching, but POUDEL teaches:
wherein the second partial model includes a larger number of layers than the first partial model. (POUDEL [0020] teaches: "[…] the number of layers in the first neural network being smaller than the number of layers in the second neural network.")
Accordingly, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention, having the teachings of NOGUCHI, LI, XIANG LI, SRIVASTAVA, and POUDEL before them, to include POUDEL’s layer number configuration in each partial model in NOGUCHI, LI, XIANG LI, and SRIVASTAVA’s information processing device. One would have been motivated to make such a combination in order to reduce computational costs when operating two branches (i.e., partial models) (POUDEL [0025]).
Regarding Claim 8:
NOGUCHI in view of LI, XIANG LI, and SRIVASTAVA teaches the elements of claim 6 as outlined above. NOGUCHI in view of LI, XIANG LI, and SRIVASTAVA is not relied upon for teaching, but POUDEL teaches:
wherein the combining layer includes a softmax layer. (POUDEL [0136] teaches: "In an embodiment, the feature fusion unit is followed by a soft-max convolutional layer, Step 663, as described with reference to Step S317 of FIG. 3, which yields the probability scores for each class label.")
Accordingly, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention, having the teachings of NOGUCHI, LI, XIANG LI, SRIVASTAVA, and POUDEL before them, to include POUDEL’s soft-max layer in NOGUCHI, LI, XIANG LI, and SRIVASTAVA’s information processing device. One would have been motivated to make such a combination in order to reduce computational costs when operating two branches (i.e., partial models) (POUDEL [0025]).
Regarding Claim 9:
NOGUCHI in view of LI, XIANG LI, SRIVASTAVA, and POUDEL teaches the elements of claim 8 as outlined above. POUDEL further teaches:
wherein the combining layer performs combining processing for the output of the first partial model and the output of the second partial model before the softmax layer. (POUDEL [0131] teaches: “In an embodiment, a single feature fusion unit (i.e., the combining layer) is used to merge (i.e., performs combining processing) the output volumes of the two sub-network branches (i.e., for the output of the first partial model and the output of the second partial model), Step S661, as described with reference to S315 of FIG. 3.” POUDEL [0136] teaches: "In an embodiment, the feature fusion unit is followed by a soft-max convolutional layer (i.e., before the softmax layer), Step 663, as described with reference to Step S317 of FIG. 3, which yields the probability scores for each class label.")
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over NOGUCHI in view of LI, XIANG LI, and SRIVASTAVA, as applied to claim 4 above, and further in view of KUMAR (US 20200410054 A1), hereinafter KUMAR.
Regarding Claim 5:
NOGUCHI in view of LI, XIANG LI, and SRIVASTAVA teaches the elements of claim 4 as outlined above. NOGUCHI in view of LI, XIANG LI, and SRIVASTAVA is not relied upon for teaching, but KUMAR teaches:
wherein the first partial model includes a first embedding layer in which an input from the input layer is embedded, (KUMAR [0027] teaches: "The first embedding layer can be configured to receive and process the first input data structure and output a first embedded data structure.")
and the second partial model includes a second embedding layer in which an input from the input layer is embedded. (KUMAR [0028] teaches: "The second embedding layer can be configured to receive and process the second input data structure and output a second embedded data structure.")
Accordingly, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention, having the teachings of NOGUCHI, LI, XIANG LI, SRIVASTAVA, and KUMAR before them, to include KUMAR’s embedding layers in NOGUCHI, LI, XIANG LI, and SRIVASTAVA’s information providing device. One would have been motivated to make such a combination in order to enable the analysis of data to improve classification (KUMAR [0015]).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Alvaro S Laham Bauzo whose telephone number is (571) 272-5650. The examiner can normally be reached Mon-Fri 7:30 AM - 11:00 AM | 1:00 PM - 5:30 PM ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Usmaan Saeed, can be reached on (571) 272-4046. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/A.S.L./Examiner, Art Unit 2146
/USMAAN SAEED/Supervisory Patent Examiner, Art Unit 2146
