DETAILED ACTION
This office action is responsive to the above identified application filed 3/1/2022. The application contains claims 1-22, all examined and rejected
Notice of Pre-AIA or AIA Status
The present application is being examined under the pre-AIA first to invent provisions.
Priority
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Information Disclosure Statement
The Information Disclosure Statements with references submitted 3/10/2022 have been considered and entered into the file.
Claim Objections
Claim 1 is objected to because of the following informalities: The claim recites the limitation “input neuron(s)” in line 5 of the claim and recites the limitation “hidden layer(s)” in line 11 of the claim. These limitations are indefinite. The Examiner has interpreted these limitations as “one or more input neurons” and “hidden layer" respectively. Appropriate correction is required.
Claim 2 objected to because of the following informalities: claim 2 recites “a method as claimed in claim 1” instead of “the method as claimed in claim 1”. Appropriate correction is required.
Claim 13 is objected to because of the following informalities: The claim recites the limitation “and/or” in line 6 of the claim. This limitation is indefinite. The Examiner has interpreted this limitation as “or." Appropriate correction is required.
Claim 19 is objected to because of the following informalities: The claim recites the limitation “output neuron(s)” in line 10 of the claim. This limitation is indefinite. The Examiner has interpreted this limitation as “one or more output neurons." Appropriate correction is required.
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claim1-22 rejected on the ground of nonstatutory double patenting as being unpatentable over claim 1-30 of U.S. Patent No. 8862527 B2. Although the claims at issue are not identical, they are not patentably distinct from each other because they have a similar scope.
Current application
U.S. Patent No. 8862527 B2
Claim 1,
A method for training an artificial neural network, said method comprising:
(i) initialising the neural network by selecting an output of the neural network to be trained and connecting an output neuron of the neural network to input neuron(s) in an input layer of the neural network for the selected output;
(ii) preparing a data set to be learnt by the neural network; and (iii) applying the prepared data set to the neural network to be learnt by applying an input vector of the prepared data set to a first hidden layer of the neural network, or an output layer of the neural network if the neural network has no hidden layer(s), and determining whether at least one neuron for the selected output in each layer of the neural network can learn to produce the associated output for the input vector, wherein:
if at least one neuron for the selected output in each layer of the neural network can learn to produce the associated output for the input vector, and if there are more input vectors of the prepared data set to learn, repeat (iii) for the next input vector, else repeat (i) to (iii) for the next output of the neural network if there are more outputs to be trained;
if no neuron in a hidden layer for the selected output of the neural network can learn to produce the associated output for the input vector, a new neuron is added to that layer to learn the associated output which could not be learnt by any other neurons in that layer for the selected output, and if there are more input vectors of the data set to learn, repeat (iii) for the next input vector, else repeat (i) to (iii) for the next output of the neural network if there are more outputs to be trained;
if the output neuron for the selected output of the neural network cannot learn to produce the associated output for the input vector, that output neuron becomes a neuron of a hidden layer of the neural network, a new neuron is added to this hidden layer to learn the associated output which could not be learnt by the output neuron, and a new output neuron is added to the neural network for the selected output, and if there are more input vectors of the data set to learn, repeat (iii) for the next input vector, else repeat (i) to (iii) for the next output of the neural network if there are more outputs to be trained.
Claim 2
The artificial neural network according to claim 1, wherein the artificial neural network is arranged to be trained by:
(i) initialising the neural network by selecting an output of the neural network to be trained and connecting an output neuron of the neural network to one or more input neuron(s) in an input layer of the neural network for the selected output;
(ii) preparing a data set to be learnt by the neural network; and (iii) applying the prepared data set to the neural network to be learnt by applying an input vector of the prepared data set to a first hidden layer of the neural network, or an output layer of the neural network if the neural network does not have at least one hidden layer, and determining whether at least one neuron for the selected output in each layer of the neural network can learn to produce the associated output for the input vector, wherein:
if at least one neuron for the selected output in each layer of the neural network can learn to produce the associated output for the input vector, and if there are more input vectors of the prepared data set to learn, repeat (iii) for the next input vector, else repeat (i) to (iii) for the next output of the neural network if there are more outputs to be trained;
if no neuron in a hidden layer for the selected output of the neural network can learn to produce the associated output for the input vector, the new neuron is added to that layer, …, to learn the associated output which could not be learnt by any other neurons in that layer for the selected output, and if there are more input vectors of the data set to learn, repeat (iii) for the next input vector, else repeat (i) to (iii) for the next output of the neural network if there are more outputs to be trained;
if the output neuron for the selected output of the neural network cannot learn to produce the associated output for the input vector, that output neuron becomes a neuron of a hidden layer of the neural network, the new neuron is added to this hidden layer, and that new neuron is updated with a modified data set formed by copying input-output associations learnt by the output neuron and modifying the input-output associations based upon the last association that could not be learned, to learn the associated output which could not be learnt by the output neuron, and a new output neuron is added to the neural network for the selected output, and if there are more input vectors of the data set to learn, repeat (iii) for the next input vector, else repeat (i) to (iii) for the next output of the neural network if there are more outputs to be trained.
Claim 2
A method as claimed in claim 1, wherein (ii) preparing the data set is performed before (i) initializing the neural network.
Claim 3
The artificial neural network as claimed in claim 2, wherein (ii) preparing the data set is performed before (i) initializing the neural network.
Claim 3
The method as claimed in claim 1, wherein the neurons of the neural network are Linear Threshold Gates (LTGs).
Claim 4
The artificial neural network as claimed in claim 2, wherein the neurons of the neural network are Linear Threshold Gates (LTGs).
Claim 4
The method as claimed in claim 3, wherein in said (iii), to determine whether an LTG can learn to produce the associated output for the input vector is to determine whether a relationship between weights and a threshold of the LTG has a solution given what the LTG has previously learnt.
Claim 5
The artificial neural network as claimed in claim 4, wherein in said (iii), to determine whether an LTG can learn to produce the associated output for the input vector comprises determining whether the input-output associations of the LTG representing a relationship between weights and a threshold of the LTG has a solution given what the LTG has previously learnt.
Claim 5
The method as claimed in claim 4, wherein said relationship is a constraint, and wherein the input vector and the LTG's weight vector form a relationship with the LTG's threshold based on the selected output of the neural network.
Claim 6
the artificial neural network as claimed in claim 5, wherein said relationship is a constraint, and wherein the input vector and the LTG's weight vector form a relationship with the LTG's threshold based on the selected output of the neural network.
Claim 6
The method as claimed in claim 5, wherein to learn a constraint is to be able to add the constraint to a constraint set of an LTG.
Claim 7
The artificial neural network as claimed in claim 6, wherein to learn a constraint is to be able to add the constraint to a constraint set of an LTG.
Claim 7
The method as claimed in claim 6, wherein to be able to add the constraint to a constraint set of an LTG there must be a solution between all the constraints.
Claim 8
The artificial neural network as claimed in claim 7, wherein to be able to add the constraint to a constraint set of an LTG there must be a solution between all the constraints.
Claim 8
The method as claimed in claim 6, wherein initialising the neural network further includes clearing the constraints set of the output LTG so that the constraints set of the output LTG is empty.
Claim 9
The artificial neural network as claimed in claim 7, wherein initialising the neural network further includes clearing the constraints set of the output LTG so that the constraints set of the output LTG is empty.
Claim 9
The method as claimed in claim 1, wherein preparing the data set to be learnt by the neural network includes in any order: converting the data set into a predefined data format before the data set is presented to the neural network for training; determining whether there are any inconsistencies in the data set before the data set is presented to the neural network for training; sorting the data set before the data set is presented to the
neural network for training; and determining whether the 0 input vector is available in the data set before the data set is presented to the neural network for training, and if the 0 input vector is available the data set, the data set is ordered so that the 0 input vector is presented to the neural network to be trained first.
Claim 10
The artificial neural network as claimed in claim 2, wherein preparing the data set to be learnt comprises: converting the data set into a predefined data format before the data set is presented to the neural network for training; determining whether there are any inconsistencies in the data set before the data set is presented to the neural network for training; sorting the data set before the data set is presented to the neural network for training; and, determining whether there is an input vector having a value of zero for all inputs available in the data set before the data set is presented to the neural network for training, and if there is an input vector having a value of zero for all inputs available in the data set, the data set is ordered so that the input vector having a value of zero for all inputs is presented to the neural network to be trained first.
Claim 10
The method as claimed in claim 9, wherein said predefined data format is binary or floating-point data format.
Claim 11
The artificial neural network as claimed in claim 10, wherein said predefined data format is binary or floating-point data format.
Claim 11
The method as claimed in claim 9, wherein determining whether there are any inconsistencies in the data set before the data set is presented to the neural network includes: determining whether there are two or more identical input vectors which produce different output.
Claim 12
The artificial neural network as claimed in claim 10, wherein determining whether there are any inconsistencies in the data set before the data set is presented to the neural network includes: determining whether there are two or more identical input vectors which produce different output.
Claim 12
The method as claimed in claim 11, wherein if it is determined that two or more identical input vectors produce a different output, only one of the input vectors is used.
Claim 13
The artificial neural network as claimed in claim 12, wherein if it is determined that two or more identical input vectors produce a different output, only one of the input vectors is used.
Claim 13
The method as claimed in claim 9, wherein sorting the data set before the data set is presented to the neural network for training includes: sorting the input vectors of the data set into two sets, separating those that output 1 from those that produce 0 for that output, and selecting one of the two sets to be trained first; sorting the data with a Self Organising Map (SOM); and/or sorting the data using any other suitable method.
Claim 14, claim 20
The artificial neural network as claimed in claim 10, wherein sorting the data set before the data set is presented to the neural network for training includes: sorting the input vectors of the data set into at least two sets, separating the input vectors that output one from the input vectors that produce zero for that output, and selecting one of the at least two sets to be trained first; and sorting the data set.
Claim 14
The method as claimed in claim 13, wherein a single list for each input layer is created from the sorted data before the data is presented to the neural network for training.
Claim 15
The artificial neural network as claimed in claim 14, wherein a single sorted data set for the input layer currently being trained is created from the at least two separated data sets before the data is presented to the neural network for training.
Claim 15
The method as claimed in claim 5, wherein if a new LTG is added to a layer to learn a constraint that could not be learnt by any other LTG in accordance with (iii): the new LTG is connected to all LTGs in the next layer which contribute to the selected output of the neural network, and the constraints set of the LTGs in the next layer which receive input from the new LTG are updated to accept input from the new LTG; if the layer with the new LTG is not the first layer of the neural network, the new LTG is connected to and receives input from all LTGs in a preceding layer which contribute to the selected output of the neural network; and, the constraints set of the new LTG is updated to include a copy of the modified constraints set of the previous last LTG in that layer and the constraint which could not be learnt by any other LTG in that layer.
Claim 16
The artificial neural network as claimed in claim 6, wherein if a new LTG is added to a hidden layer to learn a constraint that could not be learnt by any other LTG in the hidden layer in accordance with (iii): the new LTG is connected to all LTGs in a next layer which contribute to the selected output of the neural network, and a constraints set of the LTGs in the next layer which receive input from the new LTG are updated to accept input from the new LTG; if the hidden layer with the new LTG is not the first hidden layer of the neural network, the new LTG is connected to and receives input from all LTGs in a preceding hidden layer which contribute to the selected output of the neural network; and, a constraints set of the new LTG is updated to include a copy of the constraint which could not be learnt by any other LTG in that hidden layer
Claim 16
The method as claimed in claim 5, wherein if a new output LTG is added to the neural network in accordance with (iii): the new output LTG is connected to and receives input from the LTGs in the hidden layer; if the hidden layer is not the first layer of the neural network, the new LTG in the hidden layer is connected to and receives input from all LTGs in a preceding layer which contribute to the selected output of the neural network; the constraints set of the new LTG added to the hidden layer is updated to include a copy of the modified constraints set of the previous output LTG in that layer and the constraint which could not be learnt by the previous output LTG; and, the new output LTG combines its inputs in a predefined logical relationship according to what could not be learnt by the previous output LTG.
Claim 17
The artificial neural network as claimed in claim 6, wherein if a new output LTG is added to the neural network in accordance with (iii): the new output LTG is connected to and receives input from the LTGs in the hidden layer; if the hidden layer is not the first hidden layer of the neural network, the new LTG in the hidden layer is connected to and receives input from all LTGs in a preceding hidden layer which contribute to the selected output of the neural network; the constraints set of the new LTG added to the hidden layer is updated to include a copy of the constraint which could not be learnt by the previous output LTG and a modified data set, expressed for the new LTG as a modified constraints set formed from the constraints set of the previous output LTG in that hidden layer by modifying the relationships between constraints as follows, wherein T is neuron threshold:
if xi·w>T could not be learnt, modify all constraints as xi·w<T from the previous output LTGs constraints set, else
if xi·w<T could not be learnt, modify all constraints as xi·w>T from the previous output LTGs constraints set; and, the new output LTG combines its inputs in a predefined logical relationship according to what could not be learnt by the previous output LTG.
Claim 17
The method as claimed in claim 16, wherein when a new output LTG is added to the neural network in accordance with (iii), the predefined logical relationship formed between the inputs into this new output LTG is logical OR, logical AND, or any other suitable logical relationship.
Claim 18
The artificial neural network as claimed in claim 17, wherein when a new output LTG is added to the neural network in accordance with (iii), the predefined logical relationship formed between the inputs into this new output LTG is logical OR, logical AND, or any other equivalent logical relationship.
Claim 18
The method as claimed in claim 17, wherein logical OR is used if the input vector 5 that could not be learnt by the previous output LTG produces an output 1, and logical AND is used if the input vector that could not be learnt by the previous output LTG produces an output 0.
Claim 19
The artificial neural network as claimed in claim 18, wherein logical OR is used for the neural output to be learned if the input vector that could not be learnt by the previous output LTG produces an output one, and logical AND is used for the neural output to be learned if the input vector that could not be learnt by the previous output LTG produces an output zero.
Claim 19
A method for adding a new neuron into a layer of a neural network during training, the new neuron being added to the neural network when no other neuron in that layer for the selected output can learn a relationship associated with an input vector of a data set being learnt, said method including: updating the new neuron with a copy of all the modified data from a previous last neuron that contributes to the selected output of the neural network in that layer and the relationship which could not be learnt by any other neuron in that layer; and, updating the output neuron(s) to accept input from the new neuron.
Claim 24
A method for adding a new neuron into a layer of a neural network during training, the new neuron being added to the neural network when no other neuron in that layer for a selected output can learn a relationship associated with an input vector of a data set being learnt, said method including:
updating the new neuron with both the relationship which could not be learnt by any other neuron in that layer and a modified data set from a last trained neuron in that layer that contributes to the selected output of the neural network, wherein the modified data set is formed by copying all learnt relationships from the last trained neuron into the new neuron and modifying the copied relationships based upon the relationship which could not be learnt by any other neuron in that layer; and,
updating one or more output neurons to accept input from the new neuron.
Claim 20
The method as claimed in claim 19, wherein the neurons of the neural network are LTGs.
Claim 25
The method as claimed in claim 24, wherein the neurons of the neural network are LTGs.
Claim 21
The method as claimed in claim 20, wherein said relationship is a relationship between weights and a threshold of an LTG.
Claim 26
The method as claimed in claim 25, wherein said relationship is a relationship between weights and a threshold of an LTG.
Claim 22
The method as claimed in claim 20, wherein said relationship is a constraint, and wherein the input vector of the data set and an LTG's weight vector form a relationship with the LTG's threshold based on the output of the neural network.
Claim 27
The method as claimed in claim 25, wherein said relationship is a constraint, and wherein the input vector of the data set and an LTG's weight vector form a relationship with the LTG's threshold based on the output of the neural network.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 9-14 rejected under 35 U.S.C. 112, second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which applicant regards as the invention.
Claim 9 recites the limitation “the 0 vector”. There is insufficient antecedent basis for this limitation in the claim. Dependent claims inherit the deficiency of the independent claim.
Claim 14 is rejected under 35 U.S.C. 112, second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which applicant regards as the invention. Claim 14 recites the limitation “a single list for each input layer is created” in line 2 of the claim. The Examiner is unable to determine the metes and bounds of this limitation and, therefore, the claim. The limitation “a single list” is indefinite. The Examiner is unable to ascertain the content of the created list. There is insufficient antecedent basis for the limitation “each input layer.” Claim 1 recites “an input layer” (i.e., a single input layer).
Claims 15 rejected under 35 U.S.C. 112, second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which applicant regards as the invention.
Claim 15 recites the limitation “the next layer”, “the constraints set”, and “the modified constraints set”. There is insufficient antecedent basis for this limitation in the claim.
Claims 16-18 rejected under 35 U.S.C. 112, second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which applicant regards as the invention.
Claim 16 recites the limitation “the hidden layer”, “the constraints set”, and “the modified constraints set”. There is insufficient antecedent basis for this limitation in the claim. Dependent claims inherit the deficiency of the independent claim.
Claim 16 recites the limitation “if the hidden layer is not the first layer” in lines 4-5 of the claim. The meaning of this limitation is unclear to the Examiner as it appears that the conditional would always be true.
Claims 17-18 rejected under 35 U.S.C. 112, second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which applicant regards as the invention.
Claim 17 recites the limitation “the predefined logical relationship”, “the constraints set”, and “the modified constraints set”. There is insufficient antecedent basis for this limitation in the claim. Dependent claims inherit the deficiency of the independent claim.
Claims 17-18 rejected under 35 U.S.C. 112, second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which applicant regards as the invention.
Claim 17 recites the limitation “any other suitable logical relationship” in lines 4-5 of the claim. This limitation is indefinite as it is unclear to the Examiner as to what constitutes a “suitable” logical relationship. Dependent claims inherit the deficiency of the independent claim.
Claims 19-22 rejected under 35 U.S.C. 112, second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which applicant regards as the invention.
Claim 19 recites the limitation “the “output neuron(s)”. There is insufficient antecedent basis for this limitation in the claim. Dependent claims inherit the deficiency of the independent claim.
Claims 19-22 rejected under 35 U.S.C. 112, second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which applicant regards as the invention.
Claim 19 recites the limitation “the modified data”. There is insufficient antecedent basis for this limitation in the claim. The Examiner is unable to ascertain what data is modified. Dependent claims inherit the deficiency of the independent claim.
Claims 19-22 rejected under 35 U.S.C. 112, second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which applicant regards as the invention.
Claim 19 recites the limitation “modified data from a previous last neuron” in lines 6-7 of the claim. This limitation is indefinite. The Examiner is unable to ascertain what a previous last neuron is.
When the new neuron is added to the output layer, the limitation "updating the output neuron(s) to accept input from the new neuron" is indefinite as there are no output neurons to accept input from the new neuron since the new neuron is an output neuron itself. Dependent claims inherit the deficiency of the independent claim. Dependent claims inherit the deficiency of the independent claim.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of pre-AIA 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(b) the invention was patented or described in a printed publication in this or a foreign country or in public use or on sale in this country, more than one year prior to the date of application for patent in the United States.
Claims 1-8, 15-17, 19-22 are rejected under 35 U.S.C. 102(b) as being anticipated by “A New Training Algorithm for Feedforward Neural Networks” (hereinafter D1).
With regard to Claim 1,
D1 teach a method for training an artificial neural network, said method comprising:
(i) initialising the neural network by selecting an output of the neural network to be trained and connecting an output neuron of the neural network to input neuron(s) in an input layer of the neural network for the selected output (See D1, P.1297,Section 4.1, discloses that each input is connected directly to a respective output);
(ii) preparing a data set to be learnt by the neural network (See D1, P.1297,Section 4.1, discloses training each output by applying input vectors (data set to be learnt) to the output also disclosing the presence of training data show that data was prepared for training (collecting training data)); and
(iii) applying the prepared data set to the neural network to be learnt by applying an input vector of the prepared data set to a first hidden layer of the neural network, or an output layer of the neural network if the neural network has no hidden layer(s), and determining whether at least one neuron for the selected output in each layer of the neural network can learn to produce the associated output for the input vector (See D1, P.1297, Section 4.1, Fig. 3-5), wherein:
if at least one neuron for the selected output in each layer of the neural network can learn to produce the associated output for the input vector, and if there are more input vectors of the prepared data set to learn, repeat (iii) for the next input vector, else repeat (i) to (iii) for the next output of the neural network if there are more outputs to be trained (See D1, P.1297, Section 4.1, Fig. 3-5);
if no neuron in a hidden layer for the selected output of the neural network can learn to produce the associated output for the input vector, a new neuron is added to that layer to learn the associated output which could not be learnt by any other neurons in that layer for the selected output, and if there are more input vectors of the data set to learn, repeat (iii) for the next input vector, else repeat (i) to (iii) for the next output of the neural network if there are more outputs to be trained (See D1, P.1297, Section 4.1, Fig. 3-5, discloses adding an LTG (i.e., a neuron) to the network if there is no solution);
if the output neuron for the selected output of the neural network cannot learn to produce the associated output for the input vector, that output neuron becomes a neuron of a hidden layer of the neural network, a new neuron is added to this hidden layer to learn the associated output which could not be learnt by the output neuron, and a new output neuron is added to the neural network for the selected output, and if there are more input vectors of the data set to learn, repeat (iii) for the next input vector, else repeat (i) to (iii) for the next output of the neural network if there are more outputs to be trained (See D1, P.1297, Section 4.1, Fig. 3-5, discloses that if none of the hidden layers can learn a pattern, then an LTG (i.e., a neuron) is added to the hidden layer).
With regard to Claim 2,
D1 teach a method as claimed in claim 1, wherein (ii) preparing the data set is performed before (i) initializing the neural network (See D1, Section 4.1, presence of training data show that data was prepared for training (collecting training data set is preparing a data set for the neural network and it must occur before initializing the neural network)).
With regard to Claim 3,
D1 teach the method as claimed in claim 1, wherein the neurons of the neural network are Linear Threshold Gates (LTGs) (D1; P. 1294 (Section 1): discloses the use of LTGs).
With regard to Claim 4,
D1 teach the method as claimed in claim 3, wherein in said (iii), to determine whether an LTG can learn to produce the associated output for the input vector is to determine whether a relationship between weights and a threshold of the LTG has a solution given what the LTG has previously learnt (D1; P. 1294 (Section 1): discloses constraints which define relationships between the weights and the thresholds in each LTG in the NN [3]; at Section 3.1: discloses adding constraints to an initially empty constraint set; and at Section 3.4: discloses testing the constraints to determine whether there is a solution).
With regard to Claim 5,
D1 teach the method as claimed in claim 4, wherein said relationship is a constraint, and wherein the input vector and the LTG's weight vector form a relationship with the LTG's threshold based on the selected output of the neural network (D1; P. 1294 (Section 1): discloses constraints which define relationships between the weights and the thresholds in each LTG in the NN [3]; at Section 3.1: discloses adding constraints to an initially empty constraint set; and at Section 3.4: discloses testing the constraints to determine whether there is a solution).
With regard to Claim 6,
D1 teach the method as claimed in claim 5, wherein to learn a constraint is to be able to add the constraint to a constraint set of an LTG (D1; P. 1294 (Section 1): discloses constraints which define relationships between the weights and the thresholds in each LTG in the NN [3]; at Section 3.1: discloses adding constraints to an initially empty constraint set; and at Section 3.4: discloses testing the constraints to determine whether there is a solution).
With regard to Claim 7,
D1 teach the method as claimed in claim 6, wherein to be able to add the constraint to a constraint set of an LTG there must be a solution between all the constraints (D1; P. 1294 (Section 1): discloses constraints which define relationships between the weights and the thresholds in each LTG in the NN [3]; at Section 3.1: discloses adding constraints to an initially empty constraint set; and at Section 3.4: discloses testing the constraints to determine whether there is a solution).
With regard to Claim 8,
D1 teach the method as claimed in claim 6, wherein initialising the neural network further includes clearing the constraints set of the output LTG so that the constraints set of the output LTG is empty (D1; P. 1294 (Section 1): discloses constraints which define relationships between the weights and the thresholds in each LTG in the NN [3]; at Section 3.1: discloses adding constraints to an initially empty constraint set; and at Section 3.4: discloses testing the constraints to determine whether there is a solution).
With regard to Claim 15,
D1 teach the method as claimed in claim 5, wherein if a new LTG is added to a layer to learn a constraint that could not be learnt by any other LTG (Fig. 3, “Figure 3. All the inputs are connected to (a). If (a) does not have a solution, then the new constraint is learnt by (b). LTGs (a) and (b) form the first hidden layer”, Fig. 4) in accordance with (iii): the new LTG is connected to all LTGs in the next layer which contribute to the selected output of the neural network (Fig. 3, Fig. 4, LTG D is new connected to LTG C in the next layer if it contributes to the output, Fig. 5), and the constraints set of the LTGs in the next layer which receive input from the new LTG are updated to accept input from the new LTG (Fig. 3-5, P. 1279, 4.1, “If another pattern is presented to the network that neither LTGs (a) or (b) cannot learn, then another LTG (d), see figure 4, is added into this hidden layer and learns the new constraint. The output of LTG ( d) becomes input into LTG (c) also”);
if the layer with the new LTG is not the first layer of the neural network (Figs. 3-5), the new LTG is connected to and receives input from all LTGs in a preceding layer which contribute to the selected output of the neural network (Figs. 3-5, Fig. 4, “all the input are connected now to (a), (b) and (d)”) ; and, the constraints set of the new LTG is updated to include a copy of the modified constraints set of the previous last LTG in that layer and the constraint which could not be learnt by any other LTG in that layer (D1, Fig. 3-5, P. 1279, 4.1, “If another pattern is presented to the network that neither LTGs (a) or (b) cannot learn, then another LTG (d), see figure 4, is added into this hidden layer and learns the new constraint. The output of LTG ( d) becomes input into LTG (c) also This process of adding LTGs into the hidden layer is repeated until all the constraints formed from the training set have been learnt.”).
With regard to Claim 16,
D1 teach the method as claimed in claim 5, wherein if a new output LTG is added to the neural network in accordance with (iii): the new output LTG is connected to and receives input from the LTGs in the hidden layer (Fig. 4, LTG C receive input LTGs a,b,d); if the hidden layer is not the first layer of the neural network, the new LTG in the hidden layer is connected to and receives input from all LTGs in a preceding layer which contribute to the selected output of the neural network (Figs. 3-5); the constraints set of the new LTG added to the hidden layer is updated to include a copy of the modified constraints set of the previous output LTG in that layer and the constraint which could not be learnt by the previous output LTG; and, the new output LTG combines its inputs in a predefined logical relationship according to what could not be learnt by the previous output LTG (D1, Fig. 3-5, Fig. 5, “Figure 5. If (c) is tested and found not to form a threshold function, then its functioning is split .across LTGs (c) and (e) and their behavior is combined into LTG (f) which may form a threshold function. The network now has 2 hidden layers and an output layer”).
With regard to Claim 17,
D1 teach the method as claimed in claim 16, wherein when a new output LTG is added to the neural network in accordance with (iii), the predefined logical relationship formed between the inputs into this new output LTG is logical OR, logical AND, or any other suitable logical relationship (3.3, Fig. 3-5, P.1297, “Figure 4. LTG (a) or (b) do not have a solution then the new constraints is added to LTG (d); note that all the input are connected now to (a), (b) and (d). (c) combines the functioning of the (a), (b) and (c) in the output layer”, P. 1298, 4.2).
With regard to Claim 19,
D1 teach a method for adding a new neuron into a layer of a neural network during training, the new neuron being added to the neural network when no other neuron in that layer for the selected output can learn a relationship associated with an input vector of a data set being learnt (D1, Fig. 3-5, P. 1294 (Section 1): discloses constraints which define relationships between the weights and the thresholds in each LTG in the NN [3], P.1297-1298; Adding a new neuron into a layer of a neural network during training, the new neuron being added to the neural network when no other neuron in that layer for the selected output can learn a relationship associated with an input vector of a data set being learnt of applicant maps to ‘If LTG (c) cannot learn to produce the desired output once the hidden layer has; learnt the input pattern, then LTG (e) is added to the network, see figure 5. LTGs (c) and (e) form the next hidden layer and then the functioning of these LTGs are combined by LTG (f),…’ ), said method including:
updating the new neuron with a copy of all the modified data from a previous last neuron that contributes to the selected output of the neural network in that layer and the relationship which could not be learnt by any other neuron in that layer; and, updating the output neuron(s) to accept input from the new neuron (Fig. 3-5, P. 1279, 4.1, “If another pattern is presented to the network that neither LTGs (a) or (b) cannot learn, then another LTG (d), see figure 4, is added into this hidden layer and learns the new constraint. The output of LTG ( d) becomes input into LTG (c) also This process of adding LTGs into the hidden layer is repeated until all the constraints formed from the training set have been learnt”, P1298; Updating the new neuron with a copy of all the modified data from a previous last neuron that contributes to the selected output of the neural network in that layer and the relationship which could not be learnt by any other neuron in that layer; and, updating the output neuron(s) to accept input from the new neuron of applicant maps to ‘Figure 5. If (c) is tested and found not to form a threshold function, then its functioning is split across LTGs (c) and (e) and their behavior is combined into LTG (f) which may form a threshold function. The network now has 2 hidden layers and an output layer.’ of Garner6. EC: For node ‘e’ to work, is must consider all the previous nodes (copy of modified data). To accept input from the new neuron of applicant maps to the connection from node ‘e’ to node ‘f.’).
With regard to Claim 20,
D1 teach the method as claimed in claim 19, wherein the neurons of the neural network are LTGs (Fig. 3-5, P. 1296, 4, Networks of LTGs, P. 1294 (Section 1): discloses the use of LTGs).
With regard to Claim 21,
D1 teach the method as claimed in claim 20, wherein said relationship is a relationship between weights and a threshold of an LTG (P. 1293, “The algorithm presented here learns the data by resolving the data learnt by the network into relationships between the weights and the thresholds in the network”, P. 1294 (Section 1): discloses constraints which define relationships between the weights and the thresholds in each LTG in the NN [3]).
With regard to Claim 22,
D1 teach the method as claimed in claim 20, wherein said relationship is a constraint, and wherein the input vector of the data set and an LTG's weight vector form a relationship with the LTG's threshold based on the output of the neural network (Abstract, “This algorithm trains the neurons to learn the data by resolving the relationships between the weights and thresholds into constraints that define the neuron 's behavior”, P. 1294 (Section 1): discloses constraints which define relationships between the weights and the thresholds in each LTG in the NN [3]; at Section 3.1: discloses adding constraints to an initially empty constraint set; and at Section 3.4: discloses testing the constraints to determine whether there is a solution).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of pre-AIA 35 U.S.C. 103(a) which forms the basis for all obviousness rejections set forth in this Office action:
(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth in section 102, if the differences between the subject matter sought to be patented and the prior art are such that the subject matter as a whole would have been obvious at the time the invention was made to a person having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 9-12 are rejected under 35 U.S.C. 103(a) as being unpatentable over “A New Training Algorithm for Feedforward Neural Networks” (hereinafter D1) in view of Dempsey et al. (US 2002/0147694 A1) further in view of Ruggiero [US 5241620 A].
With regard to Claim 9,
D1 teach the method as claimed in claim 1, wherein preparing the data set to be learnt by the neural network includes in any order: converting the data set into a predefined data format before the data set is presented to the neural network for training (P.1298, “In this paper an algorithm was presented that fully trains a network using binary data that will determine its own topology”, P. 1299, “this technique only works for binary data”, P. 1294, “LTG has n variable binary inputs”; thus, the data set must be in a binary format).
D1 does not explicitly teach the determining whether there are any inconsistencies in the data set before the data set is presented to the neural network for training.
Dempsey teach determining whether there are any inconsistencies in the data set before the data set is presented to the neural network for training (Dempsy, ¶44, “Typically, an item of training data contains an input element, such as a vector containing a plurality of independent parameters, and an output element, which may be a single output value. In a strict sense, one item of training data conflicts with another if the two input elements are identical but the output elements or values are different. However, a broader interpretation allows two items which have very similar input elements but also contain conflicting output values to be considered to be conflicting”, ¶100, ¶120, “validated account profiles 36 are checked for conflict with the training data items 32 contained within the existing knowledge base at step 37 as described above. If no conflict is found then a validated account profile may be added to the existing knowledge base 30 to form an extended knowledge base 38 containing the validated account profile as new training data 40. If conflict is sound then a conflict resolution step 42 must be used. Two options at the conflict resolution step are shown. The first is to discard the conflicting validated account profile, preferably placing it in a conflict library 44 for future reference rather than discarding it altogether. The second is to add the conflicting validated account profile to the existing knowledge base 30 and to remove the conflicting existing item of training data 32, to form a modified knowledge base 46. Which option is chosen in the conflict resolution step will depend on the nature of the conflict and the data, as discussed above”).
D1 and Dempsey are analogous art because they are both from a similar problem-solving area. Both D1 and Dempsy address the problem of providing training data to systems including neural networks.
It would have been obvious to one of ordinary skill in the art at the time of the invention, motivated by the desire to provide training data to train a neural network, to modify the neural network teachings of D1 to incorporate the training data teachings of Dempsy for the purpose of training a neural network to provide an improved method and apparatus for retraining trainable data classifiers and to prevent erroneous by removing training data that include confliction (Dempsy, ¶¶10-11).
D1-Dempsey does not explicitly disclose sorting the data set before the data set is presented to the neural network for training; and, determining whether the 0 input vector is available in the data set before the data set is presented to the neural network for training, and if the 0 input vector is available the data set, the data set is ordered so that the 0 input vector is presented to the neural network to be trained first.
Ruggiero teach sorting the data set before the