Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
This action is in response to the amendments filed 06 August 2026. Claims 1, 3, 9, 11-13, 17, 19-21, 25, and 27-29 are amended. Claims 1-32 are pending and have been examined.
Information Disclosure Statement
The information disclosure statements (IDS) submitted on 24 April 2026, 27 July 2026, and 01 October 2026 are being considered by the examiner.
Response to Arguments
Applicant' s arguments, see page 10, filed 06 August 2026, with respect to the objections to Claims 3, 11, 19, and 27 for informalities have been fully considered and are persuasive. The objections to Claims 3, 11, 19, and 27 have been withdrawn.
APPLICANT'S ARGUMENT: Applicant argues (page 10, paragraph 2) that "Claims 3, 11, 19, and 27 are amended ... thereby rendering the objections moot."
EXAMINER'S RESPONSE: Examiner agrees. The objections to Claims 3, 11, 19, and 27 for informalities have been withdrawn in light of arguments and/or amendments.
Applicant's arguments, see pages 10-13, filed 06 August 2026, with respect to the rejections of Claims 1-32 under 35 U.S.C. 101 have been fully considered but they are not persuasive.
APPLICANT'S ARGUMENT: Applicant argues (page 10, paragraph 5) that "Applicant submits that the presently claimed generation and use of a second or inverted neural network is not something that can practically be performed in the human mind, with or without pen and paper. ... These are specific operations involving neural-network layers, outputs, parameters, and a recited feedback/update path between neural networks, and are not practically performed as mental steps."
EXAMINER'S RESPONSE: Examiner respectfully disagrees. Amended Claim 1 recites steps of inverting layers of a neural network, generating a neural network using the layers, and updating parameters of the generated network, among other mental process steps. At the claimed level of generality, the recited mental process steps of amended Claim 1 appear to be performable in the human mind, or with the aid of pen and paper.
Amended Claim 1 recites that the steps are performed by a processor comprising circuitry. Per MPEP 2106.04(a)(2)(III)(C), a claim that requires a computer may still recite a mental process "if the claimed invention is described as a concept that is performed in the human mind and applicant is merely claiming that concept performed 1) on a generic computer, or 2) in a computer environment, or 3) is merely using a computer as a tool to perform the concept." As amended Claim 1 recites no additional elements integrating the mental process steps into a practical application or providing significantly more, it is directed to the recited mental process steps, and is thus ineligible.
APPLICANT'S ARGUMENT: Applicant argues (page 11, paragraph 1) that "claims 1, 9, and 17 recite processors, circuitry, and/or instructions which if performed by one or more processors cause the one or more processors to obtain output... These additional elements recite a particular processor-implemented inter-network feedback path, and claim 25 further recites training a third neural network using the second training data, rather than merely linking an alleged exception to a generic computing environment."
Applicant argues (page 12, paragraph 1) that "the claims recite a specific manner of generating and operating a second or inverted neural network from a first neural network.... [T]hese recitations are directed to an improvement in operation of neural-network-based computing through a particular inter-network generation, feedback, and update arrangement, rather than to an abstract idea performed on generic computing components."
EXAMINER'S RESPONSE: Examiner respectfully disagrees. Per MPEP 2106.04(a)(2)(III)(C), a claim that requires a computer may still recite a mental process. At the claimed level of generality, the additional elements recited by amended Claim 1 do not integrate the mental process steps into a practical application or provide significantly more for the reasons given below.
Applicant' s arguments, see page 13, filed 06 August 2026, with respect to the rejections of Claims 13, 21, and 29 under 35 U.S.C. 112(b) have been fully considered and are persuasive. The rejections of Claims 13, 21, and 29 under 35 U.S.C. 112(b) have been withdrawn.
APPLICANT'S ARGUMENT: Applicant argues (page 13, paragraph 2) that "claims 13, 21, and 29, as amended, comply with 35 U.S.C. § 112(b)."
EXAMINER'S RESPONSE: Examiner agrees. The rejections of Claims 13, 21, and 29 under 35 U.S.C. 112(b) have been withdrawn in light of arguments and/or amendments.
Applicant's arguments, see pages 13-16, filed 06 August 2026, with respect to the rejections of Claims 1-32 under 35 U.S.C. 103 have been fully considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
APPLICANT'S ARGUMENT: Applicant argues (page 14, paragraph 1) that "Pilzer further explains that the inconsistency-aware network takes as input a concatenation including the input image, an inconsistency tensor, and disparity information, and Fig. 2 identifies the orange dotted arrows as losses detailed in Section 3.4. ... The Office Action appears to rely on Brahimi 's Decoder for the recited second neural network and on Pilzer 's backward-network/ inconsistency-aware-network arrangement for the later-recited feedback/update sequence, but the cited combination does not show how the same claimed second neural network satisfies both aspects of the claim."
EXAMINER'S RESPONSE: Examiner notes that Applicant's arguments are moot. Amended Claim 1 is now rejected in view of Jha in view of Brahimi. Jha is relied on to teach the feature of an inverted, reversed second neural network trained based on providing the output of the second network as input to the first network.
Claim Objections
The objections to Claims 3, 11, 19, and 27 are objected for informalities are withdrawn in light of arguments and/or amendments.
Claim Rejections - 35 USC § 112
The rejections of Claims 13, 21, and 29 under 35 U.S.C. 112(b) are withdrawn in light of arguments and/or amendments.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-32 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Regarding Claim 1
Step 1
Claim 1 recites one or more processors comprising: circuitry, and thus the claimed manufacture falls within a statutory category of invention.
Step 2A Prong 1
The claim recites invert one or more layers of a first neural network trained using a first dataset, which is a mental process. The claim recites generate a second neural network comprising one or more neural network layers generated by reversing the inverted one or more layers of the first neural network, which is a mental process. The claim recites update one or more parameters of the second neural network, based, at least in part, on the output of the first neural network, which is a mental process. The claim recites generate, using the updated second neural network, a second dataset comprising approximations of data in the first dataset, which is a mental process. The claim recites perform inferences corresponding to those performed by the first neural network, which is a mental process.
Thus, the claim recites an abstract idea.
Step 2A Prong 2
The additional element obtain output of the second neural network amounts to insignificant extra-solution activity (see MPEP 2106.05(g), "mere data gathering"). The additional element provide the output of the second neural network as input to the first neural network amounts to insignificant extra-solution activity (see MPEP 2106.05(g), "mere data gathering"). The additional element generated from providing output of the second neural network as input to the first neural network invokes a computer or other machinery merely as a tool to perform an existing process (MPEP 2106.05(f), "apply it"). The additional element the second dataset being usable to train a third neural network invokes a computer or other machinery merely as a tool to perform an existing process (MPEP 2106.05(f), "apply it").
Step 2B
The additional element obtain output of the second neural network is well-understood, routine, conventional activity (see MPEP 2106.05(d), "receiving or transmitting data over a network"). The additional element provide the output of the second neural network as input to the first neural network is well-understood, routine, conventional activity (see MPEP 2106.05(d), "receiving or transmitting data over a network"). The additional element generated from providing output of the second neural network as input to the first neural network invokes a computer or other machinery merely as a tool to perform an existing process (MPEP 2106.05(f), "apply it"). The additional element the second dataset being usable to train a third neural network invokes a computer or other machinery merely as a tool to perform an existing process (MPEP 2106.05(f), "apply it").
The claim lacks additional elements that integrate it into a practical application or provide significantly more, so it is directed to an abstract idea and is ineligible.
Regarding Claim 2
Step 1
Regarding Claim 2, the rejection of Claim 1 is incorporated.
Step 2A Prong 1
The claim recites invert the one or more layers of the first neural network based, at least in part, on constructing one or more inverses of the one or more layers using parameters stored in the first neural network, which is a mathematical idea.
Thus, the claim recites an abstract idea.
Step 2A Prong 2, Step 2B
The claim recites the circuitry is to invert, which invokes a computer or other machinery merely as a tool to perform an existing process (MPEP 2106.05(f), "apply it").
The claim lacks additional elements integrating it into a practical application or providing significantly more, so it is directed to an abstract idea and is ineligible.
Regarding Claim 3
Step 1
Regarding Claim 3, the rejection of Claim 1 is incorporated.
Step 2A Prong 1
The claim recites generate a second neural network comprising one or more neural network layers generated by reversing the inverted one or more layers of the first neural network (as recited in Claim 1), wherein the one or more neural network layers comprise at least one of an inverted fully-connected layer, an inverted convolutional layer, or an inverted batch normalization layer
Thus, the claim recites an abstract idea.
Step 2A Prong 2, Step 2B
The claim lacks additional elements that integrate it into a practical application or provide significantly more, so it is directed to an abstract idea and is ineligible.
Regarding Claim 4
Step 1
Regarding Claim 4, the rejection of Claim 1 is incorporated.
Step 2A Prong 1
The claim recites wherein the circuitry is further to invert one or more convolutional layers of the first neural network ... by generating one or more transpose convolutional layers based, at least in part, on a convolutional layer of the first neural network, which is a mental process, so the claim recites a judicial exception.
Step 2A Prong 2, Step 2B
The claim recites a the first neural network being pre-trained neural network, which invokes a computer or other machinery merely as a tool to perform an existing process (MPEP 2106.05(f), "apply it").
The claim lacks additional elements integrating it into a practical application or providing significantly more, so it is directed to an abstract idea and is ineligible.
Regarding Claim 5
Step 1
Regarding Claim 5, the rejection of Claim 1 is incorporated.
Step 2A Prong 1
The claim recites wherein the circuitry is further to invert one or more batch normalization layers the first neural network ... by generating one or more inverted batch normalization layers based, at least in part, on a linear transformation of the one or more batch normalization layers of the first neural network, which is a mental process, so the claim recites a judicial exception.
Step 2A Prong 2, Step 2B
The claim recites the first neural network being pre-trained neural network, which invokes a computer or other machinery merely as a tool to perform an existing process (MPEP 2106.05(f), "apply it").
The claim lacks additional elements integrating it into a practical application or providing significantly more, so it is directed to an abstract idea and is ineligible.
Regarding Claim 6
Step 1
Regarding Claim 6, the rejection of Claim 1 is incorporated.
Step 2A Prong 1
The claim recites wherein the circuitry is further to insert an activation layer, which is a mental process.
Thus, the claim recites an abstract idea.
Step 2A Prong 2, Step 2B
The claim lacks additional elements integrating it into a practical application or providing significantly more, so it is directed to an abstract idea and is ineligible.
Regarding Claim 7
Step 1
Regarding Claim 7, the rejection of Claim 1 is incorporated.
Step 2A Prong 1
The claim recites wherein the second neural network comprising the one or more neural network layers outputs a facsimile of an input to the first neural network, which is a mental process.
Thus, the claim recites an abstract idea.
Step 2A Prong 2, Step 2B
The claim recites the first neural network being pre-trained neural network, which invokes a computer or other machinery merely as a tool to perform an existing process (MPEP 2106.05(f), "apply it").
The claim lacks additional elements that integrate it into a practical application or provide significantly more, so it is directed to an abstract idea and is ineligible.
Regarding Claim 8
Step 1
Regarding Claim 8, the rejection of Claim 1 is incorporated.
Step 2A Prong 1
The claim recites fine-tune the one or more neural network layers based, at least in part, on a layer consistency loss and a reconstruction loss, which is a mathematical idea, so the claim recites an abstract idea.
Step 2A Prong 2, Step 2B
The claim recites the circuitry is ... to fine-tune, which invokes a computer or other machinery merely as a tool to perform an existing process (MPEP 2106.05(f), "apply it").
The claim lacks additional elements integrating it into a practical application or providing significantly more, so it is directed to an abstract idea and is ineligible.
Regarding Claim 9
Step 1
Claim 9 recites a system, comprising: one or more processors, and thus the claimed machine falls within a statutory category of invention.
Step 2A Prong 1
The claim recites invert one or more layers of a first neural network trained using a first dataset, which is a mental process. The claim recites generate a second neural network comprising one or more neural network layers generated by reversing the inverted one or more layers of the first neural network, which is a mental process. The claim recites update one or more parameters of the second neural network, based, at least in part, on output of the first neural network, which is a mental process. The claim recites generate, using the updated second neural network, a second dataset comprising approximations of data in the first dataset, which is a mental process. The claim recites to perform inferences corresponding to those performed by the first neural network, which is a mental process.
Thus, the claim recites an abstract idea.
Step 2A Prong 2
The additional element obtain output of the second neural network amounts to insignificant extra-solution activity (see MPEP 2106.05(g), "mere data gathering"). The additional element provide the output of the second neural network as input to the first neural network amounts to insignificant extra-solution activity (see MPEP 2106.05(g), "mere data gathering"). The additional element generated from providing output of the second neural network as input to the first neural network invokes a computer or other machinery merely as a tool to perform an existing process (MPEP 2106.05(f), "apply it"). The additional element the second dataset being usable to train a third neural network invokes a computer or other machinery merely as a tool to perform an existing process (MPEP 2106.05(f), "apply it").
Step 2B
The additional element obtain output of the second neural network is well-understood, routine, conventional activity (see MPEP 2106.05(d), "receiving or transmitting data over a network"). The additional element provide the output of the second neural network as input to the first neural network is well-understood, routine, conventional activity (see MPEP 2106.05(d), "receiving or transmitting data over a network"). The additional element generated from providing output of the second neural network as input to the first neural network invokes a computer or other machinery merely as a tool to perform an existing process (MPEP 2106.05(f), "apply it"). The additional element the second dataset being usable to train a third neural network invokes a computer or other machinery merely as a tool to perform an existing process (MPEP 2106.05(f), "apply it").
The claim lacks additional elements that integrate it into a practical application or provide significantly more, so it is directed to an abstract idea and is ineligible.
Regarding Claim 10
Step 1
Regarding Claim 10, the rejection of Claim 9 is incorporated.
Step 2A Prong 1
Claim 10 recites the abstract ideas recited by parent Claim 9.
Step 2A Prong 2, Step 2B
The claim recites a the first neural network is a pre-trained neural network, which invokes a computer or other machinery merely as a tool to perform an existing process (MPEP 2106.05(f), "apply it").
The claim lacks additional elements integrating it into a practical application or providing significantly more, so it is directed to an abstract idea and is ineligible.
Claim 11 incorporates substantively the limitations of Claim 3 in system form and is rejected under the same rationale.
Regarding Claim 12
Step 1
Regarding Claim 12, the rejection of Claim 11 is incorporated.
Step 2A Prong 1
The claim recites the inverted convolutional layer is generated based, at least in part, on transposition of a convolutional layer of the first neural network, which is a mental process, so the claim recites a judicial exception.
Step 2A Prong 2, Step 2B
The claim recites a the one or more neural network layers comprise at least the inverted convolutional layer, which invokes a computer or other machinery merely as a tool to perform an existing process (MPEP 2106.05(f), "apply it"). The claim recites a the first neural network being a pre-trained neural network, which invokes a computer or other machinery merely as a tool to perform an existing process (MPEP 2106.05(f), "apply it").
The claim lacks additional elements integrating it into a practical application or providing significantly more, so it is directed to an abstract idea and is ineligible.
Regarding Claim 13
Step 1
Regarding Claim 13, the rejection of Claim 11 is incorporated.
Step 2A Prong 1
The claim recites the inverted batch normalization layer is generated based, at least in part, on a linear transformation of a batch normalization layer of the first neural network, which is a mental process, so the claim recites a judicial exception.
Step 2A Prong 2, Step 2B
The claim recites a the one or more neural network layers comprise at least the inverted batch normalization layer, which invokes a computer or other machinery merely as a tool to perform an existing process (MPEP 2106.05(f), "apply it"). The claim recites a the first neural network being a pre-trained neural network, which invokes a computer or other machinery merely as a tool to perform an existing process (MPEP 2106.05(f), "apply it").
The claim lacks additional elements integrating it into a practical application or providing significantly more, so it is directed to an abstract idea and is ineligible.
Claims 14-16 incorporate substantively the limitations of Claims 6-8, respectively, in system form and are rejected under the same rationale.
Regarding Claim 17
Step 1
Claim 17 recites a non-transitory machine-readable medium having stored thereon a set of instructions, and thus the claimed machine falls within a statutory category of invention.
Step 2A Prong 1
The claim recites invert one or more layers of a first neural network trained using a first dataset, which is a mental process. The claim recites generate a second neural network comprising one or more neural network layers generated by reversing the inverted one or more layers of the first neural network, which is a mental process. The claim recites update one or more parameters of the second neural network, based, at least in part, on the output of the first neural network, which is a mental process. The claim recites generate, using the updated second neural network, a second dataset comprising approximations of data in the first dataset, which is a mental process. The claim recites perform inferences corresponding to those performed by the first neural network, which is a mental process.
Thus, the claim recites an abstract idea.
Step 2A Prong 2
The additional element performed by one or more processors, cause the one or more processors invokes a computer or other machinery merely as a tool to perform an existing process (MPEP 2106.05(f), "apply it"). The additional element obtain output of the second neural network amounts to insignificant extra-solution activity (see MPEP 2106.05(g), "mere data gathering"). The additional element provide the output of the second neural network as input to the first neural network amounts to insignificant extra-solution activity (see MPEP 2106.05(g), "mere data gathering"). The additional element generated from providing output of the second neural network as input to the first neural network invokes a computer or other machinery merely as a tool to perform an existing process (MPEP 2106.05(f), "apply it"). The additional element the second dataset being usable to train a third neural network invokes a computer or other machinery merely as a tool to perform an existing process (MPEP 2106.05(f), "apply it").
Step 2B
The additional element performed by one or more processors, cause the one or more processors invokes a computer or other machinery merely as a tool to perform an existing process (MPEP 2106.05(f), "apply it"). The additional element obtain output of the second neural network is well-understood, routine, conventional activity (see MPEP 2106.05(d), "receiving or transmitting data over a network"). The additional element provide the output of the second neural network as input to the first neural network is well-understood, routine, conventional activity (see MPEP 2106.05(d), "receiving or transmitting data over a network"). The additional element generated from providing output of the second neural network as input to the first neural network invokes a computer or other machinery merely as a tool to perform an existing process (MPEP 2106.05(f), "apply it"). The additional element the second dataset being usable to train a third neural network invokes a computer or other machinery merely as a tool to perform an existing process (MPEP 2106.05(f), "apply it").
The claim lacks additional elements that integrate it into a practical application or provide significantly more, so it is directed to an abstract idea and is ineligible.
Claims 18-24 incorporate substantively the limitations of Claims 10-16, respectively, in non-transitory machine-readable medium form and are rejected under the same rationale.
Regarding Claim 25
Step 1
Claim 25 recites a method, and thus the claimed process falls within a statutory category of invention.
Step 2A Prong 1
The claim recites inverting one or more layers of a first neural network trained using a first training data, which is a mental process. The claim recites generating an inverted neural network comprising one or more neural network layers based, at least in part, on reversing the inverted one or more layers of the first neural network, which is a mental process. The claim recites updating one or more parameters of the inverted neural network, based, at least in part, on the output of the first neural network, which is a mental process. The claim recites generate a second training data comprising approximations of the first training data, which is a mental process.
Thus, the claim recites an abstract idea.
Step 2A Prong 2
The additional element obtaining output of the inverted neural network amounts to insignificant extra-solution activity (see MPEP 2106.05(g), "mere data gathering"). The additional element providing the output of the inverted neural network as input to the first neural network amounts to insignificant extra-solution activity (see MPEP 2106.05(g), "mere data gathering"). The additional element generated from providing the output of the inverted neural network as input to the first neural network invokes a computer or other machinery merely as a tool to perform an existing process (MPEP 2106.05(f), "apply it"). The additional element using the updated inverted neural network invokes a computer or other machinery merely as a tool to perform an existing process (MPEP 2106.05(f), "apply it"). The additional element training a third neural network using the second training data invokes a computer or other machinery merely as a tool to perform an existing process (MPEP 2106.05(f), "apply it").
Step 2B
The additional element obtaining output of the inverted neural network is well-understood, routine, conventional activity (see MPEP 2106.05(d), "receiving or transmitting data over a network"). The additional element providing the output of the inverted neural network as input to the first neural network is well-understood, routine, conventional activity (see MPEP 2106.05(d), "receiving or transmitting data over a network"). The additional element generated from providing the output of the inverted neural network as input to the first neural network invokes a computer or other machinery merely as a tool to perform an existing process (MPEP 2106.05(f), "apply it"). The additional element using the updated inverted neural network invokes a computer or other machinery merely as a tool to perform an existing process (MPEP 2106.05(f), "apply it"). The additional element training a third neural network using the second training data invokes a computer or other machinery merely as a tool to perform an existing process (MPEP 2106.05(f), "apply it").
The claim lacks additional elements that integrate it into a practical application or provide significantly more, so it is directed to an abstract idea and is ineligible.
Regarding Claim 26
Step 1
Regarding Claim 26, the rejection of Claim 25 is incorporated.
Step 2A Prong 1
The claim recites generating an inverted neural network comprising one or more neural network layers based, at least in part, on reversing the inverted one or more layers of the first neural network (as recited in Claim 25), wherein the inverted neural network has a reversed computational flow with respect to the first neural network. which is a mental process.
Thus, the claim recites an abstract idea.
Step 2A Prong 2, Step 2B
The claim lacks additional elements integrating it into a practical application or providing significantly more, so it is directed to an abstract idea and is ineligible.
Claims 27-30 and 32 incorporate substantively the limitations of Claims 19-22 and 24, respectively, in method form and are rejected under the same rationale.
Regarding Claim 31
Step 1
Regarding Claim 31, the rejection of Claim 25 is incorporated.
Step 2A Prong 1
Claim 31 recites the abstract ideas recited by parent Claim 25.
Step 2A Prong 2, Step 2B
The additional element generating the second training data by inputting at least one of a logit or data indicative of an encoded feature to the inverted neural network invokes a computer or other machinery merely as a tool to perform an existing process (MPEP 2106.05(f), "apply it").
The claim lacks additional elements integrating it into a practical application or providing significantly more, so it is directed to an abstract idea and is ineligible.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-3, 6, 7, 9-11, 14, 15, 17-19, 22, 23, 25-27, 30, and 31 are rejected under 35 U.S.C. 103 as being unpatentable over Jha, et al., "Disentangling factors of variation with cycle-consistent variational auto-encoders" (hereinafter "Jha") in view of Brahimi, et al., "Deep interpretable architecture for plant diseases classification" (hereinafter "Brahimi").
Regarding Claim 1, Jha teaches:
One or more processors, comprising: circuitry (Jha, p. 1, "Code for the paper: github.com/ananyahjha93/cycle-consistent-vae," and source file README.md: "This repository contains the code for the paper.... The code has been tested with the following python interpreter," where one or more processors comprising circuitry is inherent is executing Python code) to:
invert one or more layers of a first neural network (Jha, source code repository, file networks.py:
class Encoder
Conv2d(in_channels=1, out_channels=16
Conv2d(in_channels=16, out_channels=32
Conv2d(in_channels=32, out_channels=64
class Decoder
ConvTranspose2d(in_channels=128, out_channels=32
ConvTranspose2d(in_channels=32, out_channels=16
ConvTranspose2d(in_channels=16, out_channels=1
, where Jha's Encoder and Decoder correspond to the instant first and second networks, respectively, and where the decoder's second and third ConvTranspose2d layers are inverted versions of the Encoder's first and second layers) ... ;
generate a second neural network comprising one or more neural network layers generated by reversing the inverted one or more layers of the first neural network (Jha, source code repository, file networks.py, where the decoder's second and third ConvTranspose2d layers are revered in order from the corresponding layers of the encoder);
obtain output of the second neural network (Jha, p. 8, Fig. 4, "Reverse cycle of the cycle-consistent VAE architecture," depicting output
x
1
'
'
of the second decoder network Dec);
provide the output of the second neural network as input to the first neural network (Jha, p. 8, Fig. 4, "Reverse cycle of the cycle-consistent VAE architecture," depicting output
x
1
'
'
of the second decoder network Dec provided as input the first encoder network Enc);
update one or more parameters of the second neural network, based, at least in part, on the output of the first neural network generated from providing the output of the second neural network as input to the first neural network (Jha, p. 6, 3.1 Cycle-Consistency: "Cycle-consistency naturally fits into the (variational) auto-encoder training framework, where the KL divergence regularized reconstruction comprises the
L
f
o
r
w
a
r
d
. We also use the reverse cycle-consistency loss to train the encoder to disentangle better. As is typical for such loss functions, we train our model by alternating between the forward and reverse losses. We discuss the details in the sections that follow.
L
c
y
c
l
i
c
=
L
f
o
r
w
a
r
d
+
L
r
e
v
e
r
s
e
L
c
y
c
l
i
c
=
E
x
∼
p
x
∥
G
F
x
-
x
∥
p
+
E
y
∼
p
y
∥
F
G
x
-
y
∥
p
(3)
," where Jha's training with reverse cycle consistency loss corresponds to the instant second network parameter update); and
generate, using the updated second neural network, a second dataset (Jha, p. 8, Fig. 4, "Reverse cycle of the cycle-consistent VAE architecture," depicting the encoder network Enc's second output
S
1
, corresponding to the instant second dataset) ....
Jha teaches inverting one or more layers of a first neural network and generating a second dataset comprising using the updated second neural network.
Jha may not explicitly teach a first neural network trained using a first dataset and generate, using the updated second neural network, a second dataset comprising approximations of data in the first dataset, the second dataset being usable to train a third neural network to perform inferences corresponding to those performed by the first neural network.
However, Brahim teaches:
a first neural network trained using a first dataset (Brahimi, source code file teacher_student.py of the cited repository, lines 15-16, "#Teacher's graph" and "base_model1 = VGG16( ... weights='imagenet' ...)," indicating pre-training of the teacher model on ImageNet dataset)
generate, using the updated second neural network, a second dataset comprising approximations of data in the first dataset, the second dataset being usable to train a third neural network (Brahimi, 3.4 Reconstructed image refinement: "The second convolution layer (green in Fig. 1) expands the tensor to three channels to use it as the Student's input," where Brahimi's second Decoder network generates inputs for the third Student network, as depicted by p. 5, Fig. 3, "Visualization images of the Teacher/Student architecture" and p. 5, 4.2 Visualization results: "The visualizations depicted on Fig. 3 represent the reconstructed three channels images used as an input for the Student," where Brahimi's reconstructed corresponds to the instant approximations) to perform inferences corresponding to those performed by the first neural network (Brahimi, p. 2, "The whole network (Teacher + Decoder + Student) is trained to minimize jointly the losses of the two classifiers (Teacher and Student)," where the teacher and student inferences correspond as indicated by the loss functions (1), (2), and (3) for corresponding inputs).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Jha regarding inverting one or more layers of a first neural network and generating a second dataset comprising using the updated second neural network with those of Brahimi regarding a first neural network trained using a first dataset and generate, using the updated second neural network, a second dataset comprising approximations of data in the first dataset, the second dataset being usable to train a third neural network to perform inferences corresponding to those performed by the first neural network.
The motivation to do so would be to facilitate training a visualization model that produces explainable results using sharp images, using a pre-trained model to simplify development (Brahimi, p. 1, 1 Introduction: "we propose a classification and visualization architecture based on multitask learning of two classifiers: Teacher and Student. This architecture represents a trainable visualization method for plant diseases classification" and p. 9, 5 Conclusion and further research: "This architecture leverages the multitask to produce a trainable visualization method. Our experiments demonstrate the benefit of adding the Student classifier to guide the architecture in order to reconstruct a sharp visualization images," and for the pre-trained first network, p. 3, 3 Proposed method: "Fig. 1 details the Teacher/Student architecture. For the sake of simplicity, VGG16 [12] architecture is used as Teacher and Student. Nevertheless, the Teacher/Student architecture is flexible and other classification architectures can be used as Teacher or Student").
Regarding Claim 9, Jha teaches:
a system, comprising: one or more processors (Jha, p. 1, "Code for the paper: github.com/ananyahjha93/cycle-consistent-vae," and source file README.md: "This repository contains the code for the paper.... The code has been tested with the following python interpreter," where one or more processors is inherent is executing Python code) to: perform those steps recited by Claim 1. Claim 9 is rejected under the same rationale as Claim 1.
Regarding Claim 17, Jha teaches:
a non-transitory machine-readable medium having stored thereon a set of instructions, which if performed by one or more processors, cause the one or more processors (Jha, p. 1, "Code for the paper: github.com/ananyahjha93/cycle-consistent-vae," and source file README.md: "This repository contains the code for the paper.... The code has been tested with the following python interpreter," where a non-transitory machine-readable medium storing instructions performed by a processor is inherent is executing Python code) to at least: perform those steps recited by Claim 1. Claim 17 is rejected under the same rationale as Claim 1.
Regarding Claim 2, the rejection of Claim 1 is incorporated.
Brahimi further teaches:
wherein the circuitry is to invert the one or more layers of the first neural network based, at least in part, on constructing one or more inverses of the one or more layers using parameters stored in the first neural network (Brahimi, source code file teacher_student.py of the cited repository,
#Teacher's graph.
base_model1 = VGG16(include_top=False, weights='imagenet',input_shape = input_shape)
...
#Decoder's graph
#Get Teacher's tensors for skip connection.
pool5 = base_model1.get_layer('block5_pool').output
...
inv_x1_0 = Add(name='merge_x1_0')([reshaped_inv_x1_0,pool5])
, where Brahimi's VGG16 is a pre-trained model using Imagenet weights, and where the inversion of the layers, such as the pool5 layer, comprise the pre-trained weights).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the Jha/Brahimi combination regarding inverting one or more layers of a first neural network trained using a first dataset with the further teachings of Brahimi regarding wherein the circuitry is to invert the one or more layers of the first neural network based, at least in part, on constructing one or more inverses of the one or more layers using parameters stored in the first neural network.
The motivation to do so would be to facilitate training a visualization model using a pre-trained model to simplify development (Brahimi, p. 3, 3 Proposed method: "Fig. 1 details the Teacher/Student architecture. For the sake of simplicity, VGG16 [12] architecture is used as Teacher and Student. Nevertheless, the Teacher/Student architecture is flexible and other classification architectures can be used as Teacher or Student"
Regarding Claim 3, the rejection of Claim 1 is incorporated. The Jha/Brahimi combination teaches:
wherein the one or more neural network layers comprise at least one of an inverted fully-connected layer, an inverted convolutional layer, or an inverted batch normalization layer (Jha, source code repository, file networks.py, line 19, where Jha's encoder model comprises three Conv2d layers).
Claims 11 and 19 incorporate substantively the limitations of Claim 3 in system and non-transitory machine-readable medium forms, respectively, and are rejected under the same rationale.
Regarding Claim 6, the rejection of Claim 1 is incorporated.
Brahimi further teaches:
wherein the circuitry is further to insert an activation layer (Brahimi, p. 2, Fig. 1: "Teacher/Student network architecture," Decoder layers "CONV layer + Sigmoid" and "CONV layer + RELU," inserted after the final DECONV layer, and p. 4, 3.4 Reconstructed image refinement: "The last decoder's convolution layer uses sigmoid activation function to scale the values in the interval [0, 1]").
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the Jha/Brahimi combination regarding inverting one or more layers of a first neural network with the further teachings of Brahimi regarding wherein the circuitry is further to insert an activation layer.
The motivation to do so would be to scale the input to the student network to better understand inter-network communication (Brahimi, p. 4, 3.4 Reconstructed image refinement: "The last decoder's convolution layer uses sigmoid activation function to scale the values in the interval [0; 1]. This final tensor is used as Student's input and can be used also as proxy to understand the communication between the Teacher and the Student").
Claims 14 and 22 incorporate substantively the limitations of Claim 6 in system and non-transitory machine-readable medium forms, respectively, and are rejected under the same rationale.
Regarding Claim 7, the rejection of Claim 1 is incorporated.
Brahimi further teaches:
wherein the second neural network comprising the one or more neural network layers outputs a facsimile of an input (Brahimi, Figure 3: Visualization images of the Teacher/Student architecture, which depicts pairs of inputs to the first Teacher network and outputs of the second Decoder network, where the images are segmented versions of the inputs, and p. 5, 4.2 Visualization results: "The visualizations depicted on Fig. 3 represent the reconstructed three channels images used as an input for the Student," where Brahimi's reconstructed images corresponds to the instant input facsimile) to the first neural network, the first neural network being a pre-trained neural network (Brahimi, source code file Teacher_student.py of the source repository, which uses a pre-trained VGG16 convolutional network as the first Teacher network, with pre-training weights indicated by the parameter weights='imagenet').
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the Jha/Brahimi combination regarding providing the output of the second neural network as input to the first neural network with the further teachings of Brahimi regarding wherein the second neural network comprising the one or more neural network layers outputs a facsimile of an input to the first neural network, the first neural network being a pre-trained neural network.
The motivation to do so would be to facilitate training a student classifier model without requiring segmentation masks (Brahimi, p. 3, 3 Proposed method: "This proposed architecture is also designed to extract the important regions for classification without using masks. For this reason, any segmentation architecture can be modified to fit the proposed architecture by modifying the loss function to include a Student classifier and avoid the segmentation masks").
Claims 15 and 23 incorporate substantively the limitations of Claim 7 in system and non-transitory machine-readable medium forms, respectively, and are rejected under the same rationale.
Regarding Claim 10, the rejection of Claim 9 is incorporated.
Brahimi further teaches:
wherein the first neural network is a pre-trained neural network (Brahimi, source code file Teacher_student.py of the source repository, which uses a pre-trained VGG16 convolutional network as the first Teacher network, with pre-training weights indicated by the parameter weights='imagenet').
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the Jha/Brahimi combination regarding inverting one or more layers of a first neural network trained using a first dataset with the further teachings of Brahimi regarding wherein the first neural network is a pre-trained neural network.
The motivation to do so would be to simplify the training of the ensemble (Brahimi, p. 3, 3 Proposed method: "Fig. 1 details the Teacher/Student architecture. For the sake of simplicity, VGG16 [12] architecture is used as Teacher and Student. Nevertheless, the Teacher/Student architecture is flexible and other classification architectures can be used as Teacher or Student")
Claim 18 incorporates substantively the limitations of Claim 10 in non-transitory machine-readable medium form and is rejected under the same rationale.
Regarding Claim 25, Jha teaches:
A method (Jha, p. 7, 3.2 Model Description: "In the next section, we introduce our non-adversarial method, based on reverse cycle-consistency, to avoid learning a degenerate solution and explicitly train the encoder to prevent information associated with specified factors from leaking into the unspecified subspace") comprising:
inverting one or more layers of a first neural network (Jha, source code repository, file networks.py:
class Encoder
Conv2d(in_channels=1, out_channels=16
Conv2d(in_channels=16, out_channels=32
Conv2d(in_channels=32, out_channels=64
class Decoder
ConvTranspose2d(in_channels=128, out_channels=32
ConvTranspose2d(in_channels=32, out_channels=16
ConvTranspose2d(in_channels=16, out_channels=1
, where Jha's Encoder and Decoder correspond to the instant first and second networks, respectively, and where the decoder's second and third ConvTranspose2d layers are inverted versions of the Encoder's first and second layers) ... ;
generating an inverted network comprising one or more neural network layers based, at least in part, on reversing the inverted one or more layers of the first neural network (Jha, source code repository, file networks.py, where the decoder's second and third ConvTranspose2d layers are revered in order from the corresponding layers of the encoder);
obtaining output of the second neural network (Jha, p. 8, Fig. 4, "Reverse cycle of the cycle-consistent VAE architecture," depicting output
x
1
'
'
of the second decoder network Dec);
providing the output of the second neural network as input to the first neural network (Jha, p. 8, Fig. 4, "Reverse cycle of the cycle-consistent VAE architecture," depicting output
x
1
'
'
of the second decoder network Dec provided as input the first encoder network Enc);
updating one or more parameters of the inverted neural network, based, at least in part, on the output of the first neural network generated from providing the output of the inverted neural network as input to the first neural network (Jha, p. 6, 3.1 Cycle-Consistency: "Cycle-consistency naturally fits into the (variational) auto-encoder training framework, where the KL divergence regularized reconstruction comprises the
L
f
o
r
w
a
r
d
. We also use the reverse cycle-consistency loss to train the encoder to disentangle better. As is typical for such loss functions, we train our model by alternating between the forward and reverse losses. We discuss the details in the sections that follow.
L
c
y
c
l
i
c
=
L
f
o
r
w
a
r
d
+
L
r
e
v
e
r
s
e
L
c
y
c
l
i
c
=
E
x
∼
p
x
∥
G
F
x
-
x
∥
p
+
E
y
∼
p
y
∥
F
G
x
-
y
∥
p
(3)
," where Jha's training with reverse cycle consistency loss corresponds to the instant second network parameter update); and
using the updated inverted neural network to generate ... second ... data (Jha, p. 8, Fig. 4, "Reverse cycle of the cycle-consistent VAE architecture," depicting the encoder network Enc's second output
S
1
, corresponding to the instant second dataset) ....
Jha teaches inverting one or more layers of a first neural network and generating a second dataset comprising using the updated second neural network.
Jha may not explicitly teach XXX.
However, Brahim teaches:
a first neural network trained using a first training data (Brahimi, source code file teacher_student.py of the cited repository, lines 15-16, "#Teacher's graph" and "base_model1 = VGG16( ... weights='imagenet' ...)," indicating pre-training of the teacher model on ImageNet dataset)
using the updated inverted neural network to generate a second training data comprising approximations of the first training data (Brahimi, 3.4 Reconstructed image refinement: "The second convolution layer (green in Fig. 1) expands the tensor to three channels to use it as the Student's input," where Brahimi's second Decoder network generates inputs for the third Student network, as depicted by p. 5, Fig. 3, "Visualization images of the Teacher/Student architecture" and p. 5, 4.2 Visualization results: "The visualizations depicted on Fig. 3 represent the reconstructed three channels images used as an input for the Student," where Brahimi's reconstructed corresponds to the instant approximations)
training a third neural network using the second training data (Brahimi, p. 4, 4.1 Classification results: "At the beginning of training, the Teacher is more accurate than the student. This may be explained by the dependency of the Student on the representation constructed by the Teacher. However, the loss and the accuracy of the Teacher and the Student converge at the end of the training because the communicated representation becomes stable," where the Student is trained on second data produced by the Decoder from Teacher network output, as in p. 2, 3 Proposed method: "The whole network (Teacher + Decoder + Student) is trained to minimize jointly the losses of the two classifiers (Teacher and Student)," where the teacher and student inferences correspond as indicated by the loss functions (1), (2), and (3) for corresponding inputs).").
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Jha regarding inverting one or more layers of a first neural network and generating a second dataset comprising using the updated second neural network with those of Brahimi regarding a first neural network trained using a first dataset and generate, using the updated inverted neural network to generate a second training data comprising approximations of the first data, and training a third neural network using the second training data.
The motivation to do so would be to facilitate training a visualization model that produces explainable results using sharp images, using a pre-trained model to simplify development (Brahimi, p. 1, 1 Introduction: "we propose a classification and visualization architecture based on multitask learning of two classifiers: Teacher and Student. This architecture represents a trainable visualization method for plant diseases classification" and p. 9, 5 Conclusion and further research: "This architecture leverages the multitask to produce a trainable visualization method. Our experiments demonstrate the benefit of adding the Student classifier to guide the architecture in order to reconstruct a sharp visualization images," and for the pre-trained first network, p. 3, 3 Proposed method: "Fig. 1 details the Teacher/Student architecture. For the sake of simplicity, VGG16 [12] architecture is used as Teacher and Student. Nevertheless, the Teacher/Student architecture is flexible and other classification architectures can be used as Teacher or Student")
Regarding Claim 26, the rejection of Claim 25 is incorporated. The Jha/Brahimi combination teaches:
wherein the inverted neural network has a reversed computational flow with respect to the first neural network (Jha, source code repository, file networks.py:
class Encoder
Conv2d(in_channels=1, out_channels=16
Conv2d(in_channels=16, out_channels=32
Conv2d(in_channels=32, out_channels=64
class Decoder
ConvTranspose2d(in_channels=128, out_channels=32
ConvTranspose2d(in_channels=32, out_channels=16
ConvTranspose2d(in_channels=16, out_channels=1
, where Jha's decoder's ConvTranspose2d layers reverse the order of the encoder's Conv2d layers).
Regarding Claim 27, the rejection of Claim 25 is incorporated. The Jha/Brahimi combination teaches:
wherein the inverted neural network comprises at least one of an inverted fully-connected layer, an inverted convolutional layer, or an inverted batch normalization layer (Jha, source code repository, file networks.py, line 19, where Jha's decoder model comprises three ConvTranspose2d that invert the corresponding encoder layers).
Regarding Claim 30, the rejection of Claim 25 is incorporated. The Jha/Brahimi combination teaches:
further comprising adding an activation layer to the inverted neural network (Brahimi, p. 2, Fig. 1: "Teacher/Student network architecture," Decoder layers "CONV layer + Sigmoid" and "CONV layer + RELU," inserted after the final DECONV layer, and p. 4, 3.4 Reconstructed image refinement: "The last decoder's convolution layer uses sigmoid activation function to scale the values in the interval [0, 1]").
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the Jha/Brahimi combination regarding inverting one or more layers of a first neural network with the further teachings of Brahimi regarding adding an activation layer to the inverted neural network.
The motivation to do so would be to scale the input to the student network to better understand inter-network communication (Brahimi, p. 4, 3.4 Reconstructed image refinement: "The last decoder's convolution layer uses sigmoid activation function to scale the values in the interval [0; 1]. This final tensor is used as Student's input and can be used also as proxy to understand the communication between the Teacher and the Student").
Regarding Claim 31, the rejection of Claim 25 is incorporated. The Brahimi/Pilzer combination teaches:
generating the second training data by inputting at least one of a logit or data indicative of an encoded feature to the inverted neural network (Brahimi, p. 2, Fig. 1: "Teacher/Student network architecture" with p. 2, 3 Proposed method: "The decoder consumes the Teacher latent representations to reconstruct an image with the same dimension of the input image," where Brahimi's latent representations corresponds to the instant encoded feature data, as in p. 3, 3 Proposed method: "The Teacher/Student network is designed to reconstruct an image containing the discriminant features formed by the Teacher to help the Student training").
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the Jha/Brahimi combination regarding generating a second training data with the further teachings of Brahimi regarding generating the second training data by inputting at least one of a logit or data indicative of an encoded feature to the inverted neural network.
The motivation to do so would be to facilitate training a network with improved discrimination capacity (Brahimi, p. 3, 3 Proposed method: "The Teacher/Student network is designed to reconstruct an image containing the discriminant features formed by the Teacher to help the Student training").
Claims 4, 12, 20, and 28 are rejected under 35 U.S.C. 103 as being unpatentable over Jha, et al., "Disentangling factors of variation with cycle-consistent variational auto-encoders" (hereinafter "Jha") in view of Brahimi, et al., "Deep interpretable architecture for plant diseases classification" (hereinafter "Brahimi") in view of Iglovikov, et al. "Ternausnet: U-net with vgg11 encoder pre-trained on imagenet for image segmentation" (hereinafter "Iglovikov").
Regarding Claim 4, the rejection of Claim 1 is incorporated. The Jha/Brahimi combination teaches:
wherein the circuitry is ... to invert one or more convolutional layers of the first neural network ... by generating one or more ... convolutional layers based, at least in part, on a convolutional layer of the first neural network (Jha, source code repository, file networks.py:
class Encoder
Conv2d(in_channels=1, out_channels=16
Conv2d(in_channels=16, out_channels=32
Conv2d(in_channels=32, out_channels=64
class Decoder
ConvTranspose2d(in_channels=128, out_channels=32
ConvTranspose2d(in_channels=32, out_channels=16
ConvTranspose2d(in_channels=16, out_channels=1
, where Jha's Encoder and Decoder correspond to the instant first and second networks, respectively, and where the decoder's second and third ConvTranspose2d layers are inverted versions of the Encoder's first and second layers).
Brahimi further teaches:
the first neural network being a pre-trained neural network (Brahimi, source code file Teacher_student.py of the source repository, which uses a pre-trained VGG16 convolutional network as the first Teacher network, with pre-training weights indicated by the parameter weights='imagenet').
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the Jha/Brahimi combination regarding inverting one or more layers of a first neural network trained using a first dataset with the further teachings of Brahimi regarding wherein the first neural network is a pre-trained neural network.
The motivation to do so would be to simplify the training of the ensemble (Brahimi, p. 3, 3 Proposed method: "Fig. 1 details the Teacher/Student architecture. For the sake of simplicity, VGG16 [12] architecture is used as Teacher and Student. Nevertheless, the Teacher/Student architecture is flexible and other classification architectures can be used as Teacher or Student").
The Jha/Brahimi combination does not explicitly teach generating one or more transpose convolutional layers based, at least in part, on a convolutional layer.
However, Iglovikov teaches:
generating one or more transpose convolutional layers based, at least in part, on a convolutional layer of the first neural network (Iglovikov, p. 2, Fig. 1, 3x3 ConvTranspose2d(stride=2)+ReLU, where the skip-connection arrows connect the transpose layers to their corresponding convolutional layers in the encoder, and p. 2, II. Network Architecture: "To construct the decoder we use transposed convolutions layers that doubles the size of a feature map while reducing the number of channels by half. ... The resultant feature map is treated by convolution operation to keep the number of channels the same as in a symmetric encoder term. This upsampling procedure is repeated 5 times to pair up with 5 max poolings, as shown in Fig. 1").
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the Jha/Brahimi combination regarding generation of a transposed convolutional layer for an inverted neural network from an untrained neural network with those of Iglovikov regarding generating one or more transpose convolutional layers based on a convolutional layer.
The motivation to do so would be to facilitate training models for segmentation tasks under circumstances where there are limited amounts of data (Iglovikov, p. 1, II. Network Architecture: "Every step in the expansive path consists of an upsampling of the feature map followed by a convolution. Hence, the expansive branch increases the resolution of the output. In order to localize, upsampled features, the expansive path combines them with high-resolution features from the contracting path via skip-connections [4]. The output of the model is a pixel-by-pixel mask that shows the class of each pixel. This architecture proved itself very useful for segmentation problems with limited amounts of data").
Regarding Claim 12, the rejection of Claim 11 is incorporated. The Jha/Brahimi combination teaches:
wherein the one or more neural network layers comprise at least the inverted convolutional layer (Jha, source code repository, file networks.py:
class Encoder
Conv2d(in_channels=1, out_channels=16
Conv2d(in_channels=16, out_channels=32
Conv2d(in_channels=32, out_channels=64
class Decoder
ConvTranspose2d(in_channels=128, out_channels=32
ConvTranspose2d(in_channels=32, out_channels=16
ConvTranspose2d(in_channels=16, out_channels=1
, where Jha's Encoder and Decoder correspond to the instant first and second networks, respectively, and where the decoder's second and third ConvTranspose2d layers are inverted versions of the Encoder's first and second layers).
Brahimi further teaches:
the first neural network being a pre-trained neural network (Brahimi, source code file Teacher_student.py of the source repository, which uses a pre-trained VGG16 convolutional network as the first Teacher network, with pre-training weights indicated by the parameter weights='imagenet').
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the Jha/Brahimi combination regarding inverting one or more layers of a first neural network trained using a first dataset with the further teachings of Brahimi regarding wherein the first neural network is a pre-trained neural network.
The motivation to do so would be to simplify the training of the ensemble (Brahimi, p. 3, 3 Proposed method: "Fig. 1 details the Teacher/Student architecture. For the sake of simplicity, VGG16 [12] architecture is used as Teacher and Student. Nevertheless, the Teacher/Student architecture is flexible and other classification architectures can be used as Teacher or Student").
The Jha/Brahimi combination does not explicitly teach generating one or more transpose convolutional layers based, at least in part, on a convolutional layer.
However, Iglovikov teaches:
the inverted convolutional layer is generated based, at least in part, on transposition of a convolutional layer of the first neural network (Iglovikov, p. 2, Fig. 1, 3x3 ConvTranspose2d(stride=2)+ReLU, where the skip-connection arrows connect the transpose layers to their corresponding convolutional layers in the encoder, and p. 2, II. Network Architecture: "To construct the decoder we use transposed convolutions layers that doubles the size of a feature map while reducing the number of channels by half. ... The resultant feature map is treated by convolution operation to keep the number of channels the same as in a symmetric encoder term. This upsampling procedure is repeated 5 times to pair up with 5 max poolings, as shown in Fig. 1").
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the Jha/Brahimi combination regarding generation of a transposed convolutional layer for an inverted neural network from an untrained neural network with those of Iglovikov regarding generating one or more transpose convolutional layers based on a convolutional layer.
The motivation to do so would be to facilitate training models for segmentation tasks under circumstances where there are limited amounts of data (Iglovikov, p. 1, II. Network Architecture: "Every step in the expansive path consists of an upsampling of the feature map followed by a convolution. Hence, the expansive branch increases the resolution of the output. In order to localize, upsampled features, the expansive path combines them with high-resolution features from the contracting path via skip-connections [4]. The output of the model is a pixel-by-pixel mask that shows the class of each pixel. This architecture proved itself very useful for segmentation problems with limited amounts of data").
Claims 20 and 28 incorporate substantively the limitations of Claim 12 in non-transitory machine-readable medium and method forms, respectively, and are rejected under the same rationale.
Claims 5, 13, 21, and 29 are rejected under 35 U.S.C. 103 as being unpatentable over Jha, et al., "Disentangling factors of variation with cycle-consistent variational auto-encoders" (hereinafter "Jha") in view of Brahimi, et al., "Deep interpretable architecture for plant diseases classification" (hereinafter "Brahimi") in view of Yamaguchi, et al. (US Patent Publication US 2021/0327456 A1, hereinafter "Yamaguchi").
Regarding Claim 5, the rejection of Claim 1 is incorporated. The Jha/Brahimi combination teaches:
invert one or more ... layers of of the first neural network ... by generating one or more inverted ... layers based, at least in part, on a ... transformation of the one or more ... layers (Jha, source code repository, file networks.py:
class Encoder
Conv2d(in_channels=1, out_channels=16
Conv2d(in_channels=16, out_channels=32
Conv2d(in_channels=32, out_channels=64
class Decoder
ConvTranspose2d(in_channels=128, out_channels=32
ConvTranspose2d(in_channels=32, out_channels=16
ConvTranspose2d(in_channels=16, out_channels=1
, where Jha's Encoder and Decoder correspond to the instant first and second networks, respectively, and where the decoder's second and third ConvTranspose2d layers are inverted versions of the Encoder's first and second layers).
Brahimi further teaches:
the first neural network being a pre-trained neural network (Brahimi, source code file Teacher_student.py of the source repository, which uses a pre-trained VGG16 convolutional network as the first Teacher network, with pre-training weights indicated by the parameter weights='imagenet').
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the Jha/Brahimi combination regarding inverting one or more layers of a first neural network trained using a first dataset with the further teachings of Brahimi regarding wherein the first neural network is a pre-trained neural network.
The motivation to do so would be to simplify the training of the ensemble (Brahimi, p. 3, 3 Proposed method: "Fig. 1 details the Teacher/Student architecture. For the sake of simplicity, VGG16 [12] architecture is used as Teacher and Student. Nevertheless, the Teacher/Student architecture is flexible and other classification architectures can be used as Teacher or Student").
The Jha/Brahimi combination does not explicitly teach invert one or more batch normalization layers of a ... neural network by generating one or more inverted batch normalization layers based, at least in part, on a linear transformation of the one or more batch normalization layers of the pre-trained neural network.
However, Yamaguchi teaches invert one or more batch normalization layers of ... a ... neural network (Yamaguchi, Fig. 21, where the corresponding adaptive batch normalization layers designated "AdaBN" are indicated to be inverse layers, e.g.,
f
4
-
1
j
_
0
and
f
4
j
_
1
, as in [0089]: "adaptive batch normalization is introduced to Normalizing Flow. Specifically, at least one inverse transformation
f
i
-
1
z
of K transformations
{
f
i
z
}
i
=
1
K
to be used in Normalizing Flow is adaptive batch normalization"), by generating one or more inverted batch normalization layers based, at least in part, on a linear transformation of the one or more batch normalization layers of the ... neural network (Yamaguchi, [0213]: "AdaFlow is a method in which adaptive batch normalization is introduced into Normalizing Flow. ... Note that, in calculation of adaptive batch normalization, scale transformation and shift transformation, that is, calculation in the expression (24d) may be omitted. In other words, it is also possible to express that the inverse transformation
f
i
_
0
-
1
z
is adaptive batch normalization in which
γ
=
1
and
β
=
0
," where Yamaguchi's equation 24d expresses the normalized output of a batch normalization layer).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the Jha/Brahimi combination regarding inversion of a convolutional layer from a pre-trained encoder with those of Yamaguchi regarding inverting one or more batch normalization layers of a neural network by generating one or more inverted batch normalization layers based on a linear transformation of the one or more batch normalization layers.
The motivation to do so would be to enable the pre-trained encoder and the trained decoder to accommodate a broader distribution of inputs without decreasing accuracy, thus increasing applicability of the model (Yamaguchi, [0089]: "to enable the second probability distribution to be adaptively learned from a small amount of adaptive learning data with a small calculation cost, adaptive batch normalization is introduced to Normalizing Flow" and [0211]: "Domain adaptation is a technique of adjusting a learned model so that, in a case where distribution of learning data to be used for model learning is different from distribution of test data which is a target of processing using the learned model, accuracy of the processing using the learned model does not degrade due to a difference between the distribution" and [0212]: "While there are various methods for domain adaptation which can be combined with a deep neural network (DNN), here, adaptive batch normalization will be described").
Regarding Claim 13, the rejection of Claim 11 is incorporated.
Brahimi further teaches:
the first neural network being a pre-trained neural network (Brahimi, source code file Teacher_student.py of the source repository, which uses a pre-trained VGG16 convolutional network as the first Teacher network, with pre-training weights indicated by the parameter weights='imagenet').
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the Jha/Brahimi combination regarding inverting one or more layers of a first neural network trained using a first dataset with the further teachings of Brahimi regarding wherein the first neural network is a pre-trained neural network.
The motivation to do so would be to simplify the training of the ensemble (Brahimi, p. 3, 3 Proposed method: "Fig. 1 details the Teacher/Student architecture. For the sake of simplicity, VGG16 [12] architecture is used as Teacher and Student. Nevertheless, the Teacher/Student architecture is flexible and other classification architectures can be used as Teacher or Student").
The Jha/Brahimi combination does not explicitly teach the one or more neural network layers comprise at least the inverted batch normalization layer and the inverted batch normalization layer is generated based, at least in part, on a linear transformation of a batch normalization layer of the first neural network.
However, Yamaguchi teaches
the one or more neural network layers comprise at least the inverted batch normalization layer (Yamaguchi, Fig. 21, where the corresponding adaptive batch normalization layers designated "AdaBN" are indicated to be inverse layers, e.g.,
f
4
-
1
j
_
0
and
f
4
j
_
1
, as in [0089]: "adaptive batch normalization is introduced to Normalizing Flow. Specifically, at least one inverse transformation
f
i
-
1
z
of K transformations
{
f
i
z
}
i
=
1
K
to be used in Normalizing Flow is adaptive batch normalization"), and the inverted batch normalization layer is generated based, at least in part, on a linear transformation of a batch normalization layer of the first neural network (Yamaguchi, [0213]: "AdaFlow is a method in which adaptive batch normalization is introduced into Normalizing Flow. ... Note that, in calculation of adaptive batch normalization, scale transformation and shift transformation, that is, calculation in the expression (24d) may be omitted. In other words, it is also possible to express that the inverse transformation
f
i
_
0
-
1
z
is adaptive batch normalization in which
γ
=
1
and
β
=
0
," where Yamaguchi's equation 24d expresses the normalized output of a batch normalization layer).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the Jha/Brahimi combination regarding inversion of a convolutional layer from a pre-trained encoder with those of Yamaguchi regarding the one or more neural network layers comprise at least the inverted batch normalization layer and the inverted batch normalization layer is generated based, at least in part, on a linear transformation of a batch normalization layer of the first neural network.
The motivation to do so would be to enable the pre-trained encoder and the trained decoder to accommodate a broader distribution of inputs without decreasing accuracy, thus increasing applicability of the model (Yamaguchi, [0089]: "to enable the second probability distribution to be adaptively learned from a small amount of adaptive learning data with a small calculation cost, adaptive batch normalization is introduced to Normalizing Flow" and [0211]: "Domain adaptation is a technique of adjusting a learned model so that, in a case where distribution of learning data to be used for model learning is different from distribution of test data which is a target of processing using the learned model, accuracy of the processing using the learned model does not degrade due to a difference between the distribution" and [0212]: "While there are various methods for domain adaptation which can be combined with a deep neural network (DNN), here, adaptive batch normalization will be described").
Claims 21 and 29 incorporate substantively the limitations of Claim 11 in non-transitory machine-readable medium and method forms, respectively, and are rejected under the same rationale.
Claims 8, 16, 24, and 32 are rejected under 35 U.S.C. 103 as being unpatentable over Jha, et al., "Disentangling factors of variation with cycle-consistent variational auto-encoders" (hereinafter "Jha") in view of Brahimi, et al., "Deep interpretable architecture for plant diseases classification" (hereinafter "Brahimi") in further view of Johnson, et al., "Perceptual Losses for Real-Time Style Transfer and Super-Resolution" (hereinafter "Johnson").
Regarding Claim 8, the rejection of Claim 1 is incorporated.
The Jha/Brahimi combination teaches inverting one or more layers of a first neural network trained using a first dataset.
The Jha/Brahimi combination may not explicitly teach fine-tune the one or more neural network layers based, at least in part, on a layer consistency loss and a reconstruction loss.
However, Johnson teaches:
fine-tune the one or more neural network layers (Johnson, p. 2, 1 Introduction: "We train feed- forward transformation networks for image transformation tasks, but rather than using per-pixel loss functions depending only on low-level pixel information, we train our networks using perceptual loss functions that depend on high-level features from a pretrained loss network") based, at least in part, on a layer consistency loss and a reconstruction loss (Johnson, p. 5, 3 Method: "The loss network
ϕ
is used to define a feature reconstruction loss
l
f
e
a
t
ϕ
and a style reconstruction loss
l
s
t
y
l
e
ϕ
that measure differences in content and style between images. For each input image
x
we have a content target
y
c
and a style target
y
s
. For style transfer, the content target
y
c
is the input image
x
and the output image
y
^
should combine the content of
x
=
y
c
with the style of
y
s
" where Johnson's feature and style reconstruction losses correspond to the instant layer consistency and reconstruction losses, respectively, where the instant layer consistency loss of the instant specification at [0094]: "if inverted input X' is perceptually close to X, then layer consistency loss 506 between them should be small," is taught by Johnson at, p. 7, 3.2 Perceptual Loss Functions, Feature Reconstruction Loss: "Using a feature reconstruction loss for training our image transformation networks encourages the output image
y
^
to be perceptually similar to the target image
y
, but does not force them to match exactly").
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of the Jha/Brahimi combination regarding fine-tuning an autoencoder based on classification loss with those of Johnson regarding fine-tuning the one or more neural network layers based on a layer consistency loss and a reconstruction loss.
The motivation to do so would be to facilitate training networks capable of performing style transfer with qualitative visual results and runtime efficiency (Johnson, p. 1, Abstract: "We show results on image style transfer, where a feed-forward network is trained to solve the optimization problem proposed by Gatys et al in real-time. Compared to the optimization-based method, our network gives similar qualitative results but is three orders of magnitude faster. We also experiment with single-image super-resolution, where replacing a per-pixel loss with a perceptual loss gives visually pleasing results").
Claims 16, 24, and 32 incorporate substantively the limitations of Claim 8 in system, non-transitory machine-readable medium, and method forms, respectively, and are rejected under the same rationale.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ROBERT N DAY whose telephone number is (703)756-1519. The examiner can normally be reached M-F 9-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner's supervisor, Kakali Chaki can be reached at (571) 272-3719. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/R.N.D./Examiner, Art Unit 2122
/KAKALI CHAKI/Supervisory Patent Examiner, Art Unit 2122
