DETAILED ACTION
Response to Arguments
Applicant’s arguments with respect to amended claims 1 and 13 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged by Applicant’s argument contained in the Remarks submitted on 01/05/2026.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Acknowledgment is made of applicant’s claim for foreign priority under 35 U.S.C. 119 (a)-(d). The certified copy has been filed in parent Application No. TW110128994, filed on 08/05/2021.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-25 are rejected under 35 U.S.C. 103 as being unpatentable over Zhu et al. "Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networks." (2020): arXiv:1703.10593v7 (“Zhu”) in view of Donahue et al US 2022/0108434 Al(“Donahue”) and in view of Niu, Shuanlong, et al. "Defect image sample generation with GAN for improving defect recognition." IEEE Transactions on Automation Science and Engineering 17.3 (2020): 1611-1622(“Niu”) and further in view of Singh et al., Generative adversarial networks for synthetic defect generation in assembly and test manufacturing. In 2020 31st Annual SEMI Advanced Semiconductor Manufacturing Conference (ASMC) 2020 Aug 24(“Singh”).
Regarding claim 1, Zhu teaches a method for training asymmetric generative adversarial network to generate an image, adapted for an electronic apparatus comprising…wherein the method comprises:
inputting a first real image belonging to a first category, a second real image belonging to a second category, and a third real image belonging to a third category(Zhu, pg., 8, As the first three rows of fig. 7 detail:
PNG
media_image1.png
390
975
media_image1.png
Greyscale
the first row's input is a first real image belonging to a first category, the second row's input is a second real image belonging to a second category, and the third row's input is a third real image belonging to a third category)
to an asymmetric generative adversarial network to train the asymmetric generative adversarial network, wherein the first category and the second category belong to first element, wherein the third category belongs to second element, wherein the asymmetric generative adversarial network comprises a first generator, a second generator, a first discriminator, and a second discriminator(Zhu, pgs. 3-4, see also fig. 3, “[G]iven training samples
x
i
i
=
1
N
∈
X
and
y
j
j
=
1
M
∈
Y
. As illustrated in Figure 3 (a), our model includes two mappings
G
:
X
→
Y
[wherein the asymmetric generative adversarial network comprises a first generator]and
F
:
Y
→
X
[a second generator]…we introduce two adversarial discriminators
D
X
[and a second discriminator]and
D
Y
[ a first discriminator]….” & Zhu, pg. 5, “We apply two techniques from recent works to stabilize our model training procedure. First, for
L
G
A
N
(Equation 1)… we replace the negative log likelihood objective by a least-squares loss… for a GAN loss
L
G
A
N
(
G
,
D
,
X
,
Y
)
we train the G to minimize
E
x
~
p
d
a
t
a
(
x
)
[
D
G
x
-
1
2
]
and train the D to minimize
E
y
~
p
d
a
t
a
(
y
)
D
y
-
1
2
+
E
x
~
p
d
a
t
a
(
x
)
[
D
G
x
2
]
[ to a asymmetric generative adversarial network to train the asymmetric generative adversarial network].” & Zhu, pg., 8, As the first three rows of fig. 7 detail:
PNG
media_image1.png
390
975
media_image1.png
Greyscale
the first row and second row’s input belong to the first and second category and are part of the first element since the first row’s ground truth represents the second row’s input and vice versa (i.e., second row’s ground truth represents the first row’s input), the third row’s input belongs to the third category and are part of the second element since its ground truth does not match the first row and/or second row’s input),
wherein the first generator maps the first real image to a first generated image so that the first discriminator does not distinguish between the first generated image and the third real image, wherein the first discriminator is updated according to the first generated image(Zhu, pg. 7, see also table 2 and fig. 5, “Table 2 assesses the performance of the labels
→
photo task on the Cityscapes…[i]n both cases, our method again [CycleGAN] outperforms the baselines[wherein the first generator maps the first real image to a first generated image so that the first discriminator does not distinguish between the first generated image and the third real image].” & see also Zhu, pg., 5, “[W]e train...the D to minimize...
E
x
~
p
d
a
t
a
(
x
)
[
D
G
x
2
]
... and update the discriminators using a history of generated images....[ wherein the first discriminator is updated according to the first generated image]”),
wherein the first generator receives the second real image and generates a second generated image accordingly, and the second generator receives the second generated image and generates a second reconstructed image accordingly(Zhu, pgs. 3-4, see also fig. 3, “[G]iven training samples
x
i
i
=
1
N
∈
X
and
y
j
j
=
1
M
∈
Y
. As illustrated in Figure 3 (a), our model includes two mappings
G
:
X
→
Y
and
F
:
Y
→
X
.” And as fig. 3(b) details (below)
PNG
media_image2.png
319
434
media_image2.png
Greyscale
in using the forward cycle-consistency loss: the first generator (i.e., G) receives a real image (i.e.,
x
∈
X
) and generates a generated image (i.e.,
Y
^
in fig. 3(b)) and the second generator (i.e., F in fig. 3(b)) receives the generated image (i.e.,
Y
^
in fig. 3(b)) and generates a reconstructed image (i.e.,
x
^
∈
X
)[ wherein the first generator receives the second real image and generates a second generated image accordingly, and the second generator receives the second generated image and generates a second reconstructed image accordingly]),1 and
the first generator executes an operation to generate a second value according to the second real image and the second reconstructed image, and updates a parameter of the first generator according to the second value(Zhu,, pgs. 4-5, “We can incentivize this behavior using a cycle consistency loss:
L
c
y
c
G
,
F
=
E
x
~
p
d
a
t
a
(
x
)
[
F
G
x
-
x
1
]
[ the first generator executes an operation to generate a second value according to the second real image and the second reconstructed image]…[o]ur full objective is:
L
G
,
F
,
D
X
,
D
Y
=
L
G
A
N
G
,
D
Y
,
X
,
Y
+
L
G
A
N
F
,
D
X
,
Y
,
X
+
λ
L
c
y
c
(
G
,
F
)
…[w]e aim to solve:
G
*
,
F
*
=
arg
min
G
,
F
max
D
x
,
D
Y
L
(
G
,
F
,
D
X
,
D
Y
)
[and updates a parameter of the first generator according to the second value].”);
and inputting a fourth real image belonging to the second category to the first generator in the asymmetric generative adversarial network [that is trained to generate a defect image](Zhu, pg., 8, As the last row’s input of fig. 6 details (as shown below) a segmented image belonging to the second category is input into the GAN + forward (i.e.,
F
(
G
(
x
)
) to produce the reconstructed image[and inputting a fourth real image belonging to the second category to the first generator in the asymmetric generative adversarial network]
PNG
media_image3.png
357
892
media_image3.png
Greyscale
).2
While teaches Zhu the asymmetric generative adversarial network Zhu does not teach:
a processor; that is trained to generate a defect image, wherein the first real image and the third real image do not have defective features, and the second real image and the fourth real image have defective features.
However, Donahue teaches:
a processor(Donahue, para. 0032, see also fig. 1, “When software is used, the operations performed by Image reconstruction training system 100 can be implemented in program code configured to run on hardware, such as a processor unit[a processor]. When firmware is used, the operations performed by Image reconstruction training system 100 can be implemented in program code and data and stored in persistent memory to run on a processor unit.”);
that is trained to generate a defect image, wherein the first real image and the third real image do not have defective features, and the second real image and the fourth real image have defective features(Donahue, para. 0022-0026, “[T]raining defect detection system by creating a hybrid between Generative Adversarial Network (GAN) and an autoencoder…[t]he difference between the reconstruction and the original image highlights anomalies that can be used for defect detection[that is trained to generate a defect image]. In CT images…[it can] successfully identify cracks, voids, and high z inclusions[and the second real image and the fourth real image have defective features]….” & Donahue, para. 0049, “The illustrative embodiments train the deep learning model with defect-free example images[wherein the first real image and the third real image do not have defective features]….”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Zhu with the teachings of Donahue the motivation to do so would be to identify more potential defects/flaws in the manufacturing process to comply with higher government/quality standards while not sacrificing overall production(Donahue, paras. 0003-0005, “The disclosure relates generally to defect detection, and more specifically to automatically identifying defects or flaws that can be potentially used for quality assurance… [d]efect detection is an invaluable tool for the quality assurance in manufacturing process . There are number of ways to detect different types of defects, but it is usually hard to find the balance between maximizing yield and minimizing the number of defective parts that makes it through the quality control process...[f]or anomaly detection to be reproducible and cost effective, an automated method is needed….”).
While Zhu in view of Donahue teaches an asymmetric generative adversarial network Zhu in view of Donahue do not teach: wherein the defect image belongs to a fourth category, and the fourth category does not have a training sample, wherein the fourth category belongs to the second element, wherein the defect image is generated based on the fourth real image of the second category with the defective features and the third real image of the third category without the defective features.
However, Niu
wherein the defect image belongs to a fourth category, and the fourth category does not have a training sample, wherein the fourth category belongs to the second element(Niu, pgs. 1614-1616, see also fig. 1 and fig. 2, “Generator G generates the fake defect image G(g)[wherein the defect image belongs to a fourth category]…[t]o achieve defect images generated from defect-free images, we introduce the cycle consistency loss
L
c
y
c
G
,
F
=
E
(
g
∈
P
r
(
g
)
|
|
F
(
G
g
-
g
1
+
E
b
∈
P
r
b
G
F
b
-
b
1
[and the fourth category does not have a training sample, wherein the fourth category belongs to the second element].”),
wherein the defect image is generated based on the fourth real image of the second category with the defective features and the third real image of the third category without the defective features(Niu, pgs. 1614-1616, see also fig. 1 and fig. 2, As fig. 2 details:
PNG
media_image4.png
474
884
media_image4.png
Greyscale
a real defect image b is inputted[based on the fourth real image of the second category with the defective features] into the generator F to output a defect free image of F(b) [and the third real image of the third category without the defective features]which is then input into the generator G to output the defect image of G(F(b))[wherein the defect image is generated]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Zhu and Donahue with the teachings of Niu the motivation to do so would be to generate a better defect recognition device by generating defects in images to augment a training dataset for a deep learning model to improve the performance of industrial defect detection(Niu, pgs. 1611-1612, “In intelligent manufacturing and defect recognition are important for quality inspection and process control. Traditional defect recognition pipelines consist mainly of the extraction and selection of defect features using the digital image processing technology, defect recognition, and classification using threshold or machine-learning methods. Therefore, traditional methods possess some disadvantages. Handcrafted Features Are Deficient…Different Types of Defects Require Different Algorithms… [a] novel generation method called SDGAN is proposed by introducing two diversity control discriminators and cycle consistency loss to generate the defect images economically with high quality and diversity using a small number of defect images and a large number of auxiliary defect-free images…SDGAN reduces the error rate and improves the robustness of defect recognition as a data set augmentation method.”).
While Zhu in view of Donahue and Niu teach wherein the first category and the second category belong to first element, wherein the third category belongs to second element, Zhu in view of Donahue and Niu do not teach: wherein the first element and the second element are different elements.
However, Singh teaches:
wherein the first element and the second element are different elements(Singh, pg., 2, see also fig. 1, “An important thing to note here is that this network requires perfectly registered, paired images of non-defect samples[wherein the first element] and defect samples[and the second element] [are different elements]....”)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Zhu in view of Donahue and Niu with the teachings of Singh the motivation to do so would be to create realistic defects in the training dataset that occur during the semiconductor manufacturing process which are hard to manually create(Singh, pg., 1, “A majority of operations in the semiconductor industry utilize automated visual inspection to assess process health and help identify process induced defects... [t]o make training data balanced, it is imperative to collect a large quantity of images with a variety of defect signatures. This can be achieved by destructive methods to artificially create defective units and image the units using vision systems. The cost associated with the destruction of the units can be huge in terms of material wastage, time, and manual labour required to create realistic defects. In some cases, it may be difficult to create a defect by hand with the same look as one would observe in the production line.”).
Regarding claim 2, Zhu in view of Donahue, Niu and Singh teaches the method for training a asymmetric generative adversarial network to generate an image according to claim 1, wherein the first generator receives the first real image and generates a first generated image accordingly, and the second generator receives the first generated image and generates a first reconstructed image accordingly(Zhu, pgs. 3-4, see also fig. 3, “[G]iven training samples
x
i
i
=
1
N
∈
X
and
y
j
j
=
1
M
∈
Y
. As illustrated in Figure 3 (a), our model includes two mappings
G
:
X
→
Y
and
F
:
Y
→
X
.” And as fig. 3(b) details (below)
PNG
media_image2.png
319
434
media_image2.png
Greyscale
in using the forward cycle-consistency loss: the first generator (i.e., G) receives a real image (i.e.,
x
∈
X
) and generates a generated image (i.e.,
Y
^
in fig. 3(b)) and the second generator (i.e., F in fig. 3(b)) receives the generated image (i.e.,
Y
^
in fig. 3(b)) and generates a reconstructed image (i.e.,
x
^
∈
X
)[ the first generator receives the first real image and generates a first generated image accordingly, and the second generator receives the first generated image and generates a first reconstructed image accordingly]),3
and the first generator executes an operation to generate a first value according to the first real image and the first reconstructed image, and updates the parameter of the first generator according to the first value(Zhu, pgs. 4-5, “We can incentivize this behavior using a cycle consistency loss:
L
c
y
c
G
,
F
=
E
x
~
p
d
a
t
a
(
x
)
[
F
G
x
-
x
1
]
[ and the first generator executes an operation to generate a first value according to the first real image and the first reconstructed image]…[o]ur full objective is:
L
G
,
F
,
D
X
,
D
Y
=
L
G
A
N
G
,
D
Y
,
X
,
Y
+
L
G
A
N
F
,
D
X
,
Y
,
X
+
λ
L
c
y
c
(
G
,
F
)
…[w]e aim to solve:
G
*
,
F
*
=
arg
min
G
,
F
max
D
x
,
D
Y
L
(
G
,
F
,
D
X
,
D
Y
)
[and updates the parameter of the first generator according to the first value]).
Regarding claim 3, Zhu in view of Donahue, Niu and Singh teaches the method for training a asymmetric generative adversarial network to generate an image according to claim 2, wherein the first discriminator distinguishes between the first generated image and a third real image belonging to a third category to generate a first discrimination value, and the first generator updates the parameter of the first generator according to the first discrimination value(Zhu, pg. 4, “For the mapping function
G
:
X
→
Y
and its discriminator
D
Y
[the first discriminator]we express the objective as:
L
G
A
N
G
,
D
Y
,
X
,
Y
=
E
y
~
p
d
a
t
a
(
y
)
log
D
Y
y
+
E
x
~
p
d
a
t
a
(
x
)
[
l
o
g
(
1
-
D
Y
(
G
x
)
]
… where G tries to generate images G(x) that look similar to images from domain Y , while
D
Y
aims to distinguish between translated samples G(x) and real samples y[distinguishes between the first generated image and a third real image belonging to a third category to generate a first discrimination value]. G aims to minimize this objective against an adversary D that tries to maximize it, i.e.,
min
G
max
D
Y
L
G
A
N
G
,
D
Y
,
X
,
Y
[and the first generator updates the parameter of the first generator according to the first discrimination value].”).
Regarding claim 4, Zhu in view of Donahue, Niu and Singh teaches the method for training a asymmetric generative adversarial network to generate an image according to claim 3, wherein the first discriminator is characterized as a plurality of third neural network weights, and the first discriminator updates a parameter of the first discriminator according to the first discrimination value(Zhu, pgs. 4-5, “[W]e express the objective as:
L
G
A
N
G
,
D
Y
,
X
,
Y
=
E
y
~
p
d
a
t
a
(
y
)
log
D
Y
y
+
E
x
~
p
d
a
t
a
(
x
)
[
l
o
g
(
1
-
D
Y
(
G
x
)
]
[according to the first discrimination value]…[o]ur full objective is:
L
G
,
F
,
D
X
,
D
Y
=
L
G
A
N
G
,
D
Y
,
X
,
Y
+
L
G
A
N
F
,
D
X
,
Y
,
X
+
λ
L
c
y
c
(
G
,
F
)
…[w]e aim to solve:
G
*
,
F
*
=
arg
min
G
,
F
max
D
x
,
D
Y
L
(
G
,
F
,
D
X
,
D
Y
)
[the first discriminator updates a parameter of the first discriminator]” & Zhu, pg.18, “For discriminator networks, we use
70
×
70
PatchGAN…[l]et Ck denote a 4
×
4 Convolution-InstanceNorm-LeakyReLU layer with k filters and stride 2. After the last layer, we apply a convolution to produce a 1-dimensional output. We do not use InstanceNorm for the first C64 layer…[t]he discriminator architecture is: C64-C128-C256-C512[the first discriminator is characterized as a plurality of third neural network weights].”).4
Regarding claim 5, Zhu in view of Donahue, Niu and Singh teaches the method for training a asymmetric generative adversarial network to generate an image according to claim 2, wherein the first generator performs a subtraction between the first real image and the first reconstructed image to generate the first value, and performs a subtraction between the second real image and the second reconstructed image to generate the second value(Zhu, pgs. 4-5, “[A]s shown in Figure 3(b), for each image x from domain X [the first real image, the second real image], the image translation cycle should be able to bring x back to the original image…[w]e can incentivize this behavior using a cycle consistency loss:
L
c
y
c
G
,
F
=
E
x
~
p
d
a
t
a
(
x
)
[
F
G
x
-
x
1
]
[the first generator performs a subtraction between the first real image and the first reconstructed image to generate the first value, and performs a subtraction between the second real image and the second reconstructed image to generate the second value]” & Zhu, pgs. 4-5, As fig. 4 details(below):
PNG
media_image5.png
547
424
media_image5.png
Greyscale
The four different input images x [the first real image, the second real image]that are given to the first generator G and then reconstructed by F(G(x)) [the first reconstructed image, the second reconstructed image]).
Regarding claim 6, Zhu in view of Donahue, Niu and Singh teaches the method for training a asymmetric generative adversarial network to generate an image according to claim 1, wherein the first generator is characterized as a plurality of first neural network weights, wherein updating the parameter of the first generator comprises: updating the plurality of first neural network weights(Zhu, pg.18, “We train our networks from scratch, with a learning rate of 0.0002… [w]e keep the same learning rate for the first 100 epochs and linearly decay the rate to zero over the next 100 epochs. Weights are initialized from a Gaussian distribution
N
0
,
0.02
[wherein updating the parameter of the first generator comprises: updating the plurality of first neural network weights]…[w]e use 6 residual blocks for 128
×
128 training images…[l]et c7s1-k denote a 7
×
7 Convolution-InstanceNorm-ReLU layer with k filters and stride 1. dk denotes a 3
×
3 Convolution-InstanceNorm-ReLU layer with k filters and stride 2…Rk denotes a residual block that contains two 3
×
3 convolutional layers with the same number of filters on both layer. uk denotes a 3
×
3 fractional-strided-Convolution- InstanceNorm-ReLU layer with k filters and stride
1
2
. The network with 6 residual blocks consists of:c7s1 64,d128,d256,R256,R256,R256,R256,R256,R256,u128,u64,c7s1-3[the first generator is characterized as a plurality of first neural network weights]”)5 while minimizing a first generation loss function(Zhu, pg. 5, “We apply two techniques from recent works to stabilize our model training procedure. First, for
L
G
A
N
… we replace the negative log likelihood objective by a least-squares loss… for a GAN loss
L
G
A
N
(
G
,
D
,
X
,
Y
)
we train the G to minimize
E
x
~
p
d
a
t
a
(
x
)
[
D
G
x
-
1
2
]
[while minimizing a first generation loss function]….”).
Regarding claim 7, Zhu in view of Donahue, Niu and Singh teaches the method for training a asymmetric generative adversarial network to generate an image according to claim 1, wherein the second generator receives the third real image and generates a third generated image accordingly, and the first generator receives the third generated image and generates a third reconstructed image accordingly(Zhu, pgs. 3-4, see also fig. 3, ““[G]iven training samples
x
i
i
=
1
N
∈
X
and
y
j
j
=
1
M
∈
Y
. As illustrated in Figure 3 (a), our model includes two mappings
G
:
X
→
Y
and
F
:
Y
→
X
.” And as fig. 3(c) details (below)
PNG
media_image6.png
330
398
media_image6.png
Greyscale
in using the backward cycle-consistency loss: the second generator (i.e., F) receives a real image (i.e.,
y
∈
Y
) and generates a generated image (i.e.,
x
^
in fig. 3(c)) and the first generator (i.e., G in fig. 3(c)) receives the generated image (i.e.,
x
^
in fig. 3(c)) and generates a reconstructed image (i.e.,
y
^
∈
Y
)[wherein the second generator receives the third real image and generates a third generated image accordingly, and the first generator receives the third generated image and generates a third reconstructed image accordingly]),6
the second discriminator distinguishes between the third generated image and the first real image to generate a second discrimination value(Zhu, pgs. 4-5, “We introduce a similar adversarial loss for the mapping function
F
:
Y
→
X
and its discriminator
D
X
[the second discriminator]as well: i.e.,
min
F
max
D
X
L
G
A
N
(
F
,
D
X
,
Y
,
X
)
[distinguishes between the third generated image and the first real image to generate a second discrimination value]” & Zhu, pg. 3, As fig. 3(a) and fig. 3(b) detail the second discriminator
D
X
distinguishes between the third generated image
X
^
and the first real image X[distinguishes between the third generated image and the first real image to generate a second discrimination value]),
and the second generator executes an operation to generate a third value according to the third real image and the third reconstructed image, and updates a parameter of the second generator according to at least one of the second discrimination value and the third value(Zhu, pgs. 4-5, “[A]s illustrated in Figure 3(c)…[w]e can incentivize this behavior using a cycle consistency loss:
L
c
y
c
G
,
F
=
E
y
~
p
d
a
t
a
(
y
)
[
G
F
y
-
y
1
]
[ and the second generator executes an operation to generate a third value according to the third real image and the third reconstructed image]…[o]ur full objective is:
L
G
,
F
,
D
X
,
D
Y
=
L
G
A
N
G
,
D
Y
,
X
,
Y
+
L
G
A
N
F
,
D
X
,
Y
,
X
+
λ
L
c
y
c
(
G
,
F
)
…[w]e aim to solve:
G
*
,
F
*
=
arg
min
G
,
F
max
D
x
,
D
Y
L
(
G
,
F
,
D
X
,
D
Y
)
[and updates a parameter of the second generator according to at least one of the second discrimination value and the third value]”).
Regarding claim 8, Zhu in view of Donahue, Niu and Singh teaches the method for training a asymmetric generative adversarial network to generate an image according to claim 7, wherein the second discriminator is characterized as a plurality of fourth neural network weights, and the second discriminator updates a parameter of the second discriminator according to the second discrimination value(Zhu, pgs. 4-5, “We introduce a similar adversarial loss for the mapping function
F
:
Y
→
X
and its discriminator
D
X
[the second discriminator]as well: i.e.,
min
F
max
D
X
L
G
A
N
(
F
,
D
X
,
Y
,
X
)
[and the second discriminator updates a parameter of the second discriminator according to the second discrimination value]” & Zhu, pg.18, “For discriminator networks, we use
70
×
70
PatchGAN…[l]et Ck denote a 4
×
4 Convolution-InstanceNorm-LeakyReLU layer with k filters and stride 2. After the last layer, we apply a convolution to produce a 1-dimensional output. We do not use InstanceNorm for the first C64 layer…[t]he discriminator architecture is: C64-C128-C256-C512[the second discriminator is characterized as a plurality of fourth neural network weights].)7
Regarding claim 9, Zhu in view of Donahue, Niu and Singh teaches the method for training a asymmetric generative adversarial network to generate an image according to claim 7, wherein the second generator is characterized as a plurality of second neural network weights, and updating the parameter of the second generator comprises: updating the plurality of second neural network weights(Zhu, pg.18, “We train our networks from scratch, with a learning rate of 0.0002… [w]e keep the same learning rate for the first 100 epochs and linearly decay the rate to zero over the next 100 epochs. Weights are initialized from a Gaussian distribution
N
0
,
0.02
[and updating the parameter of the second generator comprises: updating the plurality of second neural network weights]…[w]e use 6 residual blocks for 128
×
128 training images…[l]et c7s1-k denote a 7
×
7 Convolution-InstanceNorm-ReLU layer with k filters and stride 1. dk denotes a 3
×
3 Convolution-InstanceNorm-ReLU layer with k filters and stride 2…Rk denotes a residual block that contains two 3
×
3 convolutional layers with the same number of filters on both layer. uk denotes a 3
×
3 fractional-strided-Convolution- InstanceNorm-ReLU layer with k filters and stride
1
2
. The network with 6 residual blocks consists of:c7s1 64,d128,d256,R256,R256,R256,R256,R256,R256,u128,u64,c7s1-3[wherein the second generator is characterized as a plurality of second neural network weights]”)8
while minimizing a second generation loss function(Zhu, pgs. 4-5, “We can incentivize this behavior using a cycle consistency loss:
L
c
y
c
G
,
F
=
E
y
~
p
d
a
t
a
(
y
)
[
G
F
y
-
y
1
]
…[o]ur full objective is:
L
G
,
F
,
D
X
,
D
Y
=
L
G
A
N
G
,
D
Y
,
X
,
Y
+
L
G
A
N
F
,
D
X
,
Y
,
X
+
λ
L
c
y
c
(
G
,
F
)
…[w]e aim to solve:
G
*
,
F
*
=
arg
min
G
,
F
max
D
x
,
D
Y
L
(
G
,
F
,
D
X
,
D
Y
)
[while minimizing a second generation loss function]”).
Regarding claim 10, Zhu in view of Donahue, Niu and Singh teaches the method for training a asymmetric generative adversarial network to generate an image according to claim 7, wherein the second generator performs a subtraction between the third real image and the third reconstructed image to generate the third value(Zhu,, pgs. 4-5, “We can incentivize this behavior using a cycle consistency loss:
L
c
y
c
G
,
F
=
E
y
~
p
d
a
t
a
(
y
)
[
G
F
y
-
y
1
] [the second generator performs a subtraction between the third real image and the third reconstructed image to generate the third value]).
Regarding claim 11, Zhu in view of Donahue, Niu and Singh teaches the method for training a asymmetric generative adversarial network to generate an image according to claim 1, wherein the asymmetric generative adversarial network executes a plurality of iterative operations to train the first generator, the second generator, the first discriminator, and the second discriminator(Zhu, pg. 5, “We apply two techniques from recent works to stabilize our model training procedure. First, for
L
G
A
N
(Equation 1), we replace the negative log likelihood
objective by a least-squares loss…[t]his loss is more stable during training and generates higher quality results…[s]econd, to reduce model oscillation…[we] update the discriminators using a history of generated images rather than the ones produced by the latest generators. We keep an image buffer that stores the 50 previously created images…we set
λ
=
10
in Equation 3[i.e.,
L
G
,
F
,
D
X
,
D
Y
=
L
G
A
N
G
,
D
Y
,
X
,
Y
+
L
G
A
N
F
,
D
X
,
Y
,
X
+
λ
L
c
y
c
(
G
,
F
)
[ the first generator, the second generator, the first discriminator, and the second discriminator]. We use the Adam solver…with a batch size of 1. All networks were trained from scratch with a learning rate of 0.0002. We keep the same learning rate for the first 100 epochs and linearly decay the rate to zero over the next 100 epochs[wherein the asymmetric generative adversarial network executes a plurality of iterative operations to train the first generator, the second generator, the first discriminator, and the second discriminator]”).
Regarding claim 12, Zhu in view of Donahue, Niu and Singh teaches the method for training a asymmetric generative adversarial network to generate an image according to claim 11, wherein the iterative operations comprise: when executing a first iterative operation, updating the first discriminator and the first generator according to the first real image(Zhu, pgs. 4-5, “[G]iven training samples
x
i
i
=
1
N
∈
X
…[w]e denote the data distribution as
x
~
p
d
a
t
a
(
x
)
[ the first real image]…[f]or the mapping function
G
:
X
→
Y
and its discriminator
D
Y
… G aims to minimize this objective against an adversary D that tries to maximize it, i.e.,
min
G
max
D
Y
L
G
A
N
(
G
,
D
Y
,
X
,
Y
)
[updating the first discriminator and the first generator according to the first real image]… [a]ll networks were trained from scratch…[w]e keep the same learning rate for the first 100 epochs and linearly decay the rate to zero over the next 100 epochs[when executing a first iterative operation]”);
when executing a second iterative operation, updating the first generator according to the second real image(Zhu, pgs. 4-5, “[G]iven training samples
x
i
i
=
1
N
∈
X
…[w]e denote the data distribution as
x
~
p
d
a
t
a
(
x
)
[ the second real image]…[f]or the mapping function
G
:
X
→
Y
and its discriminator
D
Y
… G aims to minimize this objective against an adversary D that tries to maximize it, i.e.,
min
G
max
D
Y
L
G
A
N
(
G
,
D
Y
,
X
,
Y
)
[updating the first generator according to the second real image]… [a]ll networks were trained from scratch…[w]e keep the same learning rate for the first 100 epochs and linearly decay the rate to zero over the next 100 epochs[when executing a second iterative operation]”);
and when executing a third iterative operation, updating the second discriminator and the second generator according to the third real image(Zhu, pgs. 4-5, “[G]iven training samples…
y
j
j
=
1
M
∈
Y
. We denote the data distribution as…
y
~
p
d
a
t
a
(
y
)
[ the third real image]…We introduce a similar adversarial loss for the mapping function
F
:
Y
→
X
and its discriminator
D
X
as well i.e.,
min
F
max
D
X
L
G
A
N
(
F
,
D
X
,
X
,
Y
)
[updating the second discriminator and the second generator according to the third real image]… [a]ll networks were trained from scratch…[w]e keep the same learning rate for the first 100 epochs and linearly decay the rate to zero over the next 100 epochs[and when executing a third iterative operation]”).
Regarding claim 13 Donahue teaches a storage device, configured to store a real image data set and one or more instructions; and a processor, coupled to the storage device(Donahue, para. 0032, see also fig. 1, “When software is used, the operations performed by Image reconstruction training system 100 can be implemented in program code configured to run on hardware, such as a processor unit. When firmware is used, the operations performed by Image reconstruction training system 100 can be implemented in program code and data and stored in persistent memory to run on a processor unit[a storage device, configured to store a real image data set and one or more instructions; and a processor, coupled to the storage device]”) and for all other claim limitations of claim 13 they are rejected on the same basis as claim 1 since they are analogous claims.
Referring to dependent claims 14-24 they are rejected on the same basis as dependent claims 2-12 since they are analogous claims.
Regarding claim 25, Zhu in view of Donahue, Niu and Singh teaches the method for training asymmetric generative adversarial network to generate an image according to claim 1, wherein an error rate of the first discriminator is increased(Zhu, pg., 4, “We apply adversarial losses...G aims to minimize this objective against an adversary D that tries to maximize it, i.e.,
min
G
max
D
y
L
G
A
N
(
G
,
D
Y
,
X
,
Y
)
.”).9
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ADAM C STANDKE whose telephone number is (571)270-1806. The examiner can normally be reached Gen. M-F 9-9PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michael J Huntley can be reached at (303) 297-4307. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Adam C Standke/
Primary Examiner
Art Unit 2129
1 Examiner Remark: the training samples
x
i
i
=
1
N
∈
X
represent the set of real images ranging from i=1,…,N taken from the real image domain of X and
y
j
j
=
1
M
∈
Y
represent the set of real images ranging from j=1,…, M taken from the real image domain of Y. And as detailed by Zhu in section 5.1.1 Evaluation Metrics, footnote 2, “[w]e train all the models on 256 × 256 images….” Zhu, pg. 5.
2 Examiner Remarks: The claim elements that are not bolded and contained with brackets are claim elements not taught by Zhu
3 Examiner Remark: the training samples
x
i
i
=
1
N
∈
X
represent the set of real images ranging from i=1,…,N taken from the real image domain of X and
y
j
j
=
1
M
∈
Y
represent the set of real images ranging from j=1,…, M taken from the real image domain of Y. And as detailed by Zhu in section 5.1.1 Evaluation Metrics, footnote 2, “[w]e train all the models on 256 × 256 images….” Zhu, pg. 5.
4 Examiner Remark: For both generative networks G and F and for both discriminator networks
D
X
and
D
y
while they share the same neural network architecture the values of their weights are different due to the training/optimization and how their weights were initially initialized. As Zhu pg. 18 details, “we divide the objective by 2 while optimizing D, which slows down the rate at which D learns, relative to the rate of G…[and] [w]eights are initialized from a Gaussian distribution
N
(
0
,
0.02
)
.”
5 Examiner Remark: For both generative networks G and F and for both discriminator networks
D
X
and
D
y
while they share the same neural network architecture the values of their weights are different due to the training/optimization and how their weights were initially initialized. As Zhu pg. 18 details, “we divide the objective by 2 while optimizing D, which slows down the rate at which D learns, relative to the rate of G…[and] [w]eights are initialized from a Gaussian distribution
N
(
0
,
0.02
)
.”
6 Examiner Remark: the training samples
x
i
i
=
1
N
∈
X
represent the set of real images ranging from i=1,…,N taken from the real image domain of X and
y
j
j
=
1
M
∈
Y
represent the set of real images ranging from j=1,…, M taken from the real image domain of Y. And as detailed by Zhu in section 5.1.1 Evaluation Metrics, footnote 2, “[w]e train all the models on 256 × 256 images….” Zhu, pg. 5.
7 Examiner Remark: For both generative networks G and F and for both discriminator networks
D
X
and
D
y
while they share the same neural network architecture the values of their weights are different due to the training/optimization and how their weights were initially initialized. As Zhu pg. 18 details, “we divide the objective by 2 while optimizing D, which slows down the rate at which D learns, relative to the rate of G…[and] [w]eights are initialized from a Gaussian distribution
N
(
0
,
0.02
)
.”
8 Examiner Remark: For both generative networks G and F and for both discriminator networks
D
X
and
D
y
while they share the same neural network architecture the values of their weights are different due to the training/optimization and how their weights were initially initialized. As Zhu pg. 18 details, “we divide the objective by 2 while optimizing D, which slows down the rate at which D learns, relative to the rate of G…[and] [w]eights are initialized from a Gaussian distribution
N
(
0
,
0.02
)
.”
9 Examiner Remarks: Para.[0065] of Applicant’s Specification states that “the training goal of the first generator G1 is to increase the error rate of the first discriminator DB, that is, try to deceive the first discriminator DB...” which under the BRI in light of Applicant’s Specification is taught by Zhu’s adversarial loss function.