Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
Allowable Subject Matter
Claims 3 and 13 are objected to as being dependent upon a rejected base claim but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1 and 11 are rejected under 35 U.S.C. 103 as being unpatentable over Wang (Pub No. US 20230095041 A1) in view of Mironica (Pub No. US 20240420394 A).
As per claim 1, Wang teaches the claimed:
1. A method related to a data generation framework, implemented by a processor, the method comprising: obtaining a latent representation from training data by an initial encoder; (Wang [0015]-[0016]: “[0015] Afterwards, it follows a propagation of the given image (x) through an encoder of the generator. Under the term propagation, it can be understood that an input is proceeded by the encoder or generator, etc. according to a sequence of processing steps defined by its (internal) structure. Then, a first part of a latent feature map (FG.sub.G) outputted by the encoder is replaced by the generated target content code ({tilde over (c)}).
[0016] Afterwards, a propagation of the replaced first part of the latent feature map (FG.sub.A={tilde over (c)}) through a first decoder of the generator is carried out, wherein the target style code ({tilde over (s)}) is injected into the first decoder at the same time. A second part of the latent feature map (BG.sub.G) is propagated through a second decoder of the generator. The second part of the latent feature map (BG.sub.G) can correspond to the remaining part of the latent feature map without the first part of the latent feature map (FG.sub.G). The injection of the target style code ({tilde over (s)}) is carried out during the propagation. This means that while the first part of the replaced latent feature map (FG.sub.G) is iteratively proceeded by layers of the first decoder, at predefined layers, the target style code ({tilde over (s)}) is injected as an additional input of the respective layer such that the information of the target style code ({tilde over (s)}) are co-proceeded by the respective layer. More precisely, the style code is the input to a normalization layer and is used to compute mean and variance to adjust the normalization accordingly.” Wang [0036]: “Furthermore, according to an example embodiment of the present invention, it is provided that wherein the outputted transformed image of the generator is added to a training data set, wherein an optical surface defect detector is trained depending on this augmented training data set”).
combining the latent representation and first noise data to generate a noisy latent representation;
(Wang [0069]-[0070]: “[0069] The Mapping network (3) computes the content c and the style s vector from random noise z, which can provide novel contents and style vectors that are not in the training set. On the other hand, the Style-Content encoder (4) extracts the content c and the style s vector from a given reference image x.sub.ref, which provides a way to transfer an existing content (i.e. defects) in the training set onto an image with different background.
[0070] In other words, the Mapping network (3) is configured to receive random noise z and a domain y, and generates a latent code, which is forwarded through two separate parts S.sub.M, C.sub.M of the Mapping network (3). Then the two separate parts S.sub.M, C.sub.M output the contents and style vectors respectively. The parts S.sub.M, C.sub.M can be a MLP and a decoder respectively. In a further embodiment of the Mapping network (3), noise is injected into the separate part C.sub.M. The noise injection significantly increases the diversity of the output content.”).
Wang alone does not explicitly teach the claimed limitations.
However, Wang in combination with Mironica teaches the claimed:
inputting the noisy latent representation to a prediction model, and outputting an initial prediction corresponding to the noisy latent representation (Mironica [0108]-[0109]: “[0108] Diffusion models work by iteratively adding noise to the data during a forward process and then learning to recover the data by denoising the data during a reverse process. For example, during training, guided latent diffusion model 900 may take an original image 905 in a pixel space 910 as input and apply an image encoder 915 to convert original image 905 into original image features 920 in a latent space 925. Then, a forward diffusion process 930 gradually adds noise to the original image features 920 to obtain noisy features 935 (also in latent space 925) at various noise levels.
[0109] Next, a reverse diffusion process 940 (e.g., a U-Net ANN) gradually removes the noise from the noisy features 935 at the various noise levels to obtain denoised image features 945 in latent space 925. In some examples, the denoised image features 945 are compared to the original image features 920 at each of the various noise levels, and parameters of the reverse diffusion process 940 of the diffusion model are updated based on the comparison. Finally, an image decoder 950 decodes the denoised image features 945 to obtain an approximation of the existing noise in the input image in pixel space 910. Then the output image 955 is obtained as the difference between the input image 910 from which a controllable fraction of the previously predicted noise is subtracted. In some cases, an output image 955 is created at each of the various noise levels. The output image 955 can be compared to the original image 905 to train the reverse diffusion process 940.”).
by referring to first semantic mask data by the prediction model, wherein the first semantic mask data defines at least one first semantic category for the training data; (Mironica [0053]: “Segmentation component 210 is configured to perform panoptic segmentation on the input image. Panoptic segmentation involves both semantic segmentation and instance segmentation and is considered a “unified segmentation” approach. The objective of panoptic segmentation is to extract, label, and classify objects in the image.” Mironica [0075]: “At operation 325, the system performs panoptic segmentation on image. The objective of panoptic segmentation is to extract, label, and classify objects in the image. In some cases, panoptic segmentation produces one or more additional images that are segmentation masks. In some cases, operation 325 includes performing a simultaneous and unified segmentation of both the background (e.g., the surrounding context, such as the sky, grass, or pavement) and the objects (e.g., the instances, such as people, cars, or buildings). For example, operation 325 performs a simultaneous and unified segmentation of both the background and the text. In some cases, operation 325 includes analyzing an image and producing a label map that segments the image into multiple regions, each with a corresponding class label.”
Mironica [0106]: “Diffusion models are a class of generative neural networks which can be trained to generate new data with features similar to features found in training data. In particular, diffusion models can be used to generate novel images. Diffusion models can be used for various image generation tasks including image super-resolution, generation of images with perceptual metrics, conditional generation (e.g., generation based on text guidance), image inpainting, and image manipulation.”).
and updating the prediction model according to a prediction error between the initial prediction and the first noise data to generate a trained prediction model.
(Mironica [0109]: “Next, a reverse diffusion process 940 (e.g., a U-Net ANN) gradually removes the noise from the noisy features 935 at the various noise levels to obtain denoised image features 945 in latent space 925. In some examples, the denoised image features 945 are compared to the original image features 920 at each of the various noise levels, and parameters of the reverse diffusion process 940 of the diffusion model are updated based on the comparison. Finally, an image decoder 950 decodes the denoised image features 945 to obtain an approximation of the existing noise in the input image in pixel space 910. Then the output image 955 is obtained as the difference between the input image 910 from which a controllable fraction of the previously predicted noise is subtracted. In some cases, an output image 955 is created at each of the various noise levels. The output image 955 can be compared to the original image 905 to train the reverse diffusion process 940.” Mironica [0049]: “During the training process, these weights are adjusted to improve the accuracy of the result (i.e., by minimizing a loss function corresponding to the difference between the current result and the target result). The weight of an edge increases or decreases the strength of the signal transmitted between nodes. In some cases, nodes have a threshold below which a signal is not transmitted at all. In some examples, the nodes are aggregated into layers. Different layers perform different transformations on their inputs. The initial layer is known as the input layer and the last layer is known as the output layer. In some cases, signals traverse one or more layers multiple times.” The loss corresponds to the error. This can be combined with the adjusting of the model of Wang, which would be updating it. Wang [0021]: “Finally, the generator, discriminator, Mapping network and the Style-Content encoder are optimized subject to an objective that the sum of the adversarial loss and the style-content reconstruction loss is minimized by the generator and the Mapping network and the Style-Content encoder and the sum the losses is maximized by the discriminator. The optimization can be carried out by gradient descent, wherein parameters of the models are adjusted depending on estimated gradients.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the prediction model based on noise data and semantic segmentation masks as taught by Mironica with the system of Wang in order to use machine learning to analyze the features of the images of different styles of Wang and to track the effects of the noise data on the overall image.
As per claim 11, this claim is similar in scope to limitations recited in claim 1 and thus is rejected under the same rationale.
Claims 2 and 12 are rejected under 35 U.S.C. 103 as being unpatentable over Wang in view of Mironica and further in view of Koh (US 20230072293 A1) and further in view of Amirghodshi (Pub no. US 20240362757 A1).
As per claim 2, Wang alone does not explicitly teach the claimed limitations.
However, Wang in combination with Koh and Mironica teaches the claimed:
2. The method related to the data generation framework according to claim 1, wherein the prediction model comprises a first encoder and a first decoder, the first encoder is connected to the first decoder, the first encoder comprises at least one first encoder block, the
first decoder comprises at least one first decoder block, (The examiner is treating an encoder block as just a portion of the data flow with an encoder at a certain phase of the process. Wang [0123]: “At operation 1205, the user initializes an untrained model. Initialization can include defining the architecture of the model and establishing initial values for the model parameters. In some cases, the initialization can include defining hyper-parameters such as the number of layers, the resolution and channels of each layer blocks, the location of skip connections, and the like.” Wang [0015]: “Afterwards, it follows a propagation of the given image (x) through an encoder of the generator. Under the term propagation, it can be understood that an input is proceeded by the encoder or generator, etc. according to a sequence of processing steps defined by its (internal) structure. Then, a first part of a latent feature map (FG.sub.G) outputted by the encoder is replaced by the generated target content code ({tilde over (c)}). [0016] Afterwards, a propagation of the replaced first part of the latent feature map (FG.sub.A={tilde over (c)}) through a first decoder of the generator is carried out, wherein the target style code ({tilde over (s)}) is injected into the first decoder at the same time. A second part of the latent feature map (BG.sub.G) is propagated through a second decoder of the generator. The second part of the latent feature map (BG.sub.G) can correspond to the remaining part of the latent feature map without the first part of the latent feature map (FG.sub.G). The injection of the target style code ({tilde over (s)}) is carried out during the propagation. This means that while the first part of the replaced latent feature map (FG.sub.G) is iteratively proceeded by layers of the first decoder, at predefined layers, the target style code ({tilde over (s)}) is injected as an additional input of the respective layer such that the information of the target style code ({tilde over (s)}) are co-proceeded by the respective layer. More precisely, the style code is the input to a normalization layer and is used to compute mean and variance to adjust the normalization accordingly.” Wang [0017]: “Afterwards, a merging the output of the first and second decoder of the generator to an output image (G(x, {tilde over (s)}, {tilde over (c)})) of the generator is carried out. Preferably, the merging is carried out by superposing but outputs such that the output of the first decoder is put on the output of the second decoder. Preferably, this can be done by a concatenation, wherein the outputs are 3D and they are concatenated along the third dimension.”).
and the step of outputting the initial prediction corresponding to the noisy latent representation by referring to the first semantic mask data by the prediction model comprises: (The first stage is the initial predication. Mironica [0125]: “At operation 1215, the system at each stage n, starting with stage N, a reverse diffusion process is used to predict the image or image features at stage n-1. For example, the reverse diffusion process can predict the noise that was added by the forward diffusion process, and the predicted noise can be removed from the image to obtain the predicted image. In some cases, an original image is predicted at each stage of the training process.” Mironica [0110]: “In some cases, image encoder 915 and image decoder 950 are pre-trained prior to training the reverse diffusion process 940. In some examples, they are trained jointly, or the image encoder 915 and image decoder 950 and fine-tuned jointly with the reverse diffusion process 940.” Each stage has its own encoder and decoder and within the process they output the respective data at that stage.”
Koh claim 3: “The computing system of claim 1, wherein the one or more spatial observations comprises one or more spatial semantic segmentation images, the one or more guidance spatial observations comprises one or more guidance semantic segmentation images, and the one or more predicted spatial observations comprises one or more predicted semantic segmentation images.” The segmentation images can be combined with the masks of Mironica.
Koh [0080]: “A latent spatial noise tensor (e.g., random noise tensor) z.sub.t 213 can be introduced into the model in between the encoder model 207 and decoder model 209 by concatenating 208 the latent spatial noise tensor with the feature map generated by the encoder model. The latent spatial noise tensor 213 can be generated by a machine-learned noise generation model 214. The decoder model 209 can process the combined feature map to generate the one or more predicted spatial observations (e.g., predicted segmentation image(s)
PNG
media_image1.png
298
298
media_image1.png
Greyscale
and/or predicted depth image(s)
PNG
media_image2.png
298
298
media_image2.png
Greyscale
) of the environment from the target camera pose 212. In some implementations, the output of the decoder model 209 can be processed with a convolution block 211. Predicted semantic segmentation observations can be processed with a C-way softmax function. Predicted depth observations can be processed with a sigmoid function and normalized in the range (0,1)…”.).
inputting the noisy latent representation to the first encoder; (Amirghodsi [0055]: “In some cases, the mask-robust image inpainting system 102 generates or trains a GAN-based inpainting model for the task of inpainting digital images. Indeed, a generative adversarial network (“GAN”) refers to a network that is tuned or trained via an adversarial process to generate an output digital image from an input such as a noise vector. For example, a generative adversarial network includes multiple constituent neural networks such as one or more encoder neural networks and one or more generator (or decoder) neural networks. For example, an encoder neural network extracts latent code from a noise vector or from a digital image (e.g., a digital image depicting background and a foreground object). A generator neural network (or a combination of generator neural networks) generates an inpainted digital image (e.g., a digital image with some or all foreground object pixels removed or inpainted) by combining or otherwise processing extracted latent code (e.g., from the encoder neural network(s)). During training, a discriminator neural network, in competition with the generator neural network, analyzes a generated digital image to generate a realism prediction by determining whether the generated digital image is real (e.g., from a set of stored digital images) or fake (e.g., not from the set of stored digital images). Using one or more loss functions, the discriminator neural network also informs modification of parameters of encoder neural network(s), generator neural network(s), and/or the discriminator neural network to eventually generate digital images that fool the discriminator neural network into indicating that a generated digital image is a real digital image.”).
inputting the first semantic mask data and feature data to one of the at least one first decoder block, wherein the feature data is an output of one of the at least one first encoder block or an output of another one of the at least one first decoder block;
and outputting the initial prediction by the first decoder. (Koh [0080]: “A latent spatial noise tensor (e.g., random noise tensor) z.sub.t 213 can be introduced into the model in between the encoder model 207 and decoder model 209 by concatenating 208 the latent spatial noise tensor with the feature map generated by the encoder model. The latent spatial noise tensor 213 can be generated by a machine-learned noise generation model 214. The decoder model 209 can process the combined feature map to generate the one or more predicted spatial observations (e.g., predicted segmentation image(s)
PNG
media_image3.png
298
298
media_image3.png
Greyscale
and/or predicted depth image(s)
PNG
media_image4.png
298
298
media_image4.png
Greyscale
) of the environment from the target camera pose 212. In some implementations, the output of the decoder model 209 can be processed with a convolution block 211. Predicted semantic segmentation observations can be processed with a C-way softmax function. Predicted depth observations can be processed with a sigmoid function and normalized in the range (0,1). At each step during an inference phase, the predicted observations can be back-projected and added to the three-dimensional point cloud to assist future predictions.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the prediction model based on noise data as taught by Mironica with the system of Wang in order to more clearly predict the effect of an image style change through a trained machine learning model. The motivation of claim 1 is incorporated herein.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the processing of the feature map with noise data by the decoder as taught by Koh with the system of Wang as modified by Mironica in order to output the prediction based on the image and the noise data for the generated images of Wang. The motivation of claim 1 is incorporated herein.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the input of the noise data into the encoder as taught by Amirghodshi with the system of Wang as modified by Mironica in order to generate a more accurate output accounting for noise. The motivation of claim 1 is incorporated herein.
As per claim 12, this claim is similar in scope to limitations recited in claim 2, and thus is rejected under the same rationale.
Claims 4 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over Wang in view of Mironica and further in view of Yoshida (US 20230402249 A1).
As per claim 4, Wang alone does not explicitly teach the claimed limitations.
However, Wang in combination with Yoshida and Mironica teaches the claimed:
4. The method related to the data generation framework according to claim 1, wherein the prediction error is a sum of a first error and a second error, (Yoshida teaches a learning model that analyzes the effect of errors s in image data, including noise. Yoshida [0031]: “A feature value calculation unit 11, an image information reduction unit 12, a statistic value estimation unit 13, and a defect detection unit 14 are configured by cooperation of the computing means and the storage means of the defect inspection apparatus. The feature value calculation unit 11, the image information reduction unit 12, and the statistic value estimation unit 13 are trained models with parameters learned by machine learning. The parameters are stored as model parameters 10 in the storage means of the defect inspection apparatus.” Yoshida teaches error between a specific image value between an estimation and a learning image. It would be obvious to apply this measurement of error to other features. Yoshida [0045]: “The error between the estimated luminance value distribution and the learning image 3 is calculated by a model evaluation unit 31 (S105). In other words, the model evaluation unit 31 performs error evaluation based on the image statistic value estimated based on the learning image 3 and the original learning image 3. For example, for each pixel, a predetermined comparison target value (for example, mean value of Gaussian distribution) can be calculated from the estimated image statistic value, and the difference between the comparison target value and the luminance value of the corresponding pixel in the learning image 3 can be calculated as a pixel-specific error. Then, the sum of the absolute values of the pixel-specific errors or the sum of the squares of the pixel-specific errors can be regarded as the error between the estimated luminance value distribution and the learning image 3.”).
the initial prediction comprises a predicted mean and a predicted variation, (Wang teaches measuring the mean and variance through the processing of image data. Wang [0016]: “Afterwards, a propagation of the replaced first part of the latent feature map (FG.sub.A={tilde over (c)}) through a first decoder of the generator is carried out, wherein the target style code ({tilde over (s)}) is injected into the first decoder at the same time. A second part of the latent feature map (BG.sub.G) is propagated through a second decoder of the generator. The second part of the latent feature map (BG.sub.G) can correspond to the remaining part of the latent feature map without the first part of the latent feature map (FG.sub.G). The injection of the target style code ({tilde over (s)}) is carried out during the propagation. This means that while the first part of the replaced latent feature map (FG.sub.G) is iteratively proceeded by layers of the first decoder, at predefined layers, the target style code ({tilde over (s)}) is injected as an additional input of the respective layer such that the information of the target style code ({tilde over (s)}) are co-proceeded by the respective layer. More precisely, the style code is the input to a normalization layer and is used to compute mean and variance to adjust the normalization accordingly.”).
and the step of updating the prediction model according to the prediction error between the initial prediction and the first noise data comprises: calculating a difference between the predicted mean and a mean of the first noise data to determine the first error; (Yoshida [0045]: “The error between the estimated luminance value distribution and the learning image 3 is calculated by a model evaluation unit 31 (S105). In other words, the model evaluation unit 31 performs error evaluation based on the image statistic value estimated based on the learning image 3 and the original learning image 3. For example, for each pixel, a predetermined comparison target value (for example, mean value of Gaussian distribution) can be calculated from the estimated image statistic value, and the difference between the comparison target value and the luminance value of the corresponding pixel in the learning image 3 can be calculated as a pixel-specific error. Then, the sum of the absolute values of the pixel-specific errors or the sum of the squares of the pixel-specific errors can be regarded as the error between the estimated luminance value distribution and the learning image 3.”
Mironica teaches updating the model based on comparison. Mironica [0109]: “Next, a reverse diffusion process 940 (e.g., a U-Net ANN) gradually removes the noise from the noisy features 935 at the various noise levels to obtain denoised image features 945 in latent space 925. In some examples, the denoised image features 945 are compared to the original image features 920 at each of the various noise levels, and parameters of the reverse diffusion process 940 of the diffusion model are updated based on the comparison. Finally, an image decoder 950 decodes the denoised image features 945 to obtain an approximation of the existing noise in the input image in pixel space 910. Then the output image 955 is obtained as the difference between the input image 910 from which a controllable fraction of the previously predicted noise is subtracted. In some cases, an output image 955 is created at each of the various noise levels. The output image 955 can be compared to the original image 905 to train the reverse diffusion process 940”).
and calculating a difference between the predicted variation and a variation of the first noise data to determine the second error. (Mironica [0126]: “At operation 1220, the system compares predicted image (or image features) at stage n-1 to an actual image (or image features), such as the image at stage n-1 or the original input image. For example, given observed data x, the diffusion model may be trained to minimize the variational upper bound of the negative log-likelihood-log pe (x) of the training data.” This requires determining the error to find the variation of the data.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the prediction model that measures error through comparison of a target outcome and a generated outcome as taught by Mironica with the system of Wang in order to compute the noise. The motivation of claim 1 is incorporated herein.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the sum of error data as taught by Yoshida with the system of Wang as modified by Mironica to compute the overall error from different statistical aspects of the result and use these to inform the training of the prediction model. The motivation of claim 1 is incorporated herein.
As per claim 14, this claim is similar in scope to limitations recited in claim 4, and thus is rejected under the same rationale.
Claims 5-8, 10, 15-18, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Wang in view of Mironica and further in view of Amirghodsi.
As per claim 5, Wang alone does not explicitly teach the claimed limitations.
However, Wang in combination with Mironica and Amirghodshi teaches the claimed:
5. The method related to the data generation framework according to claim 1, further comprising: inputting second noise data to the trained prediction model, (Wang [0016]: “Afterwards, a propagation of the replaced first part of the latent feature map (FG.sub.A={tilde over (c)}) through a first decoder of the generator is carried out, wherein the target style code ({tilde over (s)}) is injected into the first decoder at the same time. A second part of the latent feature map (BG.sub.G) is propagated through a second decoder of the generator. The second part of the latent feature map (BG.sub.G) can correspond to the remaining part of the latent feature map without the first part of the latent feature map (FG.sub.G). The injection of the target style code ({tilde over (s)}) is carried out during the propagation. This means that while the first part of the replaced latent feature map (FG.sub.G) is iteratively proceeded by layers of the first decoder, at predefined layers, the target style code ({tilde over (s)}) is injected as an additional input of the respective layer such that the information of the target style code ({tilde over (s)}) are co-proceeded by the respective layer. More precisely, the style code is the input to a normalization layer and is used to compute mean and variance to adjust the normalization accordingly.” Mironica [0118]: “The neural network may be trained to perform the reverse process. During the reverse diffusion process 1110, the model begins with noisy data X.sub.T, such as a noisy image 1115 and denoises the data to obtain the p (x.sub.t-1 |x.sub.t). At each step t-1, the reverse diffusion process 1110 takes x.sub.t, such as first intermediate image 1120, and t as input. Here, t represents a step in the sequence of transitions associated with different noise levels, The reverse diffusion process 1110 outputs x.sub.t-1, such as second intermediate image 1125 iteratively until x.sub.T is reverted back to x.sub.0, the original image 1130. The reverse process can be represented as:” The noise data in the intermediate states is the second noise data.).
and outputting predicted noise corresponding to the second noise data by referring to second semantic mask data by the trained prediction model, wherein the second semantic mask data defines at least one second semantic category for first generated data; (Amirghodsi [0066]: “To illustrate, in some embodiments, the mask-robust image inpainting system 102 obtains the initial mask by generating the initial mask 352 from the digital image 354 utilizing the object segmentation machine-learning model 118. For example, the mask-robust image inpainting system 102 obtains (e.g., receives, captures, etc.) the digital image 354 and identifies one or more objects depicted in the digital image 354 to remove and inpaint. For instance, the mask-robust image inpainting system 102 identifies the one or more objects to remove by receiving one or more indications (e.g., a click, a pen stroke, a cursor swipe, a bounding box, a lasso, and/or a text input identifying a semantic class) of one or more objects to remove from the digital image 354. The mask-robust image inpainting system 102 processes the digital image 354 and the indication of the one or more objects through the object segmentation machine-learning model 118 to generate the initial mask 352. The mask-robust image inpainting system 102 utilizes the object segmentation machine-learning model 118 to predict portions of the digital image 354 to be removed and replaced during inpainting (i.e., in the form of the initial mask 352).” Mironica [0075]: “At operation 325, the system performs panoptic segmentation on image. The objective of panoptic segmentation is to extract, label, and classify objects in the image. In some cases, panoptic segmentation produces one or more additional images that are segmentation masks. In some cases, operation 325 includes performing a simultaneous and unified segmentation of both the background (e.g., the surrounding context, such as the sky, grass, or pavement) and the objects (e.g., the instances, such as people, cars, or buildings). For example, operation 325 performs a simultaneous and unified segmentation of both the background and the text. In some cases, operation 325 includes analyzing an image and producing a label map that segments the image into multiple regions, each with a corresponding class label.”).
generating noise removed data according to a difference between the predicted noise and the second noise data; (Wang [0070]: “In other words, the Mapping network (3) is configured to receive random noise z and a domain y, and generates a latent code, which is forwarded through two separate parts S.sub.M, C.sub.M of the Mapping network (3). Then the two separate parts S.sub.M, C.sub.M output the contents and style vectors respectively. The parts S.sub.M, C.sub.M can be a MLP and a decoder respectively. In a further embodiment of the Mapping network (3), noise is injected into the separate part C.sub.M. The noise injection significantly increases the diversity of the output content.” The noise injection is the second noise data.
Mironica [0109]: “Next, a reverse diffusion process 940 (e.g., a U-Net ANN) gradually removes the noise from the noisy features 935 at the various noise levels to obtain denoised image features 945 in latent space 925. In some examples, the denoised image features 945 are compared to the original image features 920 at each of the various noise levels, and parameters of the reverse diffusion process 940 of the diffusion model are updated based on the comparison. Finally, an image decoder 950 decodes the denoised image features 945 to obtain an approximation of the existing noise in the input image in pixel space 910. Then the output image 955 is obtained as the difference between the input image 910 from which a controllable fraction of the previously predicted noise is subtracted. In some cases, an output image 955 is created at each of the various noise levels. The output image 955 can be compared to the original image 905 to train the reverse diffusion process 940.”).
and converting the noise removed data into the first generated data by a decoder corresponding to the initial encoder. (Wang [0028]: “Furthermore, according to an example embodiment of the present invention, it is provided that noise is injected into a decoder (C.sub.M) of the Mapping Network, which outputs the style code. Preferably, an intensity of the noise is determined by a hyperparameter that has been optimized during training of the generator.” The training of the generator corresponds to the removal of noise from the noisy features as taught above by Mironica. Mironica [0110]: “In some cases, image encoder 915 and image decoder 950 are pre-trained prior to training the reverse diffusion process 940. In some examples, they are trained jointly, or the image encoder 915 and image decoder 950 and fine-tuned jointly with the reverse diffusion process 940.” The encoders and decoders correspond to each other.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the prediction model analyzing noise as taught by Mironica with the system of Wang in order to analyze the features of the style codes taught by Wang for more accurate and cleaner image generation. The motivation of claim 1 is incorporated herein.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the semantic mask segmentation categories as taught by Amirghodsi with the system of Wang as modified by Mironica in order to better classify different portions of the images to for clearer representation and to understand the effects of noise. The motivation of claim 1 is incorporated herein.
As per claim 6, Wang alone does not explicitly teach the claimed limitations.
However, Wang in combination with Mironica teaches the claimed:
6. The method related to the data generation framework according to claim 5, further comprising: inputting the noise removed data to the trained prediction model, and outputting the predicted noise corresponding to the noise removed data by referring to the second semantic mask data by the trained prediction model. (Mironica [0109]: “Next, a reverse diffusion process 940 (e.g., a U-Net ANN) gradually removes the noise from the noisy features 935 at the various noise levels to obtain denoised image features 945 in latent space 925. In some examples, the denoised image features 945 are compared to the original image features 920 at each of the various noise levels, and parameters of the reverse diffusion process 940 of the diffusion model are updated based on the comparison. Finally, an image decoder 950 decodes the denoised image features 945 to obtain an approximation of the existing noise in the input image in pixel space 910. Then the output image 955 is obtained as the difference between the input image 910 from which a controllable fraction of the previously predicted noise is subtracted. In some cases, an output image 955 is created at each of the various noise levels. The output image 955 can be compared to the original image 905 to train the reverse diffusion process 940.” Wang [0028]: “Furthermore, according to an example embodiment of the present invention, it is provided that noise is injected into a decoder (C.sub.M) of the Mapping Network, which outputs the style code. Preferably, an intensity of the noise is determined by a hyperparameter that has been optimized during training of the generator.” The training of the generator corresponds to the removal of noise from the noisy features as taught above by Mironica. Mironica [0111]: “The reverse diffusion process 940 can also be guided based on a guidance prompt 960, such as an image, a layout, a segmentation map, etc. The guidance prompt 960 can be encoded using a multimodal encoder 965 to obtain guidance features 970 in guidance space 975. The guidance features 970 can be combined with the noisy features 935 at one or more layers of the reverse diffusion process 940 to ensure that the output image 955 includes content described by the guidance prompt 960. When generating a new modified image, for instance, the new modified image will contain features from the original image since the noisy image supplied as guidance prompt 960 is based on the original image. Guidance features 970 can be combined with the noisy features 935 using a cross-attention block within the reverse diffusion process” The segmentation map includes the mask. The mask is used to apply the noise, which informs the model. Mironica [0081]: “At operation 415, the system adds color C to a Gaussian mask to create noisy image. The operations of this step refer to, or may be performed by, a noise component as described with reference to FIG. 2. In some embodiments, this operation includes applying color C to the Gaussian mask, and combining the colored Gaussian mask with the image to create the noisy image in the Figure shown next to operation 415. Some embodiments further add fractal noise in areas underneath the region(s) of the text, or on surfaces or edges that were originally in those regions. The fractal noise will cause the generative machine learning model to add fine detail in these areas.” This mask can be the second mask applied at a later stage of the process. The noisy image was input via the gaussian mask before the noise data was removed. Thus, the removed data and its effect on the model refer to the mask.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the noise data input into a learning model based on the mask as taught by Mironica with the system of Wang in order to use machine learning to analyze the effects of noise on the image styles of Wang. The motivation of claim 1 is incorporated herein.
As per claim 16, this claim is similar in scope to limitations recited in claim 6, and thus is rejected under the same rationale.
As per claim 7, Wang teaches the claimed:
7. The method related to the data generation framework according to claim 5, further comprising: obtaining a first code from reference data by a style encoder; (Wang claim 1: “computer-implemented method of training a generator for transforming a given image according to a given target foreground domain, comprising the following steps: generating a target style code and target content code by a Mapping network depending on the given target foreground domain; propagating the given image through an encoder of the generator; replacing a first part of a latent feature map outputted by the encoder with the generated target content code; propagating the replaced first part of the latent feature map through a first decoder of the generator, wherein the target style code is injected into the first decoder…”).
obtaining a second code from latent encoding by a mapping network, wherein the first code and the second code form a style code, and the style code corresponds to at least one style option; (Wang claim 1: “…wherein the target style code is injected into the first decoder, and propagating a second part of the latent feature map through a second decoder of the generator; merging output of the first and second decoder of the generator to an output image of the generator;” Wang [0072]: “The Style-Content encoder (4) is similarly build as the Mapping Network (3) and extracts the style code s=S.sub.E(x.sub.ref) and content code c=C.sub.E(x.sub.ref) for a given image x.sub.ref of domain y. In other words, the Style-Content encoder (4) is configured to receive the reference input image x.sub.ref and extracts its style and content. Similar to the Mapping network (3), the Style-Content encoder (4) determines a latent code by a first encoder, wherein the latent code is forwarded through two separate parts S.sub.E, C.sub.E. The two separate parts S.sub.E, C.sub.E output the contents and style vectors respectively. The two separate parts S.sub.E, C.sub.E can be a second and third encoder.”).
inputting first source data to a generator, and outputting a first output corresponding to the first source data by referring to the style code by the generator; (Wang claim 1: “… input image through a discriminator, wherein the image is either the output image of the generator or an image from a training data set, wherein the discriminator outputs whether its input image is either an image of the training data set or generated by the generator;
…determining a style-content reconstruction loss by determining a difference between the target style code and an extracted style code by a Style-Content encoder depending on the output image of the generator and a difference between the target content code and an extracted content code by a Style-Content encoder depending on the output image of the generator;”).
and inputting the first output to a discriminator, and outputting a second output corresponding to the first output by the discriminator, wherein the second output comprises real or fake corresponding to the at least one style option. (Wang claim 14: “wherein the discriminator outputs whether its input image is either an image of the training data set or generated by the generator; determine an adversarial loss depending on the output of the discriminator; determine a style-content reconstruction loss by determining a difference between the target style code and an extracted style code by a Style-Content encoder depending on the output image of the generator and a difference between the target content code and an extracted content code by a Style-Content encoder depending on the output image of the generator; optimize the generator, the discriminator, the Mapping network, and the Style-Content encoder, subject to an objective that a sum of the adversarial loss and the style-content reconstruction loss is minimized by the generator and the Mapping network and the Style-Content encoder and the sum of the losses is maximized by the discriminator.”
Wang [0056]: “For generating realistic scratches on different products, the scratches may only differ in few characteristics. Therefore, it is proposed to make use of the shared characteristics of defects by transferring a stylized defect-specific content from one type of background to another.” Wang [0066]: “A discriminator (2) receives an input image x.sub.in,D and encodes it into logits D.sub.y(x) that are used to discriminate if the input image is real (i.e. from the training set) or fake (i.e. synthetically generated by the generator (1)) considering the foreground domain y. In other words, each of the logits D.sub.y(x) characterizes whether the foreground domain of the image x.sub.in,D is a real image of its foreground domain y or a synthetic image. Due to the different logits of the discriminator (2), the discriminator (2) can be referred to as a multi-task discriminator.”).
As per claim 17, this claim is similar in scope to limitations recited in claim 7, and thus is rejected under the same rationale.
As per claim 8, Wang teaches the claimed:
8. The method related to the data generation framework according to claim 7, wherein the generator comprises a second encoder and a second decoder, the second encoder is connected to the second decoder, the second encoder comprises at least one second encoder block, the second decoder comprises at least one second decoder block, (Wang [0033]-[0035]: “[0033] Afterwards, it follows a step of propagating the replaced first part of a latent feature map through the first decoder and propagating the unchanged second part of a latent feature map (BG.sub.G) through the second decoder of the generator, wherein a predefined style code is injected into the first decoder during propagating the replaced part of the latent representation through a second decoder.
[0034] Afterwards, it follows a step of merging both outputs of the first and second decoder of the generator and outputting the merged output as transformed image.
[0035] For the first and second aspect of the present invention, it is provided that the background of the image characterizes, in particular shows, a product surface and the foreground is an anomaly of the background, in particular a defect.”).
and the step of outputting the first output corresponding to the first source data by referring to the style code by the generator comprises:
fusing feature data output by one of the at least one second encoder block and the style code to generate fusion data; (Wang claim 6: “A computer-implemented method of image-to-image translation according to a given target foreground domain by a generator configured to transform a given image according to a given target foreground domain, the method comprising the following steps: receiving an image and a predefined content code and predefined style code, wherein both the predefined content code and the predefined style code characterize the target foreground domain; determining a latent feature map of the image by propagating the image through an encoder of the generator; replacing a first part of a latent feature map by the predefined content code; propagating the replaced first part of a latent feature map through a first decoder of the generator and propagating the unchanged second part of a latent feature map through a second decoder of the generator wherein the predefined style code is injected into the first decoder of the generator; merging outputs of the first decode of the generator and the second decoder of the generator; and outputting the merged outputs as a transformed image.” The merged data with partially replaced feature maps is the fusion data.).
and inputting the fusion data and another feature data to one of the at least one second decoder block, wherein the another feature data is an output of one of the at least one second encoder block or an output of another one of the at least one second decoder block. (Wang claim 6:…; replacing a first part of a latent feature map by the predefined content code; propagating the replaced first part of a latent feature map through a first decoder of the generator and propagating the unchanged second part of a latent feature map through a second decoder of the generator wherein the predefined style code is injected into the first decoder of the generator; merging outputs of the first decode of the generator and the second decoder of the generator; and outputting the merged outputs as a transformed image.” The merged data is the fusion data. Wang claim 8: “8. The method as recited in claim 7, wherein the generator, the Mapping network and the Style-Content encoder, are trained by: generating a target style code and target content code by a Mapping network depending on a given target foreground domain, propagating a given image through the encoder of the generator, replacing a first part of a first latent feature map outputted by the encoder with the generated target content code, propagating the replaced first part of the first latent feature map through the first decoder of the generator, wherein the target style code is injected into the first decoder, and propagating the second part of the first latent feature map through a second decoder of the generator, merging output of the first and second decoder of the generator to an output image of the generator, propagating an input image through a discriminator, wherein the image is either the output image of the generator or an image from a training data set…” The generation of the second part of the first latent feature map is the other feature data.).
As per claim 18, this claim is similar in scope to limitations recited in claim 8, and thus is rejected under the same rationale.
As per claim 10, Wang teaches the claimed:
10. The method related to the data generation framework according to claim 7, further comprising: obtaining a third code from second latent encoding by the mapping network; (Wang [0028]: “Furthermore, according to an example embodiment of the present invention, it is provided that noise is injected into a decoder (C.sub.M) of the Mapping Network, which outputs the style code. Preferably, an intensity of the noise is determined by a hyperparameter that has been optimized during training of the generator.” Wang [0072]: “The Style-Content encoder (4) is similarly build as the Mapping Network (3) and extracts the style code s=S.sub.E(x.sub.ref) and content code c=C.sub.E(x.sub.ref) for a given image x.sub.ref of domain y. In other words, the Style-Content encoder (4) is configured to receive the reference input image x.sub.ref and extracts its style and content. Similar to the Mapping network (3), the Style-Content encoder (4) determines a latent code by a first encoder, wherein the latent code is forwarded through two separate parts S.sub.E, C.sub.E. The two separate parts S.sub.E, C.sub.E output the contents and style vectors respectively. The two separate parts S.sub.E, C.sub.E can be a second and third encoder.”).
and inputting the first generated data to the trained generator, and outputting fourth generated data corresponding to the first generated data by referring to the third code by the trained generator. (Wang claim 8. “The method as recited in claim 7, wherein the generator, the Mapping network and the Style-Content encoder, are trained by: generating a target style code and target content code by a Mapping network depending on a given target foreground domain, propagating a given image through the encoder of the generator, replacing a first part of a first latent feature map outputted by the encoder with the generated target content code, propagating the replaced first part of the first latent feature map through the first decoder of the generator, wherein the target style code is injected into the first decoder, and propagating the second part of the first latent feature map through a second decoder of the generator, merging output of the first and second decoder of the generator to an output image of the generator, propagating an input image through a discriminator, wherein the image is either the output image of the generator or an image from a training data set, wherein the discriminator outputs whether its input image is either an image of the training data set or generated by the generator, determining an adversarial loss depending on the output of the discriminator, determining a style-content reconstruction loss by determining a difference between the target style code and an extracted style code by a Style-Content encoder depending on the output image of the generator and a difference between the target content code and an extracted content code by a Style-Content encoder depending on the output image of the generator, optimizing the generator, the discriminator, the Mapping network, and the Style-Content encoder, subject to an objective that a sum of the adversarial loss and the style-content reconstruction loss is minimized by the generator and the Mapping network and the Style-Content encoder and the sum of the losses is maximized by the discriminator.”).
As per claim 20, this claim is similar in scope to limitations recited in claim 10, and thus is rejected under the same rationale.
Claims 9 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Wang in view of Mironica and further in view of Amirghodsi and further in view of Lin (Pub no. US 20210027470 A1) and further in view of Shechtman (Pub no. US 20190251401 A1).
As per claim 9, Wang alone does not explicitly teach the claimed limitations.
However, Wang in combination with Lin teaches the claimed:
9. The method related to the data generation framework according to claim 7, further comprising: updating at least one of the style encoder, the mapping network, the generator, and the discriminator according to a style error, (Wang [0021]: “Finally, the generator, discriminator, Mapping network and the Style-Content encoder are optimized subject to an objective that the sum of the adversarial loss and the style-content reconstruction loss is minimized by the generator and the Mapping network and the Style-Content encoder and the sum the losses is maximized by the discriminator. The optimization can be carried out by gradient descent, wherein parameters of the models are adjusted depending on estimated gradients.” The loss represents the error).
wherein the style error comprises a style similarity error, a source preservation error, and a content comparison error, (Wang [0027] “Furthermore, according to an example embodiment of the present invention, it is provided that a diversity loss is determined, wherein the diversity loss penalizes a similarity between two outputted images of the generator for different target style code ({tilde over (s)}) and target content code ({tilde over (c)}) for the same target foreground domain. For example, the diversity loss is determined depending on a difference between generated images by the generator in accordance with the same target foreground domain but with different predefined content and style codes.”
Lin [0081]: “The image composition system 106 utilizes the multi-level fusion neural network 616 to generate a predicted composite digital image 618 based on the easy training foreground image 604, the training background image 606, the training segmentation mask 602, and the inverted training segmentation mask 608. In one or more embodiments, the multi-level fusion neural network 616 generates the predicted composite digital image 618 as discussed above with regard to FIGS. 3-4. The image composition system 106 then uses the loss function 620 to determine the loss (i.e., error) resulting from the multi-level fusion neural network 616 based on a comparison between the predicted composite digital image 618 and a target composite digital image 622. In one or more embodiments, the image composition system 106 generates the target composite digital image 622 using the same alpha channel of the matting image used to generate the easy training foreground image 604 (e.g., the alpha channel used to composite the foreground object 610 with the pure color background 612” This is the content comparison error.
Shechtman [0070]: “In Equation 3, G.sub.i(.Math.) is the geometric prediction generator and ∘ denotes composition warp parameters. Notably, as shown in FIG. 2B, because the original background image 206 and foreground object 208 are used in each iteration of the geometric prediction neural network 202, the image composite system preserves the original images from loss of information due to multiple warping operations.” This is the source preservation loss.).
and the step of updating the at least one of the style encoder, the mapping network, the generator, and the discriminator according to the style error comprises:
calculating a difference between the two first codes respectively obtained from the first source data and second source data by the style encoder to determine the style similarity error; (Wang claim 1: “A computer-implemented method of training a generator for transforming a given image according to a given target foreground domain, comprising the following steps: generating a target style code and target content code by a Mapping network depending on the given target foreground domain; propagating the given image through an encoder of the generator; replacing a first part of a latent feature map outputted by the encoder with the generated target content code; propagating the replaced first part of the latent feature map through a first decoder of the generator, wherein the target style code is injected into the first decoder, and propagating a second part of the latent feature map through a second decoder of the generator; merging output of the first and second decoder of the generator to an output image of the generator; propagating an input image through a discriminator, wherein the image is either the output image of the generator or an image from a training data set, wherein the discriminator outputs whether its input image is either an image of the training data set or generated by the generator; determining an adversarial loss depending on the output of the discriminator; determining a style-content reconstruction loss by determining a difference between the target style code and an extracted style code by a Style-Content encoder depending on the output image of the generator and a difference between the target content code and an extracted content code by a Style-Content encoder depending on the output image of the generator; optimizing the generator, the discriminator, the Mapping network, and the Style-Content encoder, subject to an objective that a sum of the adversarial loss and the style-content reconstruction loss is minimized by the generator and the Mapping network and the Style-Content encoder and the sum of the losses is maximized by the discriminator.” The target style code and the extracted code are the different codes.” The image from a training data set is second source data.).
calculating a difference between the first source data and second generated data to determine the source preservation error, (Shechtman [0100]: To further illustrate, by comparing the generated warp parameters for the perturbed foreground object to the ground truth parameters for the same object given the same background scene, the image classification loss model 408 can identify an amount of error loss for the generated warp parameters (e.g., using mean square error or other types of mathematical comparisons). For instance, the image classification loss model 408 can provide the amount of error loss back to the lower neural network layers 402 and the higher neural network layers 404 to further train the supervised geometric prediction neural network 400 to learn more accurate warp parameters for perturbed foreground objects (e.g., produce less error loss). The process of providing error loss via back propagation to the supervised geometric prediction neural network 400 can continue until the supervised geometric prediction neural network 400 converges.”).
wherein the generator outputs third generated data corresponding to the first source data by referring to the style code, the generator outputs the second generated data corresponding to the third generated data by referring to a second style code, the second style code is obtained from second reference data by the style encoder, and the second reference data and the first source data correspond to a same one of the at least one style option; (Wang [0023]: “Furthermore, according to an example embodiment of the present invention, it is provided that a cycle consistency loss (L.sub.cyc) is determined. More precisely, the cycle consistency loss (L.sub.cyc) determines a difference between the given image and the output of the generator after two rounds of replacing the first part of the latent feature map such that after the second round the original given image should be present. More precisely, during the first round, the given image is transformed according to the target foreground domain by replacing the first part of the latent feature map by the target content code and the first part of the latent feature map of the given image is stored, wherein for the second round, the outputted image of the generator is inputted into the generator and the first part of the latent feature map is replaced with the stored one of the given image.”
Wang [0026]-[0027]: “[0026] a first difference between the predefined content code and extracted first part of the latent feature map (FG.sub.G) of the image obtained after the second round of cyclic transforming the given image. The content consistency can be additionally determined depending on a second difference between the extracted first part of the latent feature map of the given image and the first part of the latent feature map after two rounds of transforming the given image according to the target foreground domain and back to the original foreground domain of the given image. Formally, the content consistency loss can be given as: L.sub.con_cyc=E.sub.x,y,{tilde over (y)},z[|FG.sub.G(G(G(x, {tilde over (s)}, {tilde over (c)}), {tilde over (s)}, {tilde over (c)}))−{tilde over (c)}|]+E.sub.x,y,{tilde over (y)},z[|FG.sub.G(G (x, {tilde over (s)}, {tilde over (c)}))−ĉ|]). [0027] Furthermore, according to an example embodiment of the present invention, it is provided that a diversity loss is determined, wherein the diversity loss penalizes a similarity between two outputted images of the generator for different target style code ({tilde over (s)}) and target content code ({tilde over (c)}) for the same target foreground domain. For example, the diversity loss is determined depending on a difference between generated images by the generator in accordance with the same target foreground domain but with different predefined content and style codes.” Wang teaches several rounds of this transformation of the image feature map and the generation of new image data through the encoders and decoders with different style codes. ).
and calculating a difference between a first feature representation and a second feature representation, and calculating a difference between the second feature representation and a third feature representation to determine the content comparison error, (Wang [0026]: “a first difference between the predefined content code and extracted first part of the latent feature map (FG.sub.G) of the image obtained after the second round of cyclic transforming the given image. The content consistency can be additionally determined depending on a second difference between the extracted first part of the latent feature map of the given image and the first part of the latent feature map after two rounds of transforming the given image according to the target foreground domain and back to the original foreground domain of the given image. Formally, the content consistency loss can be given as: L.sub.con_cyc=E.sub.x,y,{tilde over (y)},z[|FG.sub.G(G(G(x, {tilde over (s)}, {tilde over (c)}), {tilde over (s)}, {tilde over (c)}))−{tilde over (c)}|]+E.sub.x,y,{tilde over (y)},z[|FG.sub.G(G (x, {tilde over (s)}, {tilde over (c)}))−ĉ|]).” The loss is the error.).
wherein the first feature representation is obtained from the first source data by a second encoder of the generator, the second feature representation is obtained from the third generated data by the second encoder, and the third feature representation is obtained from the reference data by the second encoder. (The training data described above is the reference data. Images generated by the generator in previous cycles are the source data. Wang [0020]: “The style-content reconstruction loss (L.sub.sty_con=E.sub.x,{tilde over (y)},z[|{tilde over (s)}−S.sub.E(G(x, {tilde over (s)}, {tilde over (c)}))|]+E.sub.x,{tilde over (y)},z[|{tilde over (c)}−C.sub.E(G(x, {tilde over (s)}, {tilde over (c)}))|]) is determined depending on a difference between the target style code ({tilde over (s)}) and an extracted style code (S.sub.E) by a Style-Content encoder depending on the outputted image (G(x, {tilde over (s)}, {tilde over (c)}) of the generator and depending on a difference between the target content code ({tilde over (c)}) and an extracted content code (C.sub.E) by a Style-Content encoder depending on the outputted image (G(c, {tilde over (s)}, {tilde over (c)})) of the generator. The Style-Content encoder is suited to determine depending on an input image, which can be either the given image or the outputted image of the generator, the content code and style code of its input. Preferably, the Style-Content encoder is a neural network comprising a first plurality of layers which are connected in series, wherein the output of said series is inputted into two encoders, which outputs a content code and style code respectively. Generally, an encoder can comprise a series of layers arranged such that they reduce the dimensionality of an input of the encoder.” There are two encoders, and the second outputs style code.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the content comparison as taught by Lin with the system of Wang in order to measure if the modifications from style and noise change the fundamental image being depicted. The motivation of claim 1 is incorporated herein.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the calculation of source preservation error as taught by Shetchman with the system of Wang as modified by Lin in order to measure if the modifications to the image with style or noise still allow the image to retain its original characteristics. The motivation of claim 1 is incorporated herein.
As per claim 19, this claim is similar in scope to limitations recited in claim 9, and thus is rejected under the same rationale.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to THOMAS JOHN FOSTER whose telephone number is (571)272-5053. The examiner can normally be reached Mon, Fri 8:30-6. Tues-Thurs 7:30-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Daniel Hajnik can be reached at 571-272-7642. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/THOMAS JOHN FOSTER/Examiner, Art Unit 2616
/HAI TAO SUN/Primary Examiner, Art Unit 2616
Prosecution Insights