DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1, 2, 4 – 8, 10 – 16, and 18 – 20 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Chandran et al. (US 2022/0156987).
Regarding independent claim 1, Chandran teaches a computer-implemented method for performing style transfer (Figures 3 – 5), the method comprising:
determining a first set of attribute values for a plurality of attributes associated with a content sample (paragraph 26: Content-based attributes 240 of content sample 226 may include distinguishing visual or physical attributes, hierarchies, or arrangements of these objects and/or shapes; paragraph 27: Style-based attributes 238 in style sample 230 may include, but are not limited to, brush strokes, lines, edges, patterns, colors, bokeh, and/or other artistic or naturally occurring attributes that define the manner in which content is depicted);
computing one or more losses based on the content sample and one or more style samples (paragraph 40: style loss 232 and content loss 234 may be determined using latent representations 216, 218, as well as a latent representation 242 generated by an encoder 208 from decoder output 210);
converting, based on the one or more losses (paragraph 45: training engine 122 may use a training technique (e.g., gradient descent and backpropagation) and/or one or more hyperparameters to iteratively update weights of kernel predictor 220 and/or decoder 206 in a way that reduces the loss function (e.g., objective function 212) associated with style loss 232 and content loss 234), the first set of attribute values into a second set of attribute values for the plurality of attributes (paragraph 29: Encoder 202 may generate, for a given content sample (e.g., content sample 226), a latent representation 216 of the content sample. Encoder 204 may generate, for a given style sample (e.g., style sample 230), a latent representation 218 of the style sample); and
generating a style transfer result based on a composite of the second set of attribute values (paragraph 39: training engine 122 inputs latent representation 218 of the training style sample into kernel predictor 220 to produce convolutional kernels 222 that reflect the feature map associated with the training style sample and convolves latent representation 218 with convolutional kernels 222 to produce convolutional output).
Regarding dependent claim 2, Chandran teaches wherein determining the first set of attribute values comprises storing the first set of attribute values in a plurality of layers corresponding to the plurality of attributes (paragraph 26: Content-based attributes 240 of content sample 226 may include distinguishing visual or physical attributes, hierarchies, or arrangements of these objects and/or shapes).
Regarding dependent claim 4, Chandran teaches wherein the one or more losses comprise at least one of an L1 loss, an L2 loss, a cosine distance, a Euclidean distance (paragraph 41: style loss 232 may be calculated as a measure of distance (e.g., cosine similarity, Euclidean distance, etc.) between latent representations 218 and 242, and content loss 234 may be calculated as a measure of distance between latent representations 216 and 242), or a perceptual loss.
Regarding dependent claim 5, Chandran teaches wherein converting the first set of attribute values into the second set of attribute values comprises: converting, based on a first loss included in the one or more losses (paragraph 45: training engine 122 may use a training technique (e.g., gradient descent and backpropagation) and/or one or more hyperparameters to iteratively update weights of kernel predictor 220 and/or decoder 206 in a way that reduces the loss function (e.g., objective function 212) associated with style loss 232 and content loss 234), a first subset of the first set of attribute values into a first subset of the second set of attribute values (paragraph 29: Encoder 202 may generate, for a given content sample (e.g., content sample 226), a latent representation 216 of the content sample); and converting, based on a second loss included in the one or more losses (paragraph 45: training engine 122 may use a training technique (e.g., gradient descent and backpropagation) and/or one or more hyperparameters to iteratively update weights of kernel predictor 220 and/or decoder 206 in a way that reduces the loss function (e.g., objective function 212) associated with style loss 232 and content loss 234), a second subset of the first set of attribute values into a second subset of the second set of attribute values (paragraph 29: Encoder 204 may generate, for a given style sample (e.g., style sample 230), a latent representation 218 of the style sample).
Regarding dependent claim 6, Chandran teaches wherein the first subset of the first set of attribute values corresponds to a first attribute included in the plurality of attributes (paragraph 29: Encoder 202 may generate, for a given content sample (e.g., content sample 226), a latent representation 216 of the content sample) and the second subset of the first set of attribute values corresponds to a second attribute included in the plurality of attributes (paragraph 29: Encoder 204 may generate, for a given style sample (e.g., style sample 230), a latent representation 218 of the style sample).
Regarding dependent claim 7, Chandran teaches wherein converting the first set of attribute values into the second set of attribute values comprises iteratively updating the first set of attribute values based on the one or more losses (paragraph 45: training engine 122 may use a training technique (e.g., gradient descent and backpropagation) and/or one or more hyperparameters to iteratively update weights of kernel predictor 220 and/or decoder 206 in a way that reduces the loss function (e.g., objective function 212) associated with style loss 232 and content loss 234. In some embodiments, hyperparameters define higher-level properties of style transfer model 200 and/or are used to control the training of style transfer model 200. For example, hyperparameters for style transfer model 200 may include, but are not limited to, batch size, learning rate, number of iterations, numbers and sizes of convolutional kernels 222 outputted by kernel predictor 220, numbers of layers in each of encoders 202 and 204 and decoder 206, and/or thresholds for pruning weights in neural network layers).
Regarding dependent claim 8, Chandran teaches wherein generating the style transfer result comprises determining a set of pixel values included in the style transfer result based on an interpolation associated with the second set of attribute values (paragraph 29: each of encoders 202, 204 may convert pixels, voxels, points, textures, and/or other information in an inputted sample (e.g., a style and/or content sample) into a number of vectors and/or matrices in a lower-dimensional latent space).
Regarding dependent claim 10, Chandran teaches wherein the plurality of attributes comprises at least one of a pixel color value (paragraph 29: each of encoders 202, 204 may convert pixels, voxels, points, textures, and/or other information in an inputted sample (e.g., a style and/or content sample) into a number of vectors and/or matrices in a lower-dimensional latent space), a background color value, a color curve, an alpha channel, a mask, a pixel displacement, a shape (paragraph 26: content sample 226 may include an image, mesh, and/or other two-dimensional (2D) or three-dimensional (3D) depiction of one or more objects (e.g. face, building, vehicle, animal, plant, road, water, etc.) and/or abstract shapes (e.g., lines, squares, round shapes, curves, polygons, etc.)), a contour (paragraph 26: content sample 226 may include an image, mesh, and/or other two-dimensional (2D) or three-dimensional (3D) depiction of one or more objects (e.g. face, building, vehicle, animal, plant, road, water, etc.) and/or abstract shapes (e.g., lines, squares, round shapes, curves, polygons, etc.)), an outline (paragraph 26: content sample 226 may include an image, mesh, and/or other two-dimensional (2D) or three-dimensional (3D) depiction of one or more objects (e.g. face, building, vehicle, animal, plant, road, water, etc.) and/or abstract shapes (e.g., lines, squares, round shapes, curves, polygons, etc.)), a lighting attribute (paragraph 68: operation 502 may be performed to generate one or more series of convolutional kernels from a first input that includes embedded and/or encoded representations of camera parameters (e.g., camera model, camera pose, focal length, etc.), lighting parameters (e.g., light sources, lighting interactions, illumination models, shading, etc.), and/or other types of parameters that affect the rendering or appearance of the scene), a haze attribute, a motion vector, a rendering attribute (paragraph 68: operation 502 may be performed to generate one or more series of convolutional kernels from a first input that includes embedded and/or encoded representations of camera parameters (e.g., camera model, camera pose, focal length, etc.), lighting parameters (e.g., light sources, lighting interactions, illumination models, shading, etc.), and/or other types of parameters that affect the rendering or appearance of the scene), or a region of the content sample (paragraph 26: Content-based attributes 240 of content sample 226 may include distinguishing visual or physical attributes, hierarchies, or arrangements of these objects and/or shapes (e.g., a face is an object that includes a recognizable arrangement of eyes, ears, nose, mouth, hair, and/or other objects, and each object inside the face is represented by a recognizable arrangement of lines, angles, polygons, and/or other abstract shapes)).
Regarding independent claim 11, Chandran teaches one or more non-transitory computer-readable media storing instructions that, when executed by one or more processors (Claim 12), cause the one or more processors to perform the steps of:
determining a first set of attribute values for a plurality of attributes associated with a content sample (paragraph 26: Content-based attributes 240 of content sample 226 may include distinguishing visual or physical attributes, hierarchies, or arrangements of these objects and/or shapes; paragraph 27: Style-based attributes 238 in style sample 230 may include, but are not limited to, brush strokes, lines, edges, patterns, colors, bokeh, and/or other artistic or naturally occurring attributes that define the manner in which content is depicted);
computing one or more losses based on the content sample and one or more style samples (paragraph 40: style loss 232 and content loss 234 may be determined using latent representations 216, 218, as well as a latent representation 242 generated by an encoder 208 from decoder output 210);
converting, based on the one or more losses (paragraph 45: training engine 122 may use a training technique (e.g., gradient descent and backpropagation) and/or one or more hyperparameters to iteratively update weights of kernel predictor 220 and/or decoder 206 in a way that reduces the loss function (e.g., objective function 212) associated with style loss 232 and content loss 234), the first set of attribute values into a second set of attribute values for the plurality of attributes (paragraph 29: Encoder 202 may generate, for a given content sample (e.g., content sample 226), a latent representation 216 of the content sample. Encoder 204 may generate, for a given style sample (e.g., style sample 230), a latent representation 218 of the style sample); and
generating a style transfer result based on a composite of the second set of attribute values (paragraph 39: training engine 122 inputs latent representation 218 of the training style sample into kernel predictor 220 to produce convolutional kernels 222 that reflect the feature map associated with the training style sample and convolves latent representation 218 with convolutional kernels 222 to produce convolutional output).
Regarding dependent claim 12, Chandran teaches wherein determining the first set of attribute values comprises generating a set of parameters representing an attribute included in the plurality of attributes (paragraph 29: Encoder 202 may generate, for a given content sample (e.g., content sample 226), a latent representation 216 of the content sample. Encoder 204 may generate, for a given style sample (e.g., style sample 230), a latent representation 218 of the style sample).
Regarding dependent claim 13, Chandran teaches wherein converting the first set of attribute values into the second set of attribute values comprises iteratively updating the set of parameters based on the one or more losses and a set of constraints associated with the set of parameters (paragraph 45: training engine 122 may use a training technique (e.g., gradient descent and backpropagation) and/or one or more hyperparameters to iteratively update weights of kernel predictor 220 and/or decoder 206 in a way that reduces the loss function (e.g., objective function 212) associated with style loss 232 and content loss 234. In some embodiments, hyperparameters define higher-level properties of style transfer model 200 and/or are used to control the training of style transfer model 200. For example, hyperparameters for style transfer model 200 may include, but are not limited to, batch size, learning rate, number of iterations, numbers and sizes of convolutional kernels 222 outputted by kernel predictor 220, numbers of layers in each of encoders 202 and 204 and decoder 206, and/or thresholds for pruning weights in neural network layers).
Regarding dependent claim 14, Chandran teaches wherein computing the one or more losses comprises: generating, via one or more neural networks (paragraph 55: in operation 402, execution engine 124 applies an encoder network and/or one or more additional neural network layers to a style sample and a content sample to produce a first latent representation of the style sample and a second latent representation of the content sample), a first set of features associated with the content sample and a second set of features associated with the one or more style samples (paragraph 55: Execution engine 124 may use a pretrained encoder such as VGG, ImageNet, ResNet, GoogLeNet, and/or Inception to convert the style sample and content sample into two separate feature maps); and
computing the one or more losses based on the first set of features and the second set of features (paragraph 44: When style loss 232 and/or content loss 234 include multiple measures of distance (e.g., between features produced by different encoder layers), objective function 212 may specify a different weighting for each measure).
Regarding dependent claim 15, Chandran teaches wherein computing the one or more losses further comprises matching the first set of features to the second set of features based on one or more distances computed between the first set of features and the second set of features (paragraph 44: When style loss 232 and/or content loss 234 include multiple measures of distance (e.g., between features produced by different encoder layers), objective function 212 may specify a different weighting for each measure).
Regarding dependent claim 16, Chandran teaches wherein generating the style transfer result comprises:
determining a first level of stylization associated with a first attribute included in the plurality of attributes and a second level of stylization associated with a second attribute included in the plurality of attributes (paragraph 44: When style loss 232 and/or content loss 234 include multiple measures of distance (e.g., between features produced by different encoder layers), objective function 212 may specify a different weighting for each measure);
determining a first interpolation associated with a first subset of the second set of attribute values based on the first level of stylization (paragraph 44: style loss 232 may include a higher weight or coefficient for the distance between lower-level features produced by earlier layers of encoder 208 from decoder output 210 and features produced by corresponding layers of encoder 204 from the style sample to increase the presence of "local" style-based attributes 238 such as lines, edges, brush strokes, colors, and/or patterns) and a second interpolation associated with a second subset of the second set of attribute values based on the second level of stylization (paragraph 44: content loss 234 may include a higher weight for the distance between higher-level "global" features produced by subsequent layers of encoder 208 from decoder output 210 and features produced by corresponding layers of encoder 202 from the content sample at higher resolutions to increase the presence of overall content-based attributes 238 such as recognizable features or shapes of objects); and
determining a set of pixel values included in the style transfer result (paragraph 36: decoder 206 may include a CNN that applies additional convolutions and/or up-sampling to the convolutional output to generate decoder output 210 that includes an image, mesh, and/or another 2D or 3D representation) based on the first interpolation and the second interpolation (paragraph 45: training engine 122 may use a training technique (e.g., gradient descent and backpropagation) and/or one or more hyperparameters to iteratively update weights of kernel predictor 220 and/or decoder 206 in a way that reduces the loss function (e.g., objective function 212) associated with style loss 232 and content loss 234).
Regarding dependent claim 18, Chandran teaches wherein the plurality of attributes comprises at least one of a pixel color value (paragraph 29: each of encoders 202, 204 may convert pixels, voxels, points, textures, and/or other information in an inputted sample (e.g., a style and/or content sample) into a number of vectors and/or matrices in a lower-dimensional latent space), a background color value, a color curve, an alpha channel, a mask, a pixel displacement, a shape (paragraph 26: content sample 226 may include an image, mesh, and/or other two-dimensional (2D) or three-dimensional (3D) depiction of one or more objects (e.g. face, building, vehicle, animal, plant, road, water, etc.) and/or abstract shapes (e.g., lines, squares, round shapes, curves, polygons, etc.)), a contour (paragraph 26: content sample 226 may include an image, mesh, and/or other two-dimensional (2D) or three-dimensional (3D) depiction of one or more objects (e.g. face, building, vehicle, animal, plant, road, water, etc.) and/or abstract shapes (e.g., lines, squares, round shapes, curves, polygons, etc.)), an outline (paragraph 26: content sample 226 may include an image, mesh, and/or other two-dimensional (2D) or three-dimensional (3D) depiction of one or more objects (e.g. face, building, vehicle, animal, plant, road, water, etc.) and/or abstract shapes (e.g., lines, squares, round shapes, curves, polygons, etc.)), a lighting attribute (paragraph 68: operation 502 may be performed to generate one or more series of convolutional kernels from a first input that includes embedded and/or encoded representations of camera parameters (e.g., camera model, camera pose, focal length, etc.), lighting parameters (e.g., light sources, lighting interactions, illumination models, shading, etc.), and/or other types of parameters that affect the rendering or appearance of the scene), a haze attribute, a motion vector, a rendering attribute (paragraph 68: operation 502 may be performed to generate one or more series of convolutional kernels from a first input that includes embedded and/or encoded representations of camera parameters (e.g., camera model, camera pose, focal length, etc.), lighting parameters (e.g., light sources, lighting interactions, illumination models, shading, etc.), and/or other types of parameters that affect the rendering or appearance of the scene), or a region of the content sample (paragraph 26: Content-based attributes 240 of content sample 226 may include distinguishing visual or physical attributes, hierarchies, or arrangements of these objects and/or shapes (e.g., a face is an object that includes a recognizable arrangement of eyes, ears, nose, mouth, hair, and/or other objects, and each object inside the face is represented by a recognizable arrangement of lines, angles, polygons, and/or other abstract shapes)).
Regarding dependent claim 19, Chandran teaches wherein the one or more losses comprise at least one of a style loss (paragraph 40: style loss 232), a content loss (paragraph 40: content loss 234), a perceptual loss, an L1 loss, or an L2 loss.
Regarding independent claim 20, Chandran teaches a system (Figures 1 and 2), comprising: one or more memories (Figure 1: Storage 114) that store instructions, and one or more processors (Figure 1: Processor(s) 102) that are coupled to the one or more memories and, when executing the instructions (paragraph 100), are configured to perform the steps of:
determining a first set of attribute values for a plurality of attributes associated with a content sample (paragraph 26: Content-based attributes 240 of content sample 226 may include distinguishing visual or physical attributes, hierarchies, or arrangements of these objects and/or shapes; paragraph 27: Style-based attributes 238 in style sample 230 may include, but are not limited to, brush strokes, lines, edges, patterns, colors, bokeh, and/or other artistic or naturally occurring attributes that define the manner in which content is depicted);
computing one or more losses based on the content sample and one or more style samples (paragraph 40: style loss 232 and content loss 234 may be determined using latent representations 216, 218, as well as a latent representation 242 generated by an encoder 208 from decoder output 210);
converting, based on the one or more losses (paragraph 45: training engine 122 may use a training technique (e.g., gradient descent and backpropagation) and/or one or more hyperparameters to iteratively update weights of kernel predictor 220 and/or decoder 206 in a way that reduces the loss function (e.g., objective function 212) associated with style loss 232 and content loss 234), the first set of attribute values into a second set of attribute values for the plurality of attributes (paragraph 29: Encoder 202 may generate, for a given content sample (e.g., content sample 226), a latent representation 216 of the content sample. Encoder 204 may generate, for a given style sample (e.g., style sample 230), a latent representation 218 of the style sample); and
generating a style transfer result based on a composite of the second set of attribute values (paragraph 39: training engine 122 inputs latent representation 218 of the training style sample into kernel predictor 220 to produce convolutional kernels 222 that reflect the feature map associated with the training style sample and convolves latent representation 218 with convolutional kernels 222 to produce convolutional output).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim(s) 3 is/are rejected under 35 U.S.C. 103 as being unpatentable over Chandran et al. (US 2022/0156987) in view of Hsiao et al. (US 2022/0084165).
Regarding dependent claim 3, Chandran teaches <<does not expressly disclose>> wherein computing the one or more losses comprises: converting, via a trained <<variational auto>>encoder, a first set of features associated with the content sample into a second set of features from a feature space associated with the one or more style samples (paragraph 31: encoders 202, 204 may use different CNNs and/or layers to convert different types of data (e.g., 2D image data and 3D mesh data) into feature embeddings Fc and Fs and/or generate feature embeddings with different sizes and/or numbers of channels from the corresponding content and style samples); and
computing the one or more losses based on the first set of features and the second set of features (paragraph 40: style loss 232 and content loss 234 may be determined using latent representations 216, 218, as well as a latent representation 242 generated by an encoder 208 from decoder output 210. For example, encoder 208 may include the same pre-trained CNN layers as encoders 202 and/or 204. As a result, encoder 208 may output latent representation 242 in the same latent space as and/or in a similar latent space to those of feature embeddings Fc and Fs).
Chandran does not expressly disclose a trained variational autoencoder. Hsiao discloses the auto-encoder network 300 receives the content image 204 and the style image 206, applies style transfer on the whole content image 204, and outputs a stylized image 208, wherein the auto-encoder network 300 includes a content encoder branch 302, the style encoder branch 304, and a decoder 306 (paragraph 38). It would have been obvious for one of ordinary skill in the art at the time of the invention (pre-AIA ) or at the time of the effective filing date of the application (AIA ) to modify Chandran's system to use a trained variational autoencoder for converting content features into content latent vectors/tensors. One would be motivated to do so because this would allow efficient data representations of the content features in an unsupervised manner.
Claim(s) 9 is/are rejected under 35 U.S.C. 103 as being unpatentable over Chandran et al. (US 2022/0156987) in view of Official Notice.
Regarding dependent claim 9, Chandran does not expressly disclose wherein generating the style transfer result comprises modifying a set of pixel values included in the content sample based on the second set of attribute values and a set of motion vectors associated with the content sample. Examiner takes Official Notice that the concept of motion vectors for modifying a set of pixel values included in the content sample and the advantage of encoding motion of an object are well known and expected in the art. It would have been obvious for one of ordinary skill in the art at the time of the invention (pre-AIA ) or at the time of the effective filing date of the application (AIA ) to modify Chandran's system to utilize motion vectors of an object in generating style transfer of the object. One would be motivated to do so because this would help determine small clips/animations/videos to be set in the desired style.
Claim(s) 17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Chandran et al. (US 2022/0156987) in view of Official Notice.
Regarding dependent claim 17, Chandran does not expressly disclose wherein generating the style transfer result comprises displacing a set of pixel values included in the content sample based on a displacement map included in the second set of attribute values. Examiner takes Official Notice that the concept of displacement map for generating the style transfer result and the advantage of creating a realistic 3D-like texture are well known and expected in the art. It would have been obvious for one of ordinary skill in the art at the time of the invention (pre-AIA ) or at the time of the effective filing date of the application (AIA ) to modify Chandran's system to incorporate a displacement map for generation of the style transfer of an attribute value. One would be motivated to do so because this would help produce a realistic 3D-like texture of the style transferred image.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JEFFREY J CHOW whose telephone number is (571)272-8078. The examiner can normally be reached 11AM-7PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Devona Faulk can be reached at 571-272-7515. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JEFFREY J CHOW/Primary Examiner, Art Unit 2618
Prosecution Insights