DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Response to Amendment
Application has been amended by preliminary amendment filed December 2, 2024. Claim 12 has been canceled and claims 15-21 have been added. Claims 1-11 and 13-21 are currently pending and under consideration.
Allowable Subject Matter
Claims 3 and 16 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 4-5 and 17-18 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 4 recites the limitation "the current network layer" in line 2. There is insufficient antecedent basis for this limitation in the claim.
Claim 5 depends from claim 4 and incorporates the same indefinite language.
Claim 17 recites the limitation “the current network layer” in line 2. There is insufficient antecedent basis for this limitation in the claim.
Claim 18 depends from claim 17 and incorporates the same indefinite language.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 1, 6, 10, 13-14 and 19 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Hsiao et al. (US 2021/0142455 A1).
REGARDING CLAIM 13, Hsiao discloses:
An electronic device, (Hsiao, Abstract and ¶30: method and apparatus, terminal device 102) comprising:
One or more processors (Hsiao, ¶32: device includes processor 101); and
A storage apparatus configured to store one or more programs, (Hsiao, ¶¶32-33: memory to store instructions for implementation) wherein
The one or more programs, when executed by the one or more processors, cause the one or more processors to (Hsiao, ¶33: instructions executed by processor to perform method):
Receive an image to be processed and a mask image of a target region in the image to be processed (Hsiao, ¶24: digital image; ¶26: mask; ¶46: CNN receives original image as input, which separates into background image and foreground image to obtain a mask, performed before image transformation network);
Process the image to be processed and the mask image based on a stylization processing system, to obtain a stylized image associated with the target region (Hsiao, ¶37: stylize an image of background or foreground; ¶46: after CNN layer obtains mask from original image, a partial stylized image is obtained by transforming the background image or the foreground image according to a selected style at the image transforming network of the CNN, and next a stylized image according to the mask and the partial stylized image at the image transforming network to output) ; and
Display the stylized image associated with the target region (Hsiao, ¶37: Final styled image will be displayed on the display 107 for review by the user).
REGARDING CLAIM 1, the device of claim 13 performs the method of claim 1, and as such claim 1 is rejected based on the same rationale as claim 13 set forth above.
REGARDING CLAIM 14, Hsiao discloses:
A non-transitory storage medium comprising computer-executable instructions, wherein the computer executable instructions, when executed by a computer processor, cause the computer processor to perform operations (Hsiao, ¶¶32-33: memory to store instructions for implementation, where instructions executed by processor to perform method, where memory can be a non-transitory computer readable storage medium)
Further regarding claim 14, the operations perform the method of claim 1, and as such the claim is further rejected based on claim 1 set forth above.
REGARDING CLAIM 19, Hsiao further discloses:
wherein the electronic device is further caused to: extract the target region from the image to be processed, to obtain a target region image (Hsiao, ¶46: original image is separated into background image an foreground image);
input the target region image into the stylization processing system, to obtain a local stylized image for the target region (Hsiao, ¶46: the separation can be done at a first layer of the CNN before the image transformation network of FIG. 5. Then a partial stylized image is obtained by transforming the background image or the foreground image according to a selected style at the image transforming network of the CNN; ¶53: stylize by transforming background or foreground image according to selected style); and
perform image fusion on the stylized image associated with the target region and the local stylized image, to obtain a target stylized image (Hsiao, ¶52: The resulted mask as shown in FIG. 7 will be used later to fuse a partial stylized image and partial original image, or to fuse an intermediate stylized image and partial original image); and
wherein displaying the stylized image associated with the target region comprises: displaying the target stylized image (Hsiao, ¶37: Final styled image will be displayed on the display 107 for review by the user)
REGARDING CLAIM 6, the device of claim 19 performs the method of claim 6, and as such claim 6 is rejected based on the same rationale as claim 19 set forth above.
REGARDING CLAIM 10, Hsiao further discloses:
wherein the image to be processed is an image comprising a facial region, and the target region is the facial region; and the processing the image to be processed and the mask image based on a stylization processing system, to obtain a stylized image associated with the target region comprises: processing the image to be processed comprising the facial region and the mask image of the facial region based on the stylization processing system, to obtain a stylized image associated with the facial region. (Hsiao, Fig. 7 and ¶26: original image for processing, where foreground includes image of person – note under BRI the facial region is any region that includes a face, which includes the depicted woman in the foreground, processed to be stylized using mask of image; ¶46: processing the foreground to be stylized)
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 2, 7, 11, 15 and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over:
Hsiao et al. (US 2021/0142455 A1) in view of
Ghosh et al. (US 2022/0222872 A1).
REGARDING CLAIM 15, the limitations incorporated from claim 13 are rejected as set forth above for claim 13. Further regarding claim 15, Ghosh discloses:
wherein the stylization processing system comprises an encoding model, (Ghosh, Fig. 2 and ¶31: encoding network 245) an image reconstruction model, (Ghosh, Fig. 2 and ¶31: decoder network 250) and an image stylization model, (Ghosh, Fig. 2 and ¶35: layer 220, where one or more styles may be selected, e.g., in the form of a lookup entry (such as a 1-D style vector) in an embedding matrix, which may then be reshaped as needed in the form of a selected style vector 255 that may be concatenated (layer 220) with the feature map 215, i.e., as an additional feature channel, to generate concatenated feature map 225)
wherein the encoding model is separately connected to the image reconstruction model and the image stylization model (Ghosh, Fig. 2 and ¶36: skip connections 235, as well as ¶35-36 discussing connection to layer 220), and
network layers in the image reconstruction model are connected to corresponding network layers in the image stylization model (Ghosh, Fig. 2 and ¶36: features of concatenated feature map 225 are passed on to decoder 250 network – note that claim does not indicate how they are connected, but rather that there is some sort of connection)
Both Hsiao and Ghosh are directed to stylization of images using neural networks. It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to modify the neural network stylization system of Hsiao, by utilizing the machine learning architecture of Ghosh, using known electronic interfacing and programming techniques. The modification merely substitutes one known type of machine learning stylization neural network model for another, yielding predictable results of introducing style to an image using neural network processing. Moreover, the modification results in an improved stylization system by allowing application of new styles in an ad hoc fashion on smaller set of input images (see Ghosh, ¶7).
REGARDING CLAIM 2, the device of claim 15 performs the method of claim 2, and as such claim 2 is rejected based on the same rationale as claim 15 set forth above.
REGARDING CLAIM 11, the limitations incorporated from claim 1 are rejected as set forth above for claim 1. Further regarding claim 11, Ghosh discloses:
further comprising: determining the image to be processed and the stylized image as an image pair in training samples, and training an end-to-end mobile end network model based on a plurality of image pairs, to obtain an end-to-end stylization network model. (Ghosh, ¶13: training of ML model to stylize input images includes obtaining a training set comprising a plurality of image pairs, including a first image and a version of the first image stylized, and refining the neural network based on the paired images)
Both Hsiao and Ghosh are directed to stylization of images using neural networks. It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to modify the neural network stylization system of Hsiao, by utilizing the machine learning architecture for a generative adversarial network training model of Ghosh, using known electronic interfacing and programming techniques. The modification results in an improved stylized image model by allowing for image generation of entirely new content using improved training using discriminator type training, requiring less cumbersome user training intervention while also allowing for more diverse, but realistic results.
REGARDING CLAIM 20, Hsiao modified by Ghosh further discloses:
wherein a training process of the image reconstruction model comprises: training an image reconstruction model to be trained and a discrimination network model based on random data and a sample image, to obtain a trained image reconstruction model. (Ghosh, ¶¶10 and 13 discloses use of generative adversarial network for training; ¶39: training of GAN including fake training images and randomly sample data of images – examiner further notes that use of random data is inherent to the use of GAN, where GAN training involves random noise vectors for generator, taking a random noise vector as input and generating data samples from which the system produces realistic data by minimizing its loss function, which measures how well it can fool the discriminator)
Both Hsiao and Ghosh are directed to stylization of images using neural networks. It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to modify the neural network stylization system of Hsiao, by utilizing the machine learning architecture for a generative adversarial network training model of Ghosh, using known electronic interfacing and programming techniques. The modification results in an improved stylized image model by allowing for image generation of entirely new content using improved training using discriminator type training, requiring less cumbersome user training intervention while also allowing for more diverse, but realistic results.
REGARDING CLAIM 7, the device of claim 20 performs the method of claim 7, and as such claim 7 is rejected based on the same rationale as claim 20 set forth above.
Claim(s) 8-9 and 21 is/are rejected under 35 U.S.C. 103 as being unpatentable over:
Hsiao et al. (US 2021/0142455 A1) in view of
Ghosh et al. (US 2022/0222872 A1) and in further view of
Xu et al. (W. Xu, C. Long, R. Wang and G. Wang, "DRB-GAN: A Dynamic ResBlock Generative Adversarial Network for Artistic Style Transfer," 2021 IEEE/CVF International Conference on Computer Vision (ICCV), Montreal, QC, Canada, 2021, pp. 6363-6372, doi: 10.1109/ICCV48922.2021.00632)
REGARDING CLAIM 21, the limitations incorporated from claim 15 are rejected as set forth above for claim 15. Further regarding claim 21, Xu discloses:
wherein a training process of the encoding model comprises: iteratively perform the following training process until a training condition is satisfied, to obtain a trained encoding model: (Xu, 6367, section 3.4 discloses use of objective lose functions with adversarial loss and perceptual loss functions for iteration on discriminator; Section 4, ¶1: training model on 600,000 iterations)
input a sample image into an encoding model to be trained, to obtain a training image code (Xu, 6365, section 3, ¶1: Given a content image x ∈ X and an arbitrary style image yc ∈ Y randomly sampled from N different style image collections; Section 3.1, ¶1: The style encoding network is to generate style code from the style image for the style transfer network on the content image);
input the training image code into a trained image reconstruction model, to obtain a reconstructed image (Xu, 6365, section 3, ¶1: Given a content image x ∈ X and an arbitrary style image yc ∈ Y randomly sampled from N different style image collections, our goal is to transfer the content image with the generator G to produce a desired synthetic image ˜xc, ensuring a consistent style with the style image yc via the discriminator D.); and
adjust a model parameter of the encoding model based on the sample image and the reconstructed image. (Xu, 6365-6366, section 3.1, ¶2: we take a classification weight to recalibrate our encoded style feature, denoted as Fs. The attention mechanism is based on an auxiliary classifier Dcls trained to predict the style classification probability wc, which is used to the likelihood of the input style image belonging to the cth category. Then, the style encoding is recalibrated as sc = ωcFs, The recalibrated style encoded feature is then fed into the weight generation module H designed as multi-layer perceptions (MLPs) to determine parameter values for Dynamic ResBlocks.)
Hsiao, Ghosh and Xu are directed to stylization of images using neural networks. It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to modify the neural network stylization system of Hsiao, by utilizing the machine learning architecture for a generative adversarial network training model of Ghosh, using the GAN training of Xu, using known electronic interfacing and programming techniques. The modification results in an improved stylized image model by allowing for image generation of entirely new content using improved training using discriminator type training, requiring less cumbersome user training intervention while also allowing for more diverse, but realistic results, while providing a more robust training of the machine learning using iterative loss function weightings to better adjust a final trained model for improved image results.
REGARDING CLAIM 8, the device of claim 21 performs the method of claim 8, and as such claim 8 is rejected based on the same rationale as claim 21 set forth above.
REGARDING CLAIM 9, the limitations incorporated from claim 2 are rejected as set forth above for claim 2. Further regarding claim 9, Xu discloses:
wherein a training method for the image stylization model comprises: performing parameter initialization on the image stylization model based on a model parameter of the image reconstruction model; and (Xu, 6365, Fig. 2 and Section 3.1, ¶1-2: As shown in Figure 2, we model the style code as the shared hyper-parameters for Dynamic ResBlocks which is designed to integrate dynamic convolution (DConv) and Adaptive Instance Normalization Normalization (AdaIN) [17] in a residual structure [13]. The style encoding network is to generate style code from the style image for the style transfer network on the content image; Attention guided feature extractor. We introduce an ar chitecture of style encoding by concatenating the features from a pre-trained VGG encoder and a learnable encoder. The parameters in the learnable encoder are updated while those in the VGG encoder are fixed.)
training the initialized image stylization model to be trained and a discrimination network model based on random data and a stylized sample image, to obtain a trained image stylization model. (Xu, 6365, section 3, content image and arbitrary style image randomly sampled for generator G; u, 6365, section 3, ¶1: Given a content image x ∈ X and an arbitrary style image yc ∈ Y randomly sampled from N different style image collections; Section 3.1, ¶1: The style encoding network is to generate style code from the style image for the style transfer network on the content image; 6367, section 3.4 discloses use of objective lose functions with adversarial loss and perceptual loss functions for iteration on discriminator; 6365-6366, section 3.1, ¶2: we take a classification weight to recalibrate our encoded style feature, denoted as Fs. The attention mechanism is based on an auxiliary classifier Dcls trained to predict the style classification probability wc, which is used to the likelihood of the input style image belonging to the cth category. Then, the style encoding is recalibrated as sc = ωcFs, The recalibrated style encoded feature is then fed into the weight generation module H designed as multi-layer perceptions (MLPs) to determine parameter values for Dynamic ResBlocks.)
Hsiao, Ghosh and Xu are directed to stylization of images using neural networks. It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to modify the neural network stylization system of Hsiao, by utilizing the machine learning architecture for a generative adversarial network training model of Ghosh, using the GAN training of Xu, using known electronic interfacing and programming techniques. The modification results in an improved stylized image model by allowing for image generation of entirely new content using improved training using discriminator type training, requiring less cumbersome user training intervention while also allowing for more diverse, but realistic results, while providing a more robust training of the machine learning using iterative loss function weightings to better adjust a final trained model for improved image results.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to WILLIAM A BEUTEL whose telephone number is (571)272-3132. The examiner can normally be reached Monday-Friday 9:00 AM - 5:00 PM (EST).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, DANIEL HAJNIK can be reached at 571-272-7642. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/WILLIAM A BEUTEL/ Primary Examiner, Art Unit 2616
Prosecution Insights