DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on January 29, 2026 has been entered.
Response to Arguments
Applicant's arguments filed January 5, 2026 have been fully considered but they are not persuasive. Applicant argues that Shayani (US 20230326159A1) fails to teach training the machine learning model to generate images with a new augmented reality stylization after having been initially trained to generate images corresponding to the plurality of augmented reality stylizations; and after training the machine learning model to generate images with the new augmented reality stylization, presenting a new option corresponding to the new augmented reality stylization on the device, the new option being visually distinguished and selectable to apply one or more augmented reality elements to the image with the new augmented reality stylization (p. 10).
In reply, the Examiner points out that Shayani describes “Style-generation engine 126 can additionally train and/or execute machine learning model 228 so that style code samples 240 and/or style code distribution 230 are conditioned on additional input…After machine learning model 228 has been trained, an additional description could be provided as input to machine learning model 228, and machine learning model 228 could use the input to generate a style code sample that captures the style-based attributes in the description” [0081]. Thus, Shayani teaches training machine learning model 228 to generate images with a new augmented reality stylization that is based on the additional input description with the style-based attributes after having been initially trained to generate images corresponding to the plurality of augmented reality stylizations; and after training machine learning model 228 to generate images with the new augmented reality stylization [0081], a new option corresponding to the new augmented reality stylization, the new option being selectable to apply with the new augmented reality stylization (visual attributes pertaining to the style can be controlled by selecting the set of augmentations, [0036]). The primary reference, Shen (US 20230281940A1), teaches presenting a plurality of options each associated with a different augmented reality stylization on the device, each option being visually distinguished and selectable to apply one or more augmented reality elements to the image with the augmented reality stylization (context-aware universal avatar editing system provides, for display within the context-aware avatar overlay editor, an avatar appearance modifier option, context-aware universal avatar editing system can provide, for display within the context-aware avatar overlay editor, options to modify various features of an avatar, context-aware universal avatar editing system can provide, for display within the context-aware avatar overlay editor, options to modify graphical assets of the avatar, such as, but not limited to, hairstyles, hair color, face styles, skin tones, body types, and/or height, [0082], [0035]). Thus, this teaching from Shayani of the new augmented reality stylization can be implemented into the options of Shen so that it presents a new option corresponding to the new augmented reality stylization on the device, the new option being visually distinguished and selectable to apply one or more augmented reality elements to the image with the new augmented reality stylization.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1-4, 15-18, and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Shen (US 20230281940A1) and Shayani (US 20230326159A1).
As per Claim 1, Shen teaches a method comprising: receiving an image depicting a real-world object (avatar can include a three-dimensional representation of a user that provides a realistic (photorealistic) portrayal of the user, [0035]); presenting a plurality of options each associated with a different augmented reality stylization of a plurality of augmented reality stylizations, each of the plurality of augmented reality stylizations being associated with modification of visual features of a facial object, the plurality of options being overlaid on the image that depicts the real-world object (context-aware universal avatar editing system provides, for display within the context-aware avatar overlay editor, an avatar appearance modifier option, context-aware universal avatar editing system can provide, for display within the context-aware avatar overlay editor, options to modify various features of an avatar, context-aware universal avatar editing system can provide, for display within the context-aware avatar overlay editor, options to modify graphical assets of the avatar, such as, but not limited to, hairstyles, hair color, face styles, skin tones, body types, and/or height, [0082], [0035]); receiving user input that selects an individual option from the plurality of options that are overlaid on the image, the individual option corresponding to an individual augmented reality stylization of the plurality of augmented reality stylizations; generating a modified image in which facial features of the real-world object are modified according to the modification of the visual features of the facial object associated with the individual augmented reality stylization selected by the user input; and presenting the modified image on a device ([0082], upon receiving a selection of the selectable option 418, the context-aware universal avatar editing system can provide the modified avatar for display within the extended-reality environment such that the user of the client device and other co-users within the extended-reality environment can view the updated appearance of the avatar, [0083]); presenting a plurality of options each associated with a different augmented reality stylization on the device, each option being visually distinguished and selectable to apply one or more augmented reality elements to the image with the augmented reality stylization [0082, 0035].
However, Shen does not teach generating, by a machine learning model, the modified image in which facial features of the real-world object are modified according to the modification of visual features of the facial object associated with the individual augmented reality stylization selected by the user input, the machine learning model trained in multiple stages having different training data sets and different conditions applied in each of the multiple stages, the machine learning model being initially trained to generate images corresponding to the plurality of augmented reality stylizations; training the machine learning model to generate images with a new augmented reality stylization after having been initially trained to generate images corresponding to the plurality of augmented reality stylizations; and after training the machine learning model to generate images with the new augmented reality stylization, presenting a new option corresponding to the new augmented reality stylization on the device, the new option being visually distinguished and selectable to apply one or more augmented reality elements to the image with the new augmented reality stylization. However, Shayani teaches generating, by a machine learning model, a modified image in which facial features are modified according to the modification of visual features of the facial object associated with the individual augmented reality stylization (training engine trains encoder, style network 214, and decoder using training data 202 that includes asset of training shapes and a corresponding set of augmented training shapes, augmented training shapes are generated by applying augmentations to the corresponding training shapes, [0044], training engine uses encoder to convert the set of augmented training shapes into a second set of training shape signatures, training engine uses style network 214 to generate training style codes from pairs of training shape signatures, where each pair of training shape signatures includes a first training shape signature representing a training shape and a second training shape signature representing an augmented training shape that is produced by applying augmentations to the training shape, training engine uses decoder to convert training shape signatures, and training style codes into training output shapes, training engine then computes losses associated with training output shapes and updates parameters or encoder, style network 214, and decoder based on losses, [0045], after machine learning model 204 is trained, execution engine uses components of machine learning model 204 to perform style transfer for additional 3D shapes, style transfer in a 3D shape involves converting a given input shape into a corresponding output shape 236 that retains the content of input shape but depicts a style that is distinct form that input shape, [0061], style transfer refers to the manipulation of an image, a video, to adopt the appearance or visual style belonging to a different piece of media, for example, a style of a first image could be transferred onto faces depicted in a second image without altering the identity of the objects in the second image, [0003], for example, style transfer techniques could be used to convert a first 3D shape that captures the structure of a generic car into a second 3D shape that reflects the specific design or look of a particular model of car, the second 3D shape could then be used in an augmented reality environment, [0004]). In step 516, training engine determines whether or not training of the machine learning model is complete. For example, training engine could determine that training is complete when conditions are met. These conditions include but are not limited to a certain number of training steps, iterations, batches, and epochs. While training of the machine learning model is not complete, training engine continues performing steps 502-514 [0101]. In step 502, training engine applies augmentations to a first input 3D shape to generate a second input 3D shape [0094]. Thus, as shown in Fig. 5, in step 516, if training of the machine learning model is not complete, training engine starts again from step 502 to perform steps 502-514 again, and in step 502, it applies a different augmentation, and so it has a different training data set and different conditions applied, and each time it starts again from step 502, it is starting the next stage. Thus, Shayani teaches the machine learning model trained in multiple stages having different training data sets and different conditions applied in each of the multiple stages [0101, 0094] (Fig. 5), the machine learning model being initially trained to generate images corresponding to the plurality of augmented reality stylizations [0044, 0045, 0004]; presenting the modified image on a device ([0061], I/O devices 108 may be configured to provide output to the end-user of computing device 100, such as displayed digital images or digital videos, [0027]). Shayani describes “Style-generation engine 126 can additionally train and/or execute machine learning model 228 so that style code samples 240 and/or style code distribution 230 are conditioned on additional input…After machine learning model 228 has been trained, an additional description could be provided as input to machine learning model 228, and machine learning model 228 could use the input to generate a style code sample that captures the style-based attributes in the description” [0081]. Thus, Shayani teaches training machine learning model 228 to generate images with a new augmented reality stylization that is based on the additional input description with the style-based attributes after having been initially trained to generate images corresponding to the plurality of augmented reality stylizations; and after training machine learning model 228 to generate images with the new augmented reality stylization [0081], a new option corresponding to the new augmented reality stylization, the new option being selectable to apply with the new augmented reality stylization (visual attributes pertaining to the style can be controlled by selecting the set of augmentations, [0036]). Shen teaches generating a modified image in which facial features of the real-world object are modified according to the modification of the visual features of the facial object associated with the individual augmented reality stylization selected by the user input [0082-0083]; presenting a plurality of options each associated with a different augmented reality stylization on the device, each option being visually distinguished and selectable to apply one or more augmented reality elements to the image with the augmented reality stylization [0082, 0035]. Thus, this teaching of the machine learning model and the new augmented reality stylization from Shayani can be implemented into the device of Shen so that it generates, by a machine learning model, a modified image in which facial features of the real-world object are modified according to the modification of visual features of the facial object associated with the individual augmented reality stylization selected by the user input; presenting a new option corresponding to the new augmented reality stylization on the device, the new option being visually distinguished and selectable to apply one or more augmented reality elements to the image with the new augmented reality stylization.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Shen to include generating, by a machine learning model, the modified image in which facial features of the real-world object are modified according to the modification of visual features of the facial object associated with the individual augmented reality stylization selected by the user input, the machine learning model trained in multiple stages having different training data sets and different conditions applied in each of the multiple stages, the machine learning model being initially trained to generate images corresponding to the plurality of augmented reality stylizations; training the machine learning model to generate images with a new augmented reality stylization after having been initially trained to generate images corresponding to the plurality of augmented reality stylizations; and after training the machine learning model to generate images with the new augmented reality stylization, presenting a new option corresponding to the new augmented reality stylization on the device, the new option being visually distinguished and selectable to apply one or more augmented reality elements to the image with the new augmented reality stylization because Shayani suggests that this way, the user does not need to manually manipulate the image to add the augmented reality stylization themselves, and the machine learning model automatically adds the augmented reality stylization in a way that precisely controls the isolation and disentanglement of style features from content features and allows the machine learning model to transfer different combinations of attributes corresponding to different styles across 3D shapes [0011].
As per Claim 2, Shen describes as shown in Fig. 7C, the context-aware universal avatar editing system 106 provides, for display, a hairstyle modification tab 720. As further shown in Fig. 7C, the context-aware universal avatar editing system 106 provides, for display, within the context-aware universal avatar editing system 106 under the hairstyle modification tab 720, a selectable graphical asset 722 that includes a hairstyle appearance change to modify a hairstyle of the avatar. As shown in Fig. 7C, the context-aware universal avatar editing system 106 provides, for display, options to modify colors of graphical assets (e.g., hair color, lipstick color, hat color) [0117]. Thus, as shown in Fig. 7C, the user could select to modify the hair (individual augmented reality stylization), and then could further select individual conditions such as modifying the hairstyle or the facial hair and modifying the hair color. Thus, Shen teaches accessing an individual condition corresponding to the individual augmented reality stylization of the plurality of augmented reality stylizations in response to receiving the user input [0082, 0117] (Fig. 7C).
However, Shen does not teach receiving, by the machine learning model, the individual condition along with the image, the machine learning model generating the modified image in response to receiving the individual condition and the image. However, Shayani teaches further comprising: accessing an individual condition (smoothing or coarsening) corresponding to the individual augmented reality stylization of the plurality of augmented reality stylizations; and receiving, by the machine learning model, the individual condition along with the image, the machine learning model generating the modified image in response to receiving the individual condition and the image (training engine trains encoder, style network 214, and decoder using training data 202 that includes asset of training shapes and a corresponding set of augmented training shapes, augmented training shapes are generated by applying augmentations to the corresponding training shapes, for example, smoothing or coarsening augmentations 206 could be applied to each of training shapes to generate a corresponding set of augmented training shapes, [0044, 0045, 0061, 0004]). This would be obvious for the reasons given in the rejection for Claim 1.
As per Claim 3, Shen does not teach further comprising: providing, as input to the machine learning model, a condition comprising the individual augmented reality stylization and the image. However, Shayani teaches further comprising: providing, as input to the machine learning model, a condition comprising the individual augmented reality stylization and the image [0044, 0045, 0061, 0004]). This would be obvious for the reasons given in the rejection for Claim 1.
As per Claim 4, Shen teaches wherein the real-world object comprises a human face (avatar can include a three-dimensional representation of a user that provides a realistic (photorealistic) portrayal of the user, [0035], avatar elements (digital faces), [0020], avatar face, [0036]), and wherein the plurality of augmented reality stylizations comprise one or more augmented reality items applied to at least a portion of the human face (context-aware universal avatar editing system displays, within the context-aware avatar overlay editor, an avatar accessory modification tab (e.g., head wear), context-aware universal avatar editing system provides, for display, within the context-aware universal avatar editing system under the avatar accessory modification tab, a selectable graphical asset that includes a head wear appearance change (e.g., hat accessory) to modify accessories of the avatar, context-aware universal avatar editing system can provide avatar accessory modification tabs for various avatar accessories (e.g., hats, glasses), [0119]).
As per Claim 15, Claim 15 is similar in scope to Claim 1, except that Claim 15 is directed to a system comprising: at least one processor; and at least one memory component having instructions stored thereon that, when executed by the at least one processor, cause the at least one processor to perform the method of Claim 1. Shen teaches a system comprising: at least one processor; and at least one memory component having instructions stored thereon that, when executed by the at least one processor, cause the at least one processor to perform the method (processor receives instructions, from a non-transitory computer readable medium (memory), and executes those instructions, thereby performing processes described herein, [0134]). Thus, Claim 15 is rejected under the same rationale as Claim 1.
As per Claims 16-18, these claims are similar in scope to Claims 2-4 respectively, and therefore are rejected under the same rationale. As per Claim 20, Claim 20 is similar in scope to Claim 15, and therefore is rejected under the same rationale.
Claim(s) 5, 7, 8, and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Shen (US 20230281940A1) and Shayani (US 20230326159A1) in view of Li (US 20250173971A1) and Demyanov (US 20220207355A1).
As per Claim 5, Shen and Shayani are relied upon for the teachings as discussed above relative to Claim 1.
However, Shen and Shayani do not teach wherein a first stage of the multiple stages comprises a first plurality of training operations comprising: accessing training data comprising a plurality of training data pairs each including respective training faces and target faces, each of the plurality of training data pairs being associated with a different stylization of a plurality of stylizations; selecting a first batch of the training data associated with a first stylization of the plurality of stylizations, the first batch of the training data comprising a first batch of training faces and a first batch of target faces corresponding to the first stylization; processing an individual training face of the first batch of training faces by the machine learning model to estimate a target face corresponding to the first stylization; computing a deviation between the estimated target face and an individual target face of the first batch of target faces; and updating the parameters of the machine learning model based on the computed deviation. However, Li teaches wherein a first stage of the multiple stages comprises a first plurality of training operations comprising: accessing training data comprising a plurality of training data pairs each including respective training faces and target faces, each of the plurality of training data pairs being associated with a different stylization of a plurality of stylizations (face, mouth area in the original image may be attached into the to-be-used image, so as to obtain the target style image with higher pairing property with the to-be-processed original image, target style image is combined with the corresponding to-be-processed original image, so as to construct a final version of training data in pairs, and a finally used target stylization conversion model may be trained with the training data with higher pairing property, [0065]); selecting a first batch of the training data associated with a first stylization of the plurality of stylizations, the first batch of the training data comprising a first batch of training faces and a first batch of target faces corresponding to the first stylization; processing an individual training face of the first batch of training faces by the machine learning model to estimate a target face corresponding to the first stylization; computing a deviation between the estimated target face and an individual target face of the first batch of target faces; and updating the parameters of the machine learning model based on the computed deviation (for each initial pairing data, an original image in each initial pairing data is used as an input of the first style model to be trained, so as to obtain a first output image corresponding to the original image, a loss value is determined on the basis of the first output image and the initial style image corresponding to the original image, so as to adjust model parameters in the first style model to be trained on the basis of the loss value, and a first loss function in the first style model to be trained is converged as a training target, so as to obtain the style model to be used, [0044], plurality of initial pairing data are acquired, the large amount of initial pairing data may be processed by using the existing first style model to be trained, so as to generate the first output image, after the first output image is obtained, the loss value between the first output image and the initial style image may be determined, when model parameters in the first style model to be trained are corrected by using the loss value, a training error, that is, a loss parameter, of the first loss function in the first style model to be trained may be used as a condition for detecting whether the first loss function reaches convergence, whether the current number of iterations is equal to a preset number of iterations, if it is detected that a convergence condition is met, the training error of the loss function is less than the preset error, iterative training may be stopped, if it is detected that the convergence condition is not met at present, other initial pairing data may be acquired to continue to train the model, until the training error of the loss function is within a preset range, after the original image including the facial information of the user is input into the model, an image having a higher matching degree with the original image may be obtained, [0045]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Shen and Shayani so that a first stage of the multiple stages comprises a first plurality of training operations comprising: accessing training data comprising a plurality of training data pairs each including respective training faces and target faces, each of the plurality of training data pairs being associated with a different stylization of a plurality of stylizations; selecting a first batch of the training data associated with a first stylization of the plurality of stylizations, the first batch of the training data comprising a first batch of training faces and a first batch of target faces corresponding to the first stylization; processing an individual training face of the first batch of training faces by the machine learning model to estimate a target face corresponding to the first stylization; computing a deviation between the estimated target face and an individual target face of the first batch of target faces; and updating the parameters of the machine learning model based on the computed deviation because Li suggests that this way, the image that is generated has a higher matching degree with the original image and is in a desired style type of the user [0045].
However, Shen, Shayani, and Li do not expressly teach randomly initializing parameters of the machine learning model. However, Demyanov teaches randomly initializing parameters of the machine learning model (the condition data and random noise is received by the generator neural network, the pre-trained GAN produces a style vector based on the random noise, [0016]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Shen, Shayani, and Li to include randomly initializing parameters of the machine learning model as suggested by Demyanov. It is well-known in the art that without randomly initializing parameters, the GAN would produce the same output repeatedly. The noise introduces randomness, allowing the GAN to explore the data distribution and create a variety of samples.
As per Claim 7, Claim 7 is similar in scope to Claim 5, except that Claim 7 is directed to performing the same method of Claim 5 on a second batch of the training data associated with a second stylization, a second batch of training faces and a second batch of target faces; a second individual training face, a second target face; a second individual target face. Li teaches determining a plurality of initial pairing data, and performing training on the basis of the plurality of initial pairing data, so as to obtain a style model to be used [0035]. The first style model may use, as the input, each original image including facial information, and outputs an image of a specific style type, an image of the 3D game character style type [0043]. It would have been obvious to one of ordinary skill in the art that after generating an image of a specific style type that is a 3D game character style type, the machine learning model can be used to generate a second image of a second specific style type. Thus, Li teaches wherein the first plurality of training operations further comprise: selecting a second batch of the training data associated with a second stylization of the plurality of stylizations, the second batch of the training data comprising a second batch of training faces and a second batch of target faces corresponding to the second stylization; processing a second individual training face of the second batch of training faces by the machine learning model to estimate a second target face corresponding to the second stylization; computing a deviation between the estimated second target face and a second individual target face of the second batch of target faces; and updating the parameters of the machine learning model based on the computed deviation between the estimated second target face and a second individual target face of the second batch of target faces [0035, 0043, 0044, 0045]. This would be obvious for the reasons given in the rejection for Claim 5.
As per Claim 8, Shen and Shayani do not teach wherein the first plurality of training operations further comprise: specifying a individual condition that represents the second stylization; and inputting the individual condition that represents the second stylization to the machine learning model. However, Li teaches wherein the first plurality of training operations further comprise: selecting a second batch of the training data associated with a second stylization; processing a second individual training face of the second batch by the machine learning model to estimate a second target face corresponding to the second stylization, as discussed in the rejection for Claim 7. It would have been obvious to one of ordinary skill in the art that Li specifies a individual condition that represents the second stylization; and inputs the individual condition that represents the second stylization to the machine learning model, in order for the machine learning model to process the second batch that is associated with a second stylization to estimate a second target face corresponding to the second stylization. This would be obvious for the reasons given in the rejection for Claim 5.
As per Claim 19, Claim 19 is similar in scope to Claim 5, and therefore is rejected under the same rationale.
Claim(s) 6 is/are rejected under 35 U.S.C. 103 as being unpatentable over Shen (US 20230281940A1), Shayani (US 20230326159A1), Li (US 20250173971A1), and Demyanov (US 20220207355A1) in view of Chandran (US012340440B2).
Shen, Shayani, Li, and Demyanov are relied upon for the teachings as discussed above relative to Claim 5.
However, Shen, Shayani, Li, and Demyanov do not teach wherein the first plurality of training operations further comprise: randomly selecting the first stylization from the plurality of stylizations, the first batch of the training data being selected in response to randomly selecting the first stylization. However, Chandran teaches wherein the first plurality of training operations further comprise: randomly selecting the first stylization from the plurality of stylizations, the first batch of the training data being selected in response to randomly selecting the first stylization (training engine trains style transfer model to perform style transfer between pairs of training content samples and training style samples 228 in a set of training data, training engine may generate each pair of samples by randomly selecting a training content sample from a set of training content samples in training data and a training style sample from a set of training style samples 228 in training data, col. 6, lines 60-67; training engine may randomly select the training style sample from a set of training style samples in the training data, col. 9, lines 32-35).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Shen, Shayani, Li, and Demyanov so that the first plurality of training operations further comprise: randomly selecting the first stylization from the plurality of stylizations, the first batch of the training data being selected in response to randomly selecting the first stylization as suggested by Chandran. It is well-known in the art to select training data in response to randomly selecting because this process is crucial for ensuring that the training set accurately reflects the overall distribution and characteristics of the data, thereby preventing bias and improving the model’s ability to generalize to unseen data.
Claim(s) 9-14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Shen (US 20230281940A1), Shayani (US 20230326159A1), Li (US 20250173971A1), and Demyanov (US 20220207355A1) in view of Ghosh (US 20220222872A1).
As per Claim 9, Shen, Shayani, Li, and Demyanov are relied upon for the teachings as discussed above relative to Claim 5.
However, Shen, Shayani, Li, and Demyanov do not expressly teach wherein the first plurality of training operations comprise: determining that the training data for each of the plurality of stylizations has been used to train the machine learning model; and in response to determining that the training data for each of the plurality of stylizations has been used to train the machine learning model, terminating the first plurality of training operations. However, Ghosh teaches wherein the first plurality of training operations comprise: determining that the training data for each of the plurality of stylizations has been used to train the machine learning model; and in response to determining that the training data for each of the plurality of stylizations has been used to train the machine learning model, terminating the first plurality of training operations (training a ML model to stylize input images into a plurality of styles, comprising: obtaining a training set comprising a first plurality of images, wherein the first plurality of images comprises a second plurality of image pairs, wherein each image pair comprises a first image and a version of the first image stylized into one of a plurality of styles, initializing a neural network configured to learn the plurality of styles from the first plurality of images, for each image pair in the second plurality of image pairs: determining a style of the stylized version of the first image from the current image pair, decomposing the first image from the current image pair into a first plurality of features using the neural network, reconstructing the first image form the current image pair using a style vector representative of the determined style to attempt to match the stylized version of the first image from the current image pair, and refining the neural network based on loss functions computed between the reconstructed first image from the current image pair and the stylized version of the first image from the current image pair, wherein refining the neural network further comprises updating the style embedding matrix entry for the determined style of the first image from the current image pair, [0013], finally, the neural network may be refined based on one or more loss functions computed between the reconstructed first image from the current image pair and the stylized version of the first image from the current image pair, updating the style embedding matrix entry for the determined style of the first image from the current image pair, [0046]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Shen, Shayani, Li, and Demyanov so that the first plurality of training operations comprise: determining that the training data for each of the plurality of stylizations has been used to train the machine learning model; and in response to determining that the training data for each of the plurality of stylizations has been used to train the machine learning model, terminating the first plurality of training operations because Ghosh suggests that this enables an image rendering pipeline to be capable of stylizing input images into a plurality of user-selectable predetermined styles—as well as learn new styles in an ad hoc fashion on small sets of input images [0007].
As per Claim 10, Shen, Shayani, and Li do not teach further comprising performing a second plurality of training operations associated with a second stage of the multiple stages after completing training the machine learning model using the training data for each of the plurality of stylizations. However, Demyanov teaches further comprising performing a second plurality of training operations associated with a second stage of the multiple stages after completing training the machine learning model using the training data for the stylization (in operation 802, image manipulation system 130 accesses a pre-trained generative adversarial network (GAN) trained on a primary image domain, [0129], in operation 810, image manipulation system 130 identifies input data of the fine-trained GAN, the input data may comprise a dataset of images from a secondary domain, the secondary image domain is different from the primary image domain, the secondary image domain is a dataset of stylized human faces (cartoon faces, anime faces), [0130], identifies training layers of the fine-tuned GAN, the identified training layers may exclude at least one layer of the fine-tuned GAN, freezes one or more layers of the pre-trained GAN, and only trains the remaining layers, [0131]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Shen, Shayani, and Li to include performing a second plurality of training operations associated with a second stage of the multiple stages after completing training the machine learning model using the training data for the stylization as suggested by Demyanov. It is well-known in the art to pre-train the GAN in the first stage, so that the GAN has already learned fundamental image generation capabilities from vast datasets, which significantly reduces the training time and computational resources needed for the second stage of new tasks, allowing for faster development and easier deployment.
However, Shen, Shayani, Li, and Demyanov do not teach training the machine learning model using the training data for each of the plurality of stylizations. However, Ghosh teaches training the machine learning model using the training data for each of the plurality of stylizations [0013, 0046]. This would be obvious for the reasons given in the rejection for Claim 9.
As per Claim 11, Shen, Shayani, and Li do not teach wherein the second plurality of training operations comprise: accessing new training data comprising new training data pairs each including respective training faces and target faces, each of the new training data pairs being associated with the new augmented reality stylization that is different from the plurality of stylizations; selecting an individual batch of the new training data associated with the new augmented reality stylization, the individual batch of the new training data comprising an individual batch of training faces and an individual batch of target faces corresponding to the new augmented reality stylization; and accessing the machine learning model with previously trained parameters corresponding to the plurality of stylizations. However, Demyanov teaches wherein the second plurality of training operations comprise: accessing new training data comprising new training data pairs each including respective training faces and target faces, each of the new training data pairs being associated with the new augmented reality stylization (image manipulation system may fine-tune the pre-trained GAN by training the pre-trained GAN on a secondary image domain, the secondary image domain may be stylized face images (cartoon face images or anime face images), the image manipulation system generates a pair of images: one human face using the pre-trained GAN and another stylized human face using the fine-tuned GAN, [0018]) that is different from the stylization (pre-trained generative adversarial network trained on a primary image domain, [0129], secondary image domain is different from the primary image domain, [0130]); selecting an individual batch of the new training data associated with the new augmented reality stylization, the individual batch of the new training data comprising an individual batch of training faces and an individual batch of target faces corresponding to the new augmented reality stylization; and accessing the machine learning model with previously trained parameters corresponding to the stylization [0018, 0129]. This would be obvious for the reasons given in the rejection for Claim 10.
However, Shen, Shayani, Li, and Demyanov do not teach that the new stylization is different from the plurality of stylizations; and the machine learning model with previously trained parameters corresponding to the plurality of stylizations. However, Ghosh teaches training the machine learning model using the training data for each of the plurality of stylizations [0013, 0046]. Thus, this teaching from Ghosh can be implemented into the first plurality of training operations of Demyanov so that the new stylization is different from the plurality of stylizations; and the machine learning model with previously trained parameters corresponding to the plurality of stylizations. This would be obvious for the reasons given in the rejection for Claim 9.
As per Claim 12, Claim 12 is similar in scope to Claim 5, except that Claim 12 is directed to performing the same method of Claim 5 for the second plurality of training operations on the individual batch of training faces, the new augmented reality stylization; the individual batch of target faces; the previously trained parameters. Shen, Shayani, and Li do not teach the second plurality of training operations on the individual batch of training faces, the new augmented reality stylization; the individual batch of target faces; the previously trained parameters. However, Demyanov teaches wherein the second plurality of training operations further comprise: processing an individual training face of the individual batch of training faces by the machine learning model to estimate a target face corresponding to the new augmented reality stylization [0018]; updating the previously trained parameters of the machine learning model (adjusts weights of neural network layers of the pre-trained GAN using weights of neural network layers of the fine-tuned GAN, [0133]). Thus, this teaching of the second plurality of training operations on the individual batch of training faces, the new augmented reality stylization, the previously trained parameters from Demyanov can be implemented into the method of Li so that the same method of Claim 5 is performed for the second plurality of training operations on the individual batch of training faces, the new augmented reality stylization; the individual batch of target faces; the previously trained parameters. Thus, Claim 12 is rejected under the same rationale as Claim 5 along with this additional teaching from Demyanov. This would be obvious for the reasons given in the rejection for Claim 10.
As per Claim 13, Shen, Shayani, and Li do not teach wherein the machine learning model is trained to generate images corresponding to the new augmented reality stylization in the second stage in less time than being trained to generate the plurality of stylizations in the first stage. However, Demyanov teaches wherein the machine learning model is trained to generate the stylization in the first stage [0129]. The machine learning model is trained to generate images corresponding to the new augmented reality stylization in the second stage [0018]. Demyanov teaches identifying training layers of the fine-tuned GAN. The identified training layers exclude at least one layer of the fine-tuned GAN. The image manipulation system freezes one or more layers of the pre-trained GAN, and only trains the remaining layers [0131]. Thus, since all of the layers were trained in order to produce the pre-trained GAN, and less than all of the layers of the pre-trained GAN are trained in the second stage, this means that the machine learning model is trained to generate images corresponding to the new augmented reality stylization in the second stage [0018] in less time than being trained to generate the stylization in the first stage [0131, 0129]. This would be obvious for the reasons given in the rejection for Claim 10.
However, Shen, Shayani, Li, and Demyanov do not teach generating the plurality of stylizations in the first stage. However, Ghosh teaches wherein the machine learning model is trained to generate the plurality of stylizations in the first stage [0013, 0046]. This would be obvious for the reasons given in the rejection for Claim 9.
As per Claim 14, Shen, Shayani, and Li do not teach wherein the second plurality of training operations further comprise: identifying a subset of the previously trained parameters associated with one or more of the plurality of stylizations; and truncating the subset of the previously trained parameters to reduce an amount of time used to train the machine learning model for the new augmented reality stylization. However, Demyanov teaches the machine learning model is trained to generate images corresponding to the new augmented reality stylization in the second stage [0018]. Demyanov teaches identifying training layers of the fine-tuned GAN. The identified training layers exclude at least one layer of the fine-tuned GAN. The image manipulation system freezes one or more layers of the pre-trained GAN, and only trains the remaining layers [0131]. Demyanov teaches the weights of the layers [0017]. Thus, the second plurality of training operations further comprise: identifying layers of the pre-trained GAN; and excluding those layers, which excludes the weights of those layers, which truncates the subset of the previously trained layers to reduce an amount of time used to train the machine learning model for the new augmented reality stylization. Thus, Demyanov teaches wherein the second plurality of training operations further comprise: identifying a subset of the previously trained parameters associated with one or more of the plurality of stylizations; and truncating the subset of the previously trained parameters to reduce an amount of time used to train the machine learning model for the new augmented reality stylization [0131, 0017, 0018]. This would be obvious for the reasons given in the rejection for Claim 10.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JONI HSU whose telephone number is (571)272-7785. The examiner can normally be reached M-F 10am-6:30pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kee Tung can be reached at (571)272-7794. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
JH
/JONI HSU/Primary Examiner, Art Unit 2611