DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-8, 11, 13-17, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Sutton (US 20240062445 A1) in view of Milman et al. (US 20190340419 A1) (Hereinafter referred to as Milman).
Regarding Claim 1, Sutton discloses A non-transitory computer-readable medium having stored thereon instructions configured to, when executed by one or more computing devices of a computer system, cause the computer system to perform operations comprising: (See Claim 24, “A computer-readable medium having non-transitory instruction included . . .” Also see Fig. 10 showing a computer system.)
obtaining digital image data of a live subject; (See [0024], “In accordance with aspects of the present disclosure, a user may, for example, use an image of the user's face to easily create an avatar having the user's likeness as a starting point for customization.”)
extracting one or more reference attributes of the live subject based on the digital image data of the live subject; (See [0027], “As shown, initially an image of a face is provided to the system as indicated at 202. The image may be digitized and facial features may be recognized at 202 from digital data corresponding to the image, for example and without limitation the existence of a face shape, head size, eye color, hair color or hair style may be detected. From the recognition of facial features, feature parameter values are extracted at 203.” In this case, the extracted feature parameter values taught by Sutton corresponds to “one or more reference attributes”.)
executing a machine learning model to extract the one or more reference attributes; (See [0036], “In some implementations, facial feature parameters and/or facial feature parameter values may be extracted using one or more neural networks trained with a machine learning algorithm configured to determine facial feature parameters and/or facial feature parameter values as discussed above.”)
by an avatar customization engine, modifying a graphical avatar based on the one or more reference attributes. (See [0028], “In this implementation, the system has access to a database of pre-generated facial models. . . A pre-generated facial model may be selected from the database based on some subset of the feature parameters and/or feature parameter values, as indicated at 204.”
Also see [0029], “Once the facial model has been selected or generated, the model is modified based on the facial feature parameters and/or facial feature parameter values, as indicated at 205.” Note, although an “avatar customization engine” is not explicitly referenced in Sutton, since Sutton already teaches its functions, then the above limitation is still taught.)
However, Sutton fails to explicitly disclose executing a machine learning model using the one or more reference attributes of the live subject as input to generate avatar customization data as output; and
by an avatar customization engine, modifying a graphical avatar based on the avatar customization data generated by the machine learning model.
Milman teaches executing a machine learning model using the one or more reference attributes of the live subject as input to generate avatar customization data as output; and (See [0017], “In one or more implementations, the avatar generation system is formed based on a framework that initially provides the digital photograph to a facial identity network, which generates a condensed parameter vector indicative of features of the depicted person's face. In such implementations, the avatar generation system then provides this facial feature vector as the input to the trained machine-learning model.” Here, similar to Sutton, Milman also teaches generating a facial feature vector (one or more reference attributes) based on an image of a user. Then, Milman then teaches to use this feature vector as input into a machine learning model.
See [0050], “In other words, these facial feature vectors are “real person” feature vectors. The avatar generation framework 120 then provides these “real person” feature vectors as input to the parameter network 302, which outputs parameterized “cartoon” feature vectors.” In other words, Milman uses the extracted feature vector as input to a machine learning model to output a cartoon/stylized form of these features. Thus, the outputted parameterized cartoon feature vector corresponds to “avatar customization data”.)
by an avatar customization engine, modifying a graphical avatar based on the avatar customization data generated by the machine learning model. (See [0050] teaching the parameterized cartoon feature vector (avatar customization data). In combination with Sutton [0029] teaching to modify an avatar based on the facial feature parameter/values, the above limitation is taught.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Sutton with Milman to include a machine learning model that takes the one or more reference attributes as input and outputs avatar customization data and then modifying the avatar based on the outputted avatar customization data.
The motivation to combine Sutton with Milman would have been obvious as both Sutton and Milman are both arts relating to generating an avatar based on an image of the user (See Milman Abstract). The benefit of including a machine learning model that outputs avatar customization data from inputted reference attributes is that it can be used to stylize a person’s features which would result in a more unique or personalized avatar.
Regarding Claim 2, Sutton in view of Milman discloses The non-transitory computer-readable medium of Claim 1, wherein the machine learning model is trained on a training set of images using a supervised learning approach. (See Sutton [0017], “FIG. 9D shows a flow diagram depicting a method for supervised training of a machine learning neural network according to aspects of the present disclosure.”
See Milman [0018], “The trained machine-learning model corresponds to the parameter-network portion, once trained. Initially, the framework is trained using supervised training of the parameter network. . . As noted just above, this relatively small set of training data includes pairs of digital photographs of persons and parameters of matching digital cartoon images, where the parameters are manually created parameterizations of the matching cartoon images to train the parameter network under ‘supervision.’” The motivation to combine would have been similar to that of Claim 1 rejection motivation.)
Regarding Claim 3, Sutton in view of Milman discloses The non-transitory computer-readable medium of Claim 2, wherein the training set of user images is labeled with corresponding avatar attributes. (See Sutton [0040], “One or more object recognition neural networks may be trained from a database of labeled objects . . .”
See Milman [0018], “The trained machine-learning model corresponds to the parameter-network portion, once trained. Initially, the framework is trained using supervised training of the parameter network. . . As noted just above, this relatively small set of training data includes pairs of digital photographs of persons and parameters of matching digital cartoon images, where the parameters are manually created parameterizations of the matching cartoon images to train the parameter network under ‘supervision.’” The motivation to combine would have been similar to that of Claim 1 rejection motivation.)
Regarding Claim 4, Sutton in view of Milman discloses The non-transitory computer-readable medium of Claim 3, wherein the one or more reference attributes comprise a first set of hexadecimal color values, (See Sutton [0027], “The image may be digitized and facial features may be recognized at 202 from digital data corresponding to the image, for example and without limitation the existence of a face shape, head size, eye color, hair color or hair style may be detected. From the recognition of facial features, feature parameter values are extracted at 203.” Here, Sutton teaches that facial features can include eye color or hair color (a first set of color values). Note that color is commonly known to be represented by a hexadecimal code value, so although not explicitly stated, it would be implicit that the color values are hexadecimal color values.)
and wherein the avatar customization data generated by the machine learning model comprises a second set of hexadecimal color values. (See Milman [0050], “In other words, these facial feature vectors are “real person” feature vectors. The avatar generation framework 120 then provides these “real person” feature vectors as input to the parameter network 302, which outputs parameterized “cartoon” feature vectors.” In this case, when combined with Sutton, since the feature vectors can include color values, then the outputted parameterized cartoon feature would also include color values which can be considered as “a second set of hexadecimal color values”. The motivation to combine would have been similar to that of Claim 1 rejection motivation.)
Regarding Claim 5, Sutton in view of Milman discloses The non-transitory computer-readable medium of Claim 1, wherein the one or more reference attributes comprise a hair color value of the live subject, (See Sutton [0027], “The image may be digitized and facial features may be recognized at 202 from digital data corresponding to the image, for example and without limitation the existence of a face shape, head size, eye color, hair color or hair style may be detected. From the recognition of facial features, feature parameter values are extracted at 203.”)
wherein the machine learning model uses the hair color value of the live subject as input to generate the avatar customization data, and wherein the avatar customization data comprises one or more avatar hair color values. (See Milman [0050] teaching inputting feature vectors into a machine learning model to obtain parameterized cartoon feature vectors. Once again, since Sutton’s feature vectors can include hair color, then the outputted parametrized carton feature vectors also include hair color and thus correspond to “avatar customization data comprises one or more avatar hair color values”. The motivation to combine would have been similar to that of Claim 1 rejection motivation.)
Regarding Claim 6, Sutton in view of Milman discloses The non-transitory computer-readable medium of Claim 1, wherein the digital image data of the live subject comprises digital image data of a face area of the live subject. (See Sutton [0020], “Here, the subject may be a person's body, a person's face, an animal, any type of inanimate object, building, a plant or similar.”)
Regarding Claim 7, Sutton in view of Milman discloses The non-transitory computer-readable medium of Claim 6, wherein the one or more reference attributes comprise a face color value of the live subject, (See Sutton [0027], “As discussed above the feature parameters may include facial landmarks such as eye width, eye shape, eye separation, eye size eye color, nose width, nose length, mouth shape, mouth location, mouth width, beard shape, beard location, beard width, beard length, hair shape, hair size, hair line location, hair color, hair texture, head shape, head length, head width, jaw shape, skin tone, or any combination thereof or similar.”)
wherein the avatar customization data comprises an avatar face color value, (See Milman [0050] teaching inputting feature vectors into a machine learning model to obtain parameterized cartoon feature vectors. Since Sutton’s feature vectors can include skin tone (face color), then the outputted parametrized carton feature vectors also include a face color and thus correspond to “avatar customization data comprises an avatar face color value”.)
wherein modifying the graphical avatar based on the avatar customization data comprises modifying a face area of the graphical avatar based on the avatar face color value, and wherein the modified face area of the graphical avatar corresponds to the face area of the live subject. (See Sutton [0029], “Once the facial model has been selected or generated, the model is modified based on the facial feature parameters and/or facial feature parameter values, as indicated at 205. The facial feature parameters and/or facial feature parameter values taken from the image of the face may be applied to the pre-set feature parameters to modify the appearance of the facial model.” The motivation to combine would have been similar to that of Claim 1 rejection motivation.)
Regarding Claim 8, Sutton in view of Milman discloses The non-transitory computer-readable medium of Claim 7, wherein the face color value of the live subject corresponds to a cosmetic worn on the face area of the live subject, and wherein the modified face area of the graphical avatar includes a depiction of the cosmetic applied to the modified face area of the graphical avatar. (See Sutton [0029], “Additionally certain modifications to the facial model may be made by adding features to the model based on certain feature parameters determined from the image. By way of example and without limitation, added features may include a beard, a moustache, a hair style, one or more tattoos, one or more piercings, one or more accessories (e.g., eyeglasses or sunglasses) one or more scars, one or more blemishes, one or more eyebrows, or any combination thereof.”)
Regarding Claim 11, Sutton in view of Milman discloses The computer-readable medium of Claim 1, wherein the digital image data of the live subject comprises digital image data of hair of the live subject, and wherein the one or more reference attributes include length, texture, or style of hair of the live subject, the operations further comprising: (See Sutton [0025], “Here, the feature parameters characterize facial landmarks and may include, but are not limited to Eye width 107, eye shape, eye separation, eye color, eye size 113, nose width 109, nose length 114, mouth shape, mouth location, mouth width 108, beard shape, beard location, beard width 110, beard length 117, hair shape, hair size, hair color, hair texture, hairline location 110, head shape, head length 115, head width, jaw shape 116, or any combination thereof or similar.”)
executing a machine learning model using the length, texture, or style of the hair of the live subject as input to generate avatar hair customization data as output; and (See Milman [0050] teaching inputting feature vectors into a machine learning model to obtain parameterized cartoon feature vectors. Since Sutton’s feature vectors can include hair shape, hair size, hair color, hair texture (length, texture, or style of the hair), then the outputted parametrized carton feature vectors also include hair shape, hair size, hair color, hair texture and thus correspond to “avatar hair customization data”.)
by the avatar customization engine, further modifying the graphical avatar based on the avatar hair customization data. (See Sutton [0029] teaching that the model is modified based on the facial feature parameters. Combined with Milman [0050] and Sutton [0025] teaching the avatar hair customization data, the above limitation is taught. The motivation to combine would have been similar to that of Claim 1 rejection motivation.)
Regarding Claim 13, Sutton in view of Milman discloses A computer-implemented method executed by one or more computing devices of a computer system, the method comprising: (See Sutton Claim 1, “A method for image-based customization, comprising: . . .” Also see Sutton Fig. 10 showing a computer system.)
obtaining digital image data of a live subject; extracting one or more reference attributes of the live subject based on the digital image data of the live subject; executing a machine learning model using the one or more reference attributes of the live subject as input to generate avatar customization data as output; and by an avatar customization engine, modifying a graphical avatar based on the avatar customization data generated by the machine learning model. (The above limitations are similar to those of Claim 1 and are therefore rejected under a similar rationale as that of Claim 1.)
Regarding Claim 14, Claim 14 contains similar limitations as to Claim 3 and is therefore rejected under a similar rationale as that of Claim 3.
Regarding Claim 15, Claim 15 contains similar limitations as to Claim 4 and is therefore rejected under a similar rationale as that of Claim 4.
Regarding Claim 16, Claim 16 contains similar limitations as to Claim 5 and is therefore rejected under a similar rationale as that of Claim 5.
Regarding Claim 17, Claim 17 contains similar limitations as to Claim 8 and is therefore rejected under a similar rationale as that of Claim 8.
Regarding Claim 19, Claim 19 contains similar limitations as to Claim 11 and is therefore rejected under a similar rationale as that of Claim 11.
Claims 9 is rejected under 35 U.S.C. 103 as being unpatentable over Sutton in view of Milman and in further view of Ghosh et al. (US 20240404225 A1) (Hereinafter referred to as Ghosh).
Regarding Claim 9, Sutton in view of Milman discloses The non-transitory computer-readable medium of Claim 1, wherein the digital image data of the live subject comprises digital image data of a face area of the live subject, and (See Sutton [0027], “As shown, initially an image of a face is provided to the system as indicated at 202.”)
executing a machine learning model using the one or more reference attributes input to generate avatar customization data as output; and (See Milman [0050] teaching inputting feature vectors into a machine learning model to obtain parameterized cartoon feature vectors.)
by the avatar customization engine, further modifying the graphical avatar based on the avatar customization data, wherein further modifying the graphical avatar based on the avatar customization data comprises modifying a face area of the graphical avatar based on the avatar customization data, and wherein the modified face area of the graphical avatar corresponds to the face area of the live subject. (See Milman [0050] teaching the avatar customization data and see Sutton [0029] teaching to modify a model based on facial feature parameters.)
However, Sutton in view of Milman fails to explicitly disclose wherein the one or more reference attributes include a live texture value of the face area of the live subject, the operations further comprising:
executing a machine learning model using the live texture value of the face area of the live subject as input to generate avatar texture customization data as output; and
by the avatar customization engine, further modifying the graphical avatar based on the avatar texture customization data, wherein further modifying the graphical avatar based on the avatar texture customization data comprises modifying a face area of the graphical avatar based on the avatar texture customization data, and wherein the modified face area of the graphical avatar corresponds to the face area of the live subject.
Ghosh teaches wherein the one or more reference attributes include a live texture value of the face area of the live subject, the operations further comprising: (See [0113], “Facial features include the overall shape of a person's face (e.g., round, oval, square, heart-shaped) and the proportions of the face (e.g., the relative positions and sizes of the eyes, nose, and mouth); the size, shape, . . . skin color and texture, as well as any distinctive marks like freckles, moles, or scars; . . .” Here, skin texture corresponds to “a live texture value of the face area of the live subject”. In combination with Sutton [0027] already teaching feature parameter values (one or more reference attributes), the above limitation is taught.)
executing a machine learning model using the live texture value of the face area of the live subject as input to generate avatar texture customization data as output; and (see Ghosh [0113] teaching skin texture as being a part of the facial features. In combination with Milman [0050] teaching inputting feature vectors into a machine learning model to obtain parameterized cartoon feature vectors, the resulting output can be considered as “avatar texture customization data”.)
by the avatar customization engine, further modifying the graphical avatar based on the avatar texture customization data, wherein further modifying the graphical avatar based on the avatar texture customization data comprises modifying a face area of the graphical avatar based on the avatar texture customization data, and wherein the modified face area of the graphical avatar corresponds to the face area of the live subject. (see Ghosh [0113] teaching skin texture. In combination with Milman [0050], they teach the avatar texture customization data. Then based on Sutton [0029] teaching to modify a model based on facial feature parameters, the above limitations are taught as the avatar’s skin texture would also be a parameter that would be now modified.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Sutton in view of Milman with Ghosh to include detecting a texture of the subject.
The motivation to combine Sutton in view of Milman with Ghosh would have been obvious as all three arts are within the same field of generating avatars from user-self images (See Ghosh Abstract). Ghosh is simply teaching that texture is a common feature to extract and work with when dealing with the generation of a virtual avatar. The benefit of including texture is that it can make the resulting avatar more unique and personalized to the user as they would now potentially share a similar texture.
Claims 12 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Sutton in view of Milman and in further view of Maric et al. (US 20200356163 A1) (Hereinafter referred to as Maric).
Regarding Claim 12, Sutton in view of Milman discloses The computer-readable medium of Claim 1, wherein the digital image data of the live subject comprises digital image data of eyes of the live subject, and wherein the one or more reference attributes comprise length, density, color, or texture of eye of the live subject, the operations further comprising: (See Sutton [0025], “Here, the feature parameters characterize facial landmarks and may include, but are not limited to Eye width 107, eye shape, eye separation, eye color, eye size 113 . . .”)
executing a machine learning model using the length, density, color, or texture of the eye of the live subject as input to generate avatar eye customization data as output; and (See Milman [0050] teaching inputting feature vectors into a machine learning model to obtain parameterized cartoon feature vectors. Combined with Sutton [0025], the outputted parameterized cartoon feature vectors would include eye related information and thus generate “avatar eye customization data”.)
by the avatar customization engine, further modifying the graphical avatar based on the avatar eye customization data. (See Sutton [0029] teaching to modify the avatar based on facial parameters. Combined with Milman [0050], they teach to modify the avatar based on avatar eye customization data.)
However, Sutton in view of Milman fails to explicitly disclose wherein the digital image data of the live subject comprises digital image data of eyelashes of the live subject, and wherein the one or more reference attributes comprise length, density, color, or texture of eyelashes of the live subject, the operations further comprising:
executing a machine learning model using the length, density, color, or texture of the eyelashes of the live subject as input to generate avatar lash customization data as output; and
by the avatar customization engine, further modifying the graphical avatar based on the avatar lash customization data.
Maric teaches wherein the digital image data of the live subject comprises digital image data of eyelashes of the live subject, and wherein the one or more reference attributes comprise length, density, color, or texture of eyelashes of the live subject, the operations further comprising: (See [0070], “At block 320, a feature (sometimes referred to as a landmark) of the face is identified. In some examples, the electronic device identifies the feature. . . features related to eyelids and eyelashes, features related to additional facial features such as moles or freckles . . . eyelash length, width of a user's nose bridge . . .” In combination with Sutton [0025], instead of extracting information related to eyes as the reference attributes, one can identify features related to eyelashes. Thus the above limitations are taught.)
executing a machine learning model using the length, density, color, or texture of the eyelashes of the live subject as input to generate avatar lash customization data as output; and (See [0070] teaching identifying eyelash features such as eyelash length. In combination with Milman [0050] and Sutton [0025], the output can be considered as “avatar lash customization data”.)
by the avatar customization engine, further modifying the graphical avatar based on the avatar lash customization data. (See [0070] teaching identifying eyelash features such as eyelash length. In combination Sutton [0029] and Milman [0050], they teach to modify the avatar based on avatar lash customization data.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Sutton in view of Mutton with Maric to include using eyelash features during the modification of an avatar from a user’s self-image.
The motivation to combine Sutton in view of Mutton with Maric would have been obvious as Sutton and Maric is in the same field of identifying features on a face (See Maric Abstract). Maric simply teaches that identifying eyelashes would be a normal and common part of the process when looking the features on a face. Sutton already teaches extracting features relating to the eyes, thus extracting features related to eyelashes would have been an obvious as eyelashes can be considered apart of the eyes portion of the face.
Regarding Claim 20, Claim 20 contains similar limitations as to Claim 12 and is therefore rejected under a similar rationale as that of Claim 12.
Allowable Subject Matter
Claims 10 and 18 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter:
Regarding Claim 10, the cited prior art does not disclose or render obvious the combination of elements cited in the claims as a whole. Specifically, the cited prior art fails to disclose or render obvious the limitations: wherein the modified face area of the graphical avatar includes a depiction of a cosmetic applied to the modified face area of the graphical avatar, the depicted cosmetic having a texture or finish based on the avatar texture customization data. Thus Claim 10 contains allowable subject matter.
Regarding Claim 18, Claim 18 contains similar limitations as to Claim 10 and therefore contains similar allowable subject matter.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Reference Zohar et al. (US 20240265608 A1) is made of record as an art that teaches extract one or more features of the given person depicted in the individual image (See [0019]), and applying a machine learning model to the extracted features of the given person to generate an avatar that resembles the given person (See [0057]), updating the avatar based on input that adjusts the one or more parameters of the avatar and the one or more parameters includes skin tone and hair color (See [0095]).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to THANG G HUYNH whose telephone number is (571)272-5432. The examiner can normally be reached Mon-Thu 7:30am-4:30pm EST | Fri 7:30am-11:30am EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kee Tung can be reached at (571)272-7794. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/T.G.H./Examiner, Art Unit 2611
/KEE M TUNG/Supervisory Patent Examiner, Art Unit 2611
Prosecution Insights