DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 17-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. The claim(s) does/do not fall within at least one of the four categories of patent eligible subject matter because the claimed computer-readable media includes embodiments to be a signal or carrier wave. This interpretation of the claimed medium being a signal is based upon the explanation given in the specification. The publication of the specification in paragraph [0085] gives an open-ended definition of the claimed computer-readable medium including some examples as to what the claimed medium could possibly be (i.e. a RAM, ROM, etc.). It is noted however that these are simply non-limiting examples of computer-readable media. Nothing in the specification explicitly forbids the claimed computer-readable media from also including other possible embodiments such as signals or carrier waves.
Claims that recite nothing but the physical characteristics of a form of energy, such as a frequency, voltage, or the strength of a magnetic field, define energy or magnetism, per se, and as such are nonstatutory natural phenomena. One useful and possibly helpful description of the claimed computer-readable media is to indicate in the claim that it is a "non-transitory" type of computer readable medium.
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claims 1, 9, 11, 17, and 19 are provisionally rejected on the ground of nonstatutory double patenting as being unpatentable over claim 1, 10, 11, 19 and 20 of copending Application No. 18.954,251 (reference application). Although the claims at issue are not identical, they are not patentably distinct from each other because the notion of the claims of the current application correspond with the claims in Application No. 18/954,251.
Below is a limitation mapping between the current application and the claims of Application No. 18/954,251.
Current Application (18/954,238)
Reference Application (18.954,251)
Claim
Limitation
Claim
Limitation
1
A method comprising: accessing information indicating a correspondence
1
A method comprising: accessing information indicating a first correspondence
between a first three-dimensional landmark of a source three-dimensional head mesh representing a first face and a second three-dimensional landmark of a target three-dimensional head mesh representing a second face
between a first three-dimensional landmark of a source three-dimensional face mesh representing a first face and a second three-dimensional landmark of a target three-dimensional head mesh representing a second face
the first three-dimensional landmark and the second three-dimensional landmark associated with a facial feature;
the first three-dimensional landmark and the second three-dimensional landmark corresponding to a facial feature;
accessing a first three-dimensional head mesh representing a third face and showing the facial feature;
accessing a two-dimensional face image representing a third face, the two-dimensional face image further showing the facial feature;
and generating a second three-dimensional head mesh representing a fourth face based on the first three-dimensional head mesh and the correspondence between the first three-dimensional landmark and the second three-dimensional landmark,
and generating a three-dimensional head mesh based on the two-dimensional face image and the first correspondence between the first three-dimensional landmark and the three-dimensional second landmark,
the second three-dimensional head mesh comprising a first facial characteristic of the second face and a second facial characteristic of the third face.
the three-dimensional head mesh representing a fourth face and including a first facial characteristic of the second face and a second facial characteristic of the third face.
9
A computer system comprising: processor circuitry;
10
A computer system comprising: processor circuitry;
and one or more computer-readable media comprising a sequence of instructions that, when executed, cause the processor circuitry to:
and one or more computer-readable media comprising a sequence of instructions that, when executed, cause the processor circuitry to:
access information indicating a correspondence between a first landmark of a source three-dimensional head mesh representing a first face and a second landmark of a target three-dimensional head mesh representing a second face, the first landmark and the second landmark associated with a facial feature;
access information indicating a first correspondence between a first three-dimensional landmark of a source three-dimensional face mesh representing a first face and a second three-dimensional landmark of a target three-dimensional head mesh representing a second face, the first three-dimensional landmark and the second three-dimensional landmark corresponding to a facial feature;
and generate a second three-dimensional head mesh representing a fourth face based on the first three-dimensional head mesh and the correspondence between the first landmark and the second landmark,
and generate a three-dimensional head mesh based on the two-dimensional face image and the first correspondence between the first three-dimensional landmark and the three-dimensional second landmark,
the second three-dimensional head mesh comprising a first facial characteristic of the second face and a second facial characteristic of the third face.
the three-dimensional head mesh representing a fourth face and including a first facial characteristic of the second face and a second facial characteristic of the third face.
11
The computer system of claim 9, wherein the sequence of instructions that, when executed,
11
The computer system of claim 10, wherein the sequence of instructions that, when executed,
further cause the processor circuitry to:
further cause the processor circuitry to:
determine a first two-dimensional landmark based on a two-dimensional representation of the first face;
and determine a second two-dimensional landmark based on a two-dimensional representation of the second face,
determine a second correspondence between a first two-dimensional landmark of a first two-dimensional facial rendering representing the first face and a second two-dimensional landmark of a second two-dimensional facial rendering representing the second face;
wherein the first three-dimensional landmark is based on the first two-dimensional landmark,
determine the first three-dimensional landmark based on the first two-dimensional landmark;
and wherein the second three-dimensional landmark is based on the second two-dimensional landmark.
wherein the first correspondence is based on the second correspondence.
17
One or more computer-readable media comprising a sequence of instructions that,
19
One or more computer-readable media comprising a sequence of instructions that,
when executed, cause processor circuitry to:
when executed, cause processor circuitry to:
access information indicating a correspondence between a first landmark of a source three-dimensional head mesh representing a first face and a second landmark of a target three-dimensional head mesh representing a second face
access information indicating a first correspondence between a first three-dimensional landmark of a source three-dimensional face mesh representing a first face and a second three-dimensional landmark of a target three-dimensional head mesh representing a second face
the first landmark and the second landmark associated with a facial feature;
the first three-dimensional landmark and the second three-dimensional landmark corresponding to a facial feature;
access a first three-dimensional head mesh representing a third face and showing the facial feature;
access a two-dimensional face image representing a third face, the two-dimensional face image further showing the facial feature
and generate a second three-dimensional head mesh representing a fourth face based on the first three-dimensional head mesh and the correspondence between the first landmark and the second landmark,
and generate a three-dimensional head mesh based on the two-dimensional face image and the first correspondence between the first three-dimensional landmark and the three-dimensional second landmark,
the second three-dimensional head mesh comprising a first facial characteristic of the second face and a second facial characteristic of the third face.
the three-dimensional head mesh representing a fourth face and including a first facial characteristic of the second face and a second facial characteristic of the third face.
19
The one or more computer-readable media of claim 17,
20
The one or more computer-readable media of claim 19,
wherein the sequence of instructions that, when executed,
wherein the sequence of instructions that, when executed,
further cause the processor circuitry to:
further cause processor circuitry to:
determine a first two-dimensional landmark based on a two-dimensional representation of the first face;
determine a second correspondence between a first two-dimensional landmark of a first two-dimensional facial rendering representing the first face
and determine a second two-dimensional landmark based on a two-dimensional representation of the second face,
and a second two-dimensional landmark of a second two-dimensional facial rendering representing the second face;
wherein the first three-dimensional landmark is based on the first two-dimensional landmark,
and determine the second three-dimensional landmark based on the second two-dimensional landmark,
and wherein the second three-dimensional landmark is based on the second two-dimensional landmark.
wherein the first correspondence is based on the second correspondence.
This is a provisional nonstatutory double patenting rejection because the patentably indistinct claims have not in fact been patented.
Allowable Subject Matter
Claims 3-4, 11-12, and 19-20 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-2, 6, 9-10, 14, 17-18 are rejected under 35 U.S.C. 102(a)(1) as anticipated over Onizuka et al. (NPL Titled: “Landmark-guided deformation transfer of template facial expressions for automatic generation of avatar blendshapes”).
As per claim 1, Onizuka teaches the claimed:
A method comprising:
accessing information indicating a correspondence between a first three-dimensional landmark of a source three-dimensional head mesh representing a first face (In figure 9, in the top row the neutral shape represents a 3D head mesh representing a first face (a person labelled “Source”).
Also, please see section 4.2, 1st paragraph where they refer to: “In this section we describe our proposed method to robustly and precisely identify the face correspondences between the source and target neutral mesh (as illustrated in fig. 2).”
In this passage, the source mesh corresponds to the claimed “a source three-dimensional head mesh”)) and a second three-dimensional landmark of a target three-dimensional head mesh representing a second face (In figure 9, in the bottom row the neutral shape represents a 3D head mesh representing a second face (a video game character).
Also, please see section 4.2, 1st paragraph where they refer to: “In this section we describe our proposed method to robustly and precisely identify the face correspondences between the source and target neutral mesh (as illustrated in fig. 2).”
In this passage, the target mesh corresponds to the claimed “a target three-dimensional head mesh”.
Also, please see section 4.2.2 in the text below where figure 4 appears. This portion recites: “the vertex v is a vertex on the source neutral mesh, L is a list of facial landmark indices on the source mesh (as described below), corr(i) is index of the ith landmark of the avatar mesh corresponding to the ith landmark of the source mesh”), the first three-dimensional landmark and the second three-dimensional landmark associated with a facial feature (Please see section 4.2.2 in the text below where figure 4 appears. This portion recites: “We use 51 facial landmarks around the eyes, eyebrows, nose and mouth as in dlib 1. The source mesh is already given a fixed set of land marks L.”);
accessing a first three-dimensional head mesh representing a third face and showing the facial feature (In figure 9, in the top row where the 1st 3D head mesh of the person represents different facial expressions. For instances, in the 2nd column, the 1st 3D head mesh of the person represents a person winking their eye. Thus, this corresponds to the claimed “third face”); and
generating a second three-dimensional head mesh representing a fourth face based on the first three-dimensional head mesh (In figure 9, in the bottom row where the 2nd 3D head mesh of the person represents different facial expressions. For instances, in the 2nd column, the 2nd 3D head mesh of the video game character (i.e. “Mario”) represents Mario winking his eye. Thus, this corresponds to the claimed “fourth face”) and the correspondence between the first three-dimensional landmark and the second three-dimensional landmark (The invention teaches that facial expressions are mapped from the source to the target meshes by using corresponding landmarks between them, e.g. please see section 4.2.2 in the text below where figure 4 appears. This portion recites: “… L is a list of facial landmark indices on the source mesh (as described below), corr(i) is index of the ith landmark of the avatar mesh corresponding to the ith landmark of the source mesh”), the second three-dimensional head mesh comprising a first facial characteristic of the second face (In figure 9 in the bottom row, 2nd column, the winking Mario character (2nd 3D head mesh) is wearing a hat and has a mustache and large nose (a first facial characteristic of the second face) and a second facial characteristic of the third face (In addition, the winking Mario character (2nd 3D head mesh) also has his left eye closed to represent the winking (a second facial characteristic of the third face)).
As per claim 2, Onizuka teaches the claimed:
The method of claim 1, wherein generating second three-dimensional head mesh comprises (In figure 9 in the bottom row, 2nd column, the winking Mario character (2nd 3D head mesh) is wearing a hat and has a mustache and large nose (a first facial characteristic of the second face):
adjusting, using a mesh retargeting technique (Please see the bottom left column on page 8. This portion recites: “Then we re-targeted these expressions to the avatar meshes with the key-shape created with our proposed method”), a vertex of the first three-dimensional head mesh based on the correspondence between the first three-dimensional landmark of a source three-dimensional head mesh (Please see bottom left column of page 5 here Onizuka teaches “By minimizing the weighted linear sum of the above three cost functions, the vertices coordinates of the avatar mesh after deformation ˜ v1,...,˜ vn can be estimated. The constrained optimization problem to solve is expressed as follows
PNG
media_image1.png
275
489
media_image1.png
Greyscale
where [v]s.neut indicates that the vertex v is a vertex on the source neutral mesh, L is a list of facial landmark indices on the source mesh (as described below), corr(i) is index of the ith landmark of the avatar mesh corresponding to the ith landmark of the source mesh”) representing a first face (In figure 9, in the top row the neutral shape represents a 3D head mesh representing a first face (a person labelled “Source”) and the second three-dimensional landmark of the target three-dimensional head mesh representing the second face (In figure 9, in the bottom row the neutral shape represents a 3D head mesh representing a second face (a video game character).
Also, please see section 4.2, 1st paragraph where they refer to: “In this section we describe our proposed method to robustly and precisely identify the face correspondences between the source and target neutral mesh (as illustrated in fig. 2).”
In this passage, the target mesh corresponds to the claimed “a target three-dimensional head mesh”.
Also, please see section 4.2.2 in the text below where figure 4 appears. This portion recites: “the vertex v is a vertex on the source neutral mesh, L is a list of facial landmark indices on the source mesh (as described below), corr(i) is index of the ith landmark of the avatar mesh corresponding to the ith landmark of the source mesh”).
As per claim 6, Onizuka teaches the claimed:
The method of claim 1, wherein generating the second three-dimensional head mesh comprises applying an affine transformation a real-world face based on a video game character face (Please see the middle right column where Onizuka discloses the calculation of the affine transformation:
PNG
media_image2.png
147
510
media_image2.png
Greyscale
Onizuka further teaches that “here, S and T are the linear parts of the affine transformations from the neutral shape to a given key shape for all the faces in the meshes of the source and avatar models, respectively. All linear transformations S are known from the source model and we aim at estimating the unknown linear transformations T and translations d of the avatar model” ).
As per claim 14, this claim is similar in scope to limitations recited in claim 6, and thus is rejected under the same rationale.
As per claims 9 and 17, these claims are similar in scope to limitations recited in claim 1, respectively, and thus are rejected under the same rationale. The system of Onizuka would have to have some type of processing circuit and a computer-readable media present in order to function and run as described by the reference.
As per claims 10 and 18, these claims are similar in scope to limitations recited in claim 2, and thus is rejected under the same rationale.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 5, 7-8, 13, and 15-16 are rejected under 35 U.S.C. 103 as being unpatentable Onizuka in view of Savchenkov et al. (Pub No. US 20200234034 A1) in further view of Comploi et al. (Pub No. US 20200312002 A1) .
As per claims 5 and 13, Onizuka alone does not explicitly teach the claimed limitations.
However, Onizuka in combination with Stemple and Comploi teaches the claimed:
The method of claim 1, wherein the method further comprises:
accessing a first real-world face image (Savchenkov [0015] “The system may include a database configured to store one or more advertisement videos. The advertisement videos may include at least a target face. The target face can be associated with a first individual. The system may further include a user information collection module configured to receive a user data associated with a user.”);
generating the source three-dimensional head mesh based on the first real-world face image (Savchenkov [0049] “The source parameters 330 can be extracted by fitting a parametric face model to the source face 315. The parametric face model may include a template mesh. Coordinates of vertices in the template mesh may depend on two parameters: a facial identity and a facial expression. Thus, the source parameters 330 may include a facial identity and facial expression corresponding to the source face 315”);
accessing a first video game character face image (Comploi Claim 1 “selecting by the processor a first avatar feature from a plurality of avatar features in the avatar feature category based on the classification of the first user feature, wherein the first avatar feature corresponds to the first user feature, and wherein the first avatar feature corresponds to the character design framework; combining by the processor the first avatar feature with an avatar representation to generate an individualized user avatar”); and
generating the target three-dimensional head mesh based on the first video game character face image (Savchenkov [0049] “The source parameters 330 can be extracted by fitting a parametric face model to the source face 315. The parametric face model may include a template mesh. Coordinates of vertices in the template mesh may depend on two parameters: a facial identity and a facial expression. Thus, the source parameters 330 may include a facial identity and facial expression corresponding to the source face 315”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use a first real-world image and a first avatar image as taught by Savchenkov and Comploi with the system of Onizuka in order to establish a baseline for the upcoming retargeting process. Doing so allows the system to have a reference point when applying changes to a specific landmark of the face.
As per claim 13, this claim is similar in scope to limitations recited in claim 5, and thus is rejected under the same rationale.
As per claim 7, Onizuka alone does not explicitly teach the claimed limitations.
However, Onizuka in combination with Savchenkov teaches the claimed:
The method of claim 1, wherein the method further comprises:
accessing a first real-world face image (Savchenkov [0015] “The system may include a database configured to store one or more advertisement videos. The advertisement videos may include at least a target face. The target face can be associated with a first individual. The system may further include a user information collection module configured to receive a user data associated with a user.”);
generating the source three-dimensional head mesh based on the first real-world face image (Savchenkov [0049] “The source parameters 330 can be extracted by fitting a parametric face model to the source face 315. The parametric face model may include a template mesh. Coordinates of vertices in the template mesh may depend on two parameters: a facial identity and a facial expression. Thus, the source parameters 330 may include a facial identity and facial expression corresponding to the source face 315”);
accessing a second real-world face image (Savchenkov [0015] “The advertisement videos may include a target face with a first individual. The method may include receiving, by a user information collection module, user data associated with a user. The user data may include an image of at least a source face. The source face can be associated with a second individual different from the first individual.”); and
generating the target three-dimensional head mesh based on the second real-world face image (Savchenkov [0049] “The source parameters 330 can be extracted by fitting a parametric face model to the source face 315. The parametric face model may include a template mesh. Coordinates of vertices in the template mesh may depend on two parameters: a facial identity and a facial expression. Thus, the source parameters 330 may include a facial identity and facial expression corresponding to the source face 315”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the first and second individuals as taught by Savchenkov with the system of Onizuka in order to have an interactive experience with multiple people.
As per claim 15, this claim is similar in scope to limitations recited in claim 7, and thus is rejected under the same rationale.
As per claim 8, Onizuka alone does not explicitly teach the claimed limitations.
However, Onizuka in combination with Comploi teaches the claimed:
The method of claim 1, wherein the method further comprises:
accessing a first video game character face image (Comploi Claim 1 “selecting by the processor a first avatar feature from a plurality of avatar features in the avatar feature category based on the classification of the first user feature, wherein the first avatar feature corresponds to the first user feature, and wherein the first avatar feature corresponds to the character design framework; combining by the processor the first avatar feature with an avatar representation to generate an individualized user avatar”);
generating the source three-dimensional head mesh based on the first video game character face image (Comploi Claim 5 “The method of claim 3, further comprising generating an avatar mesh based on the wrapped user mesh and the plurality of avatar features”);
accessing a second video game character face image (Comploi Claim 2 “The method of claim 1, further comprising: receiving by the processor color information corresponding to the user features; utilizing by the processor, the color information to classify a second user feature relative to an avatar feature category; selecting by the processor a second avatar feature from the avatar feature category based on the classification of the second user feature; and combining by the processor the second avatar feature with the avatar representation to generate the individualized user avatar”); and
generating the source three-dimensional head mesh based on the second video game character face image (Comploi Claim 5 “The method of claim 3, further comprising generating an avatar mesh based on the wrapped user mesh and the plurality of avatar features”.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the first and second individuals as taught by Comploi with the system of Onizuka in order to have an interactive experience with multiple people together in a virtual world.
As per claim 16, this claim is similar in scope to limitations recited in claim 8, and thus is rejected under the same rationale.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SARAH Y. LEE whose telephone number is (571)272-8374. The examiner can normally be reached 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Daniel F. Hajnik can be reached at (571) 272-7642. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
SARAH Y. LEE
Examiner
Art Unit 2616
/DANIEL F HAJNIK/Supervisory Patent Examiner, Art Unit 2616
Prosecution Insights