Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The IDS dated 11/06/2024 has been considered and placed in the application file.
Claim Interpretation
3. The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-11 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 1 introduces an actor’s face in (“capturing, from each of a plurality of depth-sensing digital cameras, a 2D digital image of an actor’s face” Line 5), then introduces another actor’s face in (“a set of single-perspective 3-dimensional facial landmarks on an actor’s face” Line 7). This claim is ambiguous because it does not specify whether there are multiple actors or a single actor. There is insufficient antecedent basis for this limitation in the claim. Claim 2-11 are rejected as dependent claims of claim 1.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1,3-4,10,11 are rejected under 35 U.S.C. 103 as obvious over US Patent Publication 10089522 B2, (Yu et al.) in view of US Patent Publication 20200279279 A1 (Chaudhuri)
Regarding Claim 1, Yu et al. teach A computerized method for transferring a facial expression of an actor to an avatar-generating rendering engine, comprising the following steps:
capturing, from each of a plurality of depth-sensing digital cameras, a 2D digital image of an actor’s face and a corresponding depth map of the actor’s face (Par 25 “one or more depth cameras 105 ... On the other hand, the depth camera 105 generates depth images for the user’s lower facial features that including at least one of the nose, lip, cheek, and chin.”)
obtaining, from each 2D digital image and corresponding depth map, a set of single-perspective 3-dimensional facial landmarks on an actor’s face
(Par 76 “Landmark locations associated with the user’s lower facial features is generated 720 by processing the IR or RGB images and/or the depth image of the 3D camera”, Par 77 “By using the landmark locations (and optionally 3D depth map data), FE parameters 424 for the entire face of the user is generated”, Par 59 "FE parameter generator 422 also performs fitting of landmark locations 415, 419 and 3D depth map to the model of 3D facial expression model to extract facial expression (FE) parameters")
Here, a set of single-perspective 3-dimensional facial landmarks are mapped to landmarks captured by a depth camera, which has a single perspective. As explained, Yu teaches the use of two or more depth cameras. Therefore, there could be two or more sets of single-perspective 3-dimensional facial landmarks
merging the sets of single-perspective 3-dimensional facial landmarks into a set of integrated 3-dimensional facial landmarks on the actor’s face
Figs 5A-5C show facial landmarks after merging/converting two or more sets of single-perspective 3-dimensional facial landmarks when there are two or more depth cameras.
PNG
media_image1.png
517
379
media_image1.png
Greyscale
transforming the set of integrated 3-dimensional facial landmarks into (a) a vector of expression blendshape coefficients, where each of the expression blendshape coefficients corresponds to one of a plurality of expression blendshapes, and where each such expression blendshape is associated with a different facial expression (Par 24 “processes the first images and the second images to extract facial expression parameters representing a facial expression of the user” Par 59 “performs fitting of landmark locations 415, 419 and 3D depth map to the model of 3D facial expression model to extract facial expression (FE) parameters 424 that collectively indicate the facial expressions”),
and (b) a facial orientation matrix of affine transformation values describing a spatial position and orientation of the actor’s face (Par 82: “the FE parameter generator 422 utilizes the personalized linear expression model (i.e., blendshape model) B to represent the facial mesh W, as shown in equation (8) in the world coordinate for performing the fitting optimization:
W(e)=RB(e)+t (8)
where e is the linear weight vector for the blendshape model, R is the rotation matrix and t is the translation vector computed from the Rigid Stabilization step.”
The rotation angle and the distance between the face, or each feature, and the rotational axis defines the spatial position of the face, or each facial feature.)
determining a direction of eye gaze in the actor’s face;
(Par 57 “Eye and eyebrow tracking module 414 may employ a tracking algorithm”)
outputting to a rendering engine the vector of expression blendshape coefficients, the facial orientation matrix, and the direction of eye gaze
(Par 60 “Applying the extracted facial expression parameters to a digital representation of the user to generate a graphical representation of the user”)
But Yu et al. fails to explicitly disclose
determining a direction of eye gaze in the actor’s face.
Chaudhuri teaches a determining a direction of eye gaze in the actor’s face (Par 80: “software that performs the function of Gaze tracking, which is a technology that can locate the point of a person’s gaze through computer algorithms.” “… directly infer gaze directions from observed eye shapes, such as pupil centre or iris edges.”)
It would have been obvious to a person of ordinary skill in the art at the time before the effective filing date of the claimed invention to modify Yu et al to include determining a direction of eye gaze in the actor’s face as taught by Chaudhuri. One of ordinary skill in the art would have been motivated to determine where or what an actor is looking at (Par 107 “indicates where a subject is looking and can be used to determine what a subject is looking at”).
Regarding claim 3, Yu et al. as modified by Chaudhuri teach all the limitations of claim 1, and Yu et al. further teaches identifying, in the digital image of the actor’s face, a set of single-perspective 2D facial landmarks on the actors face (Fig 5C shows crosses marking facial landmarks on a facial image) and converting the set of single-perspective 2D facial landmarks into the set of single-perspective 3D facial landmarks (Par 59: “also performs fitting of landmark locations 415, 419 and 3D depth map to the model of 3D facial expression model to extract facial expression (FE) parameters”).
Regarding claim 4, Yu et al. as modified by Chaudhuri teach all the limitations of claim 1 and 3, and Yu et al further teaches the method of claim 3, wherein the single-perspective 2D facial landmarks are found by a ML model trained to detect faces (Par 58: “To track the landmarks of the user’s lower face, the lower tracking module 418 may use the tracking algorithm, for example, using one of … (iv) deep machine learning”).
Regarding claim 10, Yu et al. as modified by Chaudhuri teach all the limitations of claim 1, and Yu et al further teaches the method of claim 1, further comprising: aligning the set of integrated 3-dimensional facial landmarks with a generic facial mesh (Par 66 “Specifically, the FE parameter generator 422 utilizes the linear PCA morphable version of the personalized neutral model M to represent the facial mesh Win the world coordinate using the following equation:
W(w,R,t)=RM(w)+t (2)
where w represents the linear weight vector for the morphable model, R is the rotation matrix and t is the translation vector. The FE parameter generator 422 determines w and (R, t) iteratively by minimizing the following energy term:
(67)
PNG
media_image2.png
31
281
media_image2.png
Greyscale
where α, β, γ represents the weights for the fitting terms; C.sub.pl is a fitting term to minimize the point-to-plane error between the volumetric mesh V and the facial mesh W, as defined by equation (4); Cm is a fitting term to minimize the point-to-point error between the 2D facial feature landmarks of the mouth, nose and silhouette, and the corresponding vertices of the facial mesh W, as defined by equation (5); Cr is a fitting term to minimize the point-to-point error between the 2D facial feature landmarks of the right eye region and the corresponding vertices of the facial mesh W, as defined by equation (6); and C.sub.l is a fitting term to minimize the point-to-point error between the 2D facial feature landmarks of the left eye region and the corresponding vertices of facial mesh W , as defined by equation (7).”);
finding a set of identity coefficients that minimizes a calculated difference between each of the plurality of identity blendshapes and its corresponding facial landmark in the set of integrated 3-dimensional facial landmarks (Cm, Cr, and Cl of Par 67);
applying a set of identity coefficients to the generic facial mesh to create a neutral mesh of the actor’s face (Par 65 “builds a personalized neutral face model by (i) estimating the rigid pose of a template neutral face model and then (ii) warping a linear principal component analysis (PCA) model of a neutral face to fit the volumetric mesh and 2D landmarks.”)
Regarding claim 11, Yu et al. as modified by Chaudhuri teach all the limitations of claim 10, and Yu et al further teaches wherein the transforming step further comprises:
aligning the set of integrated 3-dimensional facial landmarks with the neutral mesh of the actor’s face (Par 66 “where w represents the linear weight vector for the morphable model, R is the rotation matrix and t is the translation vector. The FE parameter generator 422 determines w and (R, t) iteratively by minimizing the following energy term:
PNG
media_image2.png
31
281
media_image2.png
Greyscale
where α, β, γ represents the weights for the fitting terms; C.sub.pl is a fitting term to minimize the point-to-plane error between the volumetric mesh V and the facial mesh W, as defined by equation (4); Cm is a fitting term to minimize the point-to-point error between the 2D facial feature landmarks of the mouth, nose and silhouette, and the corresponding vertices of the facial mesh W, as defined by equation (5); Cr is a fitting term to minimize the point-to-point error between the 2D facial feature landmarks of the right eye region and the corresponding vertices of the facial mesh W, as defined by equation (6); and C.sub.l is a fitting term to minimize the point-to-point error between the 2D facial feature landmarks of the left eye region and the corresponding vertices of facial mesh W, as defined by equation (7).”);
finding a set of expression coefficients that minimizes a calculated difference between each of the plurality of expression blendshapes and its corresponding facial landmark in the set of integrated 3-dimensional facial landmarks (Par 82: “A personalized linear model is a set of facial expression shapes (e.g., smile and jaw open) derived from a personalized neutral face model M. In one embodiment, the FE parameter generator 422 utilizes the personalized linear expression model (i.e., blendshape model) B to represent the facial mesh W”);
and converting the set of expression coefficients into the vector of expression blendshape coefficients (Par 82: “the FE parameter generator 422 utilizes the personalized linear expression model (i.e., blendshape model) B to represent the facial mesh W”)
Claim 2 is rejected under 35 U.S.C. 103 as obvious over US Patent Publication 10089522 B2, (Yu et al.) in view of US Patent Publication 20200279279 A1 (Chaudhuri) and in further view of Kuang et al.
Regarding claim 2, Yu et al. as modified by Chaudhuri teach all the limitations of claim 1, and Yu et al. further teaches identifying pupil position in at least one of the 2D images (Fig. 5A and 5B show landmarks identified on pupil positions). Yu et al. fails to teach Converting the pupil position to a 3D pupil position and calculating the eye gaze as a vector formed by a centroid of an eyeball mesh and the 3D pupil position.
In related endeavor, Kuang et al. teaches converting the pupil position to a 3D pupil position and calculating the eye gaze as a vector formed by a centroid of an eyeball mesh and the 3D pupil position (Fig 1 of Kuang shows a 3D eye mesh with a “visual axis” vector passing through the “eyeball center” and “pupil center”.)
It would have been obvious to a person of ordinary skill in the art at the time before the effective filing date of the claimed invention to modify the combination of Yu et al. and Chaudhuri to include converting the pupil position to a 3D pupil position and calculating the eye gaze as a vector formed by a centroid of an eyeball mesh and the 3D pupil position as taught by Kuang, since Yu et al cites deformable model fitting as a tracking algorithm for eye tracking (Par 56: "The tracking algorithm may use, for example… (ii) deformable model fitting"). Doing so results in more robust eye tracking measures to more accurately display the direction a user is looking in.
Claims 5-6 are rejected under 35 U.S.C. 103 as obvious over US Patent Publication 10089522 B2, (Yu et al.) in view of US Patent Publication 20200279279 A1 (Chaudhuri) as applied to claim 1 and in further view of US Patent Publication US 11003898 B2 (Fortune et al.)
Regarding claim 5, Yu et al. as modified by Chaudhuri teaches all the limitations of claim 1, and assembling the set of integrated 3-dimensional facial landmarks by calculating the integrated location of each visible facial landmark on the actor’s face based on the location of each corresponding facial landmark in each set of single-perspective 3-dimensional facial landmarks (P58: “By tracking the landmarks in the lower portion of the user's face, the lower face tracking module 418 generates landmark locations 419 of the lower facial features”) but Yu et al. as modified by Chaudhuri fails to explicitly disclose identifying non-visible facial landmarks in each of the sets of single-perspective 3-dimensional facial landmarks.
Fortune teaches identifying non-visible facial landmarks in each of the sets of single-perspective 3-dimensional facial landmarks (P99: “Based on the 3D model, processor 750 can calculate which control points are not visible”).
It would have been obvious to a person of ordinary skill in the art at the time before the effective filing date of the claimed invention to modify the combination of Yu et al and Chaudhuri to include identifying non-visible facial landmarks in each of the sets of single perspective 3-dimensional facial landmarks as taught by Fortune. One of ordinary skill in the art would have been motivated to hide non-visible parts of the face to increase the realistic impression (P99: “hide the parts of the images that are not visible according to the 3D model, increases the realistic impression”).
Regarding claim 6, Yu et al. as modified by Chaudhuri and Fortune teaches all the limitations of claim 5, but fails to explicitly disclose wherein each non-visible facial landmark is identified as not being visible by one of the depth-sensing digital cameras.
Fortune further teaches wherein each non-visible facial landmark is identified as not being visible by one of the depth-sensing digital cameras (P99: “Based on the 3D model, processor 750 can calculate which control points are not visible because they are occluded by other parts of the head.”)
Claim 7 is rejected under 35 U.S.C. 103 as obvious over US Patent Publication 10089522 B2, (Yu et al.) in view of US Patent Publication 20200279279 A1 (Chaudhuri) as applied to claim 1 and in further view of US Patent Publication US 11003898 B2 (Fortune et al.) as applied to claim 5 and further in view of US Patent Publication US 20140355843 A1 (Feipeng et al.)
Regarding claim 7, Yu et al. as modified by Chaudhuri and by Fortune teaches all the limitations of claim 5, but Yu et al. as modified by Chaudhuri and Fortune fails to explicitly disclose wherein each non-visible facial landmark is identified by its location being outside a bounding sphere that encompasses the actor’s face.
In related endeavor, Feipeng further teaches wherein each non-visible facial landmark is identified by its location being outside a bounding sphere that encompasses the actor’s face (P12: “discard outside points of the plurality of points that are outside of the sphere, wherein the facial point cloud comprising the points that are within the sphere”)
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to have modified the combination of Yu et al., Chaudhuri, and Fortune to include each non-visible facial landmark is identified by its location being outside a bounding sphere that encompasses the actor’s face as taught by Feipeng. One of ordinary skill in the art would have been motivated to identify facial landmarks with location outside of a bounding sphere encompassing the actor’s face in order to discard those landmarks to decrease computational complexity. (P6: “Therefore, how to decrease computational complexity and mitigate the impact of facial expression is a bottleneck in the three-dimensional face recognition technology as well as a key challenge in the research.”)
Claims 8-9 are rejected under 35 U.S.C. 103 as obvious over US Patent Publication 10089522 B2, (Yu et al.) in view of US Patent Publication 20200279279 A1 (Chaudhuri) and in further view of US Patent Publication US 11003898 B2 (Fortune et al.) and further in view of US Patent Publication US 20220309704 A1 (Shimizu).
Regarding claim 8, Yu et al. as modified by Chaudhuri and by Fortune et al. teaches all the limitations of claim 5, but Yu et al. as modified by Chaudhuri and Fortune et al. fails to explicitly disclose wherein the integrated location of each visible facial landmark is calculated by determining the centroid of corresponding facial landmarks in the sets of single-perspective 3-dimensional facial landmarks.
In related endeavor, Shimizu further teaches wherein the integrated location of each visible facial landmark is calculated by determining the centroid of corresponding facial landmarks in the sets of single-perspective 3-dimensional facial landmarks. (P144: “the position correction unit 123 may correct the plurality of landmark distances L based on the regression expression so that the regression expression representing the relationship between the landmark distance L′ and the face direction angle θ is closer to the line than the regression expression representing the relationship between the landmark distance L and the face direction angle θ is”).
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to have modified the combination of Yu et al., Chaudhuri, and Fortune to include the integrated location of each visible facial landmark is calculated by determining the centroid of corresponding facial landmarks in the sets of single-perspective 3-dimensional facial landmarks as taught by Shimizu. Doing so would correct errors in the position information of facial landmarks.
Regarding claim 9, Yu et al. as modified by Chaudhuri and by Fortune et al. teaches all the limitations of claim 5, but Yu et al. as modified by Chaudhuri and Fortune et al. fails to explicitly disclose wherein the integrated location of each visible facial landmark is calculated by selecting, from one of the sets of single-perspective 3-dimensional facial landmarks, the facial landmark that most directly faces its corresponding depth-sensing digital camera.
In related endeavor, Shimizu further teaches wherein the integrated location of each visible facial landmark is calculated by selecting, from one of the sets of single-perspective 3-dimensional facial landmarks, the facial landmark that most directly faces its corresponding depth-sensing digital camera. (P107 “Namely, the landmark selection unit 211 may select the landmark that is collected from the face image 301 including the face that faces the direction corresponding to the face direction angle θ set at the step S22.”)
It would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to have modified the combination of Yu et al. Chaudhuri, and Fortune to include wherein the integrated location of each visible facial landmark is calculated by selecting, from one of the sets of single-perspective 3-dimensional facial landmarks, the facial landmark that most directly faces its corresponding depth-sensing digital camera as taught by Shimizu. Doing so would cause the generated avatar to provide little or no feeling of strangeness as the face of the human. (P107 “Thus, the data generation apparatus 2 or the arithmetic apparatus 21 can generate the face data 221 by disposing the plurality of landmarks that correspond to the plurality of facial parts, respectively, at a position that provides little or no feeling of strangeness or in an arrangement manner that provides little or no feeling of strangeness. Namely, the data generation apparatus 2 or the arithmetic apparatus 21 can properly generate the face data 221 that indicates the landmark of the face of the virtual human 200 that provides little or no feeling of strangeness as the face of the human.”)
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JOHN PATRICK GOCO whose telephone number is (571)272-5872. The examiner can normally be reached M-Th, 7:00 am - 5:00 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jason Chan can be reached at (571) 272-3022. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JOHN P GOCO/Examiner, Art Unit 2619
/JASON CHAN/Supervisory Patent Examiner, Art Unit 2619