DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1, 2, 3, 4, 6, 7, 9, 10, 16, 17, 19, and 20 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Wood (NPL: Rendering of Eyes for Eye-Shape Registration and Gaze Estimation; from applicant’s IDS).
Regarding claim 1:
Wood teaches:
An information processing method comprising, by an information processing apparatus (Wood: Each 120×80px rendering took 5.26s on average using a commodity GPU (Nvidia GTX660), Pg. 5, Section 4.4: Rendering images, par. 1):
controlling a state of a three-dimensional model of a person in a three-dimensional virtual space (Wood: Within E we randomly configure iris color and pose eyelids according to g. (Pg. 5, Section 4: Posing the Model, par. 1) and a rendering condition for rendering the three-dimensional model (Wood: The camera is positioned to simulate changes in head pose (a), Pg. 4, Fig. 6; Wood: The environment is then randomly rotated to simulate a continuous range of head-pose, and randomly scaled in intensity to simulate changes in ambient light, Pg. 5, col. 2, par. 1; see Note 1A); and
generating, on the basis of the state of the three-dimensional model and the rendering condition, input data including a three-dimensional model image that is an image obtained by rendering the three-dimensional model (Wood: The camera is positioned to simulate changes in head pose (a). At each position, we render many eye images for different gaze directions by posing the eyeball model (b), Pg. 4, Fig. 6), and learning data including correct data regarding the three-dimensional model (Wood: For eye shape registration, we needed additional ground truth annotations of eye-region landmarks in the training images. As shown in Figure 2d, each 3D eye-region was annotated once in 3D with 28 landmarks, corresponding to the eyelids (12), iris boundary (8), and pupil boundary (8), Pg. 5, Section 4.3: Eye-Region Landmark Annotation).
Note 1A: Wood teaches repositioning the camera and environment and relighting of the environment to generate various renderings of the eye model. The Examiner interpreted the aforementioned repositioning and relighting as the rendering condition, because the specification of the present application teaches: “The state of the CG space includes, for example, a state regarding a rendering condition (virtual imaging condition) at the time of rendering. Specifically, for example, the state of the CG space includes a light beam or an illumination state of the CG space, a background of the CG space, and the like” [0036].
Regarding claim 2:
Wood teaches:
The information processing method according to claim 1 (as shown above),
wherein the state of the three-dimensional model includes at least one of a position or a posture of the three-dimensional model in the virtual space (Wood: we render many eye images for different gaze directions by posing the eyeball model (b), Pg. 4, par. 6; Wood: Within E we randomly configure iris color and pose eyelids according to g. (Pg. 5, Section 4: Posing the Model, par. 1).
Regarding claim 3:
The information processing method according to claim 2 (as shown above),
wherein the state of the three-dimensional model further includes a direction of an eyeball of the three-dimensional model in the virtual space (Wood: To avoid violating anatomical constraints, we only rendered images for valid eyeball rotations |α|≤25◦ and |β|≤35◦ [30], Pg. 5, Section 4.1: Posing the Model, par. 1).
Regarding claim 4:
Wood teaches:
The information processing method according to claim 3 (as shown above),
wherein the correct data includes line-of-sight information indicating a line-of-sight direction of the three-dimensional model (Wood: At each camera position c, we rendered multiple images with different 3D gaze vectors to simulate the eye looking in different directions, Pg. 5, Section 4.1: Posing the Model, par. 1).
Regarding claim 6:
Wood teaches:
The information processing method according to claim 3 (as shown above),
wherein the information processing apparatus further controls a position of a gaze point indicating a position at which the three-dimensional model gazes in the virtual space (Wood: Gaze vectors g were chosen by first pointing the eye directly at the camera (simulating eye-contact), Pg. 5, Section 4.1: Posing the Model, par. 1), and
controls a direction of an eyeball of the three-dimensional model on a basis of a relative position between the eyeball of the three-dimensional model and the gaze point (Wood: and then modifying the eyeball’s pitch (α) and yaw (β) angles over a chosen range, Pg. 5, Section 4.1: Posing the model, par. 1; see Note 6A).
Note 6A: The Examiner interpreted the “position” in the limitation “on a basis of a relative position between the eyeball of the three-dimensional model and the gaze point” to be a broader reading of position that generally describes how an object is situated, rather than translation along three-dimensional axes. This is because an example provided in the specification is substantially similar to a teaching in Wood:
“Fig. 16A illustrates an example in a case where the positions of the camera object 202 and the gaze point object 203 are substantially the same. In this case, the line of sight of the avatar 201 is directed toward the camera object 202.” (specification of the present application, [0089])
“Gaze vectors g were chosen by first pointing the eye directly at the camera (simulating eye-contact)” (Wood, Pg. 5, Section 4.1: Posing the Model, par. 1)
Regarding claim 7:
Wood teaches:
The information processing method according to claim 2 (as shown above),
wherein the information processing apparatus controls a relative position and posture between the three-dimensional model and a camera object used for controlling the rendering condition (Wood: The camera is positioned to simulate changes in head pose (a). At each position, we render many eye images for different gaze directions, Pg. 4, Fig. 6).
Regarding claim 9:
Wood teaches:
The information processing method according to claim 1 (as shown above),
wherein the information processing apparatus controls the state of the three-dimensional model (Wood: Within E we randomly configure iris color and pose eyelids according to g, Pg. 5, Section 4.1: Posing the Model, par. 1) and the rendering condition (Wood: The environment is then randomly rotated to simulate a continuous range of head-pose, and randomly scaled in intensity to simulate changes in ambient light, Pg. 5, col. 2, par. 1), and
widens a variation of at least one of the state of the three-dimensional model in the three-dimensional model image (Wood: Within E we randomly configure iris color, Pg. 5, Section 4.1: Posing the Model, par. 1; see Note 9A) or the correct data in a learning data set that is a set of the learning data.
Note 9A: The Examiner interpreted the modification of iris color taught by Wood as “widening a variation of a state of a three-dimensional model” because the specification of the present application teaches: “the state control unit 152 instructs the object generation unit 151 to change the characteristic of the avatar 201 so as to widen the variation of the characteristic of the avatar 201 in the learning data set. For example, the state control unit 152 instructs the object generation unit 151 to change at least one of the race, gender, age, facial makeup, facial size, or skin color of the avatar.” [0106]. Specifically, modifying the iris color could be considered modification of the facial makeup of the avatar.
Regarding claim 10:
Wood teaches:
The information processing method according to claim 9 (as shown above)
wherein the learning data set is used for learning of a learning model for estimating at least one of a state or a characteristic of a person (Wood: We then present and evaluate two separate systems trained on the resulting data (SynthesEyes): […] an appearance-based gaze estimator, Pg. 2, par. 1).
Regarding claim 16:
Wood teaches:
The information processing method according to claim 1 (as shown above),
wherein the information processing apparatus generates the three-dimensional model (Wood: We developed a realistic dynamic eye-region model which can be randomly posed to generate fully labeled training images, Pg. 3, Section 3: Dynamic Eye-Region Model) and changes a characteristic of the three-dimensional model (Wood: Eyes vary in both shape (pupillary dilation) and texture (iris color and scleral veins). To model shape variation we use blend shapes to interpolate between several different poses created for the same topological mesh [28], Pg. 3, col 2, par. 2).
Regarding claim 17:
Wood teaches:
The information processing method according to claim 1,
wherein the information processing apparatus further controls a state of the virtual space (Wood: The environment is then randomly rotated to simulate a continuous range of head-pose, and randomly scaled in intensity to simulate changes in ambient light, Pg. 5, col. 2, par. 1).
Regarding claim 19:
Wood teaches:
An information processing apparatus comprising an estimation unit that executes estimation processing regarding a person by using a learning model generated by learning using a learning data set (Wood: We then present and evaluate two separate systems trained on the resulting data (SynthesEyes): […] an appearance-based gaze estimator, Pg. 2, par. 1) that is a set of learning data generated on a basis of a state of a three-dimensional model of the person in a three-dimensional virtual space (Wood: we provide a comprehensive and detailed description of the model preparation process and rendering pipeline (see Figure 2 for an overview of the model preparation process and Figure 4 for the eye model used), Pg. 2, par. 1; see Note 19A) and a rendering condition for rendering the three-dimensional model while changing the state of the three- dimensional model (Wood: Within E we randomly configure iris color and pose eyelids according to g. (Pg. 5, Section 4: Posing the Model, par. 1) and the rendering condition (Wood: The camera is positioned to simulate changes in head pose (a), Pg. 4, Fig. 6; Wood: The environment is then randomly rotated to simulate a continuous range of head-pose, and randomly scaled in intensity to simulate changes in ambient light, Pg. 5, col. 2, par. 1; see Note 1A), the learning data including input data including a three-dimensional model image that is an image obtained by rendering the three-dimensional model (Wood: The camera is positioned to simulate changes in head pose (a). At each position, we render many eye images for different gaze directions by posing the eyeball model (b), Pg. 4, Fig. 6) and correct data regarding the three-dimensional model (Wood: For eye shape registration, we needed additional ground truth annotations of eye-region landmarks in the training images. As shown in Figure 2d, each 3D eye-region was annotated once in 3D with 28 landmarks, corresponding to the eyelids (12), iris boundary (8), and pupil boundary (8), Pg. 5, Section 4.3: Eye-Region Landmark Annotation).
Note 19A: Figure 2 showcases that the data generated by Wood is based on a three-dimensional model in a virtual space (e.g., Fig. 2b) and that the data is an eye of a person.
Regarding claim 20:
Claim 20 is substantially similar to claim 19, and is therefore rejected for similar reasons. Claim 20 contains the following notable differences:
Claim 20 claims an information processing method instead of an information processing apparatus. In the rejection of claim 19, it was shown that Wood teaches the claimed information processing apparatus. It follows that Wood teaches the corresponding information processing method.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 5 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Wood (NPL: Rendering of Eyes for Eye-Shape Registration and Gaze Estimation; from applicant’s IDS) in view of Stratou (NPL: Effect of Illumination on Automatic Expression Recognition: A Novel 3D Relightable Facial Database).
Regarding claim 5:
Wood teaches:
The information processing method according to claim 4 (as shown above),
Wood fails to teach:
wherein the three-dimensional model image includes both eyes of the three-dimensional model, and the line-of-sight information indicates one line-of-sight direction with respect to both the eyes of the three-dimensional model.
Stratou teaches:
wherein the three-dimensional model image includes both eyes of the three-dimensional model (Stratou: Fig. 1; see Note 5A), and the line-of-sight information indicates one line-of-sight direction with respect to both the eyes of the three-dimensional model (Stratou: Our new ICT-3DRFE database also includes 3D models, considers different gaze directions, Pg. 2, col. 2, par. 2).
Note 5A: Stratou showcases in Fig. 1 that the generated 3D model includes both eyes.
It would be obvious to combine the teachings of Stratou with Wood because Wood explicitly compares its teachings to Stratou: “Similarly to facial expression recognition [21], illumination effects are a critical factor.” (Wood, Pg. 2, col. 2, par. 2), where [21] refers to “G. Stratou, A. Ghosh, P. Debevec, and L.-P. Morency, “Effect of illumination on automatic expression recognition: a novel 3D relightable facial database,” in FG, 2011” as listed on Pg. 9 of Wood.
Regarding claim 18:
Wood teaches:
The information processing method according to claim 1 (as shown above),
Wood fails to teach:
wherein the state of the three-dimensional model includes at least one of an expression or a gesture of the three-dimensional model.
Stratou teaches:
wherein the state of the three-dimensional model includes at least one of an expression (Stratou: In this paper, we introduce a novel 3D Relightable Facial Expression (ICT-3DRFE) database that enables experimentation in the fields of both computer graphics and computer vision. The database contains 3D models for 23 subjects and 15 expressions, Abstract) or a gesture of the three-dimensional model.
It would be obvious to combine the teachings of Stratou with Wood because Wood explicitly compares itself to Stratou: “Similarly to facial expression recognition [21], illumination effects are a critical factor.” (Pg. 2, col. 2, par. 2), where [21] refers to “G. Stratou, A. Ghosh, P. Debevec, and L.-P. Morency, “Effect of illumination on automatic expression recognition: a novel 3D relightable facial database,” in FG, 2011” as listed on Pg. 9 of Wood.
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Wood (NPL: Rendering of Eyes for Eye-Shape Registration and Gaze Estimation; from applicant’s IDS) in view of Yang (US 20220058407 A1).
Regarding claim 8:
Wood teaches:
The information processing method according to claim 2 (as shown above), wherein a posture of the three-dimensional model includes a pose of a head of the three-dimensional model in the virtual space (Wood: The environment is then randomly rotated to simulate a continuous range of head-pose, (Pg. 5, col. 2, par. 1; see also Pg. 5, Fig. 7b).
Wood fails to explicitly teach:
wherein a posture of the three-dimensional model includes a direction of a face of the three-dimensional model in the virtual space
Yang teaches:
wherein a posture of the three-dimensional model includes a direction of a face of the three-dimensional model in the virtual space (Yang: in any of the preceding aspects, the computer-implemented method further includes warping the modified facial texture image of the 2D face onto a 3D face morphable model (3DMM) to reconstruct a 3D face model with the gaze direction from the modified facial texture image; applying a rotation matrix to the reconstructed 3D face model to change the head pose [0010], emphasis added; see also Note 8A).
Note 8A: Yang further teaches that: “Optionally, in any of the preceding aspects, wherein the estimating further includes collecting 2D face images of a driver of a vehicle with one or more head poses to generate a driver dataset” [0012]. That is, it would be obvious to one of ordinary skill in the art to include data relating to the direction of a face in the virtual space in the dataset generation algorithm taught by Wood.
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to combine the teachings of Yang with Wood. Having a posture of the three-dimensional model includes a direction of a face of the three-dimensional model in the virtual space, as in Yang, would benefit the Wood teachings by increasing the quality and performance of the output dataset: “as head and eye movement are typically independent from one another, accurate head pose and gaze estimation is a non-trivial challenge in computer vision technology” (Yang, [0003]). Furthermore, similar to Wood, Yang relates to “training neural networks, and in particular, to training a neural network with a dataset to estimate a head pose and gaze angle” (Yang, [0002]).
Claims 11, 12, and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Wood (NPL: Rendering of Eyes for Eye-Shape Registration and Gaze Estimation; from applicant’s IDS) in view of Nakanishi (NPL: Targeting and Removing Bad Training Data).
Regarding claim 11:
Wood teaches:
The information processing method according to claim 1 (as shown above),
wherein the information processing apparatus determines the state of the three-dimensional model (Wood: Within E we randomly […] pose eyelids according to g. For our generic dataset, we rendered images with up to 45◦ horizontal and vertical deviation from eye-contact, in increments of 10◦, Pg. 5, Section 4.1: Posing the Model, par. 1; ), and
determining a case where the information processing apparatus determines that the state of the three-dimensional model is abnormal (Wood: As we posed the model in this way, there was the possibility of rendering “unhelpful” images that either simulate impossible scenarios or are not useful for training. To avoid violating anatomical constraints, we only rendered images for valid eyeball rotations |α|≤25◦ and |β|≤35◦ [30]. Pg. 5, Section 4.1: Posing the model).
Wood fails to explicitly teach:
in a case where the information processing apparatus determines that the state of the three-dimensional model is abnormal, the information processing apparatus does not add the learning data based on the three-dimensional model determined to be abnormal to a learning data set that is a set of the learning data
Nakanishi teaches:
in a case where the information processing apparatus determines that the state of the three-dimensional model is abnormal (Nakanishi: Essentially, what we’re looking for are images that possess significant differences from the kinds of images we imagine a user would take and use, Pg. 6, par. 1), the information processing apparatus does not add the learning data based on the three-dimensional model determined to be abnormal to a learning data set that is a set of the learning data (Nakanishi: Artificial images are bad training examples and should be filtered out. These kinds of images are materially different from the kinds of images we imagine a user would take and use to identify a given plant, Pg. 6, par. 3).
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to combine the teachings of Nakanishi with Wood. Not adding the learning data based on the three-dimensional model determined to be abnormal to a learning data set that is a set of the learning data, as in Nakanishi, would benefit the Wood teachings by increasing the quality and performance of the output dataset: “As we found that many of our images have an adverse effect on the accuracy of our classifier, due to the diminishing relevancy of our Google Image search results turning up things like drawings, graphs and fact sheets. These put bad training examples into our pool, making our model worse at doing what we want it to do(classify a plant based on a natural photo.) We’d like to remove these, but manually examining each of the 150 files across 500+folders is a seriously daunting task. Automating this task with code or focusing a final manual clean is a much more sensible way to do things.” (Nakanishi, Pg. 2, par. 2)
Regarding claim 12:
Wood in view of Nakanishi teaches:
The information processing method according to claim 11 (as shown above),
wherein the state in which the three-dimensional model is abnormal is a state in which learning efficiency may decrease in a case where the learning data including the three-dimensional model image based on the three-dimensional model is used for learning (Nakanishi: we found that many of our images have an adverse effect on the accuracy of our classifier, due to the diminishing relevancy of our Google Image search results turning up things like drawings, graphs and fact sheets. These put bad training examples into our pool, Pg. 2, par. 2).
Regarding claim 13:
Wood teaches:
The information processing method according to claim 12,
wherein the state in which the three-dimensional model is abnormal (see Note 13B) is at least one of a state in which at least one eye of the three-dimensional model is not included in the three- dimensional model image or a state in which at least one eye of the three-dimensional model is similar to a white of eye (Wood: Before rendering, we also verified that the projected 2D pupil center in the image was within the 2D boundary of the eyelid landmarks– this prevented us from rendering images where too little of the iris was visible, Pg. 5, Section 4.1: Posing the Model; see Note 13A).
Note 13A: The Examiner interpreted the teaching where Wood avoids “rendering images where too little of the iris was visible” as determining a state where “at least one eye of the three-dimensional model is similar to a white of eye” because an image where the eye is visible, but iris is not visible would result in only the white of eye being visible.
Note 13B: It would be obvious to one of ordinary skill in the art that a rendering where eyes are not visible would be abnormal with respect to Wood because Wood is directed to “Rendering of Eyes for Eye-Shape Registration and Gaze Estimation”. When combined with the teachings of Nakanishi, it would also be obvious to consider images without visible eyes as abnormal and remove them from the training set.
Claims 14 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Wood (NPL: Rendering of Eyes for Eye-Shape Registration and Gaze Estimation; from applicant’s IDS) in view of Chatterjee (US 20220179840 A1).
Regarding claim 14:
Wood teaches:
The information processing method according to claim 1 (as shown above),
Wood fails to teach:
wherein the information processing apparatus analyzes an acquired learning data set, and replenishes a shortage of at least one of an amount or a quality of the learning data set on a basis of an analysis result of the learning data set
Chatterjee teaches:
wherein the information processing apparatus analyzes an acquired learning data set (Chatterjee: A probability engine 410 may then operate on the untrusted dataset 408 based on the trusted dataset 406 to generate bias predictions [0071]), and replenishes a shortage of at least one of an amount or a quality of the learning data set on a basis of an analysis result of the learning data set (Chatterjee: For each data sample with confidence 414 higher than a predetermined threshold, a trusted label in the trusted dataset 406 with the highest conditional probability in the conditional probability distribution for the corresponding data sample may be used to replace the observed label (e.g., untrusted label). [0072]).
Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to combine the teachings of Chatterjee with Wood. replenishes a shortage of at least one of an amount or a quality of the learning data set on a basis of an analysis result of the learning data set, as in Chatterjee, would benefit the Wood teachings by reducing bias in the training dataset: “Biases in training datasets are increasingly common. Noise can be introduced to the training dataset while labeling due to various factors, including but not limited to, automatic labeling, lack of domain expertise of manual labelers, and data poisoning attacks by adversaries. These biases often unfairly disadvantage certain groups. Machine learning models, e.g., classifiers, trained on such biased datasets can inherit the biases; thereby, reducing the accuracy of the outputs of the machine learning models during inference.” (Chatterjee, [0002])
Regarding claim 15:
Wood in view of Chatterjee teaches:
The information processing method according to claim 14 (as shown above),
wherein the information processing apparatus detects a bias of the learning data set (Chatterjee: A confidence 414 may be determined for each data sample in the untrusted dataset 408 that indicates a confidence of the bias prediction 412 [0072]),
generates the learning data for correcting the bias of the learning data set (Chatterjee: For each data sample with confidence 414 higher than a predetermined threshold, a trusted label in the trusted dataset 406 with the highest conditional probability in the conditional probability distribution for the corresponding data sample may be used to replace the observed label (e.g., untrusted label). [0072]), and
adds the learning data to the learning data set (Chatterjee: An updated training data set 418 may be generated using the recovered true labels of the untrusted dataset 408 and the trusted dataset. [0072]).
Conclusion
The Examiner identified one or more potential limitation(s) in the specification that would overcome the prior art rejections under 102(a)(1) and 103 if amended into the claims. Note that in such a situation, further search and consideration would be required:
“For example, the learning data generation unit 153 stores a script at the time of generating each piece of learning data. Each script includes, for example, the type of the avatar 201 at the time of generating the learning data, the state of the CG space, and the state of each object in the CG space. The state of each object in the CG space includes the position and posture of the avatar 201 in the CG space, the position and posture of the camera object 202 in the CG space, the position of the gaze point object 203 in the CG space, and the like” as in [0119] of the specification of the present application.
“The hidden state occurs, for example, in a case where the face of the avatar 201 exists on a line segment connecting the eyeball object of at least one eye of the avatar 201 and the camera object 202. Therefore, the hidden state can be detected on the basis of, for example, the relative position between the eyeball object of each eye of the avatar 201 and the camera object 202 and the relative position between the face of the avatar 201 and the camera object 202.” as in [0140] of the specification of the present application.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to VINCENT ALEXANDER PROVIDENCE whose telephone number is (571)270-5765. The examiner can normally be reached Monday-Thursday 8:30-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, King Poon can be reached at (571)270-0728. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/VINCENT ALEXANDER PROVIDENCE/Examiner, Art Unit 2617 /KING Y POON/Supervisory Patent Examiner, Art Unit 2617
Prosecution Insights