DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
Claims 3 and 13 are rejected under 35 U.S.C. 112(b) as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor regards as the invention.
Claim 3 recites “determining the initial 3D pose information … based on a determination that an ith video frame is the first frame in the video clips” in lines 5-6 and “obtaining corrected pose information … based on a determination that the ith video frame is not the first frame in the video clips” in lines 9-10. It appears that the ith video frame is determined as both a first frame and not a first frame. This apparent contradiction makes the scope of the claim unclear. Further, it is unclear how the last two paragraphs are connected to either or both of the ith video frame being a first frame or the ith video frame not being a first frame. Therefore, the overall scope of the claim is unclear. The prior-art rejection below is based on attempted interpretation of the unclear claim scope. Claim 13 is rejected for the same reasons as claim 3.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1-3, 8-13, and 18-20 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Rempe et al. (“HuMoR: 3D Human Motion Model for Robust Pose Estimation;” hereinafter “Rempe”).
Regarding claim 1, Rempe discloses A virtual object animation generation method, performed by a computer device (“single-character locomotion animation,” pg. 3, col. 2, para. 4), comprising: obtaining an initial video to be processed (“3D human perception from … video,” pg. 2, col. 1, para. 2), and determining initial three-dimensional (3D) pose information (“We represent the state of a moving person as a matrix x composed of … body pose joint angles,” pg. 3, sec. 3, para. 1; “our optimization is initialized by directly optimizing SMPL pose and shape parameters,” pg. 18, col. 1, para. 3), two-dimensional (2D) pose information (“We run off-the-shelf 2D pose estimation,” pg. 6, sec. 5.1, para. 2), and foot ground contact information of a target object in the initial video (“The person-ground contact ct is the probability that each of 8 body joints (left and right toes, heels, knees, and hands) is in contact with the ground at time t,” pg. 3, col. 2, para. 2); determining pose change information between two adjacent video frames in the initial video based on the initial 3D pose information of the target object (“conditioned on zt and xt-1, the decoder produces two outputs, Δθ and ct … We find the additive update Δθ improves predictive accuracy compared to direct next-step prediction,” pg. 3, col. 2, para. 2); determining to-be-corrected pose information of the target object based on the initial 3D pose information of the target object and the pose change information (Equation (4) generates to-be-corrected pose information xt based on the initial pose information xt-1 and the additive update Δθ); performing correction processing on the to-be-corrected pose information of the target object (“We next use the space of motion learned by HuMoR as a prior in TestOpt to recover pose and shape,” pg. 4, sec. 4, para. 1; “The optimization objective can be formulated as a maximum a-posteriori (MAP) estimate (see supplementary), which seeks a motion that is plausible under our generative model while closely matching observations,” pg. 5, sec. 4.2, para. 1) based on the 2D pose information (“For 2D joint positions, each with a detection confidence
σ
t
j
, we use a re-projection loss,” pg. 5, sec. 4.2, para. 3) and the foot ground contact information (“The second regularizer
ℇ
e
n
v
ensures consistency between predicted CVAE contacts, the motion, and the environment …
c
t
j
is the contact probability,” pg. 5, col. 2, para. 2) to obtain corrected pose information of the target object (Sec. 4 describes optimizing/correcting the pose information determined in Sec. 3); and retargeting the corrected pose information to a virtual object by means of motion retargeting, to generate a virtual object animation video corresponding to the initial video (“single-character locomotion animation,” pg. 3, col. 2, para. 4; Figs. 3 and 4 illustrate a virtual object animation video corresponding to the captured video).
Regarding claim 2, Rempe discloses wherein the initial 3D pose information comprises an initial pose parameter and an initial shape parameter (“Robust Estimation of temporal pose and shape,” abstract), and wherein the determining pose change information comprises: obtaining a trained motion prior model (“uses HuMoR as a strong motion prior,” pg. 2, col. 1, bullet 2); and performing prediction processing on the initial pose parameter of the target object by using the trained motion prior model to obtain the pose change information between the two adjacent video frames in the initial video (“conditioned on zt and xt-1, the decoder produces two outputs, Δθ and ct … We find the additive update Δθ improves predictive accuracy compared to direct next-step prediction,” pg. 3, col. 2, para. 2).
Regarding claim 3, Rempe discloses dividing the initial video into a plurality of video clips (“we perform optimization on batches of 3s sequences (90 frames).,” pg. 22, sec. F.4., para. 1); determining the initial 3D pose information of the target object as the to-be-corrected pose information of the target object based on a determination that an ith video frame is the first frame in the video clips, i=1, 2, . . . , or N, N being a total quantity of video frames in the initial video, and the ith video frame being a current video frame in which the target object is located (“an initial state x0 … Initial State GMM. We model pθ (x0) with a Gaussian mixture model (GMM),” pg. 3, col. 2, paras. 3-6; it is clear that the to-be-corrected pose information for an initial frame/state cannot be based on a previous frame; instead, Rempe uses a GMM to determine the initial 3D pose information which is directly used as the pose information to be optimized/corrected); obtaining corrected pose information of the target object in an (i−1)th video frame based on a determination that the ith video frame is not the first frame in the video clips, the corrected pose information comprising a corrected pose parameter and a corrected shape parameter; determining a to-be-corrected pose parameter of the target object based on pose change information of the target object from the (i−1)th video frame to the ith video frame and the corrected pose parameter of the target object in the (i−1)th video frame; and determining the initial shape parameter corresponding to the target object and the to-be-corrected pose parameter of the target object as the to-be-corrected pose information of the target object (“conditioned on zt and xt-1, the decoder produces two outputs, Δθ and ct … We find the additive update Δθ improves predictive accuracy compared to direct next-step prediction,” pg. 3, col. 2, para. 2; Equation (4) generates to-be-corrected pose information xt based on the previous corrected/optimized pose information xt-1 and the additive update Δθ; “We represent the state of a moving person as a matrix x composed … body pose joint angles … Part of the state … shape parameters,” pg. 3, sec. 3, para. 2; “optimization (TestOpt) of pose and shape,” pg. 3, sec. 3, para. 1).
Regarding claim 8, Rempe discloses obtaining a training video and a preset motion prior model, the training video comprising a plurality of training video frames (“training on the large AMASS motion capture dataset, we use HuMoR as a motion prior,” pg. 2, col. 1, para. 2); determining a kth training video frame and a (k+1)th training video frame as a kth training data pair, k=1, 2, . . . , or M−1, and M being a total quantity of video frames of the training video (“Our CVAE is trained using pairs of (xt-1, xt).,” pg. 3, sec. 3.1, para. 1); performing encoding processing on the two training video frames in the kth training data pair to obtain encoding results of the two training video frames in the kth training data pair in the latent space (“During training, given the previous state xt-1 and ground truth current state xt, the model reconstructs x̂t by sampling from the encoder distribution,” Fig. 2 caption).; determining predicted change information of the two training video frames in the kth training data pair based on the encoding results of the two training video frames in the kth training data pair in the latent space; and adjusting a parameter of the preset motion prior model based on pose change information of two video frames in each training data pair to obtain the trained motion prior model (“For a single training pair (xt-1, xt), the reconstruction loss is computed … from the decoder output x̂t = xt-1 + Δθ (zt, xt-1),” pg. 4, col. 1, para. 2).
Regarding claim 9, Rempe discloses obtaining a trained 3D pose estimation model, a trained 2D pose estimation model, and a trained foot ground contact detection model; performing pose estimation on the initial video by using the trained 3D pose estimation model to obtain the initial 3D pose information of the target object in the initial video (“We model pθ(x0) with a Gaussian mixture model (GMM),” pg. 3, col. 2, para. 6; “The initial state GMM is trained,” pg. 4, col. 2, para. 1); performing pose estimation on the initial video by using the trained 2D pose estimation model to obtain the 2D pose information of the target object (“We run off-the-shelf 2D pose estimation,” pg. 6, sec. 4.1, para. 2); and performing prediction processing on the 2D pose information of the target object by using the trained foot ground contact detection model to obtain the foot ground contact information of the target object (“learning a conditional prior, modeling the change in state and contacts,” pg. 3, col. 2, para. 4).
Regarding claim 10, Rempe discloses obtaining body mesh information of a to-be-driven virtual object; adapting the corrected pose information to the virtual object by means of motion retargeting based on the body mesh information of the virtual object, to drive the virtual object to perform an action corresponding to the corrected pose information, to generate the virtual object animation video corresponding to the initial video (“SMPL body model … that maps to body mesh,” pg. 3, sec. 3, para. 2; “single-character locomotion animation,” pg. 3, col. 2, para. 4; Figs. 3 and 4 illustrate a virtual object mesh being animated corresponding to the initial video).
Regarding claims 11-13, 18, and 19, they are rejected using the same citations and rationales described in the rejections of claims 1-3, 8, and 9, respectively.
Regarding claim 20, it is rejected using the same citations and rationales described in the rejection of claim 1.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 4-7 and 14-17 are rejected under 35 U.S.C. 103 as being unpatentable over Rempe.
Regarding claim 4, Rempe discloses determining a reprojection error of the target object based on the to-be-corrected pose information, the 2D pose information, and camera parameters of the target object (“2D joint positions … we use a re-projection loss,” pg. 5, sec. 4.2, para. 3; see Equation (13)); determining prior information of the target object based on the to-be-corrected pose information of the target object (“Motion Prior
ℇ
m
o
t
,” pg. 5, sec. 4.2, para. 2); determining a regular term error of the target object based on foot ground contact information and corrected pose information of the target object in an (i−1)th video frame, and foot ground contact information and to-be-corrected pose information of the target object in an ith video frame (“The second regularizer
ℇ
e
n
v
ensures consistency between predicted CVAE contacts, the motion, and the environment …
c
t
j
is the contact probability,” pg. 5, col. 2, para. 2); determining a total error of the target object based on the reprojection error, the prior information, and the regular term error of the target object (Equation (10) calculates a total error based on each individual error); and performing iterative optimization on the to-be-corrected pose information of the target object (“optimized for 30 iterations,” pg. 18, col. 1, para. 4) by using a gradient algorithm (“Gradients are backpropagated,” pg. 4, col. 1, para. 2) until the total error of the target object reaches a minimum value to obtain the corrected pose information of the target object (“we seek the parameters (θ, ϕ) that minimize the loss function,” pg. 4, col. 1, para. 1).
Rempe does not explicitly recite using a gradient descent algorithm.
The Examiner takes Official Notice that both the concepts and the advantages of using a gradient descent algorithm were well known and expected in the art before the effective filing date of the claimed invention, and it would have been obvious before the effective filing date of the claimed invention to apply a gradient descent algorithm in Rempe in order to increase accuracy and efficiency.
Regarding claim 5, Rempe renders obvious wherein the 2D pose information comprises reference coordinate information of each joint point of the target object and a confidence of the joint point (“2D joint positions, each with a detection confidence
σ
t
j
,” pg. 5, sec. 4.2, para. 3), and wherein the determining a reprojection error comprises (“we use a re-projection loss,” pg. 5, sec. 4.2, para. 3): determining camera coordinate information of the joint point of the target object based on the to-be-corrected pose information and the 2D pose information of the target object (“transformed from the canonical to observation (i.e. camera) frame,” pg. 5, sec. 4.2, para. 3); determining image coordinate information of the joint point based on the camera parameters and the camera coordinate information of the joint point of the target object (“pinhole projection,” pg. 5, sec. 4.2, para. 3); determining a coordinate error of the joint point based on the image coordinate information of the joint point and the reference coordinate information of the joint point (
∏
p
t
j
-
y
t
j
of Equation (13)); and determining the reprojection error of the target object based on the coordinate error of the joint point and the confidence of the joint point (see Equation (13)).
Regarding claim 6, Rempe renders obvious wherein the to-be-corrected pose information comprises a to-be-corrected pose parameter and a to-be-corrected shape parameter (“our approach optimizes for pose and shape,” pg. 2, col. 2, para. 3), the prior information comprises pose prior information and shape prior information (“a strong motion prior jointly solving for pose, body shape,” pg. 2, col. 1, bullet 2), and wherein the determining prior information comprises: determining the shape prior information of the target object based on the to-be-corrected shape parameter of the target object (“The final two regularizers are priors on the ground and shape,” pg. 5, col. 2, para. 3); performing encoding processing on the to-be-corrected pose parameter of the target object to obtain an encoding result of the to-be-corrected pose parameter in a latent space (“a conditional variational autoencoder (CVAE) which formulates the motion … as a latent variable model,” pg. 3, sec. 3, para. 3); determining the pose prior information of the target object based on the encoding result of the to-be-corrected pose parameter in the latent space (“generate the next state from xt-1 by sampling from the prior distribution and decoding,” Fig. 2 caption); and determining the prior information of the target object based on the shape prior information of the target object and the pose prior information of the target object (“we seek the shape and a sequence of SMPL pose parameters … We parameterize the optimized motion using our CVAE,” pg. 4, sec. 4.1, para. 1; “
ℇ
C
V
A
E
uses the learned conditional prior,” pg. 5, sec. 4.2, para. 2).
Regarding claim 7, Rempe renders obvious determining a position difference of each joint point of the target object and a length difference of each bone of the target object based on the corrected pose information of the target object in the (i−1)th video frame and the to-be-corrected pose information of the target object in the ith video frame (“The skeleton consistency term uses the joints Jt directly predicted by the decoder … The second summation uses bone lengths l computed from Jt at each step,” pg. 5, col. 2, para. 2); determining a foot velocity error of the target object based on the foot ground contact information of the target object in the (i−1)th video frame and the foot ground contact information of the target object in the ith video frame; determining a height difference between a foot joint point of the target object and a ground (“The contact height term,” pg. 5, col. 2, para. 2; “We automatically annotate ground contacts for 8 body joints (left and right toes, heels, knees, and hands) based on velocity and height,” pg. 16, col. 2, para. 3; “we analyze the effect of the energy terms and regularizers … the contact velocity and height terms,” pg. 21, sec. F.2., para. 2); and determining the regular term error of the target object based on the position difference of each joint point of the target object, the length difference of each bone of the target object, the foot velocity error of the target object, and the height difference between the foot joint point and the ground (the section “Regularizers” on pg. 5, col. 2 contains an equation that adds all of the individual regularizer terms).
Regarding claims 14-17, they are rejected using the same citations and rationales described in the rejections of claims 4-7, respectively.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Ryan McCulley whose telephone number is (571)270-3754. The examiner can normally be reached Monday through Friday, 8:00am - 4:30pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kee Tung can be reached at (571) 272-7794. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/RYAN MCCULLEY/Primary Examiner, Art Unit 2611
Prosecution Insights