Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 02/17/2025 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Objections
Claim 11 objected to because of the following informalities: Claim 11 depends from claim 1 and recites term "the detected rotation angle value" in lines 1 and 5 of the claim. Claims 1 and 11 provide no detection of a rotation angle value. There is insufficient antecedent basis for this limitation, and the term should be rewritten as "the rotation angle value", "the obtained rotation angle value", or otherwise rewritten to provide sufficient antecedent basis for the limitation. Appropriate correction is required.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-2, 4, 8-9, 11-13, 15, and 18-19 are rejected under 35 U.S.C. 103 as being unpatentable over Nakamura et al (US 20220108468 A1, hereinafter Nakamura) and Takemoto (US 20200133388 A1).
Regarding claim 1, Nakamura teaches a data processing method, comprising ([0082] “The storage unit stores acquired data and processed data … and any other data generated during processing.”):
obtaining a target video frame and a reference video frame from an inputted video ([0102] “joint positions and joint angles in the present frame (a frame at time t+1) are calculated by using joint position data of the previous frame (a frame at time t)”), and extracting a joint point of a detected object separately from the target video frame and the reference video frame ([0102] “iterating the steps of obtaining joint angels and joint positions of the frame at time t+1 from joint positions of the frame at time t until t=T”), the reference video frame being a previous video frame of the target video frame ([0102] “joint positions and joint angles in the present frame (a frame at time t+1) are calculated by using joint position data of the previous frame (a frame at time t)”);
determining, based on an angle value range to which the rotation angle value belongs ([0149] “the optimization calculation based on the inverse kinematics may be performed by using prior knowledge such as giving weights and restrictions to each degree of freedom of the wrists and ankles according to the range of motion”) and a confidence range to which target position confidence of the joint point in the target video frame belongs ([0088] “the term PCM may be used occasionally for a spatial distribution of a likelihood of a position of a feature point”), a virtual rotation magnitude for controlling the joint point change magnitude of the joint point within a preset magnitude range ([0109] “each joint position .sup.t+1J.sup.n of the frame at time t+1 is a function of the joint angle .sup.t+1Q, as expressed in the equation (3), the joint angle .sup.t+1Q is obtained by an optimization calculation based on the inverse kinematics, and the joint position .sup.t+1J.sup.n is obtained by a forward kinematics calculation”, [0110] “Here, the sum of the PCM values in the prediction position of each joint is used for the weight of each joint in the optimization calculation based on the inverse kinematics .sup.t+1W.sup.n”, [0149] “the optimization calculation based on the inverse kinematics may be performed by using prior knowledge such as giving weights and restrictions to each degree of freedom of the wrists and ankles according to the range of motion”, where the joint angle obtained by inverse kinematics is restricted by the range of motion, corresponding to preset magnitude range, and joint angle obtained by equation (3) corresponds to virtual rotation magnitude.);
PNG
media_image1.png
85
587
media_image1.png
Greyscale
PNG
media_image2.png
92
514
media_image2.png
Greyscale
and adjusting the rotation angle value based on the virtual rotation magnitude to obtain an adjusted rotation angle value corresponding to the joint point ([0077] “The obtained joint position data is sent to the smoothing processing unit to obtain smoothed joint positions and joint angles. A pose of the subject can be determined by the smoothed joint positions and joint angles.”).
Nakamura fails to explicitly teach obtaining a rotation angle value representing a joint point change magnitude of the joint point from the reference video frame to the target video frame, but in related field of endeavor, Takemoto teaches obtaining a rotation angle value representing a joint point change magnitude of the joint point from the reference video frame to the target video frame ([0083] “based on the difference between the joint angle obtained from the bone estimation result and the predicted joint angle in the previous frame. For example, a difference angle between the joint angle in the current frame and the joint angle in the previous frame may be calculated to obtain the predicted joint angle”);
It would have been obvious to one of ordinary skill in the art prior to the time of filing to have modified Nakamura to include obtaining a rotation angle value representing a joint point change magnitude of the joint point from the reference video frame to the target video frame as taught by Takemoto. Doing so would provide a highly accurate prediction of positions and angles of a moving object ([0088] “a more highly accurate prediction can be applied to a variable portion of a moving object not only by predicting the position, but also by predicting the joint angle.”) Regarding claim 2, Nakamura as modified by Takemoto teaches the method according to claim 1, and Nakamura further teaches wherein the obtaining a target video frame and a reference video frame from an inputted video, and extracting a joint point of a detected object separately from the target video frame and the reference video frame comprises:
obtaining the target video frame from the inputted video, obtaining the previous video frame of the target video frame from the inputted video if the target video frame is not the first video frame of the inputted video ([0075] “comprises a video obtaining unit for obtaining a video of a subject”, [0102] “joint positions and joint angles in the present frame (a frame at time t+1) are calculated by using joint position data of the previous frame (a frame at time t)”), and determining the previous video frame of the target video frame as the reference video frame ([0102] “joint positions and joint angles in the present frame (a frame at time t+1) are calculated by using joint position data of the previous frame (a frame at time t)”);
inputting the target video frame into a posture estimation model, performing posture estimation on the target video frame by using the posture estimation model, and outputting the joint point of the detected object in the target video frame (Fig. 1, [0101] “The unit for obtaining joint positions comprises a unit for obtaining candidates for joint positions estimating the candidates for the joint position based on the heatmap data”, [0102] “A video motion capture for all T frames can be conducted by iterating the steps of obtaining joint angels and joint positions of the frame at time t+1 from joint positions of the frame at time t until t=T.”);
and inputting the reference video frame into the posture estimation model, performing posture estimation on the reference video frame by using the posture estimation model, and outputting the joint point of the detected object in the reference video frame. (Fig. 1, [0101] “The unit for obtaining joint positions comprises a unit for obtaining candidates for joint positions estimating the candidates for the joint position based on the heatmap data”, [0102] “A video motion capture for all T frames can be conducted by iterating the steps of obtaining joint angels and joint positions of the frame at time t+1 from joint positions of the frame at time t until t=T.”)
PNG
media_image3.png
705
469
media_image3.png
Greyscale
Regarding claim 4, Nakamura as modified by Takemoto teaches the method according to claim 2, and Nakamura further teaches wherein the method further comprises:
outputting target coordinate information of the joint point in the target video frame by using the posture estimation model ([0077] “The obtained joint position data is sent to the smoothing processing unit to obtain smoothed joint positions and joint angles. A pose of the subject can be determined by the smoothed joint positions and joint angles. The skeletal structure of the body of the subject and the motion of the subject comprised of time series data of the pose may be displayed on the display.”); and outputting reference coordinate information of the joint point in the reference video frame by using the posture estimation model ([0102] “ joint positions and joint angles in the present frame (a frame at time t+1) are calculated by using joint position data of the previous frame (a frame at time t). A video motion capture for all T frames can be conducted by iterating the steps of obtaining joint angels and joint positions of the frame at time t+1 from joint positions of the frame at time t until t=T.”);
Nakamura fails to explicitly teach the obtaining a detected rotation angle value representing a joint point change magnitude of the joint point from the reference video frame to the target video frame comprises: generating, based on the target coordinate information of the joint point in the target video frame and the reference coordinate information of the joint point in the reference video frame, the detected rotation angle value representing the joint point change magnitude of the joint point but in related field of endeavor Takemoto further teaches obtaining a detected rotation angle value representing a joint point change magnitude of the joint point from the reference video frame to the target video frame comprises: generating, based on the target coordinate information of the joint point in the target video frame and the reference coordinate information of the joint point in the reference video frame, the detected rotation angle value representing the joint point change magnitude of the joint point ([0083] “a difference angle between the joint angle in the current frame and the joint angle in the previous frame may be calculated to obtain the predicted joint angle ”).
It would have been obvious to one of ordinary skill in the art prior to the time of filing to have further modified Nakamura and Takemoto to include the obtaining a detected rotation angle value representing a joint point change magnitude of the joint point from the reference video frame to the target video frame comprises: generating, based on the target coordinate information of the joint point in the target video frame and the reference coordinate information of the joint point in the reference video frame, the detected rotation angle value representing the joint point change magnitude of the joint point as taught by Takemoto. Doing so would provide a highly accurate prediction of positions and angles of a moving object ([0088] “a more highly accurate prediction can be applied to a variable portion of a moving object not only by predicting the position, but also by predicting the joint angle.”)
Regarding claim 8, Nakamura as modified by Takemoto teaches the method according to claim 2, and Nakamura further teaches wherein the target position confidence is confidence of target coordinate information of the joint point ([0088] “OpenPose uses a trained convolutional neural network (CNN) to generate Part Confidence Maps (PCM) … the term PCM may be used occasionally for a spatial distribution of a likelihood of a position of a feature point”) in the target video frame outputted by the posture estimation model ([0089] “OpenPose is applied to each RGB image obtained from a plurality of synchronized cameras to generates Part Confidence Maps (PCM) for 18 feature points”); and
the determining, based on an angle value range to which the detected rotation angle value belongs and a confidence range to which target position confidence of the joint point in the target video frame belongs, a virtual rotation magnitude configured for controlling the joint point change magnitude of the joint point within a preset magnitude range comprises:
determining an angle value rotation magnitude corresponding to the joint point based on the angle value range to which the detected rotation angle value belongs ([0149] “the optimization calculation based on the inverse kinematics may be performed by using prior knowledge such as giving weights and restrictions to each degree of freedom of the wrists and ankles according to the range of motion” where range of motion corresponds to angle value range);
determining a confidence rotation magnitude corresponding to the joint point based on the confidence range to which the target position confidence of the joint point in the target video frame belongs ([0088] “the term PCM may be used occasionally for a spatial distribution of a likelihood of a position of a feature point”); and
generating, by performing multiplication operation on the angle value rotation magnitude and the confidence rotation magnitude, the virtual rotation magnitude configured for controlling the joint point change magnitude of the joint point within the preset magnitude range (Eq. 3, where t+1Wn is a PCM value, corresponding to confidence, and t+1Q corresponds to the joint angle, [0109] “each joint position .sup.t+1J.sup.n of the frame at time t+1 is a function of the joint angle .sup.t+1Q”, [0149] “the optimization calculation based on the inverse kinematics may be performed by using prior knowledge such as giving weights and restrictions to each degree of freedom of the wrists and ankles according to the range of motion of the elbows and knees”)
PNG
media_image1.png
85
587
media_image1.png
Greyscale
Regarding claim 9, Nakamura as modified by Takemoto teaches the method according to claim 8, and Nakamura further teaches wherein the determining an angle value rotation magnitude corresponding to the joint point based on the angle value range to which the detected rotation angle value belongs comprises: obtaining an angle value range parameter corresponding to the angle value range to which the detected rotation angle value belongs ([0134] “a set of a plurality of rotation angles (for example, 0 degree, 30 degrees, 60 degrees, 90 degrees, 330 degrees in 30-degree increments) and an angle range corresponding to each rotation angle (for example, an angle between 15 degrees and 45 degrees corresponds to 30 degrees) are set and stored in the storage unit as a table for determining the rotation of the input image, With reference to this table, it is determined which angle range the inclination of the subject body (angle of the orthogonal projection vector) in the previous frame corresponds to”), and obtaining a part area corresponding to the joint point ([0088] “18 keypoints in the body are set (see FIG. 10B), Specifically, the 18 feature points consist of 13 joints, a nose, left and right eyes, and left and right ears.”, [0094] “Each joint of the skeletal model (FIG. 10A) corresponds to each feature point in the body adopted by the heatmap generating unit”); determining the angle value range parameter as the angle value rotation magnitude corresponding to the joint point if the part area belongs to a trunk area ([0133] “the inclination of the body of the subject (in one aspect, the inclination of the trunk) is detected. For example, a vector connecting the waist and neck of the subject is calculated”); and performing, if the part area belongs to a non-trunk area, expansion processing on the angle value range parameter to obtain an angle value expansion parameter, and determining a larger value of the angle value expansion parameter and a preset parameter as the angle value rotation magnitude corresponding to the joint point ([0118] “A threshold value may be set for this likelihood, and if the likelihood is lower than the threshold value, it may be considered that the tracking of the subject pose has failed, and the search range for the joint position candidate may be expanded to perform the search. This may be applied for some parts of the whole body pose, or for the whole body”).
Regarding claim 11, Nakamura as modified by Takemoto teaches the method according to claim 1, and Nakamura further teaches wherein the adjusting the detected rotation angle value based on the virtual rotation magnitude to obtain an adjusted rotation angle value corresponding to the joint point comprises: generating the adjusted rotation angle value corresponding to the joint point by multiplying the virtual rotation magnitude and the detected rotation angle value ([0077] “The obtained joint position data is stored in the storage unit as time series data of the joint positions. The obtained joint position data is sent to the smoothing processing unit to obtain smoothed joint positions and joint angles. A pose of the subject can be determined by the smoothed joint positions and joint angles.”, [0124] “Then the smoothing processing while maintaining the link length is performed by the optimization calculation based on the inverse kinematics using the skeletal structure of the subject and smoothed joint positions to obtain each joint angles of the subject.”).
Regarding claim 12, the computer device claim 12 (Nakamura [0061] “the invention is provided as a computer program that causes a computer to function as a memory and processor of the above system”) is similar in scope to the method claim 1, and is rejected under similar rationale.
Regarding claim 13, the computer device claim 13 is similar in scope to the method claim 2, and is rejected under similar rationale.
Regarding claim 15, the computer device claim 15 is similar in scope to the method claim 4, and is rejected under similar rationale.
Regarding claim 18, the non-transitory computer-readable storage medium (Nakamura [0061] “the invention is provided as a computer program that causes a computer to function as a memory and processor of the above system, or a computer readable medium in which the computer program is stored.”) claim 18 is similar in scope to the method claim 1, and is rejected under similar rationale.
Regarding claim 19, the computer-readable storage medium claim 19 is similar in scope to the method claim 2, and is rejected under similar rationale.
Claims 3, 14, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Nakamura and Takemoto as applied to claim 2 above, and further in view of Ren et al (US 20220237829 A1, hereinafter Ren).
Regarding claim 3, Nakamura as modified by Takemoto teaches the method according to claim 2, and Nakamura further teaches further comprising: obtaining the joint point of the detected object in the target video frame if the target video frame is the first video frame of the inputted video ([0096] “An initial pose is obtained as a starting point for motion measurement of a subject. According to the embodiment, estimations of initial pose of skeletal model and distance between adjacent joints are obtained from the pixel positions of feature points that are calculated by applying OpenPose to a distortion-corrected image. First, an initial heatmap is obtained based on an initial image”);
obtaining an updated detected rotation angle value, the updated detected rotation angle value representing the joint point change magnitude of the joint point, and the updated detected rotation angle value being determined based on target coordinate information of the joint point in the target video frame ([0127] “ the low pass filter cannot be applied for 3 frames from the start of joint angle updating. In the present embodiment, by assigning the joint positions of the first frame to a −2.sup.nd frame, a −1.sup.st frame and a 0.sup.th frame prior to the application of the filter, the smoothing process for all joint positions with little spatial errors can be performed though the calculation time is delayed by two frames.”);
determining, based on the confidence range to which the target position confidence of the joint point in the target video frame belongs ([0088] “the term PCM may be used occasionally for a spatial distribution of a likelihood of a position of a feature point”), an updated virtual rotation magnitude for controlling the joint point change magnitude of the joint point within the preset magnitude range ([0109] “each joint position .sup.t+1J.sup.n of the frame at time t+1 is a function of the joint angle .sup.t+1Q, as expressed in the equation (3), the joint angle .sup.t+1Q is obtained by an optimization calculation based on the inverse kinematics, and the joint position .sup.t+1J.sup.n is obtained by a forward kinematics calculation.”, [0110] “Here, the sum of the PCM values in the prediction position of each joint is used for the weight of each joint in the optimization calculation based on the inverse kinematics .sup.t+1W.sup.n”, [0149] “the optimization calculation based on the inverse kinematics may be performed by using prior knowledge such as giving weights and restrictions to each degree of freedom of the wrists and ankles according to the range of motion”, where the joint angle obtained by inverse kinematics is limited by the range of motion, corresponding to magnitude range.);
and adjusting the updated detected rotation angle value based on the updated virtual rotation magnitude to obtain an updated adjusted rotation angle value corresponding to the joint point ([0077] “The obtained joint position data is sent to the smoothing processing unit to obtain smoothed joint positions and joint angles. A pose of the subject can be determined by the smoothed joint positions and joint angles.”).
Nakamura as modified by Takemoto fail to explicitly teach the updated detected rotation angle value being determined based on default coordinate information corresponding to the joint point of a default detected object, and the updated adjusted rotation angle value being used for performing object driving on a virtual object in a default virtual image to obtain the virtual object in a target virtual image associated with the target video frame, and the virtual object in the default virtual image being obtained by performing object simulation on the default detected object, but in related field of endeavor Ren teaches the updated detected rotation angle value being determined based on default coordinate information corresponding to the joint point of a default detected object ([0104] “obtain a 3D model corresponding to the sample target object in the sample source image … The predictive pose transition matrix characterizes the transfer relationship between corresponding positions on the sample target object and the sample reference object.”, where sample target object in the sample source image corresponds to default detected object), and the updated adjusted rotation angle value being used for performing object driving on a virtual object in a default virtual image to obtain the virtual object in a target virtual image associated with the target video frame ([0108] “The appearance feature of the sample target object is rearranged by using the predictive pose transition matrix, and the appearance feature of the sample target object in the sample source image is correspondingly transferred to a corresponding pose position in the sample target image, to obtain a predictive target appearance feature after transfer”), and the virtual object in the default virtual image being obtained by performing object simulation on the default detected object ([0108] “and a feature extracting module in the generator extracts a global feature of the sample source image and then further extracts an appearance feature of the sample target object from the global feature. The appearance feature of the sample target object is rearranged by using the predictive pose transition matrix, and the appearance feature of the sample target object in the sample source image is correspondingly transferred to a corresponding pose position in the sample target image, to obtain a predictive target appearance feature after transfer”)
It would have been obvious to one of ordinary skill in the art prior to the time of filing to have further modified Nakamura and Takemoto to include the updated detected rotation angle value being determined based on default coordinate information corresponding to the joint point of a default detected object, and the updated adjusted rotation angle value being used for performing object driving on a virtual object in a default virtual image to obtain the virtual object in a target virtual image associated with the target video frame, and the virtual object in the default virtual image being obtained by performing object simulation on the default detected object as taught by Ren. Doing so would improve the performance of human pose and provide the expected effect to the human pose after transfer ([0039] “ thus improving the performance of human pose transfer and ensuring that the human pose after transfer can achieve the expected effect.”)
Regarding claim 14, the computer device claim 14 is similar in scope to the method claim 3, and is rejected under similar rationale.
Regarding claim 20, the computer-readable storage medium claim 20 is similar in scope to the method claim 3, and is rejected under similar rationale.
Claims 5-7, 16 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Nakamura and Takemoto as applied to claim 4 above, and further in view of Nakamura et al 2 (US 20030034979 A1, hereinafter Nakamura 2).
Regarding claim 5, Nakamura as modified by Takemoto teach the method according to claim 4, but fail to explicitly teach wherein the generating, based on the target coordinate information of the joint point in the target video frame and the reference coordinate information of the joint point in the reference video frame, the detected rotation angle value representing the joint point change magnitude of the joint point comprises: determining a target rotation matrix corresponding to the joint point based on the target coordinate information of the joint point in the target video frame; determining a reference rotation matrix corresponding to the joint point based on the reference coordinate information of the joint point in the reference video frame; determining a joint point rotation matrix of the joint point from the reference video frame to the target video frame based on the target rotation matrix and the reference rotation matrix; and converting the joint point rotation matrix into a joint point rotation vector, and generating, based on the joint point rotation vector, the detected rotation angle value representing the joint point change magnitude of the joint point.
In related field of endeavor, Nakamura 2 teaches determining a target rotation matrix corresponding to the joint point based on the target coordinate information of the joint point in the target video frame ([0093] “the target joint value of the spherical joint is explained. The joint value R.sub.i and the joint velocity .omega..sub.i are defined by a 3.times.3 rotation matrix and its angular velocity respectively When the spherical joint is given the target joint value R.sub.Di.epsilon.R.sup.3.times.3, the target velocity is obtained as follows”); determining a reference rotation matrix corresponding to the joint point based on the reference coordinate information of the joint point in the reference video frame ([0093] “the target joint value of the spherical joint is explained. The joint value R.sub.i and the joint velocity .omega..sub.i are defined by a 3.times.3 rotation matrix and its angular velocity respectively When the spherical joint is given the target joint value R.sub.Di.epsilon.R.sup.3.times.3, the target velocity is obtained as follows”); determining a joint point rotation matrix of the joint point from the reference video frame to the target video frame based on the target rotation matrix and the reference rotation matrix (Eq. 22, [0095] “and each column corresponds to rotation around x, y and z respectively. Each column is calculable like the rotation joint centering on the corresponding direction, respectively.”);
PNG
media_image4.png
35
454
media_image4.png
Greyscale
and converting the joint point rotation matrix into a joint point rotation vector, and generating, based on the joint point rotation vector ([0096] “The direction of the link when R.sub.i is a unit matrix is expressed with a unit vector d.sub.i.sup.0”), the detected rotation angle value representing the joint point change magnitude of the joint point ([0096] “The direction of the link when R.sub.i is a unit matrix is expressed with a unit vector d.sub.i.sup.0. The actual direction d.sub.i of the link is obtained by rotating d.sub.i.sup.0 around a vector a.sub.i contained in a plane perpendicular to d.sub.i.sup.0 as shown in FIG. 5. The length of a.sub.i is sin (.gamma..sub.i/2) where the rotation angle is .gamma..sub.i. The twist angle .alpha. is defined as the rotation angle required in order to make to coincident R.sub.i with the frame which is obtained by rotating R.sub.i around a.sub.i from the state of a unit matrix.”).
It would have been obvious to one of ordinary skill in the art to have further modified Nakamura and Takemoto to include determining a target rotation matrix corresponding to the joint point based on the target coordinate information of the joint point in the target video frame; determining a reference rotation matrix corresponding to the joint point based on the reference coordinate information of the joint point in the reference video frame; determining a joint point rotation matrix of the joint point from the reference video frame to the target video frame based on the target rotation matrix and the reference rotation matrix; and converting the joint point rotation matrix into a joint point rotation vector, and generating, based on the joint point rotation vector, the detected rotation angle value representing the joint point change magnitude of the joint point as taught by Nakamura 2. Doing so would generate man-like motions of the whole body for a human figure ([0015] “It is an object of this invention to provide a method of generating poses and motions of a tree structure link system which overcomes the drawbacks of the above-mentioned conventional technologies, in order to generate man-like motions of the whole body for a human figure with multiple joints and multiple (for example, 20 or more) degrees of freedom”)
Regarding claim 6, Nakamura as modified by Takemoto and Nakamura 2 teach the method according to claim 5. Nakamura further teaches wherein a quantity of joint points is S, S is a positive integer greater than 1, and S joint points comprise a target joint point ([0088] “18 keypoints in the body are set (see FIG. 10B), Specifically, the 18 feature points consist of 13 joints, a nose, left and right eyes, and left and right ears.”); and obtaining a target joint point type corresponding to the target joint point ([0094] “Each joint of the skeletal model (FIG. 10A) corresponds to each feature point in the body adopted by the heatmap generating unit”); and obtaining a sub-joint point of the target joint point from the S joint points ([0098] “An optimization calculation based on inverse kinematics requires a constant of the distance between adjacent feature points (a distance between joints), or a link length.”, where adjacent feature point corresponds to sub-joint point) if the target joint point type is a joint point type corresponding to a root joint point ([0095] “there are no keypoints provided by OpenPose which correspond to pelvis (base body) … Joint angles which cannot be determined with 18 keypoints provided by OpenPose can be determined by optimization considering a constraint such as a movable range”), and ([0124] “Then the smoothing processing while maintaining the link length is performed by the optimization calculation based on the inverse kinematics using the skeletal structure of the subject and smoothed joint positions to obtain each joint angles of the subject”); or obtaining a parent joint point of the target joint point from the S joint points if the target joint point type is a joint point type corresponding to a non-root joint point ([0149] “Further, the optimization calculation based on the inverse kinematics may be performed by using prior knowledge such as giving weights and restrictions to each degree of freedom of the wrists and ankles according to the range of motion of the elbows and knees”, where wrists and ankles correspond to a non-root joint point and elbows and knees correspond to parent joints), and ([0124] “Then the smoothing processing while maintaining the link length is performed by the optimization calculation based on the inverse kinematics using the skeletal structure of the subject and smoothed joint positions to obtain each joint angles of the subject””).
Nakamura fails to explicitly teach determining a target rotation matrix corresponding to the target joint point, but in related field of endeavor Nakamura 2 teaches determining a target rotation matrix corresponding to the target joint point ([0093] “the target joint value of the spherical joint is explained. The joint value R.sub.i and the joint velocity .omega..sub.i are defined by a 3.times.3 rotation matrix and its angular velocity respectively”, [0096] “the movable range of the spherical joint is expressed using a total of three parameters containing two parameters representing the direction of the link and one parameter representing the twist angle of the link. The direction of the link when R.sub.i is a unit matrix is expressed with a unit vector d.sub.i.sup.0.”)
It would have been obvious to one of ordinary skill in the art prior to the time of filing to have further modified Nakamura as modified by Takemoto and Nakamura 2 to further include determining a target rotation matrix corresponding to the target joint point as taught by Nakamura 2. Doing so would generate man-like motions of the whole body for a human figure ([0015] “It is an object of this invention to provide a method of generating poses and motions of a tree structure link system which overcomes the drawbacks of the above-mentioned conventional technologies, in order to generate man-like motions of the whole body for a human figure with multiple joints and multiple (for example, 20 or more) degrees of freedom”)
Regarding claim 7, Nakamura as modified by Takemoto and Nakamura 2 teach the method according to claim 5, and Nakamura 2 further teaches wherein the determining a joint point rotation matrix of the joint point from the reference video frame to the target video frame based on the target rotation matrix and the reference rotation matrix comprises: performing inverse transformation on the reference rotation matrix to obtain an inverse reference rotation matrix of the reference rotation matrix ([0071] “since the tree structure link systems such as a man, animals, robot, etc. usually have 30 or more degrees of freedom, J.sub.i is not square and has redundancy. In this case, the general solution of the equation (2) is expressed by using a pseudoinverse matrix”); and performing matrix multiplication on the inverse reference rotation matrix and the target rotation matrix to obtain the joint point rotation matrix of the joint point from the reference video frame to the target video frame (Eq. 22, [0093] “The joint value R.sub.i and the joint velocity .omega..sub.i are defined by a 3.times.3 rotation matrix and its angular velocity respectively When the spherical joint is given the target joint value R.sub.Di.epsilon.R.sup.3.times.3”).
PNG
media_image4.png
35
454
media_image4.png
Greyscale
It would have been obvious to one of ordinary skill in the art prior to the time of filing to have further modified Nakamura as modified by Takemoto and Nakamura 2 to further include wherein the determining a joint point rotation matrix of the joint point from the reference video frame to the target video frame based on the target rotation matrix and the reference rotation matrix comprises: performing inverse transformation on the reference rotation matrix to obtain an inverse reference rotation matrix of the reference rotation matrix; and performing matrix multiplication on the inverse reference rotation matrix and the target rotation matrix to obtain the joint point rotation matrix of the joint point from the reference video frame to the target video frame as taught by Nakamura 2. Doing so would generate man-like motions of the whole body for a human figure ([0015] “It is an object of this invention to provide a method of generating poses and motions of a tree structure link system which overcomes the drawbacks of the above-mentioned conventional technologies, in order to generate man-like motions of the whole body for a human figure with multiple joints and multiple (for example, 20 or more) degrees of freedom”)
Regarding claim 16, the computer device claim 16 is similar in scope to the method claim 5, and is rejected under similar rationale.
Regarding claim 17, the computer device claim 17 is similar in scope to the method claim 6, and is rejected under similar rationale.
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Nakamura and Takemoto as applied to claim 8 above, and further in view of Kim et al (US 20240029307 A1, hereinafter Kim).
Regarding claim 10, Nakamura as modified by Takemoto teach the method according to claim 8, but fail to explicitly teach wherein the determining a confidence rotation magnitude corresponding to the joint point based on the confidence range to which the target position confidence of the joint point in the target video frame belongs comprises: determining a first confidence range parameter corresponding to a first confidence range as the confidence rotation magnitude corresponding to the joint point if the confidence range to which the target position confidence of the joint point in the target video frame belongs is the first confidence range; and performing, if the confidence range to which the target position confidence of the joint point in the target video frame belongs is a second confidence range, operation processing on the target position confidence to obtain a second confidence range parameter corresponding to the second confidence range, and determining the second confidence range parameter as the confidence rotation magnitude corresponding to the joint point. In related field of endeavor, Kim teaches determining a first confidence range parameter corresponding to a first confidence range as the confidence rotation magnitude corresponding to the joint point if the confidence range to which the target position confidence of the joint point in the target video frame belongs is the first confidence range; and performing, if the confidence range to which the target position confidence of the joint point in the target video frame belongs is a second confidence range ([0079] “the apparatus 300 for synthesizing joint data may assign a binary gate value of 1 to the joint data when the confidence level of the joint data is greater than the confidence threshold, and may assign a binary gate value of 0 to the joint data when the confidence level of the joint data is less than or equal to the confidence threshold”), operation processing on the target position confidence to obtain a second confidence range parameter corresponding to the second confidence range, and determining the second confidence range parameter as the confidence rotation magnitude corresponding to the joint point (Kim Claim 2 “further comprising calculating a binary gate value of the converted joint data for calibration based on the confidence level and a confidence threshold, wherein the calculating of the rotation matrix and the translation vector includes calculating a rotation matrix … based on the converted joint data for calibration and the binary gate value.”)
It would have been obvious to one of ordinary skill in the art prior to the time of filing to have further modified Nakamura as modified by Takemoto to include determining a first confidence range parameter corresponding to a first confidence range as the confidence rotation magnitude corresponding to the joint point if the confidence range to which the target position confidence of the joint point in the target video frame belongs is the first confidence range; and performing, if the confidence range to which the target position confidence of the joint point in the target video frame belongs is a second confidence range, operation processing on the target position confidence to obtain a second confidence range parameter corresponding to the second confidence range, and determining the second confidence range parameter as the confidence rotation magnitude corresponding to the joint point as taught by Kim. Doing so would improve the precision of motion and resolve joint motion occlusion of user’s 3D motions ([0115] “the recognition range of a user's 3D motion required for indoor virtual sports can be expanded, joint motion occlusion can be resolved, and the precision of motion can be improved”)
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Saragih et al (US 20220207831 A1, hereinafter Saragih) teaches a method of updating a dynamic model of a subject based on key points obtained from frames of a video stream (Saragih Claim 1 "retrieving a first frame that includes a body image of a subject; selecting, from the first frame, multiple key points within the body image of the subject that define a hull of a body part and multiple joint points that define a joint between two body parts").
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JOHN PATRICK GOCO whose telephone number is (571)272-5872. The examiner can normally be reached M-Th, 7:00 am - 5:00 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kee Tung can be reached at (571)272-7794. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/J.P.G./ Examiner, Art Unit 2611
/KEE M TUNG/ Supervisory Patent Examiner, Art Unit 2611
Prosecution Insights