Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 1-2, 4-5, 8, and 20 are rejected under 35 U.S.C. 102(a)(1) as being unpatentable over Lu (Made of ref: Lu Y, Yu H, Ni W, Song L. 3D real-time human reconstruction with a single RGBD camera. Appl Intell (Dordr). 2023;53(8))
Rgearding claim 1, Lu teaches A 3D motion capture method, the 3D motion capture method being performed by at least one processor, the 3D motion capture method comprising: receiving an input image including a person object (Sections 3.1-3.2 and figure 2: teaches receiving a single RGBD image depicting human using an Azure Kinect camera); detecting a person region, hand regions, and a face region from at least some of the input image by using a first machine learning model ( Section 4.1, equations 3-5 and figure 2:OpenPose obtains the human joint locations; corresponding body, face, and hand boundary points/ bounding boxes are calculated and used to crop the global/body, face, and two hand regions); and recognizing 3D motion information from a person sub-image including the person region, hand sub-images including the hand regions, and a face sub-image including the face region ( Section 3.2, 4.1 and fig 2: the cropped global/body image is processed by a ResNet50 branch, while the face and hand images are processed by parallel ResNet18 branches. The Mlps predict SMPL-X parameters including body pose, hand pose, facial expression, jaw pose, eye pose, and global orientation.); wherein a character image, including a character that imitates a body motion, hand motion, and face motion of the person object included in the input image, is generated through rendering based on the 3D motion information( sections 3.2, 4.1, 5.1, and figures 2,4,5: the outputs of the parallel networks are weighted and fused into the SMPL-X model, which generated and renders a 3D human mesh reflecting the person’s body pose, hand pose, and facial expressions).
Regarding claim 2, Lu teaches The 3D motion capture method of claim 1, wherein: the receiving an input image comprises sequentially receiving a plurality of frame images including the person object (Section 3.1 and 4.2.1: Lu explains that raw RGBD outputs are sequentially entered into a buffer queue, and outputs are produced in turn. Its indoor Human dataset also contains 2,500 frames of human action data); in response to sequential reception of each of the plurality of frame images, the detecting and the recognizing are performed in real time for each of the plurality of frame images ( sections 3.2 and 5.1-5.3, figure 2 and table 1: figure 2 shows that RGBD image is processed through the human-pose estimation, global body face and hand branches. Lu reports that the complete process takes approximately 0.21 seconds per frame, including preprocessing, pose estimation, and rendering); and a plurality of character images, each including a character that imitates a body motion, hand motion, and face motion of the person object included in a corresponding one of the plurality of frame images, are generated through rendering based on the 3D motion information ( Section 3.2, 4.1 and figures 1-2: for each processed frame, the global body, face, and hand outputs are fused into SMPL-x parameters representing body pose, hand pose, facial expression, and the resulting 3D human model is rendered. Repeating this disclosed per frame process over the sequential RGBD frames produces the corresponding sequence of rendered character images).
Regarding claim 4, Lu teaches The 3D motion capture method of claim 1, wherein the recognizing comprises: recognizing body motion information from the person sub-image including the person region by using a second machine learning model ( The global/person RGBD sub-image is input into the global body subnetwork, which uses ResNet50 and an MLP to extract global body features and predict body pose parameters); recognizing hand motion information from the hand sub-images including the hand regions by using a third machine learning model (the cropped hand images are input into a separate hand subnetwork using ResNet18 and an MLP to extract hand features and contribute to predicting the SMPL-X hand pose parameters); and recognizing face motion information from the face sub-image including the face region by using a fourth machine learning model(the cropped face image is input a separate face subnetwork using ResNet18 and an MLP to extract facial features and predict the facial expression parameters).
Regarding claim 5, Lu teaches The 3D motion capture method of claim 4, wherein: the second machine learning model is a model configured to output body model parameters and camera parameters based on the person sub-image (Section 3.3, 4.1 and 1-2: Lu’s global-body FBN subnetwork processes the global/person RGBD sub image and uses an MLP to predict SMPL-X body model parameters. Lu further states that the 3D point cloud and camera parameters are predicted using FBN and SMPL-X); and the body motion information comprises body 3D joint angle information( Section 2.2 and 4.1: the predicted SMPL-X pose parameter includes a body-pose parameter. Lu explains that body posture is represented using rotation matrices. These three dimensional rotational values for the body joints correspond to the claimed 3D joint angle information), body 3D joint location information (section 5.1, fig 4 and table 1: the predicted body-pose parameters are used to position the joints of the reconstructed 3D human model. Lu evaluates these predicted joint positions using Mean Per Joint position Error in millimeters, confirming that the system produces 3D joint location information), and body 3D mesh information obtained based on the body model parameters (Lu, section 4.1, equation 1-2 and figures 1 and 4: expressly discloses that SMPL-X generates vertices and faces from the predicted shape, pose, and expression parameters and obtains the 3D human mesh from those vertices and faces).
Regarding claim 8, Lu teaches The 3D motion capture method of claim 4, wherein: the first machine learning model is a model configured to detect the person region, the hand regions, and the face region from at least some of the input image (Lu teaches an end-to-end deep-learning network called Fast Body Net (“FBN”). As part of the FBN architecture, OpenPose obtains the human joint points from the input image. Based on the OpenPose output O, FBN calculates the minimum and maximum joint coordinates and determines the center and size of a bounding box. FBN applies the same operation to the body, face, and hand nodes and performs affine transformations to obtain respective global/person, face, and hand sub-images. See section 4.1, equations (3)–(5), and Fig. 2.)and recognize the body motion information (Lu further teaches that the same overall FBN architecture includes a global body subnetwork employing ResNet-50. The global subnetwork receives the global/person RGB image, corresponding depth image, and OpenPose joint output and extracts a global feature vector . A three-layer MLP then predicts the SMPL-X parameters, including the body-pose parameters, thereby recognizing body-motion information from the person sub-image. See section 4.1 and Fig. 2.); and the second machine learning model is the same model as the first machine learning model (Lu teaches that region localization and body-motion recognition are performed within the same overall end-to-end FBN architecture: FBN uses its OpenPose component to locate and extract the body, face, and hand regions and uses its global subnetwork and MLP to predict the corresponding body-pose parameters. Therefore, the model performing the detection functions and the model recognizing the body-motion information may be regarded as the same FBN model. (Lu Abstract, section 4.1, and Fig. 2. Lu also states that the entire pipeline is implemented through PyTorch).
Rgearding claim 20, Lu teaches A motion capture system comprising: memory; and at least one processor connected to the memory and configured to execute at least one computer- readable program included in the memory; receiving an input image including a person object (Sections 3.1-3.2 and figure 2: teaches receiving a single RGBD image depicting human using an Azure Kinect camera); detecting a person region, hand regions, and a face region from at least some of the input image by using a first machine learning model ( Section 4.1, equations 3-5 and figure 2:OpenPose obtains the human joint locations; corresponding body, face, and hand boundary points/ bounding boxes are calculated and used to crop the global/body, face, and two hand regions); and recognizing 3D motion information from a person sub-image including the person region, hand sub-images including the hand regions, and a face sub-image including the face region ( Section 3.2, 4.1 and fig 2: the cropped global/body image is processed by a ResNet50 branch, while the face and hand images are processed by parallel ResNet18 branches. The Mlps predict SMPL-X parameters including body pose, hand pose, facial expression, jaw pose, eye pose, and global orientation.); wherein a character image, including a character that imitates a body motion, hand motion, and face motion of the person object included in the input image, is generated through rendering based on the 3D motion information( sections 3.2, 4.1, 5.1, and figures 2,4,5: the outputs of the parallel networks are weighted and fused into the SMPL-X model, which generated and renders a 3D human mesh reflecting the person’s body pose, hand pose, and facial expressions).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 3 and 10 are rejected under 35 U.S.C. 103 as being unpatentable over Lu (Lu Y, Yu H, Ni W, Song L. 3D real-time human reconstruction with a single RGBD camera. Appl Intell (Dordr). 2023;53(8)) in view of Gi (Made of ref: Gi, G. et al. (2020). Real Time 3D Pose Estimation of Both Human Hands via RGB-Depth Camera and Deep Convolutional Neural Networks. In: Van Toi , V., Le, T., Ngo, H., Nguyen, TH. (eds) 7th International Conference on the Development of Biomedical Engineering in Vietnam (BME7). BME 2018. IFMBE Proceedings, vol 69. Springer, Singapore. https://doi.org/10.1007/978-981-13-5859-3_81)
Regarding claim 3, Lu teaches The 3D motion capture method of claim 1, wherein: Lu fails to teach the first machine learning model comprises a hand classifier; and the detecting comprises classifying the detected hand regions as at least one of a left-hand region, a right-hand region, and a both-hand region by using the hand classifier.
Gi teaches machine learning model comprises a hand classifier( Pages 467-469, sections 2, 2.2 and figures 1-2: Gi uses a CNN hand classifier as part of its hand detection and pose estimation pipeline); and the detecting comprises classifying the detected hand regions as at least one of a left-hand region, a right-hand region, and a both-hand region by using the hand classifier ( pages 467-470, sections 2.1-2.2, 3.1 and figures 1-2: Gi detects and extracts both hands from RGB and depth images and applies the CNN classifier to each cropped depth hand image to determine whether it represents the left hand or the right hand. It would have been obvious to modify Lu’s OpenPose based hand region detection to include Gi’s CNN hand classifier so that the detected hand regions could be identified as left hand or right hand regions before estimating and rendering their respective poses, thereby correctly associating each detected hand with corresponding SMPL-X hand parameters and improving the accuracy of the reconstructed hand motions).
Regarding claim 10, Lu teaches The 3D motion capture method of claim 4, wherein: the third machine learning model comprises a hand classifier (Lu, section 4.1 and fig.2: teaches processing extracted hand sub images through its hand network to obtain hand features and predict hand pose parameters. Gi, section 2.1-2.2 and figs 1-2: teaches a CNN hand classifier operating on extracted depth hand images); and the recognizing hand motion information comprises classifying the detected hand regions as at least one of a left-hand region, a right-hand region, and a both-hand region by using the hand classifier (pages 467-470, sections 2.1-2.2, 3.1 and figures 1-2: Gi detects and extracts both hands from RGB and depth images and applies the CNN classifier to each cropped depth hand image to determine whether it represents the left hand or the right hand).
Claim(s) 6 is rejected under 35 U.S.C. 103 as being unpatentable over Lu (Lu Y, Yu H, Ni W, Song L. 3D real-time human reconstruction with a single RGBD camera. Appl Intell (Dordr). 2023;53(8)) in view of Lee (Made of ref: KR 20210135033).
Regarding claim 6, Lu teaches The 3D motion capture method of claim 5, wherein the second machine learning model is a model further configured to further output body 2D joint estimation information( Section 4.1 and fig 2: teaches processing the person/body image through its global body subnetwork and using OpenPose to obtain the person’s local body joint points. The OpenPose output represents the two dimensional joint coordinates and associated confidence information for 25 body joints) but fails to teach a body part segmentation map based on the person sub-image.
Lee Teaches a body part segmentation map based on the person sub-image ( para 34 and fig.2: teaches extracting feature information for the principal body parts from an input person image through a first neural network and predicting a body part segmentation map 300. Further teaches predicting joint coordinates 500 through a second network based on both the extracted feature information and the body part segmentation map. It would have been obvious to modify the system of Lu machine learning model to incorporate Lee body part segmentation and joint coordinate prediction network because Lee teaches that the segmentation map supplies body part location information for predicting join coordinates, thereby improving joint localization and reducing ambiguity or collisions between joints associated with different body regions).
Claim(s) 7 is rejected under 35 U.S.C. 103 as being unpatentable over Lu (Lu Y, Yu H, Ni W, Song L. 3D real-time human reconstruction with a single RGBD camera. Appl Intell (Dordr). 2023;53(8)) in view of Lee (Made of ref: KR 20210135033) in further view of Gi (Made of ref: Gi, G. et al. (2020). Real Time 3D Pose Estimation of Both Human Hands via RGB-Depth Camera and Deep Convolutional Neural Networks. In: Van Toi , V., Le, T., Ngo, H., Nguyen, TH. (eds) 7th International Conference on the Development of Biomedical Engineering in Vietnam (BME7). BME 2018. IFMBE Proceedings, vol 69. Springer, Singapore. https://doi.org/10.1007/978-981-13-5859-3_81).
Regarding claim 7, Lu in view of Lee teaches The 3D motion capture method of claim 6, further comprising classifying the detected hand regions as at least one of a left-hand region, a right-hand region, and a both-hand region by using the body 3D joint location information, the body 2D joint estimation information, and the body part segmentation map.
Lu teaches obtaining body 3D joint-location information through the predicted SMPL-X body-pose parameters and resulting articulated 3D human model. Lu also teaches using OpenPose to obtain body joint points O and using those joint points to identify and extract the local face and hand regions from the input image. (Lu, Section 4.1, Figure 2, Equations 1–5.)
Lee teaches extracting feature information for body parts contained in an image, predicting a body-part segmentation map through a first network, and predicting joint coordinates through a second network based on the extracted feature information and the body-part segmentation map. Lee further teaches that the segmentation map includes hand regions and that the predicted body-joint coordinates include separately identified left-wrist and right-wrist coordinates. (Lee, Abstract; paragraphs 76–88, 108–116, and 121–125; Figures 1, 5, and 8; claims 1 and 10–11.).
However, Lu, in view of Lee fails to expressly teach classifying each detected hand region as at least one of a left-hand region, a right-hand region, and a both-hand region using the body 3D joint-location information, body 2D joint-estimation information, and body-part segmentation map.
Gi teaches detecting and extracting both hands from RGB and depth images and applying a CNN classifier to distinguish a detected right hand from a detected left hand. (D2, pages 686–687 and Figure 1.)
It would have been obvious to a person of ordinary skill in the art before the effective filing date to incorporate Gi hand-classification technique into the combined system of Lu and Lee and to classify Lee’s segmented hand regions using Lu’s body 3D joint locations, Lu’s/OpenPose’s body 2D joint estimates, and Lee’s body-part segmentation map. The separately labeled left- and right-wrist joint locations would provide anatomical reference points for associating each segmented hand region with the corresponding side of the body, while the segmentation map would identify the image region occupied by the detected hand or hands. A region corresponding to the left wrist could thereby be classified as a left-hand region, a region corresponding to the right wrist as a right-hand region, and a region containing or associated with both wrist locations as a both-hand region.
A person of ordinary skill would have been motivated to make this modification to reliably associate the detected hand regions with the proper side of the reconstructed body, prevent left/right hand swapping, and improve hand identification when the hands overlap or appear close together. The modification would have amounted to applying Gi’s known CNN hand-classification technique to the complementary joint-location and segmentation information already produced by Lu and Lee, with the predictable result of more accurate hand-region classification.
Claim(s) 9 is rejected under 35 U.S.C. 103 as being unpatentable over Lu (Lu Y, Yu H, Ni W, Song L. 3D real-time human reconstruction with a single RGBD camera. Appl Intell (Dordr). 2023;53(8)) in view of Mehta (Dushyant Mehta, Srinath Sridhar, Oleksandr Sotnychenko, Helge Rhodin, MohammadShafiei,Hans-PeterSeidel,WeipengXu,DanCasasandChristian Theobalt. 2017. VNect: Real-time 3D Human Pose Estimation with a Single RGB Camera. ACM Trans. Graph. 36, 4, Article 44 (July 2017), 14 pages. DOI: http://dx.doi.org/10.1145/3072959.3073596).
Regarding claim 9, Lu teaches The 3D motion capture method of claim 8, wherein: receiving the input image comprises sequentially receiving a plurality of frame images having the same size (Lu, section 3.3 and figs 1-2: receives an RGB-D image sequence from a camera. The consecutively captured video frames have the camera’s fixed image dimensions);in response to sequential reception of each of the plurality of frame images, the detecting and the recognizing 3D motion information are performed in real time for each of the plurality of frame images (Lu, abstract, and section 3.3 and 4.3: performs realtime 3d human reconstruction on the RGB-D sequence, reporting approximately 25 frames per second);in the detecting and the recognizing 3D motion information for a specific one of the plurality of frame images, at least some of the specific frame image is applied as input to the first machine learning model; but fails to each and the at least some of the specific frame image applied as input to the first machine learning model: when a person region is detected from a previous frame image of the specific frame image, includes a region at a location that is the same as a location of the person region detected from the previous frame image; and when a person region is not detected from a previous frame image of the specific frame image, is all the specific frame image.
Mehta teaches the at least some of the specific frame image applied as input to the first machine learning model(Mehta, Fig.2 and section 4.1: receives a full size image at frame and extracts a person centered crop, which is supplied to its CNN for joint 2d/3d pose prediction):when a person region is detected from a previous frame image of the specific frame image, includes a region at a location that is the same as a location of the person region detected from the previous frame image (Fig, 2 and sec 4.1: extracts the person centered crop for the current frame through bounding box tracking using the previous frame key points. It calculates a bounding box around the predicted person key points and filters the box using the previous frame bounding box); and when a person region is not detected from a previous frame image of the specific frame image, is all the specific frame image (Section 4.1 : explains that the bounding box tracker begins by performing multi scale predictions on the full image for the first few frames and then narrows the processing region after location the person. It would have been obvious to modify Lu system of Realtime 3d reconstruction method with Mehta previous frame bounding box tracking technique to reduce the image area processed by the machine learning model after locating the person, thereby decreasing computational expense and maintaining real time performance).
Claim(s) 11-12 are rejected under 35 U.S.C. 103 as being unpatentable over Lu (Lu Y, Yu H, Ni W, Song L. 3D real-time human reconstruction with a single RGBD camera. Appl Intell (Dordr). 2023;53(8)) in view of Sinha (US 20170168586 A1).
Regarding claim 11, Lu teaches The 3D motion capture method of claim 4, wherein: the third machine learning model is a model configured to output hand pose parameters based on the hand sub-images; and hand 3D mesh information, and camera parameters obtained based on the hand pose parameters (Lu, section 3.3 and figs 1-2: receives an RGB-D image sequence from a camera. The consecutively captured video frames have the camera’s fixed image dimensions).
Lu Fails to teach the hand motion information comprises first wrist angle information, hand 3D joint angle information
Sinha teaches the hand motion information comprises first wrist angle information, hand 3D joint angle information ( Para 34-35 and 38-40: teaches providing hand depth-map data to a neural network trained to identify wrist rotational features and generate hand-pose parameters, including wrist orientation in six degrees of freedom, while utilizing separate neural networks to classify finger-joint angles and determine corresponding finger hand-pose parameters, which are subsequently combined with the wrist parameters to generate a complete hand-pose model. It would have been obvious to modify Lu’s hand subnetwork to incorporate Sinha’s wrist- and finger-joint-angle estimation techniques so that Lu’s SMPL-X reconstruction system could determine more detailed hand-pose parameters, including wrist orientation and individual finger-joint angles, thereby improving the accuracy and realism of the reconstructed hand motion, particularly when portions of the hand are occluded.).
Regarding claim 12, Lu in view of Sinha teaches The 3D motion capture method of claim 11, wherein: the body motion information comprises second wrist angle information obtained based on an output of the second machine learning model; and the recognizing 3D motion information further comprises correcting the second wrist angle information obtained based on the output of the second machine learning model by using the first wrist angle information obtained based on the hand pose parameters.
Lu teaches the body motion information comprises second wrist angle information obtained based on an output of the second machine learning model (Lu, section 4.1, Fig. 2, and equations 1–2: teaches that the global body sub-network processes the human-body image and predicts SMPL-X body-pose parameters, which represent the respective three-dimensional joint rotations of the body, including the wrist-joint rotation). Lu further teaches weighting and fusing the pose outputs of the global body sub-network and the hand sub-network to obtain a final pose output (Lu, section 4.1, Fig. 2, and equation 9: teaches combining the pose output of the global sub-network with the pose output of the hand sub-network according to respective pose weights g and h).
Lu fails to expressly teach correcting the second wrist angle information obtained based on the output of the second machine learning model by using first wrist angle information obtained based on hand pose parameters. Sinha teaches obtaining first wrist angle information based on hand pose parameters (Sinha, paras. 34–35 and 38–40: teaches providing hand depth-map data to a neural network trained to identify features corresponding to the rotational angle of the wrist and generating hand-pose parameters that include wrist-angle information, including the orientation of the wrist in six degrees of freedom. It would have been obvious to modify Lu’s weighted pose-fusion process to incorporate Sinha’s hand-derived wrist-angle information when generating the final wrist pose, such that the wrist angle initially determined by Lu’s global body sub-network is refined or corrected using the more detailed wrist angle obtained from the hand-pose parameters. A person of ordinary skill in the art would have been motivated to make this modification to improve the accuracy and anatomical consistency of the reconstructed wrist motion, particularly when the wrist or hand is partially occluded or occupies only a small portion of the input image).
Claim(s) 13 is rejected under 35 U.S.C. 103 as being unpatentable over Lu (Lu Y, Yu H, Ni W, Song L. 3D real-time human reconstruction with a single RGBD camera. Appl Intell (Dordr). 2023;53(8)) in view of Sinha (US 20170168586 A1) in further view of Singhose (US 20200178851 A1).
Regarding claim 13, Lu teaches The 3D motion capture method of claim 11, wherein: the body motion information comprises elbow angle information and shoulder angle information obtained based on an output of the second machine learning model; and the recognizing 3D motion information further comprises correcting the elbow angle information and shoulder angle information obtained based on the output of the second machine learning model by using the first wrist angle information.
Lu teaches the body motion information comprises elbow angle information and shoulder angle information obtained based on an output of the second machine learning model (Lu, section 4.1, Figs. 1–2, and equations 1–2: teaches that the global body sub-network predicts SMPL-X body-pose parameters, which represent the three-dimensional rotations of the body joints, including the shoulder and elbow joints).
Lu fails to expressly teach correcting the elbow angle information and shoulder angle information obtained based on the output of the second machine learning model by using first wrist angle information.
Sinha teaches obtaining first wrist angle information from hand-pose parameters (Sinha, paras. 34–35 and 38–40: teaches providing hand depth-map data to a neural network trained to identify wrist rotational features and generating hand-pose parameters that include wrist-angle information, including wrist orientation in six degrees of freedom). However, Lu and Sinha fail to expressly teach using the first wrist angle information to correct both the elbow angle information and the shoulder angle information.
Singhose teaches modeling each human arm as a serial kinematic chain extending from the shoulder joint through the elbow joint and wrist joint to the tip of the hand (Singhose, paras. 78–79 and 88, Figs. 5–6: teaches representing the shoulder as three single-degree-of-freedom revolute joints, the elbow as two single-degree-of-freedom revolute joints, and the wrist as two single-degree-of-freedom revolute joints). Singhose further teaches using inverse kinematics to calculate a joint-angle vector q based on a given position and orientation of a hand end-effector x, such that the hand or wrist orientation is used to determine the associated shoulder and elbow joint angles of the connected kinematic chain (Singhose, paras. 93–97 and equation 4. It would have been obvious to modify the combined system of Lu and Sinha to incorporate Singhose’s inverse-kinematics technique so that the wrist-angle information obtained from Sinha’s hand-pose parameters is used as distal orientation information for recalculating or correcting the shoulder and elbow angles initially output by Lu’s global body sub-network. A person of ordinary skill in the art would have been motivated to make this modification to maintain kinematic consistency throughout the connected shoulder–elbow–wrist chain, reduce anatomically unreasonable upper-limb states caused by occlusion or inaccurate body-pose estimation, and improve the accuracy and realism of the reconstructed three-dimensional body motion.).
Claim(s) 14-15 are rejected under 35 U.S.C. 103 as being unpatentable over Lu (Lu Y, Yu H, Ni W, Song L. 3D real-time human reconstruction with a single RGBD camera. Appl Intell (Dordr). 2023;53(8)) in view of Feng (Yao Feng, Haiwen Feng, Michael J. Black, and Timo Bolkart. 2021. Learning an Animatable Detailed 3D Face Model from In-The-Wild Images. ACM Trans. Graph. 40, 4, Article 88 (August 2021), 13 pages. https://doi.org/10. 1145/3450626.3459936).
Regarding claim 14, Lu teaches The 3D motion capture method of claim 4, wherein the fourth machine learning model is a model configured to output 3D face model parameters, 3D facial landmark information, a blend shape, and camera parameters based on the face sub-image.
Lu teaches a fourth machine-learning model configured to output 3D face-model parameters based on a face sub-image. In particular, Lu obtains a face sub-image by cropping the original RGB and depth images using facial joint points, extracts face features using a face subnetwork, and uses a multilayer perceptron to predict model parameters (Lu, Section 4.1, Fig. 2, and Eqs. 3–9). Lu’s SMPL-X representation includes shape parameters , expression parameters , jaw-pose parameters, and eye-pose parameters. Lu further teaches that FBN and SMPL-X predict a 3D point cloud and camera parameters (Lu, “Texture Mapping,” step 2).
Lu does not expressly teach that the fourth machine-learning model outputs 3D facial landmark information and a blend shape, together with the camera parameters, based on the face sub-image.
Feng et al. teach a machine-learning face model, DECA, that receives a facial image and uses a ResNet50 encoder followed by a fully connected layer to predict FLAME shape parameters , expression parameters , pose parameters , and camera parameters (Feng, Section 4.1 and Fig. 2). Feng further teaches that FLAME is a statistical 3D head model employing separate identity and expression spaces and pose-dependent corrective blendshapes. Feng identifies corresponding facial landmarks on the three-dimensional FLAME surface and projects those 3D facial landmarks into the image using the estimated camera model (Feng, Section 3, Section 4.1, and Eq. 5). Thus, Feng teaches producing 3D face-model parameters, three-dimensional facial-landmark information, blendshape-based facial geometry, and camera parameters from a face image. It would have been obvious to one of ordinary skill in the art before the effective filing date to modify or implement Lu’s face subnetwork using Feng’s DECA face-model encoder so that Lu’s face subnetwork additionally outputs 3D facial landmarks, blendshape-based facial geometry, and face-specific camera parameters. A person of ordinary skill would have been motivated to make this modification to improve the accuracy and expressive detail of Lu’s reconstructed face and to provide the camera and landmark information needed to align and render the reconstructed face within Lu’s SMPL-X human-body model.
Regarding claim 15, Lu teaches The 3D motion capture method of claim 4, wherein: the face motion information comprises head pose information, face shape information, and face expression information; and the recognizing face motion information comprises recognizing the head pose information, the face shape information, and the face expression information separately from the face sub-image by using the fourth machine learning model.
Lu teaches face motion information comprising head-pose information, face-shape information, and face-expression information. In particular, Lu uses the SMPL-X human representation, which includes shape parameters , expression parameters , jaw-pose parameters, eye-pose parameters, and global-orientation parameters (Lu, Section 4.1 and Eqs. 1–2). Lu further teaches cropping the face from the original RGB and depth images, extracting face features through a face subnetwork, and using a multilayer perceptron to predict the model parameters (Lu, Section 4.1, Fig. 2, and Eqs. 3–9).
Lu does not expressly teach separately recognizing the head-pose information, face-shape information, and face-expression information from the face sub-image using the fourth machine-learning model. Feng teaches a fourth machine-learning model configured to separately recognize head-pose information, face-shape information, and face-expression information from a face image. Specifically, Feng’s DECA system receives a face image and uses a ResNet50 encoder followed by a fully connected layer to regress a latent code containing separate FLAME shape parameters β, expression parameters ψ, and pose parameters θ. Feng explains that these parameters represent distinct components of the reconstructed three-dimensional face geometry, including facial identity shape, facial expression, and head or facial pose (Feng, Sections 3 and 4.1 and Fig. 2. It would have been obvious to one of ordinary skill in the art before the effective filing date to implement or modify Lu’s face subnetwork using Feng’s DECA encoder so that the head-pose information, face-shape information, and face-expression information are separately recognized from the face sub-image. A person of ordinary skill would have been motivated to make this modification because separately estimating pose, identity shape, and expression would permit independent control of the corresponding facial characteristics and improve the accuracy, expressive detail, and animation of Lu’s reconstructed three-dimensional human model).
Claim(s) 16 is rejected under 35 U.S.C. 103 as being unpatentable over Lu (Lu Y, Yu H, Ni W, Song L. 3D real-time human reconstruction with a single RGBD camera. Appl Intell (Dordr). 2023;53(8)) in view of Feng (Yao Feng, Haiwen Feng, Michael J. Black, and Timo Bolkart. 2021. Learning an Animatable Detailed 3D Face Model from In-The-Wild Images. ACM Trans. Graph. 40, 4, Article 88 (August 2021), 13 pages. https://doi.org/10. 1145/3450626.3459936) in further view of Yu (US 20200035007 A1).
Regarding claim 16, Lu teaches The 3D motion capture method of claim 15, wherein the face expression information comprises gaze information and tongue movement information.
Lu teaches that the face-expression information comprises gaze information. Specifically, Lu’s SMPL-X representation includes separate eye-pose parameters , which represent the respective orientations or poses of the eyes and therefore provide information indicating the user’s gaze direction (Lu, Section 4.1, Fig. 2, and Eqs. 1–2). Lu further extracts a face sub-image, processes the face sub-image through the face subnetwork, and predicts the facial parameters using a multilayer perceptron (Lu, Section 4.1 and Eqs. 3–9). Feng further supports recognizing gaze information because Feng’s FLAME model articulates the eyeballs and predicts pose parameters from a face image using a ResNet50 encoder and fully connected layer (Feng, Sections 3 and 4.1 and Fig. 2).
Lu in view of Feng, fails to teach that the face-expression information further comprises tongue-movement information.
Yu et al. teach facial-expression information comprising tongue-movement information. Yu discloses monitoring the mouth region using image and depth sensors to detect a physical tongue and track its movements, including the tongue’s extension, length, position, and direction. The detected tongue movement is used to animate a corresponding digital tongue forming part of a three-dimensional digital representation of the face (Yu, para[0002]–[0003], [0063]–[0065], [0077]–[0081], and [0090]–[0104]; Figs. 3–8). Yu further teaches that machine-learning, computer-vision, or image-processing algorithms may identify a tongue-tip landmark from an image and that different tongue-expression parameters control the horizontal, vertical, and depth components of the tongue’s movement (Yu, para[0063], [0091]–[0094], and [0102]–[0104]. It would have been obvious to one of ordinary skill in the art before the effective filing date to incorporate Yu’s tongue-detection and tongue-animation technique into the face subnetwork of Lu, as modified by Feng, so that the resulting facial-expression information additionally includes tongue-movement information. A person of ordinary skill would have been motivated to make this modification to capture additional expressive movements of the mouth region and produce a more complete, accurate, and realistic animation of the reconstructed face. The modification would have involved the predictable use of Yu’s known image-based tongue-tracking technique with Lu’s RGB-D facial input and parametric three-dimensional facial model).
Claim(s) 17 is rejected under 35 U.S.C. 103 as being unpatentable over Lu (Lu Y, Yu H, Ni W, Song L. 3D real-time human reconstruction with a single RGBD camera. Appl Intell (Dordr). 2023;53(8)) in view of Feng (Yao Feng, Haiwen Feng, Michael J. Black, and Timo Bolkart. 2021. Learning an Animatable Detailed 3D Face Model from In-The-Wild Images. ACM Trans. Graph. 40, 4, Article 88 (August 2021), 13 pages. https://doi.org/10. 1145/3450626.3459936) in further view of Feng ( Collaborative Regression of Expressive Bodies using Moderation}, Yao Feng and Vasileios Choutas and Timo Bolkart and Dimitrios Tzionas and Michael J. Black 2021 {https://arxiv.org/abs/2105.05301}).
Regarding claim 17, Lu teaches The 3D motion capture method of claim 15, wherein: the body motion information comprises head angle information obtained based on an output of the second machine learning model; and the recognizing 3D motion information further comprises correcting the head angle information obtained based on the output of the second machine learning model by using the head pose information.
Lu teaches that the second/global body machine-learning model produces body-motion information comprising articulated body-pose parameters. In particular, Lu’s FBN predicts SMPL-X pose parameters, including body pose θb and global orientation θg, based on features extracted by the global body subnetwork. A person of ordinary skill would have understood the rotational pose parameters associated with the neck/head portion of the SMPL-X articulated body model to constitute head-angle information obtained from the output of the second machine-learning model. (Lu, Sections 3.3 and 4.1, Figures 1–2; SMPL-X pose parameters and FBN architecture.)
As discussed with respect to claim 15, Feng et al. (DECA) teaches recognizing head-pose information from a face image. DECA applies a ResNet50 encoder and fully connected layer to the face image and separately regresses FLAME shape β, expression ψ, and pose θ parameters. The FLAME pose parameters articulate the neck, jaw, and eyeballs, and DECA expressly identifies the reconstructed head pose as a separately controllable component. (Feng et al., Sections 3 and 4.1, Figure 2.)
Lu and DECA do not expressly teach correcting the head-angle information produced by the body machine-learning model using the head-pose information obtained from the face sub-image.
However, Feng et al. (PIXIE), “Collaborative Regression of Expressive Bodies Using Moderation,” teaches integrating body and face/head estimates to obtain a more reliable head rotation. PIXIE extracts separate body and face/head image crops, supplies the crops to respective body and face/head expert encoders, and identifies the body and head experts as complementary. A learned moderator fuses the body and face/head features by a confidence-weighted sum. The fused features are supplied to a fused face/head regressor that outputs head rotation , expression , and jaw pose . (PIXIE, Section 3.2, Figure 3, Equations 1–2.)
It would have been obvious to a person of ordinary skill in the art before the effective filing date to modify Lu’s system, as supplemented by DECA, to incorporate PIXIE’s confidence-weighted body and face/head fusion. Under the resulting combination, the head-angle information initially derived from Lu’s second/global body model would be adjusted or corrected using the head-pose information recognized from the face sub-image, and the fused face/head regressor would output the corrected head rotation.
Claim(s) 18 is rejected under 35 U.S.C. 103 as being unpatentable over Lu (Lu Y, Yu H, Ni W, Song L. 3D real-time human reconstruction with a single RGBD camera. Appl Intell (Dordr). 2023;53(8)) in view of Akhter (Pose-Conditioned Joint Angle Limits for 3D Human Pose ReconstructionIjaz Akhter, Michael J. Black; Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015, pp. 1446-1455).
Regarding claim 18, Lu teaches The 3D motion capture method of claim 1, wherein: the 3D motion information comprises joint angle information; and the recognizing 3D motion information comprises correcting the joint angle information by using at least one of a body ratio of the person object included in the input image, a body ratio of the character, and joint angle restriction information associated with the character.
Lu teaches the 3D motion capture method of claim 1 and further teaches that the 3D motion information comprises joint-angle information. Particularly, Lu’s FBN outputs SMPL-X pose parameters θ, including body-pose parameters, hand-pose parameters, jaw-pose parameters, eye-pose parameters, and global-orientation parameters. These pose parameters represent rotations of the respective articulated joints of the reconstructed character and therefore constitute joint-angle information. (Lu, Section 4.1, Figure 2, Equations 1–2 and 6–9.)
Lu further teaches placing a denoising autoencoder at the output of FBN to correct unreasonable body states caused by occlusion, including palm valgus and an inwardly positioned elbow joint. Lu treats the unreasonable pose parameters as noise, filters the unreasonable states, and produces a more realistic human reconstruction. (Lu, Section 4.2, “Compression of the Parameters,” Figure 3.)
However, Lu fails to expressly teach that the correction of the joint-angle information is performed using body-ratio information or expressly defined joint-angle restriction information associated with the reconstructed character.
Akhter and Black teach correcting estimated 3D human-pose information using body-proportion and pose-conditioned joint-angle restrictions. Akhter defines a physically motivated pose prior containing pose-dependent joint-angle limits that distinguishes valid and invalid configurations of a kinematic human skeleton. During pose estimation, a penalty term or nonlinear inequality constraint is applied so that configurations violating the joint-angle limits are rejected and the estimated pose remains anatomically valid. Akhter also teaches using known body proportions and penalizing differences between estimated bone lengths and normalized mean bone lengths. (Akhter and Black, Sections 3–4, Figure 4(c), Equations 3 and 7.)
It would have been obvious to a person of ordinary skill in the art before the effective filing date to modify Lu’s joint-angle correction process to incorporate Akhter’s pose-conditioned joint-angle limits and body-proportion constraints. A person of ordinary skill would have been motivated to apply those restrictions to the SMPL-X joint-angle parameters generated by Lu’s FBN because Lu expressly recognizes that occlusion can produce anatomically unreasonable joint configurations, such as palm valgus and inward elbow positions. Employing Akhter’s known joint-angle validity prior and body-proportion constraints in Lu’s denoising or post-processing stage would predictably identify and correct those invalid joint angles, maintain anatomically permissible joint motion, and produce a more realistic reconstructed character.
Claim(s) 19 is rejected under 35 U.S.C. 103 as being unpatentable over Lu (Lu Y, Yu H, Ni W, Song L. 3D real-time human reconstruction with a single RGBD camera. Appl Intell (Dordr). 2023;53(8)) in view of Sipe (Made of ref: US 8457367 B1)
Regarding claim 19, Lu teaches A computer-readable storage medium having recorded thereon instructions for executing the method according to claim 1 on a computer.
Lu fails to teach A computer-readable non-transitory storage medium. Sipe teaches a computer-readable non-transitory storage medium (Col.40: Teaches a non-transitory storage medium. It would have been obvious for a person skilled in the art to combine the teachings of Sipe with the system of Lu to use a non-transitory storage medium).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Konovalov et al (Automatic Hand Detection in RGB-Depth Data Sequences Authors Vitaliy Konovalov, Albert Clapés, Sergio Escalera Pages 91 - 100DOI10.3233/978-1-61499-320-9-91 (Year: 2013)): relevant for teaching segmenting a body region from an RGB-depth image, projecting a two-dimensional anatomical reference location into three-dimensional space, constructing a three-dimensional graph representing the segmented body, and identifying the two hand regions based on their spatial and geodesic relationships to the anatomical reference point.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to LATRELL ANTHONY CREARY whose telephone number is (703)756-1219. The examiner can normally be reached Mon - Fri 7:30am - 4:30pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Xiao Wu can be reached on (571) 272-7761. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/LATRELL ANTHONY CREARY/Examiner, Art Unit 2613
/XIAO M WU/Supervisory Patent Examiner, Art Unit 2613