DETAILED ACTION
Status of Claims
Claims 1-20 are pending in this application, with claims 1, 11 and 17 being independent.
Notice of AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Obligation Under 37 CFR 1.56 – Joint Inventors
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Drawings
The drawings were received on September 25, 2024. These drawings are acceptable.
Claim Objections
Claim 11 is objected to because of the following informalities: the comma in line 5 of claim 11 is unnecessary and/or improper and, as such, renders claim 11 vague and indefinite. Appropriate correction is required.
Claim 17 is objected to because of the following informalities: the comma in line 7 of claim 17 is unnecessary and/or improper and, as such, renders claim 11 vague and indefinite. Appropriate correction is required.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 11-16 are rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter.
As per the published guidelines (1351 OG 212; Feb. 23, 2010) regarding subject-matter eligibility of computer readable media, the United States Patent and Trademark Office (USPTO) is obliged to give claims their broadest reasonable interpretation consistent with the specification during proceedings before the USPTO. See In re Zletz, 893 F.2d 319 (Fed. Cir. 1989) (during patent examination the pending claims must be interpreted as broadly as their terms reasonably allow). The broadest reasonable interpretation of a claim drawn to a computer readable medium (also called machine readable medium and other such variations) typically covers forms of non-transitory tangible media and transitory propagating signals per se in view of the ordinary and customary meaning of computer readable media, particularly when the specification is silent. See MPEP 2111.01. When the broadest reasonable interpretation of a claim covers a signal per se, the claim must be rejected under 35 U.S.C. § 101 as covering non-statutory subject matter. See In re Nuijten, 500 F.3d 1346, 1356-57 (Fed. Cir. 2007) (transitory embodiments are not directed to statutory subject matter) and Interim Examination Instructions for Evaluating Subject Matter Eligibility Under 35 U.S.C. § 101, Aug. 24, 2009; p. 2.
The USPTO recognizes that applicants may have claims directed to computer readable media that cover signals per se, which the USPTO must reject under 35 U.S.C. § 101 as covering both non-statutory subject matter and statutory subject matter. In an effort to assist the patent community in overcoming a rejection or potential rejection under 35 U.S.C. § 101 in this situation, the USPTO suggests the following approach. A claim drawn to such a computer readable medium that covers both transitory and non-transitory embodiments may be amended to narrow the claim to cover only statutory embodiments to avoid a rejection under 35 U.S.C. § 101 by adding the limitation "non-transitory" to the claim. Cf. Animals - Patentability, 1077 Off. Gaz. Pat. Office 24 (April 21, 1987) (suggesting that applicants add the limitation "non-human" to a claim covering a multi-cellular organism to avoid a rejection under 35 U.S.C. § 101). Such an amendment would typically not raise the issue of new matter, even when the specification is silent because the broadest reasonable interpretation relies on the ordinary and customary meaning that includes signals per se. The limited situations in which such an amendment could raise issues of new matter occur, for example, when the specification does not support a non-transitory embodiment because a signal per se is the only viable embodiment such that the amended claim is impermissibly broadened beyond the supporting disclosure. See, e.g., Gentry Gallery, Inc. v. Berkline Corp., 134 F.3d 1473 (Fed. Cir. 1998).
Regarding claims 11-16, claims 11-16 are rejected because claims 11-16 recite a “computer-readable storage medium”. Neither the claims, the specification nor the record explicitly disclose that the claimed “computer-readable storage medium” is limited to only a non-transitory medium. Thus, the examiner asserts that the claimed “computer-readable storage medium” can be a transitory signal, which is non-statutory. In order to overcome the rejection of claims 11-16 under 35 U.S.C. § 101, Examiner recommends amending claims 11-16 to explicitly recite a “non-transitory computer-readable storage medium”.
Claim Rejections – 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
Determining the scope and contents of the prior art;
Ascertaining the differences between the prior art and the claims at issue;
Resolving the level of ordinary skill in the pertinent art; and
Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-2 and 6-10 are rejected under 35 U.S.C. 103 as being unpatentable over CHENG et al. (Cheng Y, Yang B, Wang B, Tan RT. “3d human pose estimation using spatio-temporal networks with explicit occlusion training.” In Proceedings of the AAAI Conference on Artificial Intelligence 2020 Apr 3 (Vol. 34, No. 07, pp. 10631-10638); hereinafter “CHENG 2020”) in view of CHENG et al. (Cheng Y, Yang B, Wang B, Yan W, Tan RT. Occlusion-aware networks for 3d human pose estimation in video. In Proceedings of the IEEE/CVF international conference on computer vision 2019 (pp. 723-732); hereinafter “CHENG 2019”).
Regarding claim 1, CHENG 2020 discloses a method for synthesizing a full body representation of a user for application in an artificial reality environment (p. 10631, Title: “3D Human Pose Estimation”), the method comprising:
obtaining, over multiple frames (e.g., the input frames shown in Figures 1-2 and 6.), one or more body tracking signals for one or more body parts of a body of the user (e.g., the image of the body in the input video frames, i.e., video frame pixels corresponding the body) (On p. 10631, see the input video frames (i.e., multiple frames) shown in Figure 1-2 and 6. p. 10632, Figure 2: “Input Frames”. p. 10633, 2nd paragraph: “Given an input video, we first detect and track the persons by any state-of-the-art detector and tracker, such as Mask R-CNN (He et al. 2017) and PoseFlow (Xiu et al. 2018).”) in a real-world environment (p. 10631, 2nd paragraph: “the target person in wild videos.” As can been seen in Figures 1-2 and 6. the input video frames are all taken in a real-world environment.); and
based on the one or more body tracking signals, synthesizing the full body representation of the user (p. 10633, 2nd paragraph (“Methodology”): “Given an input video, we first detect and track the persons by any state-of-the-art detector and tracker, such as Mask R-CNN (He et al. 2017) and PoseFlow (Xiu et al. 2018). Subsequently, we perform the pose estimation for each person individually.” See 3D human pose estimation “Results” in Figure 1.) by:
estimating scale of the body of the user (3rd paragraph on p. 10633 (i.e., 1st paragraph under “Multi-Scale Features for Pose Estimation”): “Given a series of bounding box for a person in a video,” NOTE: To generate each bounding box (i.e., the minimum sized box that will enclose the body of the person in the video frame), the scale (i.e., height and width) of the subject must be determined. Thus, the body of the user must be detected in the input video image frames (i.e., pixels corresponding the body must be identified), and the size of the body of the user determined in order to generate a bounding box around the body of the user in the image frame, i.e., the minimum sized box that encloses the identified pixels corresponding to the body of the user.)
normalizing at least one of: A) one or more positions of the one or more corresponding body parts, estimated from the one or more body tracking signals (e.g., the pixels corresponding the body in the video frame), to be independent of the estimated scale (3rd paragraph on p. 10633: “Given a series of bounding boxes for a person in a video, we first normalize the image within each bounding box to a pre-defined fixed size, e.g., 256 × 256, and then apply High Resolution Networks (HRNet) (Sun et al. 2019) to each normalized image patch to produce K heat maps, each of which indicates the possibility of certain human joint’s location.” NOTE: The “pre-defined fixed size” is independent of the determined scale (height and width) of the bounding box enclosing the detected body of the user in the image frame.), B) one or more trajectories of the one or more body parts, estimated from the one or more body tracking signals, based on the estimated scale, C) a representation of space, surrounding the user in the real-world environment, based on the estimated scale, or D) any combination thereof (NOTE: Since alternatives limitations are recited, only one of the limitations must be met.); and
synthesizing multiple poses of the body of the user, over the respective multiple frames (See the 3D human pose “Results” in Figure 1 (p. 10631), Figure 2 (p. 106302), and Figure 6 (p. 10637). 1st paragraph on p. 10631 (“Introduction”): “3D human pose estimation from a monocular RGB video. A 3D pose is defined as the 3D coordinates of pre-defined keypoints on humans, such as shoulder, pelvis, wrist, and etc.”), using the one or more body tracking signals (e.g., the input video frames in Figures 1-2 and 6. 1st paragraph on p. 10631 (“Introduction”): ““3D human pose estimation from a monocular RGB video.”) and the at least one of A), B), C), or D), by applying a neural network (p. 10631, Abstract: “As humans in videos may appear in different scales and have various motion speeds, we apply multi-scale spatial features for 2D joints or keypoints prediction in each individual frame, and multi-stride temporal convolutional networks (TCNs) to estimate 3D joints or keypoints.” p. 10631, 2nd paragraph: “First, we consider multi-scale features both spatially and temporally to deal with persons at various distances with different speeds of motions. We use the High Resolution Network (HRNet) (Sun et al. 2019) which exploits multi-scale spatial features to produce one heat map for each keypoint. Unlike most previous works (Newell, Yang, and Deng 2016; Pavllo et al. 2019) that only use the peaks in the heat maps, we encode these maps into a latent space to incorporate more spatial information. Then, we apply temporal convolutional networks (TCNs) (Pavllo et al. 2019) to these latent features with different strides, e.g., 1, 2, 4, and 8, and concatenate them together for prediction of the 3D poses. Figure 1 shows some examples of our results.” See the network framework in Figure 2. 3rd paragraph on p. 10633: “Given a series of bounding boxes for a person in a video, we first normalize the image within each bounding box to a pre-defined fixed size, e.g., 256 × 256, and then apply High Resolution Networks (HRNet) (Sun et al. 2019) to each normalized image patch to produce K heat maps, each of which indicates the possibility of certain human joint’s location.” 5th paragraph on p. 10633: “Given a sequence of heat map embeddings {rt}, we apply TCN to them.” 8th paragraph on p. 10634 (i.e., the 3rd paragraph under “Data Augmentation for Occlusions”): “the trained multi-scale TCN”) trained on:
i) historical motion data, of other bodies of other users, captured by multiple input sensors (6th paragraph on p. 10633 (i.e., 4th paragraph under “Multi-Scale Features for Pose Estimation”): We use both 3D dataset Human3.6M (Ionescu et al. 2014) and 2D dataset Penn Action (Zhang, Zhu, and Derpanis 2013) for training. Human3.6M has multi-view captured videos and 3D ground-truths, while PENN only has 2D ground-truths for visible keypoints.” … “As Human3.6M data set provides videos from multiple views, we expect the 3D estimation results from different views should be the same after rotation alignment.” NOTE: In other words, the Human3.6M dataset is captured using multiple cameras capturing different views of the actors performing motions. 1st paragraph on p. 10635: “Data Sets. Human3.6M (Ionescu et al. 2014) is a large 3D human pose dataset. It has 3.6 million images including eleven actors performing daily-life activities, and seven actors are annotated. The 3D ground-truth is provided by the mocap system, and the intrinsic/extrinsic camera parameters are known. Similar to some existing methods (Hossain and Little 2018; Pavllo et al. 2019; Pavlakos, Zhou, and Daniilidis 2018; Yang et al. 2018), we use subjects 1, 5, 6, 7, 8 for training, and the subjects 9 and 11 for evaluation.”), and
ii) one or more masking techniques applied to the historical motion data, the one or more masking techniques accounting for lack of visibility of one or more other body parts, of the other bodies of the other users, by the multiple input sensors (p. 10634, 5th -6th paragraphs (1st-2nd paragraphs under “Data Augmentation for Occlusions”): “To make our approach capable of dealing with different occlusion cases, we perform data augmentation during the training. We use random masking of keypoints to simulate the occluded condition. Three types of occlusion are applied in the training process. The first type is the frame-wise occlusion. Given a sequence of heatmaps produced by the 2D keypoint estimator, we randomly mask several frames by setting their heatmaps to zero, indicating that the whole frame is occluded or has low confidence. Second, the point-wise occlusion is applied by randomly setting certain keypoints’ heatmaps to zero. This simulates the scenario that certain keypoints are occluded. Third, we apply area occlusion by setting a virtual occluder area. The heatmaps of keypoints located within this area are set to zero.”).
CHENG 2020 fails to explicitly disclose: “estimating scale of the body of the user by applying a machine learning model to the one or more body tracking signals.”
However, whereas CHENG 2020 is not entirely explicit as to, CHENG 2019 teaches a method for synthesizing a full body representation of a user (p. 723, Title: “Occlusion-Aware Networks for 3D Human Pose Estimation in Video”) for application in an artificial reality environment (p. 723, 1st paragraph of § 1: “Estimating 3D human poses from a monocular video is important in many applications, such as animation generation, activity recognition, human-computer interaction, and etc.”), the method comprising:
obtaining, over multiple frames (e.g., p. 723, Abstract: “from a monocular video”; p. 725, 1st paragraph of § 3: “Given an input video,” NOTE: The input video(s) comprise multiple frames), one or more body tracking signals for one or more body parts of a body of the user (e.g., pixels corresponding to the body in each of the input video frames) in a real-world environment (e.g., as shown in Figures 1-3, the input video frames are all images of one or body parts of a user/subject captured in a real-world environment. p. 724, 1st paragraph of § 2: “pose estimation for wild videos,” 3rd paragraph on p. 724 (i.e., 1st paragraph of § 3): ”Figure 2 shows an overview of our framework. Given an input video, we apply a human detector, such as Mask RCNN [12], to each frame, normalize each detected human bounding box to a fixed size while keeping the width/height ratio, and feed it to our first network, a stacked hourglass network [25], which estimates the 2D keypoints in the form of heatmaps (or confidence maps). Subsequently, our second network (2D TCN) improves the accuracy of the estimated 2D keypoints, and feed them further to our third network (3D TCN) to obtain the final 3D pose.” NOTE: In other words, the one or more body tracking signals are the video images of the body of user.); and
based on the one or more body tracking signals (p. 724, 1st paragraph in § 3: ” Given an input video,”), synthesizing the full body representation of the user (p. 724, 1st paragraph in § 3: “Figure 2 shows an overview of our framework. Given an input video, we apply a human detector, such as Mask RCNN [12], to each frame, normalize each detected human bounding box to a fixed size while keeping the width/height ratio, and feed it to our first network, a stacked hourglass network [25], which estimates the 2D keypoints in the form of heatmaps (or confidence maps). Subsequently, our second network (2D TCN) improves the accuracy of the estimated 2D keypoints, and feed them further to our third network (3D TCN) to obtain the final 3D pose.”) by:
estimating scale of the body of the user (p. 724, 1st paragraph in § 3: ”apply a human detector, such as Mask R-CNN [12], to each frame,” … “each detected human bounding box”; In other words, the Mask R-CNN detects the human body in each frame and determines a bounding box for the body. Furthermore, determining the bounding box around the body in each video frame requires detecting (i.e., estimating) the location and scale of the body in each video frame.) by applying a machine learning model (p. 724, 1st paragraph in § 3: “apply a human detector, such as Mask R-CNN”) to the one or more body tracking signals (e.g., pixels corresponding to the body in each of the input video frames. p. 724, 1st paragraph in § 3: ”input video,”) (p. 724, 1st paragraph in § 3: ”Figure 2 shows an overview of our framework. Given an input video, we apply a human detector, such as Mask R-CNN [12], to each frame, normalize each detected human bounding box to a fixed size while keeping the width/height ratio, and feed it to our first network, a stacked hourglass network [25], which estimates the 2D keypoints in the form of heatmaps (or confidence maps). Subsequently, our second network (2D TCN) improves the accuracy of the estimated 2D keypoints, and feed them further to our third network (3D TCN) to obtain the final 3D pose.”);
normalizing at least one of: A) one or more positions of the one or more corresponding body parts, estimated from the one or more body tracking signals (e.g., the pixels corresponding the body in the video frame), to be independent of the estimated scale (p. 724, 1st paragraph in § 3: ”Figure 2 shows an overview of our framework. Given an input video, we apply a human detector, such as Mask R-CNN [12], to each frame, normalize each detected human bounding box to a fixed size while keeping the width/height ratio, and feed it to our first network, a stacked hourglass network [25], which estimates the 2D keypoints in the form of heatmaps (or confidence maps). Subsequently, our second network (2D TCN) improves the accuracy of the estimated 2D keypoints, and feed them further to our third network (3D TCN) to obtain the final 3D pose.” NOTE: In each frame, the bounding box is normalized to a fixed size (i.e., size of the image of the body within the bounding box is normalized to a fixed size). Thus, since the normalized bounding box is a fixed size, the normalized size of the image of the body within the bounding box is independent of the estimated scale of the body (i.e., is independent of the inferred size of the bounding box fitting around the body in each frame).); and
synthesizing multiple poses of the body of the user, over the respective multiple frames, using the one or more body tracking signals and the at least one of A), B), C), or D), by applying a neural network (p. 724, 1st paragraph in § 3: ”Figure 2 shows an overview of our framework. Given an input video, we apply a human detector, such as Mask R-CNN [12], to each frame, normalize each detected human bounding box to a fixed size while keeping the width/height ratio, and feed it to our first network, a stacked hourglass network [25], which estimates the 2D keypoints in the form of heatmaps (or confidence maps). Subsequently, our second network (2D TCN) improves the accuracy of the estimated 2D keypoints, and feed them further to our third network (3D TCN) to obtain the final 3D pose.”).
Thus, based on the teachings of CHENG 2020 and CHENG 2019, in order to detect the bounding boxes for the person in the input video (3rd paragraph on p. 10633 of CHENG 2020), it would have been obvious to one of ordinary skill in the art to have modified the 3D human pose estimation system taught by CHENG 2020 so as to incorporate estimating scale of the body of the user by applying a machine learning model to the one or more body tracking signals, as taught by CHENG 2019.
Regarding claim 2 (depends on claim 1), CHENG 2020 discloses:
wherein the one or more body tracking signals are obtained from at least one sensor, the at least one sensor including an inertial measurement unit, an image capture device (e.g., p. 10631, 1st paragraph of “Introduction”: “from a monocular RGB video.” p. 10633, 1st paragraph of “Methodology”: “Given an input video”. NOTE: Video is captured by a video camera (i.e., “an image capture device”).), an electromyography sensor, or any combination thereof (NOTE: Since alternative limitations are presented, if one limitation has been met, then the remaining alternative limitations need not be given patentable weight.).
Regarding claim 6 (depends on claim 1), whereas CHENG 2020 may not be entirely explicit as to, CHENG 2019 further teaches:
wherein the scale of the body of the user includes at least one of height of the user (p. 725, 1st paragraph of § 3: “Given an input video, we apply a human detector, such as Mask R CNN [12], to each frame, normalize each detected human bounding box to a fixed size while keeping the width/height ratio, and feed it to our first network, a stacked hourglass network [25], which estimates the 2D keypoints in the form of heatmaps (or confidence maps). Subsequently, our second network (2D TCN) improves the accuracy of the estimated 2D keypoints, and feed them further to our third network (3D TCN) to obtain the final 3D pose. Our framework is end-to-end, for both training and testing.” NOTE: Determining the size, in particular the height, of the human bounding box likewise determines the height of the detected/tracked human in each frame of the input video.) and/or one or more bone lengths of the user ( ) (NOTE: Since alternative limitations are presented, if one limitation has been met, then the remaining alternative limitations need not be given patentable weight.).
Regarding claim 7 (depends on claim 1), CHENG 2020 discloses:
wherein the neural network includes at least one of a temporal convolutional encoder (p. 10631, 2nd paragraph of “Introduction”: “we apply temporal convolutional networks (TCNs)”), a long short-term memory network, a multi-task multi-layer perception model, or any combination thereof (NOTE: Since alternative limitations are presented, if one limitation has been met, then the remaining alternative limitations need not be given patentable weight.).
Regarding claim 8 (depends on claim 1), CHENG 2020 discloses:
wherein synthesizing the multiple poses includes at least one of D) estimating a global position and orientation of the body of the user in the representation of space, E) estimating one or more bone lengths of the user (1st – 3rd paragraphs on p. 10634: “Among all single frame discriminators, the Kinematic Chain Space (KCS) used in (Wandt and Rosenhahn 2019) is one of the most effective methods. Each bone, defined as the connection between two neighboring human keypoints such as elbow and wrist, is represented as a 3D vector bm, indicating the direction from one keypoint to its neighbor. All such vectors form a 3 × M matrix B, where M is the predefined number of bones for a human structure. They use Ψ=BTB as the features for discriminator, where the diagonal elements in Ψ indicate the square of bone length and other elements represent the weighted angle between two bones as an inner production. Inspired by their spatial KCS, we introduce a Temporal KCS(TKCS) defined as: Φ=BTt+I Bt+i −BTt Bt. where i is the temporal interval between the KCS. The diagonal elements in Φ indicates the bone length changes, and other elements denote the change of angles between two bones. Figure 4 shows an example of two neighboring bones b1 and b2. The spatial KCS measures the lengths of b1 and b2 as well as angles between them, θ12. The temporal KCS measures the bone length changes between two frames with temporal interval i, i.e., differences between b1t and b1t+I as well as b2t and b2t+I and the angle change between neighboring bones, i.e., difference between θ12 t and θ12 t+i . We concatenate the spatial KCS, temporal KCS, and the predicted keypoint coordinates, and then feed them to a TCN to build a discriminator. Such approach not only considers whether a pose is valid in individual frames, but also checks the validity of transitions across frames. We follow the procedure in the standard GAN to train the discriminator, and use it to produce a regularization loss for our predicted poses as Lgen.”), F) estimating one or more poses, of the multiple poses of the body of the user, based on anatomical body model, G) estimating a probability that one or more feet joints, of the body of the user, are in contact with ground in the real-world environment, H) estimating a probability that one or more hips, of the body of the user, are in contact with a physical object or the ground in the real-world environment, or I) any combination thereof (NOTE: Since alternative limitations are presented, if one limitation has been met, then the remaining alternative limitations need not be given patentable weight.).
Regarding claim 9 (depends on claim 1), CHENG 2020 discloses:
wherein the neural network applies at least one of a body pose reconstruction loss (p. 10633, 4th – 5th paragraphs in “Multi-Scale Features for Pose Estimation”: We use both 3D dataset Human3.6M (Ionescu et al. 2014) and 2D dataset Penn Action (Zhang, Zhu, and Derpanis 2013) for training. Human3.6M has multi-view captured videos and 3D ground-truths, while PENN only has 2D ground-truths for visible keypoints. For Human3.6M data, the 3D MSE loss is defined as: (1) L3d =(X −X3D)2, where X is our predicted 3D coordinates for all keypoints, and X3D is the 3D ground truth. As Human3.6M data set provides videos from multiple views, we expect the 3D estimation results from different views should be the same after rotation alignment. So, we define the multi-view loss as: (2) Lmv =(Rv1→v2Xv1 −Xv2)2, where Rv1→v2 is the rotation matrix from viewpoint 1 to viewpoint 2, and is precomputed from the ground-truth camera parameters. The Xv1 and Xv2 are the predicted 3D results in viewpoints 1 and 2. For the 2D dataset, we project the 3D prediction to 2D space assuming orthogonal projection, and the 2D MSE loss is defined as: (3) L2d =(Orth(X)−X2D)2, where Orth(·) is the orthogonal projection operator, and X2D is the 2D ground truth.” 5th paragraph on p. 10634: “The overall loss function for our training is defined as L =L3d + w1Lmv + w2L2d + w3L′gen, where w1,w2,w3 are set to 0.5, 0.1, 0.01, respectively, and are fixed in all our experiments.”), an anatomical representation loss, a feet sliding loss, a bone length loss, contact classification loss for feet, contact classification loss for hip, or any combination thereof (NOTE: Since alternative limitations are presented, if one limitation has been met, then the remaining alternative limitations need not be given patentable weight.).
Regarding claim 10 (depends on claim 1), CHENG 2020 and CHENG 2019 disclose:
wherein the body scale of the user is predicted without calibration to the user (Neither CHENG 2020 nor CHENG 2019 use or require any calibration to the user (i.e., person(s) detected and tracked in the input video frames.).
Claims 11-14 and 17-20 are rejected under 35 U.S.C. 103 as being unpatentable over CHENG et al. (Cheng Y, Yang B, Wang B, Tan RT. “3d human pose estimation using spatio-temporal networks with explicit occlusion training.” In Proceedings of the AAAI Conference on Artificial Intelligence 2020 Apr 3 (Vol. 34, No. 07, pp. 10631-10638); “CHENG 2020”) in view of YANG et al. (EP 4 002 198), further in view of CHENG et al. (Cheng Y, Yang B, Wang B, Yan W, Tan RT. Occlusion-aware networks for 3d human pose estimation in video. In Proceedings of the IEEE/CVF international conference on computer vision 2019 (pp. 723-732); “CHENG 2019”).
Regarding claim 11, CHENG 2020 discloses a computer-readable storage medium storing instructions, for synthesizing a full body representation of a user for application in an artificial reality environment (p. 10631, Title: “3D Human Pose Estimation”), the instructions, when executed by a computing system (One of ordinary skill in the art would have understood that the artificial intelligence method/system taught by CHENG 2020 is a computer implemented method. Further, one of ordinary skill in the art would understand that a computer-implemented method, such as the one disclosed by CHENG 2020, is customarily implemented using memory storing instructions for executing the method on a computing system. For instance, YANG et al. (EP 4 002 198) clearly teaches a related computer-implemented pose acquisition method for application in an artificial reality environment (e.g., ¶ [0024]: “applied to the field of animation and game production,” … “where the pose may be used in animation and games.”) which is implemented by “a computing device, including a processor and memory, the memory storing at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set being loaded and executed by the processor to implement the pose acquisition method” (See ¶ [0011] ). Thus, in order to implement the method taught by CHENG 2020, it would have been obvious to one of ordinary skill in the art to implemented the method taught by CHENG 2020 using a computing system, as taught by YANG et al. (EP 4 002 198).), cause the computing system to:
obtain, over multiple frames (e.g., the input frames shown in Figures 1-2 and 6.), one or more body tracking signals for one or more body parts[,] of a body of the user (e.g., the image of the body in the input video frames, i.e., video frame pixels corresponding the body) (On p. 10631, see the input video frames (i.e., multiple frames) shown in Figure 1-2 and 6. p. 10632, Figure 2: “Input Frames”. p. 10633, 2nd paragraph: “Given an input video, we first detect and track the persons by any state-of-the-art detector and tracker, such as Mask R-CNN (He et al. 2017) and PoseFlow (Xiu et al. 2018).”) in a real-world environment (p. 10631, 2nd paragraph: “the target person in wild videos.” As can been seen in Figures 1-2 and 6. the input video frames are all taken in a real-world environment.); and
synthesize the full body representation of the user (p. 10633, 2nd paragraph (“Methodology”): “Given an input video, we first detect and track the persons by any state-of-the-art detector and tracker, such as Mask R-CNN (He et al. 2017) and PoseFlow (Xiu et al. 2018). Subsequently, we perform the pose estimation for each person individually.” See 3D human pose estimation “Results” in Figure 1.) by:
estimating scale of the body of the user (3rd paragraph on p. 10633 (i.e., 1st paragraph under “Multi-Scale Features for Pose Estimation”): “Given a series of bounding box for a person in a video,” NOTE: To generate each 2D bounding box (i.e., the minimum sized box that will enclose the body of the person in the video frame), the scale (i.e., height and width) of the subject must be determined. Thus, the body of the user must be detected in the input video image frames (i.e., pixels corresponding the body must be identified), and the size of the body of the user determined in order to generate an appropriately sized bounding box around the body of the user in the image frame. In other words, to determine (or, “estimate”) the size of the size of the bounding box is to determine (or, “estimate”) the size of the body in the video frame.)
based on the estimated scale of the body of the user (3rd paragraph on p. 10633: “Given a series of bounding boxes for a person in a video,”), normalizing the one or more body tracking signals (3rd paragraph on p. 10633: “Given a series of bounding boxes for a person in a video, we first normalize the image within each bounding box to a pre-defined fixed size, e.g., 256 × 256, and then apply High Resolution Networks (HRNet) (Sun et al. 2019) to each normalized image patch to produce K heat maps, each of which indicates the possibility of certain human joint’s location.”); and
synthesizing multiple poses of the body of the user, over the respective multiple frames (e.g., the input video frames in Figures 1-2 and 6.) (See the 3D human pose “Results” in Figure 1 (p. 10631), Figure 2 (p. 106302), and Figure 6 (p. 10637). 1st paragraph on p. 10631 (“Introduction”): “3D human pose estimation from a monocular RGB video. A 3D pose is defined as the 3D coordinates of pre-defined keypoints on humans, such as shoulder, pelvis, wrist, and etc.”), using the one or more normalized body tracking signals (p. 10631, Abstract: “As humans in videos may appear in different scales and have various motion speeds, we apply multi-scale spatial features for 2D joints or keypoints prediction in each individual frame, and multi-stride temporal convolutional networks (TCNs) to estimate 3D joints or keypoints.” p. 10631, 2nd paragraph: “First, we consider multi-scale features both spatially and temporally to deal with persons at various distances with different speeds of motions. We use the High Resolution Network (HRNet) (Sun et al. 2019) which exploits multi-scale spatial features to produce one heat map for each keypoint. Unlike most previous works (Newell, Yang, and Deng 2016; Pavllo et al. 2019) that only use the peaks in the heat maps, we encode these maps into a latent space to incorporate more spatial information. Then, we apply temporal convolutional networks (TCNs) (Pavllo et al. 2019) to these latent features with different strides, e.g., 1, 2, 4, and 8, and concatenate them together for prediction of the 3D poses. Figure 1 shows some examples of our results.” See the network framework in Figure 2. 3rd paragraph on p. 10633: “Given a series of bounding boxes for a person in a video, we first normalize the image within each bounding box to a pre-defined fixed size, e.g., 256 × 256, and then apply High Resolution Networks (HRNet) (Sun et al. 2019) to each normalized image patch to produce K heat maps, each of which indicates the possibility of certain human joint’s location.” 5th paragraph on p. 10633: “Given a sequence of heat map embeddings {rt}, we apply TCN to them. As human motions may be fast or slow, we consider multi-scale features in the temporal domain. As shown in Figure 2, we apply TCN with temporal strides of 1, 2, 3, 5, 7 and concatenate these features for the final pose estimation.” 8th paragraph on p. 10634 (i.e., the 3rd paragraph under “Data Augmentation for Occlusions”): “the trained multi-scale TCN”), by applying a second machine learning model (p. 10631, Abstract: “we apply multi-scale spatial features for 2D joints or keypoints prediction in each individual frame, and multi-stride temporal convolutional networks (TCNs) to estimate 3D joints or keypoints.” p. 10631, 2nd paragraph: “First, we consider multi-scale features both spatially and temporally to deal with persons at various distances with different speeds of motions. We use the High Resolution Network (HRNet) (Sun et al. 2019) which exploits multi-scale spatial features to produce one heat map for each keypoint. Unlike most previous works (Newell, Yang, and Deng 2016; Pavllo et al. 2019) that only use the peaks in the heat maps, we encode these maps into a latent space to incorporate more spatial information. Then, we apply temporal convolutional networks (TCNs) (Pavllo et al. 2019) to these latent features with different strides, e.g., 1, 2, 4, and 8, and concatenate them together for prediction of the 3D poses. Figure 1 shows some examples of our results.” 3rd paragraph on p. 10633: “Given a series of bounding boxes for a person in a video, we first normalize the image within each bounding box to a pre-defined fixed size, e.g., 256 × 256, and then apply High Resolution Networks (HRNet) (Sun et al. 2019) to each normalized image patch to produce K heat maps, each of which indicates the possibility of certain human joint’s location.” 5th paragraph on p. 10633: “Given a sequence of heat map embeddings {rt}, we apply TCN to them.” 8th paragraph on p. 10634 (i.e., the 3rd paragraph under “Data Augmentation for Occlusions”): “the trained multi-scale TCN” 5th paragraph on p. 10633: “apply TCN with temporal strides of 1, 2, 3, 5, 7”) trained on historical motion data, of other bodies of other users, captured by multiple input sensors (6th paragraph on p. 10633 (i.e., 4th paragraph under “Multi-Scale Features for Pose Estimation”): We use both 3D dataset Human3.6M (Ionescu et al. 2014) and 2D dataset Penn Action (Zhang, Zhu, and Derpanis 2013) for training. Human3.6M has multi-view captured videos and 3D ground-truths, while PENN only has 2D ground-truths for visible keypoints.” … “As Human3.6M data set provides videos from multiple views, we expect the 3D estimation results from different views should be the same after rotation alignment.” NOTE: In other words, the Human3.6M dataset is captured using multiple cameras capturing different views of the actors performing motions. 1st paragraph on p. 10635: “Data Sets. Human3.6M (Ionescu et al. 2014) is a large 3D human pose dataset. It has 3.6 million images including eleven actors performing daily-life activities, and seven actors are annotated. The 3D ground-truth is provided by the mocap system, and the intrinsic/extrinsic camera parameters are known. Similar to some existing methods (Hossain and Little 2018; Pavllo et al. 2019; Pavlakos, Zhou, and Daniilidis 2018; Yang et al. 2018), we use subjects 1, 5, 6, 7, 8 for training, and the subjects 9 and 11 for evaluation.”).
CHENG 2020 fails to explicitly disclose: “estimating scale of the body of the user by applying a first machine learning model to the one or more body tracking signals.”
However, whereas CHENG 2020 is not entirely explicit as to, CHENG 2019 teaches: synthesizing a full body representation of a user (p. 723, Title: “Occlusion-Aware Networks for 3D Human Pose Estimation in Video”) for application in an artificial reality environment (p. 723, 1st paragraph: “Estimating 3D human poses from a monocular video is important in many applications, such as animation generation, activity recognition, human-computer interaction, and etc.”), by causing a computing system to:
obtain, over multiple frames (e.g., p. 723, Abstract: “from a monocular video”; p. 725, 1st paragraph of § 3: “Given an input video,” NOTE: The input video(s) comprise multiple frames), one or more body tracking signals for one or more body parts[,] of a body of the user (e.g., the images of the body parts of a body captured in each frame of the input video, i.e., the pixels corresponding to the parts of the body in each of the input video frames) in a real-world environment (e.g., As shown in Figures 1-3, the input video frames are all images of one or body parts of a user/subject captured in a real-world environment. p. 724, 1st paragraph of § 2: “pose estimation for wild videos,” p. 724, 3rd paragraph (i.e., 1st paragraph in § 3): ”Figure 2 shows an overview of our framework. Given an input video, we apply a human detector, such as Mask RCNN [12], to each frame, normalize each detected human bounding box to a fixed size while keeping the width/height ratio, and feed it to our first network, a stacked hourglass network [25], which estimates the 2D keypoints in the form of heatmaps (or confidence maps). Subsequently, our second network (2D TCN) improves the accuracy of the estimated 2D keypoints, and feed them further to our third network (3D TCN) to obtain the final 3D pose.”); and
synthesize the full body representation of the user (p. 724, 1st paragraph in § 3: ” Figure 2 shows an overview of our framework. Given an input video, we apply a human detector, such as Mask RCNN [12], to each frame, normalize each detected human bounding box to a fixed size while keeping the width/height ratio, and feed it to our first network, a stacked hourglass network [25], which estimates the 2D keypoints in the form of heatmaps (or confidence maps). Subsequently, our second network (2D TCN) improves the accuracy of the estimated 2D keypoints, and feed them further to our third network (3D TCN) to obtain the final 3D pose.”) by:
estimating scale of the body of the user (p. 724, 1st paragraph in § 3: ”apply a human detector, such as Mask R-CNN [12], to each frame,” … “each detected human bounding box”; In other words, the Mask R-CNN detects the human body in each frame and estimates a bounding box for the body. Determining the size of the bounding box around the body in each video frame requires determining (i.e., estimating) the location and scale of the body in each video frame.) by applying a first machine learning model (p. 724, 1st paragraph in § 3: “apply a human detector, such as Mask R-CNN [12],”) to the one or more body tracking signals (p. 724, 1st paragraph in § 3: ”input video,” p. 724, 1st paragraph in § 3: ”apply a human detector, such as Mask R-CNN [12], to each frame,” i.e., the detected pixels corresponding to the body in each of the input video frames.) (p. 724, 1st paragraph in § 3: ”Figure 2 shows an overview of our framework. Given an input video, we apply a human detector, such as Mask R-CNN [12], to each frame, normalize each detected human bounding box to a fixed size while keeping the width/height ratio, and feed it to our first network, a stacked hourglass network [25], which estimates the 2D keypoints in the form of heatmaps (or confidence maps). Subsequently, our second network (2D TCN) improves the accuracy of the estimated 2D keypoints, and feed them further to our third network (3D TCN) to obtain the final 3D pose.”);
based on the estimated scale of the body of the user (p. 724, 1st paragraph in § 3: ” each detected human bounding box”), normalizing the one or more body tracking signals (p. 724, 1st paragraph in § 3: ”Figure 2 shows an overview of our framework. Given an input video, we apply a human detector, such as Mask R-CNN [12], to each frame, normalize each detected human bounding box to a fixed size while keeping the width/height ratio, and feed it to our first network, a stacked hourglass network [25], which estimates the 2D keypoints in the form of heatmaps (or confidence maps). Subsequently, our second network (2D TCN) improves the accuracy of the estimated 2D keypoints, and feed them further to our third network (3D TCN) to obtain the final 3D pose.” NOTE: In each frame, a bounding box that fits the detected body is determined (thereby determining the size of the body), and the bounding box is normalized to a fixed size, whereby the size of the image of the body within the bounding box is also normalized to the fixed size).); and
synthesizing multiple poses of the body of the user, over the respective multiple frames, using the one or more normalized body tracking signals, by applying a second machine learning model (p. 724, 1st paragraph in § 3: ”Figure 2 shows an overview of our framework. Given an input video, we apply a human detector, such as Mask R-CNN [12], to each frame, normalize each detected human bounding box to a fixed size while keeping the width/height ratio, and feed it to our first network, a stacked hourglass network [25], which estimates the 2D keypoints in the form of heatmaps (or confidence maps). Subsequently, our second network (2D TCN) improves the accuracy of the estimated 2D keypoints, and feed them further to our third network (3D TCN) to obtain the final 3D pose.”).
Thus, based on the combined teachings of CHENG 2020 and CHENG 2019, in order to detect the bounding boxes for the person in the input video (3rd paragraph on p. 10633 of CHENG 2020), it would have been obvious to one of ordinary skill in the art to have modified the 3D human pose estimation system taught by CHENG 2020 so as to incorporate estimating scale of the body of the user by applying a machine learning model to the one or more body tracking signals, as taught by CHENG 2019.
Regarding claim 12 (depends on claim 11), CHENG 2020 discloses:
wherein the second machine learning model is further trained on one or more masking techniques applied to the historical motion data, the one or more masking techniques accounting for lack of visibility, of one or more other body parts of the other bodies of the other users, by the multiple input sensors (p. 10634, 5th -6th paragraphs (1st-2nd paragraphs under “Data Augmentation for Occlusions”): “To make our approach capable of dealing with different occlusion cases, we perform data augmentation during the training. We use random masking of keypoints to simulate the occluded condition. Three types of occlusion are applied in the training process. The first type is the frame-wise occlusion. Given a sequence of heatmaps produced by the 2D keypoint estimator, we randomly mask several frames by setting their heatmaps to zero, indicating that the whole frame is occluded or has low confidence. Second, the point-wise occlusion is applied by randomly setting certain keypoints’ heatmaps to zero. This simulates the scenario that certain keypoints are occluded. Third, we apply area occlusion by setting a virtual occluder area. The heatmaps of keypoints located within this area are set to zero.”) (6th paragraph on p. 10633 (i.e., 4th paragraph under “Multi-Scale Features for Pose Estimation”): We use both 3D dataset Human3.6M (Ionescu et al. 2014) and 2D dataset Penn Action (Zhang, Zhu, and Derpanis 2013) for training. Human3.6M has multi-view captured videos and 3D ground-truths, while PENN only has 2D ground-truths for visible keypoints.” … “As Human3.6M data set provides videos from multiple views, we expect the 3D estimation results from different views should be the same after rotation alignment.” NOTE: In other words, the Human3.6M dataset is captured using multiple cameras capturing different views of the actors performing motions. 1st paragraph on p. 10635: “Data Sets. Human3.6M (Ionescu et al. 2014) is a large 3D human pose dataset. It has 3.6 million images including eleven actors performing daily-life activities, and seven actors are annotated. The 3D ground-truth is provided by the mocap system, and the intrinsic/extrinsic camera parameters are known. Similar to some existing methods (Hossain and Little 2018; Pavllo et al. 2019; Pavlakos, Zhou, and Daniilidis 2018; Yang et al. 2018), we use subjects 1, 5, 6, 7, 8 for training, and the subjects 9 and 11 for evaluation.”).
Regarding claim 13 (depends on claim 11), CHENG 2020 and CHENG 2019 disclose:
wherein normalizing the body tracking signals (e.g., the pixel locations corresponding the body in the video frame) includes A) one or more positions of the one or more corresponding body parts to be independent of the estimated scale, based on the estimating of the scale of the body (p. 724 of CHNEG 2019, 1st paragraph in § 3: ”Figure 2 shows an overview of our framework. Given an input video, we apply a human detector, such as Mask R-CNN [12], to each frame, normalize each detected human bounding box to a fixed size while keeping the width/height ratio, and feed it to our first network, a stacked hourglass network [25], which estimates the 2D keypoints in the form of heatmaps (or confidence maps). Subsequently, our second network (2D TCN) improves the accuracy of the estimated 2D keypoints, and feed them further to our third network (3D TCN) to obtain the final 3D pose.” NOTE: In each frame, the bounding box is normalized to a fixed size (i.e., size of the image of the body within the bounding box is normalized to a fixed size). Thus, since the normalized bounding box is a fixed size, the normalized size of the image of the body within the bounding box is independent of the estimated scale of the body (i.e., is independent of the inferred size of the bounding box fitting around the body in each frame).) (3rd paragraph on p. 10633 of CHENG 2020: “Given a series of bounding boxes for a person in a video, we first normalize the image within each bounding box to a pre-defined fixed size, e.g., 256 × 256, and then apply High Resolution Networks (HRNet) (Sun et al. 2019) to each normalized image patch to produce K heat maps, each of which indicates the possibility of certain human joint’s location.” NOTE: The “pre-defined fixed size” is independent of the determined scale (height and width) of the bounding box enclosing the detected body of the user in the image frame.) and/or B) one or more trajectories of the one or more body parts based on the estimated scale.
Regarding claim 14 (depends on claim 11), CHENG 2020 discloses the instructions, when executed by the computing system, further cause the computing system to:
based on the estimated scale of the body (1st - 2nd paragraph on p. 10633: “Given an input video, we first detect and track the persons by any state-of-the-art detector and tracker, such as Mask R-CNN (He et al. 2017) and PoseFlow (Xiu et al. 2018). Subsequently, we perform the pose estimation for each person individually. Given a series of bounding boxes for a person in a video, we first normalize the image within each bounding box to a pre-defined fixed size, e.g., 256 × 256, and then apply High Resolution Networks (HRNet) (Sun et al. 2019) to each normalized image patch to produce K heat maps, each of which indicates the possibility of certain human joint’s location.” NOTE: As explained above, detecting the body and determining its bounding box requires determining the size and location of the body in the input video.), normalize a representation of space surrounding the user in the real-world environment (1st - 2nd paragraph on p. 10633: “Given an input video, we first detect and track the persons by any state-of-the-art detector and tracker, such as Mask R-CNN (He et al. 2017) and PoseFlow (Xiu et al. 2018). Subsequently, we perform the pose estimation for each person individually. Given a series of bounding boxes for a person in a video, we first normalize the image within each bounding box to a pre-defined fixed size, e.g., 256 × 256, and then apply High Resolution Networks (HRNet) (Sun et al. 2019) to each normalized image patch to produce K heat maps, each of which indicates the possibility of certain human joint’s location.” NOTE: The image within each bounding box includes not only the detected human body but also a representation of space surrounding the user in the real-world environment. The bounding box is rectangular, but the human body isn’t. Thus, the bounding box must include some of the background in the image within the bounding box, and, as such, normalizing the image within each bounding box also normalizes “a representation of space surrounding the user in the real-world environment”.),
wherein synthesizing the multiple poses of the body of the user is further based on the normalized representation of space surrounding the user in the real-world environment (The multiple poses of the body are synthesized base on the normalized images within the bounding boxes, and the normalized images within the bounding boxes include a normalized representation of space surrounding the user in the real-world environment. Thus, synthesizing the multiple poses of the body of the user is further based on the normalized representation of space surrounding the user in the real-world environment.) .
Regarding claim 17, CHENG 2020 discloses a computing system for synthesizing a full body representation of a user for application in an artificial reality environment (p. 10631, Title: “3D Human Pose Estimation”), the computing system comprising: one or more processors; and one or more memories storing instructions that, when executed by the one or more processors (Although CHENG 2020 is silent as to a computing system for implementing the machine learning framework for synthesizing 3D human poses detected in input video frames, one of ordinary skill in the art would have understood that the method taught by CHENG 2020 is a computer-implemented method performed by a computing system comprising one or more processors and one more memories storing instructions for execution by the one or more processors. For instance, YANG et al. (EP 4 002 198) clearly teaches a related computer-implemented pose acquisition method for application in an artificial reality environment (¶ [0024]: “applied to the field of animation and game production,” … “where the pose may be used in animation and games.”) which is implemented by “a computing device, including a processor and memory, the memory storing at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set being loaded and executed by the processor to implement the pose acquisition method” (¶ [0011] ). Thus, in order to implement the method taught by CHENG 2020, it would have been obvious to one of ordinary skill in the art to implemented the method taught by CHENG 2020 using a computing system, as taught by YANG et al. (EP 4 002 198).), cause the computing system to:
obtain, over multiple frames (e.g., the input frames shown in Figures 1-2 and 6.), one or more body tracking signals for one or more body parts[,] of a body of the user (e.g., the image of the body in the input video frames, i.e., video frame pixels corresponding the body) (On p. 10631, see the input video frames (i.e., multiple frames) shown in Figure 1-2 and 6. p. 10632, Figure 2: “Input Frames”. p. 10633, 2nd paragraph: “Given an input video, we first detect and track the persons by any state-of-the-art detector and tracker, such as Mask R-CNN (He et al. 2017) and PoseFlow (Xiu et al. 2018).”) in a real-world environment (p. 10631, 2nd paragraph: “the target person in wild videos.” As can been seen in Figures 1-2 and 6. the input video frames are all taken in a real-world environment.); and
based on the one or more body tracking signals, synthesize the full body representation of the user (p. 10633, 2nd paragraph (“Methodology”): “Given an input video, we first detect and track the persons by any state-of-the-art detector and tracker, such as Mask R-CNN (He et al. 2017) and PoseFlow (Xiu et al. 2018). Subsequently, we perform the pose estimation for each person individually.” See 3D human pose estimation “Results” in Figure 1.) by:
estimating scale of the body of the user (3rd paragraph on p. 10633 (i.e., 1st paragraph under “Multi-Scale Features for Pose Estimation”): “Given a series of bounding box for a person in a video,” NOTE: To generate each bounding box (i.e., the minimum sized box that will enclose the body of the person in the video frame), the scale (i.e., height and width) of the subject must be determined. Thus, the body of the user must be detected in the input video image frames (i.e., pixels corresponding the body must be identified), and the size of the body of the user determined in order to generate a bounding box around the body of the user in the image frame, i.e., the minimum sized box that encloses the identified pixels corresponding to the body of the user.)
based on the estimating of the scale of the body (3rd paragraph on p. 10633: “Given a series of bounding boxes for a person in a video,”), normalizing one or more positions of the one or more corresponding body parts (3rd paragraph on p. 10633: “Given a series of bounding boxes for a person in a video, we first normalize the image within each bounding box to a pre-defined fixed size, e.g., 256 × 256, and then apply High Resolution Networks (HRNet) (Sun et al. 2019) to each normalized image patch to produce K heat maps, each of which indicates the possibility of certain human joint’s location. The HRNet conducts repeated multi-scale fusions by exchanging the information across the parallel multi-scale subnetworks. Thus, the estimated heat maps incorporate spatial multi-scale features to provide more accurate 2D pose estimations.”), identified from the one or more body tracking signals (e.g., the pixels corresponding the human body in each video frame), to be independent of the estimated scale (3rd paragraph on p. 10633: “Given a series of bounding boxes for a person in a video, we first normalize the image within each bounding box to a pre-defined fixed size, e.g., 256 × 256, and then apply High Resolution Networks (HRNet) (Sun et al. 2019) to each normalized image patch to produce K heat maps, each of which indicates the possibility of certain human joint’s location.” NOTE: The “pre-defined fixed size” is independent of the determined scale (height and width) of the bounding box enclosing the detected body of the user in each video frame.); and
synthesizing multiple poses of the body of the user, over the respective multiple frames (See the 3D human pose “Results” in Figure 1 (p. 10631), Figure 2 (p. 106302), and Figure 6 (p. 10637). 1st paragraph on p. 10631 (“Introduction”): “3D human pose estimation from a monocular RGB video. A 3D pose is defined as the 3D coordinates of pre-defined keypoints on humans, such as shoulder, pelvis, wrist, and etc.”), using the one or more body tracking signals (e.g., the input video frames in Figures 1-2 and 6. 1st paragraph on p. 10631 (“Introduction”): ““3D human pose estimation from a monocular RGB video.”), by applying a neural network (p. 10631, Abstract: “As humans in videos may appear in different scales and have various motion speeds, we apply multi-scale spatial features for 2D joints or keypoints prediction in each individual frame, and multi-stride temporal convolutional networks (TCNs) to estimate 3D joints or keypoints.” p. 10631, 2nd paragraph: “First, we consider multi-scale features both spatially and temporally to deal with persons at various distances with different speeds of motions. We use the High Resolution Network (HRNet) (Sun et al. 2019) which exploits multi-scale spatial features to produce one heat map for each keypoint. Unlike most previous works (Newell, Yang, and Deng 2016; Pavllo et al. 2019) that only use the peaks in the heat maps, we encode these maps into a latent space to incorporate more spatial information. Then, we apply temporal convolutional networks (TCNs) (Pavllo et al. 2019) to these latent features with different strides, e.g., 1, 2, 4, and 8, and concatenate them together for prediction of the 3D poses. Figure 1 shows some examples of our results.” See the network framework in Figure 2. 3rd paragraph on p. 10633: “Given a series of bounding boxes for a person in a video, we first normalize the image within each bounding box to a pre-defined fixed size, e.g., 256 × 256, and then apply High Resolution Networks (HRNet) (Sun et al. 2019) to each normalized image patch to produce K heat maps, each of which indicates the possibility of certain human joint’s location.” 5th paragraph on p. 10633: “Given a sequence of heat map embeddings {rt}, we apply TCN to them.” 8th paragraph on p. 10634 (i.e., the 3rd paragraph under “Data Augmentation for Occlusions”): “the trained multi-scale TCN”) trained on historical motion data captured by multiple input sensors (6th paragraph on p. 10633 (i.e., 4th paragraph under “Multi-Scale Features for Pose Estimation”): We use both 3D dataset Human3.6M (Ionescu et al. 2014) and 2D dataset Penn Action (Zhang, Zhu, and Derpanis 2013) for training. Human3.6M has multi-view captured videos and 3D ground-truths, while PENN only has 2D ground-truths for visible keypoints.” … “As Human3.6M data set provides videos from multiple views, we expect the 3D estimation results from different views should be the same after rotation alignment.” NOTE: In other words, the Human3.6M dataset is captured using multiple cameras capturing different views of the actors performing motions. 1st paragraph on p. 10635: “Data Sets. Human3.6M (Ionescu et al. 2014) is a large 3D human pose dataset. It has 3.6 million images including eleven actors performing daily-life activities, and seven actors are annotated. The 3D ground-truth is provided by the mocap system, and the intrinsic/extrinsic camera parameters are known. Similar to some existing methods (Hossain and Little 2018; Pavllo et al. 2019; Pavlakos, Zhou, and Daniilidis 2018; Yang et al. 2018), we use subjects 1, 5, 6, 7, 8 for training, and the subjects 9 and 11 for evaluation.”).
CHENG 2020 fails to explicitly disclose: “estimating scale of the body of the user by applying a machine learning model to the one or more body tracking signals.”
However, whereas CHENG 2020 is not entirely explicit as to, CHENG 2019 teaches: synthesizing a full body representation of a user (p. 723, Title: “Occlusion-Aware Networks for 3D Human Pose Estimation in Video”) for application in an artificial reality environment (p. 723, 1st paragraph: “Estimating 3D human poses from a monocular video is important in many applications, such as animation generation, activity recognition, human-computer interaction, and etc.”), by causing a computing system to:
obtain, over multiple frames (e.g., p. 723, Abstract: “from a monocular video”; p. 725, 1st paragraph of § 3: “Given an input video,” NOTE: The input video(s) comprise multiple frames), one or more body tracking signals for one or more body parts[,] of a body of the user (e.g., the images of the body parts of a body captured in each frame of the input video, i.e., the pixels corresponding to the parts of the body in each of the input video frames) in a real-world environment (e.g., As shown in Figures 1-3, the input video frames are all images of one or body parts of a user/subject captured in a real-world environment. p. 724, 1st paragraph of § 2: “pose estimation for wild videos,” p. 724, 3rd paragraph (i.e., 1st paragraph in § 3): ”Figure 2 shows an overview of our framework. Given an input video, we apply a human detector, such as Mask RCNN [12], to each frame, normalize each detected human bounding box to a fixed size while keeping the width/height ratio, and feed it to our first network, a stacked hourglass network [25], which estimates the 2D keypoints in the form of heatmaps (or confidence maps). Subsequently, our second network (2D TCN) improves the accuracy of the estimated 2D keypoints, and feed them further to our third network (3D TCN) to obtain the final 3D pose.”); and
based on the one or more body tracking signals, synthesize the full body representation of the user (p. 724, 1st paragraph in § 3: ” Figure 2 shows an overview of our framework. (p. 724, 1st paragraph in § 3: ” Figure 2 shows an overview of our framework. Given an input video, we apply a human detector, such as Mask RCNN [12], to each frame, normalize each detected human bounding box to a fixed size while keeping the width/height ratio, and feed it to our first network, a stacked hourglass network [25], which estimates the 2D keypoints in the form of heatmaps (or confidence maps). Subsequently, our second network (2D TCN) improves the accuracy of the estimated 2D keypoints, and feed them further to our third network (3D TCN) to obtain the final 3D pose.”), we apply a human detector, such as Mask RCNN [12], to each frame, normalize each detected human bounding box to a fixed size while keeping the width/height ratio, and feed it to our first network, a stacked hourglass network [25], which estimates the 2D keypoints in the form of heatmaps (or confidence maps). Subsequently, our second network (2D TCN) improves the accuracy of the estimated 2D keypoints, and feed them further to our third network (3D TCN) to obtain the final 3D pose.”) by:
estimating scale of the body of the user (p. 724, 1st paragraph in § 3: ”apply a human detector, such as Mask R-CNN [12], to each frame,” … “each detected human bounding box”; In other words, the Mask R-CNN detects the human body in each frame and estimates a bounding box for the body. Determining the size of the bounding box around the body in each video frame requires determining (i.e., estimating) the location and scale of the body in each video frame.) by applying a machine learning model (p. 724, 1st paragraph in § 3: “apply a human detector, such as Mask R-CNN [12],”) to the one or more body tracking signals (p. 724, 1st paragraph in § 3: ”input video,” p. 724, 1st paragraph in § 3: ”apply a human detector, such as Mask R-CNN [12], to each frame,” i.e., the detected pixels corresponding to the body in each of the input video frames.) (p. 724, 1st paragraph in § 3: ”Figure 2 shows an overview of our framework. Given an input video, we apply a human detector, such as Mask R-CNN [12], to each frame, normalize each detected human bounding box to a fixed size while keeping the width/height ratio, and feed it to our first network, a stacked hourglass network [25], which estimates the 2D keypoints in the form of heatmaps (or confidence maps). Subsequently, our second network (2D TCN) improves the accuracy of the estimated 2D keypoints, and feed them further to our third network (3D TCN) to obtain the final 3D pose.”);
based on the estimating of the scale of the body (e.g., p. 724, 1st paragraph in § 3: ” each detected human bounding box”; i.e., each human bounding box detected by the applied human detector (e.g., Mask R-CNN). NOTE: To determine the bounding box fitting a detected human body in a video frame, the size of the detected human body must be determined.), normalizing one or more positions of the one or more corresponding body parts, identified from the one or more body tracking signals (e.g., p. 724, 1st paragraph in § 3: ”Figure 2 shows an overview of our framework. Given an input video,) (p. 724, 1st paragraph in § 3: ”Figure 2 shows an overview of our framework. Given an input video, we apply a human detector, such as Mask R-CNN [12], to each frame, normalize each detected human bounding box to a fixed size while keeping the width/height ratio, and feed it to our first network, a stacked hourglass network [25], which estimates the 2D keypoints in the form of heatmaps (or confidence maps). Subsequently, our second network (2D TCN) improves the accuracy of the estimated 2D keypoints, and feed them further to our third network (3D TCN) to obtain the final 3D pose.” NOTE: In each frame, a bounding box that fits the detected body is determined (thereby determining the size of the body), and the bounding box is normalized to a fixed size, whereby the size of the image of the body within the bounding box is also normalized to the fixed size).), to be independent of the estimated scale (In each video frame, the bounding box is normalized to a fixed size, and, as such, the size of the image of the body within the bounding box is also normalized to a fixed size. Thus, since the normalized bounding box is a fixed size, the normalized size of the image of the body and positions of the body parts within the bounding box is independent of the estimated scale of the body (i.e., is independent of the inferred size of the bounding box fitting around the detected body in each video frame). ); and
synthesizing multiple poses of the body of the user, over the respective multiple frames, using the one or more body tracking signals, by applying a neural network (p. 724, 1st paragraph in § 3: ”Figure 2 shows an overview of our framework. Given an input video, we apply a human detector, such as Mask R-CNN [12], to each frame, normalize each detected human bounding box to a fixed size while keeping the width/height ratio, and feed it to our first network, a stacked hourglass network [25], which estimates the 2D keypoints in the form of heatmaps (or confidence maps). Subsequently, our second network (2D TCN) improves the accuracy of the estimated 2D keypoints, and feed them further to our third network (3D TCN) to obtain the final 3D pose.”).
Thus, based on the teachings of CHENG 2020 and CHENG 2019, in order to detect the bounding boxes for the person in the input video (3rd paragraph on p. 10633 of CHENG 2020), it would have been obvious to one of ordinary skill in the art to have modified the 3D human pose estimation system taught by CHENG 2020 so as to incorporate estimating scale of the body of the user by applying a machine learning model to the one or more body tracking signals, as taught by CHENG 2019.
Regarding claim 18 (depends on claim 17), CHENG 2020 discloses:
wherein the neural network includes at least one of a temporal convolutional encoder (p. 10631, 2nd paragraph of “Introduction”: “we apply temporal convolutional networks (TCNs)”), a long short-term memory network, a multi-task multi-layer perception model, or any combination thereof (NOTE: Since alternative limitations are presented, if one limitation has been met, then the remaining alternative limitations need not be given patentable weight.).
Regarding claim 19 (depends on claim 17), CHENG 2020 discloses:
wherein synthesizing the multiple poses includes at least one of D) estimating a global position and orientation of the body of the user in the representation of space, E) estimating one or more bone lengths of the user (1st – 3rd paragraphs on p. 10634: “Among all single frame discriminators, the Kinematic Chain Space (KCS) used in (Wandt and Rosenhahn 2019) is one of the most effective methods. Each bone, defined as the connection between two neighboring human keypoints such as elbow and wrist, is represented as a 3D vector bm, indicating the direction from one keypoint to its neighbor. All such vectors form a 3 × M matrix B, where M is the predefined number of bones for a human structure. They use Ψ=BTB as the features for discriminator, where the diagonal elements in Ψ indicate the square of bone length and other elements represent the weighted angle between two bones as an inner production. Inspired by their spatial KCS, we introduce a Temporal KCS(TKCS) defined as: Φ=BTt+I Bt+i −BTt Bt. where i is the temporal interval between the KCS. The diagonal elements in Φ indicates the bone length changes, and other elements denote the change of angles between two bones. Figure 4 shows an example of two neighboring bones b1 and b2. The spatial KCS measures the lengths of b1 and b2 as well as angles between them, θ12. The temporal KCS measures the bone length changes between two frames with temporal interval i, i.e., differences between b1t and b1t+I as well as b2t and b2t+I and the angle change between neighboring bones, i.e., difference between θ12 t and θ12 t+i . We concatenate the spatial KCS, temporal KCS, and the predicted keypoint coordinates, and then feed them to a TCN to build a discriminator. Such approach not only considers whether a pose is valid in individual frames, but also checks the validity of transitions across frames. We follow the procedure in the standard GAN to train the discriminator, and use it to produce a regularization loss for our predicted poses as Lgen.”), F) estimating one or more poses, of the multiple poses of the body of the user, based on anatomical body model, G) estimating a probability that one or more feet joints, of the body of the user, are in contact with ground in the real-world environment, H) estimating a probability that one or more hips, of the body of the user, are in contact with a physical object or the ground in the real-world environment, or I) any combination thereof (NOTE: Since alternative limitations are presented, if one limitation has been met, then the remaining alternative limitations need not be given patentable weight.).
Regarding claim 20 (depends on claim 17), CHENG 2020 discloses:
wherein the neural network applies at least one of a body pose reconstruction loss (p. 10633, 4th – 5th paragraphs in “Multi-Scale Features for Pose Estimation”: We use both 3D dataset Human3.6M (Ionescu et al. 2014) and 2D dataset Penn Action (Zhang, Zhu, and Derpanis 2013) for training. Human3.6M has multi-view captured videos and 3D ground-truths, while PENN only has 2D ground-truths for visible keypoints. For Human3.6M data, the 3D MSE loss is defined as: (1) L3d =(X −X3D)2, where X is our predicted 3D coordinates for all keypoints, and X3D is the 3D ground truth. As Human3.6M data set provides videos from multiple views, we expect the 3D estimation results from different views should be the same after rotation alignment. So, we define the multi-view loss as: (2) Lmv =(Rv1→v2Xv1 −Xv2)2, where Rv1→v2 is the rotation matrix from viewpoint 1 to viewpoint 2, and is precomputed from the ground-truth camera parameters. The Xv1 and Xv2 are the predicted 3D results in viewpoints 1 and 2. For the 2D dataset, we project the 3D prediction to 2D space assuming orthogonal projection, and the 2D MSE loss is defined as: (3) L2d =(Orth(X)−X2D)2, where Orth(·) is the orthogonal projection operator, and X2D is the 2D ground truth.”), an anatomical representation loss, a feet sliding lose, a bone length loss, contact classification loss for feet, contact classification loss for hip, or any combination thereof (3rd – 5th paragraphs on p.10634: “We concatenate the spatial KCS, temporal KCS, and the predicted keypoint coordinates, and then feed them to a TCN to build a discriminator. Such approach not only considers whether a pose is valid in individual frames, but also checks the validity of transitions across frames. We follow the procedure in the standard GAN to train the discriminator, and use it to produce a regularization loss for our predicted poses as Lgen. In addition, to increase the robustness under different view angles, we introduce a rotational matrix as an augmentation to the generated 3D pose, as shown in the following equation: L′gen = Lgen(RX), where R is a rotational matrix Rotation(α,β,γ), and α,β, γ are rotational angles along x, y, and z axis, respectively. As the rotational angles along x and z angles should be smaller compared with rotations along y for normal human poses, in our experiments, β is randomly sampled from [−π,π] while α and γ are sampled from [−0.2π,0.2π]. The overall loss function for our training is defined as L =L3d + w1Lmv + w2L2d + w3L′gen, where w1,w2,w3 are set to 0.5, 0.1, 0.01, respectively, and are fixed in all our experiments.”) (NOTE: Since alternative limitations are presented, if one limitation has been met, then the remaining alternative limitations need not be given patentable weight.).
Claims 3-5 are rejected under 35 U.S.C. 103 as being unpatentable over CHENG et al. (Cheng Y, Yang B, Wang B, Tan RT. “3d human pose estimation using spatio-temporal networks with explicit occlusion training.” In Proceedings of the AAAI Conference on Artificial Intelligence 2020 Apr 3; Vol. 34, No. 07, pp. 10631-10638.) in view of CHENG et al. (Cheng Y, Yang B, Wang B, Yan W, Tan RT. Occlusion-aware networks for 3d human pose estimation in video. In Proceedings of the IEEE/CVF international conference on computer vision 2019; pp. 723-732.), further in view of OHASHI (US 2021/0350551).
Regarding claim 3 (depends on claim 2), whereas CHENG 2020 and CHENG 2019 are not explicit as to, OHASHI teaches:
wherein the at least one sensor is included in at least one of an artificial reality head-mounted display (¶ [0400]: “the tracker 12a, for example, may be a head mounted display (HMD).” ) and/or a device worn by the user that is external to the artificial reality head-mounted display (¶ [0050]: “As depicted in FIG. 1, the entertainment system 10 according to the present embodiment includes a plurality of trackers 12 (trackers 12a to 12e in the example of FIG. 1), the entertainment apparatus 14, a relay apparatus 16, a display 18, and a camera microphone unit 20.” ¶ [0051]: “The trackers 12 according to the present embodiment are devices that, for example, track positions and directions of the trackers 12. Each of the trackers 12 may be configured herein with, for example, various kinds of sensors such as a camera, an inertial measurement unit (IMU), a geomagnetic sensor (azimuth sensor), an acceleration sensor, a motion sensor, and a GPS (Global Positioning System) module. In addition, each of the trackers 12 may identify the position and the direction of the tracker 12 on the basis of sensing data that is measurement results by the sensors provided in the tracker 12.” ¶ [0052]: “Alternatively, for example, each of the trackers 12 may identify the position and the direction of the tracker 12 on the basis of an image captured by a camera 20a included in the camera microphone unit 20, to be described later and containing an image of the tracker 12.” ¶ [0053]: “In the present embodiment, the trackers 12a, 12b, 12c, 12d, and 12e are attached to a head, a left hand, a right hand, a left foot, and a right foot of a user, respectively. As depicted in FIG. 1, herein, the trackers 12b and 12c may be grasped by user's hands. In the present embodiment, the positions and the directions identified by the trackers 12a, 12b, 12c, 12d, and 12e correspond to positions and directions of the head, the left hand, the right hand, the left foot, and the right foot of the user, respectively. In this way, in the present embodiment, the plurality of trackers 12 identify the positions and the directions of a plurality of regions included in a user's body.” ¶ [0400]: “Furthermore, the tracker 12a, for example, may be a head mounted display (HMD). In this case, a video picture in response to a result of various types of processing such as game processing in response to the positions or the directions of the plurality of regions included in the user may be displayed on, for example, a display section of the HMD.”) (NOTE: Since alternative limitations are presented, if one limitation has been met, then the remaining alternative limitations need not be given patentable weight.”).
Thus, in order to obtain a more versatile system for synthesizing a full body representation of movement of a user having the cumulative features and/or functionalities taught by CHENG 2020, CHENG 2019 and OHASHI, it would have been obvious to one of ordinary skill in the art to have modified the pose tracking and synthesizing system taught by the combination of CHENG 2020 and CHENG 2019 so as to incorporate at least one sensor device worn by the user that is external to an artificial reality head-mounted display, as taught by OHASHI.
Regarding claim 4 (depends on claim 1), whereas CHENG 2020 and CHENG 2019 are not explicit as to, OHASHI teaches:
wherein each of the one or more body tracking signals includes a position and orientation of a corresponding body part, of the one or more body parts (e.g., ¶ [0053]: “the positions and the directions identified by the trackers 12a, 12b, 12c, 12d, and 12e correspond to positions and directions of the head, the left hand, the right hand, the left foot, and the right foot of the user, respectively.”), at a frame of the multiple frames (e.g., ¶ [0110]: “sequence containing t frames”; ¶ [0119]: “a series of t pieces of sensing data output from each of the trackers 12”) (¶ [0050]: “As depicted in FIG. 1, the entertainment system 10 according to the present embodiment includes a plurality of trackers 12 (trackers 12a to 12e in the example of FIG. 1), the entertainment apparatus 14, a relay apparatus 16, a display 18, and a camera microphone unit 20.” ¶ [0051]: “The trackers 12 according to the present embodiment are devices that, for example, track positions and directions of the trackers 12. Each of the trackers 12 may be configured herein with, for example, various kinds of sensors such as a camera, an inertial measurement unit (IMU), a geomagnetic sensor (azimuth sensor), an acceleration sensor, a motion sensor, and a GPS (Global Positioning System) module. In addition, each of the trackers 12 may identify the position and the direction of the tracker 12 on the basis of sensing data that is measurement results by the sensors provided in the tracker 12.” ¶ [0052]: “Alternatively, for example, each of the trackers 12 may identify the position and the direction of the tracker 12 on the basis of an image captured by a camera 20a included in the camera microphone unit 20, to be described later and containing an image of the tracker 12.” ¶ [0053]: “In the present embodiment, the trackers 12a, 12b, 12c, 12d, and 12e are attached to a head, a left hand, a right hand, a left foot, and a right foot of a user, respectively. As depicted in FIG. 1, herein, the trackers 12b and 12c may be grasped by user's hands. In the present embodiment, the positions and the directions identified by the trackers 12a, 12b, 12c, 12d, and 12e correspond to positions and directions of the head, the left hand, the right hand, the left foot, and the right foot of the user, respectively. In this way, in the present embodiment, the plurality of trackers 12 identify the positions and the directions of a plurality of regions included in a user's body.” ¶ [0064]: “In the present embodiment, at a time of, for example, executing a game program by the entertainment apparatus 14, various types of processing such as game processing in response to the positions or directions of the plurality of regions included in the user's body in a skeleton model 40 depicted in FIG. 3 is executed. A video picture in response to a result of the processing is then displayed on, for example, the display 18.” ¶ [0101]: “FIG. 5 is a diagram depicting an example of the estimation of the direction of the chest node 42f using the learned machine learning model.” ¶ [0102]: “As described above, in the present embodiment, it is assumed, for example, that the position and the direction of each of the trackers 12a to 12e are identified at the predetermined sampling rate. It is also assumed that data indicating the position and the direction of each tracker 12 is transmitted to the entertainment apparatus 14 in response to identification of the position and the direction of the tracker 12.” ¶ [0103]: “It is further assumed that region data indicating the position, the posture, or the motion about any of the regions of the body is generated on the basis of the data indicating the position and the direction of each tracker 12 transmitted in this way. In the present embodiment, the region data is repeatedly generated in this way.” ¶ [0110] Learning may be executed herein in advance by data indicating a direction of the lumbar made to correspond to a combination of the direction of the head, the angular speed of the left hand, and the angular speed of the right hand. In this case, the supervisory data contained in the learning data described above may be, for example, the data indicating the direction of the lumbar. In addition, the supervisory data may be generated in a similar manner as that described above on the basis of, for example, the sensing data output from each of the trackers 12 attached to the head, the left hand, the right hand, and the lumbar of the user making various motions, and an image sequence containing t frames of the user making various motions captured from the external camera.” ¶ [0112]: “Furthermore, learning may be executed in advance by data indicating an angular speed of the chest made to correspond to a combination of the direction of the head, the angular speed of the left hand, and the angular speed of the right hand. In this case, the supervisory data contained in the learning data described above may be, for example, data indicating the direction of the chest. In addition, the supervisory data may be generated in a similar manner as that described above on the basis of, for example, the sensing data output from each of the trackers 12 attached to the head, the left hand, the right hand, and the chest of the user making various motions, and the image sequence containing t frames of the user making various motions captured from the external camera. An angular speed of the chest node 42f may be then estimated. In this case, the data D4 depicted in FIGS. 4 and 5 corresponds to an estimation result of the angular speed of the chest node 42f.” ¶ [0119]: “On the basis of, for example, a series of t pieces of sensing data output from each of the trackers 12 attached to the left hand and the left wrist when the user having the trackers 12 attached thereto makes various motions, a series of t pieces of region data made to correspond to the series of t respective pieces of sensing data may be generated herein. In addition, on the basis of the t-th sensing data supervisory data indicating the direction of the left wrist that is made to correspond to the t-th sensing data may be generated. Learning data containing the series of t pieces of region data and the supervisory data may be then generated.” ¶ [0121]: “In another alternative, on the basis of, for example, an image sequence containing t frames of the user making various motions captured from an external camera, learning data containing a series of t pieces of region data made to correspond to the t frames, respectively and supervisory data indicating the direction of the left wrist that is made to correspond to the t-th frame may be generated.” ¶ [0141]: “Learning may be executed herein in advance by data indicating the angular speed of the left wrist made to correspond to a combination of the direction of the left hand, the angular speed of the left hand, and the position or speed of the left hand. In this case, the supervisory data contained in the learning data described above may be, for example, the data indicating the angular speed of the left wrist. In addition, the supervisory data may be generated in a similar manner as that described above on the basis of, for example, the sensing data output from each of the trackers 12 attached to the left hand and the left wrist of the user making various motions, and an image sequence containing t frames of the user making various motions captured from the external camera.”).
Thus, in order to obtain a more versatile system for synthesizing a full body representation of movement of a user having the cumulative features and/or functionalities taught by CHENG 2020, CHENG 2019 and OHASHI, it would have been obvious to one of ordinary skill in the art to have modified the pose tracking and synthesizing system taught by the combination of CHENG 2020 and CHENG 2019 so that each of the one or more body tracking signals includes a position and orientation of a corresponding body part, of the one or more body parts, at a frame of the multiple frames, as taught by OHASHI.
Regarding claim 5 (depends on claim 4), CHENG 2020 and CHENG 2019 disclose:
wherein each of the one or more body tracking signals further includes a confidence value for the position and orientation of the corresponding body part (3rd paragraph on p. 10633 of CHENG 2020: “Given a series of bounding boxes for a person in a video, we first normalize the image within each bounding box to a pre-defined fixed size, e.g., 256 × 256, and then apply High Resolution Networks (HRNet) (Sun et al. 2019) to each normalized image patch to produce K heat maps, each of which indicates the possibility of certain human joint’s location.” 1st paragraph of § 3 on p. 725 of CHENG 2019: “Figure 2 shows an overview of our framework. Given an input video, we apply a human detector, such as Mask R CNN [12], to each frame, normalize each detected human bounding box to a fixed size while keeping the width/height ratio, and feed it to our first network, a stacked hourglass network [25], which estimates the 2D keypoints in the form of heatmaps (or confidence maps). Subsequently, our second network (2D TCN) improves the accuracy of the estimated 2D keypoints, and feed them further to our third network (3D TCN) to obtain the final 3D pose. Our framework is end-to-end, for both training and testing.” ), the confidence value being generated based on visibility of the corresponding body part by one or more sensors capturing the respective body tracking signal (3rd paragraph on p. 10633 of CHENG 2020: “Given a series of bounding boxes for a person in a video,” 1st paragraph of § 3 on p. 725 of CHENG 2019: “Given an input video, we apply a human detector, such as Mask R CNN [12], to each frame,” 1st paragraph of § 3.1 on p. 725 of CHENG 2019: “Given a bounding box containing a person, our first network outputs a set of heatmaps, expressed as { Mi }, where i ∈ [1,K] and K is the number of predefined keypoints. The network processes the bounding box frame-by-frame, individually,” ).
Claims 15-16 are rejected under 35 U.S.C. 103 as being unpatentable over CHENG et al. (Cheng Y, Yang B, Wang B, Tan RT. “3d human pose estimation using spatio-temporal networks with explicit occlusion training.” In Proceedings of the AAAI Conference on Artificial Intelligence 2020 Apr 3 (Vol. 34, No. 07, pp. 10631-10638); “CHENG 2020”) in view of YANG et al. (EP 4 002 198), further in view of CHENG et al. (Cheng Y, Yang B, Wang B, Yan W, Tan RT. Occlusion-aware networks for 3d human pose estimation in video. In Proceedings of the IEEE/CVF international conference on computer vision 2019 (pp. 723-732); “CHENG 2019”), and further yet in view of OHASHI (US 2021/0350551).
Regarding claim 15 (depends on claim 11), whereas CHENG 2020 and CHENG 2019 are not explicit as to, OHASHI teaches:
wherein each of the one or more body tracking signals includes a position and orientation of a corresponding body part, of the one or more body parts (e.g., ¶ [0053]: “the positions and the directions identified by the trackers 12a, 12b, 12c, 12d, and 12e correspond to positions and directions of the head, the left hand, the right hand, the left foot, and the right foot of the user, respectively.”), at a frame of the multiple frames (e.g., ¶ [0110]: “sequence containing t frames”; ¶ [0119]: “a series of t pieces of sensing data output from each of the trackers 12”) (¶ [0050]: “As depicted in FIG. 1, the entertainment system 10 according to the present embodiment includes a plurality of trackers 12 (trackers 12a to 12e in the example of FIG. 1), the entertainment apparatus 14, a relay apparatus 16, a display 18, and a camera microphone unit 20.” ¶ [0051]: “The trackers 12 according to the present embodiment are devices that, for example, track positions and directions of the trackers 12. Each of the trackers 12 may be configured herein with, for example, various kinds of sensors such as a camera, an inertial measurement unit (IMU), a geomagnetic sensor (azimuth sensor), an acceleration sensor, a motion sensor, and a GPS (Global Positioning System) module. In addition, each of the trackers 12 may identify the position and the direction of the tracker 12 on the basis of sensing data that is measurement results by the sensors provided in the tracker 12.” ¶ [0052]: “Alternatively, for example, each of the trackers 12 may identify the position and the direction of the tracker 12 on the basis of an image captured by a camera 20a included in the camera microphone unit 20, to be described later and containing an image of the tracker 12.” ¶ [0053]: “In the present embodiment, the trackers 12a, 12b, 12c, 12d, and 12e are attached to a head, a left hand, a right hand, a left foot, and a right foot of a user, respectively. As depicted in FIG. 1, herein, the trackers 12b and 12c may be grasped by user's hands. In the present embodiment, the positions and the directions identified by the trackers 12a, 12b, 12c, 12d, and 12e correspond to positions and directions of the head, the left hand, the right hand, the left foot, and the right foot of the user, respectively. In this way, in the present embodiment, the plurality of trackers 12 identify the positions and the directions of a plurality of regions included in a user's body.” ¶ [0064]: “In the present embodiment, at a time of, for example, executing a game program by the entertainment apparatus 14, various types of processing such as game processing in response to the positions or directions of the plurality of regions included in the user's body in a skeleton model 40 depicted in FIG. 3 is executed. A video picture in response to a result of the processing is then displayed on, for example, the display 18.” ¶ [0101]: “FIG. 5 is a diagram depicting an example of the estimation of the direction of the chest node 42f using the learned machine learning model.” ¶ [0102]: “As described above, in the present embodiment, it is assumed, for example, that the position and the direction of each of the trackers 12a to 12e are identified at the predetermined sampling rate. It is also assumed that data indicating the position and the direction of each tracker 12 is transmitted to the entertainment apparatus 14 in response to identification of the position and the direction of the tracker 12.” ¶ [0103]: “It is further assumed that region data indicating the position, the posture, or the motion about any of the regions of the body is generated on the basis of the data indicating the position and the direction of each tracker 12 transmitted in this way. In the present embodiment, the region data is repeatedly generated in this way.” ¶ [0110] Learning may be executed herein in advance by data indicating a direction of the lumbar made to correspond to a combination of the direction of the head, the angular speed of the left hand, and the angular speed of the right hand. In this case, the supervisory data contained in the learning data described above may be, for example, the data indicating the direction of the lumbar. In addition, the supervisory data may be generated in a similar manner as that described above on the basis of, for example, the sensing data output from each of the trackers 12 attached to the head, the left hand, the right hand, and the lumbar of the user making various motions, and an image sequence containing t frames of the user making various motions captured from the external camera.” ¶ [0112]: “Furthermore, learning may be executed in advance by data indicating an angular speed of the chest made to correspond to a combination of the direction of the head, the angular speed of the left hand, and the angular speed of the right hand. In this case, the supervisory data contained in the learning data described above may be, for example, data indicating the direction of the chest. In addition, the supervisory data may be generated in a similar manner as that described above on the basis of, for example, the sensing data output from each of the trackers 12 attached to the head, the left hand, the right hand, and the chest of the user making various motions, and the image sequence containing t frames of the user making various motions captured from the external camera. An angular speed of the chest node 42f may be then estimated. In this case, the data D4 depicted in FIGS. 4 and 5 corresponds to an estimation result of the angular speed of the chest node 42f.” ¶ [0119]: “On the basis of, for example, a series of t pieces of sensing data output from each of the trackers 12 attached to the left hand and the left wrist when the user having the trackers 12 attached thereto makes various motions, a series of t pieces of region data made to correspond to the series of t respective pieces of sensing data may be generated herein. In addition, on the basis of the t-th sensing data supervisory data indicating the direction of the left wrist that is made to correspond to the t-th sensing data may be generated. Learning data containing the series of t pieces of region data and the supervisory data may be then generated.” ¶ [0121]: “In another alternative, on the basis of, for example, an image sequence containing t frames of the user making various motions captured from an external camera, learning data containing a series of t pieces of region data made to correspond to the t frames, respectively and supervisory data indicating the direction of the left wrist that is made to correspond to the t-th frame may be generated.” ¶ [0141]: “Learning may be executed herein in advance by data indicating the angular speed of the left wrist made to correspond to a combination of the direction of the left hand, the angular speed of the left hand, and the position or speed of the left hand. In this case, the supervisory data contained in the learning data described above may be, for example, the data indicating the angular speed of the left wrist. In addition, the supervisory data may be generated in a similar manner as that described above on the basis of, for example, the sensing data output from each of the trackers 12 attached to the left hand and the left wrist of the user making various motions, and an image sequence containing t frames of the user making various motions captured from the external camera.”).
Thus, in order to obtain a more versatile system for synthesizing a full body representation of movement of a user having the cumulative features and/or functionalities taught by CHENG 2020, CHENG 2019 and OHASHI, it would have been obvious to one of ordinary skill in the art to have modified the pose tracking and synthesizing system taught by the combination of CHENG 2020 and CHENG 2019 so that each of the one or more body tracking signals includes a position and orientation of a corresponding body part, of the one or more body parts, at a frame of the multiple frames, as taught by OHASHI.
Regarding claim 16 (depends on claim 15), CHENG 2020 and CHENG 2019 disclose:
wherein each of the one or more body tracking signals further includes a confidence value for the position and orientation of the corresponding body part (3rd paragraph on p. 10633 of CHENG 2020: “Given a series of bounding boxes for a person in a video, we first normalize the image within each bounding box to a pre-defined fixed size, e.g., 256 × 256, and then apply High Resolution Networks (HRNet) (Sun et al. 2019) to each normalized image patch to produce K heat maps, each of which indicates the possibility of certain human joint’s location.” 1st paragraph of § 3 on p. 725 of CHENG 2019: “Figure 2 shows an overview of our framework. Given an input video, we apply a human detector, such as Mask R CNN [12], to each frame, normalize each detected human bounding box to a fixed size while keeping the width/height ratio, and feed it to our first network, a stacked hourglass network [25], which estimates the 2D keypoints in the form of heatmaps (or confidence maps). Subsequently, our second network (2D TCN) improves the accuracy of the estimated 2D keypoints, and feed them further to our third network (3D TCN) to obtain the final 3D pose. Our framework is end-to-end, for both training and testing.” ), the confidence value being generated based on visibility of the corresponding body part by one or more sensors capturing the respective body tracking signal (3rd paragraph on p. 10633 of CHENG 2020: “Given a series of bounding boxes for a person in a video,” 1st paragraph of § 3 on p. 725 of CHENG 2019: “Given an input video, we apply a human detector, such as Mask R CNN [12], to each frame,” 1st paragraph of § 3.1 on p. 725 of CHENG 2019: “Given a bounding box containing a person, our first network outputs a set of heatmaps, expressed as { Mi }, where i ∈ [1,K] and K is the number of predefined keypoints. The network processes the bounding box frame-by-frame, individually,” ).
Conclusion
At present, it is not apparent to the examiner which part of the application could serve as a basis for new and allowable claims. However, should the applicant nevertheless regard some particular matter as patentable, the examiner encourages applicant to appropriately amend the claims to include such matter and to indicate in the REMARKS the difference(s) between the prior art and the claimed invention as well as the significance thereof.
Furthermore, should applicant decide to amend the claims, examiner respectfully requests that the applicant please indicate in the REMARKS from which page(s), line(s) or claim(s) of the originally filed application that any amendments are derived. See MPEP § 2163(II)(A) (There is a strong presumption that an adequate written description of the claimed invention is present in the specification as filed, Wertheim, 541 F.2d at 262, 191 USPQ at 96; however, with respect to newly added or amended claims, applicant should show support in the original disclosure for the new or amended claims.).
A shortened statutory period for reply to this action is set to expire THREE MONTHS from the mailing date of this action. Extensions of time may be available under the provisions of 37 CFR 1.136(a). In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action. Failure to reply within the set or extended period for reply will, by statute, cause the application to become ABANDONED (35 USC § 133).
Relevant Prior Art
The following prior art, although not relied upon, is made of record since it is considered pertinent to applicant's disclosure:
SHIRATORI et al. (US 20120327194) discloses using body-mounted cameras to accurately reconstruct the motion of a subject. Outward-looking cameras are attached to the limbs of the subject, and the joint angles and root pose that define the subject's configuration are estimated through a non-linear optimization, which can incorporate image matching error and temporal continuity of motion. Instrumentation of the environment is not required, allowing for motion capture over extended areas and in outdoor settings.
KOIKE et al. (US 20210035326) discloses a motion measurement system including a wide-angle camera configured to capture in the periphery of an image at least a part of a body of a subject when the wide-angle camera is mounted on the body, a feature point extractor configured to extract feature points from the image, and a 3D pose estimator configured to estimate 3D pose data of the subject by using the feature points.
Contact Information
Any inquiry concerning this communication or earlier communications from the examiner should be directed to VINCENT PEREN who can be reached by telephone at (571) 270-7781, or via email at vincent.peren@uspto.gov. The examiner can normally be reached on Monday-Friday from 10:00 A.M. to 6:00 P.M.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, KING POON, can be reached at telephone number (571)272-7440. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from Patent Center. Status information for published applications may be obtained from Patent Center. Status information for unpublished applications is available through Patent Center for authorized users only. Should you have questions about access to Patent Center, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) Form at https://www.uspto.gov/patents/uspto-automated- interview-request-air-form.
/VINCENT PEREN/
Examiner, Art Unit 2617
/KING Y POON/Supervisory Patent Examiner, Art Unit 2617
Prosecution Insights