Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
The Amendment filed 7/02/2026 has been entered. Claims 1, 4-15 and 17-20 remain pending in the application.
Response to Arguments
Applicant argues the content of amended claim 1 is not disclosed by Pontoon, Jiang, or a combination of the two. Specifically, the applicant states that
“1. The Height-Based Further-Skeleton Features require more than merely resizing an avatar or uniformly scaling an avatar according to a user height. The amended independent claims require determining a skeleton length of a limb of the human body from a limb image, determining a further skeleton length of the human body based on that limb skeleton length, obtaining a height of the human body, and determining the further skeleton length based on both the height of the human body and the skeleton length of the limb of the human body.
Ponton fails to disclose this claimed data relationship.
2. Ponton uses physical HTC Vive trackers and controllers to directly obtain positions and dimensions of body portions, and then resizes avatar bone chains using tracker-derived measurements and/or general scaling. The cited portions of Ponton fail to teach first determining a human further skeleton length from an image-derived limb skeleton length and a human body height, and then adjusting a corresponding further skeleton length of a virtual character based on the determined human further skeleton length.
3. To the extent Ponton discusses a height of the user, Ponton uses height in a different manner, such as for uniform avatar scaling or for positioning certain avatar portions in the tracker-based fitting framework. Such use of height is not the same as determining a further skeleton length of the human body based on both the height of the human body and the skeleton length of the limb of the human body as recited in the amended independent claims.
Therefore, Ponton fails to disclose or render obvious at least the Height-Based Further- Skeleton Features as recited in amended claim 1.”
Regarding the content of the numbered paragraph 1 of the applicant’s argument, Ponton does not merely accept a user height to scale an avatar. Ponton, as evidenced by Ponton 3.3.2 cited in the previous rejection of claim 2, teaches that the user’s arm length can be obtained to resize the length of arm bones.
In paragraph 2 it is stated that Ponton fails “to teach first determining a human skeleton length from an image-derived limb skeleton length”, this has already been conceded in the original office action as Ponton is never claimed to teach an image-derived limb length. Jiang is combined with Ponton specifically to remedy this deficiency, as evidenced by Jiang 3 and Fig. 1 cited in the previous rejection of claim 1. The skeleton length determined from this first image, which the combination of Jiang and Ponton has already been shown to teach, is referred to as the “human further skeleton length”. Applicant also argues that Ponton does not teach determining a “further skeleton length” based on the “human further skeleton length”.
As Jian combined with Ponton teaches the “human further skeleton length”, Ponton 3.3.2 cited in the previous rejection of claim 2 teaches determining a “further skeleton length” based on the “human further skeleton length”. Here, the “human further skeleton length” is the length of the arms, based on the length of the arms Ponton teaches that the shoulder width can be changed.
PNG
media_image1.png
498
560
media_image1.png
Greyscale
Fig. 7 of Ponton is provided for clarification, typically “shoulder width” is not a direct measure of bone length, however in Ponton’s skeleton the shoulder can be seen to be the bone connecting the end of the arm to the spine. Thus, Ponton teaches adjusting a further skeleton length, the length of a shoulder bone, based on a “human further skeleton length”, the arm lengths.
It is conceded that the use of height by Ponton in regards to determining a “further skeleton length” is not as written in amended claim 1. Specifically, wherein determining the further skeleton length of the human body based on the skeleton length of the limb of the human body comprises: determining the further skeleton length of the human body based on the height of the human body and the skeleton length of the limb of the human body. Ponton teaches determining skeleton lengths based on height, and determining a “further skeleton length” based on another skeleton length, but does not teach that the user’s height and length of another skeleton are used to determine the length of a skeleton.
Applicant’s arguments, see pages 9-13 of remarks/arguments, filed 7/02/2026, with respect to the rejection(s) of claim(s) 1, 20, and 15 under 35 USC 103 have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of Fujita (US 20230076859 A1).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim 1, 4, 6, 7, 10-12, 15, 17-20, are rejected under 35 U.S.C. 103 as being unpatentable over Ponton (Fitted avatars: automatic skeleton adjustment for self-avatars in virtual reality) in view of Jiang (Skeletor: Skeletal Transformers for Robust Body-Pose Estimation) and further in view of Fujita (US 20230076859 A1).
Regarding claims 15, 1, 20,
Ponton teaches:
An electronic device, comprising: a processor and a memory, wherein the memory is configured to store a computer program, (Ponton 4.2 “Participants are first briefed about the steps of the experiment and the total duration … they put on the three VIVE trackers, the HTC Vive PRO wireless HMD and take one controller in each hand.” Note: Ponton teaches the use of an HTC Vive HMD (head mounted display), this HMD contains its own processor and memory.) and the processor is configured to call and run the computer program stored in the memory, to perform acts of displaying a virtual character, the acts comprising: obtaining limb data of a human body; (Ponton 1 “In this paper, we present a method to fit an avatar to the dimensions of a user rapidly. We use the HTC VIVE Head Mounted Display (HMD), with two VIVE wand controllers (one for each hand), and three additional VIVE trackers, attached with straps to the user’s waist and feet. The user needs to follow a simple sequence of movements to capture the exact location of relevant joints and lengths of limbs. Once the avatar has been adjusted, we can animate it in real time” Note: Ponton teaches that its process starts with recording limb information, particularly length, via VIVE trackers. Despite their name VIVE trackers are not merely isolated tracking devices instead employing a method where a sensor outputs a sweep of lasers to get info on a room. When the lasers hit trackers, which are designed to recognize when this occurs, their position information in the room can be obtained.) determining a skeleton length of a corresponding limb of the human body based on the limb data; (Ponton 3 “the user follows a short sequence of exercises to automatically extract the location and dimensions of their joints and limbs” 3.3.2 “To resize the avatar bones based on the user’s limb measurements, we focus on five bone chains: the two arm chains, the two leg chains and the spine chain. Our method stretches or shrinks the length of each of those chains while keeping the ratios between the bones’ length for each chain. Leg chain: The avatar’s lower and upper leg bones are resized according to the participant’s leg height” Note: Ponton teaches that its recording system measures limb and body dimensions. The limb heights that are determined, such as leg height, are the claims skeleton lengths of a corresponding limb as it is this value which is used to determine the skeleton length.) and displaying a virtual character, a skeleton length of a limb of the virtual character being determined based on the skeleton length of the limb of the human body;(Ponton 3.3 “In order to have a virtual avatar matching the user’s body as close as possible, we follow a two-pass process: (1) uniform avatar scaling, and (2) skeleton tuning … A uniform scaling is applied based on the user’s height to preserve the avatar’s proportions … To resize the avatar bones based on the user’s limb measurements, we focus on five bone chains: the two arm chains, the two leg chains and the spine chain” Note: Ponton teaches that the user’s bone measurements are used to determine the length of a virtual avatar/character’s skeleton.)wherein the the acts further comprise: determining a further skeleton length of the human body based on the skeleton length of the limb of the human body; (
PNG
media_image1.png
498
560
media_image1.png
Greyscale
Ponton 3.3.2 “Arm chain: Similarly, the lower and upper arm bones are resized to match the participant’s arm length. Spine chain: First we adjust the shoulder width and then the spine length. The avatar’s shoulder width is adjusted by scaling along the axis defined by the vector connecting both shoulder joints: v = p(JRshoulder) − p(JLshoulder) / ‖ p(JRshoulder)−p(JLshoulder) ‖ According to this axis, shoulder joints are positioned so that they are at a distance d apart. The distance d is given by the distance between the shoulders’ centers of rotation from Step 2: d =distance(p(CRshoulder), p(CLshoulder))” Note: The claim states a “further skeleton length” is determined from the limb length of the human body meaning a bone length that is not the limb is determined using a limb length. In Fig. 7 we see the two arm bones whose total length is the user’s arm length connected to a shoulder bone length that is in turn connected to an upper chest bone linking the chain of bones to the spine. As the width of the shoulder bones is adjusted to be accurate to the user based first on the length and position of the arms Ponton teaches determining a further skeleton length using the skeleton length of a limb.) and adjusting a further skeleton length of the virtual character based on the further skeleton length of the human body. (Ponton 3.2.2 “ The distance d is given by the distance between the shoulders’ centers of rotation from Step 2: d =distance(p(CRshoulder), p(CLshoulder)) Then we need to resize the vertical dimension of the bone chain comprising the avatar’s spine, chest and upper chest bones, to match the avatar’s shoulder height with the height of the shoulder centers of rotation h(CLshoulder) and h(CRshoulder).” Note: Previously it was established that Ponton teaches a further skeleton length based on the skeleton length of a limb of the body, specifically shoulder bones had their length adjusted and scaled based on arm bone length. The provided citation is a continuation of the last which specifies that this further skeleton length of the shoulders is used in adjusting another further skeleton length, specifically in adjusting the length of the spine, chest and upper chest bones.)
While Ponton teaches that limb info can be obtained via a sensor that sweeps a room with lasers as it leverages HTC Vive trackers it does not teach that an imaging system with a camera is used to obtain limb recordings. This is taught in Jiang which teaches obtaining a limb image of a human body; (
PNG
media_image2.png
590
788
media_image2.png
Greyscale
Jiang 3 “In this section we introduce the architecture of Skeletor, a novel deep learning network that learns about the shape and motion of 3D skeletons from video in an unsupervised manner. Given an image sequence V = (f1,f2,··· ,fT), our goal is to estimate an accurate 3D pose of the skeleton in each frame. We define 3D pose as P = (P1,P2,··· ,PT), where Pi = (Ji 1,Ji 2,··· ,Ji N) is the 3D skeleton in the ith frame and Ji k = (xi k,yi k,zi k) is the 3D position of joint k. An overview of our approach is seen in Figure 1. First, 2D Pose Estimation extracts a noisy and/or partially occluded 2D skeleton in image coordinates. This is then lifted into 3D using either regression or inverse kinematics (IK) to produce a preliminary 3D Pose Estimation, where errors in the 2D skeleton can effect the 3D estimate. Skeletor then provides 3D Pose Refinement of the 3D skeleton.” Note: Jiang teaches that from images of a human body containing limbs, as seen in Fig. 1, a skeleton with accurate lengths and joint positions is created to fit the corresponding limbs in the limb image.) determining a skeleton length of a corresponding limb of the human body based on the limb image; (Jiang 3, Fig. 1, cited above, teach the determining of a skeleton length for limbs of a human body based on a corresponding image of them.)
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Ponton with Jiang where the determination of a virtual character’s skeleton lengths uses limb measurements obtained from a camera’s images.
There are several reasons that would motivate one to do so, one is offering an easier more accessible method to use the invention. Performing actions with specialized laser sweeping and trackers requires more effort and specialized equipment as opposed to simply taking images with a depth camera, a piece of technology found in many modern phones.
While Ponton teaches determining skeleton length based on height, and determining skeleton length based on another skeleton length, it does not teach that a skeleton length can be determined based on both the height and skeleton length of another limb. This is taught by Fujita wherein determining the further skeleton length of the human body based on the skeleton length of the limb of the human body comprises: determining the further skeleton length of the human body based on the height of the human body and the skeleton length of the limb of the human body.(Fujita ¶81 “First, the calculation unit 23 performs a ratio calculation process of calculating a length ratio of a skeleton portion (a leg K2, an arm K3, a torso K4) corresponding to the target portion to a first skeleton K1 which is a representative skeleton forming a part of a body height LH (step SP1). Next, the calculation unit 23 performs an estimated length calculation process of calculating an estimated length of the first skeleton K1 (step SP2).” ¶84 “Next, the calculation unit 23 calculates an estimated length of the leg K2 by multiplying the calculated estimated length of the first skeleton K1 by the ratio (=K2/K1) of the length of the leg K2 with respect to the length of the first skeleton K1, and calculates an estimated length of the leg K5 indicating the entire leg by adding, to the calculated estimated length of the leg K2, the second specified value LB equivalent to a predetermined leg correction value (step SP3).” ¶113 “The calculation unit 23 calculates the length of the leg K2 equivalent to the skeleton of the leg based on the ratios among the skeletons K1 to K4 including the first skeleton K1 and the above-described first length, and calculates, as the length of the leg K5 indicating the entire leg, the length obtained by adding, to the calculation result, the second specified value LB equivalent to the leg correction value.” ¶57 “second specified value LB equivalent to a length between the second position PL which is a foot position of the skeleton and a sole position. ”Note: Fujita teaches that the entire leg length K5 can be calculated via the height and the length of another limb. Using the skeleton length of the foot, which is specifically the foot position and the sole position, and the height the entire leg length can be calculated. The input height LH is divided with a ratio to obtain K2, K2 is then added to the length of the foot, LB, to obtain the entire leg length K5.)
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Ponton with Fujita where determining a further skeleton length of the human body based on the skeleton length of the limb of the human body comprises determining the further skeleton length based on the height and skeleton length of a limb of the body.
There are several reasons that would motivate one to do so, limb lengths are often proportional to an individual’s height and other limbs of the body. Rather than having to directly calculate every limb length from direct measurements the system could be made more efficient by only taking an initial measurement of a limb and height and then using those measurements to determine the lengths of other limbs.
Regarding claims 17, 4,
Ponton teaches:
The method of claim 1, wherein determining the skeleton length of the corresponding limb of the human body based on the limb data comprises: identifying joint points of the limb of the human body based on the limb data; (Ponton 3.2 “Our method expects a humanoid avatar with at least three joints per limb, two in the spine and one for the neck. In order to correctly map the user’s body to the avatar, we need to infer the location of some of the user’s body joints … Since several body joints are centers of rotation when moving a limb, we can compute their location by calculating the center of a point cloud forming a sphere, which is captured by the tracker located at the end of the limb. We use a least squares solution for estimating the average center of rotation and the radius of the sphere. This method is used for the left and right arm and for the neck, as follows: Shoulders: position of the center of rotation p(Cshoulder) of the point cloud obtained with the controllers tip positions p(Twrist) when moving it with the straight arms.” Note: Ponton teaches that from its data on limbs, in the given example the arms are used, their motion and dimensions used to calculate a joint point for the limb, in this case the shoulder joint.) determining three-dimensional (3D) coordinates of the joint points of the limb of the human body based on the limb data; (
PNG
media_image3.png
428
1420
media_image3.png
Greyscale
Ponton 3.2 “Arm length: radius of the fitted sphere obtained from the hand controllers, which corresponds to the distance: L arm = distance(p(Cshoulder), p(Twrist)” Note: As seen in Fig. 5 a 3D point cloud is leveraged to determine the shoulder joint position in 3D space, or in other words a 3D coordinate for the joint is found seen in the blue and yellow 3D points in Fig. 5. We know that similar 3D coordinate positions for the arms are also known as seen in Fig. 5 as well from the red and green circles and dots displaying real 3D coordinate positions of the arm as it moves through space.) and determining the skeleton length of the limb of the human body based on the 3D coordinates of the joint points of the limb of the human body. (Ponton 3.2 and Fig. 5, cited above, teaches the equation for determining arm length. A 3D position for the joint is subtracted from 3D coordinate data of the arm. Specifically, Fig. 5 teaches the arms are moved in circles in 3D space to obtain an accurate radius measurement which is subtracted from the joint position. Note: It has been established that Ponton teaches that 3D joint and limb coordinate/point data is leveraged to find limb bone length. Ponton, as seen in Fig. 5 and 3.2 uses the wrist to shoulder joint to determine arm length so that it accurately measures the bone length, excluding the hand by measuring only to the wrist.)
While Ponton teaches obtaining limb data from the user via its previously detailed HTC Vive trackers which leverage a laser scanning across a room to register with key tracker points it does not teach that limb data can be obtained in the form of an image. This is taught in Jiang which teaches identifying joint points of the limb of the human body based on the limb image; determining three-dimensional (3D) coordinates of the joint points of the limb of the human body based on the limb image;
(
PNG
media_image2.png
590
788
media_image2.png
Greyscale
Jiang 3 “In this section we introduce the architecture of Skeletor, a novel deep learning network that learns about the shape and motion of 3D skeletons from video in an unsupervised manner. Given an image sequence V = (f1,f2,··· ,fT), our goal is to estimate an accurate 3D pose of the skeleton in each frame. We define 3D pose as P = (P1,P2,··· ,PT), where Pi = (Ji 1,Ji 2,··· ,Ji N) is the 3D skeleton in the ith frame and Ji k = (xi k,yi k,zi k) is the 3D position of joint k. An overview of our approach is seen in Figure 1. First, 2D Pose Estimation extracts a noisy and/or partially occluded 2D skeleton in image coordinates. This is then lifted into 3D using either regression or inverse kinematics (IK) to produce a preliminary 3D Pose Estimation, where errors in the 2D skeleton can effect the 3D estimate. Skeletor then provides 3D Pose Refinement of the 3D skeleton.” Note: Jiang teaches that from an initial 2D skeleton estimation of bone lengths and joint positions an accurate 3D skeleton is obtained with 3D joint positions and limb lengths all derived from input images of the user’s body with limbs visible.)
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Ponton with Jiang where the determining of skeleton length from 3D limb and joint coordinate data originates from limb images.
There are several reasons that would motivate one to do so, one is having an easier less resource intensive method of data collection. Ponton’s HTC Vive trackers require a laser sweeper and a number of trackers which require more space, more set up, and more specialized equipment as opposed to obtaining limb data from images taken with a camera.
Regarding claim 6,
Ponton teaches:
The method of claim 4,
Ponton leverages its previously detailed HTC Vive tracker method for obtaining limb data and joint points, the use of a human body key point identification network to obtain the joint points of limbs is instead taught in Jiang wherein identifying the joint points of the limb of the human body based on the limb image comprises: inputting the limb image into a human body key point identification network to obtain the joint points of the limb of the human body. (Fig. 1. Cited above, details a skeleton created from a limb image of a human showing that key joint points, skeleton lengths,
4.1 “We first run OpenPose [6] on the video footage to get the 2D skeletons and their confidence values” Note: Jiang teaches the use of a human body key point identification network to obtain joint points when it uses OpenPose, a known human body key point detection system that leverages a neural network. As seen in Fig. 1, key bone lengths and joints are determined by the initial 2D skeleton created by OpenPose.)
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Ponton with Jiang where the identification of key joint points for limbs is handled by a human body key point identification network.
There are several reasons that would motivate one to do so, one of which is gaining the benefits of the fast and accurate computer vision results neural networks are able to output, another is to obtain the joint positions without needing a series of trackers and scanners instead only using a camera.
Regarding claim 7,
Ponton teaches:
The method of claim 3, wherein obtaining the height of the human body comprises: receiving the height of the human body input by a user or detecting the height of the human body. (Ponton 3.3.1, 3.3.2, cited in claim 3 above, teaches that user height is used and detected via the HMD’s height, which is at eye level providing a sufficient measurement to use as user height.)
Regarding claims 18, 10,
Ponton teaches:
The device of claim 15, further comprising: obtaining posture information of the human body; and displaying the virtual character comprises: displaying the virtual character based on the posture information of the human body. (
PNG
media_image4.png
348
990
media_image4.png
Greyscale
Ponton 1 “Once the avatar has been adjusted, we can animate it in real time using only those six trackers.” 4.2.2 “To induce embodiment we ask participants to perform a sequence of steps, the first one involves moving freely in front of a mirror to observe how the virtual avatar mimics their movements” Note: Ponton teaches that the avatar mimics the user’s motion in real time using 6 trackers, seen in Fig. 3. The numerous trackers at different core points like hands, feet, waist/root, and head provide an abundance of position data to allow the virtual character to accurately represent the live posture and pose of the user.)
Regarding claims 19, 11,
Ponton teaches:
The device of claim 15, further comprising: in response to detecting that a user starts an application, displaying an adjustment option of the skeleton length of the virtual character; (
PNG
media_image5.png
548
1092
media_image5.png
Greyscale
Note: Fig. 2 teaches the ability for a user to choose an adjustment option that will adjust the skeleton of a virtual character to their body. This is taught in the Fig. 2 description which states the user, in order to set up the framework, chooses an avatar themselves then stands in the initial T-pose to start the automatic limb and joint data extraction process to adjust the skeleton. Thus, Ponton offers the user a choice to adjust a skeleton length of a virtual character by selecting a virtual character and initiating the tracking process.) in response to a determination instruction for the adjustment option, enabling an adjustment function of the skeleton length of the virtual character; (Ponton 3 “Our framework provides a pipeline to create fitted avatars to the specific dimensions of the user. To achieve this, the user needs three HTC trackers attached to the feet and back, an HMD, and a controller in each hand … Next, the user follows a short sequence of exercises to automatically extract the location and dimensions of their joints and limbs” Note: Ponton teaches that to create fitted avatars, a process initiated by the user as described previously, the user follows the adjustment function to adjust the model. In Ponton the adjustment function involves a sequence of exercise which will allow the location and dimensions of limbs to be extracted to later be used in fitting the virtual character.) and displaying prompt information for prompting the user to extend both hands forward and kicking forward with a left leg or a right leg. (
PNG
media_image6.png
292
1052
media_image6.png
Greyscale
Ponton 3 “Once the avatar has been adjusted to the user’s dimensions, the final step consists of walking inside a virtual body that matches the user’s dimensions, making a T-pose inside the avatar and pressing a trigger to compute the exact offset between trackers and joints” Note: Ponton teaches that both hands should be extended forwards (and in all other rotational directions) synchronously as part of the exercise that automatically extracts the location and dimensions of their joints and limbs. Once this is done the process can be finalized by walking to move the left and right foot trackers to get a final end result of joint and limb data. While the claim states that the left or right leg should be “kicked” forward Ponton’s walking motion of moving both legs forward is analogous.)
Regarding claim 12,
The method of claim 10, wherein the obtaining posture information of the human body comprises: determining the posture information of the human body through inertial data measured by an inertial sensor (IMU) worn by the human body. (Ponton 1, 4.2.2, Fig. 1, cited in claim 10 teach how Ponton leverages trackers to determine accurate live posture information of the user. Specifically, Ponton teaches the use of HTC Vive trackers, all HTC Vive trackers perform their tracking in part by leveraging an IMU. Thus, Ponton teaches the use of IMUs worn by the user to determine posture information.)
Claims 5, 8, 9, are rejected under 35 U.S.C. 103 as being unpatentable over Ponton (Fitted avatars: automatic skeleton adjustment for self-avatars in virtual reality) in view of Jiang (Skeletor: Skeletal Transformers for Robust Body-Pose Estimation), in view of Fujuta (US20230076859A1) and further in view of Fei (WO 2018187171 A1).
Regarding claim 5,
Ponton teaches:
The method of claim 4, wherein a head-mounted device uses a binocular camera to capture the limb image; and determining the 3D coordinates of the joint points of the limb of the human body based on the limb image comprises:
Ponton does not however teach leveraging the binocular parallax principle, the fact that binocular vision allows for depth perception, to determine 3D coordinates of joints and limbs. This is taught in Fei which teaches determining the 3D coordinates of the joint points of the limb of the human body based on the limb image and a binocular parallax principle. (Fei ¶40 “The method 500 includes a number of steps, some of which may be optional. In some embodiments, the method 500 can be implemented by an apparatus for hand tracking. The apparatus may comprise the head mounted display 103, the hand tracking camera module 102 attached to the head mounted display 103 …and comprising at least one of a pair of stereo cameras or a depth camera, a processor, and a non-transitory computer-readable storage medium storing instructions that, when executed by the processor, cause the processor to perform the method 500.” ¶41 “The method 500 may comprise: (block 501 ) causing the hand tracking camera module to capture an image of at least one physical hand of a user wearing the head mounted display, (block 502) obtaining the image in a current frame and least one of determined skeleton joints … executing a 3D Hand Skeleton Joints Recognition algorithm to determine 3D skeleton joints of the physical hand in 26 degrees-of-freedom in the current frame, causing the head mounted display to render a virtual hand resembling the physical hand in a virtual environment for viewing by the user based at least on the determined 3D skeleton joints,” Note: Fei teaches that a 3D coordinates for joint points of a limb, in this case the arm’s hand, are determined based on a limb image and the binocular parallax principle. Fei teaches using the binocular parallax principle when it states it can use either a depth camera or a pair of stereo cameras, the stereo cameras working together to provide depth information is an example of the binocular parallax principle. From the images taken and depth info obtained from the stereo cameras 3D skeleton joints are determined for the body part in the image.)
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Ponton with Fei where the determination of 3D coordinate data for a skeleton comes from a limb image and leveraging the binocular parallax principle.
There are several reasons that would motivate one to do so, one of which is to gain the benefits of an efficient low-cost way to obtain 3D coordinates without specialized equipment or measuring, instead only relying on the depth data the binocular parallax principle provides from binocular imaging to obtain 3D information.
Regarding claim 8,
Ponton teaches:
The method of claim 1, wherein the virtual character is displayed in a virtual reality scene of the head-mounted device (
PNG
media_image7.png
394
1054
media_image7.png
Greyscale
Note: The virtual reality scene the user interacts with via the HMD is seen above in Fig. 9, and the real relative locations of the HMD and trackers/controllers are shown as well to demonstrate the virtual character is properly fit to the user via the skeleton adjustments.)
While Ponton teaches the use of a head-mounted device, also called a head-mounted display (HMD), it does not teach that its HMD can take limb images of the user, this is taught in Fei which teaches wherein obtaining the limb image of the human body comprises: collecting the limb image through a camera of a head-mounted device; wherein the virtual character is displayed in an extended reality scene of the head-mounted device. (Fei ¶40 “The apparatus may comprise the head mounted display 103, the hand tracking camera module 102 attached to the head mounted display 103 …and comprising at least one of a pair of stereo cameras or a depth camera, a processor, and a non-transitory computer-readable storage medium storing instructions that, when executed by the processor, cause the processor to perform the method 500.” ¶41 ““The method 500 may comprise: (block 501 ) causing the hand tracking camera module to capture an image of at least one physical hand of a user wearing the head mounted display, (block 502) obtaining the image in a current frame and least one of determined skeleton joints … executing a 3D Hand Skeleton Joints Recognition algorithm to determine 3D skeleton joints of the physical hand in 26 degrees-of-freedom in the current frame, causing the head mounted display to render a virtual hand resembling the physical hand in a virtual environment for viewing by the user based at least on the determined 3D skeleton joints,” Note: Fei teaches that images from the HMD are used to make 3D skeleton joints of the limb in the image, and that those 3D skeleton joints are used by the HMD to render a virtual character’s hand displayed in a virtual environment. The application’s specifications states “XR refers to a collective term of a variety of technologies such as virtual reality (VR), augmented reality (AR), and mixed reality (MR) by combining reality and virtuality by a computer to create a virtual environment for human-computer interaction. By fusing the visual interaction techniques of the three, a seamless transition “immersion” between the virtual world and the real world is brought to the experiencer. XR devices are typically worn on a user’s head, and thus an XR device is also referred to as a head-mounted device.”, as Fei teaches the use of a HMD and incorporation external physical reality into its virtual reality, Fei’s virtual environment meets the definition of the claim’s extended reality scene.)
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Ponton with Fei where the HMD used to display a virtual character images the user during use to display a virtual character of the user in an extended reality environment.
There are several reasons that would motivate one to do so, an abundance of limb image data from a variety of views and limb positions will allow for more accurate renditions of the limbs. This benefit can easily be accomplished via imaging with the HMD while the user is using it instead of having to take separate images manually.
Regarding claim 9,
Ponton teaches:
The method of claim 8,
Ponton does not teach limb imaging, doing so is taught in Fei which teaches wherein the limb image is a limb image of a human body wearing the head-mounted device. (Fei ¶40, 41, cited in the previous claim, details that the HMD photographs users while wearing it, thus all photos used are photos of a human body that is wearing the HMD.)
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Ponton with Fei where the data collected to determine body information is limb images of a user with an HMD.
There are several reasons that would motivate one to do so, when imaging is done of the user wearing the HMD it can be obtained live, while the user is interacting with the environment displayed in the HMD, in the process avoiding unnecessary steps of removing the HMD then taking images.
Claim 13 is rejected under 35 U.S.C. 103 as being unpatentable over Ponton (Fitted avatars: automatic skeleton adjustment for self-avatars in virtual reality) in view of Jiang (Skeletor: Skeletal Transformers for Robust Body-Pose Estimation) in view of Fujuta (US20230076859A1) and further in view of Ahuja (ControllerPose: Inside-Out Body Capture with VR Controller Cameras).
Regarding claim 13,
Ponton teaches:
The method of claim 1
While Ponton teaches individually determining skeleton lengths for the arms, legs, and other bones and joints that comprise the upper and lower body it does not teach that this is done through imaging. Upper and lower body imaging to determine specific skeleton lengths is taught in Ahuja wherein the limb image comprises an upper limb image and a lower limb image, the upper limb image comprises a left upper limb image and a right upper limb image, and the lower limb image comprises a left lower limb image and a right lower limb image; (Ahuja Abstract “By virtue of the hands operating in front of the user during many VR interactions, our controller-borne cameras can capture a superior view of the body for digitization. Our pipeline composites multiple camera views together, performs 3D body pose estimation, uses this data to control a rigged human model with inverse kinematics, and exposes the resulting user avatar to end user applications.” Fig. 4
PNG
media_image8.png
694
866
media_image8.png
Greyscale
Note: As seen in Fig. 4, Ahuja’s cameras located on the user’s controllers easily captures upper and lower body images. As seen the upper/lower limb images both image the left and right upper/lower limbs respectively, that is to say both arms and both legs are captured.) and the determining a skeleton length of a corresponding limb of the human body based on the limb image comprises: determining a skeleton length of an upper limb of the human body based on the upper limb image; and determining a skeleton length of a lower
PNG
media_image9.png
522
1460
media_image9.png
Greyscale
limb of the human body based on the left lower limb image and the right lower limb image. (
Note: As seen on the left side of Fig. 6 our previously shown images in Fig. 4 are input to a neural network which outputs a 3D skeleton with lengths that is used to make a rigged avatar fit to the user. It is shown that the skeleton lengths of all upper and lower body limbs are determined as the skeleton shown has both arms and both legs and is in a 3D graph which shows that the length and other distance information is determined for the skeleton. Fig. 6 provides an external reference photo allowing us to see that its skeleton distances and positions are accurate to the pose the user is in when imaged.)
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Ponton with Ahuja where the determining of individual arm and leg skeleton lengths from data obtained of the upper and lower body are images of the upper and lower body.
There are several reasons that would motivate one to do so, Ponton enjoys the benefits of accurate limb skeleton lengths via its system of collecting data on individual portions of the body but does so at the cost of using multiple trackers and other equipment. One could obtain similarly accurate skeleton lengths by evaluating specific portions of the body without the cost and set up required from specialized equipment by simply using a camera to image different parts of the body.
Claim 14 is rejected under 35 U.S.C. 103 as being unpatentable over Ponton (Fitted avatars: automatic skeleton adjustment for self-avatars in virtual reality) in view of Jiang (Skeletor: Skeletal Transformers for Robust Body-Pose Estimation) in view of Fujuta (US20230076859A1) and further in view of Martinez (OpenPose: Whole-Body Pose Estimation)
Regarding claim 14,
Ponton teaches:
The method of claim 4, wherein the limb comprises an upper limb and a lower limb, and joint points of the upper limb comprise: a shoulder node, an elbow node, and a wrist node; and joint points of the lower limb comprise: a hip node, a knee node, and an ankle node. (
PNG
media_image1.png
498
560
media_image1.png
Greyscale
Note: The claim defines nodes as joint points of limbs, Fig. 7 present’s Ponton’s present joints for its various bones. It can be seen that shoulder, wrist, and hip joints/nodes are clearly shown by Ponton with identical naming labels. Knee and elbow nodes are also taught, referred to by Ponton as lower arm/leg joints respectively. In this limb differentiation we can also see that Ponton clearly separates limbs into upper/lower arms and legs. An ankle node is also taught by Ponton as its foot joint can be seen to be positioned exactly at the ankle, allowing for the foot to move at that position.)
While Ponton details many key positions for its upper and lower body portions it does not denote a finger node or a toe node, doing so is taught in Martinez which teaches points of the upper limb comprise: a finger node; (Martinez 5 Conclusion “We evaluate our method on multiple keypoint detection benchmarks and compare it to the state-of-the-art (our previous work, OpenPose), considerably outperforming it in both training and testing speed as well as slightly improving its accuracy. We qualitatively show in Fig. 8a that our face and hand detectors generalize better to in-the-wild images, benefiting from their indirect exposure to the immense body datasets.”
PNG
media_image10.png
900
1358
media_image10.png
Greyscale
Note: Martinez teaches a key point detector usable on hands to automatically determine key points. As seen in (a) and (b) above key points for the hand include ones for fingers.) and points of the lower limb comprise: a toe node; (
PNG
media_image11.png
502
794
media_image11.png
Greyscale
Note: As seen in Fig. 4’s (a) image specific toe key points labelled 0, 1, 3, and 4 are identified from an image.)
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine Ponton with Martinez where the several key joint points identified from limb images consist of finger and toe nodes specifically, alongside the other previously mentioned nodes taught by Ponton.
There are several reasons that would motivate one to do so, one is to gain a higher level of detail and accuracy in the movements of a virtual character’s hands and feet by leveraging key points on the user that correspond to them. Furthermore, in claim 4 it was shown why one would seek to modify Ponton with Jiang where Jiang implements specific joint positions/nodes from limb images. The method Jiang uses to obtain these joint points/nodes is specifically stated to be the same key point identification method in Martinez, OpenPose.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ALAN GREGORY HAKALA whose telephone number is (571)272-7863. The examiner can normally be reached 8:00am-5:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, King Poon can be reached at (571) 270-0728. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ALAN GREGORY HAKALA/Examiner, Art Unit 2617
/KING Y POON/Supervisory Patent Examiner, Art Unit 2617
Prosecution Insights