DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted is considered by the examiner.
Response to Arguments
Applicant’s arguments with respect to claim(s) 1-8, 10-15, 21-26 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-3, 5, 7-8, 11-13, 15, 21-22, 24 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lefaudeux et al. (US Publication Number 2022/0083125 A1, Lefaudeux) in view of Balakrishnan et al. (US Publication Number 2015/0106052 A1, hereinafter “Balakrishnan”).
(1) regarding claim 1:
As shown in figs. 1 & 5, Lefaudeux disclosed a computer-implemented method (para. [0041], note that a computer implemented method for training a generator is disclosed. Also see para. [0017], the artificial reality system 100 may comprise a headset 104, a controller 106, and a computing system 108), comprising:
generating an upper-body avatar for a user of a headset (para. [0041], note that the goal is to have the Generator 405 learn how to generate, based on an upper body pose 205, upper body poses 215);
causing a the lower-body posture to be applied a lower-body model based on the predicted pace of motions (para. [0044], note that Generator 405 may receive one or more contextual information associated with the artificial reality system 100. In particular embodiments, the lower body pose 215 is generated by further processing the contextual information using the machine-learning model); and
merging the lower-body model with the upper-body avatar to form a full-body avatar for the user of the headset (para. [0046], note that the system may generate, based on the upper body pose 205 and the lower body pose 215, a full body pose 425. FIG. 6 illustrates generating a full body pose from a generated upper body pose and a generated lower body pose).
Lefaudeux disclosed most of the subject matter as described as above except for specifically teaching obtaining motion data from a wrist-worn device of the user of the headset; and predicting a pace of motions for a lower-body portion of the user of the headset based, at least in part, on a separation between peaks in a time scale of motion data from the wrist-worn device.
However, Balakrishnan disclosed obtaining motion data from a wrist-worn device of the user of the headset (para. [0046], note that user 124 may be associated with (e.g., possess, carry, wear, and/or interact with) any number of devices, such as portable device 112, shoe-mounted device 126, wrist-worn device 128 and/or a sensing location, such as sensing location 130, which may comprise a physical device or a location that is used to collect information); and predicting a pace of motions for a lower-body portion of the user of the headset based, at least in part, on a separation between peaks in a time scale of motion data from the wrist-worn device (para. [0128], note that specific peak (or peaks) within the data (such as for example, data obtained within a buffer and/or data obtained during a time frame) may be utilized. In one embodiment, "bounce peaks," "arm swing peaks," and/or other peaks may be identified. For example, many users "bounce" upon landing their feet when running. This bounce may provide frequency peaks within the data. Other peaks (and/or valleys) may be present within the sensor data. For example, many users often swing their arms in a predictable manner during running and/or walking to provide "arm swing peaks", Also see fig. 12A).
At the time of filing for the invention, it would have been obvious to a person of ordinary skilled in the art to obtaining motion data from a wrist-worn device of the user of the headset; and predicting a pace of motions for a lower-body portion of the user of the headset based, at least in part, on a separation between peaks in a time scale of motion data from the wrist-worn device. The suggestion/motivation for doing so would have been in order to identify and quantify, using the one or more mathematical models, the motion data as linear travel motion associated with the user running or walking (abs.). Therefore, it would have been obvious to combine Lefaudeux with Balakrishnan to obtain the invention as specified in claim 1.
(2) regarding claim 2:
Lefaudeux further disclosed the computer-implemented method of claim 1, wherein generating an upper-body avatar comprises tracking a face gesture of the user with a sensor mounted on the headset (para. [0028], note that computing system 108 may determine upper body pose 200 utilizing one or more poses corresponding to a plurality of predetermined body parts of user 102, for example and not by way of limitation, a joint pose of one or more joints comprising upper body pose 205 (e.g., a head pose 210 or a wrist pose 220). Note that the head pose 210 associated with user 102 may be determined based on sensor data associated with headset 104 worn by user 102, para. [0029]).
(3) regarding claim 3:
Lefaudeux disclosed most of the subject matter as described as above except for specifically teaching wherein the lower-body posture is further based on an image of at least one of an arm, a leg, a hand or a foot of the user, captured with a camera mounted on the headset.
However, Balakrishnan disclosed wherein tracking a lower-body posture of the user comprises collecting an image of at least one of an arm, a leg, a hand or a foot of the user with a camera mounted on the headset (para [0132], note that in certain embodiments, arm swing data may be utilized to determine the dual foot step frequency (see axis 1104). For example, if a wrist-worn device is configured to measure arm swings, such data may be interpreted as the single foot frequency).
At the time of filing for the invention, it would have been obvious to a person of ordinary skilled in the art to teach wherein the lower-body posture is further based on an image of at least one of an arm, a leg, a hand or a foot of the user, captured with a camera mounted on the headset. The suggestion/motivation for doing so would have been in order to efficiently capture human motion with a sparse set of tracking data for real-time prediction of the lower-body pose only from the tracking signals of the upper-body joints (abs.). Therefore, it would have been obvious to combine Lefaudeux with Balakrishnan to obtain the invention as specified in claim 3.
(4) regarding claim 5:
Lefaudeux further disclosed the computer-implemented method of claim 1, wherein the lower-body posture of the user is further based on an acceleration signal from an inertial measurement unit in the headset (para. [0018], note that one or more controllers of the headset may be equipped with at least one inertial measurement units (IMUs) to collect and send sensor data to the computing system. Also see fig. 8, step 830, para. [0070]).
(5) regarding claim 7:
Lefaudeux further disclosed the computer-implemented method of claim 1, wherein the lower-body posture of the user is further based on identifying a sitting posture or a standing posture of the user (para. [0040], note that the Generator 405 may be configured to receive an upper body pose 205 and generate a temporal sequence of lower body poses 215, for example, a sequence of lower body poses of a user standing, sitting, or walking).
(6) regarding claim 8:
Lefaudeux further disclosed the computer-implemented method of claim 1, wherein the lower-body posture of the user is further based on a physical constraint for lower-body motion (para. [0045], note that the Generator 405 may receive and utilize one or more of these physical constraints to generate more realistic lower body poses, or a more realistic temporal sequence of lower body poses).
(7) regarding claim 22:
Lefaudeux further disclosed the non-transitory computer-readable storage medium of claim 21, wherein the lower-body posture of the user is further based on a determined facial expression of the user (para. [0020], note that one or more cameras may be positioned to capture one or more images associated with various perspectives, for example and not by way of limitation, one or more cameras associated with headset 104 that face downward (e.g. towards the feet of user 102 while standing). Also look, para. [0028], note that computing system 108 may determine upper body pose 200 utilizing one or more poses corresponding to a plurality of predetermined body parts of user 102, for example and not by way of limitation, a joint pose of one or more joints comprising upper body pose 205 (e.g., a head pose 210 or a wrist pose 220). Note that the head pose 210 associated with user 102 may be determined based on sensor data associated with headset 104 worn by user 102, para. [0029.).
(8) regarding claim 24:
Lefaudeux disclosed most of the subject matter as described as above except for specifically teaching wherein the lower-body posture of the user is further based on a determined heart rate signal for the user.
However, Balakrishnan disclosed wherein the lower-body posture of the user is further based on a determined heart rate signal for the user (para. [0098], note that the number of samples and/or data points stored in a buffer may be dependent upon the sensor type from which the data is received. For example, data received from an accelerometer may occupy more storage space than data received from a heart rate monitor).
At the time of filing for the invention, it would have been obvious to a person of ordinary skilled in the art to teach wherein the lower-body posture of the user is further based on a determined heart rate signal for the user. The suggestion/motivation for doing so would have been in order to efficiently capture human motion with a sparse set of tracking data for real-time prediction of the lower-body pose only from the tracking signals of the upper-body joints (abs.). Therefore, it would have been obvious to combine Lefaudeux with Balakrishnan to obtain the invention as specified in claim 24.
The proposed rejection of claims 1-3, 5 renders obvious the steps of the system claims 11-13,15 (see fig. 1, the system) and the non-transitory computer-readable claim 21 because these steps occur in the operation of the proposed rejection as discussed above. Thus, the arguments similar to that presented above for claims 1-5 are equally applicable to claims 11-13, 15 and 21.
Claim(s) 4, 6, 10, 14, 23, 25-26 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lefaudeux and Balakrishnan, and further in view of Yang et al. (NPL, “LoBSTr: Real-time Lower-body Pose Prediction from Sparse Upper-body Tracking Signals”, 2021).
(1) regarding claim 4:
Lefaudeux disclosed most of the subject matter as described as above except for specifically teaching wherein the lower-body posture of the user is further based on an image of the lower-body posture of the user, the image received from a mobile device.
However, Yang disclosed wherein tracking a lower-body posture of the user comprises receiving, from a mobile device, an image of the lower-body posture of the user, the image received from a mobile device (page 5, section 3.4, para. [0001], note that HTC Vive Pro set is used as a tracking device: HMD, two hand-held controllers, and a tracker on the pelvis).
At the time of filing for the invention, it would have been obvious to a person of ordinary skilled in the art to teach wherein tracking a lower-body posture of the user comprises receiving, from a mobile device, an image of the lower-body posture of the user. The suggestion/motivation for doing so would have been in order to efficiently capture human motion with a sparse set of tracking data for real-time prediction of the lower-body pose only from the tracking signals of the upper-body joints (abs.). Therefore, it would have been obvious to combine Lefaudeux and Balakrishnan with Yang to obtain the invention as specified in claim 4.
(2) regarding claim 6:
Lefaudeux disclosed most of the subject matter as described as above except for specifically teaching wherein the lower-body posture of the user is further based on an identified running motion or a walking motion from a frequency in an acceleration signal from an inertial measurement unit in the headset.
However, Yang disclosed wherein tracking a lower-body posture of the user comprises identifying a running motion or a walking motion from a frequency in an acceleration signal from an inertial measurement unit in the headset (page 3, Section 2.2., para. [0004], note that several studies proposed methods to predict the lower-body pose from the upper-body information. Jiang et al. [JYF16] developed a method to generate lower-body motion by blending 8 walking animations with different directions based on the body direction and velocity, which are predicted from off-the-shelf VR devices).
At the time of filing for the invention, it would have been obvious to a person of ordinary skilled in the art to teach wherein tracking a lower-body posture of the user comprises identifying a running motion or a walking motion from a frequency in an acceleration signal from an inertial measurement unit in the headset. The suggestion/motivation for doing so would have been in order to efficiently capture human motion with a sparse set of tracking data for real-time prediction of the lower-body pose only from the tracking signals of the upper-body joints (abs.). Therefore, it would have been obvious to combine Lefaudeux and Balakrishnan with Yang to obtain the invention as specified in claim 6.
(3) regarding claim 10:
Lefaudeux disclosed most of the subject matter as described as above except for specifically teaching wherein merging the lower-body model with the upper-body avatar comprises removing object collisions of the lower-body model with an object in an immersive reality application.
However, Yang disclosed wherein merging the lower-body model with the upper-body avatar comprises removing object collisions of the lower-body model with an object in an immersive reality application (page 2, Section 1, para. [0004], note that to remove foot-skating and floating artifacts, LoB-STr predicts feet contact states, which are used to post-process the lower-body pose with inverse kinematics to preserve the contact).
At the time of filing for the invention, it would have been obvious to a person of ordinary skilled in the art to teach wherein merging the lower-body model with the upper-body avatar comprises removing object collisions of the lower-body model with an object in an immersive reality application. The suggestion/motivation for doing so would have been in order to efficiently capture human motion with a sparse set of tracking data for real-time prediction of the lower-body pose only from the tracking signals of the upper-body joints (abs.). Therefore, it would have been obvious to combine Lefaudeux and Balakrishnan with Yang to obtain the invention as specified in claim 10.
(4) regarding claim 23:
Lefaudeux disclosed most of the subject matter as described as above except for specifically teaching wherein the lower-body posture of the user is further based on A) a detected height of the headset relative to the ground and B) a determined height of user.
However, Yang disclosed wherein the lower-body posture of the user is further based on A) a detected height of the headset relative to the ground and B) a determined height of user (page 6, 4.1.3 Input Representations Comparison, para. [0001], note that the test set of wild signals is captured from subjects with 157, 171, and 184 cm heights. The subjects are asked to perform a sequence of actions; walk, run, sit/standup, carry and move, static gestures, and free movement. Also see figs. 5-7).
At the time of filing for the invention, it would have been obvious to a person of ordinary skilled in the art to teach wherein the lower-body posture of the user is further based on A) a detected height of the headset relative to the ground and B) a determined height of user. The suggestion/motivation for doing so would have been in order to efficiently capture human motion with a sparse set of tracking data for real-time prediction of the lower-body pose only from the tracking signals of the upper-body joints (abs.). Therefore, it would have been obvious to combine Lefaudeux and Balakrishnan with Yang to obtain the invention as specified in claim 23.
The proposed rejection of claims 4, 6 renders obvious the steps of the system claim 4 (see fig. 1, the system) and the non-transitory computer-readable claims 25-26 because these steps occur in the operation of the proposed rejection as discussed above. Thus, the arguments similar to that presented above for claims 4 and 6 are equally applicable to claims 14, 25-26.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Winold et al. (US Publication Number 2022/0035443 A1) disclose a system for generating complementary data for a visual display that includes one or plurality of wearable sensors that collect tracking data for a users position, orientation, and movement.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communication from the examiner should be directed to Hilina K Demeter whose telephone number is (571) 270-1676.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, King Y. Poon could be reached at (571) 270- 0728. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about PAIR system, see http://pari-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/HILINA K DEMETER/Primary Examiner, Art Unit 2617