DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Allowable Subject Matter
Claims 11-14 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-3, 5-7 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ballan et al. (US Patent Number 2023/0359286-A1, hereinafter “Ballan”) in view of Erivantcev et al. (US Patent Number 2020/0033937-A1, hereinafter “Erivantcev”).
Regarding claim 1, Ballan teaches: An apparatus for pose generation, comprising: at least one memory; and at least one processor coupled to the at least one memory, the at least one processor being configured to: ([0027], "FIGS. 1A and 1B show pictorial views of AR glasses 100, according to a possible implementation of the present disclosure. The AR glasses 100 are an example of a head-mounted display, or headset, suitable for use in creating an immersive experience for a user. The AR glasses 100 include, among other components, a projector 104, a display area 105, a control system 112 that includes an image processor 114, a camera 116, a frame 123, lenses 127, arms 130, and one or more IMUs 150.")
generate, using a machine learning (ML) model, ([0044], "FIGS. 7-12 present additional details related to development of a machine learning algorithm able to estimate, from IMU measurements, the 6DoF pose of a mobile device, e.g., an AR headset such as AR glasses 100.")
a first estimated body pose based on sensor information received from one or more sensors of a set of sensors; ([0030], "FIG. 2 is a high-level block diagram illustrating a tracking system 200, according to a possible implementation of the present disclosure. The tracking system 200 combines image data 202 from the camera 116 with IMU data 204 from the IMU 150 to produce a final device pose 216. The device pose 216 represents the position and orientation of the AR headset. ")
generate, using a physics-based model, a second estimated body pose based on the sensor information; ([0030], "A fused pose provider 214 integrates a mapping pose stream component 210 with an IMU-based 6DoF pose stream component 212 to produce a composite device pose, e.g., the final device pose 216.")
detect a failure of the physics-based model; and based on detection of the failure of the physics-based model, switch from generating ([0031], "From time t=0 until the mapping algorithm 206 is initialized at time t.sub.i, the mapping algorithm 206 cannot provide the device pose 216. Instead, the IMU-based 6DoF fallback pose stream component 212 can provide a continuous AR experience by supporting 6DoF poses while the original mapping pose stream component 210 is initializing.")
Ballan does not teach: The poses for the switched and switching models to be skeletal.
However, Erivantcev does teach: Poses of model reconstruing to be skeletal. (Erivantcev, [0025], "At least some techniques disclosed herein allow the determination of calibration parameters of the measurements of the inertial measurement unit such that the calibrated measurements of the inertial measurement unit are relative to an known orientation, such as the orientation of the sensor device in which the inertial measurement unit is installed, the orientation of the arm or hand of a user to which the sensor device is attached, or the orientation of a skeleton model of the user in a reference pose. A stereo camera integrated in a head mount display (HMD) can be used to capture images of sensor modules on the user.")
At the time the invention was made, it would have been obvious to one of ordinary skill in the art to modify secondary device pose model fallback (as taught by Ballan) to include skeletal models (as taught by Erivantcev)because such a modification is the result of simple substitution of one known element for another producing a predictable result. More specifically, the device pose of Ballan and skeletal pose of Erivantcev perform the same general and predictable function, the predictable function being tracking the body location of the user. Since each individual element and its function are shown in the prior art, albeit shown in separate references, the difference between the claimed subject matter and the prior art rests not on any individual element or function but in the very combination itself - that is in the substitution of modify secondary device pose model fallback (as taught by Ballan) by replacing it with skeletal models (as taught by Erivantcev). Thus, the simple substitution of one known element for another producing a predictable result renders the claim obvious.
Regarding claim 2, Ballan in view of Erivantcev teaches: The apparatus of claim 1, wherein the physics-based model comprises a reinforcement learning (RL) based model for predicting control signals to apply in a physics engine. ([0036], "In some implementations, while the mapping algorithm 206 re-initializes, the IMU-based 6DoF 208 supplies poses to the fused pose provider 214. In some implementations, internally, the IMU-based 6DoF 208 runs an IMU integrator 406, the output of which is fed to a neural network, e.g., the DeepIO network 408. In some implementations, the DeepIO network 408, in turn, corrects integration errors using past device motions estimated by the mapping algorithm 206, prior to the reset and feeds a pose update back to the IMU integrator 406 for a next iteration.")
Regarding claim 3, Ballan in view of Erivantcev teaches: The apparatus of claim 1, wherein the sensor information comprises pose information for the set of sensors. ([0029], "FIG. 1B shows a rear pictorial view of the AR glasses 100, illustrating placement of the IMU 150, the projector 104, and the display area 105. Images captured by the camera 116 are projected by the projector 104 onto the display area 105, located on an inside surface of one of the lenses 127, e.g., the right lens, of the AR glasses 100. One or more IMUs 150 can be located at a fixed corner, or on a moveable arm 130 of the AR glasses 100. Each IMU 150 includes micro-electromechanical (MEMs) devices such as accelerometers and gyroscopes that can measure orientation and motion of the AR glasses 100 to which they are attached, and by inference, motion of the user's head relative to the environment. In some implementations, IMU data can be processed along with image data in the image processor 114. Further, the image processor 114 located on the HMD, e.g., headset, or AR glasses, can also implement the tracking algorithm(s) described herein.")
Regarding claim 5, Ballan in view of Erivantcev teaches: The apparatus of claim 1, wherein the set of sensors include fewer sensors than joints in the skeletal pose. (Erivantcev, [0026], "In general, the kinematics of a user can be modeled using a skeleton model having a set of rigid parts/portions connected by joints. For example, the head, the torso, the left and right upper arms, the left and right forearms, the palms, phalange bones of fingers, metacarpal bones of thumbs, upper legs, lower legs, and feet can be considered as rigid parts that are connected via various joints, such as the neck, shoulders, elbows, wrist, and finger joints."; [0060], "The computing device (141) can have a prediction model (116) and a motion processor (145). The measurements of the IMUs (e.g., 131, 121) from the head module (111), arm modules (e.g., 113 and 115), and/or hand modules (e.g., 117 and 119)"; Examiner's Note - Prior art shows embodiment that has 7 sensors but at least 13 joints not including the fingers.)
Regarding claim 6, Ballan in view of Erivantcev teaches: The apparatus of claim 1, wherein the set of sensors include fewer sensors than joints in the skeletal pose. (Erivantcev, [0026], "In general, the kinematics of a user can be modeled using a skeleton model having a set of rigid parts/portions connected by joints. For example, the head, the torso, the left and right upper arms, the left and right forearms, the palms, phalange bones of fingers, metacarpal bones of thumbs, upper legs, lower legs, and feet can be considered as rigid parts that are connected via various joints, such as the neck, shoulders, elbows, wrist, and finger joints."; [0060], "The computing device (141) can have a prediction model (116) and a motion processor (145). The measurements of the IMUs (e.g., 131, 121) from the head module (111), arm modules (e.g., 113 and 115), and/or hand modules (e.g., 117 and 119)"; Examiner's Note - Prior art shows embodiment that has 7 sensors but at least 13 joints not including the fingers.)
Regarding claim 7, Ballan in view of Erivantcev teaches: The apparatus of claim 6, wherein the set of sensors include one or more hand controllers. (Erivantcev, [0060], "The computing device (141) can have a prediction model (116) and a motion processor (145). The measurements of the IMUs (e.g., 131, 121) from the head module (111), arm modules (e.g., 113 and 115), and/or hand modules (e.g., 117 and 119)")
Regarding claim 15, claim 15 has been analyzed with regard to claim 1 and is rejected for the same reasons of obviousness as used above as well as in accordance with Ballan further teaching on: A method ([0030], "FIG. 2 is a high-level block diagram illustrating a tracking system 200, according to a possible implementation of the present disclosure. The tracking system 200 combines image data 202 from the camera 116 with IMU data 204 from the IMU 150 to produce a final device pose 216. The device pose 216 represents the position and orientation of the AR headset. ")
Regarding claim 16, claim 16 has been analyzed with regard to claim 2 and is rejected for the same reasons of obviousness as used above as well as in accordance with Ballan further teaching on: A method ([0030], "FIG. 2 is a high-level block diagram illustrating a tracking system 200, according to a possible implementation of the present disclosure. The tracking system 200 combines image data 202 from the camera 116 with IMU data 204 from the IMU 150 to produce a final device pose 216. The device pose 216 represents the position and orientation of the AR headset. ")
Regarding claim 17, claim 17 has been analyzed with regard to claim 3 and is rejected for the same reasons of obviousness as used above as well as in accordance with Ballan further teaching on: A method ([0030], "FIG. 2 is a high-level block diagram illustrating a tracking system 200, according to a possible implementation of the present disclosure. The tracking system 200 combines image data 202 from the camera 116 with IMU data 204 from the IMU 150 to produce a final device pose 216. The device pose 216 represents the position and orientation of the AR headset. ")
Regarding claim 19, claim 19 has been analyzed with regard to claim 5 and is rejected for the same reasons of obviousness as used above as well as in accordance with Ballan further teaching on: A method ([0030], "FIG. 2 is a high-level block diagram illustrating a tracking system 200, according to a possible implementation of the present disclosure. The tracking system 200 combines image data 202 from the camera 116 with IMU data 204 from the IMU 150 to produce a final device pose 216. The device pose 216 represents the position and orientation of the AR headset. ")
Regarding claim 20, claim 20 has been analyzed with regard to claim 6 and is rejected for the same reasons of obviousness as used above as well as in accordance with Ballan further teaching on: A method ([0030], "FIG. 2 is a high-level block diagram illustrating a tracking system 200, according to a possible implementation of the present disclosure. The tracking system 200 combines image data 202 from the camera 116 with IMU data 204 from the IMU 150 to produce a final device pose 216. The device pose 216 represents the position and orientation of the AR headset. ")
Claim(s) 8-10 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ballan et al. (US Patent Number 2023/0359286-A1, hereinafter “Ballan”) and Erivantcev et al. (US Patent Number 2020/0033937-A1, hereinafter “Erivantcev”) in view of Degges et al. (CN Patent Number 113383294-A, hereinafter “Degges”).
Regarding claim 8, Ballan in view of Ericantcev does not teach: The apparatus of claim 1, wherein, to detect the failure of the physics-based model, the at least one processor is configured to: determine a difference between the second estimated body pose and the first estimated body pose; and detect the failure based on a comparison between the difference and a threshold.
However, Degges does teach: The apparatus of claim 1, wherein, to detect the failure of the physics-based model, the at least one processor is configured to: determine a difference between the second estimated body pose and the first estimated body pose; and detect the failure based on a comparison between the difference and a threshold. (Degges, [0009], "Example 9 is the method according to example 1, further comprising: comparing the first difference with a threshold value; determining that the first difference exceeds the threshold; and in response to determining that the first difference exceeds the threshold: stopping the first processing stack; and allowing the second processing stack to continue to run.")
At the time the invention was made, it would have been obvious to one of ordinary skill in the art to modify fallback model (as taught by Ballan in view of Ericantcev) to include fallback based on difference between first and second model being greater than some threshold (as taught by Degges) because such a modification is the result of combining prior art elements according to known methods to yield predictable results. More specifically, fallback criteria as modified by being a difference being greater then some threshold can yield a predictable result of allowing for errors between both models to be corrected. Thus, a person of ordinary skill would have appreciated including in fallback model the ability to do fallback based on difference between first and second model being greater than some threshold since the claimed invention is merely a combination of old elements, and in the combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that the results of the combination were predictable.
Regarding claim 9, Ballan in view of Ericantcev and Degges teaches: The apparatus of claim 1, wherein, to detect the failure of the physics-based model, the at least one processor is configured to: determine a difference between the second estimated body pose and the first estimated body pose; and detect the failure based on a comparison between the difference and a threshold. (Degges, [0009], "Example 9 is the method according to example 1, further comprising: comparing the first difference with a threshold value; determining that the first difference exceeds the threshold; and in response to determining that the first difference exceeds the threshold: stopping the first processing stack; and allowing the second processing stack to continue to run.")
Regarding claim 10, Ballan in view of Ericantcev and Degges teaches: The apparatus of claim 1, wherein, to detect the failure of the physics-based model, the at least one processor is configured to: determine a difference between the second estimated body pose and the first estimated body pose; and detect the failure based on a comparison between the difference and a threshold. (Degges, [0009], "Example 9 is the method according to example 1, further comprising: comparing the first difference with a threshold value; determining that the first difference exceeds the threshold; and in response to determining that the first difference exceeds the threshold: stopping the first processing stack; and allowing the second processing stack to continue to run.")
Claim(s) 4 and 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ballan et al. (US Patent Number 2023/0359286-A1, hereinafter “Ballan”) and Erivantcev et al. (US Patent Number 2020/0033937-A1, hereinafter “Erivantcev”) in view of Black et al. (US Patent Number 2018/0315230-A1, hereinafter “Black”).
Regarding claim 4, Ballan in view of Erivantcev does no teach: The apparatus of claim 1, wherein the skeletal pose comprises a skinned multi-person linear model (SMPL) skeleton.
However, Black does teach: The apparatus of claim 1, wherein the skeletal pose comprises a skinned multi-person linear model (SMPL) skeleton. (Black, [0033], “FIG. 1: (left) SMPL model (orange) fit to ground truth 3D meshes (gray). (right) Unity 5.0 game engine screenshot showing bodies from the CAESAR dataset animated in real time.”)
At the time the invention was made, it would have been obvious to one of ordinary skill in the art to modify secondary device pose model fallback (as taught by Ballan in view of Erivantcev) to include skinned multi-person linear model (as taught by Black) because such a modification is the result of simple substitution of one known element for another producing a predictable result. More specifically, the device pose of Ballan in view of Erivantcev and skinned multi-person linear model of Black perform the same general and predictable function, the predictable function being tracking the body location of the user. Since each individual element and its function are shown in the prior art, albeit shown in separate references, the difference between the claimed subject matter and the prior art rests not on any individual element or function but in the very combination itself - that is in the substitution of modify secondary device pose model fallback (as taught by Ballan in view of Erivantcev) by replacing it with skinned multi-person linear model (as taught by Black). Thus, the simple substitution of one known element for another producing a predictable result renders the claim obvious.
Regarding claim 18, claim 18 has been analyzed with regard to claim 4 and is rejected for the same reasons of obviousness as used above as well as in accordance with Ballan further teaching on: A method ([0030], "FIG. 2 is a high-level block diagram illustrating a tracking system 200, according to a possible implementation of the present disclosure. The tracking system 200 combines image data 202 from the camera 116 with IMU data 204 from the IMU 150 to produce a final device pose 216. The device pose 216 represents the position and orientation of the AR headset. ")
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Jinsu Hwang whose telephone number is (703)756-1370. The examiner can normally be reached Mon -Thu 10am-8am EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matthew Bella can be reached at (571) 272-7778. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JINSU HWANG/Examiner, Art Unit 2667
/MATTHEW C BELLA/Supervisory Patent Examiner, Art Unit 2667
Prosecution Insights