DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 2-21 (Claims 2, 12 and 21 are independent claims) are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The claim(s) recite(s) a process. This judicial exception is not integrated into a practical application because the claims recite a mathematical concept. The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception because the claims recite determining a first pose; receiving IMU data of the controller; calculating a second pose of the controller; and estimating based on the calculated second pose; calculating the second pose based on the estimated pose of the controller relative to the environment and relative to the hand. Although the claim recites receiving IMU data of the controller, the additional limitation refers to mere data gathering to perform the mathematical calculations.
Claims that are noted above as being rejected but that are not specifically cited below are rejected based on similar rational of the independent claim.
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claims 2-11 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1-10 of U.S. Patent No. 12,153,724 B2. Although the claims at issue are not identical, they are not patentably distinct from each other because it is clear that all the elements of the application claims 2-11 are to be found in patent claims 1-10 (as the application claims 2-11 fully encompasses patent claims 1-10). The difference between the application claims 2-11 and the patent claims 1-10 lies in the fact that the patent claim includes many more elements and is thus much more specific. Thus the invention of claims 1-10 of the patent is in effect a “species” of the “generic” invention of the application claims 2-11. It has been held that the generic invention is “anticipated” by the “species”. See In re Goodman, 29 USPQ2d 2010 (Fed. Cir. 1993). Since application claims 2-11 are anticipated by claims 1-10 of the patent, it is not patentably distinct from claims 1-10 of the patent.
18/920,449
(New) A method comprising, by a computing system: determining a first pose of a hand of a user holding a controller associated with a wearable device worn by a user in an environment based on a first image depicting at least a part of the hand;
determining a first pose of the controller based on the first pose of the hand and a first inertial measurement unit (IMU) data of the controller;
receiving a second IMU data of the controller;
calculating a second pose of the controller by adjusting the first pose of the controller based on the second IMU data; and estimating, based on the calculated second pose of the controller, a second pose of the hand,
wherein calculating the second pose of the controller comprises: estimating a pose of the controller relative to the environment based on the first pose of the controller and the second IMU data of the controller; estimating a pose of the controller relative to the hand based on the adjusted pose of the controller relative to the environment and the second IMU data of the controller; and calculating the second pose of the controller based on the estimated pose of the controller relative to the environment and the estimated pose of the controller relative to the hand.
3. (New) The method of Claim 2, wherein the first image is captured at a first frequency, wherein the first IMU data of the controller and the second IMU data of the controller are received at a second frequency, wherein the second frequency is higher than the first frequency.
4. (New) The method of Claim 2, further comprising: receiving IMU data of the wearable device to estimate a pose of the wearable device; and updating the first pose of the hand of the user based on the pose of the wearable device.
5. (New) The method of Claim 4, wherein the pose of the wearable device is estimated based on the IMU data of the wearable device and the first image.
6. (New) The method of Claim 2, further comprising: capturing a second image using one or more cameras, the second image depicting at least a part of the hand of the user holding the controller in the environment; identifying one or more features from the second image; and estimating a third pose of the hand based on the one or more features identified from the second image.
7. (New) The method of Claim 6, wherein the computer system utilizes information associated with a subset area of the second image for identifying the one or more features from the second image.
8. (New) The method of Claim 6, wherein a frequency of estimating the second pose of the hand is higher than a frequency of estimating the third pose of the hand.
9. (New) The method of Claim 2, wherein the wearable device comprises: one or more cameras configured to capture images; a hand-tracking unit configured to estimate the pose of the hand of the user; and a controller-tracking unit configured to estimate the pose of the controller.
10. (New) The method of Claim 2, wherein determining the first pose of the controller comprises: further determining the first pose of the controller based on an estimated grip that defines a relative pose between the hand of the user and the controller and a predicted pose of the controller based on the first IMU data of the controller.
11. (New) The method of Claim 10, wherein estimating the second pose of the hand comprises applying an inverse of the estimated grip to the calculated second pose of the controller.
12153724 B2
1. A method
comprising, by a computing system: determining a first pose of a hand of a user holding a controller associated with a wearable device worn by a user in an environment based on a first image depicting at least a part of the hand;
determining a first pose of the controller based on the first pose of the hand and first inertial measurement unit (IMU) data of the controller;
receiving second IMU data of the controller;
calculating a second pose of the controller by adjusting the first pose of the controller based on the second IMU data;
estimating, based on the calculated second pose of the controller, a second pose of the hand; and determining, based on the estimated second pose of the hand, a subset area of a second image depicting at least a part of the hand of the user, wherein the computing system identifies one or more features from the determined subset area of the second image to determine a third pose of the hand that is used for determining a third pose of the controller,
wherein calculating the second pose of the controller comprises: estimating a pose of the controller relative to the environment based on the first pose of the controller and the second IMU data of the controller; estimating a pose of the controller relative to the hand based on the adjusted pose of the controller relative to the environment and the second IMU data of the controller; and calculating the second pose of the controller based on the estimated pose of the controller relative to the environment and the estimated pose of the controller relative to the hand.
2. The method of claim 1, wherein the first image and the second image are captured at a first frequency, wherein the first IMU data of the controller and the second IMU data of the controller are received at a second frequency, wherein the second frequency is higher than the first frequency.
3. The method of claim 1, further comprising: receiving IMU data of the wearable device to estimate a pose of the wearable device; and updating the first pose of the hand of the user based on the pose of the wearable device.
4. The method of claim 3, wherein the pose of the wearable device is estimated based on the IMU data of the wearable device and the first image.
5. The method of claim 1, further comprising: capturing the second image using one or more cameras, the second image depicting at least a part of the hand of the user holding the controller in the environment; identifying one or more features from the second image; and estimating the third pose of the hand based on the one or more features identified from the second image of the user.
6. The method of claim 5, wherein the computer system utilizes information associated with the determined subset area of the second image for identifying the one or more features from the second image.
7. The method of claim 5, wherein a frequency of estimating the second pose of the hand is higher than a frequency of estimating the third pose of the hand.
8. The method of claim 1, wherein the wearable device comprises: one or more cameras configured to capture images; a hand-tracking unit configured to estimate the pose of the hand of the user; and a controller-tracking unit configured to estimate the pose of the controller.
9. The method of claim 1, wherein determining the first pose of the controller comprises: determining the first pose of the controller based on the first pose of the hand of the user, an estimated grip that defines a relative pose between the hand of the user and the controller, and a predicted pose of the controller based on the first IMU data of the controller.
10. The method of claim 1, wherein estimating the second pose of the hand comprises applying an inverse of the estimated grip to the calculated second pose of the controller.
12. (New) One or more computer-readable non-transitory non-volatile storage media embodying software that is operable when executed, by a computer system, to: determine a first pose of a hand of a user holding a controller associated with a wearable device worn by a user in an environment based on a first image depicting at least a part of the hand; determine a first pose of the controller based on the first pose of the hand and first inertial measurement unit (IMU) data of the controller;
receive a second IMU data of the controller; calculate a second pose of the controller by adjusting the first pose of the controller based on the second IMU data; estimate, based on the calculated second pose of the controller, a second pose of the hand, wherein calculating the second pose of the controller comprises: estimating a pose of the controller relative to the environment based on the first pose of the controller and the second IMU data of the controller; estimating a pose of the controller relative to the hand based on the adjusted pose of the controller relative to the environment and the second IMU data of the controller; and calculating the second pose of the controller based on the estimated pose of the controller relative to the environment and the estimated pose of the controller relative to the hand.
13. (New) The media of Claim 12, wherein the first image is captured at a first frequency, wherein the first IMU data of the controller and the second IMU data of the controller are received at a second frequency, wherein the second frequency is higher than the first frequency.
14. (New) The media of Claim 12, wherein the software is further operable when executed to: receive IMU data of the wearable device to estimate a pose of the wearable device; and update the first pose of the hand of the user based on the pose of the wearable device.
15. (New) The media of Claim 14, wherein the pose of the wearable device is estimated based on the IMU data of the wearable device and the first image.
16. (New) The media of Claim 12, wherein the software is further operable when executed to: capture a second image using one or more cameras, the second image depicting at least a part of the hand of the user holding the controller in the environment; identify one or more features from the second image; and estimate a third pose of the hand based on the one or more features identified from the second image.
17. (New) The media of Claim 16, wherein the computer system utilizes information associated with a subset area of the second image for identifying the one or more features from the second image.
18. (New) The media of Claim 16, wherein a frequency of estimating the second pose of the hand is higher than a frequency of estimating the third pose of the hand.
19. (New) The media of Claim 12, wherein the wearable device comprises: one or more cameras configured to capture images; a hand-tracking unit configured to estimate the pose of the hand of the user; and a controller-tracking unit configured to estimate the pose of the controller.
20. (New) The media of Claim 12, wherein the software is further operable when executed to determine the first pose of the controller by further determining the first pose of the controller based on an estimated grip that defines a relative pose between the hand of the user and the controller and a predicted pose of the controller based on the first IMU data of the controller, wherein estimating the second pose of the hand comprises applying an inverse of the estimated grip to the calculated second pose of the controller.
10. One or more computer-readable non-transitory storage media embodying software that is operable when executed to: capture, using one or more cameras implemented in a wearable device worn by a user, a first image depicting at least a part of a hand of the user holding a controller in an environment; identify one or more features from the first image to estimate a pose of the hand of the user; estimate a first pose of the controller based on the pose of the hand of the user and an estimated grip that defines a relative pose between the hand of the user and the controller; receive first inertial measurement unit (IMU) data of the controller; estimate a second pose of the controller by updating the first pose of the controller using the first IMU data of the controller; receive second IMU data of the controller; calculate an IMU-predicted pose of the controller based on the second IMU data; estimate an IMU-predicted pose of the hand by applying an inverse of the estimated grip to the IMU-predicted pose of the controller; and determine, based on the estimated IMU-predicted pose of the hand, a search range of the hand when capturing a second image depicting at least a part of the hand of the user.
11. The media of claim 10, wherein the first image is captured at a first frequency and the IMU data of the controller is received at a second frequency, wherein the second frequency is higher than the first frequency.
12. The media of claim 10, wherein the software is further operable when executed to: receive IMU data of the wearable device to estimate a pose of the wearable device; and update the pose of the hand of the user based on the pose of the wearable device.
13. The media of claim 12, wherein the pose of the wearable device is estimated based on the IMU data of the wearable device and the first image.
14. The media of claim 10, wherein the software is further operable when executed to: estimate a second pose of the hand based on the IMU-predicted pose of the hand.
15. The media of claim 14, wherein the software is further operable when executed to: capture the second image using the one or more cameras, the second image depicting at least a part of the hand of the user holding the controller in the environment; identify the one or more features from the second image; and estimate a third pose of the hand based on the one or more features identified from the second image of the user.
16. The media of claim 15, wherein a frequency of estimating the second pose of the hand is higher than a frequency of estimating the third pose of the hand.
17. The media of claim 10, wherein the wearable device comprises: the one or more cameras configured to capture images; a hand-tracking unit configured to estimate the pose of the hand of the user; and a controller-tracking unit configured to estimate the second pose of the controller.
18. The media of claim 10, wherein estimating the second pose of the controller comprises: estimating a pose of the controller relative to the environment based on the estimated grip and the estimated pose of the hand of the user relative to the environment; adjusting the pose of the controller relative to the environment based on the IMU data of the controller; estimating a pose of the controller relative to the hand based on the adjusted pose of the controller relative to the environment and the IMU of the controller; and estimating the second pose of the controller based on the adjusted pose of the controller relative to the environment and the estimated pose of the controller relative to the hand.
21. (New) A computing system comprising: one or more processors; and one or more computer-readable non-transitory non-volatile storage media coupled to one or more of the processors and comprising instructions operable when executed by the one or more of the processors to cause the system to: determine a first pose of a hand of a user holding a controller associated with a wearable device worn by a user in an environment based on a first image depicting at least a part of the hand; determine a first pose of the controller based on the first pose of the hand and first inertial measurement unit (IMU) data of the controller;
receive a second IMU data of the controller; calculate a second pose of the controller by adjusting the first pose of the controller based on the second IMU data; and estimate, based on the calculated second pose of the controller, a second pose of the hand, wherein calculating the second pose of the controller comprises: estimating a pose of the controller relative to the environment based on the first pose of the controller and the second IMU data of the controller; estimating a pose of the controller relative to the hand based on the adjusted pose of the controller relative to the environment and the second IMU data of the controller; and calculating the second pose of the controller based on the estimated pose of the controller relative to the environment and the estimated pose of the controller relative to the hand.
19. A system comprising: one or more processors; and one or more computer-readable non-transitory storage media coupled to one or more of the processors and comprising instructions operable when executed by the one or more of the processors to cause the system to: capture, using one or more cameras implemented in a wearable device worn by a user, a first image depicting at least a part of a hand of the user holding a controller in an environment; identify one or more features from the first image to estimate a pose of the hand of the user; estimate a first pose of the controller based on the pose of the hand of the user and an estimated grip that defines a relative pose between the hand of the user and the controller; receive first inertial measurement unit (IMU) data of the controller; estimate a second pose of the controller by updating the first pose of the controller using the first IMU data of the controller;
receive second IMU data of the controller; calculate an IMU-predicted pose of the controller based on the second IMU data; estimate an IMU-predicted pose of the hand by applying an inverse of the estimated grip to the IMU-predicted pose of the controller; and determine, based on the estimated IMU-predicted pose of the hand, a search range of the hand when capturing a second image depicting at least a part of the hand of the user.
20. The system of claim 19, wherein the instructions are further operable when executed to: receive IMU data of the wearable device to estimate a pose of the wearable device; and update the pose of the hand of the user based on the pose of the wearable device.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 2-21 is/are rejected under 35 U.S.C. 103 as being unpatentable over Chang et al., U.S. Patent Publication Number 2021/0263592 A1, in view of Fei et al, U.S. Patent Publication Number 2018/0285636 A1.
Regarding claim 2, Chang discloses a method comprising, by a computing system: determining a first pose of a hand of a user holding a controller associated with a wearable device worn by a user in an environment based on a first image depicting at least a part of the hand (paragraph 0130, wearable imaging devices having a field of view that at least partially and at least temporarily includes hand, allowing the determination of the pose of hand; FIGS. 8A and 11; hand holding controller); determining a first pose of the controller based on the first pose of the hand and a first inertial measurement unit (IMU) data of the controller (paragraph 0091, positions and orientations of handheld device and hand may be determined; paragraph 0115, handheld device may include an IMU; IMU may include one or more accelerometers (e.g., three), one or more gyroscopes (e.g., three); convert raw measurements into processed data);
receiving a second IMU data of the controller (paragraph 0115, handheld device may include an IMU; IMU may include one or more accelerometers (e.g., three), one or more gyroscopes (e.g., three); convert raw measurements into processed data; paragraph 0136, handheld pose data may include capturing sensor data using the handheld device pose sensor);
calculating a second pose of the controller by adjusting the first pose of the controller based on the second IMU data (paragraph 0136, position of handheld device is determined based on the sensor data, for example, processing and analyzing the sensor data); and estimating, based on the calculated second pose of the controller, a second pose of the hand (paragraph 0011, using the handheld device pose data to augment the hand pose data; paragraph 0099, one or more such estimated or assumed parameters may be determined based at least in part on; data acquired or output by one or more sensors of handheld device (e.g., one or more cameras, one or more inertial measurement units (IMUs); paragraph 0141, using the hand pose data to augment the handheld device pose data includes using the hand pose data in any determination of the position and/or orientation of the handheld device).
However, it is noted that Chang discloses using hand pose data in any determination of the position and/or orientation of the handheld device and vice versa, but fails to specifically disclose wherein calculating the second pose of the controller comprises: estimating a pose of the controller relative to the environment based on the first pose of the controller and the second IMU data of the controller; estimating a pose of the controller relative to the hand based on the adjusted pose of the controller relative to the environment and the second IMU data of the controller; and calculating the second pose of the controller based on the estimated pose of the controller relative to the environment and the estimated pose of the controller relative to the hand.
Fei discloses determining a first pose of a hand of a user holding a controller associated with a wearable device worn by a user in an environment based on a first image depicting at least a part of the hand (paragraph 0006, obtaining the image in a current frame; in response to determining that the physical hand holds an object; determine 3D skeleton joints of the physical hand); determining a first pose of the controller based on the first pose of the hand and a first inertial measurement unit (IMU) data of the controller (paragraph 0047, determine 3D object pose; paragraph 0052, user may hold a controller (with a motion sensor such as IMU) ); receiving a second IMU data of the controller (paragraph 0055, can obtain the controller’s IMU information ); (paragraph 0055, can use the IMU information to: continue tracking the 3D rotation and 3D position of the controller); and estimating, based on the calculated second pose of the controller, a second pose of the hand, wherein calculating the second pose of the controller comprises: estimating a pose of the controller relative to the environment based on the first pose of the controller and the second IMU data of the controller (paragraph 0055, when hand and controller move out of camera’s field of view, use the IMU data to continue tracking the 3D rotation and 3D position of the controller and the user’s hand; paragraph 0038, all movement of the device can be estimated based on vision-based hand tracking); estimating a pose of the controller relative to the hand based on the adjusted pose of the controller relative to the environment and the second IMU data of the controller (paragraph 0037, 3D rotation and 3D position of the hand; e.g., when the hand is out of the hand tracking camera’s field of view coverage; detection of some action, e.g., grabbing, etc.; peripheral device that provides 3D rotational information, such as a 3D controller); and calculating the second pose of the controller based on the estimated pose of the controller relative to the environment and the estimated pose of the controller relative to the hand (paragraph 0037, calculate new 3D rotation and 3D position of the hand and the controller).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include in the hand and controller estimating as disclosed by Chang, using the estimated controller and adjusted pose information of the IMU data as disclosed by Fei, to detect 3D rotation and 3D positioning of a hand when it is out of field of view of a camera, to further provide new rotation and position with updated IMU data captured for a fusion algorithm.
Regarding claim 3, Chang discloses wherein the first image is captured at a first frequency, wherein the first IMU data of the controller and the second IMU data of the controller are received at a second frequency, wherein the second frequency is higher than the first frequency (paragraph 0125, IMUs may be engineered such that they remain stable up to 50-100 milliseconds; that may enable pose updates to be reported at a rate of 10-20 HZ; by keeping the IMUs stable at a reasonable rate, the rate of pose updates may be decreased to 10-20 Hz (as compared to higher frequencies)).
Regarding claim 4, Chang discloses further comprising: receiving IMU data of the wearable device to estimate a pose of the wearable device; and updating the first pose of the hand of the user based on the pose of the wearable device (paragraph 0107, IMU may be operative coupled; to detect the position and orientation of each component relative to each other and/or relative to a coordinate system; paragraph 0091, position and orientation of wearable device may be used as the reference frame; accordingly, the positions and orientations of handheld device and hand may be determined with respect to the position and orientation of wearable device).
Regarding claim 5, Chang discloses wherein the pose of the wearable device is estimated based on the IMU data of the wearable device and the first image (paragraph 0118, imaging device may determine its position and/or orientation with respect to wearable device by capturing an image; paragraph 0120, wearable device may include an IMU).
Regarding claim 6, Chang discloses further comprising: capturing a second image using one or more cameras, the second image depicting at least a part of the hand of the user holding the controller in the environment (paragraph 0130, field of view that at least partially and at least temporarily includes hand); identifying one or more features from the second image (paragraph 0131, set of hand key points that may be identified based on captured image or video of a user’s hand); and estimating a third pose of the hand based on the one or more features identified from the second image (paragraph 0130, field of view that at least partially and at least temporarily includes hand, allowing the determination of the pose of the hand; paragraph 0131, set of hand key points that may be identified based on captured image or video of a user’s hand).
Regarding claim 7, Chang discloses wherein the computer system utilizes information associated with a subset area of the second image for identifying the one or more features from the second image (paragraph 0130, hand key points may be identified using various image analysis techniques, such as, for example, by training a machine learning model (e.g., a neural network) using a set of labeled image; may receive images or a stream of images).
Regarding claim 8, Chang discloses wherein a frequency of estimating the second pose of the hand is higher than a frequency of estimating the third pose of the hand (paragraph 0125, IMUs may be engineered such that they remain stable up to 50-100 milliseconds; that may enable pose updates to be reported at a rate of 10-20 HZ; by keeping the IMUs stable at a reasonable rate, the rate of pose updates may be decreased to 10-20 Hz (as compared to higher frequencies)).
Regarding claim 9, Chang discloses wherein the wearable device comprises: one or more cameras configured to capture images (FIG. 7; paragraph 0119, wearable device may include one or more imaging devices, 710; wearable imaging device may be optical device such as cameras and may capture still or moving images); a hand-tracking unit configured to estimate the pose of the hand of the user (paragraph 0121, wearable system may include a computing apparatus (e.g., one or more processors and an associated memory) for performing a localization of handheld device); and a controller-tracking unit configured to estimate the pose of the controller (paragraph 0121, computing apparatus may comprise a mapping database to detect pose, to determine the coordinates of real objects).
Regarding claim 10, Chang discloses wherein determining the first pose of the controller comprises: further determining the first pose of the controller based on an estimated grip that defines a relative pose between the hand of the user and the controller and a predicted pose of the controller based on the first IMU data of the controller (Chang 2021/0263592 paragraph 0113, user may hold handheld device in their left or right hand by actively gripping handheld device; paragraph 0114, handheld device may include one or more fiducials).
Regarding claims 12-20, they are rejected based upon similar rational as above claims 1-10. Chang further discloses one or more computer-readable non-transitory non-volatile storage media embodying software that is operable when executed, by a computer system (paragraph 0148).
Regarding claim 21, it is rejected based upon similar rational as above claim 1. Chang further discloses a computing system comprising: one or more processors; and one or more computer-readable non-transitory non-volatile storage media coupled to one or more of the processors and comprising instructions operable when executed by the one or more of the processors to cause the system (paragraph 0150).
Claim(s) 11 is/are rejected under 35 U.S.C. 103 as being unpatentable over Chang in view of Fei as applied to claim 10 above, and further in view of Gupta et al., U.S. Patent Publication Number 2019/0250708 A1.
Regarding claim 11, Chang discloses paragraph 0113, user may hold handheld device in their left or right hand by actively gripping handheld device.
it is noted that Chang in view of Fei fails to disclose wherein estimating the second pose of the hand comprises applying an inverse of the estimated grip to the calculated second pose of the controller.
Gupta discloses estimating the second pose of the hand comprises applying an inverse of the estimated grip to the calculated second pose of the controller (paragraph 0096, during the mapping of the index and middle fingers, the effective angle of rotation in hand pose 2 (abduction) is much smaller than hand pose 6 (flexion); the vector component of quaternion corresponding to each hand pose is an inverse function).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include in the grip as disclosed by Chang, the inverse function for the grip as disclosed by Gupta, to accurately represent hand poses performed by the user.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Hesch et al., U.S. Patent Publication Number 2019/0033988 A1
Hesch discloses determining a first pose of a hand of a user holding a controller associated with a wearable device worn by a user in an environment based on a first image depicting at least a part of the hand (paragraph 0029, electronic device HMD includes a plurality of sensors to capture information about the pose (position and orientation) of the electronic and information about the pose of object in the local environment relative to the electronic device); determining a first pose of the controller based on the first pose of the hand and a first inertial measurement unit (IMU) data of the controller (paragraph 0025, electronic device is configured to received IMU data captured at the hand-held controller).
Wan et al., U.S. Patent Publication Number 2021/0239993 A1
Wan discloses paragraph 0036, determines its pose by performing multi-view analysis of the image data; can identify spatial features (e.g., anatomical features, marker and/or gestures associated with the hand-held controller) present in one or more of capture image frames of the image data; FIG. 5A and FIG. 5B; paragraph 0039, determines a grasp pattern in which the fingers of the hand are wrapped around the controller in approximately a “thumbs up” configuration) and a first inertial measurement unit (IMU) data of the controller (paragraph 0037, receives non-image sensor data, such as gyroscopic data, accelerometer data, and/or IMU data from the controller); paragraph 007, capture 3DoF readings for enabling detection of the relative pose of the controller; receives rotational information indicating rotational movement of the controller as the user moves, for example his hand; paragraph 0037, enabling detection of a relative pose; paragraph 0038, uses the 3DoF position information and 3DoF rotation information to correlate changes in position and orientation of spatial features (e.g., anatomical features such as the thumb or thumbnail, markers, and/or gestures) observed in one image frame with spatial feature observed in a subsequent image frame; paragraph 0038, based on positional information captured by the depth information (or in some embodiments, capture image and non-image sensor data) and rotation movement information provided by the controller, thereby extending 5DoF tracking to 3DoF controllers.
Alatise et al., “Pose Estimation of a Mobile Robot Based on Fusion of IMU Data and Vision Data Using an Extended Kalman Filter”, MDPI, Sensors, 21 September 2017; pages 1-22
Alatise discloses page 6, section 2.3., determine the pose estimation of an object; visual data was obtained; page 8, section 3.2.; tracks the pose of a mobile robot with a camera in relative to the referenced object; acquire images via camera; page 7, IMU method provides orientation of the body (b) with respect to (wrt) world frame; section 3.1., IMU-Based Pose Estimation; estimate the current pose and to reduce drifts and errors of the sensors; page 7, section 3.1., inertial sensors, accelerometer, gyroscope; page 7, section 3.1., IMU-Based Pose Estimation; inertial sensors which used a Kalman filter; page 8, section 3.2., estimate the 3D camera from the 2D image plane; the relationship between a point in the world frame and its projection in the image plane; page 8, calculate the angular rate; angular rate is integrated to determine the orientation from the gyroscope; page 8, section 3.2., calculate the pose as a function of the recognized object.
Balan et al., U.S. Patent Publication Number 2019/0325274 A1
Balan discloses paragraph 0022, acquire image data of the surrounding environment of a handheld object held by the user; paragraph 0024, handheld objects are depicted as controllers; FIG. 1; paragraph 0002, determining a pose of a handheld object in a surrounding environment; inertial measurement (IMU) data from an IMU of the handheld object is received; paragraph 0024, handheld object comprise inertial measurement unit (IMU) (which may include an accelerometer, gyroscope, magnetometer, and/or other suitable sensor) that provides output related to changes in position and orientation of the handheld objects; paragraph 0035, IMU data is received ; IMU data may be adjusted via intrinsic calibration data;
paragraph 0033, determine a pose of the handheld object with respect to the world; paragraph 0028, a relative-to-HMD pose (position and orientation) of the handheld object may be estimated by tracking the positions; data from the IMU on the handheld object further information tracking; paragraph 0039, predict an expected pose of the controller in a next frame based upon HMD motion and the handheld object pose.
Nienstedt et al., U.S. Patent Publication Number 2020/0026348 A1
Nienstedt discloses paragraph 0051, position and/or orientation of the handheld device is calculated based on the IMU data; paragraph 0050, IMU data; indicative of at least rotational movement of handheld device; may include linear accelerations; paragraph 0050-0051, may include linear accelerations or raw data from which linear accelerations may be calculated; position and/or orientation of handheld device with respect to wearable device is calculated (using previous known and/or estimated orientations);
paragraph 0030, estimation of the position and/or orientation of the handheld device with respect to the wearable device; estimated localization.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Motilewa Good-Johnson whose telephone number is (571)272-7658. The examiner can normally be reached Monday - Friday 6am-2:30pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jason Chan can be reached at 571-272-3022. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
MOTILEWA . GOOD JOHNSON
Primary Examiner
Art Unit 2616
/MOTILEWA GOOD-JOHNSON/Primary Examiner, Art Unit 2619
Prosecution Insights