Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on April 15, 2026 has been entered.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 21-40 are rejected under 35 U.S.C. 103 as being unpatentable over Yokokawa (US 2022/0163800) in view of Nienstedt et al (US 2020/0026348).
As per claim 21 Yokokawa discloses: A computer-implemented method, comprising:
accessing an image comprising information associated with a hand 502-902 of a user or a handheld device 212 & 504-904, wherein the image is captured by a first camera 208 associated with a headset 200 {[0033] The HMD 200 also may include one or more outward-oriented cameras 208 for imaging objects such as hands of the wearer of the HMD 200.};
generating a vision-based pose 1108 estimation {figure 11} for the handheld device 212 & 504-904 by processing the image {figure 3:306 & figures 5-9 [0044] FIG. 11 illustrates an example ML module or engine 1100 that may be used in which initial hand detection 1102 need not be used. Instead, as described previously left and right keypoint estimation stages 1104, 1106 (details of the right stage 1104 only shown for clarity) may receive multiple images 1108 of a hand holding a controller, cropped if desired and up-res′d using super-resolution if desired according to principles discussed elsewhere herein.};
generating a map-based pose 1112 & 1114 estimation {figure 11} for the handheld device 212 & 504-904 based at least in part upon one or more images captured using a second camera 208 associated with the handheld device 212 & 504-904 { [0036] Commencing at block 300 an image is received from, e.g., the camera 208 of a controller 212 that may be held by a human hand.}; and
generating a final pose 1118 estimation {figure 11} for the handheld device 212 & 504-904 based on the vision-based pose 1108 estimation and the map-based pose 1112 & 1114 estimation { [0045] The key NNs 1110 produce both two dimensional (2D) and 1D heatmaps 1112, 1114, from which keypoints 1116 are derived for altering the pose of a template hand 1118 according to the keypoints 1116. Model parameters are learned by optimizing min E(θ).}.
Regarding claim 21 Yokokawa is silent as to: generating a map-based pose estimation for the handheld device by processing one or more images captured by a second camera mounted on the handheld device to construct a spatial map of a physical environment and localize the handheld device within the spatial map. With respect to claim 21 Nienstedt et al depicts in figure 3 and discloses: generating a map-based pose estimation for the handheld device 304 by processing one or more images captured by a second camera 326 A/326B mounted on the handheld device to construct a spatial map of a physical environment and localize the handheld device 304 within the spatial map { [0066] At step 1006, an image (“world image”) is captured by front handheld imaging device 326A. The world image may contain surrounding features 344. & [0086] At step 1310, an image (“world image”) is captured by front handheld imaging device 326A containing surrounding features 344.}.
It would have been obvious to a person having ordinary skill in the art at the time the invention was effectively filed to the method of Yokokawa with generating a map-based pose estimation for the handheld device by processing one or more images captured by a second camera mounted on the handheld device to construct a spatial map of a physical environment and localize the handheld device within the spatial map as taught by Nienstedt et al. The rationale is as follows: one of ordinary skill in the art at the time the invention was effectively filed would have been motivated to a method with generating a map-based pose estimation for the handheld device by processing one or more images captured by a second camera mounted on the handheld device to construct a spatial map of a physical environment and localize the handheld device within the spatial map so as “to create a desired cumulative field of view.” See [0037] of Nienstedt et al.
As per claim 22 Yokokawa discloses: The method of claim 21, wherein generating the map-based pose estimation for the handheld device includes:
accessing a first set of images of the one or more images captured using the second camera 208 to build the spatial map of a portion of the physical environment, wherein the map-based pose 1112 & 1114 estimation for the handheld device 212 & 504-904 is based at least in part upon the map { [0047] With respect to the example heat map technique discussed herein, in one non-limiting implementation, K heatmaps of size W0×H0,{H1, H2, . . . , Hk} may be estimated, where each heatmap Hk indicates the location confidence of the kth keypoint of the virtual hand to be rendered. (K keypoints in total).}.
As per claim 23 Yokokawa discloses: The method of claim 21, wherein the map-based pose 1112 & 1114 estimation is generated using Simultaneous Localization and Mapping (SLAM) engine { [0047] “Efficient Object Localization Using Convolutional Networks”, Tompson et al., arXiv:1411.4280v3 (June, 2015) describes such an approach in which heatmaps are generated by running an image through multiple resolution banks in parallel to simultaneously capture features at a variety of scales. The output is a discrete heatmap instead of continuous regression. A heatmap predicts the probability of the joint occurring at each pixel. A multi-resolution CNN architecture (coarse heatmap model) is used to implement a sliding window detector to produce a coarse heatmap output. This is but one example heatmap technique that may be used. Note: SLAM has not been set forth with any specificity to distinguish over the applied prior art}.
As per claim 24 Yokokawa discloses: The method of claim 21, wherein the vision-based pose 1108 estimation is based at least in part upon sensor data from a sensor associated with the handheld device 212 & 504-904 { [0038] Proceeding to block 308, the cropped image can be analyzed to determine the pose of the hand based on both the image and the signals from the controller sensors 216, 218.}.
As per claim 25 Yokokawa discloses: The method of claim 21, wherein the vision-based pose 1108 estimation is based at least in part upon metadata associated with the image { [0034] The controller 212 may include one or more processors 220 configured to send signals from the sensors 216, 218 and control key 214 to other components using one or more network interfaces 222. The controller may also include one or more position sensors 223 such as inertial sensors, global positioning satellite sensors, accelerometers, magnetometers, gyroscopes, and combinations thereof.}.
As per claim 26 Yokokawa discloses: The method of claim 21, wherein at least one of the vision-based pose 1108 estimation, the map-based pose 1112 & 1114 estimation, or the final pose 1118 estimation comprises information for six degrees of freedom (6DoF) associated with the handheld device 212 & 504-904 { [0003] Tracking a hand based on sensors on a controller can yield “dead zones” for parts of the hands that are not located near a sensor and for parts of the hand such as the thumb that can assume a wide degree of freedom of movement. Note: it is seen that the handheld device has six degrees of freedom}.
As per claim 27 Yokokawa discloses: The method of claim 21, wherein the vision-based pose 1108 estimation is generated using a machine-learning model { [0041] Indeed, FIG. 3 illustrates that the cropped region of the controller with hand may be input to a ML module at block 308, with corresponding touch signals from the controller sensors 216, 218 generated at the same time the image was generated being input to the ML module at block 310. The ML module uses both the sensor signals and controller/hand image to output at block 312 a virtual image of a complete hand in the same pose as it is in grasping the controller in the cropped region generated at block 306. The virtual image is presented on a display such as the HMD 200 at block 314.}.
As per claim 28 Yokokawa discloses: The method of claim 27, wherein the machine-learning model comprises a neural network { [0044] The images 1108 may be processed through key neural networks 1110, such as but not limited to convolutional neural networks (CNN).}.
As per claim 29 Yokokawa discloses: The method of claim 21, wherein the final pose 1118 estimation is used as user input { [0038] Proceeding to block 308, the cropped image can be analyzed to determine the pose of the hand based on both the image and the signals from the controller sensors 216, 218. & Figure 11}.
As per claim 30 Yokokawa discloses: A system, comprising:
one or more processors 24; and
one or more memories 28 coupled to at least one of the one or more processors 24, wherein the one or more memories 28 comprise computer-readable program instructions, which when executed by at least one of the one or more processors 24, cause the system to:
access an image comprising information associated with a hand 502-902 of a user or a handheld device 212 & 504-904, wherein the image is captured by a first camera 208 associated with a headset 200 {[0033] The HMD 200 also may include one or more outward-oriented cameras 208 for imaging objects such as hands of the wearer of the HMD 200.};
generate a vision-based pose 1108 estimation for the handheld device 212 & 504-904 by processing the image {figure 3:306 & figures 5-9};
generate a map-based pose 1112 & 1114 estimation for the handheld device 212 & 504-904 based at least in part upon one or more images captured using a second camera 208 associated with the handheld device 212 & 504-904 { [0036] Commencing at block 300 an image is received from, e.g., the camera 208 of a controller 212 that may be held by a human hand.}; and
generate a final pose 1118 estimation for the handheld device 212 & 504-904 based on the vision-based pose 1108 estimation and the map-based pose 1112 & 1114 estimation {[0044] FIG. 11 illustrates an example ML module or engine 1100 that may be used in which initial hand detection 1102 need not be used. Instead, as described previously left and right keypoint estimation stages 1104, 1106 (details of the right stage 1104 only shown for clarity) may receive multiple images 1108 of a hand holding a controller, cropped if desired and up-res′d using super-resolution if desired according to principles discussed elsewhere herein. The images 1108 may be processed through key neural networks 1110, such as but not limited to convolutional neural networks (CNN).}.
Regarding claim 30 Yokokawa is silent as to: generating a map-based pose estimation for the handheld device by processing one or more images captured by a second camera mounted on the handheld device to construct a spatial map of a physical environment and localize the handheld device within the spatial map. With respect to claim 30 Nienstedt et al depicts in figure 3 and discloses: generating a map-based pose estimation for the handheld device 304 by processing one or more images captured by a second camera 326 A/326B mounted on the handheld device to construct a spatial map of a physical environment and localize the handheld device 304 within the spatial map { [0066] At step 1006, an image (“world image”) is captured by front handheld imaging device 326A. The world image may contain surrounding features 344. & [0086] At step 1310, an image (“world image”) is captured by front handheld imaging device 326A containing surrounding features 344.}.
It would have been obvious to a person having ordinary skill in the art at the time the invention was effectively filed to the method of Yokokawa with generating a map-based pose estimation for the handheld device by processing one or more images captured by a second camera mounted on the handheld device to construct a spatial map of a physical environment and localize the handheld device within the spatial map as taught by Nienstedt et al. The rationale is as follows: one of ordinary skill in the art at the time the invention was effectively filed would have been motivated to a method with generating a map-based pose estimation for the handheld device by processing one or more images captured by a second camera mounted on the handheld device to construct a spatial map of a physical environment and localize the handheld device within the spatial map so as “to create a desired cumulative field of view.” See [0037] of Nienstedt et al.
As per claim 31 Yokokawa discloses: The system of claim 30, wherein the instructions, that cause the system to generate the map-based pose estimation for the handheld device include instructions that cause the system to:
access a first set of images of the one or more images captured using the second camera 208 to build the spatial map of a portion of the physical environment, wherein the map-based pose 1112 & 1114 estimation for the handheld device 212 & 504-904 is based at least in part upon the spatial map { [0047] With respect to the example heat map technique discussed herein, in one non-limiting implementation, K heatmaps of size W0×H0,{H1, H2, . . . , Hk} may be estimated, where each heatmap Hk indicates the location confidence of the kth keypoint of the virtual hand to be rendered. (K keypoints in total).}.
As per claim 32 Yokokawa discloses: The system of claim 30, wherein the vision-based pose 1108 estimation is based at least in part upon sensor data from a sensor associated with the handheld device 212 & 504-904 { [0038] Proceeding to block 308, the cropped image can be analyzed to determine the pose of the hand based on both the image and the signals from the controller sensors 216, 218.}.
As per claim 33 Yokokawa discloses: The system of claim 30, wherein at least one of the vision-based pose 1108 estimation, the map-based pose 1112 & 1114 estimation, or the final pose 1118 estimation comprises information for six degrees of freedom (6DoF) associated with the handheld device 212 & 504-904 { [0003] Tracking a hand based on sensors on a controller can yield “dead zones” for parts of the hands that are not located near a sensor and for parts of the hand such as the thumb that can assume a wide degree of freedom of movement. Note: it is seen that the handheld device has six degrees of freedom}.
As per claim 34 Yokokawa discloses: The system of claim 30, wherein the vision-based pose 1108 estimation is generated using a machine-learning model { [0041] Indeed, FIG. 3 illustrates that the cropped region of the controller with hand may be input to a ML module at block 308, with corresponding touch signals from the controller sensors 216, 218 generated at the same time the image was generated being input to the ML module at block 310. The ML module uses both the sensor signals and controller/hand image to output at block 312 a virtual image of a complete hand in the same pose as it is in grasping the controller in the cropped region generated at block 306. The virtual image is presented on a display such as the HMD 200 at block 314.}.
As per claim 35 Yokokawa discloses: The system of claim 30, wherein the final pose 1118 estimation is used as user input { [0038] Proceeding to block 308, the cropped image can be analyzed to determine the pose of the hand based on both the image and the signals from the controller sensors 216, 218. & Figure 11}.
As per claim 36 Yokokawa discloses: A non-transitory computer-readable storage medium including computer- readable instructions embodied therein, which when executed by one or more processors 24, cause a computer system to:
access an image comprising information associated with a hand 502-902 of a user or a handheld device 212 & 504-904, wherein the image is captured by a first camera 208 associated with a headset 200 {[0033] The HMD 200 also may include one or more outward-oriented cameras 208 for imaging objects such as hands of the wearer of the HMD 200.};
generate a vision-based pose 1108 estimation for the handheld device 212 & 504-904 by processing the image {figure 3:306 & figures 5-9};
generate a map-based pose 1112 & 1114 estimation for the handheld device 212 & 504-904 based at least in part upon one or more images captured using a second camera 208 associated with the handheld device 212 & 504-904 { [0036] Commencing at block 300 an image is received from, e.g., the camera 208 of a controller 212 that may be held by a human hand.}; and
generate a final pose 1118 estimation for the handheld device 212 & 504-904 based on the vision-based pose 1108 estimation and the map-based pose 1112 & 1114 estimation {[0044] FIG. 11 illustrates an example ML module or engine 1100 that may be used in which initial hand detection 1102 need not be used. Instead, as described previously left and right keypoint estimation stages 1104, 1106 (details of the right stage 1104 only shown for clarity) may receive multiple images 1108 of a hand holding a controller, cropped if desired and up-res′d using super-resolution if desired according to principles discussed elsewhere herein. The images 1108 may be processed through key neural networks 1110, such as but not limited to convolutional neural networks (CNN).}.
Regarding claim 36 Yokokawa is silent as to: generating a map-based pose estimation for the handheld device by processing one or more images captured by a second camera mounted on the handheld device to construct a spatial map of a physical environment and localize the handheld device within the spatial map. With respect to claim 36 Nienstedt et al depicts in figure 3 and discloses: generating a map-based pose estimation for the handheld device 304 by processing one or more images captured by a second camera 326 A/326B mounted on the handheld device to construct a spatial map of a physical environment and localize the handheld device 304 within the spatial map { [0066] At step 1006, an image (“world image”) is captured by front handheld imaging device 326A. The world image may contain surrounding features 344. & [0086] At step 1310, an image (“world image”) is captured by front handheld imaging device 326A containing surrounding features 344.}.
It would have been obvious to a person having ordinary skill in the art at the time the invention was effectively filed to the method of Yokokawa with generating a map-based pose estimation for the handheld device by processing one or more images captured by a second camera mounted on the handheld device to construct a spatial map of a physical environment and localize the handheld device within the spatial map as taught by Nienstedt et al. The rationale is as follows: one of ordinary skill in the art at the time the invention was effectively filed would have been motivated to a method with generating a map-based pose estimation for the handheld device by processing one or more images captured by a second camera mounted on the handheld device to construct a spatial map of a physical environment and localize the handheld device within the spatial map so as “to create a desired cumulative field of view.” See [0037] of Nienstedt et al.
As per claim 37 Yokokawa discloses: The non-transitory computer-readable storage medium of claim 36, wherein the instructions that cause the computer system to generate the map-based pose estimation for the handheld device include instructions that cause the computer system to: access a first set of images of the one or more images captured using the second camera 208 to build the spatial map of a portion of the physical environment, wherein the map-based pose 1112 & 1114 estimation for the handheld device 212 & 504-904 is based at least in part upon the spatial map { [0047] With respect to the example heat map technique discussed herein, in one non-limiting implementation, K heatmaps of size W0×H0,{H1, H2, . . . , Hk} may be estimated, where each heatmap Hk indicates the location confidence of the kth keypoint of the virtual hand to be rendered. (K keypoints in total).}.
As per claim 38 Yokokawa discloses: The non-transitory computer-readable storage medium of claim 36, wherein the vision-based pose 1108 estimation, the map-based pose 1112 & 1114 estimation, or the final pose 1118 estimation comprises information for six degrees of freedom (6DoF) associated with the handheld device 212 & 504-904 { [0003] Tracking a hand based on sensors on a controller can yield “dead zones” for parts of the hands that are not located near a sensor and for parts of the hand such as the thumb that can assume a wide degree of freedom of movement. Note: it is seen that the handheld device has six degrees of freedom}.
As per claim 39 Yokokawa discloses: The non-transitory computer-readable storage medium of claim 36, wherein the vision-based pose 1108 estimation is generated using a machine-learning model { [0041] Indeed, FIG. 3 illustrates that the cropped region of the controller with hand may be input to a ML module at block 308, with corresponding touch signals from the controller sensors 216, 218 generated at the same time the image was generated being input to the ML module at block 310. The ML module uses both the sensor signals and controller/hand image to output at block 312 a virtual image of a complete hand in the same pose as it is in grasping the controller in the cropped region generated at block 306. The virtual image is presented on a display such as the HMD 200 at block 314.}.
As per claim 40 Yokokawa discloses: The non-transitory computer-readable storage medium of claim 36, wherein the final pose 1118 estimation is used as user input { [0038] Proceeding to block 308, the cropped image can be analyzed to determine the pose of the hand based on both the image and the signals from the controller sensors 216, 218. & Figure 11}.
Response to Arguments
Applicant’s arguments with respect to the claims have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DAVID D DAVIS whose telephone number is (571)272-7572. The examiner can normally be reached Monday - Friday, 8 a.m. - 4 p.m..
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Ke Xiao can be reached at 571-272-7776. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DAVID D DAVIS/Primary Examiner, Art Unit 2627
DDD
Prosecution Insights