DETAILED ACTION
This action is responsive to 06/29/2025.
Claims 1, 3-14, and 16-20 are pending.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Claim Objections
Claims 13, 14, and 19 are objected to because of the following informalities:
In line 4 of claim 13 and line 3 of claim 19, change “the carrier” to “the movable carrier”.
In line 2 of claim 14, insert a comma after “display device” to separate “a head-mounted display device” from “a first motion sensor”.
Appropriate correction is required.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1, 4-7, 11-14, and 17-20 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Gorur Sheshagiri et al. (US Pub. 2020/0271450), hereinafter Sheshagiri.
Regarding claim 1, Sheshagiri discloses a method for determining a relative pose (see, for example, fig. 12), which is used for determining a relative pose of a head-mounted display device relative to a movable carrier (see, for example, figs. 4A-4B with description in [0144]-[0147], which discloses a user 404 wearing a head mounted display(HMD 408), and, a method of calculating relative pose information relative to a mobile platform 402 (movable carrier)), the head-mounted display device being provided with an image sensor (the one or more sensors 130 on the HMD 408 can include an image sensor 134-see figs. 4A-4B and [0150]) and a first motion sensor (IMU sensor 132 on the HMD-see [0097]), the movable carrier is provided with a second motion sensor (sensors of the car 402 can include inertial measurement unit (IMU) sensors 132A and 132B-see fig. 4A and [0146]), and the head-mounted display device being located inside the movable carrier (see fig. 4B-HMD 408), the method comprising: acquiring a current image (sensor data (e.g., 208-see fig. 3 and [0097]) can include image data from image sensor 134 on the HMD (see [0097] and [0150])), first inertial data, and second inertial data respectively by the image sensor (the one or more sensors 130 on the HMD 408 can be used to calculate motion parameters associated with the user 404-see [0149]. Inertia data can be measured different times, e.g., at first time t1 (see fig. 3 and [0141]) and at second time t2 (see fig. 3 and [0142])), the first motion sensor (i.e., data from one or more IMUs 132 on an HMD-see [0097] and [0149]), and the second motion sensor (i.e., data from IMU sensors 132 on the car-see [0097] and [0146]), at a current time (the sensor data can be captured at a first location at time t1 (see fig. 3 and [0141]) and at a second location at time t2 (see fig. 3 and [0142])); predicting a first pose of the head-mounted display device relative to the movable carrier based on the first inertial data and the second inertial data (motion parameters calculated by the IMU sensors132A and 132B can be used to calculate a pose of a user 404, e.g., relative to the car 402 and/or an outside of the car 402 (see [0147]), and image sensors 134B can be used to capture one or more images of the user 404 and/or the HMD 408, which can be used to calculate a head pose of the user 404 (see [0148]), or the one or more sensors 130 on the HMD 408 can be used to calculate motion parameters associated with the user 404 … the motion parameters can be relative to the car 402 and/or external environment of the car-see [0149]); predicting a second pose of the head-mounted display device relative to the movable carrier at the current time, based on the current image and a pre-constructed offline map model of the movable carrier (see fig. 3 and, for example, [0140]-[0143], which illustrates a mobile map 304 moving within a global map 302, wherein image sensor 208 within the global map 304 can detect features 306 within the mobile map 304 and features 308 outside of the mobile map but within the global map 302, and the tracked features can be used to adapt relative pose information used to provide virtual content and XR experiences to a user with a mobile platform associated with the mobile map 304); fusing the first pose with the second pose (data from the sensors 132A-B and 134A-B on the car 402 and/or the sensors 130 on the HMD408 can be fused for greater accuracy in calculating pose information (see, for example, [0123] and [0151]). Fusion can be done using, for example, extended Kalman filter (EKF) model, Kalman filters, or any other suitable tracking models and/or algorithms (see [0122], [0129], and [0184]); and determining the relative pose of the head-mounted display device relative to the movable carrier at the current time (see, for example, figs. 2B and 12, steps 1206-1210 with description in [0120]-[0139], and [0199]-[0202]).
Regarding claim 14, Sheshagiri discloses an extended reality system (see figs. 1-4), comprising a head-mounted display device (a head mounted display device (HMD 408)-see fig. 4A), a first motion sensor (IMU sensor on the HMD-see [0097]), a second motion sensor arranged in a movable carrier (IMU sensors 132 on the car (movable carrier)-see fig. 4A-4B and [0097), and an apparatus for determining a relative pose of a head- mounted display device relative to the movable carrier (see, for example, fig. 2B-pose estimation engine 104), wherein the head-mounted display device is provided with an image sensor (image sensor 134 on the HMD-see, for example, [0097]), and the head-mounted display device being located inside the movable carrier (see fig. 4A), the extended reality system is configured to: acquire a current image (sensor data (e.g., 208-see fig. 3 and [0097]) can include image data from image sensor 134 on the HMD (see [0097] and [0150])), first inertial data and second inertial data respectively by the image sensor (the one or more sensors 130 on the HMD 408 can be used to calculate motion parameters associated with the user 404-see [0149]. Inertia data can be measured different times, e.g., at first time t1 (see fig. 3 and [0141]) and at second time t2 (see fig. 3 and [0142])), the first motion sensor (i.e., data from one or more IMUs 132 on an HMD-see [0097] and [0149]) and the second motion sensor (i.e., data from IMU sensors 132 on the car-see [0097] and [0146]), at a current time (the sensor data can be captured at a first location at time t1 (see fig. 3 and [0141]) and at a second location at time t2 (see fig. 3 and [0142])); predict a first pose of the head-mounted display device relative to the movable carrier based on the first inertial data and the second inertial data (motion parameters calculated by the IMU sensors132A and 132B can be used to calculate a pose of a user 404, e.g., relative to the car 402 and/or an outside of the car 402 (see [0147]), and image sensors 134B can be used to capture one or more images of the user 404 and/or the HMD 408, which can be used to calculate a head pose of the user 404 (see [0148]), or the one or more sensors 130 on the HMD 408 can be used to calculate motion parameters associated with the user 404 … the motion parameters can be relative to the car 402 and/or external environment of the car-see [0149]); predict a second pose of the head-mounted display device relative to the movable carrier at the current time, based on the current image and a pre-constructed offline map model of the movable carrier (see fig. 3 and, for example, [0140]-[0143], which illustrates a mobile map 304 moving within a global map 302, wherein image sensor 208 within the global map 304 can detect features 306 within the mobile map 304 and features 308 outside of the mobile map but within the global map 302, and the tracked features can be used to adapt relative pose information used to provide virtual content and XR experiences to a user with a mobile platform associated with the mobile map 304); fuse the first pose with the second pose (data from the sensors 132A-B and 134A-B on the car 402 and/or the sensors 130 on the HMD408 can be fused for greater accuracy in calculating pose information (see, for example, [0123] and [0151]). Fusion can be done using, for example, extended Kalman filter (EKF) model, Kalman filters, or any other suitable tracking models and/or algorithms (see [0122], [0129], and [0184]); and determine the relative pose of the head-mounted display device relative to the movable carrier at the current time (see, for example, figs. 2B and 12, steps 1206-1210 with description in [0120]-[0139], and [0199]-[0202]).
Regarding claims 4 and 17, Sheshagiri discloses wherein predicting a second pose of the head-mounted display device relative to the movable carrier at the current time based on the current image and a pre-constructed offline map model comprises: obtaining the offline map model of the movable carrier, the offline map model comprising predetermined reference feature points of the movable carrier and reference visual descriptors thereof; extracting a 2D feature point and a corresponding visual descriptor from the current image; matching the visual descriptor corresponding to the 2D feature point with the reference visual descriptors to determine a reference feature point corresponding to the 2D feature point from the offline map model; and determining the second pose based on the 2D feature point and the reference feature point corresponding thereto (see, for example, figs. 2B and 3 with description in, for example, [0140]-[0143], which discloses a mobile map 304 and a global map 302, wherein, an image sensor 208 within the mobile map 304 can detect features 306 within the mobile map 304 and features 308 outside of the mobile map 304 but within the global map 302, and will continue to track these features while the mobile map is at a first location and the features 306 and 308 are within range and visibility. The tracked features can be used to adapt relative pose information used to provide virtual content and XR experiences to a user within a mobile platform associated with the mobile map 304).
Regarding claims 5 and 18, Sheshagiri discloses wherein determining a reference feature point corresponding to the 2D feature point from the offline map model based on the visual descriptor corresponding to the 2D feature point and the reference visual descriptors comprises: determining the reference feature point corresponding to the 2D feature point based on degrees of similarity between the visual descriptor corresponding to the 2D feature point and the reference visual descriptors in the offline map model, respectively (see, for example, [0075], which discloses that content synthesis engine 112 can use information about one or more frames of reference to match, map, or synchronize features in content, objects, and/or real-world environments (or maps of real-world environments such as mobile maps and global maps), model objects and/or scenes with merged perspectives, produce realistic spatio-temporal content, incorporate motion dynamics of an environment, etc.).
Regarding claims 6 and 19, Sheshagiri discloses wherein the offline map model is constructed in the following manner: obtaining a sample image of the movable carrier and 3D coordinates of spatial points in the movable carrier (see, for example, fig. 3 with description in [0070]-[0072], which discloses mapper 108 that can generate a mobile map that charts, plots, models, or identifies objects, space, and/or characteristics (e.g., shape, volume, size, position, etc.) of the mobile platform, and a global map that chats, plots, models or identifies objects, space, and/or characteristics of the outside environment … the mobile map and the global map can be 2D or 3D grid or models of the mobile platform and the outside environment); calculating a point cloud model of the movable carrier based on the obtained 3D coordinates of the spatial points in the movable carrier (see figs. 6-9. One or more external sources 128 (145-see fig. 1) can provide this information in the form of a public and/or private cloud-see [0063]); and associating the sample image with the point cloud model (see fig. 1 with description in [0060] and [0076]-[0078]); and determining an image descriptor corresponding to a point in the point cloud model to obtain point cloud data with the image descriptor, thereby constructing the offline map model, wherein the point in the point cloud model and the image descriptor thereof are respectively used as a reference feature point and a reference visual descriptor (see, for example, [0071]-mapper 108 can use data from tracker 106 to identify the location of mobile maps or correct artifacts in a mobile map caused by movement of an associated mobile platform. The mobile mapper can perform operations to map virtual objects or content to features in a map of the mobile platform (e.g., a mobile map) and/or map of the outside environment (e.g., a global map)).
Regarding claim 7, Sheshagiri discloses wherein, after obtaining point cloud data with the image descriptor and before constructing the offline map model, the method further comprises: obtaining a structural model of the movable carrier (see fig. 2C which illustrates a mapper 108, and, for example, fig. 8B with description in [0167]-[0169], which illustrates an example view 810 of a mobile platform 812, particularly, both internal and external features of the mobile platform, which are mapped into a mobile map of the mobile platform or car 812); registering the point cloud model with the structural model to determine an interior point in the point cloud model (see fig. 8B); and updating coordinates of the interior point in the point cloud based on coordinates corresponding to the interior point in the structural model (see fig. 2A with description in [0135]-[0139]-mapper 108 can receive segmented image (and/or corresponding information or analysis) to generate, select and/or retrieve one or more maps).
Regarding claim 8, Sheshagiri discloses wherein the method further comprises: registering the point cloud model with the structural model to determine an exterior point in the point cloud data; and removing the exterior point from the point cloud model (see fig. 8B with further description in [0167]-[0169], wherein the mobile platform is a car (mobile carrier), the view 810 depicts an inside view of the car 812 and features 814 and 816 visible from the car, wherein, wherein features 814can include visual features outside of a mobile map associated with the car 812 (see [0167), and features 816 can include visual features inside of the mobile map of the car 812, i.e., exterior and interior features are separately mapped).
Regarding claims 11 and 20, Sheshagiri discloses wherein the sample image comprises images obtained by photographing the movable carrier from a plurality of viewing angles and/or in a plurality of lighting environments (i.e., one or more image sensors 134 can capture image and/or video data-see figs. 1, 3-4, and 8B-9 with description in [0062], [0097]. The plurality of cameras 134 (see figs. 4A-4B) necessarily take images from different viewing angles); and determining an image descriptor corresponding to a point in the point cloud model to obtain point cloud data with the image descriptor comprises: determining image descriptors corresponding to the point in the point cloud model respectively from a plurality of viewing angles and/or in a plurality of lighting environments to obtain the point cloud data (see, for example, figs. 1, 2C, and 9, with further description in [0076]-[0077], which disclose an environment classifier 114 of the virtual content processing system 102 to identify and classify a user’s environment, or whether a user has entered or is entering a different environment. Different environments necessarily have different lighting situations).
Regarding claim 12, Sheshagiri discloses further comprising: processing the relative pose and the structural model by the head-mounted display device to generate a projected image (the virtual content processing system 102 can use these images and measurements to compute the pose of the user and the pose of the car, and provide a realistic and immersive extended reality (XR) experience for the user in the car-see [0097]).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 3, 13, and 16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Sheshagiri in view of Rogers et al. (US Pub. 2021/0055545), hereinafter Rodgers.
Regarding claims 3 and 16, Sheshagiri discloses wherein predicting a first pose of the head-mounted display device relative to the movable carrier based on the first inertial data and the second inertial data comprises: predicting the first pose of the head-mounted display device relative to the movable carrier at the current time based on the first inertial data, the second inertial data as in claim 1 above.
However, Sheshagiri does not appear to expressly disclose and a historical pose of the head-mounted display device relative to the movable carrier at the previous time.
Rodgers, in for example, [0008]-[0011], [0096], and fig. 8, teaches a computer-implemented method for predicting poses in a virtual reality environment using historical data and other sensor information (see [0046]-[0047]).
Therefore, it would have been obvious to a person of ordinary skill in the art before the effectively filing date of the claimed invention to incorporate teachings of Rodgers with the invention of Sheshagiri by using historical data to determine a pose a virtual reality environment, as taught by Rodgers, in order to provide an advantage of improved accuracy and reduction of perceived errors (see [0034]).
regarding claim 13, Rodgers is further relied upon to teach further comprising: using the relative pose as a historical pose of the head-mounted display device relative to the movable carrier at next time (see, for example, [0008]-[0011], [0096], and fig. 8, which teaches a computer-implemented method for predicting poses in a virtual reality environment using historical data and other sensor information (see [0046]-[0047])).
Claim 9 is/are rejected under 35 U.S.C. 103 as being unpatentable over Sheshagiri in view of Yin et al. (US Pub. 2022/0182593), hereinafter Yin.
Regarding claim 9, Sheshagiri does not appear expressly disclose wherein the interior point and the exterior point are determined in the following manner: registering the point cloud model with the structural model to determine from the structural model a similar point that is closest to the point in the point cloud model; and if a distance between the similar point and the point in the point cloud model is greater than a preset threshold, determining the point in the point cloud model as the exterior point; or if a distance between the similar point and the point in the point cloud model is not greater than the preset threshold, determining the point in the point cloud model as the interior point.
Yin, in for example, [0107], teaches an iteration where, for each point of a source point cloud, finding a nearest point by using a K-D tree to obtain a corresponding point-to-point set, then removing a point and its nearest point whose distance is greater than a threshold to obtain optimized point-to-point set.
Therefore, it would have been obvious to a person of ordinary skill in the art before the effectively filing date of the claimed invention to incorporate teachings of Yin with the invention of Sheshagiri differentiating between an interior point and an exterior point in a point cloud model based on a distance between similar points meeting a threshold requirement, as taught by Yin, which constitutes combining prior art elements according to known methods to yield predictable results.
Allowable Subject Matter
Claim 10 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter: The limitations recited in claim 10 are not taught or suggested by any of the applied references.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SARDIS F AZONGHA whose telephone number is (571)270-7706. The examiner can normally be reached 10AM-7:00PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Ke Xiao can be reached at (571)272-7776. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SARDIS F AZONGHA/Primary Examiner, Art Unit 2627
Prosecution Insights