Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the response to this Office action, the Office respectfully requests that support be shown for language added to any original claims on amendment and any new claims. That is, indicate support for newly added claim language by specifically pointing to page(s) and line numbers in the specification and/or drawing figure(s). This will assist the Office in prosecuting this application.
The Office has cited particular figures, elements, paragraphs and/or columns and line numbers in the references as applied to the claims for the convenience of the applicant. Although the specified citations are representative of the teachings in the art and are applied to the specific limitations within the individual claim, other passages and figures may apply as well. It is respectfully requested from the applicant, in preparing the responses, to fully consider each of the cited references in entirety as potentially teaching all or part of the claimed invention, as well as the context of the passage disclosed by the Office.
Status of Claims
- Applicant’s Preliminary Amendment filed June 29, 2025 is acknowledged.
- Claim(s) 3, 5-8, 11-13, 15 is/are amended
- Claim(s) 14 is/are canceled
- Claim(s) 16-21 is/are new
- Claim(s) 1-13, 15-21 is/are pending in the application.
Priority
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
The application is a U.S. National Phase Application of PCT International Application No. PCT/CN2023/140605 filed on December 21, 2023.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on June 29, 2025 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Specification
The specification has not been checked to the extent necessary to determine the presence of all possible minor errors. Applicant’s cooperation is requested in correcting any errors of which applicant may become aware in the specification.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 1-13, 15-21 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention.
Claim 1 recites “ A method for determining a relative pose, wherein an auxiliary positioning apparatus and a target head-mounted display device corresponding to the auxiliary positioning apparatus are located inside a movable carrier, the auxiliary positioning apparatus being configured to be fixedly connected to the carrier and being provided with an image sensor and a first inertial measurement unit, and the target head-mounted display device being provided with a second inertial measurement unit, the method comprising: obtaining an image, captured by the image sensor, of a user wearing the target head- mounted display device, first inertial data captured by the first inertial measurement unit, and second inertial data captured by the second inertial measurement unit; determining, based on the image, pose data of the target head-mounted display device relative to the auxiliary positioning apparatus; performing data fusion on the first inertial data, the second inertial data, and the pose data, and performing fusion detection; and when the fusion detection passes, determining a relative pose of the target head-mounted display device relative to the carrier based on the first inertial data, the second inertial data, and the pose data”.
Initially, the claim requires “pose data of the target head-mounted display device relative to the auxiliary position apparatus” to be determined based on the image data. Applicant’s disclosure has not provided a description of how said “pose data” is determined based on the image data. How is the pose data determined? What corresponds to “pose data”? How does the “pose data” based on the image differ from “a relative pose” in the preamble? How does “pose data” based on the image differ from “a relative pose of the target head-mounted display device relative to the carrier”?
Examiner is unable to discern what corresponds to “performing data fusion on the first inertial data, the second inertial data, and the pose data “ What type of data is “pose data” which is “based on the image data”? How is derived “pose data” fused with inertial data?
Specifically, how is “fusion detection” performed? How is a determination of “detection passes” made? How is a relative pose determined? What steps, methods, algorithms are used? Applicant’s disclosure has not provided any specific examples of how these recited features are accomplished but merely states that “pose data” is determined, “performing data fusion”, “performing fusion detection”, “fusion detection passes”, “determine relative pose”.
Claim 2 recites “wherein performing data fusion on the first inertial data, the second inertial data, and the pose data, and performing fusion detection comprises: determining a difference between the first inertial data and the second inertial data; determining a confidence based on a residual between the pose data and the difference; and in the case where a determined confidence is greater than or equal to a preset confidence threshold, determining that the fusion detection passes”
The features of claim 2 do not provide any further assistance in determining how “data fusion” is performed and introduces yet further ambiguities in reciting “a residual between the pose data and the difference”. What corresponds to “a residual”?
Claim 3 recites “wherein determining, based on the image, pose data of the target head-mounted display device relative to the auxiliary positioning apparatus comprises: when a plurality of head-mounted display devices are included in the image, respectively determining poses of the plurality of head-mounted display devices relative to the auxiliary positioning apparatus to obtain a plurality of candidate poses.”
Applicant’s disclosure has not provided any specific examples of how these recited features are accomplished but merely states that “determining poses of the plurality of head-mounted display devices relative to the auxiliary positioning apparatus to obtain a plurality of candidate poses”. How are different candidate poses determined?
Claim 4 recites “wherein when a plurality of head- mounted display devices are included in the image, respectively determining poses of the plurality of head-mounted display devices relative to the auxiliary positioning apparatus to obtain a plurality of candidate poses comprises: identifying the image using a pre-trained identification model to determine at least one user included in the image; and for a user of the identified at least one user, in response to the user wearing a head- mounted display device, determining a head pose of the user as a candidate pose corresponding to the user, to obtain the plurality of candidate poses”
Applicant’s disclosure has not provided any specific examples of how these recited features are accomplished but merely states that “determining poses of the plurality of head-mounted display devices relative to the auxiliary positioning apparatus to obtain a plurality of candidate poses”. How are different candidate poses determined?
Claim 5 recites “wherein the auxiliary positioning apparatus is in one-to-one correspondence with the target head-mounted display device; and performing data fusion on the first inertial data, the second inertial data, and the pose data, and performing fusion detection comprises: performing data fusion on the plurality of candidate poses respectively with the first inertial data and the second inertial data to determine confidences of the plurality of candidate poses; and when there is a confidence greater than or equal to a preset first confidence threshold, determining that the fusion detection passes”
Applicant’s disclosure has not provided any specific examples of how these recited features are accomplished but merely states that “determining poses of the plurality of head-mounted display devices relative to the auxiliary positioning apparatus to obtain a plurality of candidate poses”. How are different candidate poses determined? How is a confidence determined? What steps, methods, algorithms are used?
Claim 6 recites “wherein before determining a relative pose of the target head-mounted display device relative to the carrier based on the first inertial data, the second inertial data, and the pose data, the method further comprises: determining a candidate pose with a highest confidence as the pose data of the target head-mounted display device relative to the auxiliary positioning apparatus.”
Applicant’s disclosure has not provided any specific examples of how these recited features are accomplished but merely states that “determining poses of the plurality of head-mounted display devices relative to the auxiliary positioning apparatus to obtain a plurality of candidate poses”. How are different candidate poses determined? How is a confidence determined? What steps, methods, algorithms are used to determine determining a candidate pose with a highest confidence? What features correspond to “a highest confidence”?
Claim 7 recites “further comprising: abandoning tracking of the head pose of the user if the confidence of the candidate pose corresponding to the user is less than a preset second confidence threshold.”
Applicant’s disclosure has not provided any specific examples of how these recited features are accomplished but merely states that “abandoning… if the confidence of the candidate pose corresponding to the user is less than a preset second confidence threshold” How is confidence determined? How is a second confidence threshold determined/selected?
Claim 8 recites “wherein the auxiliary positioning apparatus corresponds to a plurality of target head-mounted display devices; and the second inertia data is obtained by: obtaining inertial data captured respectively by a plurality of second inertial measurement units corresponding to the plurality of head-mounted display devices to obtain a plurality of sets of candidate inertial data as the second inertial data.”
This claim appears to require acquiring additional data to obtain addition sets of candidate inertial data, but does not appear to perform any function with the acquired data. What is accomplished by obtaining additional inertial data from a plurality of target… to obtain a plurality of set of candidate inertial data as the second inertial data? How are sets of candidate inertial data used?
Claim 9 recites “wherein performing data fusion on the first inertial data, the second inertial data, and the pose data, and performing fusion detection comprises: fusing the first inertial data respectively with each set of candidate inertial data of the plurality of sets of candidate inertial data, and each candidate pose of the plurality of candidate poses, to determine a plurality of candidate confidences corresponding to each candidate pose of the plurality of candidate poses; and when there is a confidence greater than or equal to a preset third confidence threshold, determining that the fusion detection passes.”
What steps, methods, algorithms are used? Applicant’s disclosure has not provided any specific examples of how these recited features are accomplished.
Claim 10 recites “wherein determining a relative pose of the target head-mounted display device relative to the carrier based on the first inertial data, the second inertial data, and the pose data comprises: determining a head-mounted display device corresponding to each candidate pose of the plurality of candidate poses based on candidate inertial data corresponding to the highest candidate confidence of each candidate pose of the plurality of candidate poses; and respectively determining a relative pose of each head-mounted display device of the plurality of head-mounted display devices relative to the carrier based on the first inertial data, a candidate pose corresponding to each head-mounted display device of the plurality of head- mounted display devices, and the candidate inertial data corresponding to the candidate pose.”
What steps, methods, algorithms are used? Applicant’s disclosure has not provided any specific examples of how these recited features are accomplished.
Claim 11 recites “further comprising: abandoning tracking of the head pose of the user if a plurality of candidate confidences corresponding to the user are all less than a preset fourth confidence threshold.”
What steps, methods, algorithms are used? Applicant’s disclosure has not provided any specific examples of how these recited features are accomplished.
Claim 12 recites “further comprising: sending the relative pose to the target head-mounted display device, so that the target head-mounted display device performs rendering process based on the relative pose”
What steps, methods, algorithms are used? Applicant’s disclosure has not provided any specific examples of how these recited features are accomplished.
Claims 13,15-21 recite similar claim limitations as claim 1-7, and thus are rejected under similar rational as claims 1 detail above
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 15-21 are rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. The claim(s) does/do not fall within at least one of the four categories of patent eligible subject matter because claim 15 recites “computer-readable storage medium configured to store a computer program therein” which appears to cover both transitory and non-transitory embodiments. Applicant’s disclosure indicates:
Page 20 indicates “The readable medium may be a readable signal medium or a readable storage medium The readable storage medium may include, for example, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus or device, or any combination thereof. More specific examples (a non-exhaustive list) of the readable storage medium include an electrical connection with one or more conducting wires, a portable disk, a hard disk, a random access memory (RAM), a read only memory (ROM), an erasable programmable read only memory (EPROM or flash memory), an optical fiber, a portable compact disk read only memory(CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof”
While Applicant’s disclosure appears to explicitly contemplate a computer readable signal medium encompassing a propagated data signal, as best understood by Examiner, Applicant’s disclosure of a computer readable storage medium appears to encompass embodiments which are not considered as statutory subject matter under 35 USC 101. Specifically in Page 20 of Applicant’s disclosure indicates “A readable storage medium may include, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system”. Further, on page 20 Applicant indicates “a computer readable storage medium may be electronic, magnetic, optical electromagnetic, infrared” which, as best understood by Examiner correspond, to electro-magnetic or optical signals. Particularly in view of page 20 where “readable storage medium include an electrical connection with one or more conducting wires ". Therefore, as best understood by Examiner, Applicant’s disclosed “computer readable storage medium" as disclosed by Applicant, covers forms of non-transitory tangible media and transitory propagating signals per se in view of the ordinary and customary meaning of Applicant’s disclosure in paragraph 0020.
The United States Patent and Trademark Office (USPTO) is required to give claims their broadest reasonable interpretation consistent with the specification during proceedings before the USPTO. See In re Zletz, 893 F.2d 319 (Fed. Cir. 1989) (during patent examination the pending claims must be interpreted as broadly as their terms reasonably allow). The broadest reasonable interpretation of a claim drawn to a computer readable medium (also called machine readable medium and other such variations) typically covers forms of non-transitory tangible media and transitory propagating signals per se in view of the ordinary and customary meaning of computer readable media, particularly when the specification is silent. In this instant, page 20 explicitly discloses the invention may include a propagated data signal with computer readable program code embodied therein. A claim drawn to such a computer readable medium that covers both transitory and non-transitory embodiments may be amended to narrow the claim to cover only statutory embodiments to avoid a rejection under 35 U.S.C. § 101 by adding the limitation “non-transitory” to the claim.
Claims 16-21 are dependent on claim 15 and, therefore, inherit the non-statutory subject matter and are rejected using the same reasoning above.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have 208been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-3, 5, 7-17, 19, 21 is/are rejected under 35 U.S.C. 103 as being unpatentable over Rothkopf, U.S. Patent Publication No. 20180081426 in view of Koseki et al, U.S. Patent Publication No. 20150352437, Foxlin et al, U.S. Patent Publication No. 20040149036 and Foxlin, U.S. Patent Publication No. 20040073360 (Foxlin2).
Consider claim 1, Rothkopf teaches a method for determining a relative pose, wherein an auxiliary positioning apparatus (see Rothkopf figure 2A, element 208, figure 7A, element 702) and a target head-mounted display device (see Rothkopf figure 2A, element 204) corresponding to the auxiliary positioning apparatus are located inside a movable carrier (see Rothkopf figure 2A, element 200), the auxiliary positioning apparatus being configured to be fixedly connected to the carrier (see Rothkopf paragraph 0039 where vehicle 200 includes an additional inertial measurement device 208 built into vehicle 20) and being provided with an image sensor (see Rothkopf paragraph 0065 where a vehicle may use light detection and ranging or cameras to correct for drift) and a first inertial measurement unit (see Rothkopf figure 2A, element 208), and the target head-mounted display device being provided with a second inertial measurement unit (see Rothkopf paragraph 039 where User 202 is wearing user device 204, which is a head-mounted display that includes an inertial measurement device 206.),
the method comprising: obtaining an image, captured by the image sensor, by the first inertial measurement unit, and second inertial data captured by the second inertial measurement unit (see Rothkopf paragraph 0031 where relative inertial measurement system may also include one or more processors that receive inertial measurements from the first and second inertial measurement devices or the one or more processors may be included in the user device. The one or more processors may determine a relative motion of the user device relative to motion of the non-fixed reference frame (e.g the vehicle) based on differences between the inertial measurements received from the first inertial measurement device and the inertial measurements received from the second inertial measurement device. For example, the one or more processors may determine that the user turned his head 20 degrees by subtracting an inertial measurement from the second inertial measurement device indicating the vehicle turned 90 degrees from an inertial measurement from the first inertial measurement device indicating that the user's head turned on overall amount of 110 degrees. In a similar manner, various other relative inertial motions may be determined by the one or more processors based on received inertial measurements from the first and second inertial measurement devices, such as relative acceleration, relative velocity, position, relative position within the non-fixed reference frame (e.g. the vehicle), three-dimensional orientation within the non-fixed reference frame (e.g. the vehicle), and orientation in three dimensional space with regard to a reference frame outside of the vehicle such as the earth and paragraph 0074 where relative motion measurements may be corrected based on tracking data to account for errors in measurement (e.g. drift) and paragraph 0065 where a vehicle may use light detection and ranging or cameras to correct for drift);
Rothkopf does not appear to explicitly disclose obtaining an image, captured by the image sensor, of a user wearing the target head- mounted display device; determining, based on the image, pose data of the target head-mounted display device relative to the auxiliary positioning apparatus.
In a related field of endeavor, tracking pose/position of a head mounted display, Koseki teaches obtaining an image, captured by the image sensor, of a user wearing the target head- mounted display device; determining, based on the image, pose data of the target head-mounted display device relative to the auxiliary positioning apparatus (see Koseki figure 1, element 1100, 1300, 1310 and paragraph 0097-0103, 0256 where the posture of the HMD 1310 is detected using the image captured by the range sensor unit 1100 and the detection results of the 6-axis sensor).
One of ordinary skill would have been motivated to have modified Rothkopf to have tracked a pose/posture of a head mounted display(HMD) as disclosed by Koseki by capturing an image of a user wearing the target and determining based on an image pose/posture of the HMD so as to correct for drift as suggested by Rothkopf using known techniques with predictable results.
Rothkopf/Koseki discloses using image data to correct for drift and multiple inertial measurement devices which, as best understood by Examiner, correspond to the recited feature of performing data fusion on the first inertial data, the second inertial data, and the pose data, and performing fusion detection.
Rothkopf/Koseki is silent regarding when the fusion detection passes, determining a relative pose of the target head-mounted display device relative to the carrier based on the first inertial data, the second inertial data, and the pose data.
In are related field of endeavor, Foxlin teaches tracking motion relative to a moving platform in applications such as motion-base driving and flight simulators, conventional VE systems deployed on board ships, and a range of live vehicular applications such as driver's or pilot's vision enhancement, helmet-mounted cueing systems, and advanced human-machine interfaces to improve pilots' situational awareness and control capability (see Foxlin paragraph 0004). Further Foxlin teaches a similar sensor arrangement (see Foxlin figure 3C, figure 8, element 810, 820, 800 and paragraphs 0079-0098 specifically for example paragraph 0083 where IS-1200 system also includes a base unit or computer, which receives data from one or more sensing devices through its sensor ports, and fuses the data using flexible sensor fusion software as described in pending U.S. patent application Ser. No. 10/639,242 and paragraph 0097-0098 where optical system illustrated in the above embodiment is one example of optical aiding, in which measurements made by a head-mounted camera are combined with measurements from two cockpit-fixed cameras to obtain especially high robustness. However, the system can also be operated using only one or more head-mounted cameras (in which case at least one reference IMU fixed to the vehicle should also be provided for compensation of the vehicle motion), or using only cockpit-mounted cameras (in which case a simple IMU instead of the combined IMU/camera unit is used on the helmet or tracked object)).
Foxlin2 (corresponding to U.S. patent application Ser. No. 10/639,242) teaches a returning a failure condition when pose-recovery algorithm (PRA) routine were not sufficiently consistent to produce a high confidence pose estimate or returns a presumably valid pose estimate otherwise (see Foxlin2 paragraph 0242-0243) so as to either abandon failed condition measurements or initialize a filter and calculate an initial uncertainty.
One of ordinary skill would have been motivated to have further modified Rothkopf with the teachings of Foxlin/Foxlin2 to determine whether a data set of measurements should be abandoned as failed or passed on for further processing according to a confidence value using known techniques with predictable results.
Consider claim 2, Rothkopf as modified by Koseki, Foxlin and Foxlin2 teaches all the limitations of claim 1 and further teaches wherein performing data fusion on the first inertial data, the second inertial data, and the pose data, and performing fusion detection comprises: determining a difference between the first inertial data and the second inertial data (see Rothkopf paragraph 0043 where The difference between the inertial measurements from inertial measurement devices 206 and 208 indicate the relative motion of user device 204 (and user 202's head) within the reference frame of vehicle 200);
determining a confidence based on element 1100, 1300, 1310 and paragraph 0097-0103, 0256 where the posture of the HMD 1310 is detected using the image captured by the range sensor unit 1100 and the detection results of the 6-axis sensor); and
in the case where a determined confidence is greater than or equal to a preset confidence threshold, determining that the fusion detection passes (see Foxlin2 paragraph 0242-0243).
Rothkopf is silent regarding a residual. Foxlin2 teaches calculating and “innovation” measurement corresponding to the difference between an expected measurement based on an estimated pose and an actual sensor measurement so as to process a degree of uncertainty in how closely an actual measurement will match a prediction and calculate a search window or range about the estimated measurement that the sensor should search for the target (see Foxlin2 paragraphs 0260, 0242-0243, 0284-0292).
One of ordinary skill would have been motivated to have further modified Rothkopf with the teachings of Foxlin2 to have a residual as disclosed by Foxlin2 so as to process a degree of uncertainty in how closely an actual measurement will match a prediction and calculate a search window or range about the estimated measurement that the sensor should search for the target using known techniques with predictable results.
Consider claim 3, Rothkopf as modified by Koseki, Foxlin and Foxlin2 teaches all the limitations of claim 1 and further teaches wherein determining, based on the image, pose data of the target head-mounted display device relative to the auxiliary positioning apparatus comprises: when a plurality of head-mounted display devices are included in the image, respectively determining poses of the plurality of head-mounted display devices relative to the auxiliary positioning apparatus to obtain a plurality of candidate poses (see Rothkopf paragraph 0060 where In some embodiments, a system may include multiple ones of user devices 602, 604, 606, 608, or 610 along with a head mounted display such as system 500 illustrated in FIG. 5. In some embodiments, a vehicle in which a user device, such as user devices 602, 604, 606, 608, and/or 610, is used may be a public transportation vehicle such as a subway, bus, tram, etc. In some embodiments, a public transportation vehicle may include an inertial measurement device and may broadcast inertial measurements of the public transportation vehicle to riders of the public transportation vehicle. Thus user devices, such as such as user devices 602, 604, 606, 608, and/or 610, may determine relative movements relative to the public transportation vehicle based on received inertial measurements broadcast to users of the public transportation vehicle. In some embodiments, a vehicle, such as a public transportation vehicle, may include predictive inertial data in inertial measurements broadcast to riders of the public transportation vehicle.).
Consider claim 5, Rothkopf as modified by Koseki, Foxlin and Foxlin2 teaches all the limitations of claim 3 and further teaches wherein the auxiliary positioning apparatus is in one-to-one (replication of parts) correspondence with the target head-mounted display device (see Rothkopf paragraph 0060 where a system may include multiple ones of user devices 602, 604, 606, 608, or 610 along with a head mounted display such as system 500 illustrated in FIG. 5); and
performing data fusion on the first inertial data, the second inertial data, and the pose data, and performing fusion detection comprises: performing data fusion on the plurality of candidate poses respectively with the first inertial data and the second inertial data to determine confidences of the plurality of candidate poses (see Foxlin2 paragraph 0408 where a single object can be tracked or multiple object can be tracked); and
when there is a confidence greater than or equal to a preset first confidence threshold, determining that the fusion detection passes (see Foxlin2 paragraph 0242-0243).
Consider claim 7, Rothkopf as modified by Koseki, Foxlin and Foxlin2 teaches all the limitations of claim 3 and further teaches further comprising: abandoning tracking of the head pose of the user if the confidence of the candidate pose corresponding to the user is less than a preset second confidence threshold Foxlin2 paragraph 0242-0243).
Consider claim 12, Rothkopf as modified by Koseki, Foxlin and Foxlin2 teaches all the limitations of claim 1 and further teaches further comprising: sending the relative pose to the target head-mounted display device, so that the target head-mounted display device performs rendering process based on the relative pose (see Rothkopf paragraph 0066 where images in a view that correlate to the terrain or that indicate speed of the vehicle may improve comfort of a user wearing a head-mounted display while travelling in a vehicle and may reduce symptoms of sickness or nausea).
Consider claim 13, Rothkopf teaches an extended reality system comprising an auxiliary positioning apparatus (see Rothkopf figure 2A, element 208, figure 7A, element 702), a head-mounted display device (see Rothkopf figure 2A, element 204) communicatively connected to the auxiliary positioning apparatus, and an apparatus for determining a relative pose of a target head-mounted display device relative to a movable carrier (see Rothkopf figure 2A, element 200),
wherein the head-mounted display device and the apparatus are located in the movable carrier, the auxiliary positioning apparatus is fixedly connected to the movable carrier (see Rothkopf paragraph 0039 where vehicle 200 includes an additional inertial measurement device 208 built into vehicle 20) and provided with an image sensor (see Rothkopf paragraph 0065 where a vehicle may use light detection and ranging or cameras to correct for drift) and a first inertial measurement unit (see Rothkopf figure 2A, element 208), and the head-mounted display device is provided with a second inertial measurement unit (see Rothkopf paragraph 039 where User 202 is wearing user device 204, which is a head-mounted display that includes an inertial measurement device 206.); and
the apparatus for determining a relative pose is configured to: obtain an image including at least one head-mounted display device captured by the image sensor (implicit see Rothkopf paragraph 0074 where relative motion measurements may be corrected based on tracking data to account for errors in measurement (e.g. drift) and paragraph 0065 where a vehicle may use light detection and ranging or cameras to correct for drift), first inertial data of at least one target head-mounted display device, and second inertial data of the auxiliary positioning apparatus (see Rothkopf paragraph 0031 where relative inertial measurement system may also include one or more processors that receive inertial measurements from the first and second inertial measurement devices or the one or more processors may be included in the user device. The one or more processors may determine a relative motion of the user device relative to motion of the non-fixed reference frame (e.g the vehicle) based on differences between the inertial measurements received from the first inertial measurement device and the inertial measurements received from the second inertial measurement device. For example, the one or more processors may determine that the user turned his head 20 degrees by subtracting an inertial measurement from the second inertial measurement device indicating the vehicle turned 90 degrees from an inertial measurement from the first inertial measurement device indicating that the user's head turned on overall amount of 110 degrees. In a similar manner, various other relative inertial motions may be determined by the one or more processors based on received inertial measurements from the first and second inertial measurement devices, such as relative acceleration, relative velocity, position, relative position within the non-fixed reference frame (e.g. the vehicle), three-dimensional orientation within the non-fixed reference frame (e.g. the vehicle), and orientation in three dimensional space with regard to a reference frame outside of the vehicle such as the earth and paragraph 0074 where relative motion measurements may be corrected based on tracking data to account for errors in measurement (e.g. drift) and paragraph 0065 where a vehicle may use light detection and ranging or cameras to correct for drift);
determine a relative pose of each target head-mounted display device (see Rothkopf paragraph 0060 where In some embodiments, a system may include multiple ones of user devices 602, 604, 606, 608, or 610 along with a head mounted display such as system 500 illustrated in FIG. 5. In some embodiments, a vehicle in which a user device, such as user devices 602, 604, 606, 608, and/or 610, is used may be a public transportation vehicle such as a subway, bus, tram, etc. In some embodiments, a public transportation vehicle may include an inertial measurement device and may broadcast inertial measurements of the public transportation vehicle to riders of the public transportation vehicle. Thus user devices, such as such as user devices 602, 604, 606, 608, and/or 610, may determine relative movements relative to the public transportation vehicle based on received inertial measurements broadcast to users of the public transportation vehicle. In some embodiments, a vehicle, such as a public transportation vehicle, may include predictive inertial data in inertial measurements broadcast to riders of the public transportation vehicle.) relative to the carrier based on the first inertial data of each target head-mounted display device, the second inertial data of the auxiliary positioning apparatus, and the determined pose data of each target head-mounted display device relative to the auxiliary positioning apparatus (see Rothkopf paragraph 0031 where relative inertial measurement system may also include one or more processors that receive inertial measurements from the first and second inertial measurement devices or the one or more processors may be included in the user device. The one or more processors may determine a relative motion of the user device relative to motion of the non-fixed reference frame (e.g the vehicle) based on differences between the inertial measurements received from the first inertial measurement device and the inertial measurements received from the second inertial measurement device. For example, the one or more processors may determine that the user turned his head 20 degrees by subtracting an inertial measurement from the second inertial measurement device indicating the vehicle turned 90 degrees from an inertial measurement from the first inertial measurement device indicating that the user's head turned on overall amount of 110 degrees. In a similar manner, various other relative inertial motions may be determined by the one or more processors based on received inertial measurements from the first and second inertial measurement devices, such as relative acceleration, relative velocity, position, relative position within the non-fixed reference frame (e.g. the vehicle), three-dimensional orientation within the non-fixed reference frame (e.g. the vehicle), and orientation in three dimensional space with regard to a reference frame outside of the vehicle such as the earth and paragraph 0074 where relative motion measurements may be corrected based on tracking data to account for errors in measurement (e.g. drift) and paragraph 0065 where a vehicle may use light detection and ranging or cameras to correct for drift).
Rothkopf does not appear to explicitly disclose obtaining an image, captured by the image sensor, of at least one target head- mounted display device; determining, based on the image, pose data of the target head-mounted display device relative to the auxiliary positioning apparatus.
In a related field of endeavor, tracking pose/position of a head mounted display, Koseki teaches obtaining an image, captured by the image sensor, of a user wearing the target head- mounted display device; determining, based on the image, pose data of the target head-mounted display device relative to the auxiliary positioning apparatus (see Koseki figure 1, element 1100, 1300, 1310 and paragraph 0097-0103, 0256 where the posture of the HMD 1310 is detected using the image captured by the range sensor unit 1100 and the detection results of the 6-axis sensor).
One of ordinary skill would have been motivated to have modified Rothkopf to have tracked a pose/posture of a head mounted display(HMD) as disclosed by Koseki by capturing an image of a user wearing the target and determining based on an image pose/posture of the HMD so as to correct for drift as suggested by Rothkopf using known techniques with predictable results.
Rothkopf/Koseki discloses using image data to correct for drift and multiple inertial measurement devices, which as best understood by Examiner, correspond to the recited feature of performing data fusion on the first inertial data, the second inertial data, and the pose data, and performing fusion detection.
Rothkopf/Koseki is silent regarding determine whether a target head-mounted display device is included in the image.
In are related field of endeavor, Foxlin teaches tracking motion relative to a moving platform in applications such as motion-base driving and flight simulators, conventional VE systems deployed on board ships, and a range of live vehicular applications such as driver's or pilot's vision enhancement, helmet-mounted cueing systems, and advanced human-machine interfaces to improve pilots' situational awareness and control capability (see Foxlin paragraph 0004). Further Foxlin teaches a similar sensor arrangement (see Foxlin figure 3C, figure 8, element 810, 820, 800 and paragraphs 0079-0098 specifically for example paragraph 0083 where IS-1200 system also includes a base unit or computer, which receives data from one or more sensing devices through its sensor ports, and fuses the data using flexible sensor fusion software as described in pending U.S. patent application Ser. No. 10/639,242 and paragraph 0097-0098 where optical system illustrated in the above embodiment is one example of optical aiding, in which measurements made by a head-mounted camera are combined with measurements from two cockpit-fixed cameras to obtain especially high robustness. However, the system can also be operated using only one or more head-mounted cameras (in which case at least one reference IMU fixed to the vehicle should also be provided for compensation of the vehicle motion), or using only cockpit-mounted cameras (in which case a simple IMU instead of the combined IMU/camera unit is used on the helmet or tracked object)).
Foxlin2 (corresponding to U.S. patent application Ser. No. 10/639,242) teaches a returning a failure condition when pose-recovery algorithm (PRA) routine were not sufficiently consistent to produce a high confidence pose estimate or returns a presumably valid pose estimate otherwise (see Foxlin2 paragraph 0242-0243) so as to either abandon failed condition measurements or initialize a filter and calculate an initial uncertainty.
One of ordinary skill would have been motivated to have further modified Rothkopf with the teachings of Foxlin/Foxlin2 to determine whether a data set of measurements should be abandoned as not having a target present or passed for including a target so as to further process data according to a confidence value using known techniques with predictable results.
Claim 14 cancelled
Consider claim 15, Rothkopf teaches a computer-readable storage medium configured to store a computer program therein, wherein the computer program, when executed by a processor, implements the method for determining a relative pose (see Rothkopf paragraph 0057), wherein an auxiliary positioning apparatus (see Rothkopf figure 2A, element 208, figure 7A, element 702) and a target head-mounted display device (see Rothkopf figure 2A, element 204) corresponding to the auxiliary positioning apparatus are located inside a movable carrier (see Rothkopf figure 2A, element 200), the auxiliary positioning apparatus being configured to be fixedly connected to the carrier (see Rothkopf paragraph 0039 where vehicle 200 includes an additional inertial measurement device 208 built into vehicle 20) and being provided with an image sensor (see Rothkopf paragraph 0065 where a vehicle may use light detection and ranging or cameras to correct for drift) and a first inertial measurement unit (see Rothkopf figure 2A, element 208), and the target head-mounted display device being provided with a second inertial measurement unit (see Rothkopf paragraph 039 where User 202 is wearing user device 204, which is a head-mounted display that includes an inertial measurement device 206.), the method comprising:
obtaining an image, captured by the image sensor, one or more processors based on received inertial measurements from the first and second inertial measurement devices, such as relative acceleration, relative velocity, position, relative position within the non-fixed reference frame (e.g. the vehicle), three-dimensional orientation within the non-fixed reference frame (e.g. the vehicle), and orientation in three dimensional space with regard to a reference frame outside of the vehicle such as the earth and paragraph 0074 where relative motion measurements may be corrected based on tracking data to account for errors in measurement (e.g. drift) and paragraph 0065 where a vehicle may use light detection and ranging or cameras to correct for drift);
Rothkopf does not appear to explicitly disclose obtaining an image, captured by the image sensor, of a user wearing the target head- mounted display device; determining, based on the image, pose data of the target head-mounted display device relative to the auxiliary positioning apparatus.
In a related field of endeavor, tracking pose/position of a head mounted display, Koseki teaches obtaining an image, captured by the image sensor, of a user wearing the target head- mounted display device; determining, based on the image, pose data of the target head-mounted display device relative to the auxiliary positioning apparatus (see Koseki figure 1, element 1100, 1300, 1310 and paragraph 0097-0103, 0256 where the posture of the HMD 1310 is detected using the image captured by the range sensor unit 1100 and the detection results of the 6-axis sensor).
One of ordinary skill would have been motivated to have modified Rothkopf to have tracked a pose/posture of a head mounted display(HMD) as disclosed by Koseki by capturing an image of a user wearing the target and determining based on an image pose/posture of the HMD so as to correct for drift as suggested by Rothkopf using known techniques with predictable results.
Rothkopf/Koseki discloses using image data to correct for drift and multiple inertial measurement devices which, as best understood by Examiner, correspond to the recited feature of performing data fusion on the first inertial data, the second inertial data, and the pose data, and performing fusion detection.
Rothkopf/Koseki is silent regarding when the fusion detection passes, determining a relative pose of the target head-mounted display device relative to the carrier based on the first inertial data, the second inertial data, and the pose data.
In are related field of endeavor, Foxlin teaches tracking motion relative to a moving platform in applications such as motion-base driving and flight simulators, conventional VE systems deployed on board ships, and a range of live vehicular applications such as driver's or pilot's vision enhancement, helmet-mounted cueing systems, and advanced human-machine interfaces to improve pilots' situational awareness and control capability (see Foxlin paragraph 0004). Further Foxlin teaches a similar sensor arrangement (see Foxlin figure 3C, figure 8, element 810, 820, 800 and paragraphs 0079-0098 specifically for example paragraph 0083 where IS-1200 system also includes a base unit or computer, which receives data from one or more sensing devices through its sensor ports, and fuses the data using flexible sensor fusion software as described in pending U.S. patent application Ser. No. 10/639,242 and paragraph 0097-0098 where optical system illustrated in the above embodiment is one example of optical aiding, in which measurements made by a head-mounted camera are combined with measurements from two cockpit-fixed cameras to obtain especially high robustness. However, the system can also be operated using only one or more head-mounted cameras (in which case at least one reference IMU fixed to the vehicle should also be provided for compensation of the vehicle motion), or using only cockpit-mounted cameras (in which case a simple IMU instead of the combined IMU/camera unit is used on the helmet or tracked object)).
Foxlin2 (corresponding to U.S. patent application Ser. No. 10/639,242) teaches a returning a failure condition when pose-recovery algorithm (PRA) routine were not sufficiently consistent to produce a high confidence pose estimate or returns a presumably valid pose estimate otherwise (see Foxlin2 paragraph 0242-0243) so as to either abandon failed condition measurements or initialize a filter and calculate an initial uncertainty.
One of ordinary skill would have been motivated to have further modified Rothkopf with the teachings of Foxlin/Foxlin2 to determine whether a data set of measurements should be abandoned as failed or passed on for further processing according to a confidence value using known techniques with predictable results.
Claims 8-12, 16-17, 19, 21 recite similar claim limitations as claims 1-3, 5, 7, and thus are rejected under similar rational as claims 1-3, 5, 7, detail above.
Claim(s) 4, 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Rothkopf, U.S. Patent Publication No. 20180081426, Koseki et al, U.S. Patent Publication No. 20150352437, Foxlin et al, U.S. Patent Publication No. 20040149036 and Foxlin, U.S. Patent Publication No. 20040073360 (Foxlin2) in view of Dorn, U.S. Patent Publication No. 20230065018.
Consider claim 4, Rothkopf as modified by Koseki, Foxlin and Foxlin2 teaches all the limitations of claim 3 and further teaches wherein when a plurality of head- mounted display devices are included in the image, respectively determining poses of the plurality of head-mounted display devices relative to the auxiliary positioning apparatus to obtain a plurality of candidate poses (see Rothkopf paragraph 0060 where In some embodiments, a system may include multiple ones of user devices 602, 604, 606, 608, or 610 along with a head mounted display such as system 500 illustrated in FIG. 5. In some embodiments, a vehicle in which a user device, such as user devices 602, 604, 606, 608, and/or 610, is used may be a public transportation vehicle such as a subway, bus, tram, etc. In some embodiments, a public transportation vehicle may include an inertial measurement device and may broadcast inertial measurements of the public transportation vehicle to riders of the public transportation vehicle. Thus user devices, such as such as user devices 602, 604, 606, 608, and/or 610, may determine relative movements relative to the public transportation vehicle based on received inertial measurements broadcast to users of the public transportation vehicle. In some embodiments, a vehicle, such as a public transportation vehicle, may include predictive inertial data in inertial measurements broadcast to riders of the public transportation vehicle.) comprises:
for a user of the identified at least one user, in response to the user wearing a head- mounted display device, determining a head pose of the user as a candidate pose corresponding to the user, to obtain the plurality of candidate poses Rothkopf paragraph 0031 where relative inertial measurement system may also include one or more processors that receive inertial measurements from the first and second inertial measurement devices or the one or more processors may be included in the user device. The one or more processors may determine a relative motion of the user device relative to motion of the non-fixed reference frame (e.g the vehicle) based on differences between the inertial measurements received from the first inertial measurement device and the inertial measurements received from the second inertial measurement device. For example, the one or more processors may determine that the user turned his head 20 degrees by subtracting an inertial measurement from the second inertial measurement device indicating the vehicle turned 90 degrees from an inertial measurement from the first inertial measurement device indicating that the user's head turned on overall amount of 110 degrees. In a similar manner, various other relative inertial motions may be determined by the one or more processors based on received inertial measurements from the first and second inertial measurement devices, such as relative acceleration, relative velocity, position, relative position within the non-fixed reference frame (e.g. the vehicle), three-dimensional orientation within the non-fixed reference frame (e.g. the vehicle), and orientation in three dimensional space with regard to a reference frame outside of the vehicle such as the earth and paragraph 0074 where relative motion measurements may be corrected based on tracking data to account for errors in measurement (e.g. drift) and paragraph 0065 where a vehicle may use light detection and ranging or cameras to correct for drift and Koseki figure 1, element 1100, 1300, 1310 and paragraph 0097-0103, 0256 where the posture of the HMD 1310 is detected using the image captured by the range sensor unit 1100 and the detection results of the 6-axis sensor).
Rothkopf/Koseki is silent regarding identifying the image using a pre-trained identification model to determine at least one user included in the image.
In a related field of endeavor, object identification using images, Dorn teaches a matching routine to check whether the at least one orientation object is also contained or imaged in the image or video so as to identify when a target is present in a capture image (see Dorn paragraphs 0010, 0014 where matching routine is used to check whether the at least one orientation object is also contained or imaged in the image or video. This part of the image here is the described image region in the image. If such a match between model and image region is identified, a relative position of the orientation object in relation to the glasses is determined by the processor circuit.).
One of ordinary skill would have been motivated to have modified Rothkopf with the teachings of Dorn to have a matching routine so as to check whether the at least one orientation object is also contained or imaged in the image or video using known techniques with predictable results.
Claim 18 recites similar claim limitations as claim 4, and thus is rejected under similar rational as claim 4, detail above.
Claim(s) 6, 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Rothkopf, U.S. Patent Publication No. 20180081426, Koseki et al, U.S. Patent Publication No. 20150352437, Foxlin et al, U.S. Patent Publication No. 20040149036 and Foxlin, U.S. Patent Publication No. 20040073360 (Foxlin2) in view of Fung et al, U.S. Patent Publication No. 20180032825.
Consider claim 6, Rothkopf as modified by Koseki, Foxlin and Foxlin2 teaches all the limitations of claim 3. Rothkopf is silent regarding wherein before determining a relative pose of the target head-mounted display device relative to the carrier based on the first inertial data, the second inertial data, and the pose data, the method further comprises: determining a candidate pose with a highest confidence as the pose data of the target head-mounted display device relative to the auxiliary positioning apparatus.
In a related field of endeavor, calculating head pose point (HPP), Fung teaches that calculation of a HPP can be based on confidence levels so as to avoid wasting processing by calculating for a low confidence value (see Fung paragraph 0059).
One of ordinary skill would have been motivated to have further modified Rothkopf as recited to have calculated a high confidence pose so as to avoid wasting processing by calculating for a low confidence value using known techniques with predictable results.
Claim 20 recites similar claim limitations as claim 6, and thus is rejected under similar rational as claim 6, detail above.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Haar et al, U.S. Patent Publication No. 20200218065(determining pose), Cohen et al, U.S. Patent Publication No. 20200169855 (locating mobile devices within a vehicle), Benfold et al, U.S. Patent Publication No. 20230277934 (camera re-localization determination using prior pose model), Haar et al, U.S. Patent Publication No. 20190180717 (gauging augmented reality goggles in a transportation vehicle), Hudman, U.S. Patent Publication No. 20210250570 (compensation optic), Trythall, U.S. Patent Publication No. 20180301076 (figure 5), Mackie et al, U.S. Patent Publication No. 20240327027 (augment reality system for aircraft pilots), Gorur Sheshagiri, U.S. Patent Publication No. 20200271450 (providing immersive extended reality experiences on moving platforms), Ma et al, U.S. Patent Publication No. 20230098910 (tracking head mounted display), Holland et al, U.S. Patent Publication No. 20210125664 (pose estimation in extended reality), Lahr et al, U.S. Patent Publication No. 10775881 (high assurance head tracker), Lochmann, U.S. Patent Publication No. 20240328810 (head mounted display in a motor vehicle).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Dorothy H Harris whose telephone number is (571)270-7539. The examiner can normally be reached Monday - Friday 8am - 4pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, William Boddie can be reached at 571-272-0666. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Dorothy Harris/Primary Examiner, Art Unit 2625
Prosecution Insights