DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1, 3–5, 11, 13 and 20 are rejected under 35 U.S.C. § 102(a)(2) as being anticipated by Wang (U.S. 12,685,934 B2).
Regarding claim 1, Wang discloses a map alignment method, applicable to a multi-device system, wherein the multi-device system is operable in a physical environment and comprises a host device (Fig. 2, 21 a VR device) and a client device (Fig. 2, 23 a mobile phone), and the map alignment method comprises:
obtaining, by the host device and the client device, a host key frame (Per Fig. 3, Wang discloses a first screen image—i.e., a host key frame—from his VR device. Wang col. 12 lines 21–27. [r]eceiving a first screen image sent from the VR device.) and a client key frame1 at a preset time point, respectively; (Per Fig. 2, Wang’s mobile phone 23 captures real scene images—i.e., a client key frame(s)—communicating his VR device 21. Ibid. col. 9 lines 7–23. A camera 231 in the mobile phone 23 can collect real scene images.)
generating, by the host device, a first client pose according to the host key frame; and (Per Fig. 3, Wang discloses pose information based on his VR device’s coordinates. Ibid. col. 11 lines 58–60. [a]cquiring 3D key point coordinates and a pose of a target device in a VR world coordinate system determined by a VR device.)
aligning, by the client device, a client map (a client map construed as 2D key point coordinates) established by the client device detecting the physical environment (Per Fig. 3 at step 303, Wang discloses 2D key point coordinates of the target device in a camera coordinate system. Ibid. col. 11 line 57 – col. 12 line 5. [a]cquiring 2D key point coordinates of the target device in a camera coordinate system based on the target image.) with a host map established by the host device detecting the physical environment according to the first client pose. (Per Fig. 3 at step 304, Wang discloses 3D key point coordinates based on a pose of the target device in a VR world coordinate system such that calibration information—i.e., aligning spatial information—is determined. Ibid. [d]etermining calibration information of the image acquisition device in the VR world coordinate system according to the 2D key point coordinates, the 3D key point coordinates, the pose of the target device and a pose of the image acquisition device.)
Regarding claim 11, Wang discloses a multi-device system, operable in a physical environment, and comprising:
a host device, configured to establish a host map by detecting the physical environment; and (Fig. 2, 21 a VR device)
a client device, configured to establish a client map by detecting the physical environment, (Fig. 2, 23 a mobile phone)
wherein the host device and the client device are configured to obtain a host key frame and (Per Fig. 3, Wang discloses a first screen image—i.e., a host key frame—from his VR device. Wang col. 12 lines 21–27. [r]eceiving a first screen image sent from the VR device.) a client key frame at a preset time point, respectively, and (Per Fig. 2, Wang’s mobile phone 23 captures real scene images—i.e., a client key frame(s)—communicating his VR device 21. Ibid. col. 9 lines 7–23. A camera 231 in the mobile phone 23 can collect real scene images.)
wherein the host device is configured to generate a first client pose according to the host key frame, (Per Fig. 3, Wang discloses pose information based on his VR device’s coordinates. Ibid. col. 11 lines 58–60. [a]cquiring 3D key point coordinates and a pose of a target device in a VR world coordinate system determined by a VR device.) and the client device is configured to align the client map with the host map according to the first client pose. (Per Fig. 3 at step 304, Wang discloses 3D key point coordinates based on a pose of the target device in a VR world coordinate system such that calibration information—i.e., aligning spatial information—is determined. Ibid. [d]etermining calibration information of the image acquisition device in the VR world coordinate system according to the 2D key point coordinates, the 3D key point coordinates, the pose of the target device and a pose of the image acquisition device.)
Regarding claim 20, Wang discloses a non-transitory computer readable storage medium with a computer program to execute a map alignment method applicable to a multi-device system, wherein the multi-device system is operable in a physical environment and comprises a host device and a client device, and the map alignment method comprises:
obtaining, by the host device and the client device, a host key frame (Per Fig. 3, Wang discloses a first screen image—i.e., a host key frame—from his VR device. Wang col. 12 lines 21–27. [r]eceiving a first screen image sent from the VR device.) and a client key frame at a preset time point, respectively; (Per Fig. 2, Wang’s mobile phone 23 captures real scene images—i.e., a client key frame(s)—communicating his VR device 21. Ibid. col. 9 lines 7–23. A camera 231 in the mobile phone 23 can collect real scene images.)
generating, by the host device, a first client pose according to the host key frame; and (Per Fig. 3, Wang discloses pose information based on his VR device’s coordinates. Ibid. col. 11 lines 58–60. [a]cquiring 3D key point coordinates and a pose of a target device in a VR world coordinate system determined by a VR device.)
aligning, by the client device, a client map (a client map construed as 2D key point coordinates) established by the client device detecting the physical environment (Per Fig. 3 at step 303, Wang discloses 2D key point coordinates of the target device in a camera coordinate system. Ibid. col. 11 line 57 – col. 12 line 5. [a]cquiring 2D key point coordinates of the target device in a camera coordinate system based on the target image.) with a host map established by the host device detecting the physical environment according to the first client pose. (Per Fig. 3 at step 304, Wang discloses 3D key point coordinates based on a pose of the target device in a VR world coordinate system such that calibration information—i.e., aligning spatial information—is determined. Ibid. [d]etermining calibration information of the image acquisition device in the VR world coordinate system according to the 2D key point coordinates, the 3D key point coordinates, the pose of the target device and a pose of the image acquisition device.)
Regarding claim 3, Wang discloses the map alignment method, wherein generating, by the host device, the first client pose according to the host key frame comprises: calculating, by the host device, pose data of at least one trackable object134 on the client device from the host key frame, to generate the first client pose. (Per Fig. 3, Wang discloses pose information based on his VR device’s coordinates. Wang col. 11 lines 58–60. [a]cquiring 3D key point coordinates and a pose of a target device in a VR world coordinate system determined by a VR device.)
Regarding claim 4, Wang discloses the map alignment method, wherein aligning, by the client device, the client map with the host map according to the first client pose comprises:
replacing, by the client device, a second client pose corresponding to the client key frame by the first client pose; and (Per Fig. 3 at step 306, Wang discloses a second screen image from a first screen image. Wang col. 12 lines 31–35. A second screen image from the perspective of the image acquisition device can be determined according to the position information and orientation information of the camera in the VR world coordinate system, as well as the first screen image.)
transforming, by the client device, a plurality of map points according to a transformation data configured to transform the second client pose into the first client pose. (Per Fig. 1A at step 104, Wang discloses 3D key points correspond to 2D key points of the target device. Ibid. col. 9 lines 62–67. The corresponding 3D keys and 2D keys can correspond to the same feature points of the target device.)
Regarding claim 5, Wang discloses the map alignment method, further comprising:
generating, by the client device, a second client pose corresponding to the client key frame through a feature extraction based localization technology. (Per Fig. 1A at step 103, Wang discloses a key point extraction model in his neural network to render a plurality of a sample images of the target device. Wang col. 9 lines 35–45. [o]btaining 2D key point coordinates of the target device output by the key point extraction model, wherein the key point extraction model is a convolutional neural network model trained based on sample information comprising a plurality of sample images of the target device)
Regarding claim 13, it has been rejected in the same manner as claim 3.
Regarding claim 14, it has been rejected in the same manner as claim 4.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 2 and 12 are rejected under 35 U.S.C. § 103 as being unpatentable over Wang in view of Dixit et al. (U.S. 2021/0110646 A1).
Regarding claim 2, Wang fails to specifically disclose the map alignment method, wherein before obtaining, by the host device and the client device, the host key frame and the client key frame at the preset time point, respectively, the map alignment method further comprises: synchronizing, by the host device and the client device, time between the host device and the client device; and notifying, by the host device, the client device of the preset time point when the host key frame and the client key frame are obtained.
In related art, Dixit discloses the map alignment method, wherein before obtaining, by the host device and the client device, the host key frame and the client key frame at the preset time point, respectively, the map alignment method further comprises:
synchronizing, by the host device and the client device, time between the host device and the client device; and (Per Fig. 1, Dixit’s console service 104 synchronizes maps of AR machine environments. Dixit Spec. ¶48. The console service 104 can also be configured to respond to a request to synchronize map data by receiving updates to the map data,)
notifying, by the host device, the client device of the preset time point when the host key frame and the client key frame are obtained. (Per Fig. 1, Dixit’s console service 104 responds to his AR console data. Ibid. The console service 104 can further be configured to respond to a request to download map data by searching the AR console data store 106 for map data associated with a location specified in the request,)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to incorporate the teachings of Dixit into the teachings of Wang to identify a location of computing device rendering a map of a current environment thereof. Ibid. ¶52.
Regarding claim 12, it has been rejected in the same manner as claim 2.
Claims 6 and 15 are rejected under 35 U.S.C. § 103 as being unpatentable over Wang in view of Tao (U.S. 12,229,899 B2).
Regarding claim 6, Wang fails to specifically disclose the map alignment method, further comprising: determining, by the client device, if the client device is available for obtaining the client device key frame or not.
In related art, Tao discloses the map alignment method, further comprising: determining, by the client device, if the client device is available for obtaining the client device key frame or not. (Per Fig. 4, Tao discloses a method 400 where his client device 492 is connected to his host device with host bundle adjustment data. Tao col. 5 lines 51–62. [s]ince the client maps and the host map are updated with the same host bundle adjustment data, the drift of the host device 491,)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to incorporate the teachings of Tao into the teachings of Wang to prevent incorrect tracking position optimizing 3D structure and parameters. Ibid. col. 4 lines 1–9.
Regarding claim 15, it has been rejected in the same manner as claim 6.
Claims 7 and 16 are rejected under 35 U.S.C. § 103 as being unpatentable over Wang in view of Tao and further in view of Liao (CN 116664806 A).
Regarding claim 7, Wang as modified by Tao, discloses the claimed invention, but fails to specifically disclose the map alignment method, further comprising:
when the client device is available for obtaining the client key frame, determining, by the host device, if the host device is available for obtaining the host key frame or not and if at least one trackable object on the client device is visible to the host device or not, wherein when the host device is available for obtaining the host key frame and the at least one trackable object on the client device is visible to the host device, the host device and the client device obtain the host key frame and the client key frame at the preset time point, respectively.
In related art, Liao discloses the map alignment method, further comprising:
when the client device is available for obtaining the client key frame, determining, by the host device, if the host device is available for obtaining the host key frame or not and if at least one trackable object on the client device is visible to the host device or not, (Liao discloses a third user device with which a third user is not identified in a video scene. Liao Spec. ¶170. [w]herein the third user is not in the multi-person video scene, and in the real-world view of the camera of the third user device, based on the first augmented reality data,)
wherein when the host device is available for obtaining the host key frame and the at least one trackable object on the client device is visible to the host device, the host device and the client device obtain the host key frame and the client key frame at the preset time point, respectively. (Liao discloses if a target anchor point is identified related to a third user, his device determines the point based on a first position information in an object data information. Ibid. If a target anchor point is identified, the third user device determines the target anchor point based on the first position information in the object data information corresponding to at least one augmented reality presentation information added by the second user in the first augmented reality data and the real-time pose information of the third user device relative to the target anchor point.)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to incorporate the teachings of Liao into the teachings of Wang and Tao to overcome existing technologies where another user could not see the other user’s location in a physical environment. Ibid. ¶2.
Regarding claim 16, it has been rejected in the same manner as claim 7.
Claims 8–10 and 17–19 are rejected under 35 U.S.C. § 103 as being unpatentable over Wang in view of Liao.
Regarding claim 8, Wang fails to specifically disclose the map alignment method, wherein after the first client pose is updated and becomes a third client pose, the map alignment method further comprises: transmitting, by the host device, the third client pose to the client device.
In related art, Liao discloses the map alignment method, wherein after the first client pose is updated and becomes a third client pose, the map alignment method further comprises:
transmitting, by the host device, the third client pose to the client device. (Per Fig. 3 at step S31, Liao’s third user device discloses augmented reality data corresponding to a first user and a second user. Liao Spec. ¶146. In step S31, the third user device acquires first augmented reality data provided by the first user device in a multi-user video scene involving the first user and the second user, wherein the third user is not in the multi-user video scene.)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to incorporate the teachings of Liao into the teachings of Wang to overcome existing technologies where another user could not see the other user’s location in a physical environment. Ibid. ¶2.
Regarding claim 9, Wang as modified by Liao, discloses the map alignment method, further comprising:
updating, by the client device, the client map according to the third client pose. (Liao’s third user equipment discloses updated target anchor point in a physical environment. Liao Spec. ¶149. [t]he third user equipment identifies the updated target anchor point in the real-world view of the camera based on the first incremental data.)
Regarding claim 10, Wang as modified by Liao, discloses the map alignment method, wherein updating, by the client device, the client map according to the third client pose comprises:
replacing, by the client device, the first client pose corresponding to the client key frame by the third client pose. (Liao discloses that his first user equipment overlays augmented reality information based on third incremental data. Liao Spec. ¶151. [s]o that the first user equipment overlays at least one augmented reality presentation information onto the first video stream based on the third incremental data and the first real-time pose information.)
Regarding claim 17, it has been rejected in the same manner as claim 8.
Regarding claim 18, it has been rejected in the same manner as claim 9.
Regarding claim 19, it has been rejected in the same manner as claim 10.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to BENEDICT LEE whose telephone number is (571)270-0390. The examiner can normally be reached 10:00-17:00 (EST).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Stephen R. Koziol can be reached at (408) 918-7630. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/BENEDICT E LEE/Examiner, Art Unit 2665
/Stephen R Koziol/Supervisory Patent Examiner, Art Unit 2665
1 A client key frame can be also construed as a second screen image. See his Fig. 3 at step 306 and col. 12 lines 28–35. [a]cquiring a second screen image from the perspective of the image acquisition device based on the calibration information and a first screen image.
Prosecution Insights