Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Specification
The disclosure is objected to because it contains an embedded hyperlink and/or other form of browser-executable code located on page 19, paragraph 100, line 4. Applicant is required to delete the embedded hyperlink and/or other form of browser-executable code; references to websites should be limited to the top-level domain name without any prefix such as http:// or other browser-executable code. See MPEP § 608.01.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 2-3, 16-26, 27 and 33 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claims 2, 16, 22, 24 and 27 recite the limitation "the server" in:
Claim 2, lines 10 and 13
Claim 16, lines 14 and 16
Claim 22, line 3
Claim 24, line 4
Claim 27, lines 12 and 15
There is insufficient antecedent basis for this limitation in these claims because “the server” is lacking antecedent basis and is only correctly referred to in independent claims 1, 29, 32 and 35, therefore, the independent claims 2, 16 and 27 and their dependent claims, are all rejected due to the lack of antecedent basis in regards to “the server”. This can be corrected by removing “the server” mentioned in those claims and would help in distinguishing the different independent claims from one another, especially due to the similarity between independent claims 1 and 2.
Claim 33 recites the limitation "the error threshold" in:
Claim 33, line 2
There is insufficient antecedent basis for this limitation in this claim because “the error threshold” is lacking antecedent basis and is not ever referred to in independent claim 32, which claim 33 is dependent from and therefore it is rejected due to the lack of antecedent basis in regards to “the error threshold”. This can be corrected by potentially changing “the error threshold” to “an error threshold”.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 29, 32 and 35 are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Meier et al. (U.S. Patent: #10,121,247 B2), hereinafter Meier.
Regarding claim 29, Meier discloses a method for performing 3D, three-dimensional, reconstruction for use in a model of a physical environment captured by at least one sensor of a mobile device (FIG. 3 and Col. 6, Lines 24-28 teach that FIG. 3 shows a flowchart of a method according to an embodiment of the invention generating a geometrical model of the environment based on environment data acquired by sensors of a mobile system and tracking a device based on the generated environment model), the method being performed by a server (Col. 5, Lines 11-20 teach that according to a further embodiment, generating the first geometrical model or a part of the first geometrical model is performed by a processing device of the mobile system, and the first geometrical model is transferred from the mobile system to the mobile device. For example, the first geometrical model is transferred from the mobile system to the mobile device via a server computer or via a point to point communication between the mobile system and the mobile device or via a broadcast or multicast communication (e.g. the mobile system broadcasts data).), the method comprising:
receiving a 3D reconstruction request from the mobile device, wherein the 3D reconstruction request comprises data based on sensor data obtained by sensors of the mobile device (Col. 15, Lines 1-11 teach that if environmental data captured by sensors of a mobile system during the mobile system travelling in an environment are available, the environmental data could be transferred to a mobile device of a user (e.g. passenger or driver of the mobile system) who prepares to get off the mobile system. Transferring the data from the mobile system to the mobile device may be via a server computer or based on a point to point communication. A geometrical model of the environment could be reconstructed based on the environmental data received in the mobile device. The user may then use the geometrical model for tracking.);
performing a central 3D reconstruction based on the 3D reconstruction request (Col. 17, Lines 6-9 teach that generating the first geometrical model or a part of the first geometrical model may be performed by a server computer, and the environmental data is transferred from the mobile system to the server computer.);
and sending a result of the central 3D reconstruction to the mobile device (Col. 17, Lines 9-12 teach that then, the first geometrical model is transferred from the server computer to the mobile device, e.g. via the mobile system or based on a point to point communication.).
Regarding claim 32, Meier discloses a server for performing 3D, three-dimensional, reconstruction for use in a model of a physical environment captured by at least one sensor of a mobile device (Col. 15, Lines 16-21 teach that the geometrical model may also be generated in a server computer based on the environmental data. In this case, the captured environmental data is transferred from the mobile system to the server computer and then the generated geometrical model is transferred from the server computer to the mobile device.), the server comprising:
a processor (Col. 16, Lines 61-64 teach that generating the first geometrical model or a part of the first geometrical model may be performed at processor devices of the mobile system, and the first geometrical model is transferred from the mobile system to the mobile device.);
and a memory storing instructions (Col. 6, Lines 4-14 teach that according to another aspect, the invention is also related to a computer program product comprising software code sections which are adapted to perform a method according to the invention. Particularly, the software code sections are contained on a computer readable medium which are non-transitory. The software code sections may be loaded into a memory of one or more processing devices as described herein. Any used processing devices may communicate via a communication network, e.g. via a server computer or a point to point communication, as described herein.) that, when executed by the processor, cause the server to:
receive a 3D reconstruction request from the mobile device, wherein the 3D reconstruction request comprises data based on sensor data obtained by sensors of the mobile device (Col. 15, Lines 1-11 teach that if environmental data captured by sensors of a mobile system during the mobile system travelling in an environment are available, the environmental data could be transferred to a mobile device of a user (e.g. passenger or driver of the mobile system) who prepares to get off the mobile system. Transferring the data from the mobile system to the mobile device may be via a server computer or based on a point to point communication. A geometrical model of the environment could be reconstructed based on the environmental data received in the mobile device. The user may then use the geometrical model for tracking.);
perform a central 3D reconstruction based on the 3D reconstruction request (Col. 17, Lines 6-9 teach that generating the first geometrical model or a part of the first geometrical model may be performed by a server computer, and the environmental data is transferred from the mobile system to the server computer.);
and send a result of the central 3D reconstruction to the mobile device (Col. 17, Lines 9-12 teach that then, the first geometrical model is transferred from the server computer to the mobile device, e.g. via the mobile system or based on a point to point communication.).
Regarding claim 34, Meier discloses everything claimed as applied above (see claim 32), in addition, Meier discloses wherein the result of the central 3D reconstruction comprises a pose of the mobile device determined by the server (Col. 9, Lines 47-57 teach that it is also possible to transfer environmental data ED to another computer, e.g. a server computer remote from the mobile device and mobile system, and create a geometrical model Md of the environment based on the environmental data ED on such server computer, e.g. by an application running on the server computer. In such configuration, the server computer is communicating in a client-server architecture with the mobile device and mobile system as client devices. Then, the environmental data ED and/or the geometrical model Md is transferred from the server computer to the mobile device.).
Regarding claim 35, the computer program steps correlate to and are rejected similarly to the method steps of claim 29. Additionally, Meier discloses a computer program product comprising a non-transitory computer readable medium storing a computer program for performing 3D, three-dimensional, reconstruction for use in a model of a physical environment captured by at least one sensor of a mobile device (Col. 6, Lines 4-14 teach that according to another aspect, the invention is also related to a computer program product comprising software code sections which are adapted to perform a method according to the invention. Particularly, the software code sections are contained on a computer readable medium which are non-transitory. The software code sections may be loaded into a memory of one or more processing devices as described herein. Any used processing devices may communicate via a communication network, e.g. via a server computer or a point to point communication, as described herein.).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-3, 16-20, 22, 25-27 and 33 are rejected under 35 U.S.C. 103 as being unpatentable over Meier in view of Lawlor et al. (Pub. No.: US 2021/0089572 A1), hereinafter Lawlor.
Regarding claim 1, Meier discloses a method for performing 3D, three-dimensional, reconstruction for use in a model of a physical environment captured by at least one sensor of a mobile device (FIG. 3 and Col. 6, Lines 24-28 teach that FIG. 3 shows a flowchart of a method according to an embodiment of the invention generating a geometrical model of the environment based on environment data acquired by sensors of a mobile system and tracking a device based on the generated environment model),
the method being performed by a system comprising the mobile device and a server (Col. 5, Lines 11-20 teach that according to a further embodiment, generating the first geometrical model or a part of the first geometrical model is performed by a processing device of the mobile system, and the first geometrical model is transferred from the mobile system to the mobile device. For example, the first geometrical model is transferred from the mobile system to the mobile device via a server computer or via a point to point communication between the mobile system and the mobile device or via a broadcast or multicast communication (e.g. the mobile system broadcasts data).), the method comprising:
obtaining, by the mobile device, sensor data from sensors of the mobile device (Col. 10, Lines 16-23 teach that for example, a geometrical model of a real environment may be generated by depth data of the environment provided by depth sensors of a mobile system, for example from range sensors or time of flight cameras mounted in the mobile system, while driving the mobile system in the environment. Many methods could be employed for reconstructing a 3D surface of the real environment from depth data. Push broom scanners may be used to create a 3D surface);
determining, by the mobile device, a pose estimate of the mobile device based on the sensor data (Col. 12, Lines 21-25 teach that after feature matching, correspondences between features from feature set FA and feature set FB are created. The correspondences could be 2D-2D or 2D-3D. Based on the correspondences, a camera pose relative to the environment or to the one of previous camera poses is determined. Additionally, Col. 19, Lines 60-64 teach that using one captured camera image without depth data generates model information of the environment with undetermined metric scale. The model information with undetermined metric scale may be used to estimate camera poses when the camera undergoes a pure rotation.). However, Meier fails to disclose estimating, by the mobile device, a pose error of the pose estimate.
Lawlor discloses estimating, by the mobile device, a pose error of the pose estimate (Paragraph 58 teaches that referring back to the satellite-based positioning system example, the predicting module 207 can use the meta-data fields listed in Table 1 as predictors X to predict/estimate the 3D error y described above as a pose error. In short, the metadata fields related to positions, velocities, and orientations of cameras, the confidence (e.g., standard deviations) of the reported positions, velocities, and orientations, the number of GNSS satellites visible at the time of collection, the horizontal and vertical dilution of precision of the GNSS constellation at the time of collection, etc. Additionally, paragraph 59 teaches that in other embodiments, wherein the sensor system includes a LiDAR system or a Radar system, the data model module 205 can similarly determine the estimated location of each survey point 415, except in these instances, the point position is within a point cloud generated by the LiDAR or Radar system rather than on an image plane as described above. The model module 205, for instance, identifies the positions of the survey points 415a-415c in the cloud and measures the distance from them to the surveyed positions to determine the pose error associated with the sensor system.). Since Meier teaches the initial steps for performing 3D reconstruction using a mobile device to estimate poses from different sensor data and Lawlor teaches using sensor data from mobile devices to create and estimate pose errors, it would have been obvious to a person having ordinary skill in the art to combine the teachings together, so that any of the estimated poses discovered from the sensor data could then utilize the data from the sensors to create estimated pose errors for further uses.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Meier to incorporate the functions of Lawlor, so that the combined features together would allow for the sensors data to be combined with the estimated poses, to establish estimated pose errors that would help with improving the accuracy of data, especially when used with such systems as SLAM or LiDAR.
Furthermore, Meier in view of Lawlor disclose comparing, by the mobile device, the pose error against an error threshold (Paragraph 64 of Lawlor teaches that in another embodiment, the predicting module 207 flags certain sensor system pose data when the aggregation of minimum distances is greater than an error threshold.);
performing, by the mobile device, a device 3D reconstruction when the pose error is determined to be smaller than the error threshold, resulting in updates to a device 3D model (FIGS 6A-6B and paragraph 61 teach that FIGS. 6A-6B are diagrams of user interfaces illustrating examples of predicted pose errors of different classes for images captured in Washington DC by a sensor system (e.g., GPS, IMU, camera, LiDAR, Radar, etc.), according to one embodiment. FIG. 6A show a map view 600 and a histogram 610 of the predicted probabilities for the “good” class (0<=y<0.2 meter) for a number of images captured in Washington DC. The histogram 610 is a representation of an estimated probability distribution of 0<pose error<0.2 meter data points, with a mean μ=______ meter and a standard deviation σ=______. Additionally, Col. 16, Lines 5-15 of Meier teach that after parking (see FIG. 4, depiction 405), the geometrical model is transferred to the user's mobile device 408 (or generated at the user's mobile device 408 from transferred environmental data) equipped with a camera (e.g. a smart phone). Then, tracking the mobile device in the parking lot can ideally continue seamlessly based on at least one image captured by the camera of the mobile device 408 (see FIG. 4, depiction 405). In contrast to existing approaches, the present invention provides the user with an initial and up-to-date geometrical model of the environment on his mobile device.);
sending, by the mobile device, a 3D reconstruction request to the server, when the pose error is determined to be greater than the error threshold, wherein the 3D reconstruction request comprises data based on the sensor data (Col. 16, Line 61 through Col. 17, Line 5 of Meier teach that generating the first geometrical model or a part of the first geometrical model may be performed at processor devices of the mobile system, and the first geometrical model is transferred from the mobile system to the mobile device. The first geometrical model may be transferred from the mobile system to the mobile device via a server computer or based on a point to point communication. The first geometrical model or a part of the first geometrical model may also be generated by processor device(s) of the mobile device, after the environmental data is transferred from the mobile system to the mobile device via a server computer or based on a point to point communication. Additionally, paragraph 64 of Lawlor teaches that in another embodiment, the predicting module 207 flags certain sensor system pose data when the aggregation of minimum distances is greater than an error threshold.);
performing, by the server, a central 3D reconstruction based on the 3D reconstruction request (Col. 17, Lines 6-9 of Meier teach that generating the first geometrical model or a part of the first geometrical model may be performed by a server computer, and the environmental data is transferred from the mobile system to the server computer.);
sending, by the server, a result of the central 3D reconstruction to the mobile device (Col. 17, Lines 9-12 of Meier teach that then, the first geometrical model is transferred from the server computer to the mobile device, e.g. via the mobile system or based on a point to point communication.);
and performing, by the mobile device, a 3D model fusion of a device 3D model in the mobile device and the result of the central 3D reconstruction, wherein the device 3D model, at least partly, is a result of previous device 3D reconstruction (Col. 18, Line 53 through Col. 19, Line 8 of Meier teach that in another embodiment, data acquired by at least one sensor of the mobile system may not be sufficient to create at least part of the first geometrical model. Similarly, data acquired by a camera of the mobile device may also not be sufficient to create at least part of the first geometrical model. However, at least part of the first geometrical model may be created by using both data acquired by the at least one sensor of the mobile system and by the camera of the mobile device. For example, it may not be possible to create the first geometrical model by using only one image captured by a camera of the mobile system or by using only one image captured by a camera of the mobile device. However, it may be possible to create the first geometrical model by using the image captured by the camera of the mobile system and the image captured by the camera of the mobile device. Further, it may not be possible to create at least part of the first geometrical model with a correct metric scale by using either data acquired by at least one sensor of the mobile system or data acquired by a camera of the mobile device. However, it may be possible to create at least part of the first geometrical model with a correct metric scale by using, both, data acquired by the at least one sensor of the mobile system and by the camera of the mobile device.).
Regarding claim 2, the method steps correspond to and are rejected similarly to the method steps of claim 1 (see claim 1 above). These claims are practically identical to the claims of claim 1 but would not necessarily be considered identical if the 112(b) issue (See 112(b) rejection above) is addressed and mention of “the server” would be removed from the claim language.
Regarding claim 3, Meier in view of Lawlor disclose everything claimed as applied above (see claim 2), in addition, Meier in view of Lawlor disclose wherein the sensors include at least a camera (Col. 8, Lines 17-19 of Meier teach that a camera attached to a mobile device is an appropriate sensor for tracking the device and reconstructing a geometrical model of the environment. Additionally, Col. 10, Line 64 through Col. 11, Line 5 of Meier teach that referring now to FIG. 1, given at least one camera, a process of creating or generating a geometrical model and/or computing camera poses based on images captured by the at least one camera may consist of feature detection (step 102 or 105), feature description (step 102 or 105), feature matching (step 106), triangulation (step 107) and optionally (global) map refinement which adjusts triangulation positions and/or camera poses, and/or removes and/or adds points from the triangulation.).
Regarding claim 16, the mobile device steps correspond to and are rejected similarly to the method steps of claim 1 or claim 2 (see claims 1 or 2 above). Additionally, Meier discloses a mobile device for performing 3D, three-dimensional, reconstruction for use in a model of a physical environment captured by at least one sensor of the mobile device (Col. 5, Lines 11-20 teach that according to a further embodiment, generating the first geometrical model or a part of the first geometrical model is performed by a processing device of the mobile system, and the first geometrical model is transferred from the mobile system to the mobile device. For example, the first geometrical model is transferred from the mobile system to the mobile device via a server computer or via a point to point communication between the mobile system and the mobile device or via a broadcast or multicast communication (e.g. the mobile system broadcasts data).), the mobile device comprising:
a processor (Col. 16, Lines 61-64 teach that generating the first geometrical model or a part of the first geometrical model may be performed at processor devices of the mobile system, and the first geometrical model is transferred from the mobile system to the mobile device.);
and a memory storing instructions (Col. 6, Lines 4-14 teach that according to another aspect, the invention is also related to a computer program product comprising software code sections which are adapted to perform a method according to the invention. Particularly, the software code sections are contained on a computer readable medium which are non-transitory. The software code sections may be loaded into a memory of one or more processing devices as described herein. Any used processing devices may communicate via a communication network, e.g. via a server computer or a point to point communication, as described herein.)
Regarding claim 17, Meier in view of Lawlor disclose everything claimed as applied above (see claim 16), in addition, Meier in view of Lawlor disclose wherein the sensors include at least a camera (Col. 8, Lines 17-19 of Meier teach that a camera attached to a mobile device is an appropriate sensor for tracking the device and reconstructing a geometrical model of the environment. Additionally, Col. 10, Line 64 through Col. 11, Line 5 of Meier teach that referring now to FIG. 1, given at least one camera, a process of creating or generating a geometrical model and/or computing camera poses based on images captured by the at least one camera may consist of feature detection (step 102 or 105), feature description (step 102 or 105), feature matching (step 106), triangulation (step 107) and optionally (global) map refinement which adjusts triangulation positions and/or camera poses, and/or removes and/or adds points from the triangulation.).
Regarding claim 18, Meier in view of Lawlor disclose everything claimed as applied above (see claim 16), in addition, Meier in view of Lawlor disclose wherein the instructions to determine a pose estimate comprise instructions that, when executed by the processor, cause the mobile device to determine the pose estimated based on a SLAM, simultaneous localisation and mapping, procedure (Col. 6, Lines 19-20 of Meier teaches that FIG. 1 shows a flowchart of a method according to an embodiment of the invention using SLAM and additionally Col. 20, Lines 10-22 teach that tracking the mobile device and/or generating the second geometrical model could also be realized by monocular vision based Simultaneous Localization and Mapping (SLAM). Generating the second geometrical model may also include reconstruction algorithms that do not run simultaneously, but use batch/quasi-offline reconstruction methods. Monocular vision based SLAM is about moving a single camera in a real environment to determine camera poses relative to the real environment and create a model of the real environment. SLAM is often employed for tracking when at least part of the environment has an unknown geometrical model.).
Regarding claim 19, Meier in view of Lawlor disclose everything claimed as applied above (see claim 18), in addition, Meier in view of Lawlor disclose wherein the instructions to estimate a pose error comprise instructions that, when executed by the processor, cause the mobile device to determine a pose error based on an odometry component that is usable for the SLAM procedure (Paragraph 91 of Lawlor teaches that by way of example, the feature correspondence data records 709 can be associated with one or more of the node records 703, road segment records 705, and/or POI data records 707 to support localization or visual odometry based on the features stored therein and the corresponding estimated quality and/or pose errors of the features. In this way, the records 709 can also be associated with or used to classify the characteristics or metadata of the corresponding records 703, 705, and/or 707. Additionally, Col. 13, Lines 17-32 of Meier teach that in most cases, a geometrical model of an unknown environment created based on environmental data captured by sensors of a mobile system should be superior to a model created from a hand-held mobile device in terms of accuracy and consistency. This is because there are more sensors in the mobile system that can be used to cross-check the (intermediate) results of the reconstruction process. For example, correspondences can be validated via the overlap between two camera images from the same time or via predictions of object positions from the odometry; specifically, the steering angle of the front wheels and the speed of the mobile system can be used to predict how the image of a certain real object may have moved from one to the other camera frame (camera image), where the prediction is dependent on the depth of the real object relative to the camera.).
Regarding claim 20, Meier in view of Lawlor disclose everything claimed as applied above (see claim 18), in addition, Meier in view of Lawlor disclose wherein the instructions to estimate a pose error comprise instructions that, when executed by the processor, cause the mobile device to reduce the pose error when a localisation of the mobile device in the SLAM procedure occurs (Paragraph 66 of Lawlor teaches that in one embodiment, the geographic database 123 includes representations of features and/or other related geographic features determined from feature correspondences to facilitate visual odometry to increase localization accuracy.).
Regarding claim 22, Meier in view of Lawlor disclose everything claimed as applied above (see claim 16), in addition, Meier in view of Lawlor disclose wherein the result of the central 3D reconstruction comprises a pose of the mobile device determined by the server (Col. 9, Lines 47-57 of Meier teach that it is also possible to transfer environmental data ED to another computer, e.g. a server computer remote from the mobile device and mobile system, and create a geometrical model Md of the environment based on the environmental data ED on such server computer, e.g. by an application running on the server computer. In such configuration, the server computer is communicating in a client-server architecture with the mobile device and mobile system as client devices. Then, the environmental data ED and/or the geometrical model Md is transferred from the server computer to the mobile device.).
Regarding claim 25, Meier in view of Lawlor disclose everything claimed as applied above (see claim 16), in addition, Meier in view of Lawlor disclose wherein the 3D reconstruction request comprises at least part of a most recent device 3D model and sensor data obtained after determining the most recent device 3D model (Col. 18, Lines 1-5 of Meier teach that the mobile device may be tracked according to the created first geometrical model and images captured by the camera of the mobile device within a set time period, preferably within 24 hours, after the acquisition process or a part of the acquisition process. Additionally, Col. 18, Lines 23-33 of Meier teach that in one embodiment, the generation of the first geometrical model may be based at least in part on mobile system state data acquired in an acquisition process by at least one sensor of a mobile system. For example, at least part of the environmental data for generating the first geometrical model may be acquired by an independent sensor that is not part of the mobile system. The acquired at least part of the environmental data may be used together with mobile system state data acquired by one or more sensors of the mobile system in order to create at least part of the first geometrical model.).
Regarding claim 26, Meier in view of Lawlor disclose everything claimed as applied above (see claim 16), in addition, Meier in view of Lawlor disclose wherein the 3D reconstruction request comprises the pose estimate (Col. 17, Lines 41-52 of Meier teach that the first geometrical model may also be generated according to a pose of the mobile system relative to the environment. The position of the mobile system relative to the environment may be obtained from GPS. However, GPS is not accurate, especially when the mobile system is inside buildings. There are many different sensors that may be installed in the environment, like cameras (e.g. security camera), and have known positions in the environment. It is possible to perform object recognition or pose estimation based on images of the mobile system captured by the security camera in order to determine the pose of the mobile system relative to the environment.).
Regarding claim 27, the computer program steps correlate to and are rejected similarly to the mobile device steps of claim 16. Additionally, Meier discloses a computer program product comprising a non-transitory computer readable medium storing a computer program for performing 3D, three-dimensional, reconstruction for use in a model of a physical environment captured by at least one sensor of a mobile device (Col. 6, Lines 4-14 of Meier teach that according to another aspect, the invention is also related to a computer program product comprising software code sections which are adapted to perform a method according to the invention. Particularly, the software code sections are contained on a computer readable medium which are non-transitory. The software code sections may be loaded into a memory of one or more processing devices as described herein. Any used processing devices may communicate via a communication network, e.g. via a server computer or a point to point communication, as described herein.).
Claim 21 is rejected under 35 U.S.C. 103 as being unpatentable over Meier in view of Lawlor as applied to claim 20 above, and further in view of Knorr et al. (U.S. Patent: #10,445,895 B2), hereinafter Knorr.
Regarding claim 21, Meier in view of Lawlor disclose everything claimed as applied above (see claim 20), however, Meier in view of Lawlor fail to disclose wherein the instructions to estimate a pose error comprise instructions that, when executed by the processor, cause the mobile device to set the pose error to zero when a localisation of the mobile device in the SLAM procedure occurs.
Knorr discloses wherein the instructions to estimate a pose error comprise instructions that, when executed by the processor, cause the mobile device to set the pose error to zero when a localisation of the mobile device in the SLAM procedure occurs (Col. 28, Lines 42-50 teach that according to a further embodiment, the method treats the spatial distance between coordinate systems F and W as zero, thereby ignoring the translational movement for the pose of coordinate system (camera) F induced by a rotation of coordinate system (camera) W and vice versa, which leads to an error for poses of cameras F1 and F2 corresponding to poses of cameras W1 and W2 that is less or equal to the actual distance between coordinate system (camera) F and (camera) W.). Since Meier in view of Lawlor teach the initial mobile device steps for generating 3D reconstruction using pose error estimates and data when implementing SLAM procedures and Knorr teaches a function for setting coordinates and pose data to zero within a system used for 3D reconstruction, it would have been obvious to a person having ordinary skill in the art to combine the teachings together, so that the combined functions would allow for the ability to change and set a pose error to zero.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Meier in view of Lawlor to incorporate the functions of Knorr, so that the combined features together would help improve the accuracy of the localization and position information being used when performing SLAM.
Claims 23-24 and 33 are rejected under 35 U.S.C. 103 as being unpatentable over Meier in view of Lawlor as applied to claim 20 above, and further in view of Hasegawa et al. (Pub. No.: US 2023/0360236 A1), hereinafter Hasegawa.
Regarding claim 23, Meier in view of Lawlor disclose everything claimed as applied above (see claim 16), however, Meier in view of Lawlor fail to disclose further comprising instructions that, when executed by the processor, cause the mobile device to adjust the error threshold to decrease the error threshold when a quality of a device 3D reconstruction is greater than a quality threshold.
Hasegawa disclose further comprising instructions that, when executed by the processor, cause the mobile device to adjust the error threshold to decrease the error threshold when a quality of a device 3D reconstruction is greater than a quality threshold (Paragraph 94 through 97 teach that (7) In addition, the calibration device 300 calculates the estimated movement amount of the camera 20, from the three-dimensional coordinate information obtained by the conversion. (8) Then, the calibration device 300 calculates from the estimated movement amount of the camera 20 calculated in the above (7) and the LiDAR movement amount 120, the relative position and the relative pose between the camera 20 and the LiDAR 30. (9) Next, the calibration device 300 evaluates a change amount between the relative position and the relative pose obtained in the process of (8), and the relative position and the relative pose obtained in the process of (1). When the change amount is equal to or greater than a threshold value, the calibration device 300 performs the processes of (2) to (8) again using, the relative position and the relative pose calculated in the process of (8) since the relative position and the relative pose with higher accuracy may be calculated.). Since Meier in view of Lawlor teach the initial mobile device steps for estimating pose errors using camera sensor data when performing 3D reconstruction and can utilize error thresholds to make different decisions and Hasegawa teaches making changes and adjustments to camera sensor data that will help improve and reduce pose error data by making adjustments to the poses to improve their overall accuracy, it would have been obvious to a person having ordinary skill in the art to combine the teachings together, so that if a particular error threshold is triggered, adjustments could then be made to the help improve the overall quality of the 3D reconstruction that would be produced.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Meier in view of Lawlor to incorporate the functions of Hasegawa, so that the combined features together would allow for more improved accuracy and precision when generating the 3D reconstruction by being able to adjust and lower the error threshold when needed.
Regarding claim 24, Meier in view of Lawlor and Hasegawa disclose everything claimed as applied above (see claim 23), in addition, Meier in view of Lawlor and Hasegawa disclose wherein the instructions to adjust comprise instructions that, when executed by the processor, cause the mobile device to adjust the error threshold based on a central pose error indication received from the server (Paragraph 99 of Hasegawa teaches that when the change amount is equal to or greater than the threshold value, the calibration device 300 performs the processes of (2) to (8) again using the relative position and the relative pose calculated in the process of (8) since the relative position and the relative pose with higher accuracy may be calculated. Additionally, paragraph 75 of Lawlor teaches communications between the network nodes are typically effected by exchanging discrete packets of data. Each packet typically comprises (1) header information associated with a particular protocol, and (2) payload information that follows the header information and contains information that may be processed independently of that particular protocol. In some protocols, the packet includes (3) trailer information following the payload and indicating the end of the payload information. The header includes information such as the source of the packet, its destination, the length of the payload, and other properties used by the protocol and paragraph 105 of Lawlor teaches that for example, in wireless handheld devices, such as mobile telephones like cell phones, the communications interface 870 includes a radio band electromagnetic transmitter and receiver called a radio transceiver. In certain embodiments, the communications interface 870 enables connection to the communication network 121 for predicting a pose error for a sensor system based on a trained machine learning model.).
Regarding claim 33, Meier discloses everything claimed as applied above (see claim 32), however, Meier fails to disclose further comprising instructions that, when executed by the processor, cause the server to adjust the error threshold based on a central pose error indication received from the server.
Lawlor discloses comprising instructions that, when executed by the processor, cause the server to adjust a central pose error indication received from the server (Paragraph 75 teaches that communications between the network nodes are typically effected by exchanging discrete packets of data. Each packet typically comprises (1) header information associated with a particular protocol, and (2) payload information that follows the header information and contains information that may be processed independently of that particular protocol. In some protocols, the packet includes (3) trailer information following the payload and indicating the end of the payload information. The header includes information such as the source of the packet, its destination, the length of the payload, and other properties used by the protocol. Additionally, paragraph 105 teaches that for example, in wireless handheld devices, such as mobile telephones like cell phones, the communications interface 870 includes a radio band electromagnetic transmitter and receiver called a radio transceiver. In certain embodiments, the communications interface 870 enables connection to the communication network 121 for predicting a pose error for a sensor system based on a trained machine learning model.). Since Meier teaches the initial steps for performing 3D reconstruction using a server to estimate poses from different sensor data and Lawlor teaches using sensor data from mobile devices to create and estimate pose errors and share potential pose error data from a server, it would have been obvious to a person having ordinary skill in the art to combine the teachings together, so that any of the estimated poses discovered from the server sensor data could then utilize the data from the sensors to create estimated pose errors for further uses.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Meier to incorporate the functions of Lawlor, so that the combined features together would allow for the sensors data from the server to be combined with the estimated poses, to establish estimated pose errors that would help with improving the accuracy of data, especially when used with such systems as SLAM or LiDAR.
Furthermore, Meier in view of Lawlor fail to disclose adjust the error threshold.
Hasegawa discloses adjust the error threshold (Paragraph 99 teaches that when the change amount is equal to or greater than the threshold value, the calibration device 300 performs the processes of (2) to (8) again using the relative position and the relative pose calculated in the process of (8) since the relative position and the relative pose with higher accuracy may be calculated.). Since Meier in view of Lawlor teach the initial server steps for estimating pose errors using camera sensor data when performing 3D reconstruction and can utilize error thresholds to make different decisions and Hasegawa teaches making changes and adjustments to camera sensor data that will help improve and reduce pose error data by making adjustments to the poses to improve their overall accuracy, it would have been obvious to a person having ordinary skill in the art to combine the teachings together, so that if a particular error threshold is triggered, adjustments could then be made to the help improve the overall quality of the 3D reconstruction that would be produced.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Meier in view of Lawlor to incorporate the functions of Hasegawa, so that the combined features together would allow for more improved accuracy and precision when generating the 3D reconstruction by being able to adjust and lower the error threshold when needed.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Fang et al. (Pub. No.: US 2021/0110599 A1) teaches a depth camera based three dimensional reconstruction method and apparatus, device and storage medium
Pi et al. (Pub. No.: US 2023/0038594 A1) teaches a SLAM-based electronic device and method for calculating pose errors of surrounding environmental objects
Jones et al. (U.S. Patent: #10,269,147 B2) teaches a system for providing camera positions and point cloud estimation 3D reconstruction.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to George Renze whose telephone number is (703)756-5811. The examiner can normally be reached Monday-Friday 9:00am - 6:00pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Xiao Wu can be reached at (571) 272-7761. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/G.R./Examiner, Art Unit 2613
/XIAO M WU/Supervisory Patent Examiner, Art Unit 2613