Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections – 35 U.S.C. § 112
The following is a quotation of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
Rejection under 35 U.S.C. 112(a): Written Description
Claims 1–20 are rejected under 35 U.S.C. 112(a) as failing to comply with the written description requirement. The claims contain subject matter that was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, at the time the application was filed, had possession of the claimed invention.
Independent claims 1, 9, and 17, as amended, recite “a scenario where the vehicle is located and in which communication with a cloud network is prevented.” The original disclosure does not describe a scenario in which communication with a cloud network is prevented. The specification instead describes a network environment that “may be poor” (Specification [0026], [0030]), in which transmitting data to the cloud “may take a relatively long time” ( [0026]), and in which vehicle-side positioning avoids “slow uploading of data or slow obtaining of matching results due to an influence of network transmission conditions” ( [0030], [0058]). The specification further contemplates that the fused map “may be obtained through technologies such as Wi-Fi, cellular, Bluetooth, etc.” ( [0058]).
A network environment that is poor or slow is a network with which communication occurs, albeit with degraded performance. It is not a network with which communication is prevented. The original disclosure therefore describes only a degraded network condition, and does not reasonably convey possession of the distinct condition now claimed, in which communication with the cloud network is prevented. See Ariad Pharms., Inc. v. Eli Lilly & Co., 598 F.3d 1336, 1351 (Fed. Cir. 2010) (en banc); MPEP 2163. When an amendment introduces subject matter into the claims that lacks support in the application as originally filed, the claims are rejected under 35 U.S.C. 112(a). MPEP 2163.06, 608.04.
Applicant may amend the claims to remove the unsupported subject matter, or point out with particularity where support for the limitation is found in the application as originally filed. Claims 2–8, 10–16, and 18–20 depend from claims 1, 9, and 17, respectively, inherit the unsupported limitation, and are rejected for the same reason.
Rejection under 35 U.S.C. 112(b): Indefiniteness
Claims 1–20 are rejected under 35 U.S.C. 112(b) as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor regards as the invention. During examination, a claim is indefinite when it contains words or phrases whose meaning is unclear. In re Packard, 751 F.3d 1307, 1310, 1314 (Fed. Cir. 2014); MPEP 2173.02(I).
Claims 1, 9, and 17 each recite “before a vehicle enters a scenario where the vehicle is located.” This phrase is internally inconsistent. Before the vehicle enters the scenario, the vehicle is not located in that scenario. It is therefore unclear whether “where the vehicle is located” refers to the vehicle’s location at the time of the uploading step, which would contradict “before a vehicle enters,” or to a scenario in which the vehicle will subsequently be located. The phrase is therefore susceptible to more than one reasonable interpretation, and fails to clearly define the metes and bounds of the claimed timing relationship between the uploading and storing steps and the scenario.
For purposes of examination, the phrase is interpreted as referring to a scenario that the vehicle subsequently enters and within which the vehicle is subsequently positioned.
Claims 2–8, 10–16, and 18–20 depend from claims 1, 9, and 17, respectively, inherit the indefinite language, and are rejected for the same reasons.
Claim Interpretation: “uploading from the cloud network to the vehicle”
Claims 1, 9, and 17 each recite “uploading from the cloud network to the vehicle.” The specification uses “uploading” to describe transfer of data from the vehicle to the cloud (Specification [0030], [0058]: “slow uploading of data”), and describes transfer from the cloud to the vehicle as the fused maps being “stored in the cloud server for the vehicle to download and use” ( [0042]). Because the claims expressly recite the direction of transfer, the limitation is interpreted as a transfer of the fused map from the cloud network to the vehicle. Applicant may consider replacing “uploading” with “downloading” or “receiving” to conform the claim terminology to the specification.
Claim Rejections – 35 U.S.C. § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Amended Claims
Claims 1, 9, and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Luo et al. (US 2019/0065863 A1) (hereinafter "Luo") in view of Flint et al. (US 2016/0350926 A1) (hereinafter "Flint").
Regarding Claim 1,
Disclosure by Luo
Luo teaches:
A method for vehicle positioning, comprising:
See at least:
“In contrast, the method 10 according to the present disclosure achieves localization precision in the order of few centimeters substantially free from the above-mentioned drawbacks in some existing approaches.” (Luo [0073])
Rationale: (Express.) Luo expressly teaches a localization method. The localized platform is a vehicle equipped with the sensors ( [0058]), and the programs run on a computing device that is part of a vehicle ( [0055]). Localization of a vehicle is vehicle positioning.
where the vehicle is located
See at least:
“these sensors are used to collect data in an environment.” (Luo [0068])
Rationale: (Express.) Luo's vehicle-mounted sensors collect data in the environment in which the vehicle is present. That environment is the scenario where the vehicle is located.
uploading from the cloud network to the vehicle
See at least:
“The network interface 63 is configured to access program instructions and data accessed by the program instructions stored remotely through a network (not shown).” (Luo [0109])
Rationale: (Implicit / PHOSITA-obvious.) Luo expressly teaches that data accessed by the localization programs is stored remotely and accessed through a network by the vehicle-side network interface. Luo's remote storage is an Internet or virtual server reached over a wireless network ( [0102]–[0103]), which is a cloud network. Luo does not expressly identify the global map as that remotely stored data. However, Luo's localization process expressly uses the global map as the data against which the camera-derived features are matched ( [0071]). A person of ordinary skill in the art (PHOSITA) would have found remote delivery of the global map to the vehicle through Luo's disclosed network path to be a predictable implementation. Luo does not expressly describe distributing the map to multiple vehicles. However, centralized storage permits map data to be maintained at one location and distributed to vehicle clients as needed, which is a predictable implementation benefit rather than a teaching of Luo. That delivery is a transfer from the cloud network to the vehicle. The limitation is interpreted consistent with its expressly recited direction of transfer, from the cloud network to the vehicle.
and storing in a memory of the vehicle
See at least:
“The memory 69 is configured to store program instructions to be executed by the processor 61 and data accessed by the program instructions.” (Luo [0112])
Rationale: (PHOSITA-obvious.) Luo expressly teaches a memory configured to store data accessed by the localization programs, and separately teaches execution of the localization operations by a computing device that is part of a vehicle ( [0055]). Luo does not expressly identify memory 69 as vehicle memory. In the proposed on-vehicle implementation, a PHOSITA would have stored the retrieved map in vehicle-accessible memory so that the map remains locally available for the recited matching operation after network communication becomes unavailable. This is a predictable implementation of Luo's disclosed memory and vehicle-resident processing.
wherein the fused map includes a previously-constructed point cloud basemap of the scenario
See at least:
“Further, in an embodiment, the global map is constructed, based on data from the LiDAR, using LiDAR mapping. The global map includes a 3D city-scale map.” (Luo [0069])
Rationale: (Express under the broadest reasonable interpretation.) Luo's global map is constructed from LiDAR data, and Luo confirms that the global map consists of 3D points by estimating the "distribution of 3D points within the voxels" ( [0081]). The global map is constructed in operation 13 (FIG. 1), before the feature matching of operation 15, so it is previously constructed. It is the reference map against which the vehicle is localized, and is therefore a previously-constructed point cloud basemap of the scenario. Its inclusion within the claimed fused map results from the combination with Flint, addressed below.
in the vehicle, capturing at least one image frame of a surrounding environment of the vehicle
See at least:
“The LiDAR sensor and the cameras are mounted on the roof of the vehicle.” (Luo [0058])
Rationale: (Express.) Luo's cameras are mounted on the vehicle and capture images of the environment around the vehicle ( [0068]). Capture therefore occurs in the vehicle, and each captured image is an image frame of the vehicle's surrounding environment.
within the scenario
See at least:
“In an embodiment, images of the environment are captured by the camera in approximately 30 Hz.” (Luo [0068])
Rationale: (Express.) Luo's image frames are captured of the environment in which the vehicle operates, which is the same environment the global map describes. The frames are therefore captured within the scenario.
through a camera unit
See at least:
“In an embodiment, the camera includes a monocular camera, which is calibrated by multiple shots instead of single shot.” (Luo [0065])
Rationale: (Express.) Luo's camera, whether monocular or stereo ( [0065]), is a camera unit through which the image frames are captured.
and extracting a plurality of feature points from the at least one image frame;
See at least:
“In an embodiment, the 3D submap is constructed, based on images from the camera, using visual SLAM. Reference of visual SLAM can be made to, for example, "ORB-SLAM: a versatile and accurate monocular SLAM system" by Raul Mur-Artal et al., IEEE Transactions on Robotics 31.5 (2015): 1147-1163” (Luo [0069])
Rationale: (PHOSITA-obvious, with Luo's identified ORB-SLAM technique as evidence.) Luo constructs its submap from camera image frames using visual SLAM, and specifically identifies ORB-SLAM as the visual SLAM technique. It was well known to a PHOSITA that visual SLAM, including the ORB-SLAM technique Luo identifies, operates by detecting and extracting keypoint features (feature points) from each captured image frame. The "sparse 3D points" of Luo's submap ( [0070]) are the 3D positions triangulated from those image feature points. Extracting a plurality of feature points from the at least one image frame was therefore the conventional operation of the visual SLAM technique Luo expressly selects. Luo additionally incorporates the cited ORB-SLAM publication by reference ( [0004], [0006]).
in the vehicle, performing a matching of the plurality of feature points with point cloud data in the point cloud basemap
See at least:
“Next, in operation 15, the features extracted from the 3D submap are matched against the features extracted from the global map so that in operation 16 invalid feature correspondences are removed.” (Luo [0071])
Rationale: (Express under the broadest reasonable interpretation.) Luo matches the camera-derived features, including sparse 3D feature points ( [0070]), against features extracted from the 3D point data of the LiDAR global map ( [0081]–[0082]). Matching camera feature points against features drawn from the point cloud is matching the feature points "with point cloud data in the point cloud basemap." The claim does not preclude intermediate processing such as triangulation. Luo's operations are executed by the computing device that is part of the vehicle ( [0055]), so the matching is performed in the vehicle.
to determine a position of the vehicle
See at least:
“Subsequently, in operation 17, location of the 3D submap is iteratively estimated until a distance between corresponding features is minimized.” (Luo [0072])
Rationale: (Express.) Luo's submap is built from the vehicle's own camera images and vehicle poses ( [0075]). Estimating the submap's location in the global map therefore determines the position of the vehicle, which is Luo's stated localization objective ( [0073]).
according to a result of the matching,
See at least:
“In operation 58, based on the position of the 3D submap in operation 51, an iterative estimation of location of the 3D submap is performed until distance between corresponding features reaches a predetermined value.” (Luo [0097])
Rationale: (Express.) Luo's position estimate is driven by the distances between the corresponding features established by the matching step, after invalid correspondences are removed ( [0096]). The position is therefore determined according to a result of the matching.
in the vehicle, measuring a relative displacement of the vehicle within the scenario
See at least:
“Vehicle poses, including position and orientation, are collected in an "east north up" (ENU) coordinate by the inertial navigation module in approximately 50 Hz.” (Luo [0068])
Rationale: (Implicit; alternatively PHOSITA-obvious.) Luo's vehicle-mounted inertial navigation module measures the vehicle's pose as the vehicle moves through the environment. A PHOSITA would have understood that an IMU senses accelerations and angular rates, from which the change in the vehicle's position and orientation relative to a prior state is derived (dead reckoning). Luo's inertial measurements therefore implicitly measure a relative displacement of the vehicle within the scenario. Alternatively, deriving relative displacement from IMU measurements would have been an obvious and conventional use of the IMU Luo discloses.
through an inertial measurement unit
See at least:
“The inertial navigation module in an embodiment according to the present disclosure includes a global navigation satellite system (GNSS)-inertial measurement unit (IMU).” (Luo [0059])
Rationale: (Express.) Luo expressly teaches that the inertial navigation module includes an IMU.
and updating the position of the vehicle
See at least:
“The benefits of using GNSS with an IMU are that the IMU may be calibrated by the GNSS signals and that the IMU can provide position and angle updates at a quicker rate than GNSS. For high dynamic vehicles, IMU fills in the gaps between GNSS positions.” (Luo [0059])
Rationale: (Express as to IMU position updates; PHOSITA-obvious as applied to the map-matched position.) Luo expressly teaches using the IMU to update the vehicle's position at a high rate between lower-rate absolute position fixes. Luo describes this with respect to GNSS fixes. It would have been obvious to apply the same complementary relationship to Luo's own absolute, map-matched position, for three reasons. First, Luo already uses the same inertial navigation module within its localization pipeline ( [0077]). Second, Luo's camera operates at a lower rate than its inertial module (approximately 30 Hz versus approximately 50 Hz, [0068]). Third, Luo teaches that the IMU and absolute sources "are complementary and are often employed together" ( [0059]). The result, IMU updates between visual fixes, is predictable.
according to the relative displacement,
See at least:
“Additionally, GNSS may lose its signal and the IMU can continue to compute the position and angle during the period of lost GNSS signal.” (Luo [0059])
Rationale: (Implicit; alternatively PHOSITA-obvious.) Luo teaches that the IMU continues to compute the position while the absolute fix is unavailable. A PHOSITA would have understood that continued position computation to involve propagating the prior position based on inertially measured motion, thereby updating the position according to the relative displacement. Alternatively, updating position in this manner would have been an obvious and conventional use of the IMU-based position computation Luo discloses.
Claim Limitations Not Explicitly Disclosed by Luo
Luo does not explicitly teach the following claim limitations:
before a vehicle enters a scenario
and in which communication with a cloud network is prevented,
a previously-constructed fused map of the scenario,
and a previously-constructed vector map describing the scenario;
within the vector map
without transferring data to the cloud network; and
within the vector map
without transferring data to the cloud network.
Disclosure by Flint
Flint teaches or, in combination with Luo, renders obvious:
before a vehicle enters a scenario
See at least:
“In some implementations, the interior/exterior of a building (e.g., a warehouse, shopping mall or residential home) is traversed using an electronic computing device having a visual-based inertial navigation system according to the present disclosure to record a map of the building.” (Flint [0071])
Rationale: (PHOSITA-obvious from the combination.) Flint teaches visual-inertial navigation within building interiors that the platform enters and traverses, and teaches that cellular signals cannot reach such indoor environments ( [0003]). Luo teaches that data used by its localization programs is remotely stored and accessed over a network ( [0109]). A PHOSITA would have been motivated to retrieve and locally store the map before entering an area known to lack cellular availability, so that localization could continue without depending on network access inside that area. This is a predictable use of the known technique of pre-loading data before an anticipated loss of connectivity.
and in which communication with a cloud network is prevented,
See at least:
“Since visual-based inertial navigation systems do not require signals from GPS or cell towers, such systems may be used indoors where GPS and cell signals cannot reach or are unavailable due to interference.” (Flint [0003])
Rationale: (PHOSITA-obvious from the combination.) Flint expressly teaches indoor environments in which cell signals "cannot reach." Luo's remote network may be implemented over 3G (Luo [0102]: "Examples of wireless networks may include WiFi®, Bluetooth®, and 3G."). In the combined system, with Luo's remote map access implemented over its disclosed 3G embodiment, Flint's cellular-denied environment would prevent communication with the cloud network. This passage appears in Flint's background. It is cited because Flint's detailed description presupposes, but does not restate, the signal-denied environment for which the system is designed.
a previously-constructed fused map of the scenario,
See at least:
“The positions may be used to compute the path 10, which is stored in memory of the device and/or overlaid on an image of the floor plan, as shown in FIG. 1, and which can be output to a display for a user to view.” (Flint [0029])
Rationale: (PHOSITA-obvious from the combination.) Luo teaches the previously-constructed point cloud map component (Luo [0069]). Flint teaches a previously-existing geometric floor-plan map of the environment on which computed positions are overlaid. Luo expressly establishes vehicle position in the coordinate frame of its point cloud map, transforming the submap coordinate into "a coordinate of the global map" (Luo [0077]). Flint expresses computed positions on its floor-plan map (Flint [0069]). Because the combined system determines position in Luo's global-map coordinate frame and represents that position on Flint's floor-plan map, the two representations require a known spatial correspondence, i.e., registration to a common reference frame. A PHOSITA would have found it obvious to establish that correspondence as part of map preparation and to store and distribute the registered maps as a single fused map of the scenario, rather than maintaining them as unrelated, side-by-side maps. In the proposed combination, both map components would be made available before entry into the cellular-denied environment (Flint [0003]), so that localization and position representation can continue locally thereafter. Delivering them as a single, already-registered dataset is a predictable way to do so. Performing the registration once during map preparation would also avoid repeating the same registration operation on each vehicle and promote consistent map alignment across vehicles, which is a predictable implementation benefit. Flint further recognizes that on-device navigation processing "can be computationally burdensome, especially for devices ... that may have limited computation power" (Flint [0026]). That provides additional reason to perform non-time-critical preparatory processing before deployment rather than on the vehicle. In the fused map, Luo's point cloud map serves feature-matching localization and Flint's floor-plan map serves geometric representation and position display. Each component performs its known function, and the result is predictable. The fused map transferred to the vehicle is therefore previously constructed.
and a previously-constructed vector map describing the scenario;
See at least:
“The floor plan is an overview of a generic hardware store including the different sections (e.g., plumbing section, tools and hardware section, etc.), where the shaded regions represent shelves and the white regions represent the aisles and other areas of the store in which a customer may move.” (Flint [0029])
Rationale: (PHOSITA-obvious.) Flint teaches a floor-plan map that describes the geometry of the environment: its shelves, aisles, and traversable areas. The floor plan exists before navigation, since the computed path is overlaid onto it, so it is previously constructed. Flint does not state the storage format of the floor plan. Flint's floor plan is composed of geometric elements (the lines and rectangles bounding the shelves, aisles, and sections shown in FIG. 1). A PHOSITA would have recognized that such a geometric map may be stored either as a raster image or as the geometric elements themselves. Implementing it as vector graphic elements (lines, rectangles, polygons) would have been a predictable choice among a limited number of known representation formats, yielding the known benefits of a compact representation that can be scaled for display without loss.
within the vector map
See at least:
“The position and orientation estimates determined by the SWF module can be output to the display as points on a map (e.g., before or after post-processing) to provide a user of the electronic computing device visual information about the device's current and past positions.” (Flint [0069])
Rationale: (Express as to positions within the map; PHOSITA-obvious as applied to the combination.) Flint expressly expresses the visually determined position as points within the map describing the environment. In the combination, the position Luo determines by feature matching is therefore determined within the floor-plan (vector) map.
without transferring data to the cloud network; and
See at least:
“In some cases, the SWF module is configured to be a part of the electronic computing device that is being moved. In other implementations, the SWF module is separate from the electronic computing device, e.g., the SWF module may be operating on a server, whereas the IMU and image sensor are in the electronic computing device.” (Flint [0069])
Rationale: (PHOSITA-obvious design choice.) Flint expressly identifies two alternatives: performing the visual-based position determination on the moving device itself, or transferring data to a server for processing. Given those identified alternatives, it would have been obvious to select the on-device embodiment, a choice among a finite number of identified, predictable solutions (KSR). Read together with the preceding limitations, in the combined system the position determination is performed in a scenario in which communication with the cloud network is prevented, and the on-device embodiment is selected. The position determination therefore proceeds without transferring data to the cloud network.
within the vector map
See at least:
“For example, the post-processing module 116 may include a program that uses the model parameters output by the SWF module 108 to determine a path history of the device 100 and store the path history in device memory and/or overlay the path history on a map that is output to a display, such as shown in FIG. 1.” (Flint [0042])
Rationale: (Express as to updated positions within the map; PHOSITA-obvious as applied to the combination.) Flint teaches that each successive position update ("as each new set of model parameters is output," [0042]) is added to the path history and overlaid on the floor-plan map. In the combination, Luo's IMU-updated position is therefore updated within the vector map.
without transferring data to the cloud network.
See at least:
“Using features tracked in the images and corresponding information obtained from an IMU also operating on the device, the SWF obtains estimates for the inertial state of the device (e.g., position, orientation, velocity, and/or gyro and accelerometer biases), which may be referred to as simply the "state" of the device.” (Flint [0028])
Rationale: (PHOSITA-obvious design choice.) Flint teaches that the IMU operates on the device and that the inertial-state (position) estimate is obtained from it. Together with Flint's identified on-device processing alternative ( [0069]), it would have been obvious to perform the inertial position update locally. In the combined system, that update is performed in a scenario in which communication with the cloud network is prevented, using the on-device embodiment. The position update therefore proceeds without transferring data to the cloud network.
Motivation to Combine Luo and Flint
Therefore, given the teachings as a whole, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having Luo and Flint before them, to modify Luo’s vehicle positioning method to (i) retrieve Luo’s previously-constructed LiDAR point cloud global map from Luo’s remote network storage and store it in vehicle memory, associated with a vector floor-plan map of the environment as a fused map, before the vehicle enters an environment in which cellular communication with that remote network is unavailable; (ii) perform Luo’s feature matching and IMU-based position updating on the vehicle without transferring data to the cloud network; and (iii) express the determined and updated positions within the vector map, as taught by Flint.
Both references are in the same field of endeavor, localization of a moving platform using a camera and an inertial measurement unit (both classified in CPC G01C 21/165), and both address the same recognized problem of GPS unavailability (Luo [0003]; Flint [0002]).
Luo teaches that data used by its localization programs is stored remotely and accessed over a network ( [0109]), and Luo’s localization process expressly uses the global map as the data against which camera-derived features are matched ( [0071]). A PHOSITA would therefore have understood Luo’s remote-access path to be a natural source for delivering the global map to the vehicle. Luo further discloses that its wireless network may be 3G ( [0102]). Flint teaches that its system is used indoors "where GPS and cell signals cannot reach" ( [0003]), including building interiors such as warehouses ( [0071]). A PHOSITA would have been motivated to extend Luo’s high-precision, map-based localization to such environments to improve the reliability and availability of vehicle positioning. To do so, the PHOSITA would retrieve and locally store the map before entering an area known to lack cellular availability, so that localization could continue without depending on network access inside that area. Luo already provides both remote map access ( [0109]) and vehicle-resident memory and processing ( [0055], [0112]).
Flint expressly identifies on-device processing as an alternative to server-based processing ( [0069]). Selecting that alternative, and omitting network transfer during positioning, is a choice among a finite number of identified, predictable solutions. It is especially apt where cellular communication is unavailable.
Flint further teaches that expressing position on a map of the environment provides the user visual information about current and past positions ( [0069]), improving situational awareness and control. Because a position determined in Luo’s point cloud map (whose coordinate frame Luo establishes, [0077]) is to be displayed on the floor-plan map, the two maps must be registered to a common reference frame, which would have motivated integrating them into a single fused map in which the point cloud map serves localization and the floor-plan map serves geometric representation and display. A PHOSITA would have performed that registration as part of map preparation, before the fused map is stored remotely and transferred to the vehicle. In the proposed combination, both map components would be made available before entry so that positioning and position representation can continue locally. Registering once during map preparation also avoids repeating the same operation on each vehicle and promotes consistent alignment across vehicles.
The modification combines known elements according to known methods, with no change in their respective functions. It yields the predictable result of continuous, accurate vehicle positioning in communication-denied environments. KSR Int’l Co. v. Teleflex Inc., 550 U.S. 398, 416–417 (2007).
Regarding Claim 9,
Disclosure by Luo
Luo discloses:
A device for vehicle positioning, comprising:
See at least:
“FIG. 6 is a block diagram of a system 60 for localization, in accordance with some embodiments.” (Luo [0098])
Rationale: (Express.) Luo's system 60 is an apparatus that performs localization. Its computing device is part of a vehicle ( [0055]), so it is a device for vehicle positioning.
a memory having stored computer instructions thereon; and
See at least:
“The memory 69 is configured to store program instructions to be executed by the processor 61 and data accessed by the program instructions.” (Luo [0112])
Rationale: (Express.) Luo expressly discloses a memory storing program instructions, which are computer instructions.
a processor,
See at least:
“Referring to FIG. 6, the system 60 includes a processor 61, a computer server 62, a network interface 63, an input and output (I/O) device 65, a storage device 67, a memory 69, and a bus or network 68.” (Luo [0099])
Rationale: (Express.) Luo expressly discloses a processor.
wherein the instructions, when executed by the processor, cause the processor to perform a method for vehicle positioning, the method comprising:
See at least:
“The processor 61 is configured to execute program instructions that include a tool module configured to perform a method as described and illustrated with reference to FIGS. 1 to 5.” (Luo [0104])
Rationale: (Express.) Luo's processor executes the stored instructions to perform the localization method of FIGS. 1–5. That method achieves centimeter-level localization of the vehicle ( [0073]), so it is a method for vehicle positioning.
where the vehicle is located
See at least:
“these sensors are used to collect data in an environment.” (Luo [0068])
Rationale: (Express.) Luo's vehicle-mounted sensors collect data in the environment in which the vehicle is present. That environment is the scenario where the vehicle is located.
uploading from the cloud network to the vehicle
See at least:
“The network interface 63 is configured to access program instructions and data accessed by the program instructions stored remotely through a network (not shown).” (Luo [0109])
Rationale: (Implicit / PHOSITA-obvious.) Luo expressly discloses that data accessed by the localization programs is stored remotely and accessed through a network by the vehicle-side network interface. Luo's remote storage is an Internet or virtual server reached over a wireless network ( [0102]–[0103]), which is a cloud network. Luo does not expressly identify the global map as that remotely stored data. However, Luo's localization process expressly uses the global map as the data against which the camera-derived features are matched ( [0071]). A person of ordinary skill in the art (PHOSITA) would have found remote delivery of the global map to the vehicle through Luo's disclosed network path to be a predictable implementation. Luo does not expressly describe distributing the map to multiple vehicles. However, centralized storage permits map data to be maintained at one location and distributed to vehicle clients as needed, which is a predictable implementation benefit rather than a teaching of Luo. That delivery is a transfer from the cloud network to the vehicle. The limitation is interpreted consistent with its expressly recited direction of transfer, from the cloud network to the vehicle.
and storing in a memory of the vehicle
See at least:
“The memory 69 is configured to store program instructions to be executed by the processor 61 and data accessed by the program instructions.” (Luo [0112])
Rationale: (PHOSITA-obvious.) Luo expressly discloses a memory configured to store data accessed by the localization programs, and separately discloses execution of the localization operations by a computing device that is part of a vehicle ( [0055]). Luo does not expressly identify memory 69 as vehicle memory. In the proposed on-vehicle implementation, a PHOSITA would have stored the retrieved map in vehicle-accessible memory so that the map remains locally available for the recited matching operation after network communication becomes unavailable. This is a predictable implementation of Luo's disclosed memory and vehicle-resident processing.
wherein the fused map includes a previously-constructed point cloud basemap of the scenario
See at least:
“Further, in an embodiment, the global map is constructed, based on data from the LiDAR, using LiDAR mapping. The global map includes a 3D city-scale map.” (Luo [0069])
Rationale: (Express under the broadest reasonable interpretation.) Luo's global map is constructed from LiDAR data, and Luo confirms that the global map consists of 3D points by estimating the "distribution of 3D points within the voxels" ( [0081]). The global map is constructed in operation 13 (FIG. 1), before the feature matching of operation 15, so it is previously constructed. It is the reference map against which the vehicle is localized, and is therefore a previously-constructed point cloud basemap of the scenario. Its inclusion within the claimed fused map results from the combination with Flint, addressed below.
in the vehicle, capturing at least one image frame of a surrounding environment of the vehicle
See at least:
“The LiDAR sensor and the cameras are mounted on the roof of the vehicle.” (Luo [0058])
Rationale: (Express.) Luo's cameras are mounted on the vehicle and capture images of the environment around the vehicle ( [0068]). Capture therefore occurs in the vehicle, and each captured image is an image frame of the vehicle's surrounding environment.
within the scenario
See at least:
“In an embodiment, images of the environment are captured by the camera in approximately 30 Hz.” (Luo [0068])
Rationale: (Express.) Luo's image frames are captured of the environment in which the vehicle operates, which is the same environment the global map describes. The frames are therefore captured within the scenario.
through a camera unit
See at least:
“In an embodiment, the camera includes a monocular camera, which is calibrated by multiple shots instead of single shot.” (Luo [0065])
Rationale: (Express.) Luo's camera, whether monocular or stereo ( [0065]), is a camera unit through which the image frames are captured.
and extracting a plurality of feature points from the at least one image frame;
See at least:
“In an embodiment, the 3D submap is constructed, based on images from the camera, using visual SLAM. Reference of visual SLAM can be made to, for example, "ORB-SLAM: a versatile and accurate monocular SLAM system" by Raul Mur-Artal et al., IEEE Transactions on Robotics 31.5 (2015): 1147-1163” (Luo [0069])
Rationale: (PHOSITA-obvious, with Luo's identified ORB-SLAM technique as evidence.) Luo constructs its submap from camera image frames using visual SLAM, and specifically identifies ORB-SLAM as the visual SLAM technique. It was well known to a PHOSITA that visual SLAM, including the ORB-SLAM technique Luo identifies, operates by detecting and extracting keypoint features (feature points) from each captured image frame. The "sparse 3D points" of Luo's submap ( [0070]) are the 3D positions triangulated from those image feature points. Extracting a plurality of feature points from the at least one image frame was therefore the conventional operation of the visual SLAM technique Luo expressly selects. Luo additionally incorporates the cited ORB-SLAM publication by reference ( [0004], [0006]).
in the vehicle, performing a matching of the plurality of feature points with point cloud data in the point cloud basemap
See at least:
“Next, in operation 15, the features extracted from the 3D submap are matched against the features extracted from the global map so that in operation 16 invalid feature correspondences are removed.” (Luo [0071])
Rationale: (Express under the broadest reasonable interpretation.) Luo matches the camera-derived features, including sparse 3D feature points ( [0070]), against features extracted from the 3D point data of the LiDAR global map ( [0081]–[0082]). Matching camera feature points against features drawn from the point cloud is matching the feature points "with point cloud data in the point cloud basemap." The claim does not preclude intermediate processing such as triangulation. Luo's operations are executed by the computing device that is part of the vehicle ( [0055]), so the matching is performed in the vehicle.
to determine a position of the vehicle
See at least:
“Subsequently, in operation 17, location of the 3D submap is iteratively estimated until a distance between corresponding features is minimized.” (Luo [0072])
Rationale: (Express.) Luo's submap is built from the vehicle's own camera images and vehicle poses ( [0075]). Estimating the submap's location in the global map therefore determines the position of the vehicle, which is Luo's stated localization objective ( [0073]).
according to a result of the matching,
See at least:
“In operation 58, based on the position of the 3D submap in operation 51, an iterative estimation of location of the 3D submap is performed until distance between corresponding features reaches a predetermined value.” (Luo [0097])
Rationale: (Express.) Luo's position estimate is driven by the distances between the corresponding features established by the matching step, after invalid correspondences are removed ( [0096]). The position is therefore determined according to a result of the matching.
in the vehicle, measuring a relative displacement of the vehicle within the scenario
See at least:
“Vehicle poses, including position and orientation, are collected in an "east north up" (ENU) coordinate by the inertial navigation module in approximately 50 Hz.” (Luo [0068])
Rationale: (Implicit; alternatively PHOSITA-obvious.) Luo's vehicle-mounted inertial navigation module measures the vehicle's pose as the vehicle moves through the environment. A PHOSITA would have understood that an IMU senses accelerations and angular rates, from which the change in the vehicle's position and orientation relative to a prior state is derived (dead reckoning). Luo's inertial measurements therefore implicitly measure a relative displacement of the vehicle within the scenario. Alternatively, deriving relative displacement from IMU measurements would have been an obvious and conventional use of the IMU Luo discloses.
through an inertial measurement unit
See at least:
“The inertial navigation module in an embodiment according to the present disclosure includes a global navigation satellite system (GNSS)-inertial measurement unit (IMU).” (Luo [0059])
Rationale: (Express.) Luo expressly discloses that the inertial navigation module includes an IMU.
and updating the position of the vehicle
See at least:
“The benefits of using GNSS with an IMU are that the IMU may be calibrated by the GNSS signals and that the IMU can provide position and angle updates at a quicker rate than GNSS. For high dynamic vehicles, IMU fills in the gaps between GNSS positions.” (Luo [0059])
Rationale: (Express as to IMU position updates; PHOSITA-obvious as applied to the map-matched position.) Luo expressly discloses using the IMU to update the vehicle's position at a high rate between lower-rate absolute position fixes. Luo describes this with respect to GNSS fixes. It would have been obvious to apply the same complementary relationship to Luo's own absolute, map-matched position, for three reasons. First, Luo already uses the same inertial navigation module within its localization pipeline ( [0077]). Second, Luo's camera operates at a lower rate than its inertial module (approximately 30 Hz versus approximately 50 Hz, [0068]). Third, Luo teaches that the IMU and absolute sources "are complementary and are often employed together" ( [0059]). The result, IMU updates between visual fixes, is predictable.
according to the relative displacement,
See at least:
“Additionally, GNSS may lose its signal and the IMU can continue to compute the position and angle during the period of lost GNSS signal.” (Luo [0059])
Rationale: (Implicit; alternatively PHOSITA-obvious.) Luo teaches that the IMU continues to compute the position while the absolute fix is unavailable. A PHOSITA would have understood that continued position computation to involve propagating the prior position based on inertially measured motion, thereby updating the position according to the relative displacement. Alternatively, updating position in this manner would have been an obvious and conventional use of the IMU-based position computation Luo discloses.
Claim Limitations Not Explicitly Disclosed by Luo
Luo does not explicitly disclose the following claim limitations:
before a vehicle enters a scenario
and in which communication with a cloud network is prevented,
a previously-constructed fused map of the scenario,
and a previously-constructed vector map describing the scenario;
within the vector map
without transferring data to the cloud network; and
within the vector map
without transferring data to the cloud network.
Disclosure by Flint
Flint discloses or, in combination with Luo, renders obvious:
before a vehicle enters a scenario
See at least:
“In some implementations, the interior/exterior of a building (e.g., a warehouse, shopping mall or residential home) is traversed using an electronic computing device having a visual-based inertial navigation system according to the present disclosure to record a map of the building.” (Flint [0071])
Rationale: (PHOSITA-obvious from the combination.) Flint discloses visual-inertial navigation within building interiors that the platform enters and traverses, and teaches that cellular signals cannot reach such indoor environments ( [0003]). Luo teaches that data used by its localization programs is remotely stored and accessed over a network ( [0109]). A PHOSITA would have been motivated to retrieve and locally store the map before entering an area known to lack cellular availability, so that localization could continue without depending on network access inside that area. This is a predictable use of the known technique of pre-loading data before an anticipated loss of connectivity.
and in which communication with a cloud network is prevented,
See at least:
“Since visual-based inertial navigation systems do not require signals from GPS or cell towers, such systems may be used indoors where GPS and cell signals cannot reach or are unavailable due to interference.” (Flint [0003])
Rationale: (PHOSITA-obvious from the combination.) Flint expressly discloses indoor environments in which cell signals "cannot reach." Luo's remote network may be implemented over 3G (Luo [0102]: "Examples of wireless networks may include WiFi®, Bluetooth®, and 3G."). In the combined system, with Luo's remote map access implemented over its disclosed 3G embodiment, Flint's cellular-denied environment would prevent communication with the cloud network. This passage appears in Flint's background. It is cited because Flint's detailed description presupposes, but does not restate, the signal-denied environment for which the system is designed.
a previously-constructed fused map of the scenario,
See at least:
“The positions may be used to compute the path 10, which is stored in memory of the device and/or overlaid on an image of the floor plan, as shown in FIG. 1, and which can be output to a display for a user to view.” (Flint [0029])
Rationale: (PHOSITA-obvious from the combination.) Luo teaches the previously-constructed point cloud map component (Luo [0069]). Flint discloses a previously-existing geometric floor-plan map of the environment on which computed positions are overlaid. Luo expressly establishes vehicle position in the coordinate frame of its point cloud map, transforming the submap coordinate into "a coordinate of the global map" (Luo [0077]). Flint expresses computed positions on its floor-plan map (Flint [0069]). Because the combined system determines position in Luo's global-map coordinate frame and represents that position on Flint's floor-plan map, the two representations require a known spatial correspondence, i.e., registration to a common reference frame. A PHOSITA would have found it obvious to establish that correspondence as part of map preparation and to store and distribute the registered maps as a single fused map of the scenario, rather than maintaining them as unrelated, side-by-side maps. In the proposed combination, both map components would be made available before entry into the cellular-denied environment (Flint [0003]), so that localization and position representation can continue locally thereafter. Delivering them as a single, already-registered dataset is a predictable way to do so. Performing the registration once during map preparation would also avoid repeating the same registration operation on each vehicle and promote consistent map alignment across vehicles, which is a predictable implementation benefit. Flint further recognizes that on-device navigation processing "can be computationally burdensome, especially for devices ... that may have limited computation power" (Flint [0026]). That provides additional reason to perform non-time-critical preparatory processing before deployment rather than on the vehicle. In the fused map, Luo's point cloud map serves feature-matching localization and Flint's floor-plan map serves geometric representation and position display. Each component performs its known function, and the result is predictable. The fused map transferred to the vehicle is therefore previously constructed.
and a previously-constructed vector map describing the scenario;
See at least:
“The floor plan is an overview of a generic hardware store including the different sections (e.g., plumbing section, tools and hardware section, etc.), where the shaded regions represent shelves and the white regions represent the aisles and other areas of the store in which a customer may move.” (Flint [0029])
Rationale: (PHOSITA-obvious.) Flint discloses a floor-plan map that describes the geometry of the environment: its shelves, aisles, and traversable areas. The floor plan exists before navigation, since the computed path is overlaid onto it, so it is previously constructed. Flint does not state the storage format of the floor plan. Flint's floor plan is composed of geometric elements (the lines and rectangles bounding the shelves, aisles, and sections shown in FIG. 1). A PHOSITA would have recognized that such a geometric map may be stored either as a raster image or as the geometric elements themselves. Implementing it as vector graphic elements (lines, rectangles, polygons) would have been a predictable choice among a limited number of known representation formats, yielding the known benefits of a compact representation that can be scaled for display without loss.
within the vector map
See at least:
“The position and orientation estimates determined by the SWF module can be output to the display as points on a map (e.g., before or after post-processing) to provide a user of the electronic computing device visual information about the device's current and past positions.” (Flint [0069])
Rationale: (Express as to positions within the map; PHOSITA-obvious as applied to the combination.) Flint expressly expresses the visually determined position as points within the map describing the environment. In the combination, the position Luo determines by feature matching is therefore determined within the floor-plan (vector) map.
without transferring data to the cloud network; and
See at least:
“In some cases, the SWF module is configured to be a part of the electronic computing device that is being moved. In other implementations, the SWF module is separate from the electronic computing device, e.g., the SWF module may be operating on a server, whereas the IMU and image sensor are in the electronic computing device.” (Flint [0069])
Rationale: (PHOSITA-obvious design choice.) Flint expressly identifies two alternatives: performing the visual-based position determination on the moving device itself, or transferring data to a server for processing. Given those identified alternatives, it would have been obvious to select the on-device embodiment, a choice among a finite number of identified, predictable solutions (KSR). Read together with the preceding limitations, in the combined system the position determination is performed in a scenario in which communication with the cloud network is prevented, and the on-device embodiment is selected. The position determination therefore proceeds without transferring data to the cloud network.
within the vector map
See at least:
“For example, the post-processing module 116 may include a program that uses the model parameters output by the SWF module 108 to determine a path history of the device 100 and store the path history in device memory and/or overlay the path history on a map that is output to a display, such as shown in FIG. 1.” (Flint [0042])
Rationale: (Express as to updated positions within the map; PHOSITA-obvious as applied to the combination.) Flint discloses that each successive position update ("as each new set of model parameters is output," [0042]) is added to the path history and overlaid on the floor-plan map. In the combination, Luo's IMU-updated position is therefore updated within the vector map.
without transferring data to the cloud network.
See at least:
“Using features tracked in the images and corresponding information obtained from an IMU also operating on the device, the SWF obtains estimates for the inertial state of the device (e.g., position, orientation, velocity, and/or gyro and accelerometer biases), which may be referred to as simply the "state" of the device.” (Flint [0028])
Rationale: (PHOSITA-obvious design choice.) Flint discloses that the IMU operates on the device and that the inertial-state (position) estimate is obtained from it. Together with Flint's identified on-device processing alternative ( [0069]), it would have been obvious to perform the inertial position update locally. In the combined system, that update is performed in a scenario in which communication with the cloud network is prevented, using the on-device embodiment. The position update therefore proceeds without transferring data to the cloud network.
Motivation to Combine Luo and Flint
Therefore, given the teachings as a whole, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having Luo and Flint before them, to configure the instructions stored in Luo’s memory 69 and executed by Luo’s processor 61 to (i) retrieve Luo’s previously-constructed LiDAR point cloud global map from Luo’s remote network storage and store it in vehicle memory, associated with a vector floor-plan map of the environment as a fused map, before the vehicle enters an environment in which cellular communication with that remote network is unavailable; (ii) perform Luo’s feature matching and IMU-based position updating on the vehicle without transferring data to the cloud network; and (iii) express the determined and updated positions within the vector map, as taught by Flint.
Both references are in the same field of endeavor, localization of a moving platform using a camera and an inertial measurement unit (both classified in CPC G01C 21/165), and both address the same recognized problem of GPS unavailability (Luo [0003]; Flint [0002]).
Luo teaches that data used by its localization programs is stored remotely and accessed over a network ( [0109]), and Luo’s localization process expressly uses the global map as the data against which camera-derived features are matched ( [0071]). A PHOSITA would therefore have understood Luo’s remote-access path to be a natural source for delivering the global map to the vehicle. Luo further discloses that its wireless network may be 3G ( [0102]). Flint teaches that its system is used indoors "where GPS and cell signals cannot reach" ( [0003]), including building interiors such as warehouses ( [0071]). A PHOSITA would have been motivated to extend Luo’s high-precision, map-based localization to such environments to improve the reliability and availability of vehicle positioning. To do so, the PHOSITA would retrieve and locally store the map before entering an area known to lack cellular availability, so that localization could continue without depending on network access inside that area. Luo already provides both remote map access ( [0109]) and vehicle-resident memory and processing ( [0055], [0112]).
Flint expressly identifies on-device processing as an alternative to server-based processing ( [0069]). Selecting that alternative, and omitting network transfer during positioning, is a choice among a finite number of identified, predictable solutions. It is especially apt where cellular communication is unavailable.
Flint further teaches that expressing position on a map of the environment provides the user visual information about current and past positions ( [0069]), improving situational awareness and control. Because a position determined in Luo’s point cloud map (whose coordinate frame Luo establishes, [0077]) is to be displayed on the floor-plan map, the two maps must be registered to a common reference frame, which would have motivated integrating them into a single fused map in which the point cloud map serves localization and the floor-plan map serves geometric representation and display. A PHOSITA would have performed that registration as part of map preparation, before the fused map is stored remotely and transferred to the vehicle. In the proposed combination, both map components would be made available before entry so that positioning and position representation can continue locally. Registering once during map preparation also avoids repeating the same operation on each vehicle and promotes consistent alignment across vehicles. Implementing the method as processor-executed instructions stored in memory is the conventional and predictable way to realize such a method in a vehicle computing device, as Luo itself discloses ( [0104], [0112]).
The modification combines known elements according to known methods, with no change in their respective functions. It yields the predictable result of continuous, accurate vehicle positioning in communication-denied environments. KSR Int’l Co. v. Teleflex Inc., 550 U.S. 398, 416–417 (2007).
Claim Limitations Not Explicitly Disclosed by the Combination of All References
After combining the teachings of Luo and Flint, all limitations of Claim 9 are disclosed or rendered obvious.
Regarding Claim 17,
Disclosure by Luo
Luo discloses:
A non-transitory computer-readable storage medium storing instructions
See at least:
“In some embodiments in accordance with the present disclosure, a non-transitory, i.e., non-volatile, computer readable storage medium is provided. The non-transitory computer readable storage medium is stored with one or more programs.” (Luo [0055])
Rationale: (Express.) Luo expressly discloses a non-transitory computer-readable storage medium storing programs, i.e., instructions.
that cause a processor to perform a method for vehicle positioning, the method comprising:
See at least:
“When the program is executed by the processing unit of a computing device, i.e., that are part of a vehicle, the computing device is caused to conduct specific operations set forth below in accordance with some embodiments of the present disclosure.” (Luo [0055])
Rationale: (Express.) Luo's stored programs, when executed by the vehicle's processing unit, cause it to perform the localization operations. Those operations achieve centimeter-level localization of the vehicle ( [0073]), so they are a method for vehicle positioning.
where the vehicle is located
See at least:
“these sensors are used to collect data in an environment.” (Luo [0068])
Rationale: (Express.) Luo's vehicle-mounted sensors collect data in the environment in which the vehicle is present. That environment is the scenario where the vehicle is located.
uploading from the cloud network to the vehicle
See at least:
“The network interface 63 is configured to access program instructions and data accessed by the program instructions stored remotely through a network (not shown).” (Luo [0109])
Rationale: (Implicit / PHOSITA-obvious.) Luo expressly discloses that data accessed by the localization programs is stored remotely and accessed through a network by the vehicle-side network interface. Luo's remote storage is an Internet or virtual server reached over a wireless network ( [0102]–[0103]), which is a cloud network. Luo does not expressly identify the global map as that remotely stored data. However, Luo's localization process expressly uses the global map as the data against which the camera-derived features are matched ( [0071]). A person of ordinary skill in the art (PHOSITA) would have found remote delivery of the global map to the vehicle through Luo's disclosed network path to be a predictable implementation. Luo does not expressly describe distributing the map to multiple vehicles. However, centralized storage permits map data to be maintained at one location and distributed to vehicle clients as needed, which is a predictable implementation benefit rather than a teaching of Luo. That delivery is a transfer from the cloud network to the vehicle. The limitation is interpreted consistent with its expressly recited direction of transfer, from the cloud network to the vehicle.
and storing in a memory of the vehicle
See at least:
“The memory 69 is configured to store program instructions to be executed by the processor 61 and data accessed by the program instructions.” (Luo [0112])
Rationale: (PHOSITA-obvious.) Luo expressly discloses a memory configured to store data accessed by the localization programs, and separately discloses execution of the localization operations by a computing device that is part of a vehicle ( [0055]). Luo does not expressly identify memory 69 as vehicle memory. In the proposed on-vehicle implementation, a PHOSITA would have stored the retrieved map in vehicle-accessible memory so that the map remains locally available for the recited matching operation after network communication becomes unavailable. This is a predictable implementation of Luo's disclosed memory and vehicle-resident processing.
wherein the fused map includes a previously-constructed point cloud basemap of the scenario
See at least:
“Further, in an embodiment, the global map is constructed, based on data from the LiDAR, using LiDAR mapping. The global map includes a 3D city-scale map.” (Luo [0069])
Rationale: (Express under the broadest reasonable interpretation.) Luo's global map is constructed from LiDAR data, and Luo confirms that the global map consists of 3D points by estimating the "distribution of 3D points within the voxels" ( [0081]). The global map is constructed in operation 13 (FIG. 1), before the feature matching of operation 15, so it is previously constructed. It is the reference map against which the vehicle is localized, and is therefore a previously-constructed point cloud basemap of the scenario. Its inclusion within the claimed fused map results from the combination with Flint, addressed below.
in the vehicle, capturing at least one image frame of a surrounding environment of the vehicle
See at least:
“The LiDAR sensor and the cameras are mounted on the roof of the vehicle.” (Luo [0058])
Rationale: (Express.) Luo's cameras are mounted on the vehicle and capture images of the environment around the vehicle ( [0068]). Capture therefore occurs in the vehicle, and each captured image is an image frame of the vehicle's surrounding environment.
within the scenario
See at least:
“In an embodiment, images of the environment are captured by the camera in approximately 30 Hz.” (Luo [0068])
Rationale: (Express.) Luo's image frames are captured of the environment in which the vehicle operates, which is the same environment the global map describes. The frames are therefore captured within the scenario.
through a camera unit
See at least:
“In an embodiment, the camera includes a monocular camera, which is calibrated by multiple shots instead of single shot.” (Luo [0065])
Rationale: (Express.) Luo's camera, whether monocular or stereo ( [0065]), is a camera unit through which the image frames are captured.
and extracting a plurality of feature points from the at least one image frame;
See at least:
“In an embodiment, the 3D submap is constructed, based on images from the camera, using visual SLAM. Reference of visual SLAM can be made to, for example, "ORB-SLAM: a versatile and accurate monocular SLAM system" by Raul Mur-Artal et al., IEEE Transactions on Robotics 31.5 (2015): 1147-1163” (Luo [0069])
Rationale: (PHOSITA-obvious, with Luo's identified ORB-SLAM technique as evidence.) Luo constructs its submap from camera image frames using visual SLAM, and specifically identifies ORB-SLAM as the visual SLAM technique. It was well known to a PHOSITA that visual SLAM, including the ORB-SLAM technique Luo identifies, operates by detecting and extracting keypoint features (feature points) from each captured image frame. The "sparse 3D points" of Luo's submap ( [0070]) are the 3D positions triangulated from those image feature points. Extracting a plurality of feature points from the at least one image frame was therefore the conventional operation of the visual SLAM technique Luo expressly selects. Luo additionally incorporates the cited ORB-SLAM publication by reference ( [0004], [0006]).
in the vehicle, performing a matching of the plurality of feature points with point cloud data in the point cloud basemap
See at least:
“Next, in operation 15, the features extracted from the 3D submap are matched against the features extracted from the global map so that in operation 16 invalid feature correspondences are removed.” (Luo [0071])
Rationale: (Express under the broadest reasonable interpretation.) Luo matches the camera-derived features, including sparse 3D feature points ( [0070]), against features extracted from the 3D point data of the LiDAR global map ( [0081]–[0082]). Matching camera feature points against features drawn from the point cloud is matching the feature points "with point cloud data in the point cloud basemap." The claim does not preclude intermediate processing such as triangulation. Luo's operations are executed by the computing device that is part of the vehicle ( [0055]), so the matching is performed in the vehicle.
to determine a position of the vehicle
See at least:
“Subsequently, in operation 17, location of the 3D submap is iteratively estimated until a distance between corresponding features is minimized.” (Luo [0072])
Rationale: (Express.) Luo's submap is built from the vehicle's own camera images and vehicle poses ( [0075]). Estimating the submap's location in the global map therefore determines the position of the vehicle, which is Luo's stated localization objective ( [0073]).
according to a result of the matching,
See at least:
“In operation 58, based on the position of the 3D submap in operation 51, an iterative estimation of location of the 3D submap is performed until distance between corresponding features reaches a predetermined value.” (Luo [0097])
Rationale: (Express.) Luo's position estimate is driven by the distances between the corresponding features established by the matching step, after invalid correspondences are removed ( [0096]). The position is therefore determined according to a result of the matching.
in the vehicle, measuring a relative displacement of the vehicle within the scenario
See at least:
“Vehicle poses, including position and orientation, are collected in an "east north up" (ENU) coordinate by the inertial navigation module in approximately 50 Hz.” (Luo [0068])
Rationale: (Implicit; alternatively PHOSITA-obvious.) Luo's vehicle-mounted inertial navigation module measures the vehicle's pose as the vehicle moves through the environment. A PHOSITA would have understood that an IMU senses accelerations and angular rates, from which the change in the vehicle's position and orientation relative to a prior state is derived (dead reckoning). Luo's inertial measurements therefore implicitly measure a relative displacement of the vehicle within the scenario. Alternatively, deriving relative displacement from IMU measurements would have been an obvious and conventional use of the IMU Luo discloses.
through an inertial measurement unit
See at least:
“The inertial navigation module in an embodiment according to the present disclosure includes a global navigation satellite system (GNSS)-inertial measurement unit (IMU).” (Luo [0059])
Rationale: (Express.) Luo expressly discloses that the inertial navigation module includes an IMU.
and updating the position of the vehicle
See at least:
“The benefits of using GNSS with an IMU are that the IMU may be calibrated by the GNSS signals and that the IMU can provide position and angle updates at a quicker rate than GNSS. For high dynamic vehicles, IMU fills in the gaps between GNSS positions.” (Luo [0059])
Rationale: (Express as to IMU position updates; PHOSITA-obvious as applied to the map-matched position.) Luo expressly discloses using the IMU to update the vehicle's position at a high rate between lower-rate absolute position fixes. Luo describes this with respect to GNSS fixes. It would have been obvious to apply the same complementary relationship to Luo's own absolute, map-matched position, for three reasons. First, Luo already uses the same inertial navigation module within its localization pipeline ( [0077]). Second, Luo's camera operates at a lower rate than its inertial module (approximately 30 Hz versus approximately 50 Hz, [0068]). Third, Luo teaches that the IMU and absolute sources "are complementary and are often employed together" ( [0059]). The result, IMU updates between visual fixes, is predictable.
according to the relative displacement,
See at least:
“Additionally, GNSS may lose its signal and the IMU can continue to compute the position and angle during the period of lost GNSS signal.” (Luo [0059])
Rationale: (Implicit; alternatively PHOSITA-obvious.) Luo teaches that the IMU continues to compute the position while the absolute fix is unavailable. A PHOSITA would have understood that continued position computation to involve propagating the prior position based on inertially measured motion, thereby updating the position according to the relative displacement. Alternatively, updating position in this manner would have been an obvious and conventional use of the IMU-based position computation Luo discloses.
Claim Limitations Not Explicitly Disclosed by Luo
Luo does not explicitly disclose the following claim limitations:
before a vehicle enters a scenario
and in which communication with a cloud network is prevented,
a previously-constructed fused map of the scenario,
and a previously-constructed vector map describing the scenario;
within the vector map
without transferring data to the cloud network; and
within the vector map
without transferring data to the cloud network.
Disclosure by Flint
Flint discloses or, in combination with Luo, renders obvious:
before a vehicle enters a scenario
See at least:
“In some implementations, the interior/exterior of a building (e.g., a warehouse, shopping mall or residential home) is traversed using an electronic computing device having a visual-based inertial navigation system according to the present disclosure to record a map of the building.” (Flint [0071])
Rationale: (PHOSITA-obvious from the combination.) Flint discloses visual-inertial navigation within building interiors that the platform enters and traverses, and teaches that cellular signals cannot reach such indoor environments ( [0003]). Luo teaches that data used by its localization programs is remotely stored and accessed over a network ( [0109]). A PHOSITA would have been motivated to retrieve and locally store the map before entering an area known to lack cellular availability, so that localization could continue without depending on network access inside that area. This is a predictable use of the known technique of pre-loading data before an anticipated loss of connectivity.
and in which communication with a cloud network is prevented,
See at least:
“Since visual-based inertial navigation systems do not require signals from GPS or cell towers, such systems may be used indoors where GPS and cell signals cannot reach or are unavailable due to interference.” (Flint [0003])
Rationale: (PHOSITA-obvious from the combination.) Flint expressly discloses indoor environments in which cell signals "cannot reach." Luo's remote network may be implemented over 3G (Luo [0102]: "Examples of wireless networks may include WiFi®, Bluetooth®, and 3G."). In the combined system, with Luo's remote map access implemented over its disclosed 3G embodiment, Flint's cellular-denied environment would prevent communication with the cloud network. This passage appears in Flint's background. It is cited because Flint's detailed description presupposes, but does not restate, the signal-denied environment for which the system is designed.
a previously-constructed fused map of the scenario,
See at least:
“The positions may be used to compute the path 10, which is stored in memory of the device and/or overlaid on an image of the floor plan, as shown in FIG. 1, and which can be output to a display for a user to view.” (Flint [0029])
Rationale: (PHOSITA-obvious from the combination.) Luo teaches the previously-constructed point cloud map component (Luo [0069]). Flint discloses a previously-existing geometric floor-plan map of the environment on which computed positions are overlaid. Luo expressly establishes vehicle position in the coordinate frame of its point cloud map, transforming the submap coordinate into "a coordinate of the global map" (Luo [0077]). Flint expresses computed positions on its floor-plan map (Flint [0069]). Because the combined system determines position in Luo's global-map coordinate frame and represents that position on Flint's floor-plan map, the two representations require a known spatial correspondence, i.e., registration to a common reference frame. A PHOSITA would have found it obvious to establish that correspondence as part of map preparation and to store and distribute the registered maps as a single fused map of the scenario, rather than maintaining them as unrelated, side-by-side maps. In the proposed combination, both map components would be made available before entry into the cellular-denied environment (Flint [0003]), so that localization and position representation can continue locally thereafter. Delivering them as a single, already-registered dataset is a predictable way to do so. Performing the registration once during map preparation would also avoid repeating the same registration operation on each vehicle and promote consistent map alignment across vehicles, which is a predictable implementation benefit. Flint further recognizes that on-device navigation processing "can be computationally burdensome, especially for devices ... that may have limited computation power" (Flint [0026]). That provides additional reason to perform non-time-critical preparatory processing before deployment rather than on the vehicle. In the fused map, Luo's point cloud map serves feature-matching localization and Flint's floor-plan map serves geometric representation and position display. Each component performs its known function, and the result is predictable. The fused map transferred to the vehicle is therefore previously constructed.
and a previously-constructed vector map describing the scenario;
See at least:
“The floor plan is an overview of a generic hardware store including the different sections (e.g., plumbing section, tools and hardware section, etc.), where the shaded regions represent shelves and the white regions represent the aisles and other areas of the store in which a customer may move.” (Flint [0029])
Rationale: (PHOSITA-obvious.) Flint discloses a floor-plan map that describes the geometry of the environment: its shelves, aisles, and traversable areas. The floor plan exists before navigation, since the computed path is overlaid onto it, so it is previously constructed. Flint does not state the storage format of the floor plan. Flint's floor plan is composed of geometric elements (the lines and rectangles bounding the shelves, aisles, and sections shown in FIG. 1). A PHOSITA would have recognized that such a geometric map may be stored either as a raster image or as the geometric elements themselves. Implementing it as vector graphic elements (lines, rectangles, polygons) would have been a predictable choice among a limited number of known representation formats, yielding the known benefits of a compact representation that can be scaled for display without loss.
within the vector map
See at least:
“The position and orientation estimates determined by the SWF module can be output to the display as points on a map (e.g., before or after post-processing) to provide a user of the electronic computing device visual information about the device's current and past positions.” (Flint [0069])
Rationale: (Express as to positions within the map; PHOSITA-obvious as applied to the combination.) Flint expressly expresses the visually determined position as points within the map describing the environment. In the combination, the position Luo determines by feature matching is therefore determined within the floor-plan (vector) map.
without transferring data to the cloud network; and
See at least:
“In some cases, the SWF module is configured to be a part of the electronic computing device that is being moved. In other implementations, the SWF module is separate from the electronic computing device, e.g., the SWF module may be operating on a server, whereas the IMU and image sensor are in the electronic computing device.” (Flint [0069])
Rationale: (PHOSITA-obvious design choice.) Flint expressly identifies two alternatives: performing the visual-based position determination on the moving device itself, or transferring data to a server for processing. Given those identified alternatives, it would have been obvious to select the on-device embodiment, a choice among a finite number of identified, predictable solutions (KSR). Read together with the preceding limitations, in the combined system the position determination is performed in a scenario in which communication with the cloud network is prevented, and the on-device embodiment is selected. The position determination therefore proceeds without transferring data to the cloud network.
within the vector map
See at least:
“For example, the post-processing module 116 may include a program that uses the model parameters output by the SWF module 108 to determine a path history of the device 100 and store the path history in device memory and/or overlay the path history on a map that is output to a display, such as shown in FIG. 1.” (Flint [0042])
Rationale: (Express as to updated positions within the map; PHOSITA-obvious as applied to the combination.) Flint discloses that each successive position update ("as each new set of model parameters is output," [0042]) is added to the path history and overlaid on the floor-plan map. In the combination, Luo's IMU-updated position is therefore updated within the vector map.
without transferring data to the cloud network.
See at least:
“Using features tracked in the images and corresponding information obtained from an IMU also operating on the device, the SWF obtains estimates for the inertial state of the device (e.g., position, orientation, velocity, and/or gyro and accelerometer biases), which may be referred to as simply the "state" of the device.” (Flint [0028])
Rationale: (PHOSITA-obvious design choice.) Flint discloses that the IMU operates on the device and that the inertial-state (position) estimate is obtained from it. Together with Flint's identified on-device processing alternative ( [0069]), it would have been obvious to perform the inertial position update locally. In the combined system, that update is performed in a scenario in which communication with the cloud network is prevented, using the on-device embodiment. The position update therefore proceeds without transferring data to the cloud network.
Motivation to Combine Luo and Flint
Therefore, given the teachings as a whole, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having Luo and Flint before them, to store on Luo’s non-transitory computer-readable storage medium instructions causing the vehicle’s processor to (i) retrieve Luo’s previously-constructed LiDAR point cloud global map from Luo’s remote network storage and store it in vehicle memory, associated with a vector floor-plan map of the environment as a fused map, before the vehicle enters an environment in which cellular communication with that remote network is unavailable; (ii) perform Luo’s feature matching and IMU-based position updating on the vehicle without transferring data to the cloud network; and (iii) express the determined and updated positions within the vector map, as taught by Flint.
Both references are in the same field of endeavor, localization of a moving platform using a camera and an inertial measurement unit (both classified in CPC G01C 21/165), and both address the same recognized problem of GPS unavailability (Luo [0003]; Flint [0002]).
Luo teaches that data used by its localization programs is stored remotely and accessed over a network ( [0109]), and Luo’s localization process expressly uses the global map as the data against which camera-derived features are matched ( [0071]). A PHOSITA would therefore have understood Luo’s remote-access path to be a natural source for delivering the global map to the vehicle. Luo further discloses that its wireless network may be 3G ( [0102]). Flint teaches that its system is used indoors "where GPS and cell signals cannot reach" ( [0003]), including building interiors such as warehouses ( [0071]). A PHOSITA would have been motivated to extend Luo’s high-precision, map-based localization to such environments to improve the reliability and availability of vehicle positioning. To do so, the PHOSITA would retrieve and locally store the map before entering an area known to lack cellular availability, so that localization could continue without depending on network access inside that area. Luo already provides both remote map access ( [0109]) and vehicle-resident memory and processing ( [0055], [0112]).
Flint expressly identifies on-device processing as an alternative to server-based processing ( [0069]). Selecting that alternative, and omitting network transfer during positioning, is a choice among a finite number of identified, predictable solutions. It is especially apt where cellular communication is unavailable.
Flint further teaches that expressing position on a map of the environment provides the user visual information about current and past positions ( [0069]), improving situational awareness and control. Because a position determined in Luo’s point cloud map (whose coordinate frame Luo establishes, [0077]) is to be displayed on the floor-plan map, the two maps must be registered to a common reference frame, which would have motivated integrating them into a single fused map in which the point cloud map serves localization and the floor-plan map serves geometric representation and display. A PHOSITA would have performed that registration as part of map preparation, before the fused map is stored remotely and transferred to the vehicle. In the proposed combination, both map components would be made available before entry so that positioning and position representation can continue locally. Registering once during map preparation also avoids repeating the same operation on each vehicle and promotes consistent alignment across vehicles. Embodying the method as instructions on a non-transitory medium is the conventional and predictable way to distribute and execute such a method, as Luo itself discloses ( [0055]).
The modification combines known elements according to known methods, with no change in their respective functions. It yields the predictable result of continuous, accurate vehicle positioning in communication-denied environments. KSR Int’l Co. v. Teleflex Inc., 550 U.S. 398, 416–417 (2007).
Dependent Claims 2–8, 10–16, and 18–20
Claims 2, 10, and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Luo in view of Flint.
Claims 3, 4, 11, 12, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Luo in view of Flint, and further in view of Zhao et al.
Claims 5 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Luo in view of Flint, further in view of Zhao et al., and further in view of Zhou et al.
Claims 6, 14, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Luo in view of Flint, further in view of Zhao et al., and further in view of Farshidi et al.
Claims 7 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Luo in view of Flint, further in view of Zhao et al., Farshidi et al., Lin et al., and Liang et al.
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Luo in view of Flint, and further in view of Huang et al.
Claim 16 is rejected under 35 U.S.C. 103 as being unpatentable over Luo in view of Flint, further in view of Zhao et al., Farshidi et al., Lin et al., Liang et al., and Huang et al.
Zhao et al., Zhou et al., Farshidi et al., Lin et al., Liang et al., and Huang et al. are as cited in the Non-Final Office Action dated April 1, 2026.
Regarding Claim 2,
The combination of Luo and Flint establishes the method of Claim 1, which is the basis for Claim 2.
Luo teaches or renders obvious:
whereinSee at least: “The one or more programs comprise instructions, which when executed by a computing device, cause the computing device to perform...” ([0010]).Rationale: Luo discloses a method implemented through a series of executed operations. Thus, the recited “wherein” clause is properly understood as introducing additional conditions on how the previously recited localization method is carried out.
determining the position of the vehicle
See at least: “vehicle poses, including position and orientation, are collected in an ‘east north up’ (ENU) coordinate by the inertial navigation module in approximately 50 Hz” ([0068]) and “location of the 3D submap is iteratively estimated until a distance between corresponding features is minimized” ([0072]).
Rationale: Luo expressly discloses determining vehicle-related position/location within the localization workflow. In particular, Luo discloses estimation of the location of the 3D submap and collection of vehicle poses, thereby supporting the claimed position-determination aspect.
according to the result of the matching
See at least: “the features extracted from the 3D submap are matched against the features extracted from the global map” ([0071]) and “location of the 3D submap is iteratively estimated until a distance between corresponding features is minimized” ([0072]).Rationale: Luo expressly discloses that position determination follows from the feature-matching process. Thus, Luo teaches determining position according to the result of matching.
is executed at a first frequency,
See at least: “images of the environment are captured by the camera in approximately 30 Hz. LiDAR scans are collected in the form of a sweep in approximately 20 Hz” ([0068]).Rationale: Luo does not expressly use the phrase “first frequency” for the matching-based position determination. However, Luo expressly discloses the operative frequencies of the sensed data used in the matching/localization workflow, namely camera images at approximately 30 Hz and LiDAR scans at approximately 20 Hz. In view of these teachings, one of ordinary skill in the art would have understood the matching-based determination of position to be executed at a frequency associated with, and practically bounded by, the incoming image/LiDAR data used for the matching process. Accordingly, Luo at least renders obvious that such matching-based position determination is executed at a first operating frequency.
Luo teaches or renders obvious the matching-based position-determination portion of Claim 2, including that such determination is performed at an operating frequency associated with the image/LiDAR localization pipeline. The more specific comparative relationship between that frequency and a higher inertial-update frequency is more directly shown by Flint.
Flint teaches:
and updating the position of the vehicle
See at least: “The propagation algorithm takes a sequence of inertial readings between a first time and a second time together with the inertial state of the device at the first time and produces an estimate for the inertial state of the device at the second time” ([0036]) and “Using features tracked in the images and corresponding information obtained from an IMU also operating on the device, the SWF obtains estimates for the inertial state of the device (e.g., position, orientation, velocity...)” ([0028]).
Rationale: Flint expressly discloses updating device position/state through IMU-based propagation between successive times. Thus, Flint teaches the claimed updating of position.
according to the relative displacement
See at least: “During recording of the images and inertial data, the visual-based inertial navigation system calculates and stores estimates of the state of the computing device relative to a starting point, a process commonly referred to as ‘dead reckoning’” ([0029]) and “The IMU 104 may include ... a tri-axial gyroscope and accelerometer, for recording inertial data...” ([0030]).Rationale: Flint expressly discloses dead reckoning based on inertial measurements, which one of ordinary skill in the art would have understood to constitute updating position according to relative displacement.
is executed at a second frequency,
See at least: “the module 106 receives the inertial readings from the IMU 104 and identifies readings that occur at or close to the time at which each keyframe was captured ... and containing all the inertial readings occurring between those endpoints” ([0034]-[0035]) and “The propagation algorithm takes a sequence of inertial readings between a first time and a second time ... and produces an estimate for the inertial state of the device at the second time” ([0036]).Rationale: Flint expressly discloses repeated inertial-readings-based propagation between image keyframes. Thus, Flint teaches that the inertial-update process is executed at its own operating frequency driven by the sequence of IMU readings, i.e., a second frequency.
wherein the first frequency is lower than the second frequency.
See at least: “the pre-processing module 106 also is configured to select a subset of the frames received from the image sensor 102 as ‘keyframes’” ([0032]) and “the module 106 receives the inertial readings from the IMU 104 ... and containing all the inertial readings occurring between those endpoints.” ([0034]); see also “The propagation algorithm takes a sequence of inertial readings between a first time and a second time ... and produces an estimate...” ([0036]).See also Luo further: “images of the environment are captured by the camera in approximately 30 Hz. LiDAR scans are collected in the form of a sweep in approximately 20 Hz. Vehicle poses, including position and orientation, are collected ... by the inertial navigation module in approximately 50 Hz.” ([0068]).
Rationale: Flint expressly teaches that inertial readings are processed between image keyframes, thereby supporting a higher-rate inertial-update process relative to the lower-rate image/keyframe-based processing. Flint alone does not provide the specific numerical comparison. Luo, however, expressly discloses numerical rates showing camera images at approximately 30 Hz, LiDAR sweeps at approximately 20 Hz, and inertial-navigation-module vehicle poses at approximately 50 Hz. In view of these combined teachings, one of ordinary skill in the art would have understood that the matching-based position determination associated with the camera/LiDAR localization pipeline is executed at a lower frequency than the inertial relative-displacement update, which is executed at the higher IMU-driven frequency. Accordingly, the claimed relationship that the first frequency is lower than the second frequency is rendered obvious by the combination of Luo and Flint.
Motivation to Combine Luo and Flint
Therefore, given the teachings as a whole, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having Luo and Flint before them, to modify Luo’s map-matching-based vehicle-localization process so that the matching-based position determination is performed at the lower camera/LiDAR processing frequency while the relative-displacement-based position updating is performed at the higher inertial-measurement frequency taught by Flint, because Luo already discloses a multi-sensor localization architecture including camera, LiDAR, and inertial navigation inputs, and Flint teaches the known and compatible practice of propagating position/state using IMU readings between successive image/keyframe updates. One of ordinary skill in the art would have made such modification to improve responsiveness, continuity, and smoothness of vehicle localization between lower-rate map-matching determinations, with the predictable benefit of providing higher-rate pose updating while preserving the accuracy advantages of the lower-rate feature-matching correction. Such a modification would merely apply a known higher-rate inertial-update technique to a known lower-rate map-matching localization framework and would not alter the principle of operation of either reference or render either reference unsatisfactory for its intended purpose. Therefor, after combining the teachings of Luo and Flint, Claim 2 is rendered obvious by the combination of Luo in view of Flint. In particular, Luo teaches the matching-based position-determination portion of the claimed method and discloses the operative camera/LiDAR and inertial frequencies, while Flint teaches inertial-readings-based position propagation and updating between image/keyframe events. The claimed frequency relationship, namely that the first frequency is lower than the second frequency, would have been understood by one of ordinary skill in the art from Luo’s disclosed rates and Flint’s higher-rate IMU propagation framework.
Regarding Claim 10,
The combination of Luo and Flint establishes the device of Claim 9, which is the basis for Claim 10.
Luo teaches or renders obvious:
whereinSee at least: “The processor 61 is configured to execute program instructions that include a tool module configured to perform a method as described and illustrated with reference to FIGS. 1 to 5.” ([0104]).
Rationale: Luo expressly discloses a device whose processor-executed instructions perform the recited localization operations. Thus, the “wherein” clause properly introduces additional conditions on execution of the device-implemented method.
determining the position of the vehicle
See at least: “computing location of the 3D submap in the global map, using the inertial navigation module, and aligning the 3D submap with the global map.” ([0105]) and “Subsequently, in operation 17, location of the 3D submap is iteratively estimated until a distance between corresponding features is minimized.” ([0072]).
Rationale: Luo expressly discloses determining vehicle-related location within the localization process. In particular, Luo discloses computing location of the 3D submap in the global map and iteratively estimating that location, thereby teaching the claimed position-determination aspect.
according to the result of the matching
See at least: “the features extracted from the 3D submap are matched against the features extracted from the global map” ([0071]) and “location of the 3D submap is iteratively estimated until a distance between corresponding features is minimized.” ([0072]).
Rationale: Luo expressly discloses that location determination follows from the matching process between features extracted from the 3D submap and the global map. Thus, Luo teaches determining position according to the result of the matching.
is executed at a first frequency,
See at least: “images of the environment are captured by the camera in approximately 30 Hz. LiDAR scans are collected in the form of a sweep in approximately 20 Hz.” ([0068]).
Rationale: Luo does not expressly use the phrase “first frequency” for the matching-based position determination. However, Luo expressly discloses the operating rates of the sensor data used for the map-matching localization process, namely approximately 30 Hz for camera images and approximately 20 Hz for LiDAR sweeps. In view of these teachings, one of ordinary skill in the art would have understood the matching-based position determination to be executed at a frequency associated with, and practically bounded by, the image/LiDAR localization pipeline. Accordingly, Luo at least renders obvious that such matching-based position determination is executed at a first operating frequency.
Luo teaches or renders obvious the matching-based position-determination portion of Claim 10, including that such determination is performed at an operating frequency associated with the image/LiDAR localization workflow. The more specific higher-frequency inertial-update relationship is more directly shown by Flint, with Luo supplying the express numerical rates.
Flint teaches:
and updating the position of the vehicle
See at least: “Using features tracked in the images and corresponding information obtained from an IMU also operating on the device, the SWF obtains estimates for the inertial state of the device (e.g., position, orientation, velocity, and/or gyro and accelerometer biases)” ([0028]) and “The propagation algorithm takes a sequence of inertial readings between a first time and a second time together with the inertial state of the device at the first time and produces an estimate for the inertial state of the device at the second time.” ([0036]).
Rationale: Flint expressly discloses updating device position/state through IMU-based propagation between successive times. Thus, Flint teaches the claimed updating of position.
according to the relative displacement
See at least: “During recording of the images and inertial data, the visual-based inertial navigation system calculates and stores estimates of the state of the computing device relative to a starting point, a process commonly referred to as ‘dead reckoning.’” ([0029]) and “The portable electronic computing device also includes an IMU having components (e.g., accelerometer(s) and gyroscope(s)) that record inertial data such as linear acceleration and angular velocity of the device.” ([0029]).
Rationale: Flint expressly discloses dead reckoning based on inertial measurements, which one of ordinary skill in the art would have understood to constitute updating position according to relative displacement.
is executed at a second frequency,
See at least: “the pre-processing module 106 also is configured to select a subset of the frames received from the image sensor 102 as ‘keyframes’” ([0032]) and “the module 106 receives the inertial readings from the IMU 104 and identifies readings that occur at or close to the time at which each keyframe was captured ... and containing all the inertial readings occurring between those endpoints.” ([0034]).
See also: “The propagation algorithm takes a sequence of inertial readings between a first time and a second time together with the inertial state of the device at the first time and produces an estimate for the inertial state of the device at the second time.” ([0036]).
Rationale: Flint expressly discloses repeated IMU-readings-based propagation between image keyframes. Thus, Flint teaches that the inertial-update process is executed at its own operating frequency driven by the sequence of inertial readings, i.e., a second frequency.
wherein the first frequency is lower than the second frequency.
See at least: “the pre-processing module 106 also is configured to select a subset of the frames received from the image sensor 102 as ‘keyframes’” ([0032]) and “the module 106 receives the inertial readings from the IMU 104 ... and containing all the inertial readings occurring between those endpoints.” ([0034]); see also “The propagation algorithm takes a sequence of inertial readings between a first time and a second time ... and produces an estimate...” ([0036]).See also Luo further: “images of the environment are captured by the camera in approximately 30 Hz. LiDAR scans are collected in the form of a sweep in approximately 20 Hz. Vehicle poses, including position and orientation, are collected ... by the inertial navigation module in approximately 50 Hz.” ([0068]).
Rationale: Flint expressly teaches that inertial readings are processed between image keyframes, thereby supporting a higher-rate inertial-update process relative to the lower-rate image/keyframe-based processing. Flint alone does not provide the specific numerical comparison. Luo, however, expressly discloses numerical rates showing camera images at approximately 30 Hz, LiDAR sweeps at approximately 20 Hz, and inertial-navigation-module vehicle poses at approximately 50 Hz. In view of these combined teachings, one of ordinary skill in the art would have understood that the matching-based position determination associated with the camera/LiDAR localization pipeline is executed at a lower frequency than the inertial relative-displacement update, which is executed at the higher IMU-driven frequency. Accordingly, the claimed relationship that the first frequency is lower than the second frequency is rendered obvious by the combination of Luo and Flint.
Motivation to Combine Luo and Flint
Therefore, given the teachings as a whole, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having Luo and Flint before them, to modify Luo’s device-implemented map-matching-based vehicle-positioning process so that the matching-based position determination is performed at the lower camera/LiDAR processing frequency while the relative-displacement-based position updating is performed at the higher inertial-measurement frequency taught by Flint, because Luo already discloses a multi-sensor localization architecture including camera, LiDAR, and inertial-navigation inputs, and Flint teaches the known and compatible practice of propagating position/state using IMU readings between successive image/keyframe updates. One of ordinary skill in the art would have made such modification to improve responsiveness, continuity, and smoothness of vehicle positioning between lower-rate map-matching determinations, with the predictable benefit of providing higher-rate pose updating while preserving the accuracy advantages of the lower-rate feature-matching correction. Such a modification would merely apply a known higher-rate inertial-update technique to a known lower-rate map-matching localization framework and would not alter the principle of operation of either reference or render either reference unsatisfactory for its intended purpose. Therefore, after combining the teachings of Luo and Flint, Claim 10 is rendered obvious by the combination of Luo in view of Flint. In particular, Luo teaches the device framework and the matching-based position-determination portion of the claimed positioning method, including the operative camera/LiDAR frequencies, while Flint teaches inertial-readings-based position propagation and updating between image/keyframe events. The claimed frequency relationship, namely that the first frequency is lower than the second frequency, is supported by Luo’s express numerical rates and Flint’s teaching of higher-rate IMU propagation between image/keyframe updates.
Regarding Claim 18,
The combination of Luo and Flint establishes the non-transitory computer-readable storage medium of Claim 17, which is the basis for Claim 18.
Luo renders obvious:
whereinSee at least: “The one or more programs comprise instructions, which when executed by a computing device, cause the computing device to perform...” ([0010]) and “When the program is executed by the processing unit of a computing device ... the computing device is caused to conduct specific operations...” ([0055]).Rationale: The reference expressly discloses that the stored instructions, when executed, cause the processor/computing device to carry out the recited localization operations. Thus, the “wherein” clause properly introduces additional execution conditions on that storage-medium-implemented method.
determining the position of the vehicleSee at least: “computing, in response to features from a 3D submap and features from a global map, matching score...” ([0010]) and “In yet still another embodiment, the method further comprises: refining location of the 3D submap.” ([0021]); see also “refining location of the 3D submap comprises: performing an iterative estimation of location of the 3D submap...” ([0022]).Rationale: The reference expressly discloses determining vehicle-related location within the localization process through matching and subsequent refinement of the 3D submap location. One of ordinary skill in the art would have understood such refined localization to constitute determining vehicle position.
according to the result of the matching
See at least: “computing, in response to features from a 3D submap and features from a global map, matching score...” ([0010]) and “selecting, for each feature in the 3D submap, a corresponding feature with the highest matching score from the global map...” ([0010]).Rationale: The reference expressly discloses that the localization process relies on matching features from the 3D submap and the global map. Thus, position determination according to the result of the matching is taught.
is executed at a first frequency,
See at least: “images of the environment are captured by the camera in approximately 30 Hz. LiDAR scans are collected in the form of a sweep in approximately 20 Hz.” ([0068]).Rationale: Although Luo does not expressly recite a “first frequency” for the matching-based position determination, Luo expressly discloses the operating rates of the camera and LiDAR data used in the map-matching localization pipeline. In view of these teachings, one of ordinary skill in the art would have understood the matching-based position determination to be executed at a frequency associated with, and practically bounded by, the image/LiDAR localization process. Accordingly, this first-frequency aspect is at least rendered obvious by Luo.
Luo teaches or renders obvious the matching-based position-determination portion of Claim 18, including that such determination is performed at an operating frequency associated with the image/LiDAR localization workflow. The more specific higher-frequency inertial-update relationship is more directly shown by Flint, with Luo supplying the express numerical rates.
Flint discloses:
and updating the position of the vehicle
See at least: “Using features tracked in the images and corresponding information obtained from an IMU also operating on the device, the SWF obtains estimates for the inertial state of the device (e.g., position, orientation, velocity...)” ([0028]) and “The propagation algorithm takes a sequence of inertial readings between a first time and a second time together with the inertial state of the device at the first time and produces an estimate for the inertial state of the device at the second time.” ([0036]).
Rationale: The reference expressly discloses updating device position/state through IMU-based propagation between successive times. Thus, Flint teaches the claimed updating of position.
according to the relative displacement
See at least: “During recording of the images and inertial data, the visual-based inertial navigation system calculates and stores estimates of the state of the computing device relative to a starting point, a process commonly referred to as ‘dead reckoning.’” ([0029]) and “the portable electronic computing device also includes an IMU having components ... that record inertial data such as linear acceleration and angular velocity...” ([0029]).
Rationale: The reference expressly discloses dead reckoning based on inertial measurements, which one of ordinary skill in the art would have understood to constitute updating position according to relative displacement.
is executed at a second frequency,
See at least: “the pre-processing module 106 also is configured to select a subset of the frames received from the image sensor 102 as ‘keyframes’” ([0032]) and “the module 106 receives the inertial readings from the IMU 104 ... and containing all the inertial readings occurring between those endpoints.” ([0034]); see also “The propagation algorithm takes a sequence of inertial readings between a first time and a second time ... and produces an estimate...” ([0036]).Rationale: The reference expressly discloses repeated IMU-readings-based propagation between image keyframes. Thus, Flint teaches that the inertial-update process is executed at its own operating frequency driven by the sequence of inertial readings, i.e., a second frequency.
wherein the first frequency is lower than the second frequency.See at least: “the pre-processing module 106 also is configured to select a subset of the frames received from the image sensor 102 as ‘keyframes’” ([0032]) and “the module 106 receives the inertial readings from the IMU 104 ... and containing all the inertial readings occurring between those endpoints.” ([0034]); see also “The propagation algorithm takes a sequence of inertial readings between a first time and a second time ... and produces an estimate...” ([0036]).See also Luo: “images of the environment are captured by the camera in approximately 30 Hz. LiDAR scans are collected in the form of a sweep in approximately 20 Hz. Vehicle poses, including position and orientation, are collected ... by the inertial navigation module in approximately 50 Hz.” ([0068]).
Rationale: Flint expressly teaches that inertial readings are processed between image keyframes, thereby supporting a higher-rate inertial-update process relative to the lower-rate image/keyframe-based processing. Flint alone does not provide the specific numerical comparison. Luo, however, expressly discloses numerical rates showing camera images at approximately 30 Hz, LiDAR sweeps at approximately 20 Hz, and inertial-navigation-module vehicle poses at approximately 50 Hz. In view of these combined teachings, one of ordinary skill in the art would have understood that the matching-based position determination associated with the camera/LiDAR localization pipeline is executed at a lower frequency than the inertial relative-displacement update, which is executed at the higher IMU-driven frequency. Accordingly, the claimed relationship that the first frequency is lower than the second frequency is rendered obvious by the combination of Luo and Flint.
Motivation to Combine Luo and Flint
Therefore, given the teachings as a whole, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having Luo and Flint before them, to modify Luo’s storage-medium-implemented map-matching-based vehicle-positioning process so that the matching-based position determination is performed at the lower camera/LiDAR processing frequency while the relative-displacement-based position updating is performed at the higher inertial-measurement frequency taught by Flint, because Luo already discloses a multi-sensor localization architecture including camera, LiDAR, and inertial-navigation inputs, and Flint teaches the known and compatible practice of propagating position/state using IMU readings between successive image/keyframe updates. One of ordinary skill in the art would have made such modification to improve responsiveness, continuity, and smoothness of vehicle positioning between lower-rate map-matching determinations, with the predictable benefit of providing higher-rate pose updating while preserving the accuracy advantages of the lower-rate feature-matching correction. Such a modification would merely apply a known higher-rate inertial-update technique to a known lower-rate map-matching localization framework and would not alter the principle of operation of either reference or render either reference unsatisfactory for its intended purpose. Therefore, after combining the teachings of Luo and Flint, Claim 18 is rendered obvious by the combination of Luo in view of Flint. In particular, Luo teaches the non-transitory storage-medium framework and the matching-based position-determination portion of the claimed positioning method, including the operative camera/LiDAR frequencies, while Flint teaches inertial-readings-based position propagation and updating between image/keyframe events. The claimed frequency relationship, namely that the first frequency is lower than the second frequency, is supported by Luo’s express numerical rates and Flint’s teaching of higher-rate IMU propagation between image/keyframe updates.
Claims 3, 4, 11, 12, and 19 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Luo, in view of Flint, and in view of Zhao (Automatic Vector-Based Road Structure Mapping Using Multibeam LiDAR).
Regarding Claim 3,
The combination of Luo, Flint, and Zhao establishes the method of Claim 1, which is the basis for Claim 3.
Luo teaches or renders obvious:
whereinSee at least: “The one or more programs comprise instructions, which when executed by a computing device, cause the computing device to perform...” ([0010]).Rationale: The reference expressly discloses a processor-executed localization method. Thus, the “wherein” clause properly introduces additional conditions on the previously established method.
the point cloud basemap includes point cloud data
See at least: “constructing a global map comprises: obtaining the data from the LiDAR; and constructing a city-scale 3D map based on the data from the LiDAR, using LiDAR mapping.” ([0014]) and “the unstructured features may include sparse 3D points.” ([0017]); see also “The global map includes a 3D city-scale map.” ([0076]).Rationale: Luo expressly discloses a LiDAR-derived 3D global map and sparse 3D point features. One of ordinary skill in the art would have understood such LiDAR-derived 3D map data to correspond to point-cloud-type map data. Accordingly, Luo teaches or at least renders obvious the point-cloud-data aspect of the claimed point cloud basemap.
describing measurement of objects within the scenario
See at least: “obtaining the data from the LiDAR; and constructing a city-scale 3D map based on the data from the LiDAR” ([0014]) and “the unstructured features may include sparse 3D points.” ([0017]).Rationale: Luo expressly discloses LiDAR-acquired 3D map data and 3D point-based features representing measured environmental content. Thus, Luo teaches that the point-cloud-type map data describes measured objects/features in the mapped scene, although Luo does not expressly tie those objects to roads and intersections specifically.
and the vector map includes vector graphic elements
See at least: “the structured features include at least one of planes, straight lines and curved lines...” ([0017]) and “The extracted features include structured features such as planes, straight lines and curved lines...” ([0082]).
Rationale: Luo does not expressly use the phrase “vector graphic elements.” However, Luo expressly discloses structured geometric features such as planes, straight lines, and curved lines. One of ordinary skill in the art would have recognized such structured geometric descriptors as vector-like elements of a map representation. Accordingly, Luo at least renders obvious the vector-element aspect of the claim.
Luo renders obvious the point-cloud-data aspect of Claim 3 and further provides a strong basis for treating structured geometric map features as vector-like elements. The road-specific, intersection-specific, subset-generation, and explicit mapping/construction aspects are more directly shown by Zhao.
Zhao teaches:
describing measurement of objects within the scenario
See at least: “The LiDAR-based SLAM generates accurate maps represented by occupancy grids or 3D point clouds...” (page 2) and “the 3D point cloud of a scene” and “the 2D projection” and “the road boundaries extracted in one frame” and “the multiframe probabilistic fusion” (page 5, Figure 2 and accompanying text).
Rationale: Zhao expressly discloses LiDAR-based mapping from 3D point clouds of a scene and extraction of road-boundary objects from those measurements. Thus, Zhao teaches point-cloud data describing measured objects within the mapped scenario.
at roads and intersections within the scenario,
See at least: “The high-precision high-definition (HD) map of the road environment is now recognized as one of the cornerstones for autonomous driving. The reliable mapping of road boundaries, lanes, and other road structures...” (page 2) and “Figure 11c-I, II, and VI show successfully mapped intersections; all the road boundaries along the trajectory are mapped...” (page 13).
Rationale: Zhao expressly discloses that its mapping concerns road structures in road environments and further expressly identifies mapped intersections. Accordingly, Zhao teaches the road- and intersection-specific context missing from Luo.
and the vector map includes vector graphic elements
See at least: “we propose a vector-based SLAM method for the road structure mapping using vehicle-mounted multibeam LiDAR. We propose using polylines as the primary mapping element instead of grid maps or point clouds because the vector-based representation is lightweight and precise.” (page 2) and “Vector primitives, for example, polygons and line segments, are widely adopted as the representation of road geometry...” (page 3).
Rationale: Zhao expressly discloses vector-based mapping and expressly identifies polylines, polygons, and line segments as vector-based mapping elements. Thus, Zhao teaches the claimed vector graphic elements.
describing geometric characteristics of roads and intersections within the scenario,See at least: “The high-precision high-definition (HD) map of the road environment ... The reliable mapping of road boundaries, lanes, and other road structures...” (page 2) and “Vector primitives ... are widely adopted as the representation of road geometry...” (page 3).Rationale: Zhao expressly discloses vector-based representation of road geometry and road structures, and further shows mapped intersections in the resulting vectorized map discussion. One of ordinary skill in the art would have understood such vector elements to describe geometric characteristics of roads and intersections within the scenario.
and wherein,
See at least: “We explored a combined pipeline of road structure extraction and vectorization, vector-based local map matching, optimization and vector reconstruction.” (page 2)Rationale: Zhao expressly discloses an ordered mapping pipeline in which additional map-construction operations are performed. Thus, the clause properly introduces further map-preconstruction steps.
the method further comprises pre-constructing the fused map by:
See at least: “We explored a combined pipeline of road structure extraction and vectorization, vector-based local map matching, optimization and vector reconstruction.” (page 2) and “First, the LVMs are fused without concatenation... Finally, a reconstruction method is employed to globally reconstruct the vector map...” (page 9).
Rationale: Zhao expressly discloses a mapping workflow in which road-structure data is extracted, vectorized, fused, and reconstructed into a vector map. One of ordinary skill in the art would have understood this to be a pre-construction process for creating the map used later for localization.
creating corresponding point cloud data subsets
See at least: “the virtual scan method ... is applied to eliminate obstacles unrelated to the road boundary...” (page 5) and “the sparse extraction results can be enhanced which generate the local grid map (LGM) of road boundaries” (page 6). and “The fused LVMs ... To filter out the ghost effects, we utilize the trajectories of the vehicle and employ ‘probing’ of the innermost road boundaries along the trajectory.” (page 9).
Rationale: Zhao expressly discloses extracting road-boundary-related subsets from larger point-cloud measurements and then operating on those localized/fused map portions. Although Zhao does not use the exact phrase “point cloud data subsets,” one of ordinary skill in the art would have understood the extracted road-boundary portions and local maps derived from the point-cloud scene data to constitute corresponding point-cloud-data subsets for later map generation.
respectively for each road and each intersection within the scenarioSee at least: “The data collecting path is approximately 2 km in length, passing through all road segments, including 3 cross-intersections and 8 T-junctions...” (page 13) and “Figure 11c-I, II, and VI show successfully mapped intersections; all the road boundaries along the trajectory are mapped...” (page 13).
Rationale: Zhao expressly discloses mapping across multiple road segments and intersections, including cross-intersections and T-junctions. Thus, Zhao teaches road- and intersection-specific map generation, which one of ordinary skill in the art would have understood as requiring corresponding subsets or local portions associated with those respective road/intersection regions.
to generate the point cloud data; and
See at least: “To fuse multiframes locally with high precision, we develop a reckoning system...” (page 5) and “As a result, the sparse extraction results can be enhanced which generate the local grid map (LGM) of road boundaries” (page 6).
Rationale: Zhao expressly discloses generating locally fused map data from multiple LiDAR-derived frames. Thus, Zhao teaches generating point-cloud-derived map data from extracted/fused subsets of the measured scene.
mapping the point cloud data with the vector graphic elementsSee at least: “The LGM generated in the previous step is still a grid map-based representation of road boundaries. To convert the grid map into a vector-based map...” (page 6) and “we employ the virtual scan method to locally vectorize the LGM” (page 6); see also “In Figure 9e, the ordered points belonging to a sector of the road boundaries are connected to realize the initial vectorization.” (page 11).
Rationale: Zhao expressly discloses converting LiDAR/point-cloud-derived road-boundary map data into vector-based mapping elements through vectorization. In other words, Zhao teaches mapping the point-cloud-derived road data with vector graphic elements.
to construct the fused map.
See at least: “The optimization and reconstruction of local vector-based maps into a global vector map of the road structure is achieved.” (page 3) and “Finally, a reconstruction method is employed to globally reconstruct the vector map from sampled LVMs.” (page 9).Rationale: Zhao expressly discloses constructing a final global vector map by fusing, reconstructing, and vectorizing the road-structure data. In view of Luo’s point-cloud/global-map framework and Zhao’s vector-road-map construction workflow, one of ordinary skill in the art would have found it obvious to construct the claimed fused map by associating point-cloud-derived road/intersection data with vector graphic elements.
Motivation to Combine Luo, Flint, and Zhao
Therefore, given the teachings as a whole, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having Luo, Flint, and Zhao before them, to modify Luo’s camera/LiDAR map-based vehicle-localization method, as further informed by Flint’s IMU-based relative-displacement updating used for the parent Claim 1 combination, so that the fused map used in localization includes road- and intersection-specific point-cloud data subsets and corresponding vector graphic road-geometry elements constructed through a vectorization and reconstruction workflow as taught by Zhao, because Luo already discloses a LiDAR-derived global map, a camera-derived submap, and structured/unstructured map features for localization, Flint teaches a known and compatible inertial-update technique for maintaining vehicle position between map-matching determinations, and Zhao teaches a known and compatible technique for generating vector-based road-structure maps from LiDAR-derived road-boundary data, including road and intersection mapping, local subset generation, vectorization, and reconstruction. One of ordinary skill in the art would have made such modification to provide a more lightweight, road-geometry-aware, and structured fused map representation for localization while preserving the continuous vehicle-position updating already supported by the Luo/Flint combination, with the predictable benefit of improving map usability and localization performance in road/intersection environments. Such a modification would merely apply a known vector-road-map generation technique to a known LiDAR-based localization framework, while retaining known inertial-update processing, and would not alter the principle of operation of any reference or render any reference unsatisfactory for its intended purpose. Therefore, after combining the teachings of Luo, Flint, and Zhao, Claim 3 is rendered obvious by the combination of Luo in view of Flint and Zhao. In particular, Luo teaches the underlying camera/LiDAR localization framework and point-cloud/global-map features, Flint teaches the IMU-based updating portion carried through from parent Claim 1, and Zhao teaches road- and intersection-specific vector-map construction from LiDAR-derived road-structure data, including point-cloud-derived subsets, vector graphic elements, and reconstruction of the fused/vector map. The additional map-content and pre-construction aspects of Claim 3 would have been obvious to one of ordinary skill in the art in view of Zhao’s disclosed vector-road-mapping architecture and its predictable integration with the Luo/Flint localization framework.
Regarding Claim 4,
The combination of Luo, Flint, and Zhao establishes the method of Claim 3, which is the basis for Claim 4.
Zhao renders obvious:
whereinSee at least: “We explored a combined pipeline of road structure extraction and vectorization, vector-based local map matching, optimization and vector reconstruction.” (page 2)Rationale: Zhao expressly discloses an ordered mapping pipeline with additional processing stages. Thus, the “wherein” clause properly introduces additional conditions on the subset-generation portion of the already established method.
creating corresponding point cloud data subsetsSee at least: “Figure 2. Road boundary extraction. (a) the 3D point cloud of a scene ... (d) the road boundaries extracted in one frame. (e) the multiframe probabilistic fusion.” (page 5) and “the sparse extraction results can be enhanced which generate the local grid map (LGM) of road boundaries” (page 6)Rationale: Zhao expressly discloses beginning with scene point-cloud data, extracting road-boundary portions therefrom, and then generating local map representations from those extracted portions. One of ordinary skill in the art would have understood those extracted road-boundary portions and local maps to constitute corresponding subsets of the original point-cloud data for subsequent processing.
respectively for each road and each intersection within the scenarioSee at least: “The data collecting path is approximately 2 km in length, passing through all road segments, including 3 cross-intersections and 8 T-junctions...” (page 13) and “Figure 11c-I, II, and VI show successfully mapped intersections; all the road boundaries along the trajectory are mapped...” (page 13)Rationale: Zhao expressly discloses mapping multiple road segments and multiple intersections, including cross-intersections and T-junctions. Accordingly, Zhao teaches road- and intersection-specific local mapping, which one of ordinary skill in the art would have understood as involving corresponding subsets or local portions associated with the respective road and intersection regions.
comprises:See at least: “We explored a combined pipeline of road structure extraction and vectorization...” (page 2)Rationale: Zhao expressly discloses an open-ended sequence of mapping operations, thereby supporting the recited transitional phrase for the additional subset-processing steps.
obtaining dense point cloud data subsetsSee at least: “However, the extracted road boundary in one frame is sparse and can be incomplete; therefore, it should be further densified by the multiframe probabilistic fusion.” (page 5) and “To fuse multiframes locally with high precision...” (page 5)Rationale: Zhao expressly discloses that sparse single-frame road-boundary extraction is further densified by multiframe probabilistic fusion. One of ordinary skill in the art would have understood the densified local fused boundary representation to constitute dense point-cloud-derived subsets for the corresponding road/intersection regions.
for each road and each intersection within the scenario,See at least: “all road segments, including 3 cross-intersections and 8 T-junctions...” (page 13) and “all the road boundaries along the trajectory are mapped” and “successfully mapped intersections” (page 13)Rationale: Zhao expressly discloses dense local road-boundary mapping across multiple road segments and intersections. Thus, the densified subset generation is taught in the road- and intersection-specific context required by the claim.
and determining whether a first total amount of dataSee at least: “Since the vectorized result can be noisy and dense due to the selected angular resolution of the virtual scan...” (page 7) and “The simplified representation can be reduced to merely averaging 6% of the points in the original polyline...” (page 7)Rationale: Zhao does not expressly recite computing a “first total amount of data” in those words. However, Zhao expressly distinguishes between denser and lighter-weight representations and expressly quantifies that simplification reduces the retained representation to about 6% of the original points. In view of these teachings, one of ordinary skill in the art would have understood that the mapping pipeline evaluates whether the amount of point/polyline data is sufficiently large to warrant simplification or sparsification. Accordingly, this data-amount determination is at least implicit in Zhao, or would have been obvious therefrom.
for respective dense point cloud data subsetsSee at least: “the extracted road boundary in one frame is sparse and can be incomplete; therefore, it should be further densified by the multiframe probabilistic fusion.” (page 5) and “the fused local grid map of road boundaries (LGM)” (page 11 figure description)Rationale: Zhao expressly discloses locally fused, densified road-boundary representations derived from multiframe point-cloud data. One of ordinary skill in the art would have understood these fused local representations to be the relevant dense subsets whose amount of data may be evaluated for subsequent simplification.
exceeds a predetermined threshold,See at least: “Since the vectorized result can be noisy and dense due to the selected angular resolution of the virtual scan, we employ the Ramer-Douglas-Peucker algorithm [41] to optimize polylines in the LVM.” (page 7) and “The comparison of the error statistics ... the matching with simplified-LVMs outperforms raw-LVM-based matching ... because of its lightweight and less noisy polyline-based representation.” (page 11)Rationale: Zhao does not expressly state a “predetermined threshold” for a first total amount of data. However, Zhao expressly teaches that when the representation is noisy and dense, a simplification operation is applied to reduce the amount of retained map data. One of ordinary skill in the art would have understood that, in implementing such density-driven simplification, some threshold or criterion must be used to decide when the dense representation should be retained and when it should be simplified. Use of a predetermined threshold for that decision would have been a routine and predictable implementation choice.
wherein when the first total amount of data does not exceed the predetermined threshold,See at least: “For a clear description, we refer to the simplified version of LVM as simplified LVM and the original version as raw LVM.” (page 7)Rationale: Zhao expressly distinguishes between an original, denser representation and a simplified representation. Although Zhao does not expressly recite the decision branch in the exact words of the claim, one of ordinary skill in the art would have understood that if the amount of data is not excessive, the original denser representation may simply be retained rather than simplified. This branch is therefore rendered obvious by Zhao’s express raw-versus-simplified framework.
taking the respective dense point cloud data subsets as the point cloud data,See at least: “the 3D point cloud of a scene” and “the road boundaries extracted in one frame” and “the multiframe probabilistic fusion” (page 5)Rationale: Zhao expressly discloses use of the extracted and fused road-boundary data as the operative point-cloud-derived map content. Where simplification is not invoked, one of ordinary skill in the art would have understood the dense subsets themselves to serve as the point-cloud data used downstream.
and when the first total amount of data exceeds the predetermined threshold,See at least: “Since the vectorized result can be noisy and dense due to the selected angular resolution of the virtual scan, we employ the Ramer-Douglas-Peucker algorithm [41] to optimize polylines in the LVM.” (page 7)Rationale: Zhao expressly teaches that when the representation is dense and noisy, simplification/optimization is applied. Although Zhao does not expressly frame this in terms of a threshold comparison, the conditional relationship between excessive density and subsequent simplification is clearly taught, and the use of a predetermined threshold to trigger that simplification would have been an obvious implementation detail.
converting the respective dense point cloud data subsetsSee at least: “The LGM generated in the previous step is still a grid map-based representation of road boundaries. To convert the grid map into a vector-based map...” (page 6) and “we employ the virtual scan method to locally vectorize the LGM.” (page 6)Rationale: Zhao expressly discloses converting denser local road-boundary representations into a more compact representation. Although the claim specifically recites converting dense point-cloud subsets into sparse point-cloud subsets, Zhao’s express teaching of transforming dense local road-boundary data into a lighter-weight representation strongly supports the claimed conversion concept, at least as rendered obvious to one of ordinary skill in the art.
into sparse point cloud data subsetsSee at least: “the extracted road boundary in one frame is sparse and can be incomplete” (page 5) and “The simplified representation can be reduced to merely averaging 6% of the points in the original polyline...” (page 7)Rationale: Zhao expressly discloses sparse extracted road-boundary data and expressly discloses a simplified representation using a greatly reduced number of points. Thus, Zhao teaches the concept of sparsifying a denser representation into a lighter-weight, reduced-point subset. Even if Zhao’s simplified product is discussed in a vectorized/polyline context, one of ordinary skill in the art would have found it obvious to apply the same density-reduction principle to point-cloud subsets used in the fused-map construction pipeline.
to generate the point cloud data.See at least: “the sparse extraction results can be enhanced which generate the local grid map (LGM) of road boundaries” (page 6) and “The simplified representation can be reduced to merely averaging 6% of the points in the original polyline ... which is both lightweight and beneficial to the following matching process.” (page 7)Rationale: Zhao expressly discloses generation of operative local map data from extracted/fused road-boundary information and further discloses that a simplified, reduced-point representation is beneficial for subsequent matching. Accordingly, Zhao teaches or at least renders obvious generating the point-cloud/map data from either the denser or reduced subset representation, depending on the implemented density-handling choice.
Zhao teaches or renders obvious the dense/sparse subset-processing concept of Claim 4, including dense local road/intersection data generation, sparsification of overly dense representations, and reduced-point representations beneficial for later matching. The specific “predetermined threshold” wording is not expressly disclosed, but is a routine and predictable implementation detail for deciding when to retain a dense subset and when to convert it into a sparser subset.
Motivation to Combine Luo, Flint, and Zhao
Therefore, given the teachings as a whole, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having Luo, Flint, and Zhao before them, to modify the Luo/Flint/Zhao combination used for Claims 1 and 3 so that the road- and intersection-specific point-cloud subsets generated during fused-map pre-construction are retained in denser form when the amount of subset data is acceptable, and are converted into sparser subsets when the amount of subset data becomes excessive, because Zhao already teaches densification of sparse road-boundary data through multiframe fusion, further teaches that dense/noisy local representations are simplified into much lighter-weight representations for efficiency and improved matching, Luo teaches a point-cloud/global-map localization framework in which such point-based map content is used for matching-based position determination, and Flint remains part of the parent-claim combination for the IMU-based updating aspect of Claim 1. One of ordinary skill in the art would have made such modification to control map-data volume and noise while preserving the utility of the point-cloud subsets for subsequent matching and localization, with the predictable benefit of improved storage efficiency and matching performance. The use of a predetermined threshold to decide when to retain dense data and when to sparsify it would have been a routine design choice within the level of ordinary skill in the art. Such a modification would not alter the principle of operation of any reference or render any reference unsatisfactory for its intended purpose. Therefore, After combining the teachings of Luo, Flint, and Zhao, Claim 4 is rendered obvious by the combination of Luo in view of Flint and Zhao. In particular, Luo teaches the underlying point-cloud/global-map localization framework, Flint remains part of the parent-claim combination for the IMU-based updating aspect of Claim 1, and Zhao teaches dense local road/intersection map generation, subsequent simplification of overly dense/noisy representations, and reduced-point representations that are beneficial for later matching. The remaining aspects concerning an explicit predetermined threshold and the branch between retaining dense subsets versus converting them into sparse subsets would have been obvious to one of ordinary skill in the art as routine implementation choices in view of Zhao’s disclosed densification/simplification workflow and its predictable integration with the Luo/Flint localization framework.
Regarding Claim 11,
The combination of Luo, Flint, and Zhao establishes the device of Claim 9, which is the basis for Claim 11.
Luo renders obvious:
whereinSee at least: “The processor 61 is configured to execute program instructions that include a tool module configured to perform a method…” ([0104]).Rationale: Luo expressly discloses a device whose processor-executed instructions perform the recited localization operations. Thus, the “wherein” clause properly introduces further conditions on that already established device-implemented method.
The point cloud basemap includes point cloud dataSee at least: “constructing a global map comprises: obtaining the data from the LiDAR; and constructing a city-scale 3D map based on the data from the LiDAR, using LiDAR mapping.” ([0014]) and “the unstructured features may include sparse 3D points.” ([0017]); see also “The global map includes a 3D city-scale map.” ([0076]).Rationale: Luo expressly discloses a LiDAR-derived 3D global map and sparse 3D point features. One of ordinary skill in the art would have understood such LiDAR-derived 3D map data to correspond to point-cloud-type map data. Accordingly, Luo teaches or at least renders obvious the point-cloud-data aspect of the claimed point cloud basemap.
Describing measurement of objects within the scenarioSee at least: “obtaining the data from the LiDAR; and constructing a city-scale 3D map based on the data from the LiDAR” ([0014]) and “the unstructured features may include sparse 3D points.” ([0017]).Rationale: Luo expressly discloses LiDAR-acquired 3D map data and 3D point-based features representing measured environmental content. Thus, Luo teaches that the point-cloud-type map data describes measured objects/features in the mapped scene, although Luo does not expressly tie those objects to roads and intersections specifically.
And the vector map includes vector graphic elementsSee at least: “the structured features include at least one of planes, straight lines and curved lines…” ([0017]) and “The extracted features include structured features such as planes, straight lines and curved lines…” ([0082]).Rationale: Luo does not expressly use the phrase “vector graphic elements.” However, Luo expressly discloses structured geometric features such as planes, straight lines, and curved lines. One of ordinary skill in the art would have recognized such structured geometric descriptors as vector-like elements of a map representation. Accordingly, Luo at least renders obvious the vector-element aspect of the claim.
Luo teaches or renders obvious the device-based point-cloud/global-map framework of Claim 11 and further provides a strong basis for treating structured geometric map features as vector-like elements. The road-specific, intersection-specific, subset-generation, and explicit mapping/construction aspects are more directly shown by Zhao, while Flint remains part of the parent-claim combination for Claim 9.
Zhao teaches:
describing measurement of objects within the scenarioSee at least: “The LiDAR-based SLAM generates accurate maps represented by occupancy grids or 3D point clouds...” (page 2) and “the 3D point cloud of a scene” and “the road boundaries extracted in one frame” and “the multiframe probabilistic fusion” (page 5, Figure 2 and accompanying text).Rationale: Zhao expressly discloses LiDAR-based mapping from 3D point clouds of a scene and extraction of road-boundary objects from those measurements. Thus, Zhao teaches point-cloud data describing measured objects within the mapped scenario.
at roads and intersections within the scenario,See at least: “The high-precision high-definition (HD) map of the road environment is now recognized as one of the cornerstones for autonomous driving. The reliable mapping of road boundaries, lanes, and other road structures...” (page 2) and “Figure 11c-I, II, and VI show successfully mapped intersections; all the road boundaries along the trajectory are mapped...” (page 13).Rationale: Zhao expressly discloses that its mapping concerns road structures in road environments and further expressly identifies mapped intersections. Accordingly, Zhao teaches the road- and intersection-specific context missing from Luo.
and the vector map includes vector graphic elementsSee at least: “we propose a vector-based SLAM method for the road structure mapping using vehicle-mounted multibeam LiDAR. We propose using polylines as the primary mapping element instead of grid maps or point clouds because the vector-based representation is lightweight and precise.” (page 2) and “Vector primitives, for example, polygons and line segments, are widely adopted as the representation of road geometry...” (page 3).Rationale: Zhao expressly discloses vector-based mapping and expressly identifies polylines, polygons, and line segments as vector-based mapping elements. Thus, Zhao teaches the claimed vector graphic elements.
describing geometric characteristics of roads and intersections within the scenario,See at least: “The high-precision high-definition (HD) map of the road environment ... The reliable mapping of road boundaries, lanes, and other road structures...” (page 2) and “Vector primitives ... are widely adopted as the representation of road geometry...” (page 3).Rationale: Zhao expressly discloses vector-based representation of road geometry and road structures, and further shows mapped intersections in the resulting vectorized map discussion. One of ordinary skill in the art would have understood such vector elements to describe geometric characteristics of roads and intersections within the scenario.
and wherein,See at least: “We explored a combined pipeline of road structure extraction and vectorization, vector-based local map matching, optimization and vector reconstruction.” (page 2).Rationale: Zhao expressly discloses an ordered mapping pipeline in which additional map-construction operations are performed. Thus, the clause properly introduces further map-preconstruction steps.
the method further comprises pre-constructing the fused map by:See at least: “We explored a combined pipeline of road structure extraction and vectorization, vector-based local map matching, optimization and vector reconstruction.” (page 2) and “First, the LVMs are fused without concatenation... Finally, a reconstruction method is employed to globally reconstruct the vector map...” (page 9).Rationale: Zhao expressly discloses a mapping workflow in which road-structure data is extracted, vectorized, fused, and reconstructed into a vector map. One of ordinary skill in the art would have understood this to be a pre-construction process for creating the map used later for localization.
creating corresponding point cloud data subsetsSee at least: “Figure 2. Road boundary extraction. (a) the 3D point cloud of a scene ... (d) the road boundaries extracted in one frame. (e) the multiframe probabilistic fusion.” (page 5) and “the sparse extraction results can be enhanced which generate the local grid map (LGM) of road boundaries” (page 6).Rationale: Zhao expressly discloses beginning with scene point-cloud data, extracting road-boundary portions therefrom, and then generating local map representations from those extracted portions. One of ordinary skill in the art would have understood those extracted road-boundary portions and local maps to constitute corresponding subsets of the original point-cloud data for subsequent processing.
respectively for each road and each intersection within the scenarioSee at least: “The data collecting path is approximately 2 km in length, passing through all road segments, including 3 cross-intersections and 8 T-junctions...” (page 13) and “Figure 11c-I, II, and VI show successfully mapped intersections; all the road boundaries along the trajectory are mapped...” (page 13).Rationale: Zhao expressly discloses mapping across multiple road segments and intersections, including cross-intersections and T-junctions. Thus, Zhao teaches road- and intersection-specific map generation, which one of ordinary skill in the art would have understood as involving corresponding subsets or local portions associated with the respective road and intersection regions.
to generate the point cloud data; andSee at least: “To fuse multiframes locally with high precision, we develop a reckoning system...” (page 5) and “As a result, the sparse extraction results can be enhanced which generate the local grid map (LGM) of road boundaries” (page 6).Rationale: Zhao expressly discloses generating locally fused map data from multiple LiDAR-derived frames. Thus, Zhao teaches generating point-cloud-derived map data from extracted/fused subsets of the measured scene.
mapping the point cloud data with the vector graphic elementsSee at least: “The LGM generated in the previous step is still a grid map-based representation of road boundaries. To convert the grid map into a vector-based map...” (page 6) and “we employ the virtual scan method to locally vectorize the LGM” (page 6); see also “In Figure 9e, the ordered points belonging to a sector of the road boundaries are connected to realize the initial vectorization.” (page 11).Rationale: Zhao expressly discloses converting LiDAR/point-cloud-derived road-boundary map data into vector-based mapping elements through vectorization. In other words, Zhao teaches mapping the point-cloud-derived road data with vector graphic elements.
to construct the fused map.See at least: “The optimization and reconstruction of local vector-based maps into a global vector map of the road structure is achieved.” (page 3) and “Finally, a reconstruction method is employed to globally reconstruct the vector map from sampled LVMs.” (page 9).Rationale: Zhao expressly discloses constructing a final global vector map by fusing, reconstructing, and vectorizing the road-structure data. In view of Luo’s point-cloud/global-map framework and Zhao’s vector-road-map construction workflow, one of ordinary skill in the art would have found it obvious to construct the claimed fused map by associating point-cloud-derived road/intersection data with vector graphic elements.
Motivation to Combine Luo, Flint, and Zhao
Therefore, given the teachings as a whole, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having Luo, Flint, and Zhao before them, to modify Luo’s device-based camera/LiDAR map-based vehicle-localization system, as further informed by Flint’s IMU-based relative-displacement updating used for the parent Claim 9 combination, so that the fused map used by the device includes road- and intersection-specific point-cloud data subsets and corresponding vector graphic road-geometry elements constructed through a vectorization and reconstruction workflow as taught by Zhao, because Luo already discloses a device framework having processor-executed localization using a LiDAR-derived global map, a camera-derived submap, and structured/unstructured map features, Flint teaches a known and compatible inertial-update technique for maintaining vehicle position between map-matching determinations, and Zhao teaches a known and compatible technique for generating vector-based road-structure maps from LiDAR-derived road-boundary data, including road and intersection mapping, local subset generation, vectorization, and reconstruction. One of ordinary skill in the art would have made such modification to provide a more lightweight, road-geometry-aware, and structured fused map representation for use by Luo’s localization device while preserving the continuous vehicle-position updating already supported by the Luo/Flint combination, with the predictable benefit of improving map usability and localization performance in road/intersection environments. Such a modification would merely apply a known vector-road-map generation technique to a known LiDAR-based localization device, while retaining known inertial-update processing, and would not alter the principle of operation of any reference or render any reference unsatisfactory for its intended purpose. Therefore, after combining the teachings of Luo, Flint, and Zhao, Claim 11 is rendered obvious by the combination of Luo in view of Flint and Zhao. In particular, Luo teaches the underlying device framework and point-cloud/global-map localization features, Flint teaches the IMU-based updating portion carried through from parent Claim 9, and Zhao teaches road- and intersection-specific vector-map construction from LiDAR-derived road-structure data, including point-cloud-derived subsets, vector graphic elements, and reconstruction of the fused/vector map. The additional map-content and pre-construction aspects of Claim 11 would have been obvious to one of ordinary skill in the art in view of Zhao’s disclosed vector-road-mapping architecture and its predictable integration with the Luo/Flint device-based localization framework.
Regarding Claim 12,
The combination of Luo, Flint, and Zhao establishes the device of Claim 11, which is the basis for Claim 12.
Zhao teaches or renders obvious:
whereinSee at least: “We explored a combined pipeline of road structure extraction and vectorization, vector-based local map matching, optimization and vector reconstruction.” (page 2)Rationale: Zhao expressly discloses an ordered mapping pipeline with additional processing stages. Thus, the “wherein” clause properly introduces additional conditions on the subset-generation portion of the already established device-based method.
creating corresponding point cloud data subsetsSee at least: “Figure 2. Road boundary extraction. (a) the 3D point cloud of a scene ... (d) the road boundaries extracted in one frame. (e) the multiframe probabilistic fusion.” (page 5) and “the sparse extraction results can be enhanced which generate the local grid map (LGM) of road boundaries” (page 6)Rationale: Zhao expressly discloses beginning with scene point-cloud data, extracting road-boundary portions therefrom, and generating local map representations from those extracted portions. One of ordinary skill in the art would have understood those extracted road-boundary portions and local maps to constitute corresponding subsets of the original point-cloud data for subsequent processing.
respectively for each road and each intersection within the scenarioSee at least: “The data collecting path is approximately 2 km in length, passing through all road segments, including 3 cross-intersections and 8 T-junctions...” (page 13) and “Figure 11c-I, II, and VI show successfully mapped intersections; all the road boundaries along the trajectory are mapped...” (page 13)Rationale: Zhao expressly discloses mapping across multiple road segments and intersections, including cross-intersections and T-junctions. Thus, Zhao teaches road- and intersection-specific map generation, which one of ordinary skill in the art would have understood as involving corresponding subsets or local portions associated with the respective road and intersection regions.
comprises:See at least: “We explored a combined pipeline of road structure extraction and vectorization...” (page 2)Rationale: Zhao expressly discloses an open-ended sequence of mapping operations, thereby supporting the recited transitional phrase for the additional subset-processing steps.
obtaining dense point cloud data subsetsSee at least: “However, the extracted road boundary in one frame is sparse and can be incomplete; therefore, it should be further densified by the multiframe probabilistic fusion.” (page 5) and “To fuse multiframes locally with high precision...” (page 5)Rationale: Zhao expressly discloses that sparse single-frame road-boundary extraction is further densified by multiframe probabilistic fusion. One of ordinary skill in the art would have understood the densified local fused boundary representation to constitute dense point-cloud-derived subsets for the corresponding road/intersection regions.
for each road and each intersection within the scenario,See at least: “all road segments, including 3 cross-intersections and 8 T-junctions...” (page 13) and “all the road boundaries along the trajectory are mapped” and “successfully mapped intersections” (page 13)Rationale: Zhao expressly discloses dense local road-boundary mapping across multiple road segments and intersections. Thus, the densified subset generation is taught in the road- and intersection-specific context required by the claim.
and determining whether a first total amount of dataSee at least: “Since the vectorized result can be noisy and dense due to the selected angular resolution of the virtual scan...” (page 7) and “The simplified representation can be reduced to merely averaging 6% of the points in the original polyline...” (page 7)Rationale: Zhao does not expressly recite computing a “first total amount of data” in those exact words. However, Zhao expressly distinguishes between denser and lighter-weight representations and expressly quantifies that simplification reduces the retained representation to about 6% of the original points. In view of these teachings, one of ordinary skill in the art would have understood that the mapping pipeline evaluates whether the amount of point/polyline data is sufficiently large to warrant simplification or sparsification. Accordingly, this data-amount determination is at least implicit in Zhao, or would have been obvious therefrom.
for respective dense point cloud data subsetsSee at least: “the extracted road boundary in one frame is sparse and can be incomplete; therefore, it should be further densified by the multiframe probabilistic fusion.” (page 5) and “the fused local grid map of road boundaries (LGM)” (page 11 figure description)Rationale: Zhao expressly discloses locally fused, densified road-boundary representations derived from multiframe point-cloud data. One of ordinary skill in the art would have understood these fused local representations to be the relevant dense subsets whose amount of data may be evaluated for subsequent simplification.
exceeds a predetermined threshold,See at least: “Since the vectorized result can be noisy and dense due to the selected angular resolution of the virtual scan, we employ the Ramer-Douglas-Peucker algorithm [41] to optimize polylines in the LVM.” (page 7) and “The comparison of the error statistics ... the matching with simplified-LVMs outperforms raw-LVM-based matching ... because of its lightweight and less noisy polyline-based representation.” (page 11)Rationale: Zhao does not expressly state a “predetermined threshold” for a first total amount of data. However, Zhao expressly teaches that when the representation is noisy and dense, a simplification operation is applied to reduce the amount of retained map data. One of ordinary skill in the art would have understood that, in implementing such density-driven simplification, some threshold or criterion must be used to decide when the dense representation should be retained and when it should be simplified. Use of a predetermined threshold for that decision would have been a routine and predictable implementation choice.
wherein when the first total amount of data does not exceed the predetermined threshold,See at least: “For a clear description, we refer to the simplified version of LVM as simplified LVM and the original version as raw LVM.” (page 7)Rationale: Zhao expressly distinguishes between an original, denser representation and a simplified representation. Although Zhao does not expressly recite the decision branch in the exact words of the claim, one of ordinary skill in the art would have understood that if the amount of data is not excessive, the original denser representation may simply be retained rather than simplified. This branch is therefore rendered obvious by Zhao’s express raw-versus-simplified framework.
taking the respective dense point cloud data subsets as the point cloud data,See at least: “the 3D point cloud of a scene” and “the road boundaries extracted in one frame” and “the multiframe probabilistic fusion” (page 5)Rationale: Zhao expressly discloses use of the extracted and fused road-boundary data as the operative point-cloud-derived map content. Where simplification is not invoked, one of ordinary skill in the art would have understood the dense subsets themselves to serve as the point-cloud data used downstream.
and when the first total amount of data exceeds the predetermined threshold,See at least: “Since the vectorized result can be noisy and dense due to the selected angular resolution of the virtual scan, we employ the Ramer-Douglas-Peucker algorithm [41] to optimize polylines in the LVM.” (page 7)Rationale: Zhao expressly teaches that when the representation is dense and noisy, simplification/optimization is applied. Although Zhao does not expressly frame this in terms of a threshold comparison, the conditional relationship between excessive density and subsequent simplification is clearly taught, and the use of a predetermined threshold to trigger that simplification would have been an obvious implementation detail.
converting the respective dense point cloud data subsetsSee at least: “The LGM generated in the previous step is still a grid map-based representation of road boundaries. To convert the grid map into a vector-based map...” (page 6) and “we employ the virtual scan method to locally vectorize the LGM.” (page 6)Rationale: Zhao expressly discloses converting denser local road-boundary representations into a more compact representation. Although the claim specifically recites converting dense point-cloud subsets into sparse point-cloud subsets, Zhao’s express teaching of transforming dense local road-boundary data into a lighter-weight representation strongly supports the claimed conversion concept, at least as rendered obvious to one of ordinary skill in the art.
into sparse point cloud data subsetsSee at least: “the extracted road boundary in one frame is sparse and can be incomplete” (page 5) and “The simplified representation can be reduced to merely averaging 6% of the points in the original polyline...” (page 7)Rationale: Zhao expressly discloses sparse extracted road-boundary data and expressly discloses a simplified representation using a greatly reduced number of points. Thus, Zhao teaches the concept of sparsifying a denser representation into a lighter-weight, reduced-point subset. Even if Zhao’s simplified product is discussed in a vectorized/polyline context, one of ordinary skill in the art would have found it obvious to apply the same density-reduction principle to point-cloud subsets used in the fused-map construction pipeline.
to generate the point cloud data.See at least: “the sparse extraction results can be enhanced which generate the local grid map (LGM) of road boundaries” (page 6) and “The simplified representation can be reduced to merely averaging 6% of the points in the original polyline ... which is both lightweight and beneficial to the following matching process.” (page 7)Rationale: Zhao expressly discloses generation of operative local map data from extracted/fused road-boundary information and further discloses that a simplified, reduced-point representation is beneficial for subsequent matching. Accordingly, Zhao teaches or at least renders obvious generating the point-cloud/map data from either the denser or reduced subset representation, depending on the implemented density-handling choice.
Zhao teaches or renders obvious the dense/sparse subset-processing concept of Claim 12, including dense local road/intersection data generation, sparsification of overly dense representations, and reduced-point representations beneficial for later matching. The specific “predetermined threshold” wording is not expressly disclosed, but is a routine and predictable implementation detail for deciding when to retain a dense subset and when to convert it into a sparser subset.
Motivation to Combine Luo, Flint, and Zhao
Therefore, given the teachings as a whole, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having Luo, Flint, and Zhao before them, to modify the Luo/Flint/Zhao combination used for Claims 9 and 11 so that the road- and intersection-specific point-cloud subsets generated during fused-map pre-construction for the device are retained in denser form when the amount of subset data is acceptable, and are converted into sparser subsets when the amount of subset data becomes excessive, because Zhao already teaches densification of sparse road-boundary data through multiframe fusion, further teaches that dense/noisy local representations are simplified into much lighter-weight representations for efficiency and improved matching, Luo teaches a device-based point-cloud/global-map localization framework in which such point-based map content is used for matching-based position determination, and Flint remains part of the parent-claim combination for the IMU-based updating aspect of Claim 9. One of ordinary skill in the art would have made such modification to control map-data volume and noise while preserving the utility of the point-cloud subsets for subsequent matching and localization by the device, with the predictable benefit of improved storage efficiency and matching performance. The use of a predetermined threshold to decide when to retain dense data and when to sparsify it would have been a routine design choice within the level of ordinary skill in the art. Such a modification would not alter the principle of operation of any reference or render any reference unsatisfactory for its intended purpose. Therefore, after combining the teachings of Luo, Flint, and Zhao, Claim 12 is rendered obvious by the combination of Luo in view of Flint and Zhao. In particular, Luo teaches the underlying device-based point-cloud/global-map localization framework, Flint remains part of the parent-claim combination for the IMU-based updating aspect of Claim 9, and Zhao teaches dense local road/intersection map generation, subsequent simplification of overly dense/noisy representations, and reduced-point representations that are beneficial for later matching. The remaining aspects concerning an explicit predetermined threshold and the branch between retaining dense subsets versus converting them into sparse subsets would have been obvious to one of ordinary skill in the art as routine implementation choices in view of Zhao’s disclosed densification/simplification workflow and its predictable integration with the Luo/Flint/Zhao device-based localization framework.
Regarding Claim 19,
The combination of Luo, Flint, and Zhao establishes the non-transitory computer-readable storage medium of Claim 17, which is the basis for Claim 19.
Luo discloses or renders obvious:
whereinSee at least: “The one or more programs comprise instructions, which when executed by a computing device, cause the computing device to perform...” ([0010]).Rationale: The reference expressly discloses that the stored instructions cause a computing device/processor to carry out the localization method. Thus, the “wherein” clause properly introduces additional conditions on that already established storage-medium-implemented method.
the point cloud basemap includes point cloud dataSee at least: “constructing a global map comprises: obtaining the data from the LiDAR; and constructing a city-scale 3D map based on the data from the LiDAR, using LiDAR mapping.” ([0014]) and “the unstructured features may include sparse 3D points.” ([0017]); see also “The global map includes a 3D city-scale map.” ([0076]).Rationale: The reference expressly discloses a LiDAR-derived 3D global map and sparse 3D point features. One of ordinary skill in the art would have understood such LiDAR-derived 3D map data to correspond to point-cloud-type map data. Accordingly, Luo teaches or at least renders obvious the point-cloud-data aspect of the claimed point cloud basemap.
describing measurement of objects within the scenarioSee at least: “obtaining the data from the LiDAR; and constructing a city-scale 3D map based on the data from the LiDAR” ([0014]) and “the unstructured features may include sparse 3D points.” ([0017]).Rationale: The reference expressly discloses LiDAR-acquired 3D map data and 3D point-based features representing measured environmental content. Thus, Luo teaches that the point-cloud-type map data describes measured objects/features in the mapped scene, although Luo does not expressly tie those objects to roads and intersections specifically.
and the vector map includes vector graphic elementsSee at least: “the structured features include at least one of planes, straight lines and curved lines...” ([0017]) and “The extracted features include structured features such as planes, straight lines and curved lines...” ([0082]).Rationale: Luo does not expressly use the phrase “vector graphic elements.” However, Luo expressly discloses structured geometric features such as planes, straight lines, and curved lines. One of ordinary skill in the art would have recognized such structured geometric descriptors as vector-like elements of a map representation. Accordingly, Luo at least renders obvious the vector-element aspect of the claim.
Luo teaches or renders obvious the non-transitory-storage-medium-based point-cloud/global-map framework of Claim 19 and further provides a strong basis for treating structured geometric map features as vector-like elements. The road-specific, intersection-specific, subset-generation, and explicit mapping/construction aspects are more directly shown by Zhao, while Flint remains part of the parent-claim combination for Claim 17.
Zhao discloses:
describing measurement of objects within the scenarioSee at least: “The LiDAR-based SLAM generates accurate maps represented by occupancy grids or 3D point clouds...” (page 2) and “the 3D point cloud of a scene” and “the 2D projection” and “the road boundaries extracted in one frame” and “the multiframe probabilistic fusion” (page 5, Figure 2 and accompanying text).Rationale: Zhao expressly discloses LiDAR-based mapping from 3D point clouds of a scene and extraction of road-boundary objects from those measurements. Thus, Zhao teaches point-cloud data describing measured objects within the mapped scenario.
at roads and intersections within the scenario,See at least: “The high-precision high-definition (HD) map of the road environment is now recognized as one of the cornerstones for autonomous driving. The reliable mapping of road boundaries, lanes, and other road structures...” (page 2) and “Figure 11c-I, II, and VI show successfully mapped intersections; all the road boundaries along the trajectory are mapped...” (page 13).Rationale: Zhao expressly discloses that its mapping concerns road structures in road environments and further expressly identifies mapped intersections. Accordingly, Zhao teaches the road- and intersection-specific context missing from Luo.
and the vector map includes vector graphic elementsSee at least: “we propose a vector-based SLAM method for the road structure mapping using vehicle-mounted multibeam LiDAR. We propose using polylines as the primary mapping element instead of grid maps or point clouds because the vector-based representation is lightweight and precise.” (page 2) and “Vector primitives, for example, polygons and line segments, are widely adopted as the representation of road geometry...” (page 3).Rationale: Zhao expressly discloses vector-based mapping and expressly identifies polylines, polygons, and line segments as vector-based mapping elements. Thus, Zhao teaches the claimed vector graphic elements.
describing geometric characteristics of roads and intersections within the scenario,See at least: “The high-precision high-definition (HD) map of the road environment ... The reliable mapping of road boundaries, lanes, and other road structures...” (page 2) and “Vector primitives ... are widely adopted as the representation of road geometry...” (page 3).Rationale: Zhao expressly discloses vector-based representation of road geometry and road structures, and further shows mapped intersections in the resulting vectorized map discussion. One of ordinary skill in the art would have understood such vector elements to describe geometric characteristics of roads and intersections within the scenario.
and wherein,See at least: “We explored a combined pipeline of road structure extraction and vectorization, vector-based local map matching, optimization and vector reconstruction.” (page 2).Rationale: Zhao expressly discloses an ordered mapping pipeline in which additional map-construction operations are performed. Thus, the clause properly introduces further map-preconstruction steps.
the method further comprises pre-constructing the fused map by:See at least: “We explored a combined pipeline of road structure extraction and vectorization, vector-based local map matching, optimization and vector reconstruction.” (page 2) and “First, the LVMs are fused without concatenation... Finally, a reconstruction method is employed to globally reconstruct the vector map...” (page 9).Rationale: Zhao expressly discloses a mapping workflow in which road-structure data is extracted, vectorized, fused, and reconstructed into a vector map. One of ordinary skill in the art would have understood this to be a pre-construction process for creating the map used later for localization.
creating corresponding point cloud data subsetsSee at least: “Figure 2. Road boundary extraction. (a) the 3D point cloud of a scene ... (d) the road boundaries extracted in one frame. (e) the multiframe probabilistic fusion.” (page 5) and “the sparse extraction results can be enhanced which generate the local grid map (LGM) of road boundaries” (page 6).Rationale: Zhao expressly discloses beginning with scene point-cloud data, extracting road-boundary portions therefrom, and then generating local map representations from those extracted portions. One of ordinary skill in the art would have understood those extracted road-boundary portions and local maps to constitute corresponding subsets of the original point-cloud data for subsequent processing.
respectively for each road and each intersection within the scenarioSee at least: “The data collecting path is approximately 2 km in length, passing through all road segments, including 3 cross-intersections and 8 T-junctions...” (page 13) and “Figure 11c-I, II, and VI show successfully mapped intersections; all the road boundaries along the trajectory are mapped...” (page 13).Rationale: Zhao expressly discloses mapping across multiple road segments and intersections, including cross-intersections and T-junctions. Thus, Zhao teaches road- and intersection-specific map generation, which one of ordinary skill in the art would have understood as involving corresponding subsets or local portions associated with the respective road and intersection regions.
to generate the point cloud data; andSee at least: “To fuse multiframes locally with high precision, we develop a reckoning system...” (page 5) and “As a result, the sparse extraction results can be enhanced which generate the local grid map (LGM) of road boundaries” (page 6).Rationale: Zhao expressly discloses generating locally fused map data from multiple LiDAR-derived frames. Thus, Zhao teaches generating point-cloud-derived map data from extracted/fused subsets of the measured scene.
mapping the point cloud data with the vector graphic elementsSee at least: “The LGM generated in the previous step is still a grid map-based representation of road boundaries. To convert the grid map into a vector-based map...” (page 6) and “we employ the virtual scan method to locally vectorize the LGM” (page 6); see also “In Figure 9e, the ordered points belonging to a sector of the road boundaries are connected to realize the initial vectorization.” (page 11).Rationale: Zhao expressly discloses converting LiDAR/point-cloud-derived road-boundary map data into vector-based mapping elements through vectorization. In other words, Zhao teaches mapping the point-cloud-derived road data with vector graphic elements.
to construct the fused map.See at least: “The optimization and reconstruction of local vector-based maps into a global vector map of the road structure is achieved.” (page 3) and “Finally, a reconstruction method is employed to globally reconstruct the vector map from sampled LVMs.” (page 9).Rationale: Zhao expressly discloses constructing a final global vector map by fusing, reconstructing, and vectorizing the road-structure data. In view of Luo’s point-cloud/global-map framework and Zhao’s vector-road-map construction workflow, one of ordinary skill in the art would have found it obvious to construct the claimed fused map by associating point-cloud-derived road/intersection data with vector graphic elements.
Motivation to Combine Luo, Flint, and Zhao
Therefore, given the teachings as a whole, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having Luo, Flint, and Zhao before them, to modify Luo’s storage-medium-based camera/LiDAR map-based vehicle-localization method, as further informed by Flint’s IMU-based relative-displacement updating used for the parent Claim 17 combination, so that the fused map used by the processor-executed instructions includes road- and intersection-specific point-cloud data subsets and corresponding vector graphic road-geometry elements constructed through a vectorization and reconstruction workflow as taught by Zhao, because Luo already discloses a non-transitory storage medium storing instructions for localization using a LiDAR-derived global map, a camera-derived submap, and structured/unstructured map features, Flint teaches a known and compatible inertial-update technique for maintaining vehicle position between map-matching determinations, and Zhao teaches a known and compatible technique for generating vector-based road-structure maps from LiDAR-derived road-boundary data, including road and intersection mapping, local subset generation, vectorization, and reconstruction. One of ordinary skill in the art would have made such modification to provide a more lightweight, road-geometry-aware, and structured fused map representation for use by Luo’s storage-medium-implemented localization method while preserving the continuous vehicle-position updating already supported by the Luo/Flint combination, with the predictable benefit of improving map usability and localization performance in road/intersection environments. Such a modification would merely apply a known vector-road-map generation technique to a known LiDAR-based localization framework, while retaining known inertial-update processing, and would not alter the principle of operation of any reference or render any reference unsatisfactory for its intended purpose. Therefore, after combining the teachings of Luo, Flint, and Zhao, Claim 19 is rendered obvious by the combination of Luo in view of Flint and Zhao. In particular, Luo teaches the underlying non-transitory storage-medium framework and point-cloud/global-map localization features, Flint teaches the IMU-based updating portion carried through from parent Claim 17, and Zhao teaches road- and intersection-specific vector-map construction from LiDAR-derived road-structure data, including point-cloud-derived subsets, vector graphic elements, and reconstruction of the fused/vector map. The additional map-content and pre-construction aspects of Claim 19 would have been obvious to one of ordinary skill in the art in view of Zhao’s disclosed vector-road-mapping architecture and its predictable integration with the Luo/Flint storage-medium-based localization framework.
Claims 5 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Luo, in view of Flint, in view of Zhao, and Zhou (Non-iterative denoising algorithm based on a dual threshold for a 3D point cloud)
Regarding Claim 5,
The combination of Luo, Flint, and Zhao establishes the method of Claim 4, which is the basis for Claim 5.
Zhao teaches or renders obvious:
whereinSee at least: “We explored a combined pipeline of road structure extraction and vectorization, vector-based local map matching, optimization and vector reconstruction.” (page 2)Rationale: Zhao expressly discloses an ordered mapping pipeline with additional processing stages. Thus, the “wherein” clause properly introduces further conditions on the already established sparse-subset processing workflow.
converting the respective dense point cloud data subsetsSee at least: “The LGM generated in the previous step is still a grid map-based representation of road boundaries. To convert the grid map into a vector-based map...” (page 6) and “Since the vectorized result can be noisy and dense ... we employ the Ramer-Douglas-Peucker algorithm [41] to optimize polylines in the LVM.” (page 7)Rationale: This dense-to-sparse conversion concept was already relied upon in establishing Claim 4. It is repeated here only as antecedent context for the new Claim 5 limitations concerning what is done after the sparse subsets have already been generated. The newly added limitations begin with the second-stage evaluation of those sparse subsets.
Zhou teaches or renders obvious:
determining whether a second total amount of dataSee at least: “In this study, a non-iterative dual threshold framework for 3D point cloud denoising is proposed.” (page 2) and “Finally, we calculate the small threshold of the 3D point cloud ... where Nsum is the number of all points in P′.” (page 4)Rationale: Zhou expressly discloses threshold-based point-cloud processing and expressly uses the number of all points in the point cloud as part of the threshold framework. One of ordinary skill in the art would have understood this as evaluating the amount of point-cloud data to determine how further point-cloud reduction should proceed. Accordingly, the claimed determination of a second total amount of data is expressly suggested by Zhou’s point-count-based threshold processing.
of respective sparse point cloud data subsetsSee at least: “The first stage removes the drift points using a small threshold. The second stage removes the drift points using a large threshold.” (page 2) and “After the non-iterative small threshold denoising of the 3D point cloud, we get P″ ... The remaining drift points are continuously deleted by the non-iterative large threshold denoising algorithm.” (page 4)Rationale: Zhou expressly discloses that, after a first-stage reduction, a remaining point-cloud representation is subjected to a second threshold stage. One of ordinary skill in the art would have understood the post-first-stage representation to be a sparser subset of the original point cloud on which the second-stage decision is made.
exceeds the predetermined threshold,See at least: “The first stage removes the drift points using a small threshold. The second stage removes the drift points using a large threshold.” (page 2) and “If there is only one point in the Vi,j, it is identified as a drift point; otherwise the points are regarded as non-noise points.” (page 4)Rationale: Zhou expressly discloses predetermined thresholds governing whether point-cloud data is retained or removed in a two-stage denoising process. Thus, Zhou directly supports the claimed threshold-comparison concept applied to already sparse subsets.
wherein when the second total amount of data does not exceed the predetermined threshold,See at least: “If there is only one point in the Vi,j, it is identified as a drift point; otherwise the points are regarded as non-noise points.” (page 4)Rationale: Zhou expressly discloses a threshold-governed branch in which points are either removed or retained. One of ordinary skill in the art would have understood that where the post-conversion sparse data does not exceed the operative threshold, the existing sparse subset may simply be retained for downstream use.
taking the respective sparse point cloud data subsets as the point cloud data,See at least: “We can then retain the non-noise points and delete the drift points.” (page 4)Rationale: Zhou expressly discloses retaining the point-cloud data that survives threshold-based filtering. Accordingly, Zhou teaches the claimed branch in which the sparse subsets are taken as the operative point-cloud data.
and when the second total amount of data exceeds the predetermined threshold,See at least: “The second stage removes the drift points using a large threshold.” (page 2) and “The remaining drift points are continuously deleted by the non-iterative large threshold denoising algorithm.” (page 4)Rationale: Zhou expressly teaches that when the reduced representation still warrants further processing, additional threshold-based reduction is applied. Thus, Zhou supports the claimed second conditional branch when the second amount of data still exceeds the threshold.
Zhao further teaches or renders obvious the road/intersection-specific treatment of the already sparse subsets:
manually sampling the sparse point cloud data subset for each roadSee at least: “A probe-based sampling method is then employed to sample the fused road boundaries.” (page 8–9) and “the sampled nodes in the proposed method are not included in the optimization” (page 8)Rationale: Zhao expressly discloses sampling road-boundary data after fusion. Zhao does not expressly recite that the sampling is “manual.” However, one of ordinary skill in the art would have understood that, where an automatically produced sparse road subset still remains too large after threshold-based reduction, manual or user-directed sampling of the road subset would have been a routine and predictable implementation option for further controlling road data density.
at a predetermined interval,See at least: “The key difference is that we sample the nodes evenly in the referenced polyline...” (page 7) and “A probe-based sampling method is then employed to sample the fused road boundaries.” (page 8–9)Rationale: Zhao expressly teaches even sampling of nodes and probe-based sampling of fused road boundaries. One of ordinary skill in the art would have understood even sampling to correspond to sampling at a predetermined interval. Accordingly, this interval-based sampling aspect is at least rendered obvious by Zhao.
and taking the sparse point cloud data subset for each intersectionSee at least: “Figure 11c-I, II, and VI show successfully mapped intersections” (page 13) and “the global structure of the map was not taken into consideration, which is crucial for the vectorization of complex intersections.” (page 8)Rationale: Zhao expressly emphasizes the importance of preserving detailed structure at intersections and expressly discusses successful mapped intersections. One of ordinary skill in the art would have understood from these teachings that intersection subsets are significant structural components that may appropriately be retained without the same level of reduction applied to road-only segments.
and the manually sampled sparse point cloud data subset for each roadSee at least: “we sample the nodes evenly in the referenced polyline” (page 7) and “A probe-based sampling method is then employed to sample the fused road boundaries.” (page 8–9)Rationale: Zhao expressly teaches sampling road-boundary representations and even node sampling. Thus, Zhao teaches the road-specific sampled subset aspect of the claim, at least as rendered obvious when applied after the second threshold evaluation taught by Zhou.
as the point cloud data.See at least: “The simplified representation can be reduced to merely averaging 6% of the points ... which is both lightweight and beneficial to the following matching process.” (page 7) and “the matching with simplified-LVMs outperforms raw-LVM-based matching ... in both accuracy and efficiency” (page 11)Rationale: Zhao expressly teaches using the reduced/sampled representation as the operative representation for downstream matching. Accordingly, Zhao teaches taking the retained intersection subset together with the sampled road subset as the point-cloud data used for subsequent processing.
Motivation to Combine Luo, Flint, Zhao, and Zhou
Therefore, given the teachings as a whole, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having Luo, Flint, Zhao, and Zhou before them, to modify the Luo/Flint/Zhao combination used for Claims 1, 3, and 4 so that, after the dense road- and intersection-specific point-cloud subsets have already been converted into sparse point-cloud subsets, those already sparse subsets are subjected to a second threshold-based data-volume evaluation as taught by Zhou, and, if the remaining sparse road-segment data still exceeds the acceptable threshold, the road subsets are further sampled at an even, predetermined interval while retaining the intersection subsets as taught by Zhao, because Zhou teaches a dual-threshold point-cloud reduction framework based on point-cloud amount and threshold-driven retention/further-reduction logic, Zhao teaches road-boundary sampling and even node spacing together with explicit preservation of important intersection geometry, Luo provides the overall point-cloud/global-map localization framework in which the resulting point-cloud data is used for map matching and positioning, and Flint remains part of the parent-claim combination for the IMU-based updating aspect of Claim 1. One of ordinary skill in the art would have made such modification to further control data volume and noise after the initial dense-to-sparse conversion while preserving important intersection structure and maintaining lightweight road-segment data for matching, with the predictable benefit of improved efficiency, storage economy, and matching performance. The use of a second threshold and predetermined-interval road sampling would have been a routine and predictable design choice in view of Zhou’s threshold-driven point-cloud reduction and Zhao’s express even-sampling disclosures. Such a modification would not alter the principle of operation of any reference or render any reference unsatisfactory for its intended purpose. Therefore, after combining the teachings of Luo, Flint, Zhao, and Zhou, Claim 5 is rendered obvious by the combination of Luo in view of Flint, Zhao, and Zhou. In particular, Luo teaches the underlying point-cloud/global-map localization framework, Flint remains part of the parent-claim combination for the IMU-based updating aspect of Claim 1, Zhao teaches road/intersection-specific subset generation together with road-boundary sampling, even node spacing, and use of reduced representations for matching, and Zhou teaches a second threshold-based point-cloud reduction framework based on point-cloud amount and threshold-driven retention/further-reduction logic. The remaining aspects concerning explicit manual sampling and retaining intersection subsets while further sampling road subsets would have been obvious to one of ordinary skill in the art as routine implementation choices in view of Zhao’s road-specific sampling disclosures, Zhao’s emphasis on intersection structure, and Zhou’s disclosed multi-threshold point-cloud reduction workflow.
Regarding Claim 13,
The combination of Luo, Flint, Zhao, and Zhou establishes the device of Claim 12, which is the basis for Claim 13.
Zhou discloses or renders obvious:
whereinSee at least: “The first stage removes the drift points using a small threshold. The second stage removes the drift points using a large threshold.” (page 2).Rationale: Zhou expressly discloses a multi-stage processing pipeline in which additional conditions are applied after an initial reduction stage. Thus, the “wherein” clause properly introduces further conditions on the already established device-based sparse-subset processing workflow.
converting the respective dense point cloud data subsetsSee at least: “The first stage removes the drift points using a small threshold.” (page 2) and “After the non-iterative small threshold denoising of the 3D point cloud, we get P″ ...” (page 4).Rationale: The initial dense-to-sparse conversion concept was already relied upon in establishing Claim 12. It is repeated here only as antecedent context for the new Claim 13 limitations concerning what is done after the sparse subsets have already been generated. The newly added limitations begin with the second-stage evaluation of those sparse subsets.
into sparse point cloud data subsetsSee at least: “After the non-iterative small threshold denoising of the 3D point cloud, we get P″ ...” (page 4).Rationale: Zhou expressly discloses a post-first-stage point-cloud representation that is sparser than the original. Thus, Zhou supports the antecedent sparse-subset context on which the second-stage evaluation is carried out.
to generate the point cloud dataSee at least: “The remaining drift points are continuously deleted by the non-iterative large threshold denoising algorithm.” (page 4).Rationale: Zhou expressly discloses generation of a further-refined operative point-cloud representation after threshold-based reduction. Thus, Zhou supports the use of the sparse subsets as the point-cloud data subjected to additional evaluation.
comprises:See at least: “In this study, a non-iterative dual threshold framework for 3D point cloud denoising is proposed.” (page 2).Rationale: Zhou expressly discloses an open-ended, staged point-cloud processing workflow, thereby supporting the recited transitional phrase for the further sparse-subset processing steps.
determining whether a second total amount of dataSee at least: “Finally, we calculate the small threshold of the 3D point cloud ... where Nsum is the number of all points in P′.” (page 4) and “The first stage removes the drift points using a small threshold. The second stage removes the drift points using a large threshold.” (page 2).Rationale: Zhou expressly discloses threshold-based point-cloud processing and expressly uses the number of all points in the point cloud as part of the threshold framework. One of ordinary skill in the art would have understood this as evaluating the amount of point-cloud data to determine whether further point-cloud reduction should proceed. Accordingly, the claimed determination of a second total amount of data is expressly suggested by Zhou’s point-count-based threshold processing.
of respective sparse point cloud data subsetsSee at least: “After the non-iterative small threshold denoising of the 3D point cloud, we get P″ ...” (page 4) and “The second stage removes the drift points using a large threshold.” (page 2).Rationale: Zhou expressly discloses that, after a first-stage reduction, a remaining point-cloud representation is subjected to a second threshold stage. One of ordinary skill in the art would have understood the post-first-stage representation to be a sparser subset of the original point cloud on which the second-stage decision is made.
exceeds the predetermined threshold,See at least: “The first stage removes the drift points using a small threshold. The second stage removes the drift points using a large threshold.” (page 2) and “If there is only one point in the Vi,j, it is identified as a drift point; otherwise the points are regarded as non-noise points.” (page 4).Rationale: Zhou expressly discloses predetermined thresholds governing whether point-cloud data is retained or removed in a two-stage denoising process. Thus, Zhou directly supports the claimed threshold-comparison concept applied to already sparse subsets.
wherein when the second total amount of data does not exceed the predetermined threshold,See at least: “If there is only one point in the Vi,j, it is identified as a drift point; otherwise the points are regarded as non-noise points.” (page 4).Rationale: Zhou expressly discloses a threshold-governed branch in which points are either removed or retained. One of ordinary skill in the art would have understood that where the post-conversion sparse data does not exceed the operative threshold, the existing sparse subset may simply be retained for downstream use.
taking the respective sparse point cloud data subsets as the point cloud data,See at least: “We can then retain the non-noise points and delete the drift points.” (page 4).Rationale: Zhou expressly discloses retaining the point-cloud data that survives threshold-based filtering. Accordingly, Zhou teaches the claimed branch in which the sparse subsets are taken as the operative point-cloud data.
and when the second total amount of data exceeds the predetermined threshold,See at least: “The second stage removes the drift points using a large threshold.” (page 2) and “The remaining drift points are continuously deleted by the non-iterative large threshold denoising algorithm.” (page 4).Rationale: Zhou expressly teaches that when the reduced representation still warrants further processing, additional threshold-based reduction is applied. Thus, Zhou supports the claimed second conditional branch when the second amount of data still exceeds the threshold.
Zhao further teaches or renders obvious the road/intersection-specific treatment of the already sparse subsets:
manually sampling the sparse point cloud data subset for each roadSee at least: “A probe-based sampling method is then employed to sample the fused road boundaries.” (page 8–9) and “the sampled nodes in the proposed method are not included in the optimization” (page 8).Rationale: Zhao expressly discloses sampling road-boundary data after fusion. Zhao does not expressly recite that the sampling is “manual.” However, one of ordinary skill in the art would have understood that, where an automatically produced sparse road subset still remains too large after threshold-based reduction, manual or user-directed sampling of the road subset would have been a routine and predictable implementation option for further controlling road data density in a device-based workflow.
at a predetermined interval,See at least: “The key difference is that we sample the nodes evenly in the referenced polyline...” (page 7) and “A probe-based sampling method is then employed to sample the fused road boundaries.” (page 8–9).Rationale: Zhao expressly teaches even sampling of nodes and probe-based sampling of fused road boundaries. One of ordinary skill in the art would have understood even sampling to correspond to sampling at a predetermined interval. Accordingly, this interval-based sampling aspect is at least rendered obvious by Zhao.
and taking the sparse point cloud data subset for each intersectionSee at least: “Figure 11c-I, II, and VI show successfully mapped intersections” (page 13) and “the global structure of the map was not taken into consideration, which is crucial for the vectorization of complex intersections.” (page 8).Rationale: Zhao expressly emphasizes the importance of preserving detailed structure at intersections and expressly discusses successful mapped intersections. One of ordinary skill in the art would have understood from these teachings that intersection subsets are significant structural components that may appropriately be retained without the same level of reduction applied to road-only segments.
and the manually sampled sparse point cloud data subset for each roadSee at least: “we sample the nodes evenly in the referenced polyline” (page 7) and “A probe-based sampling method is then employed to sample the fused road boundaries.” (page 8–9).Rationale: Zhao expressly teaches sampling road-boundary representations and even node sampling. Thus, Zhao teaches the road-specific sampled subset aspect of the claim, at least as rendered obvious when applied after the second threshold evaluation taught by Zhou.
as the point cloud data.See at least: “The simplified representation can be reduced to merely averaging 6% of the points ... which is both lightweight and beneficial to the following matching process.” (page 7) and “the matching with simplified-LVMs outperforms raw-LVM-based matching ... in both accuracy and efficiency” (page 11).Rationale: Zhao expressly teaches using the reduced/sampled representation as the operative representation for downstream matching. Accordingly, Zhao teaches taking the retained intersection subset together with the sampled road subset as the point-cloud data used for subsequent processing.
Zhao and Zhou together teach or render obvious the new Claim 13 limitations without remapping the already-established Claim 12 conversion step. Zhou supplies the second-threshold evaluation and threshold-based retain/further-reduce logic for the already sparse subsets, while Zhao supplies the road-specific sampling, even-interval sampling concept, intersection preservation rationale, and use of the resulting reduced representation for subsequent matching in the device-based context.
Motivation to Combine Luo, Flint, Zhao, and Zhou
Therefore, given the teachings as a whole, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having Luo, Flint, Zhao, and Zhou before them, to modify the Luo/Flint/Zhao combination used for Claims 9, 11, and 12 so that, after the dense road- and intersection-specific point-cloud subsets have already been converted into sparse point-cloud subsets for the device, those already sparse subsets are subjected to a second threshold-based data-volume evaluation as taught by Zhou, and, if the remaining sparse road-segment data still exceeds the acceptable threshold, the road subsets are further sampled at an even, predetermined interval while retaining the intersection subsets as taught by Zhao, because Zhou teaches a dual-threshold point-cloud reduction framework based on point-cloud amount and threshold-driven retention/further-reduction logic, Zhao teaches road-boundary sampling and even node spacing together with explicit preservation of important intersection geometry, Luo provides the overall device-based point-cloud/global-map localization framework in which the resulting point-cloud data is used for map matching and positioning, and Flint remains part of the parent-claim combination for the IMU-based updating aspect of Claim 9. One of ordinary skill in the art would have made such modification to further control data volume and noise after the initial dense-to-sparse conversion while preserving important intersection structure and maintaining lightweight road-segment data for matching, with the predictable benefit of improved efficiency, storage economy, and matching performance. The use of a second threshold and predetermined-interval road sampling would have been a routine and predictable design choice in view of Zhou’s threshold-driven point-cloud reduction and Zhao’s express even-sampling disclosures. Such a modification would not alter the principle of operation of any reference or render any reference unsatisfactory for its intended purpose. Therefore, after combining the teachings of Luo, Flint, Zhao, and Zhou, Claim 13 is rendered obvious by the combination of Luo in view of Flint, Zhao, and Zhou. In particular, Luo teaches the underlying device-based point-cloud/global-map localization framework, Flint remains part of the parent-claim combination for the IMU-based updating aspect of Claim 9, Zhao teaches road/intersection-specific subset generation together with road-boundary sampling, even node spacing, and use of reduced representations for matching, and Zhou teaches a second threshold-based point-cloud reduction framework based on point-cloud amount and threshold-driven retention/further-reduction logic. The remaining aspects concerning explicit manual sampling and retaining intersection subsets while further sampling road subsets would have been obvious to one of ordinary skill in the art as routine implementation choices in view of Zhao’s road-specific sampling disclosures, Zhao’s emphasis on intersection structure, and Zhou’s disclosed multi-threshold point-cloud reduction workflow.
Claims 6, 14, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Luo, in view of Flint, in view of Zhao, and Farshidi (Robust sequential view planning for object recognition using multiple cameras).
Regarding Claim 6,
The combination of Luo, Flint, and Zhao establishes the method of Claim 3, which is the basis for Claim 6.
Luo teaches or renders obvious:
and whereinSee at least: “constructing at least one 3D submap comprises: obtaining images from a camera...” ([0013]).Rationale: The reference expressly discloses a camera-based image-acquisition step within the localization process. Thus, the additional “and wherein” clause properly introduces further camera-based image-processing detail.
capturing at least one image frame of a surrounding environment of the vehicle within the scenario through the camera unitSee at least: “constructing at least one 3D submap comprises: obtaining images from a camera...” ([0013]) and “images of the environment are captured by the camera in approximately 30 Hz.” ([0068]).Rationale: The reference expressly discloses capturing images of the surrounding environment using a camera as part of the localization workflow. One of ordinary skill in the art would have understood such captured images to correspond to image frames obtained through the camera unit.
Luo teaches the baseline vehicle-localization framework and the baseline camera-image-capture portion of Claim 6. Luo does not expressly disclose the plurality of cameras, the multiway image capture at different angles, the frame-by-frame feature extraction across multiple ways, the confidence calculation, or the preferred-frame selection. Those limitations remain outstanding after Luo.
Flint teaches:
and extracting the plurality of feature points from the at least one image frameSee at least: “Images recorded by the image sensor 102 and processed by the module 106 may also be referred to as ‘frames.’” ([0031]) and “The pre-processing module 106 performs feature tracking within the recorded frames...” ([0031]).Rationale: Flint expressly discloses frames and expressly discloses feature tracking within those frames. Thus, Flint teaches extracting or tracking feature points from image frames.
comprises:See at least: “The pre-processing module 106 generates feature tracks by identifying one or more image features in a first frame and then matching those one or more image features...” ([0031]).Rationale: Flint expressly discloses an open-ended feature-processing workflow performed on frames. Thus, the recited transitional phrase is supported for the additional frame-processing operations.
extracting feature points from each way of image framesSee at least: “The pre-processing module 106 calculates ‘feature tracks,’ in which a feature track is a sequence of two-dimensional points representing the locations of a single feature tracked across two or more frames...” ([0031]).Rationale: Flint expressly discloses identifying image features in frames and tracking them across frames. One of ordinary skill in the art would have understood this to correspond to extracting feature points from each image set once multiple ways of image frames are available.
in the multiway of image frames;See at least: “the electronic computing system may further include... an image detection unit ... configured to receive multiple images captured by the image detection unit and derive the image data from multiple images” ([0008]) and “for each window, the multiple first variables represent 3D positions and/or orientations of image features across multiple images” ([0010]).Rationale: Flint expressly teaches processing image features across multiple images. Thus, Flint teaches the multi-image context in which feature points are extracted once the multiway image set is provided.
selecting a preferred frame from the multiway of image framesSee at least: “The pre-processing module 106 also is configured to select a subset of the frames received from the image sensor 102 as ‘keyframes.’” ([0032]) and “A frame may be designated as a keyframe by the module 106 based on one or more parameters, such as a desired keyframe frequency, or statistics related to the number of image features identified in a current frame or matched in a pair of frames.” ([0032]).Rationale: Flint expressly teaches selecting certain frames from a larger set of frames as keyframes and further teaches that the selection may be based on feature-related statistics. This directly supports the claimed concept of selecting a preferred frame from multiple frames based on feature-related criteria.
taking the feature points in the preferred frameSee at least: “the feature locations associated with FALSE elements ... will be ignored, whereas feature locations associated with TRUE elements ... will be used for further calculations.” ([0034]).Rationale: Flint expressly teaches using feature locations from selected/retained frames for subsequent calculations while ignoring others. Thus, Flint teaches taking the feature points in the selected preferred frame for downstream use.
as the plurality of feature points.See at least: “The pre-processing module 106 generates feature tracks by identifying one or more image features in a first frame...” ([0031]) and “feature locations associated with TRUE elements ... will be used for further calculations.” ([0034]).Rationale: Flint expressly teaches identifying image features in frames and using retained feature locations for further calculations. Accordingly, Flint teaches taking the feature points from the selected frame as the operative plurality of feature points.
After Luo, Flint, and Zhao, the following limitations are not explicitly disclosed: the camera unit includes a plurality of cameras; arranged around the vehicle; capturing multiway of image frames of the surrounding environment of the vehicle; at different angles; through the plurality of cameras;; calculating confidences of the extracted feature points; from each way of image frames;; and according to the confidences of each way of image frames;
Farshidi teaches:
the camera unit includes a plurality of camerasSee at least: “While prior relevant research in active object recognition/pose estimation has mostly focused on single-camera systems, we propose two multi-camera solutions...” and “multiple cameras simultaneously acquire images from different view angles...” (page 1).See also: “We have developed an active two-camera vision system...” (page 3).Rationale: Farshidi expressly discloses a plurality of cameras, specifically a two-camera system, used together for image acquisition. Thus, Farshidi teaches the plural-camera aspect of the claim.
arranged around the vehicle,See at least: “multiple cameras simultaneously acquire images from different view angles...” (page 1) and “The second unit moves along the y-axis on a linear track on the side of the workspace.” (page 3).Rationale: Farshidi does not expressly recite cameras arranged around a vehicle. However, Farshidi expressly teaches a plural-camera arrangement positioned to obtain different views of the target from different surrounding locations. In view of Luo’s autonomous-vehicle localization context, one of ordinary skill in the art would have found it obvious to arrange the plural cameras around the vehicle so as to obtain surrounding views for localization, because doing so is the predictable vehicle implementation of Farshidi’s multi-angle plural-camera concept.
capturing multiway of image frames of the surrounding environment of the vehicleSee at least: “multiple cameras simultaneously acquire images from different view angles...” (page 1) and “Such scenarios can also motivate the use of multiple cameras which can provide simultaneous views of the object from different viewing angles.” (page 8).Rationale: Farshidi expressly discloses simultaneous acquisition of multiple images from different viewpoints. One of ordinary skill in the art would have understood such simultaneous multiple views to correspond to capturing a multiway set of image frames.
at different anglesSee at least: “multiple cameras simultaneously acquire images from different view angles...” (page 1).Rationale: Farshidi expressly teaches acquisition of images at different view angles.
through the plurality of cameras;See at least: “we propose two multi-camera solutions...” and “multiple cameras simultaneously acquire images...” (page 1).Rationale: Farshidi expressly teaches acquiring the multi-angle images through a plurality of cameras.
calculating confidences of the extracted feature pointsSee at least: “the recognition algorithms attempt to classify the object, if its identity/pose can be determined with a high confidence level. Otherwise, the algorithms would compute the next most informative camera positions...” (page 1) and “Two performance indices are used to quantify the quality of observations...” (page 3).Rationale: Farshidi does not expressly recite computing a confidence value for each extracted feature point in those exact words. However, Farshidi expressly teaches evaluating confidence level and quantifying observation quality for different views. One of ordinary skill in the art would have understood that, in a multi-view feature-selection workflow, confidence values can be assigned to feature information extracted from each view as a routine way of quantifying which view is more informative.
from each way of image frames;See at least: “multiple cameras simultaneously acquire images from different view angles...” and “The camera positions at each recognition step are selected based on two statistical metrics quantifying the quality of the observations...” (page 1).Rationale: Farshidi expressly teaches evaluating observations obtained from different camera views. Thus, applying a confidence or quality assessment to each way of image frames is at least rendered obvious by Farshidi’s per-view observation-quality framework.
according to the confidences of each way of image frames; andSee at least: “the recognition algorithms attempt to classify the object, if its identity/pose can be determined with a high confidence level. Otherwise, the algorithms would compute the next most informative camera positions...” (page 1) and “The camera positions at each recognition step are selected based on two statistical metrics quantifying the quality of the observations...” (page 1).Rationale: Farshidi expressly teaches confidence- and quality-based evaluation of different views and selection of camera positions based on those measures. In view of Flint’s frame-selection teaching, one of ordinary skill in the art would have found it obvious to select a preferred frame according to confidence values associated with each way of image frames.
Motivation to Combine Luo, Flint, Zhao, and Farshidi
Therefore, given the teachings as a whole, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having Luo, Flint, Zhao, and Farshidi before them, to modify the Luo/Flint/Zhao combination used for Claims 1 and 3 so that the camera unit includes a plurality of cameras arranged to acquire multiple image frames of the surrounding environment from different angles, and so that feature points are extracted from the multiway images, observation-quality or confidence values are evaluated for the feature information from each view, and a preferred frame is selected for use in downstream localization, because Luo already discloses the vehicle-localization framework in which camera-derived feature points are used for localization, Flint teaches extracting feature points from frames and selecting frames based on feature-related statistics, Zhao remains part of the parent-claim combination for the Claim 3 fused-map limitations, and Farshidi teaches simultaneous multi-camera image acquisition from different viewing angles together with confidence- and quality-based evaluation of different observations. One of ordinary skill in the art would have made such modification to improve viewpoint coverage, reduce ambiguity, and select the most informative camera view for localization, with the predictable benefit of more reliable feature-point extraction and improved localization performance in complex road and intersection environments. Such a modification would merely apply known multi-camera observation-selection techniques and known frame/feature-selection techniques to a known vehicle-localization framework and would not alter the principle of operation of any reference or render any reference unsatisfactory for its intended purpose. Therefore, after combining the teachings of Luo, Flint, Zhao, and Farshidi, Claim 6 is rendered obvious by the combination of Luo in view of Flint, Zhao, and Farshidi. In particular, Luo teaches the underlying vehicle-localization framework and baseline camera-based image capture, Flint teaches extraction of feature points from frames and frame selection based on feature-related statistics, Zhao remains part of the parent-claim combination for the Claim 3 fused-map limitations, and Farshidi teaches a plurality of cameras simultaneously acquiring images from different viewing angles together with confidence- and observation-quality-based evaluation of different views. The remaining aspects concerning explicitly arranging the plurality of cameras around the vehicle would have been obvious to one of ordinary skill in the art in view of Farshidi’s multi-angle plural-camera architecture and the predictable adaptation of that architecture to Luo’s autonomous-vehicle setting.
Regarding Claim 14,
The combination of Luo, Flint, and Zhao establishes the device of Claim 11, which is the basis for Claim 14.
Luo discloses or renders obvious:
whereinSee at least: “constructing at least one 3D submap comprises: obtaining images from a camera...” ([0013]).Rationale: The reference expressly discloses a camera-based image-acquisition step within the localization process. Thus, the additional “and wherein” clause properly introduces further camera-based image-processing detail.
capturing at least one image frame of a surrounding environment of the vehicle within the scenario through the camera unitSee at least: “constructing at least one 3D submap comprises: obtaining images from a camera...” ([0013]) and “images of the environment are captured by the camera in approximately 30 Hz.” ([0068]).Rationale: The reference expressly discloses capturing images of the surrounding environment using a camera as part of the localization workflow. One of ordinary skill in the art would have understood such captured images to correspond to image frames obtained through the camera unit.
Luo teaches the baseline device-localization framework and the baseline camera-image-capture portion of Claim 14. Luo does not expressly disclose the plurality of cameras, the multiway image capture at different angles, the frame-by-frame feature extraction across multiple ways, the confidence calculation, or the preferred-frame selection. Those limitations remain outstanding after Luo.
Flint teaches:
and extracting the plurality of feature points from the at least one image frameSee at least: “Images recorded by the image sensor 102 and processed by the module 106 may also be referred to as ‘frames.’” ([0031]) and “the pre-processing module 106 performs feature tracking within the recorded frames...” ([0031]).Rationale: Flint expressly discloses frames and expressly discloses feature tracking within those frames. Thus, Flint teaches extracting or tracking feature points from image frames.
comprises:See at least: “The pre-processing module 106 generates feature tracks by identifying one or more image features in a first frame and then matching those one or more image features with one or more corresponding image features in consecutive frames.” ([0031]).Rationale: Flint expressly discloses an open-ended feature-processing workflow performed on frames. Thus, the recited transitional phrase is supported for the additional frame-processing operations.
extracting feature points from each way of image framesSee at least: “The pre-processing module 106 calculates ‘feature tracks,’ in which a feature track is a sequence of two-dimensional points representing the locations of a single feature tracked across two or more frames...” ([0031]).Rationale: Flint expressly discloses identifying image features in frames and tracking them across frames. One of ordinary skill in the art would have understood this to correspond to extracting feature points from each image set once multiple ways of image frames are available.
in the multiway of image frames;See at least: “the electronic computing system may further include... an image detection unit ... configured to receive multiple images captured by the image detection unit and derive the image data from multiple images” ([0008]) and “for each window, the multiple first variables represent 3D positions and/or orientations of image features across multiple images” ([0010]).Rationale: Flint expressly teaches use of multiple images and image features across multiple images. Thus, Flint teaches the multi-image context in which feature points are extracted once the multiway image set is provided.
selecting a preferred frame from the multiway of image framesSee at least: “The pre-processing module 106 also is configured to select a subset of the frames received from the image sensor 102 as ‘keyframes.’” ([0032]) and “A frame may be designated as a keyframe by the module 106 based on one or more parameters, such as a desired keyframe frequency, or statistics related to the number of image features identified in a current frame or matched in a pair of frames.” ([0032]).Rationale: Flint expressly teaches selecting certain frames from a larger set of frames as keyframes and further teaches that the selection may be based on feature-related statistics. This directly supports the claimed concept of selecting a preferred frame from multiple frames based on feature-related criteria.
taking the feature points in the preferred frameSee at least: “the feature locations associated with FALSE elements ... will be ignored, whereas feature locations associated with TRUE elements ... will be used for further calculations.” ([0034]).Rationale: Flint expressly teaches using feature locations from selected or retained frames for subsequent calculations while ignoring others. Thus, Flint teaches taking the feature points in the selected preferred frame for downstream use.
as the plurality of feature points.See at least: “The pre-processing module 106 generates feature tracks by identifying one or more image features in a first frame...” ([0031]) and “feature locations associated with TRUE elements ... will be used for further calculations.” ([0034]).Rationale: Flint expressly teaches identifying image features in frames and using retained feature locations for further calculations. Accordingly, Flint teaches taking the feature points from the selected frame as the operative plurality of feature points.
After Luo, Flint, and Zhao, the following limitations are not explicitly disclosed: the camera unit includes a plurality of cameras; arranged around the vehicle; capturing multiway of image frames of the surrounding environment of the vehicle; at different angles; through the plurality of cameras;; calculating confidences of the extracted feature points; from each way of image frames;; and according to the confidences of each way of image frames;
Farshidi discloses:
the camera unit includes a plurality of camerasSee at least: “While prior relevant research in active object recognition/pose estimation has mostly focused on single-camera systems, we propose two multi-camera solutions...” and “multiple cameras simultaneously acquire images from different view angles...” (page 1).See also: “We have developed an active two-camera vision system...” (page 3).Rationale: Farshidi expressly discloses a plurality of cameras, specifically a two-camera system, used together for image acquisition. Thus, Farshidi teaches the plural-camera aspect of the claim.
arranged around the vehicle,See at least: “multiple cameras simultaneously acquire images from different view angles...” (page 1, Abstract) and “The first pan/tilt unit is mounted on an x–y gantry frame. The second unit moves along the y-axis on a linear track on the side of the workspace.” (page 3).Rationale: Farshidi does not expressly recite cameras arranged around a vehicle. However, Farshidi expressly teaches a plural-camera arrangement positioned to obtain different views of the target from different surrounding locations. In view of Luo’s autonomous-vehicle localization context, one of ordinary skill in the art would have found it obvious to arrange the plural cameras around the vehicle so as to obtain surrounding views for localization, because doing so is the predictable vehicle implementation of Farshidi’s multi-angle plural-camera concept.
capturing multiway of image frames of the surrounding environment of the vehicleSee at least: “multiple cameras simultaneously acquire images from different view angles...” (page 1) and “Such scenarios can also motivate the use of multiple cameras which can provide simultaneous views of the object from different viewing angles.” (page 8).Rationale: Farshidi expressly discloses simultaneous acquisition of multiple images from different viewpoints. One of ordinary skill in the art would have understood such simultaneous multiple views to correspond to capturing a multiway set of image frames.
at different anglesSee at least: “multiple cameras simultaneously acquire images from different view angles...” (page 1).Rationale: Farshidi expressly teaches acquisition of images at different view angles.
through the plurality of cameras;See at least: “we propose two multi-camera solutions...” and “multiple cameras simultaneously acquire images from different view angles...” (page 1, Abstract).Rationale: Farshidi expressly teaches acquiring the multi-angle images through a plurality of cameras.
calculating confidences of the extracted feature pointsSee at least: “the recognition algorithms attempt to classify the object, if its identity/pose can be determined with a high confidence level. Otherwise, the algorithms would compute the next most informative camera positions...” (page 1, Abstract) and “Two performance indices are used to quantify the quality of observations...” (page 3).Rationale: Farshidi does not expressly recite computing a confidence value for each extracted feature point in those exact words. However, Farshidi expressly teaches evaluating confidence level and quantifying observation quality for different views. One of ordinary skill in the art would have understood that, in a multi-view feature-selection workflow, confidence values can be assigned to feature information extracted from each view as a routine way of quantifying which view is more informative.
from each way of image frames;See at least: “The observation vector g includes the eigenspace coefficients corresponding to the images obtained from the two cameras, i.e. g = [g1 g2]” (page 4) and “the cameras’ observations ... are conditionally independent of each other” (page 4).Rationale: Farshidi expressly teaches separate observations from the different camera views and evaluates those observations within a probabilistic framework. Thus, applying a confidence or quality assessment to each way of image frames is at least rendered obvious by Farshidi’s per-view observation framework.
according to the confidences of each way of image frames;See at least: “the recognition algorithms attempt to classify the object, if its identity/pose can be determined with a high confidence level. Otherwise, the algorithms would compute the next most informative camera positions...” (page 1) and “Two performance indices are used to quantify the quality of observations in the context of state estimation and subsequently choose the next best positions of the cameras.” (page 3).Rationale: Farshidi expressly teaches confidence- and quality-based evaluation of different views and selection of camera positions based on those measures. In view of Flint’s frame-selection teaching, one of ordinary skill in the art would have found it obvious to select a preferred frame according to confidence values associated with each way of image frames.
Motivation to Combine Luo, Flint, Zhao, and Farshidi
Therefore, given the teachings as a whole, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having Luo, Flint, Zhao, and Farshidi before them, to modify the Luo/Flint/Zhao combination used for Claims 9 and 11 so that the device’s camera unit includes a plurality of cameras arranged to acquire multiple image frames of the surrounding environment from different angles, and so that feature points are extracted from the multiway images, observation-quality or confidence values are evaluated for the feature information from each view, and a preferred frame is selected for use in downstream localization, because Luo already discloses the device-based vehicle-localization framework in which camera-derived feature points are used for localization, Flint teaches extracting feature points from frames and selecting frames based on feature-related statistics, Zhao remains part of the parent-claim combination for the Claim 11 fused-map limitations, and Farshidi teaches simultaneous multi-camera image acquisition from different viewing angles together with confidence- and quality-based evaluation of different observations. One of ordinary skill in the art would have made such modification to improve viewpoint coverage, reduce ambiguity, and select the most informative camera view for localization, with the predictable benefit of more reliable feature-point extraction and improved localization performance in complex road and intersection environments. Such a modification would merely apply known multi-camera observation-selection techniques and known frame/feature-selection techniques to a known device-based vehicle-localization framework and would not alter the principle of operation of any reference or render any reference unsatisfactory for its intended purpose. Therefore, after combining the teachings of Luo, Flint, Zhao, and Farshidi, Claim 14 is rendered obvious by the combination of Luo in view of Flint, Zhao, and Farshidi. In particular, Luo teaches the underlying device-based vehicle-localization framework and baseline camera-based image capture, Flint teaches extraction of feature points from frames and frame selection based on feature-related statistics, Zhao remains part of the parent-claim combination for the Claim 11 fused-map limitations, and Farshidi teaches a plurality of cameras simultaneously acquiring images from different viewing angles together with confidence- and observation-quality-based evaluation of different views. The remaining aspects concerning explicitly arranging the plurality of cameras around the vehicle would have been obvious to one of ordinary skill in the art in view of Farshidi’s multi-angle plural-camera architecture and the predictable adaptation of that architecture to Luo’s autonomous-vehicle setting.
Regarding Claim 20,
The combination of Luo, Flint, and Zhao establishes the non-transitory computer-readable storage medium of Claim 19, which is the basis for Claim 20.
Luo renders obvious:
whereinSee at least: “The one or more programs comprise instructions, which when executed by a computing device, cause the computing device to perform...” ([0010]).Rationale: The reference expressly discloses that the stored instructions cause a computing device to carry out the localization method. Thus, the “wherein” clause properly introduces additional conditions on that already established storage-medium-implemented method.
capturing at least one image frame of a surrounding environment of the vehicle within the scenario through the camera unitSee at least: “constructing at least one 3D submap comprises: obtaining images from a camera...” ([0013]) and “images of the environment are captured by the camera in approximately 30 Hz.” ([0068]).Rationale: The reference expressly discloses capturing images of the surrounding environment using a camera as part of the localization workflow. One of ordinary skill in the art would have understood such captured images to correspond to image frames obtained through the camera unit.
Luo teaches the baseline storage-medium vehicle-localization framework and the baseline camera-image-capture portion of Claim 20. Luo does not expressly disclose the plurality of cameras, the multiway image capture at different angles, the frame-by-frame feature extraction across multiple ways, the confidence calculation, or the preferred-frame selection. Those limitations remain outstanding after Luo.
Flint discloses:
and extracting the plurality of feature points from the at least one image frameSee at least: “Images recorded by the image sensor 102 and processed by the module 106 may also be referred to as ‘frames.’” ([0031]) and “The pre-processing module 106 performs feature tracking within the recorded frames...” ([0031]).Rationale: Flint expressly discloses frames and expressly discloses feature tracking within those frames. Thus, Flint teaches extracting or tracking feature points from image frames.
comprises:See at least: “The pre-processing module 106 generates feature tracks by identifying one or more image features in a first frame and then matching those one or more image features with one or more corresponding image features in consecutive frames.” ([0031]).Rationale: Flint expressly discloses an open-ended feature-processing workflow performed on frames. Thus, the recited transitional phrase is supported for the additional frame-processing operations.
extracting feature points from each way of image framesSee at least: “The pre-processing module 106 calculates ‘feature tracks,’ in which a feature track is a sequence of two-dimensional points representing the locations of a single feature tracked across two or more frames...” ([0031]).Rationale: Flint expressly discloses identifying image features in frames and tracking them across frames. One of ordinary skill in the art would have understood this to correspond to extracting feature points from each image set once multiple ways of image frames are available.
in the multiway of image frames;See at least: “the electronic computing system may further include... an image detection unit ... configured to receive multiple images captured by the image detection unit and derive the image data from multiple images” ([0008]) and “for each window, the multiple first variables represent 3D positions and/or orientations of image features across multiple images” ([0010]).Rationale: Flint expressly teaches use of multiple images and image features across multiple images. Thus, Flint teaches the multi-image context in which feature points are extracted once the multiway image set is provided.
selecting a preferred frame from the multiway of image framesSee at least: “The pre-processing module 106 also is configured to select a subset of the frames received from the image sensor 102 as ‘keyframes.’” ([0032]) and “A frame may be designated as a keyframe by the module 106 based on one or more parameters, such as a desired keyframe frequency, or statistics related to the number of image features identified in a current frame or matched in a pair of frames.” ([0032]).Rationale: Flint expressly teaches selecting certain frames from a larger set of frames as keyframes and further teaches that the selection may be based on feature-related statistics. This directly supports the claimed concept of selecting a preferred frame from multiple frames based on image-feature-related criteria.
taking the feature points in the preferred frameSee at least: “the feature locations associated with FALSE elements ... will be ignored, whereas feature locations associated with TRUE elements ... will be used for further calculations.” ([0034]).Rationale: Flint expressly teaches using feature locations from selected or retained frames for subsequent calculations while ignoring others. Thus, Flint teaches taking the feature points in the selected preferred frame for downstream use.
as the plurality of feature points.See at least: “The pre-processing module 106 generates feature tracks by identifying one or more image features in a first frame...” ([0031]) and “feature locations associated with TRUE elements ... will be used for further calculations.” ([0034]).Rationale: Flint expressly teaches identifying image features in frames and using retained feature locations for further calculations. Accordingly, Flint teaches taking the feature points from the selected frame as the operative plurality of feature points.
After Luo, Flint, and Zhao, the following limitations are not explicitly disclosed: the camera unit includes a plurality of cameras; arranged around the vehicle; capturing multiway of image frames of the surrounding environment of the vehicle; at different angles; through the plurality of cameras;; calculating confidences of the extracted feature points; from each way of image frames;; and according to the confidences of each way of image frames;
Farshidi discloses:
the camera unit includes a plurality of camerasSee at least: “While prior relevant research in active object recognition/pose estimation has mostly focused on single-camera systems, we propose two multi-camera solutions...” and “multiple cameras simultaneously acquire images from different view angles...” (page 1).See also: “We have developed an active two-camera vision system...” (page 3).Rationale: Farshidi expressly discloses a plurality of cameras, specifically a two-camera system, used together for image acquisition. Thus, Farshidi teaches the plural-camera aspect of the claim.
arranged around the vehicle,See at least: “multiple cameras simultaneously acquire images from different view angles...” (page 1) and “The first pan/tilt unit is mounted on an x–y gantry frame. The second unit moves along the y-axis on a linear track on the side of the workspace.” (page 3).Rationale: Farshidi does not expressly recite cameras arranged around a vehicle. However, Farshidi expressly teaches a plural-camera arrangement positioned to obtain different views of the target from different surrounding locations. In view of Luo’s autonomous-vehicle localization context, one of ordinary skill in the art would have found it obvious to arrange the plural cameras around the vehicle so as to obtain surrounding views for localization, because doing so is the predictable vehicle implementation of Farshidi’s multi-angle plural-camera concept.
capturing multiway of image frames of the surrounding environment of the vehicleSee at least: “multiple cameras simultaneously acquire images from different view angles...” (page 1) and “Such scenarios can also motivate the use of multiple cameras which can provide simultaneous views of the object from different viewing angles.” (page 1078 / page 8 of the article).Rationale: Farshidi expressly discloses simultaneous acquisition of multiple images from different viewpoints. One of ordinary skill in the art would have understood such simultaneous multiple views to correspond to capturing a multiway set of image frames.
at different anglesSee at least: “multiple cameras simultaneously acquire images from different view angles...” (page 1).Rationale: Farshidi expressly teaches acquisition of images at different view angles.
through the plurality of cameras;See at least: “we propose two multi-camera solutions...” and “multiple cameras simultaneously acquire images from different view angles...” (page 1).Rationale: Farshidi expressly teaches acquiring the multi-angle images through a plurality of cameras.
calculating confidences of the extracted feature pointsSee at least: “the recognition algorithms attempt to classify the object, if its identity/pose can be determined with a high confidence level. Otherwise, the algorithms would compute the next most informative camera positions...” (page 1) and “Two performance indices are used to quantify the quality of observations...” (page 1073 / page 3 of the article).Rationale: Farshidi does not expressly recite computing a confidence value for each extracted feature point in those exact words. However, Farshidi expressly teaches evaluating confidence level and quantifying observation quality for different views. One of ordinary skill in the art would have understood that, in a multi-view feature-selection workflow, confidence values can be assigned to feature information extracted from each view as a routine way of quantifying which view is more informative.
from each way of image frames;See at least: “g = [g1 g2], where g1 and g2 are observation vectors from Camera 1 and 2, respectively” and “We assume that the cameras’ observations ... are conditionally independent of each other.” (page 1075 / page 4 of the article).Rationale: Farshidi expressly teaches separate observations from the different camera views and evaluates those observations within a probabilistic framework. Thus, applying a confidence or quality assessment to each way of image frames is at least rendered obvious by Farshidi’s per-view observation framework.
according to the confidences of each way of image frames; andSee at least: “the recognition algorithms attempt to classify the object, if its identity/pose can be determined with a high confidence level. Otherwise, the algorithms would compute the next most informative camera positions...” (page 1) and “Two performance indices are used to quantify the quality of observations in the context of state estimation and subsequently choose the next best positions of the cameras.” (page 1073 / page 3 of the article).Rationale: Farshidi expressly teaches confidence- and quality-based evaluation of different views and selection of camera positions based on those measures. In view of Flint’s frame-selection teaching, one of ordinary skill in the art would have found it obvious to select a preferred frame according to confidence values associated with each way of image frames.
Motivation to Combine Luo, Flint, Zhao, and Farshidi
Therefore, given the teachings as a whole, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having Luo, Flint, Zhao, and Farshidi before them, to modify the Luo/Flint/Zhao combination used for Claims 17 and 19 so that the storage-medium-implemented localization method uses a camera unit having a plurality of cameras arranged to acquire multiple image frames of the surrounding environment from different angles, and so that feature points are extracted from the multiway images, observation-quality or confidence values are evaluated for the feature information from each view, and a preferred frame is selected for use in downstream localization, because Luo already discloses the storage-medium-based vehicle-localization framework in which camera-derived feature points are used for localization, Flint teaches extracting feature points from frames and selecting frames based on feature-related statistics, Zhao remains part of the parent-claim combination for the Claim 19 fused-map limitations, and Farshidi teaches simultaneous multi-camera image acquisition from different viewing angles together with confidence- and quality-based evaluation of different observations. One of ordinary skill in the art would have made such modification to improve viewpoint coverage, reduce ambiguity, and select the most informative camera view for localization, with the predictable benefit of more reliable feature-point extraction and improved localization performance in complex road and intersection environments. Such a modification would merely apply known multi-camera observation-selection techniques and known frame/feature-selection techniques to a known storage-medium-based vehicle-localization framework and would not alter the principle of operation of any reference or render any reference unsatisfactory for its intended purpose. Therefore, after combining the teachings of Luo, Flint, Zhao, and Farshidi, Claim 20 is rendered obvious by the combination of Luo in view of Flint, Zhao, and Farshidi. In particular, Luo teaches the underlying non-transitory storage-medium vehicle-localization framework and baseline camera-based image capture, Flint teaches extraction of feature points from frames and frame selection based on feature-related statistics, Zhao remains part of the parent-claim combination for the Claim 19 fused-map limitations, and Farshidi teaches a plurality of cameras simultaneously acquiring images from different viewing angles together with confidence- and observation-quality-based evaluation of different views. The remaining aspects concerning explicitly arranging the plurality of cameras around the vehicle would have been obvious to one of ordinary skill in the art in view of Farshidi’s multi-angle plural-camera architecture and the predictable adaptation of that architecture to Luo’s autonomous-vehicle setting.
Claims 7 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Luo, in view of Flint, in view of Zhao, in view of Farshidi, in view of Lin (Autonomous Vehicle Localization with Prior Visual Point Cloud Map Constraints in GNSS-Challenged Environments), and in view of Liang (Fusion of LIDAR Point Clouds and Large Scale Vector Maps for the Reconstruction of 3-D Road Models).
Regarding Claim 7,
The combination of Luo, Flint, Zhao, and Farshidi establishes the method of Claim 6, which is the basis for Claim 7.
Luo teaches or renders obvious:
whereinSee at least: “In another embodiment, before computing matching score, the method further comprises: constructing at least one 3D submap; and constructing a global map.” ([0012]).Rationale: Luo expressly discloses an ordered matching workflow operating on a submap and a global map. Thus, the “wherein” clause properly introduces the more specific matching details now recited.
performing a matching of the plurality of feature pointsSee at least: “the features extracted from the 3D submap are matched against the features extracted from the global map...” ([0071]).Rationale: Luo expressly discloses matching the camera-derived 3D-submap features against the LiDAR-derived global-map features. One of ordinary skill in the art would have understood the camera-derived features to correspond to the claimed plurality of feature points used for matching.
with point cloud data in the point cloud basemapSee at least: “constructing a city-scale 3D map based on the data from the LiDAR, using LiDAR mapping.” ([0014]) and “the unstructured features include sparse 3D points.” ([0017]).Rationale: Luo expressly discloses a LiDAR-based global map and sparse 3D point features. Accordingly, Luo teaches matching against point-cloud-type data in the basemap.
to determine the position of the vehicle within the vector mapSee at least: “Subsequently, in operation 17, location of the 3D submap is iteratively estimated until a distance between corresponding features is minimized.” ([0072]).Rationale: Luo expressly discloses determining the submap location from the matching result. In the already established parent-claim combination, the point-cloud basemap and vector-map framework is present; therefore, Luo’s location estimation corresponds to determining vehicle position within that mapped framework.
according to the result of the matchingSee at least: “computing... matching score...” ([0010]) and “the features extracted from the 3D submap are matched against the features extracted from the global map...” ([0071]).Rationale: Luo expressly teaches that location estimation follows from the feature-matching process. Thus, Luo teaches determining position according to the result of the matching.
comprises:See at least: “before computing matching score, the method further comprises...” ([0011]) and “In another embodiment, before computing matching score, the method further comprises...” ([0012]).Rationale: Luo expressly discloses an open-ended matching workflow with constituent sub-steps. Thus, the transitional phrase is supported.
Luo teaches the core matching-and-position-determination framework of Claim 7. Luo does not expressly disclose the more specific local-part restriction to current-position road/intersection subsets within a predetermined range, nor the explicit fallback from local-part matching to all point-cloud data. Those limitations remain outstanding after Luo.
Zhao teaches:
the part of the point cloud data includes point cloud data subsetsSee at least: “We explored the following: (1) the extraction and vectorization of road structures based on multiframe probabilistic fusion; (2) the efficient vector-based matching between frames of road structures...” (page 2) and “Most of the current methods usually focus on the road boundary detection in a single frame... Therefore, a multiframe fusion is required for robust and precise road structure detection.” (page 4).Rationale: Zhao expressly discloses extracting and matching local road-structure data generated from multiframe fusion. One of ordinary skill in the art would have understood these local fused road-structure portions to constitute subsets of the overall point-cloud/map data.
for the roads and intersectionsSee at least: “The high-precision high-definition (HD) map of the road environment is now recognized as one of the cornerstones for autonomous driving. The reliable mapping of road boundaries, lanes, and other road structures...” (page 2).Rationale: Zhao expressly teaches road-structure mapping, including road boundaries and related road structures, in autonomous-driving environments. In the already established parent-claim context, these local subsets correspond to roads and intersections.
Zhao supplies the road-structure subset context for the local-part matching recited in Claim 7. Zhao does not expressly disclose current-position-based range restriction or fallback to matching against all point-cloud data. Those limitations remain outstanding after Luo, Flint, and Zhao.
Lin teaches:
performing a matching of the plurality of feature pointsSee at least: “the current visual point cloud frame is matched with the candidate sub-map by NDT.” (page 4) and “With reliable initial pose estimation, current visual point cloud frames obtained from a stereo camera is matched with the candidate sub-map...” (page 7).Rationale: Lin expressly discloses matching the current camera-derived point-cloud/frame information against a candidate sub-map. In view of the parent-claim combination, this corresponds to matching the claimed plurality of feature points.
with a part of the point cloud data in the point cloud basemapSee at least: “The main idea was to segment the sub-map from a large point cloud...” (page 5) and “the current visual point cloud frame is matched with the candidate sub-map...” (page 4/page 7).Rationale: Lin expressly discloses segmenting a sub-map from a large prior point cloud and matching the current frame against that candidate sub-map. Thus, Lin teaches matching against a part of the point-cloud basemap rather than the whole map.
corresponding to a current position of the vehicle,See at least: “where Lvechicle is the position of the vehicle...” and “Bcenter is the new center point coordinates of the sub-map.” (page 5).Rationale: Lin expressly discloses that the sub-map is centered and updated based on the vehicle position. Thus, Lin teaches that the matched sub-map corresponds to the current position of the vehicle.
whereinSee at least: “The principle of segmentation is as follows...” followed by the position-based sub-map update rule (page 5).Rationale: Lin expressly discloses a conditional segmentation rule tied to vehicle position. Thus, the clause properly introduces the more specific position/range limitation.
within a predetermined rangeSee at least: “τ is the segmentation threshold, we set it to be 80 m in this paper.” (page 5).Rationale: Lin expressly discloses a predetermined threshold distance used to define when the vehicle is near the sub-map edge and when the sub-map should be reset. One of ordinary skill in the art would have understood this threshold-based sub-map extent to correspond to a predetermined range around the current position.
of the current position of the vehicle,See at least: “where Lvechicle is the position of the vehicle, Bedge is edge of the sub-map, τ is the segmentation threshold...” (page 5).Rationale: Lin expressly ties the sub-map definition to the current vehicle position. Thus, the predetermined range is expressly anchored to the current position of the vehicle.
whereinSee at least: “If |Lvechicle − Bedge| > τ, continue; else, Bcenter = Lvechicle.” (page 5).Rationale: Lin expressly discloses a conditional decision rule for the local sub-map. Thus, the clause properly introduces the succeeding/failing local-matching logic in Claim 7.
when the matching of the plurality of feature points with the part of the point cloud data succeeds,See at least: “If the transformation parameters can make the two scans match well, the probability density of the transformed points in the candidate sub-map will be large.” (page 7).Rationale: Lin expressly discloses the condition in which matching with the candidate sub-map succeeds, i.e., when the scans match well under the NDT metric.
determining the position of the vehicle within the vector mapSee at least: “Finally, the matching result was used to update pose prediction based on the last frame for accurate localization.” (page 1/page 2/page 4).Rationale: Lin expressly discloses using the successful sub-map matching result to update pose prediction for accurate localization. Thus, Lin teaches determining the vehicle position based on successful local-part matching.
according to the result of the matchingSee at least: “Then, the current visual point cloud frame is matched with the candidate sub-map by NDT. Finally, the matching result was used to update pose prediction...” (page 4).Rationale: Lin expressly teaches that pose refinement follows from the matching result. Thus, this limitation is expressly taught.
and when the matching of the plurality of feature points with the part of the point cloud data fails,
See at least: “It is time-consuming to match the visual point cloud generated by a stereo camera with the whole prior visual point cloud map.” (page 5) and “the current visual point cloud frame is matched with the candidate sub-map...” (page 4/page 7).
Rationale: Lin expressly contrasts whole-map matching with candidate-sub-map matching. Although Lin does not expressly recite a failure branch in those exact words, one of ordinary skill in the art would have understood that if matching against the candidate sub-map does not succeed, a predictable fallback is to broaden the search from the local part to the whole prior point-cloud map, especially because Lin expressly identifies whole-map matching as the broader baseline from which sub-map matching is an efficiency improvement.
performing a matching of the plurality of feature pointsSee at least: “It is time-consuming to match the visual point cloud generated by a stereo camera with the whole prior visual point cloud map.” (page 5).Rationale: Lin expressly teaches matching against the whole prior visual point cloud map as the broader matching operation. Thus, Lin supports the fallback full-map matching concept.
with all the point cloud data in the point cloud basemap,See at least: “match the visual point cloud generated by a stereo camera with the whole prior visual point cloud map.” (page 5).Rationale: Lin expressly discloses whole-map matching. One of ordinary skill in the art would have understood the whole prior point cloud map to correspond to all the point-cloud data in the basemap.
Lin teaches the current-position candidate-submap matching, the predetermined-range restriction, successful local matching followed by localization update, and a supportable PHOSITA-obvious fallback from local-part matching to whole-map matching if local matching does not succeed. What remains after Lin is the explicit road/intersection subset content and the explicit mapping relationship between the point-cloud basemap and the vector map.
Liang teaches:
for the roads and intersectionsSee at least: “The roadsides can be paired. The centerlines can be determined...” and “a node is generated by the intersection of two centerlines for the crossroads...” (pages 2–4) and “The test area includes highways, arterial streets, local streets, and traffic islands.” (pages 4–6).Rationale: Liang expressly discloses road networks, road segments, centerlines, and crossroads/intersections. Thus, Liang directly supports the road-and-intersection organization of the local point-cloud subsets.
determining the position of the vehicle within the vector mapSee at least: “we fuse LIDAR point clouds and large scale vector maps to reconstruct the 3-D road models” (page 1) and “the roads can be represented as a parametric model” / “the 3-D road models by flat quadrilaterals ... are presented as ribbons” (pages 1 and 4–7).Rationale: Liang expressly discloses a fused representation in which LiDAR point-cloud information is organized with vector-map geometry. Thus, once the local/submap point-cloud match is successful, the point-cloud result can be used within the vector-map representation to determine position.
and a mapping relationship between the point cloud basemap and the vector map,See at least: “we fuse LIDAR point clouds and large scale vector maps to reconstruct the 3-D road models” (page 1) and “the large scale vector maps used in this investigation record the boundaries of street blocks ... Then we can compute the heights of the road surface from LIDAR point clouds along those centerlines.” (pages 1–2).Rationale: Liang expressly discloses a fused relationship between LiDAR point clouds and vector-map geometry, where the vector-map structure guides and organizes the point-cloud-derived road model. Thus, Liang teaches the claimed mapping relationship between the point cloud basemap and the vector map.
and determining the position of the vehicle within the vector mapSee at least: “the centerlines are connected to form road networks” and “the complete set of the planimetric road networks” (pages 2–6).Rationale: Liang expressly discloses a coherent vectorized road-network representation tied to fused point-cloud content. In view of the already established Luo/Lin localization workflow, one of ordinary skill in the art would have used the full-map match result within that point-cloud/vector-map relationship to determine position in the vector map.
according to the result of the matching of the plurality of feature points with all of the point cloud data.See at least: Lin’s whole-map matching disclosure, “match the visual point cloud ... with the whole prior visual point cloud map” (page 5), together with Liang’s fusion of LiDAR point clouds and vector maps (page 1).Rationale: Lin teaches the full-map matching fallback, and Liang teaches the point-cloud/vector-map relationship needed to interpret the result within the vector map. Accordingly, the combined teachings render obvious determining position in the vector map according to the result of matching with all point-cloud data.
Motivation to Combine Luo, Flint, Zhao, Farshidi, Lin, and Liang
Therefore, given the teachings as a whole, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having Luo, Flint, Zhao, Farshidi, Lin, and Liang before them, to modify the Luo/Flint/Zhao/Farshidi combination so that the matching operation is first performed against a local part of the point-cloud basemap corresponding to the current vehicle position and containing road/intersection subsets within a predetermined range, and so that, if that local matching succeeds, the vehicle position is determined in the vector map using the matching result and the point-cloud/vector-map relationship, but if the local matching does not succeed, the matching is broadened to the whole point-cloud basemap, because Luo already teaches the underlying feature-to-map matching and localization framework, Flint and Farshidi remain part of the parent Claim 6 combination, Zhao teaches road-structure subset organization, Lin teaches current-position-based candidate-submap segmentation and matching with the candidate sub-map rather than the whole map, and Liang teaches the fused relationship between LiDAR point clouds and vector-map road geometry for roads and intersections. One of ordinary skill in the art would have made such modification to improve efficiency by attempting localized matching first while preserving robustness through whole-map fallback if localized matching is unsuccessful, with the predictable benefit of faster matching, more stable localization, and meaningful position determination within the fused point-cloud/vector-map representation. Such a modification would merely combine known local-submap matching, known whole-map matching, and known point-cloud/vector-map fusion techniques in a predictable way and would not alter the principle of operation of any reference or render any reference unsatisfactory for its intended purpose. Therefore, after combining the teachings of Luo, Flint, Zhao, Farshidi, Lin, and Liang, Claim 7 is rendered obvious by the combination of Luo in view of Flint, Zhao, Farshidi, Lin, and Liang. In particular, Luo teaches the underlying feature-to-point-cloud matching and position-determination framework, Flint and Farshidi remain part of the parent Claim 6 combination, Zhao teaches road-structure subset context, Lin teaches current-position candidate-submap matching within a predetermined range and supports the full-map fallback, and Liang teaches the road/intersection organization and the mapping relationship between point-cloud data and vector-map geometry. The remaining aspects concerning the explicit failure-triggered fallback from local-part matching to all point-cloud data would have been obvious to one of ordinary skill in the art in view of Lin’s express contrast between candidate-submap matching and whole-map matching and the predictable need to broaden the search when localized matching is unsuccessful.
Regarding Claim 15,
The combination of Luo, Flint, Zhao, and Farshidi establishes the device of Claim 14, which is the basis for Claim 15.
Luo renders obvious:
whereinSee at least: “In another embodiment, before computing matching score, the method further comprises: constructing at least one 3D submap; and constructing a global map.” ([0012]).Rationale: Luo expressly discloses an ordered matching workflow operating on a submap and a global map. Thus, the “wherein” clause properly introduces the more specific matching details now recited.
performing a matching of the plurality of feature points
See at least: “computing, in response to features from a 3D submap and features from a global map, matching score...” ([0010]) and “the features extracted from the 3D submap are matched against the features extracted from the global map...” ([0071]).Rationale: Luo expressly discloses matching camera-derived submap features against global-map features. In the parent-claim combination, those camera-derived features correspond to the claimed plurality of feature points.
with point cloud data in the point cloud basemap
See at least: “constructing a city-scale 3D map based on the data from the LiDAR, using LiDAR mapping.” ([0014]) and “the unstructured features include sparse 3D points.” ([0017]).Rationale: Luo expressly discloses a LiDAR-based global map and sparse 3D point features. Accordingly, Luo teaches matching against point-cloud-type data in the basemap.
to determine the position of the vehicle within the vector map
See at least: “Subsequently, in operation 17, location of the 3D submap is iteratively estimated until a distance between corresponding features is minimized.” ([0072]).Rationale: Luo expressly discloses determining the submap location from the matching result. In the already established parent-claim combination, the point-cloud basemap and vector-map framework is present; therefore, Luo’s location estimation corresponds to determining vehicle position within that mapped framework.
according to the result of the matching
See at least: “computing... matching score...” ([0010]) and “the features extracted from the 3D submap are matched against the features extracted from the global map...” ([0071]).Rationale: Luo expressly teaches that location estimation follows from the feature-matching process. Thus, Luo teaches determining position according to the result of the matching.
comprises:See at least: “before computing matching score, the method further comprises...” ([0011]) and “In another embodiment, before computing matching score, the method further comprises...” ([0012]).Rationale: Luo expressly discloses an open-ended matching workflow with constituent sub-steps. Thus, the transitional phrase is supported.
Luo teaches the core matching-and-position-determination framework of Claim 15. Luo does not expressly disclose the more specific local-part restriction to current-position road/intersection subsets within a predetermined range, nor the explicit fallback from local-part matching to all point-cloud data. Those limitations remain outstanding after Luo.
Zhao discloses:
the part of the point cloud data includes point cloud data subsets
See at least: “the sparse extraction results can be enhanced which generate the local grid map (LGM) of road boundaries” and “we first propose a local vectorization method...” (page 6).Rationale: Zhao expressly discloses local road-boundary map portions generated from fused local data. One of ordinary skill in the art would have understood these local fused road-structure portions to constitute subsets of the overall point-cloud/map data.
for the roads and intersectionsSee at least: “the global structure of the map was not taken into consideration, which is crucial for the vectorization of complex intersections” and “A probe-based sampling method is then employed to sample the fused road boundaries.” (pages 8–9).Rationale: Zhao expressly discloses road boundaries and expressly emphasizes complex intersections in the map construction workflow. Thus, Zhao directly supports the roads-and-intersections content of the local subsets.
Zhao supplies the road/intersection subset context for the local-part matching recited in Claim 15. Zhao does not expressly disclose current-position-based range restriction or fallback to matching against all point-cloud data. Those limitations remain outstanding after Luo, Flint, and Zhao.
Lin discloses:
performing a matching of the plurality of feature pointsSee at least: “the current visual point cloud frame is matched with the candidate sub-map by NDT.” (page 1 / introduction summary) and “With reliable initial pose estimation, current visual point cloud frames obtained from a stereo camera is matched with the candidate sub-map...” (page 7).Rationale: Lin expressly discloses matching current camera-derived frame data against a candidate sub-map. In view of the parent-claim combination, this corresponds to matching the claimed plurality of feature points.
with a part of the point cloud data in the point cloud basemapSee at least: “The main idea was to segment the sub-map from a large point cloud...” (page 5) and “the current visual point cloud frame is matched with the candidate sub-map...” (page 7).Rationale: Lin expressly discloses segmenting a candidate sub-map from a large prior point cloud and matching against that candidate sub-map. Thus, Lin teaches matching against a part of the point-cloud basemap rather than the whole map.
corresponding to a current position of the vehicle,See at least: “where Lvechicle is the position of the vehicle...” and “Bcenter is the new center point coordinates of the sub-map.” (page 5).Rationale: Lin expressly discloses that the sub-map is centered and updated based on vehicle position. Thus, Lin teaches that the matched sub-map corresponds to the current position of the vehicle.
whereinSee at least: “The principle of segmentation is as follows...” followed by the position-based sub-map update rule (page 5).Rationale: Lin expressly discloses a conditional segmentation rule tied to vehicle position. Thus, the clause properly introduces the more specific position/range limitation.
within a predetermined rangeSee at least: “τ is the segmentation threshold, we set it to be 80 m in this paper.” (page 5) and “In this paper, the threshold is set to 80 m.” (page 16).Rationale: Lin expressly discloses a predetermined threshold distance used to define the local sub-map extent. One of ordinary skill in the art would have understood this threshold-based sub-map extent to correspond to a predetermined range around the current position.
of the current position of the vehicle,See at least: “where Lvechicle is the position of the vehicle, Bedge is edge of the sub-map, τ is the segmentation threshold...” (page 5).Rationale: Lin expressly ties the sub-map definition to the current vehicle position. Thus, the predetermined range is expressly anchored to the current position of the vehicle.
whereinSee at least: “If |Lvechicle − Bedge| > τ, continue else, Bcenter = Lvechicle” (page 5).Rationale: Lin expressly discloses a conditional decision rule for the local sub-map. Thus, the clause properly introduces the succeeding/failing local-matching logic in Claim 15.
when the matching of the plurality of feature points with the part of the point cloud data succeeds,See at least: “If the transformation parameters can make the two scans match well, the probability density of the transformed points in the candidate sub-map will be large.” (page 7).Rationale: Lin expressly discloses the condition in which matching with the candidate sub-map succeeds, i.e., when the scans match well under the NDT metric.
determining the position of the vehicle within the vector mapSee at least: “Finally, the matching result was used to update pose prediction based on the last frame for accurate localization.” (page 1 / introduction summary) and “the current visual point cloud frame is matched with the candidate sub-map ... to update pose prediction based on the last frame for accurate localization.” (page 7).Rationale: Lin expressly discloses using the successful candidate-submap matching result to update pose prediction for accurate localization. Thus, Lin teaches determining the vehicle position based on successful local-part matching.
according to the result of the matchingSee at least: “Then, the current visual point cloud frame is matched with the candidate sub-map by NDT. Finally, the matching result was used to update pose prediction...” (page 1 / page 7).Rationale: Lin expressly teaches that pose refinement follows from the matching result. Thus, this limitation is expressly taught.
and when the matching of the plurality of feature points with the part of the point cloud data fails,See at least: “It is time-consuming to match the visual point cloud generated by a stereo camera with the whole prior visual point cloud map.” (page 5) and “the current visual point cloud frame is matched with the candidate sub-map...” (page 7).Rationale: Lin expressly contrasts whole-map matching with candidate-submap matching. Although Lin does not expressly recite a failure branch in those exact words, one of ordinary skill in the art would have understood that if matching against the candidate sub-map does not succeed, a predictable fallback is to broaden the search from the local part to the whole prior point-cloud map, especially because Lin expressly identifies whole-map matching as the broader baseline from which sub-map matching is an efficiency improvement.
performing a matching of the plurality of feature pointsSee at least: “It is time-consuming to match the visual point cloud generated by a stereo camera with the whole prior visual point cloud map.” (page 5).Rationale: Lin expressly teaches matching against the whole prior visual point-cloud map as the broader matching operation. Thus, Lin supports the fallback full-map matching concept.
with all the point cloud data in the point cloud basemap,See at least: “match the visual point cloud generated by a stereo camera with the whole prior visual point cloud map.” (page 5).Rationale: Lin expressly discloses whole-map matching. One of ordinary skill in the art would have understood the whole prior point-cloud map to correspond to all the point-cloud data in the basemap.
Lin teaches the current-position candidate-submap matching, the predetermined-range restriction, successful local matching followed by localization update, and a supportable PHOSITA-obvious fallback from local-part matching to whole-map matching if local matching does not succeed.
Liang discloses:
and a mapping relationship between the point cloud basemap and the vector map,See at least: “we fuse LIDAR point clouds and large scale vector maps to reconstruct the 3-D road models” (page 1) and “The roadsides can be paired. The centerlines can be determined...” / “a node is generated by the intersection of two centerlines for the crossroads...” (pages 2–4).Rationale: Liang expressly discloses a fused relationship between LiDAR point clouds and vector-map road geometry, including road and intersection structure. Thus, Liang teaches the claimed mapping relationship between the point-cloud basemap and the vector map.
determining the position of the vehicle within the vector mapSee at least: “the complete set of the planimetric road networks” and “Finally, the road models are reconstructed by combining with two flat quadrilaterals from the paired roadsides and presented as ribbons.” (pages 4–6).Rationale: Liang expressly discloses a coherent vectorized road-network representation tied to fused point-cloud content. In view of the already established Luo/Lin localization workflow, one of ordinary skill in the art would have used the match result within that point-cloud/vector-map relationship to determine position in the vector map.
and determining the position of the vehicle within the vector mapSee at least: “we fuse LIDAR point clouds and large scale vector maps to reconstruct the 3-D road models” (page 1).Rationale: Liang expressly teaches that the vector-map model is organized from the point-cloud/vector fusion itself. Thus, the matched point-cloud result can be interpreted in the vector-map domain through the disclosed fusion relationship.
according to the result of the matching of the plurality of feature points with all of the point cloud data.See at least: Lin’s whole-map matching disclosure, “match the visual point cloud generated by a stereo camera with the whole prior visual point cloud map” (page 5), together with Liang’s point-cloud/vector-map fusion, “we fuse LIDAR point clouds and large scale vector maps...” (page 1).Rationale: Lin teaches the full-map matching fallback, and Liang teaches the point-cloud/vector-map relationship needed to interpret the result within the vector map. Accordingly, the combined teachings render obvious determining position in the vector map according to the result of matching with all point-cloud data.
Motivation to Combine Luo, Flint, Zhao, Farshidi, Lin, and Liang
Therefore, given the teachings as a whole, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having Luo, Flint, Zhao, Farshidi, Lin, and Liang before them, to modify the Luo/Flint/Zhao/Farshidi combination used for Claims 9, 11, and 14 so that the matching operation in the device is first performed against a local part of the point-cloud basemap corresponding to the current vehicle position and containing road/intersection subsets within a predetermined range, and so that, if that local matching succeeds, the device determines the vehicle position in the vector map using the matching result and the point-cloud/vector-map relationship, but if the local matching does not succeed, the device broadens the matching to the whole point-cloud basemap, because Luo already teaches the underlying feature-to-map matching and localization framework, Flint and Farshidi remain part of the parent Claim 14 combination, Zhao teaches road-structure subset organization, Lin teaches current-position-based candidate-submap segmentation and matching with the candidate sub-map rather than the whole map, and Liang teaches the fused relationship between LiDAR point clouds and vector-map road geometry for roads and intersections. One of ordinary skill in the art would have made such modification to improve efficiency by attempting localized matching first while preserving robustness through whole-map fallback if localized matching is unsuccessful, with the predictable benefit of faster matching, more stable localization, and meaningful position determination within the fused point-cloud/vector-map representation. Such a modification would merely combine known local-submap matching, known whole-map matching, and known point-cloud/vector-map fusion techniques in a predictable way and would not alter the principle of operation of any reference or render any reference unsatisfactory for its intended purpose. Therefore, after combining the teachings of Luo, Flint, Zhao, Farshidi, Lin, and Liang, Claim 15 is rendered obvious by the combination of Luo in view of Flint, Zhao, Farshidi, Lin, and Liang. In particular, Luo teaches the underlying feature-to-point-cloud matching and position-determination framework in the device context, Flint and Farshidi remain part of the parent Claim 14 combination, Zhao teaches road/intersection subset context, Lin teaches current-position candidate-submap matching within a predetermined range and supports the full-map fallback, and Liang teaches the mapping relationship between point-cloud data and vector-map geometry. The remaining aspects concerning the explicit failure-triggered fallback from local-part matching to all point-cloud data would have been obvious to one of ordinary skill in the art in view of Lin’s express contrast between candidate-submap matching and whole-map matching and the predictable need to broaden the search when localized matching is unsuccessful.
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Luo, in view of Flint, and in view of Huang (Vision-based Semantic Mapping and Localization for Autonomous Indoor Parking).
Regarding Claim 8,
The combination of Luo and Flint establishes the method of Claim 1, which is the basis for Claim 8.
After Luo and Flint, the following limitations are not explicitly disclosed: the scenario includes an indoor parking lot,; the method further comprises; presenting the vector map; and the position of the vehicle within the vector map; and to a driver of the vehicle.
Huang teaches:
the scenario includes an indoor parking lot,
See at least: “In this paper, we proposed a novel and practical solution for the real-time indoor localization of autonomous driving in parking lots.” (page 1) and “However, parking in a large indoor parking lot without human intervention is still an unsolved problem.” (page 1).Rationale: Huang expressly discloses that the localization scenario is an indoor parking lot. Thus, Huang directly teaches this limitation.
the method further comprisesSee at least: the pipeline on page 3 showing an “online part” that outputs “Stable Parking Map” and “Vehicle Position,” together with the accompanying description that the method performs online localization in the parking lot. (page 3).Rationale: Huang expressly discloses that its online localization workflow produces additional outputs beyond localization computation itself. Thus, Huang teaches that the method further comprises additional output operations.
presenting the vector mapSee at least: “Finally, a two-dimensional map of parking slots can be robustly established which is distinguished from the traditional feature-based or point-cloud map for its stability, re-usability, lightweight and human readable.” (page 1) and the pipeline on page 3 showing “Stable Parking Map.”Rationale: Huang expressly discloses a human-readable two-dimensional parking map and, in the page-3 pipeline, a stable parking map as an online system output. One of ordinary skill in the art would have understood this human-readable stable parking map to be a presented map representation corresponding to the claimed vector map.
and the position of the vehicle within the vector mapSee at least: the page-3 pipeline showing “Stable Parking Map” and “Vehicle Position” as online outputs, and “Then the vehicle drives automatically ... according to the real-time localization.” (pages 5–6).Rationale: Huang expressly discloses vehicle position as an online system output and expressly discloses real-time localization in the indoor parking lot. In view of Huang’s human-readable map output, one of ordinary skill in the art would have understood the vehicle position to be shown within that map representation.
to a driver of the vehicle.See at least: “Finally, a two-dimensional map of parking slots can be robustly established ... human readable.” (page 1), the page-3 pipeline showing “Stable Parking Map” and “Vehicle Position,” and “During the online experiment, the vehicle is first operated by a human driver to initialize the parking map.” (pages 5–6).Rationale: Huang does not expressly recite, in those exact words, that the map and vehicle position are presented “to a driver of the vehicle.” However, Huang expressly teaches a human-readable parking map, expressly outputs stable parking map and vehicle position in the online pipeline, and expressly operates in a vehicle context involving a human driver. One of ordinary skill in the art would have understood that a human-readable parking map and vehicle-position output in an autonomous indoor-parking system would predictably be presented to the driver through a vehicle display or HMI so the driver can monitor localization and parking status. This is a straightforward and predictable use of Huang’s disclosed outputs.
Motivation to Combine Luo, Flint and Huang
Therefore, given the teachings as a whole, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having Luo, Flint and Huang before them, to adapt Luo’s autonomous-vehicle localization method so that, in an indoor parking lot scenario, the method outputs a human-readable map together with the vehicle’s current position and presents that information to the driver, because Luo already teaches the baseline vehicle-localization method using camera- and LiDAR-derived map information, while Huang teaches autonomous indoor-parking localization using a human-readable parking map together with vehicle-position output. One of ordinary skill in the art would have made this modification to improve driver awareness, usability, and monitoring of vehicle localization during indoor parking, with the predictable benefit of giving the driver a clear view of the vehicle’s location within the mapped parking environment. Such a modification would merely apply Huang’s known indoor-parking map/position output technique to Luo’s known localization framework and would not alter the principle of operation of either reference or render either reference unsatisfactory for its intended purpose. Therefore, after combining the teachings of Luo, Flint and Huang, in view of the already established Claim 1 combination including Flint, Claim 8 is rendered obvious by the combination of Luo in view Flint, and in view of of Huang. In particular, Luo teaches the underlying vehicle-localization framework, Flint remains part of the parent-claim combination for Claim 1, and Huang teaches the indoor parking lot scenario together with the online output of a human-readable parking map and vehicle position. The remaining aspect concerning presentation of that information to a driver of the vehicle would have been obvious to one of ordinary skill in the art in view of Huang’s express teaching that the map is human readable and produced as part of the online localization workflow.
Claim 16 is rejected under 35 U.S.C. 103 as being unpatentable over Luo, in view of Flint, in view of Zhao, in view of Farshidi, in view of Lin, in view of Liang, and in view of Huang.
Regarding Claim 16,
The combination of Luo, Flint, Zhao, Farshidi, Lin, and Liang establishes the device of Claim 15, which is the basis for Claim 16.
After the combination of Luo, Flint, Zhao, Farshidi, Lin, and Liang, the following limitations are not explicitly disclosed: the scenario includes an indoor parking lot,; the method further comprises; presenting the vector map; and the position of the vehicle within the vector map; and to a driver of the vehicle.
Huang discloses:
the scenario includes an indoor parking lot,
See at least: “we proposed a novel and practical solution for the real-time indoor localization of autonomous driving in parking lots” (page 1) and “However, parking in a large indoor parking lot without human intervention is still an unsolved problem.” (page 1).Rationale: Huang expressly discloses that the localization scenario is an indoor parking lot. Thus, Huang directly teaches this limitation.
the method further comprises
See at least: “Our system is implemented on an autonomous driving vehicle and tested in real parking lots.” (page 1) and “Finally, a two-dimensional map of parking slots can be robustly established...” (page 1).
Rationale: Huang expressly discloses an operational localization workflow that produces additional outputs beyond the underlying localization computation. Thus, Huang teaches that the method further comprises additional output operations.
presenting the vector map
See at least: “Finally, a two-dimensional map of parking slots can be robustly established which is distinguished from the traditional feature-based or point-cloud map for its stability, re-usability, lightweight and human readable.” (page 1).
Rationale: Huang expressly discloses a human-readable two-dimensional parking map. One of ordinary skill in the art would have understood this human-readable parking map to be a presented map representation corresponding to the claimed vector map.
and the position of the vehicle within the vector map
See at least: “we proposed a novel and practical solution for the real-time indoor localization of autonomous driving in parking lots” (page 1) and “Our system is implemented on an autonomous driving vehicle and tested in real parking lots.” (page 1).
Rationale: Huang expressly discloses real-time indoor localization for autonomous driving in parking lots and a human-readable parking map. In view of these teachings, one of ordinary skill in the art would have understood the vehicle position to be shown within that map representation so the localization result is meaningful to the user.
to a driver of the vehicle.
See at least: “Finally, a two-dimensional map of parking slots can be robustly established ... human readable.” (page 1) and “Our system is implemented on an autonomous driving vehicle and tested in real parking lots.” (page 1).
Rationale: Huang expressly teaches a human-readable parking map and an autonomous-driving vehicle context. One of ordinary skill in the art would have understood that a human-readable parking map and localization result in a vehicle-based indoor-parking system would predictably be presented to the driver through a vehicle display or HMI so the driver can monitor localization and parking status. This is a straightforward and predictable use of Huang’s disclosed outputs.
Motivation to Combine Luo, Flint, Zhao, Farshidi, Lin, Liang, and Huang
Therefore, given the teachings as a whole, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having Luo, Flint, Zhao, Farshidi, Lin, Liang, and Huang before them, to modify the Luo/Flint/Zhao/Farshidi/Lin/Liang combination used for Claim 15 so that, in an indoor parking lot scenario, the device further outputs a human-readable map together with the vehicle’s current position and presents that information to the driver, because Luo teaches the underlying localization device framework, Flint, Zhao, Farshidi, Lin, and Liang remain part of the parent-claim combination for Claim 15, and Huang teaches real-time indoor-parking localization using a human-readable parking map in an autonomous-driving vehicle context. One of ordinary skill in the art would have made this modification to improve driver awareness, usability, and monitoring of vehicle localization during indoor parking, with the predictable benefit of giving the driver a clear view of the vehicle’s location within the mapped parking environment. Such a modification would merely apply Huang’s known indoor-parking map/presentation output technique to the already-established Claim 15 localization framework and would not alter the principle of operation of any reference or render any reference unsatisfactory for its intended purpose. Therefore, after combining the teachings of Luo, Flint, Zhao, Farshidi, Lin, Liang, and Huang, Claim 16 is rendered obvious by the combination of Luo in view of Flint, Zhao, Farshidi, Lin, Liang, and Huang. In particular, Luo teaches the underlying device-based localization framework, Flint, Zhao, Farshidi, Lin, and Liang remain part of the parent-claim combination for Claim 15, and Huang teaches the indoor parking lot scenario together with a human-readable parking map in an autonomous-driving vehicle context. The remaining aspect concerning presentation of the vector map and vehicle position to a driver of the vehicle would have been obvious to one of ordinary skill in the art in view of Huang’s express teaching that the map is human readable and used in a vehicle-based indoor-parking localization system.
For each of the foregoing rejections, the combination of Luo and Flint establishes the limitations of the amended independent claim from which the rejected claim depends, for the reasons set forth above for claims 1, 9, and 17, respectively. The findings regarding the additional limitations of each dependent claim, the teachings of Zhao, Zhou, Farshidi, Lin, Liang, and Huang relied upon for those limitations, and the corresponding reasons to combine, are as set forth in the Non-Final Office Action dated April 1, 2026, and are incorporated herein by reference. Those findings and reasons are unchanged, because the additional limitations of the dependent claims were not amended. Applicant has not separately challenged them, as addressed in the Response to Arguments below.
Response to Arguments
Applicant’s arguments filed 06/30/2026 with the amendment responding to the Non-Final Office Action dated April 1, 2026 have been fully considered but are not persuasive. The amendments to independent claims 1, 9, and 17 necessitated revised citations and an updated rationale, which are set forth in the rejections above. To the extent Applicant’s arguments are directed to the newly added limitations, they are addressed by those rejections and further below.
A. Applicant’s argument that Luo’s map is constructed by the vehicle within the scenario rather than pre-constructed outside of it
Applicant argues that Luo “explicitly provides that the map is constructed by the vehicle within the full scenario itself, and is not pre-constructed outside of the full scenario,” relying on Luo [0012]–[0014]. Examiner respectfully disagrees for the following reasons.
1. The argument relies on features not recited in the claims.
Applicant’s characterization of the claimed invention turns on four features:
the fused map covers “a full scenario”;
the map is uploaded “while the previously-constructed fused map may still be obtained from a cloud network in an unobstructed manner”;
the map is “pre-constructed outside of the full scenario”;
the map is constructed by an entity other than the vehicle.
None of these features appears in amended claims 1, 9, or 17. The claims recite “a previously-constructed fused map of the scenario,” a “previously-constructed point cloud basemap of the scenario,” and a “previously-constructed vector map describing the scenario.” They do not recite a “full” scenario. They do not require construction outside the scenario or by any particular entity. They recite no unobstructed condition of the cloud network at the time of upload. Although the claims are interpreted in light of the specification, limitations from the specification are not read into the claims. In re Van Geuns, 988 F.2d 1181, 1184 (Fed. Cir. 1993); MPEP 2145(VI).
2. The claimed “previously-constructed” fused map.
Under the broadest reasonable interpretation, the claims require that the fused map already be constructed when it is uploaded from the cloud network to the vehicle before the vehicle enters the scenario. The claims place no further restriction on where, when, or by whom the map was constructed.
The rejection does not rely on Luo’s sequence of constructing the global map before feature matching to establish this timing. It relies on the combination of Luo and Flint, in which Luo’s global map is retrieved from remote storage before the vehicle enters a cellular-denied environment, as explained in Section A.4 below. A map retrieved from remote storage necessarily exists, and has therefore already been constructed, at the time it is retrieved.
The claims further require that the map uploaded before entry be the previously-constructed fused map, including both the point cloud basemap and the vector map. In the proposed combination, the point cloud map and the floor-plan map would be registered and integrated into the fused map as part of map preparation, before the fused map is stored remotely and transferred to the vehicle.
First, in the proposed combination, both map components would be made available before entry into the cellular-denied environment (Flint [0003]), so that localization and position representation can continue locally thereafter. Delivering them as a single, already-registered dataset is a predictable way to do so.
Second, registering the two maps to a common reference frame is a map-preparation operation. Performing it once during map preparation avoids repeating the same registration operation on each vehicle and promotes consistent map alignment across vehicles, which is a predictable implementation benefit. Flint further recognizes that on-device navigation processing “can be computationally burdensome, especially for devices ... that may have limited computation power” (Flint [0026]). That provides additional reason to perform non-time-critical preparatory processing before deployment rather than on the vehicle. Luo likewise treats map construction as an operation separate from localization (Luo FIG. 1, operation 13) and stores localization data remotely for network access (Luo [0109]).
A person of ordinary skill in the art would therefore have found it obvious to construct the fused map before storing it remotely and transferring it to the vehicle, yielding the predictable result of a complete positioning dataset available locally upon entry. The fused map uploaded to the vehicle before entry in the combined system is a previously-constructed fused map within the meaning of the claims.
3. Applicant’s reading of Luo does not establish nonobviousness.
Applicant relies on Luo [0012]–[0014], which describe what is performed “[i]n another embodiment” and “[i]n still another embodiment,” to argue that Luo’s vehicle constructs its own map within the scenario. Those paragraphs do not state that every localization uses a global map constructed during the same traversal, or that the map is constructed at the time the vehicle enters the environment in which it is localized.
Luo describes the global map as “a 3D city-scale map” constructed “using LiDAR mapping” (Luo [0069]). Luo separately constructs a local, camera-based 3D submap from the vehicle’s images ( [0069], [0075]) and locates that submap within the global map by transforming “a coordinate of the 3D submap ... to a coordinate of the global map” ( [0077]). A reference is relied upon for all that it would reasonably teach a person of ordinary skill in the art, including embodiments other than those Applicant emphasizes. In re Mouttet, 686 F.3d 1322, 1331 (Fed. Cir. 2012); Merck & Co. v. Biocraft Labs., Inc., 874 F.2d 804, 807 (Fed. Cir. 1989); MPEP 2123.
More importantly, even accepting Applicant’s reading of Luo’s map-construction embodiment, the argument does not overcome the rejection. A global map constructed from data collected during a prior traversal of the environment is still a previously-constructed map. Retrieving that already-constructed map, integrated with the floor-plan map as discussed above, from remote storage before a later entry into a cellular-denied environment would have been obvious for the reasons set forth below.
4. The rejection is based on the combined teachings of Luo and Flint.
The rejection does not rely on Luo alone for acquiring the map before the vehicle enters the scenario. As set forth in the rejection above, the combination of Luo and Flint renders the pre-entry acquisition obvious:
Luo teaches that data used by its localization programs is “stored remotely” and accessed “through a network” ( [0109]), over a wireless network that may be 3G ( [0102]).
Luo’s localization process expressly uses the global map as the data against which camera-derived features are matched ( [0071]).
Luo teaches a memory configured to store data accessed by the localization programs ( [0112]) and separately discloses execution of those programs by a computing device that is part of a vehicle ( [0055]).
Flint teaches operation “indoors where GPS and cell signals cannot reach” (Flint [0003]), including building interiors such as warehouses (Flint [0071]).
For the reasons of reliability and continuity of positioning articulated in the rejection, a person of ordinary skill in the art would have been motivated to retrieve the already-constructed fused map from remote storage before entering an area known to lack cellular availability, so that localization could continue without depending on network access inside that area. In the resulting on-vehicle implementation, storing the retrieved map in vehicle-accessible memory would have been an ordinary and predictable implementation, because it permits local use of the map after network connectivity becomes unavailable.
Applicant’s remarks do not address this articulated reason to combine. Applicant asserts that “[n]either Flint nor any other reference(s) cited by Examiner remedy this deficiency of Luo”. However, Applicant does not explain why Flint’s cellular-denied environment, combined with Luo’s remote-data architecture, would not have motivated pre-loading the localization map before entry. To the extent Applicant relies on the fact that neither reference individually contains the complete claimed arrangement, that does not address what their combined teachings would have suggested to a person of ordinary skill in the art. See In re Keller, 642 F.2d 413, 426 (CCPA 1981); In re Merck & Co., 800 F.2d 1091, 1097 (Fed. Cir. 1986); MPEP 2145(IV). Attorney argument does not take the place of evidence in the record. In re Pearson, 494 F.2d 1399, 1405 (CCPA 1974); MPEP 2145(I).
B. Applicant’s argument regarding Flint
Applicant argues that Flint “makes no mention of prior acquisition of a pre-constructed fused map of a scenario ..., wherein the pre-constructed fused map includes a pre-constructed point cloud basemap ... and a pre-constructed vector map.” This argument is not persuasive. Flint is not relied upon as teaching every aspect of the fused map, or as independently teaching prior acquisition. As set forth in the rejection above:
Luo teaches the previously-constructed point cloud basemap (Luo [0069], [0081]).
Flint teaches the previously-existing geometric floor-plan map of the environment on which computed positions are expressed (Flint [0029], [0042], [0069]). Implementing that floor plan as vector graphic elements would have been a predictable choice among known representation formats.
Integrating the two maps into a single fused map follows from the need to register them to a common reference frame, because Luo establishes vehicle position in the coordinate frame of its global map (Luo [0077]) and the combination expresses that position on Flint’s floor-plan map. For the reasons set forth in Section A.2, that integration would have been performed as part of map preparation, before the fused map is stored remotely and transferred to the vehicle.
The pre-entry acquisition follows from the combination as explained in Section A.4 above.
Notably, Applicant acknowledges that Flint “relates to image-based navigation within an environment after cloud network access is obstructed”. That is the very teaching for which Flint is relied upon. It supplies the motivation to acquire the map before entering such an environment and to perform positioning on the device, as Flint expressly contemplates by identifying on-device processing as an alternative to server-based processing (Flint [0069]). Applicant’s remarks do not address why this teaching, applied to Luo, would not have motivated the claimed arrangement.
C. Applicant’s characterization of the prior Office Action
Applicant quotes the prior Office Action’s statement that Luo discloses “that the maps used for localization are prepared before matching and position estimation are performed”. The present rejection, necessitated by amendment, does not rest on that statement. As set forth above, the rejection relies on the following, which Applicant’s remarks do not address:
Luo’s disclosure of remotely stored data accessed through a network (Luo [0109]);
Luo’s memory storing data accessed by the localization programs, and its vehicle-resident execution (Luo [0055], [0112]);
Luo’s disclosure of a 3G wireless network (Luo [0102]);
Luo’s coordinate transformation into the global map (Luo [0077]);
Flint’s cellular-denied operating environment, floor-plan map, and on-device processing alternative (Flint [0003], [0028], [0029], [0042], [0069], [0071]).
D. Applicant’s statement of support for the amendments
Applicant states that the amended limitations “are fully supported in [0029]-[0031] of the Application as filed”. Examiner respectfully disagrees as to the limitation “in which communication with a cloud network is prevented,” for the reasons set forth in the rejection under 35 U.S.C. 112(a) above.
The specification describes a network environment that “may be poor” ( [0026], [0030]), giving rise to “slow uploading of data or slow obtaining of matching results due to an influence of network transmission conditions” ( [0030]). It further contemplates obtaining the fused map through “Wi-Fi, cellular, Bluetooth, etc.” ([0058]). A network that is poor or slow is not a network with which communication is prevented.
Applicant’s remarks characterize the scenario as one of “communication obstruction” in which cloud communication is “interrupted”. Those characterizations are not tied to any disclosure in the original specification describing communication as prevented, and Applicant has not identified any such disclosure. Accordingly, the rejection under 35 U.S.C. 112(a) set forth above is proper.
E. Dependent claims
With respect to claims 2–8, 10–16, and 18–20, Applicant relies on the same arguments presented for the respective independent claims. Applicant does not separately contest the additional limitations of the dependent claims, or the teachings of Zhao, Zhou, Farshidi, Lin, Liang, or Huang as relied upon for those limitations. Applicant’s arguments concerning the limitations inherited from claims 1, 9, and 17 are not persuasive for the reasons set forth above, and no separate arguments have been presented concerning the additional limitations. Accordingly, the rejections of the dependent claims are maintained.
Examiner Response Conclusion
For the foregoing reasons, the rejections of claims 1–20 under 35 U.S.C. 103 are maintained, as revised to address the amended limitations, and claims 1–20 are further rejected under 35 U.S.C. 112(a) and 35 U.S.C. 112(b) as set forth above.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to OLUWABUSAYO ADEBANJO AWORUNSE whose telephone number is (571)272-4311. The examiner can normally be reached M - F (8:30AM - 5PM).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jelani Smith can be reached at (571) 270-3969. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/OLUWABUSAYO ADEBANJO AWORUNSE/Examiner, Art Unit 3662
/JELANI A SMITH/Supervisory Patent Examiner, Art Unit 3662
