DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This is a Non-Final Action on the Merits. Claims 1-30 are currently pending and are addressed below.
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on March 9th, 2026 has been entered.
Response to Amendments
The amendment filed on March 9th, 2026 has been considered and entered. Accordingly, claims 1, 13, and 25 have been amended.
Response to Arguments
The Applicant states (Amend. 8-9) that “each of claims 1, 13, and 25 recites an improvement for localizing an object and that the various features of the claims directed to, for example, the "receiving", "matching," and two "determining" steps are in fact limitations, or restrictions, on how the improvement is accomplished. The incorporation of additional elements into each of claims 1, 13, and 25, as amended, further integrate additional limitations/restrictions on how the improvement is accomplished. As such, Applicant respectfully submits that the claims are patent eligible.” The examiner respectfully disagrees. Amended claim 1 at most discusses an abstract idea to determine the location of an object. Even if, for the sake of the argument, the determination is a new idea, “a claim for a new abstract idea is still an abstract idea.” Synopsys, Inc. v. Mentor Graphics Corp., 839 F.3d 1138, 1151 (Fed. Cir. 2016) (emphasis omitted); see also Intellectual Ventures I LLC v. Symantec Corp., 838 F.3d 1307, 1321 (Fed. Cir. 2016) (“A narrow claim directed to an abstract idea, however, is not necessarily patent-eligible.”). Furthermore, the applicants assert arguments for (1) an improvement for the localization of an object and (2) a more general improvement to an existing technological process. However, the applicant’s arguments are not persuasive because applicant’s claim 1 fails to recite (1) any limitations detailing “low demand services”, how to efficiently “uninstall and then reinstall a service” or how management of services are allowed to be more efficient, and (2) any limitations detailing how “allowing the service requester to receive a desired quality of service” or how “not experiencing a delay or difference in quality of service even if the requested service had its processing priority lowered and needed to be reconfigured” is achieved. When a claim directed to an abstract idea contains no restriction on how an asserted improvement is accomplished and the asserted improvement is not described in the claim, then the claim does not become patent eligible. See Intellectual Ventures I LLC v. Symantec Corp., 838 F.3d 1307, 1316 (Fed. Cir. 2016); see also MPEP 2106.04(d)(1) (“Second, if the specification sets forth an improvement in technology, the claim must be evaluated to ensure that the claim itself reflects the disclosed improvement. That is, the claim includes the components or steps of the invention that provide the improvement described in the specification”).
The Applicant’s arguments with respect to claims 1-30 have been considered but are moot in view of the newly formulated grounds of rejections necessitated by the applicant’s amendments.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-30 are rejected under 35 U.S.C. § 101 because the claimed invention is directed to a judicial exception (i.e., an abstract idea) without significantly more.
In sum, claims 1-30 are rejected under 35 U.S.C. §101 because the claimed invention is directed to a judicial exception to patentability (i.e., a law of nature, a natural phenomenon, or an abstract idea) and do not include an inventive concept that is something “significantly more” than the judicial exception under the January 2019 patentable subject matter eligibility guidance (2019 PEG) analysis which follows.
Under the 2019 PEG step 1 analysis, it must first be determined whether the claims are directed to one of the four statutory categories of invention (i.e., process, machine, manufacture, or composition of matter). Applying step 1 of the analysis for patentable subject matter to the claims, it is determined that the claims are directed to the statutory category of a process. Therefore, we proceed to step 2A, Prong 1.
Revised Guidance Step 2A – Prong 1
Under the 2019 PEG step 2A, Prong 1 analysis, it must be determined whether the claims recite an abstract idea that falls within one or more designated categories of patent ineligible subject matter (i.e., organizing human activity, mathematical concepts, and mental processes) that amount to a judicial exception to patentability.
Here, with respect to independent claims 1, 13, and 25, the claims recite the abstract idea of determining the localization of an object based on map comparisons, and mentally determine “generate, based on sensor data obtained from one or more sensors associated with the object, a predicted map comprising a plurality of predicted nodes associated with a predicted location of the object within an environment; match at least one node in a first graph representation of the plurality of predicted nodes with at least one node in a second graph representation of the plurality of HD nodes to determine one or more pairs of matched nodes between the predicted map and the HD map; determine, based on a comparison between nodes in each pair of the one or more pairs of matched nodes between the predicted map and the HD map, a respective node score for each pair of the one or more pairs of matched nodes; and determine, based on the respective node score for each pair of the one or more pairs of matched nodes between the predicted map and the HD map, a location of the object within the environment”, where these claims fall within one or more of the three enumerated 2019 PEG categories of patent ineligible subject matter, specifically, a mental process, that can be performed in the human mind since each of the above steps could alternatively be performed in the human mind or with the aid of pen and paper. This conclusion follows from CyberSource Corp. v. Retail Decisions, Inc., where our reviewing court held that section 101 did not embrace a process defined simply as using a computer to perform a series of mental steps that people, aware of each step, can and regularly do perform in their heads. 654 F.3d 1366, 1373 (Fed. Cir. 2011); see also In re Grams, 888 F.2d 835, 840–41 (Fed. Cir. 1989); In re Meyer, 688 F.2d 789, 794–95 (CCPA 1982); Elec. Power Group, LLC v. Alstom S.A., 830 F. 3d 1350, 1354–1354 (Fed. Cir. 2016) (“we have treated analyzing information by steps people go through in their minds, or by mathematical algorithms, without more, as essentially mental processes within the abstract-idea category”).
Additionally, mental processes remain unpatentable even when automated to reduce the burden on the user of what once could have been done with pen and paper. See CyberSource, 654 F.3d at 1375 (“That purely mental processes can be unpatentable, even when performed by a computer, was precisely the holding of the Supreme Court in Gottschalk v. Benson.”). These limitations, as drafted, are a simple process that under their broadest reasonable interpretation, covers the performance of the limitations of the mind. For example, the claim limitation encompasses mentally determine a location of an obstacle based on map comparisons provided by the car’s sensors while traveling, or alternatively, mentally determine a location of an obstacle based on map comparisons based on observations by a human.
For example, a human could mentally and with the aid of pen and paper determine a location of an obstacle based on map comparisons.
Revised Guidance Step 2A – Prong 2
Under the 2019 PEG step 2A, Prong 2 analysis, the identified abstract idea to which the claim is directed does not include limitations that integrate the abstract idea into a practical application, since the additional elements of a vehicle sensors, processor, and memory are merely generic components used as a tool (“apply it”) to implement the abstract idea. (See, e.g., MPEP §2106.05(f)). See Alice, 573 U.S. at 223 (“[T]he mere recitation of a generic computer cannot transform a patent-ineligible abstract idea into a patent-eligible invention.”)
In addition, the limitation “receive a high definition (HD) map comprising a plurality of HD nodes associated with a HD location of the object within the environment” constitutes insignificant presolution activity that merely gathers data and, therefore, do not integrate the exception into a practical application. See In re Bilski, 545 F.3d 943, 963 (Fed. Cir. 2008) (en banc), aff' d on other grounds, 561 U.S. 593 (2010) (characterizing data gathering steps as insignificant extra-solution activity); see also CyberSource, 654 F.3d at 1371–72 (noting that even if some physical steps are required to obtain information from a database (e.g., entering a query via a keyboard, clicking a mouse), such data-gathering steps cannot alone confer patentability); OIP Techs., Inc. v. Amazon.com, Inc., 788 F.3d 1359, 1363 (Fed. Cir. 2015) (presenting offers and gathering statistics amounted to mere data gathering). Accord Guidance, 84 Fed. Reg. at 55 (citing MPEP § 2106.05(g)).
In addition, merely “[u]sing a computer to accelerate an ineligible mental process does not make that process patent-eligible.” Bancorp Servs., L.L.C. v. Sun Life Assur. Co. of Canada (U.S.), 687 F.3d 1266, 1279 (Fed. Cir. 2012); see also CLS Bank Int’l v. Alice Corp. Pty. Ltd., 717 F.3d 1269, 1286 (Fed. Cir. 2013) (en banc) (“simply appending generic computer functionality to lend speed or efficiency to the performance of an otherwise abstract concept does not meaningfully limit claim scope for purposes of patent eligibility.”), aff’d, 573 U.S. 208 (2014). Accordingly, the additional element of a processor does not transform the abstract idea into a practical application of the abstract idea.
Revised Guidance Step 2B
Under the 2019 PEG step 2B analysis, the additional elements are evaluated to determine whether they amount to something “significantly more” than the recited abstract idea. (i.e., an innovative concept). Here, the additional elements, such as: a processor, a sensor, and a memory does not amount to an innovative concept since, as stated above in the step 2A, Prong 2 analysis, the claims are simply using the additional elements as a tool to carry out the abstract idea (i.e., “apply it”) on a computer or computing device and/or via software programming. (See, e.g., MPEP §2106.05(f)). The additional elements are specified at a high level of generality to simply implement the abstract idea and are not themselves being technologically improved. (See, e.g., MPEP §2106.05 I.A.). See Alice, 573 U.S. at 223 (“[T]he mere recitation of a generic computer cannot transform a patent-ineligible abstract idea into a patent-eligible invention.”). Thus, these elements, taken individually or together, do not amount to “significantly more” than the abstract ideas themselves.
The additional elements of the dependent claims 2-12, 14-24, and 26-30 merely refine and further limit the abstract idea of the independent claims and do not add any feature that is an “inventive concept” which cures the deficiencies of their respective parent claim under the 2019 PEG analysis. None of the dependent claims considered individually, including their respective limitations, include an “inventive concept” of some additional element or combination of elements sufficient to ensure that the claims in practice amount to something “significantly more” than patent-ineligible subject matter to which the claims are directed.
The elements of the instant claimed invention, when taken in combination do not offer substantially more than the sum of the functions of the elements when each is taken alone. The claims as a whole, do not amount to significantly more than the abstract idea itself because the claims do not effect an improvement to another technology or technical field; the claims do not amount to an improvement to the functioning of an electronic device itself which implements the abstract idea (e.g., the general purpose computer and/or the computer system which implements the process are not made more efficient or technologically improved); the claims do not perform a transformation or reduction of a particular article to a different state or thing (i.e., the claims do not use the abstract idea in the claimed process to bring about a physical change. See, e.g., Diamond v. Diehr, 450 U.S. 175 (1981), where a physical change, and thus patentability, was imparted by the claimed process; contrast, Parker v. Flook, 437 U.S. 584 (1978), where a physical change, and thus patentability, was not imparted by the claimed process); and the claims do not move beyond a general link of the use of the abstract idea to a particular technological environment (e.g., “for localizing an object. . . vehicle sensors” claim 1).
Accordingly, claims 1-30 are rejected under 35 USC 101 as being drawn to an abstract idea without significantly more, and thus are ineligible.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-2, 5, 7-8, 11, 13-14, 17, 19-20, 23, 25, and 27-28 are rejected under 35 U.S.C. 103 as being unpatentable over Chang (US 20200134896 A1) (“Chang”) in view of Kawasaki (US 20210383213 A1) (“Kawasaki”) in view of Garimella (US 20220274625 A1) (“Garimella”) in view of Kroepfl (US 20210063200 A1) (“Kroepfl”).
With respect to claim 1, Chang teaches an apparatus for localizing an object, the apparatus comprising: at least one memory; and at least one processor coupled to the at least one memory and configured to:
receive a high definition (HD) map comprising a plurality of HD nodes associated with a HD location of the object within the environment (See at least Chang Paragraph 85 “In an example, the map data is high density (HD) map data. An HD map is a 3D map with a high density, for example, a centimeter-level density, that may be used for autonomous driving. The HD map includes, for example, line information related to a road center line and a boundary line, and information related to a traffic light, a traffic sign, a curb, a road surface marking, and various structures in a form of 3D digital data. The HD map is established by, for example, a mobile mapping system (MMS). The MMS, a 3D space information investigation system equipped with various sensors, obtains minute position information using a moving object equipped with sensors such as a camera, a lidar, and a GPS to measure a position and geographic features.”);
Chang fails to explicitly disclose generate, based on sensor data obtained from one or more sensors associated with the object, a predicted map comprising a plurality of predicted nodes associated with a predicted location of the object within an environment; match at least one node in a first graph representation of the plurality of predicted nodes with at least one node in a second graph representation of the plurality of HD nodes to determine one or more pairs of matched nodes between the predicted map and the HD map; determine, based on a comparison between nodes in each pair of the one or more pairs of matched nodes between the predicted map and the HD map, a respective node score for each pair of the one or more pairs of matched nodes; and determine, based on the respective node score for each pair of the one or more pairs of matched nodes between the predicted map and the HD map, a location of the object within the environment.
Kawasaki teaches to generate, based on sensor data obtained from one or more sensors associated with the object, a predicted map comprising a plurality of predicted nodes associated with a predicted location of the object within an environment (See at least Kawasaki Paragraphs 50-52 “The environment map generator 103 generates an environment map. The environment map includes map information expressing, on a map, an environment around the moving object at a reference time point. For example, the environment map generator 103 generates an environment map using the moving object information and the environmental information at the reference time point … The cumulative map generator 104 generates a cumulative map. The cumulative map includes map information expressing, on a map, a plurality of positions indicated by the moving object information acquired at a plurality of time points (first time point) equal to or earlier than the reference time point. In this manner, the cumulative map is generated so as to cumulatively store, on the map, not only the position at the reference time point but also the positions indicated by the moving object information acquired earlier than the reference time”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the apparatus of Chang to generate, based on sensor data obtained from one or more sensors associated with the object, a predicted map comprising a plurality of predicted nodes associated with a predicted location of the object within an environment, as taught by Kawasaki as disclosed above, in order to ensure accurate object detection in dynamic environments (Kawasaki Paragraph 22 “In view of this, there has been a demand for a technique capable of predicting the position of other moving objects existing in the surroundings by using information obtained from sensors such as cameras and laser sensors attached to moving objects (automobiles, autonomous mobile robots, or the like) alone without using map information prepared in advance.”).
Chang in view of Kawasaki fails to explicitly disclose match at least one node in a first graph representation of the plurality of predicted nodes with at least one node in a second graph representation of the plurality of HD nodes to determine one or more pairs of matched nodes between the predicted map and the HD map; determine, based on a comparison between nodes in each pair of the one or more pairs of matched nodes between the predicted map and the HD map, a respective node score for each pair of the one or more pairs of matched nodes; and determine, based on the respective node score for each pair of the one or more pairs of matched nodes between the predicted map and the HD map, a location of the object within the environment.
Garimella teaches match at least one node in a first graph representation of the plurality of predicted nodes with at least one node in a second graph representation of the plurality of HD nodes to determine one or more pairs of matched nodes between the predicted map and the HD map (See at least Garimella FIG. 7 and Paragraph 17 “In some examples, the vectorization component of the autonomous vehicle may determine a rotational offset for a map element, before or during the encoding process. For instance, the vectorization component may match a map element detected in the environment to a map element template, based on the size, shape, and/or type of the element. To illustrate, crosswalks 110, 112, and 114 each may be matched to a common crosswalk template having a predetermined shape and/or orientation. After matching crosswalks 110-114 to the crosswalk template, the vectorization component may determine a rotational offset for each of the crosswalks 110-114, to represent the difference between the orientation of the crosswalk and the predetermined orientation of the crosswalk template. A rotational offset for an individual crosswalks (or other map element) may be stored as an attribute associated with the vectorized representation of the crosswalk, thereby allowing each crosswalk in the environment to be encoded in the same predetermined orientation, to reduce variation and improve the efficiency and accuracy of the encoding process. As described below, rotational offsets and/or other types of canonical local coordinate frames (e.g., size and shape offsets) may be determined for any map element represented by a polyline, and these offsets (e.g., canonical local frames) may be stored as attributed of the nodes and/or edge features within a graph structure (e.g., a GNN) used to represent the environment. Of course, though depicted in FIG. 1 for illustrative purposes as being performed after receiving the map elements, it is contemplated that such map elements may be previously vectorized such that the vectorized elements may be received directly.” | Paragraph 68 “Each of the individual line segment vectors representing the lane 302 may be provided to a template converter/encoder 308 of the vectorization component 230. As shown in this example, the template converter/encoder 308 may be implemented into a single system, network, etc., but in other examples a template converter and encoder may be implemented separately. In some cases, each line segment vector may be provided separately to the template converter/encoder 308, while in other cases the associated line segment vectors for a lane object (or other map element) may be provided as a group to the template converter/encoder 308. The template converter/encoder 308 in this example may include the functionality to match the map element (e.g., lane) to an associated map element template (e.g., a lane template). Based on the matching, the template converter/encoder 308 may convert the individual line segment vectors 304 and 306 into vectors complying with the map element template. As shown in box 310, in some cases the template converter/encoder 308 may determine a rotational offset between a lane segment and/or polyline 312 based on a template, and may convert segment and/or polyline to a rotated lane segment and/or polyline 314 that corresponds to the predetermined orientation of the template.” | Paragraphs 96-98 “If the GNN component 232 determines in operation 710 that an entity node is not to be added or updated to the GNN based on the perceived entities (710: No), then process 700 may return to operation 708 to await additional perception data from the sensor and perception systems of the autonomous vehicle. However, if the GNN component 232 determines in operation 710 that a new entity node is to be added to the GNN or an existing entity node is to be updated (710: Yes), then at operation 712 one or more entity nodes may be generated and/or updated based on the perceived entities, within the appropriate graph structure(s) of the GNN. As described above, to add a new entity node to the GNN, the vectorization component 230 may define one or more polylines representing the entity, vectorize the individual line segments of the polylines, convert and/or encode the vectorized polylines, and then aggregate the encoded polylines into a node data structure compatible with the GNN. At operation 714, the GNN component 232 may determine edge feature data and generate and/or update any edge features necessary based on any nodes added or updated in operations 706 and/or 712. In some examples, for any new node and/or when the position or orientation of an existing entity node changes, the GNN component 232 may determine updated relative positions values and/or updated relative yaw values for each edge connected to the affected node. At operation 716, the entity prediction component 234 may use the GNN to determine a predicted future states for one or more entities represented in the GNN. As described above, the GNN may be configured to execute one or more inference process to update the node states and/or edge features of the GNN, providing an inferred (or predicted) future state representation for all of the objects in the environment. In some examples, entity prediction component 234 may access the updated state of the GNN, retrieve and decode one or more portions of the updated GNN, and use the updated GNN data determine predicted entity states. Predictions of states for entities may include predicted locations, velocities, trajectories, orientations, and the like, at various time intervals beyond the current time. In some examples, to perform a prediction of an entity state based on an updated GNN, the entity prediction component 234 may select and decode specific portions of the GNN including the entity and selected edges/nodes that are more closely positioned or related to the entity, rather than decoding the entire GNN.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the apparatus of Chang in view of Kawasaki to match at least one node in a first graph representation of the plurality of predicted nodes with at least one node in a second graph representation of the plurality of HD nodes to determine one or more pairs of matched nodes between the predicted map and the HD map, as taught by Garimella as disclosed above, in order to ensure accurate object detection (Garimella Paragraph 11 “As discussed below, the various techniques described herein provide technical improvements in the environment modeling and predictive capabilities of the autonomous vehicles, as well as technical advantages in reducing computing resources and improving efficiency of the prediction and decision-making components of autonomous vehicles.”).
Chang in view of Kawasaki in view of Garimella fails to explicitly disclose to determine, based on a comparison between nodes in each pair of the one or more pairs of matched nodes between the predicted map and the HD map, a respective node score for each pair of the one or more pairs of matched nodes; and determine, based on the respective node score for each pair of the one or more pairs of matched nodes between the predicted map and the HD map, a location of the object within the environment.
Kroepfl teaches to determine, based on a comparison between nodes in each pair of the one or more pairs of matched nodes between a 2D or 3D object representation and the HD map, a respective node score for each pair of the one or more pairs of matched nodes; and determine, based on the respective node score for each pair of the one or more pairs of matched nodes between a 2D or 3D object representation and the HD map, a location of the object within the environment (See at least Kroepfl FIG. 9A and Paragraph 135 “With reference to FIG. 9A, cost space 902 may represent a cost space for a sensor modality—e.g., a camera based cost space generated according to FIGS. 10A-10C. For example, the cost space 902 may represent the likelihood that a vehicle 1500 is currently in each of a plurality of poses—e.g., represented by points of the cost space 902—at a current frame. The cost space 902, although represented in 2D in FIG. 9A, may correspond to a 3D cost space (e.g., with (x, y, z) locations and/or axis angles for each of the x, y, and z axes). As such, the sensor data 102—e.g., before or after pre-processing—and/or the outputs 204 (e.g., detections of landmark locations in 2D image space and/or 3D image space) may be compared against the map data 108 for each of the plurality of poses. Where a pose does not match up well with the map data 108, the cost may be high, and the point in the cost space corresponding to the pose may be represented as such—e.g., represented in red, or with respect to FIG. 9A, represented in non-dotted or white portions. Where a pose does match up well with the map data 108, the cost may be low, and the point in the cost space corresponding to the pose may be represented as such—e.g., represented in green, or with respect to FIG. 9A, represented by the dotted points. For example, with reference to FIG. 9A, where the cost space 902 corresponds to visualization 1002 of FIG. 10A, the dotted portions 908 may correspond to the low cost for the poses along the diagonal where a sign 1010 may match up well with the predictions or outputs 204 of the DNNs 202. For example, at a pose on the left bottom of the dotted portions of the cost space, the predictions of the sign may line up well with the sign 1010 from the map data 108, and similarly on the upper right portion of the dotted portions, the predictions of the sign from the corresponding poses may also line up well with the sign 1010. As such, these points may be represented with low cost. However, due to noise and the high number of low cost poses, a single cost space 902 may not be accurate for localization—e.g., the vehicle 1500 cannot be located at each of the poses represented by the dotted portions 908. As such, an aggregate cost space 904 may be generated via cost space aggregation 804, as described herein.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the apparatus of Chang in view of Kawasaki in view of Garimella to determine, based on a comparison between nodes in each pair of the one or more pairs of matched nodes between a 2D or 3D object representation and the HD map, a respective node score for each pair of the one or more pairs of matched nodes; and determine, based on the respective node score for each pair of the one or more pairs of matched nodes between a 2D or 3D object representation and the HD map, a location of the object within the environment, as taught by Kroepfl as disclosed above, such that the determination of the node score and object location is made between a comparison between the predicted map and the HD map, in order to ensure accurate object detection and localization (Kroepfl Paragraph 6 “Embodiments of the present disclosure relate to approaches for map creation and localization for autonomous driving applications”).
With respect to claim 2, and similarly claim 14, Chang in view of Kawasaki in view of Garimella in view of Kroepfl teach that the one or more sensors comprise at least one of one or more cameras, one or more radar sensors, or one or more light detection and ranging (LIDAR) sensors (See at least Chang Paragraph 84 “The capturing device is fixed to a location on the vehicle such as, for example, a windshield, a dashboard, or a rear-view mirror of the vehicle, to capture driving images of a view in front of the vehicle. The capturing device includes, for example, a vision sensor, an image sensor, or a device that performs a similar function.” | Paragraph 148 “The sensor(s) 1310 include, for example, an image sensor, a vision sensor, an acceleration sensor, a gyro sensor, a GPS sensor, an IMU sensor, a Radar, and a Lidar. The sensor(s) 1310 acquire or capture an input image including a driving image of a vehicle. The sensor(s) 1310 senses information such as, for example, a speed, an acceleration, a travelling direction, and a steering angle of the vehicle, in addition to localization information such as, for example, GPS coordinates, a position, and an orientation of the vehicle is sensed by the sensor(s) 1310”).
With respect to claim 5, and similarly claim 17, Chang in view of Kawasaki in view of Garimella in view of Kroepfl teach that the at least one processor is configured to match at least one of the plurality of predicted nodes with at least one of the plurality of HD nodes using a machine learning system (See at least Chang Paragraphs 143-145 “In an example, the neural network 1230 is trained to generate a first image including a directivity corresponding to an object included in the input image 1210 based on the input image 1210. The neural network 1230 is implemented on a hardware-based model comprising a framework or a structure of a number of layers or operations to provide for many different machine learning algorithms to work together, process complex data inputs, and recognize patterns. The neural network 1230 is implemented in various structures such as, for example, a convolutional neural network (CNN), a deep neural network (DNN), an n-layer neural network, a recurrent neural network (RNN), or a bidirectional long short term memory (BLSTM). The DNN includes, for example, a fully connected network, a CNN, a deep convolutional network, or a recurrent neural network (RNN), a deep belief network, a bi-directional neural network, a restricted Boltzman machine, or may include different or overlapping neural network portions respectively with full, convolutional, recurrent, and/or bi-directional connections. The neural network 1230 maps, based on deep learning, input data and output data that are in a non-linear relationship, to perform, for example, an object classification, an object recognition, a speech recognition, or an image recognition. The neural network may be implemented as an architecture having a plurality of layers including an input image, feature maps, and an output. In the neural network, a convolution operation between the input image, and a filter referred to as a kernel, is performed, and as a result of the convolution operation, the feature maps are output. Here, the feature maps that are output are input feature maps, and a convolution operation between the output feature maps and the kernel is performed again, and as a result, new feature maps are output. Based on such repeatedly performed convolution operations, results of recognition of characteristics of the input image via the neural network may be output. In an example, the neural network 1230 estimates the object included in the input image 1210 in a form of the distance field map 1250. For example, when the first image includes directivity information toward a close object as in the distance field map 1250, a directivity of optimization is determined by utilizing gradient descent. Further, when a probability distribution indicating a degree of closeness to the object is present all over the image as in the distance field map 1250, an amount of data for training increases, and thus the performance of the neural network improves when compared to a case of training with sparse data.”).
With respect to claim 7, and similarly claims 19 and 27, Chang in view of Kawasaki in view of Garimella in view of Kroepfl teach that the comparison between the nodes in each pair of the one or more pairs of matched nodes is based on a displacement between the nodes in each pair of the one or more pairs of matched nodes (See at least Chang Paragraph 135-136 “A localization apparatus calculates a score by matching the first image 1110 and the second image 1120 as shown in an image 1130. The localization apparatus calculates the score by summing up values of pixels corresponding to the object included in the second image 1120, among a plurality of pixels included in the first image 1110. For example, the plurality of pixels included in the first image 1110 has values between “0” and “1” based on distances to an adjacent object. Each pixel has a value close to “1” as being close to the adjacent object and has a value close to “0” as being far from the adjacent object. The localization apparatus extracts pixels matching the second image 1120 from the plurality of pixels included in the first image 1110, and calculates the score by summing up values of the extracted pixels.”).
With respect to claim 8, and similarly claims 20 and 28, Chang in view of Kawasaki in view of Garimella in view of Kroepfl teach that the predicted map comprises a plurality of predicted polylines, each polyline of the plurality of predicted polylines connecting at least two nodes of the plurality of predicted nodes, and wherein the HD map comprises a plurality of HD polylines, each HD polyline of the plurality of HD polylines connecting at least two HD nodes of the plurality of HD nodes (See at least Hess Paragraphs 25-27 “In particular embodiments, a feature extracted from an image may include a number of two-dimensional vectors or/and three-dimensional vectors in an area of interest in the image. The computing system may firstly identify the area of interest associated with the feature. For example, the computing system may determine a point of interest in the image and determine an area around the point of interest using a particular geometric shape (e.g., a circle, a square, a triangle, a polygon, etc.) and with a particular size. In particular embodiments, the computing system may determine a number of point pairs based on one or more rules and the geometric shape of the area of interest. Each point pair may include a point and an opposite point. For example, for an edge line, the points and the respective opposite points may be located at different side of the edge line. The two-dimensional or three-dimensional vectors may be determined based on the difference between the pixel value at these points and respective opposite points. For example, the computing system may subtract the values at these points to the values at respective opposite points to determine the corresponding two-dimensional or three-dimensional vectors. The two-dimensional or three-dimensional vectors may only have relative small changes when the images are captured under different lighting conditions, and therefore may be reliably used to identify, compare, and calibrate the associated features. In particular embodiments, each image captured by the camera 212 may be associated with a GPS location as determined by a GPS sensor of the vehicle 210 to indicate where the image is captured. The GPS location together with the image itself may be used to determine relative positions (e.g., distances, directions) of objects or features in surrounding environment with respect to the vehicle 210. For example, the distance and geometric relationship between the vehicle 210 and the billboard 230 may be determined based on one or more images captured by the camera 212 using computer vision algorithms. The accurate location of objects or features captured in the image may be further determined based on the relative positions and the GSP location of the vehicle 210. The location information may be used to identify areas in high definition maps that contain the high definition features matching the features extracted from the images. In particular embodiments, the vehicle 210 may localize itself with respect to one or more objects in the surrounding environment. For example, the vehicle 210 may determine the precise position of the billboard 230 based on one or more images captured by the camera 212. The vehicle 210 may further determine, from multiple view angles, precise geometric properties (e.g., a corner position, a corner angle, an edge direction, an edge length, etc.) of the billboard 230 in the three-dimensional space based on a high definition map, calibrated images, or/and uncalibrated images captured by the camera 212. The geometric properties of the billboard 230 may be determined with respect to the vehicle pose. The vehicle 210 may determine its own position in the three-dimensional space based on the GPS location and the precise geometric properties of one or more objects in the surrounding environment. In particular embodiments, the position of the vehicle 210 or an object in the surrounding environment may be determined with accuracy and precision at one-centimeter level. In particular embodiments, vehicle 210 may determine its location using, for example, a GPS sensor, simultaneous location and mapping (SLAM), visual SLAM, dead reckoning, or other localization techniques.”).
With respect to claim 11, and similarly claim 23, Chang in view of Kawasaki in view of Garimella in view of Kroepfl teach that the HD map comprises vectorized representations of the environment (See at least Hess Paragraph 14 “To solve the aforementioned problems, particular embodiments may generate a calibration model for each vehicle camera based on a reference map (e.g., high definition map) and use the calibration model to calibrate images captured by the camera. In particular embodiments, a computing system, which may be located on the vehicle itself or on a remote server(s), may extract a number of features (e.g., an object, a corner, a shape, a pattern, a three-dimension vector, etc.) from one or more images captured by a vehicle camera. The extracted features may be associated with one or more objects captured in the images. The system may identify, in the reference map, one or more reference features (e.g., high definition features) that match (e.g., being associated with the same objects) the features extracted from the images”).
With respect to claim 13, Chang teaches a method for localizing an object, the method comprising:
receive a high definition (HD) map comprising a plurality of HD nodes associated with a HD location of the object within the environment (See at least Chang Paragraph 85 “In an example, the map data is high density (HD) map data. An HD map is a 3D map with a high density, for example, a centimeter-level density, that may be used for autonomous driving. The HD map includes, for example, line information related to a road center line and a boundary line, and information related to a traffic light, a traffic sign, a curb, a road surface marking, and various structures in a form of 3D digital data. The HD map is established by, for example, a mobile mapping system (MMS). The MMS, a 3D space information investigation system equipped with various sensors, obtains minute position information using a moving object equipped with sensors such as a camera, a lidar, and a GPS to measure a position and geographic features.”);
Chang fails to explicitly disclose generate, based on sensor data obtained from one or more sensors associated with the object, a predicted map comprising a plurality of predicted nodes associated with a predicted location of the object within an environment; match at least one node in a first graph representation of the plurality of predicted nodes with at least one node in a second graph representation of the plurality of HD nodes to determine one or more pairs of matched nodes between the predicted map and the HD map; determine, based on a comparison between nodes in each pair of the one or more pairs of matched nodes between the predicted map and the HD map, a respective node score for each pair of the one or more pairs of matched nodes; and determine, based on the respective node score for each pair of the one or more pairs of matched nodes between the predicted map and the HD map, a location of the object within the environment.
Kawasaki teaches to generate, based on sensor data obtained from one or more sensors associated with the object, a predicted map comprising a plurality of predicted nodes associated with a predicted location of the object within an environment (See at least Kawasaki Paragraphs 50-52 “The environment map generator 103 generates an environment map. The environment map includes map information expressing, on a map, an environment around the moving object at a reference time point. For example, the environment map generator 103 generates an environment map using the moving object information and the environmental information at the reference time point … The cumulative map generator 104 generates a cumulative map. The cumulative map includes map information expressing, on a map, a plurality of positions indicated by the moving object information acquired at a plurality of time points (first time point) equal to or earlier than the reference time point. In this manner, the cumulative map is generated so as to cumulatively store, on the map, not only the position at the reference time point but also the positions indicated by the moving object information acquired earlier than the reference time”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of Chang to generate, based on sensor data obtained from one or more sensors associated with the object, a predicted map comprising a plurality of predicted nodes associated with a predicted location of the object within an environment, as taught by Kawasaki as disclosed above, in order to ensure accurate object detection in dynamic environments (Kawasaki Paragraph 22 “In view of this, there has been a demand for a technique capable of predicting the position of other moving objects existing in the surroundings by using information obtained from sensors such as cameras and laser sensors attached to moving objects (automobiles, autonomous mobile robots, or the like) alone without using map information prepared in advance.”).
Chang in view of Kawasaki fails to explicitly disclose match at least one node in a first graph representation of the plurality of predicted nodes with at least one node in a second graph representation of the plurality of HD nodes to determine one or more pairs of matched nodes between the predicted map and the HD map; determine, based on a comparison between nodes in each pair of the one or more pairs of matched nodes between the predicted map and the HD map, a respective node score for each pair of the one or more pairs of matched nodes; and determine, based on the respective node score for each pair of the one or more pairs of matched nodes between the predicted map and the HD map, a location of the object within the environment.
Garimella teaches match at least one node in a first graph representation of the plurality of predicted nodes with at least one node in a second graph representation of the plurality of HD nodes to determine one or more pairs of matched nodes between the predicted map and the HD map (See at least Garimella FIG. 7 and Paragraph 17 “In some examples, the vectorization component of the autonomous vehicle may determine a rotational offset for a map element, before or during the encoding process. For instance, the vectorization component may match a map element detected in the environment to a map element template, based on the size, shape, and/or type of the element. To illustrate, crosswalks 110, 112, and 114 each may be matched to a common crosswalk template having a predetermined shape and/or orientation. After matching crosswalks 110-114 to the crosswalk template, the vectorization component may determine a rotational offset for each of the crosswalks 110-114, to represent the difference between the orientation of the crosswalk and the predetermined orientation of the crosswalk template. A rotational offset for an individual crosswalks (or other map element) may be stored as an attribute associated with the vectorized representation of the crosswalk, thereby allowing each crosswalk in the environment to be encoded in the same predetermined orientation, to reduce variation and improve the efficiency and accuracy of the encoding process. As described below, rotational offsets and/or other types of canonical local coordinate frames (e.g., size and shape offsets) may be determined for any map element represented by a polyline, and these offsets (e.g., canonical local frames) may be stored as attributed of the nodes and/or edge features within a graph structure (e.g., a GNN) used to represent the environment. Of course, though depicted in FIG. 1 for illustrative purposes as being performed after receiving the map elements, it is contemplated that such map elements may be previously vectorized such that the vectorized elements may be received directly.” | Paragraph 68 “Each of the individual line segment vectors representing the lane 302 may be provided to a template converter/encoder 308 of the vectorization component 230. As shown in this example, the template converter/encoder 308 may be implemented into a single system, network, etc., but in other examples a template converter and encoder may be implemented separately. In some cases, each line segment vector may be provided separately to the template converter/encoder 308, while in other cases the associated line segment vectors for a lane object (or other map element) may be provided as a group to the template converter/encoder 308. The template converter/encoder 308 in this example may include the functionality to match the map element (e.g., lane) to an associated map element template (e.g., a lane template). Based on the matching, the template converter/encoder 308 may convert the individual line segment vectors 304 and 306 into vectors complying with the map element template. As shown in box 310, in some cases the template converter/encoder 308 may determine a rotational offset between a lane segment and/or polyline 312 based on a template, and may convert segment and/or polyline to a rotated lane segment and/or polyline 314 that corresponds to the predetermined orientation of the template.” | Paragraphs 96-98 “If the GNN component 232 determines in operation 710 that an entity node is not to be added or updated to the GNN based on the perceived entities (710: No), then process 700 may return to operation 708 to await additional perception data from the sensor and perception systems of the autonomous vehicle. However, if the GNN component 232 determines in operation 710 that a new entity node is to be added to the GNN or an existing entity node is to be updated (710: Yes), then at operation 712 one or more entity nodes may be generated and/or updated based on the perceived entities, within the appropriate graph structure(s) of the GNN. As described above, to add a new entity node to the GNN, the vectorization component 230 may define one or more polylines representing the entity, vectorize the individual line segments of the polylines, convert and/or encode the vectorized polylines, and then aggregate the encoded polylines into a node data structure compatible with the GNN. At operation 714, the GNN component 232 may determine edge feature data and generate and/or update any edge features necessary based on any nodes added or updated in operations 706 and/or 712. In some examples, for any new node and/or when the position or orientation of an existing entity node changes, the GNN component 232 may determine updated relative positions values and/or updated relative yaw values for each edge connected to the affected node. At operation 716, the entity prediction component 234 may use the GNN to determine a predicted future states for one or more entities represented in the GNN. As described above, the GNN may be configured to execute one or more inference process to update the node states and/or edge features of the GNN, providing an inferred (or predicted) future state representation for all of the objects in the environment. In some examples, entity prediction component 234 may access the updated state of the GNN, retrieve and decode one or more portions of the updated GNN, and use the updated GNN data determine predicted entity states. Predictions of states for entities may include predicted locations, velocities, trajectories, orientations, and the like, at various time intervals beyond the current time. In some examples, to perform a prediction of an entity state based on an updated GNN, the entity prediction component 234 may select and decode specific portions of the GNN including the entity and selected edges/nodes that are more closely positioned or related to the entity, rather than decoding the entire GNN.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of Chang in view of Kawasaki to match at least one node in a first graph representation of the plurality of predicted nodes with at least one node in a second graph representation of the plurality of HD nodes to determine one or more pairs of matched nodes between the predicted map and the HD map, as taught by Garimella as disclosed above, in order to ensure accurate object detection (Garimella Paragraph 11 “As discussed below, the various techniques described herein provide technical improvements in the environment modeling and predictive capabilities of the autonomous vehicles, as well as technical advantages in reducing computing resources and improving efficiency of the prediction and decision-making components of autonomous vehicles.”).
Chang in view of Kawasaki in view of Garimella fails to explicitly disclose to determine, based on a comparison between nodes in each pair of the one or more pairs of matched nodes between the predicted map and the HD map, a respective node score for each pair of the one or more pairs of matched nodes; and determine, based on the respective node score for each pair of the one or more pairs of matched nodes between the predicted map and the HD map, a location of the object within the environment.
Kroepfl teaches to determine, based on a comparison between nodes in each pair of the one or more pairs of matched nodes between a 2D or 3D object representation and the HD map, a respective node score for each pair of the one or more pairs of matched nodes; and determine, based on the respective node score for each pair of the one or more pairs of matched nodes between a 2D or 3D object representation and the HD map, a location of the object within the environment (See at least Kroepfl FIG. 9A and Paragraph 135 “With reference to FIG. 9A, cost space 902 may represent a cost space for a sensor modality—e.g., a camera based cost space generated according to FIGS. 10A-10C. For example, the cost space 902 may represent the likelihood that a vehicle 1500 is currently in each of a plurality of poses—e.g., represented by points of the cost space 902—at a current frame. The cost space 902, although represented in 2D in FIG. 9A, may correspond to a 3D cost space (e.g., with (x, y, z) locations and/or axis angles for each of the x, y, and z axes). As such, the sensor data 102—e.g., before or after pre-processing—and/or the outputs 204 (e.g., detections of landmark locations in 2D image space and/or 3D image space) may be compared against the map data 108 for each of the plurality of poses. Where a pose does not match up well with the map data 108, the cost may be high, and the point in the cost space corresponding to the pose may be represented as such—e.g., represented in red, or with respect to FIG. 9A, represented in non-dotted or white portions. Where a pose does match up well with the map data 108, the cost may be low, and the point in the cost space corresponding to the pose may be represented as such—e.g., represented in green, or with respect to FIG. 9A, represented by the dotted points. For example, with reference to FIG. 9A, where the cost space 902 corresponds to visualization 1002 of FIG. 10A, the dotted portions 908 may correspond to the low cost for the poses along the diagonal where a sign 1010 may match up well with the predictions or outputs 204 of the DNNs 202. For example, at a pose on the left bottom of the dotted portions of the cost space, the predictions of the sign may line up well with the sign 1010 from the map data 108, and similarly on the upper right portion of the dotted portions, the predictions of the sign from the corresponding poses may also line up well with the sign 1010. As such, these points may be represented with low cost. However, due to noise and the high number of low cost poses, a single cost space 902 may not be accurate for localization—e.g., the vehicle 1500 cannot be located at each of the poses represented by the dotted portions 908. As such, an aggregate cost space 904 may be generated via cost space aggregation 804, as described herein.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the apparatus of Chang in view of Kawasaki in view of Garimella to determine, based on a comparison between nodes in each pair of the one or more pairs of matched nodes between a 2D or 3D object representation and the HD map, a respective node score for each pair of the one or more pairs of matched nodes; and determine, based on the respective node score for each pair of the one or more pairs of matched nodes between a 2D or 3D object representation and the HD map, a location of the object within the environment, as taught by Kroepfl as disclosed above, such that the determination of the node score and object location is made between a comparison between the predicted map and the HD map, in order to ensure accurate object detection and localization (Kroepfl Paragraph 6 “Embodiments of the present disclosure relate to approaches for map creation and localization for autonomous driving applications”).
With respect to claim 25 Chang teaches a non-transitory computer-readable storage medium comprising instructions stored thereon which, when executed by at least one processor, causes the at least one processor to:
receive a high definition (HD) map comprising a plurality of HD nodes associated with a HD location of the object within the environment (See at least Chang Paragraph 85 “In an example, the map data is high density (HD) map data. An HD map is a 3D map with a high density, for example, a centimeter-level density, that may be used for autonomous driving. The HD map includes, for example, line information related to a road center line and a boundary line, and information related to a traffic light, a traffic sign, a curb, a road surface marking, and various structures in a form of 3D digital data. The HD map is established by, for example, a mobile mapping system (MMS). The MMS, a 3D space information investigation system equipped with various sensors, obtains minute position information using a moving object equipped with sensors such as a camera, a lidar, and a GPS to measure a position and geographic features.”);
Chang fails to explicitly disclose generate, based on sensor data obtained from one or more sensors associated with the object, a predicted map comprising a plurality of predicted nodes associated with a predicted location of the object within an environment; match at least one node in a first graph representation of the plurality of predicted nodes with at least one node in a second graph representation of the plurality of HD nodes to determine one or more pairs of matched nodes between the predicted map and the HD map; determine, based on a comparison between nodes in each pair of the one or more pairs of matched nodes between the predicted map and the HD map, a respective node score for each pair of the one or more pairs of matched nodes; and determine, based on the respective node score for each pair of the one or more pairs of matched nodes between the predicted map and the HD map, a location of the object within the environment.
Kawasaki teaches to generate, based on sensor data obtained from one or more sensors associated with the object, a predicted map comprising a plurality of predicted nodes associated with a predicted location of the object within an environment (See at least Kawasaki Paragraphs 50-52 “The environment map generator 103 generates an environment map. The environment map includes map information expressing, on a map, an environment around the moving object at a reference time point. For example, the environment map generator 103 generates an environment map using the moving object information and the environmental information at the reference time point … The cumulative map generator 104 generates a cumulative map. The cumulative map includes map information expressing, on a map, a plurality of positions indicated by the moving object information acquired at a plurality of time points (first time point) equal to or earlier than the reference time point. In this manner, the cumulative map is generated so as to cumulatively store, on the map, not only the position at the reference time point but also the positions indicated by the moving object information acquired earlier than the reference time”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the apparatus of Chang to generate, based on sensor data obtained from one or more sensors associated with the object, a predicted map comprising a plurality of predicted nodes associated with a predicted location of the object within an environment, as taught by Kawasaki as disclosed above, in order to ensure accurate object detection in dynamic environments (Kawasaki Paragraph 22 “In view of this, there has been a demand for a technique capable of predicting the position of other moving objects existing in the surroundings by using information obtained from sensors such as cameras and laser sensors attached to moving objects (automobiles, autonomous mobile robots, or the like) alone without using map information prepared in advance.”).
Chang in view of Kawasaki fails to explicitly disclose match at least one node in a first graph representation of the plurality of predicted nodes with at least one node in a second graph representation of the plurality of HD nodes to determine one or more pairs of matched nodes between the predicted map and the HD map; determine, based on a comparison between nodes in each pair of the one or more pairs of matched nodes between the predicted map and the HD map, a respective node score for each pair of the one or more pairs of matched nodes; and determine, based on the respective node score for each pair of the one or more pairs of matched nodes between the predicted map and the HD map, a location of the object within the environment.
Garimella teaches match at least one node in a first graph representation of the plurality of predicted nodes with at least one node in a second graph representation of the plurality of HD nodes to determine one or more pairs of matched nodes between the predicted map and the HD map (See at least Garimella FIG. 7 and Paragraph 17 “In some examples, the vectorization component of the autonomous vehicle may determine a rotational offset for a map element, before or during the encoding process. For instance, the vectorization component may match a map element detected in the environment to a map element template, based on the size, shape, and/or type of the element. To illustrate, crosswalks 110, 112, and 114 each may be matched to a common crosswalk template having a predetermined shape and/or orientation. After matching crosswalks 110-114 to the crosswalk template, the vectorization component may determine a rotational offset for each of the crosswalks 110-114, to represent the difference between the orientation of the crosswalk and the predetermined orientation of the crosswalk template. A rotational offset for an individual crosswalks (or other map element) may be stored as an attribute associated with the vectorized representation of the crosswalk, thereby allowing each crosswalk in the environment to be encoded in the same predetermined orientation, to reduce variation and improve the efficiency and accuracy of the encoding process. As described below, rotational offsets and/or other types of canonical local coordinate frames (e.g., size and shape offsets) may be determined for any map element represented by a polyline, and these offsets (e.g., canonical local frames) may be stored as attributed of the nodes and/or edge features within a graph structure (e.g., a GNN) used to represent the environment. Of course, though depicted in FIG. 1 for illustrative purposes as being performed after receiving the map elements, it is contemplated that such map elements may be previously vectorized such that the vectorized elements may be received directly.” | Paragraph 68 “Each of the individual line segment vectors representing the lane 302 may be provided to a template converter/encoder 308 of the vectorization component 230. As shown in this example, the template converter/encoder 308 may be implemented into a single system, network, etc., but in other examples a template converter and encoder may be implemented separately. In some cases, each line segment vector may be provided separately to the template converter/encoder 308, while in other cases the associated line segment vectors for a lane object (or other map element) may be provided as a group to the template converter/encoder 308. The template converter/encoder 308 in this example may include the functionality to match the map element (e.g., lane) to an associated map element template (e.g., a lane template). Based on the matching, the template converter/encoder 308 may convert the individual line segment vectors 304 and 306 into vectors complying with the map element template. As shown in box 310, in some cases the template converter/encoder 308 may determine a rotational offset between a lane segment and/or polyline 312 based on a template, and may convert segment and/or polyline to a rotated lane segment and/or polyline 314 that corresponds to the predetermined orientation of the template.” | Paragraphs 96-98 “If the GNN component 232 determines in operation 710 that an entity node is not to be added or updated to the GNN based on the perceived entities (710: No), then process 700 may return to operation 708 to await additional perception data from the sensor and perception systems of the autonomous vehicle. However, if the GNN component 232 determines in operation 710 that a new entity node is to be added to the GNN or an existing entity node is to be updated (710: Yes), then at operation 712 one or more entity nodes may be generated and/or updated based on the perceived entities, within the appropriate graph structure(s) of the GNN. As described above, to add a new entity node to the GNN, the vectorization component 230 may define one or more polylines representing the entity, vectorize the individual line segments of the polylines, convert and/or encode the vectorized polylines, and then aggregate the encoded polylines into a node data structure compatible with the GNN. At operation 714, the GNN component 232 may determine edge feature data and generate and/or update any edge features necessary based on any nodes added or updated in operations 706 and/or 712. In some examples, for any new node and/or when the position or orientation of an existing entity node changes, the GNN component 232 may determine updated relative positions values and/or updated relative yaw values for each edge connected to the affected node. At operation 716, the entity prediction component 234 may use the GNN to determine a predicted future states for one or more entities represented in the GNN. As described above, the GNN may be configured to execute one or more inference process to update the node states and/or edge features of the GNN, providing an inferred (or predicted) future state representation for all of the objects in the environment. In some examples, entity prediction component 234 may access the updated state of the GNN, retrieve and decode one or more portions of the updated GNN, and use the updated GNN data determine predicted entity states. Predictions of states for entities may include predicted locations, velocities, trajectories, orientations, and the like, at various time intervals beyond the current time. In some examples, to perform a prediction of an entity state based on an updated GNN, the entity prediction component 234 may select and decode specific portions of the GNN including the entity and selected edges/nodes that are more closely positioned or related to the entity, rather than decoding the entire GNN.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the apparatus of Chang in view of Kawasaki to match at least one node in a first graph representation of the plurality of predicted nodes with at least one node in a second graph representation of the plurality of HD nodes to determine one or more pairs of matched nodes between the predicted map and the HD map, as taught by Garimella as disclosed above, in order to ensure accurate object detection (Garimella Paragraph 11 “As discussed below, the various techniques described herein provide technical improvements in the environment modeling and predictive capabilities of the autonomous vehicles, as well as technical advantages in reducing computing resources and improving efficiency of the prediction and decision-making components of autonomous vehicles.”).
Chang in view of Kawasaki in view of Garimella fails to explicitly disclose to determine, based on a comparison between nodes in each pair of the one or more pairs of matched nodes between the predicted map and the HD map, a respective node score for each pair of the one or more pairs of matched nodes; and determine, based on the respective node score for each pair of the one or more pairs of matched nodes between the predicted map and the HD map, a location of the object within the environment.
Kroepfl teaches to determine, based on a comparison between nodes in each pair of the one or more pairs of matched nodes between a 2D or 3D object representation and the HD map, a respective node score for each pair of the one or more pairs of matched nodes; and determine, based on the respective node score for each pair of the one or more pairs of matched nodes between a 2D or 3D object representation and the HD map, a location of the object within the environment (See at least Kroepfl FIG. 9A and Paragraph 135 “With reference to FIG. 9A, cost space 902 may represent a cost space for a sensor modality—e.g., a camera based cost space generated according to FIGS. 10A-10C. For example, the cost space 902 may represent the likelihood that a vehicle 1500 is currently in each of a plurality of poses—e.g., represented by points of the cost space 902—at a current frame. The cost space 902, although represented in 2D in FIG. 9A, may correspond to a 3D cost space (e.g., with (x, y, z) locations and/or axis angles for each of the x, y, and z axes). As such, the sensor data 102—e.g., before or after pre-processing—and/or the outputs 204 (e.g., detections of landmark locations in 2D image space and/or 3D image space) may be compared against the map data 108 for each of the plurality of poses. Where a pose does not match up well with the map data 108, the cost may be high, and the point in the cost space corresponding to the pose may be represented as such—e.g., represented in red, or with respect to FIG. 9A, represented in non-dotted or white portions. Where a pose does match up well with the map data 108, the cost may be low, and the point in the cost space corresponding to the pose may be represented as such—e.g., represented in green, or with respect to FIG. 9A, represented by the dotted points. For example, with reference to FIG. 9A, where the cost space 902 corresponds to visualization 1002 of FIG. 10A, the dotted portions 908 may correspond to the low cost for the poses along the diagonal where a sign 1010 may match up well with the predictions or outputs 204 of the DNNs 202. For example, at a pose on the left bottom of the dotted portions of the cost space, the predictions of the sign may line up well with the sign 1010 from the map data 108, and similarly on the upper right portion of the dotted portions, the predictions of the sign from the corresponding poses may also line up well with the sign 1010. As such, these points may be represented with low cost. However, due to noise and the high number of low cost poses, a single cost space 902 may not be accurate for localization—e.g., the vehicle 1500 cannot be located at each of the poses represented by the dotted portions 908. As such, an aggregate cost space 904 may be generated via cost space aggregation 804, as described herein.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the apparatus of Chang in view of Kawasaki in view of Garimella to determine, based on a comparison between nodes in each pair of the one or more pairs of matched nodes between a 2D or 3D object representation and the HD map, a respective node score for each pair of the one or more pairs of matched nodes; and determine, based on the respective node score for each pair of the one or more pairs of matched nodes between a 2D or 3D object representation and the HD map, a location of the object within the environment, as taught by Kroepfl as disclosed above, such that the determination of the node score and object location is made between a comparison between the predicted map and the HD map, in order to ensure accurate object detection and localization (Kroepfl Paragraph 6 “Embodiments of the present disclosure relate to approaches for map creation and localization for autonomous driving applications”).
Claims 3-4, 15-16, and 26 are rejected under 35 U.S.C. 103 as being unpatentable over Chang (US 20200134896 A1) (“Chang”) in view of Kawasaki (US 20210383213 A1) (“Kawasaki”) in view of Garimella (US 20220274625 A1) (“Garimella”) in view of Kroepfl (US 20210063200 A1) (“Kroepfl”) further in view of Kim (US 20200217667 A1) (“Kim”).
With respect to claim 3, and similarly claims 15 and 26, Chang in view of Kawasaki in view of Garimella in view of Kroepfl fail to explicitly disclose that the HD map is based on positioning sensor data obtained from one or more positioning sensors associated with the object.
Kim teaches that the HD map is based on positioning sensor data obtained from one or more positioning sensors associated with the object. (See at least Kim Paragraph 35 “The VEPP unit 270 can then combine the VIO position estimate with information from the GNSS unit 230 to provide a highly-accurate vehicle position estimate in a global frame to the map fusion unit 280. The map fusion unit 280 works to provide a vehicle position estimate within a map frame, based on the position estimate from the VEPP unit 270, as well as information from a map database 250 and a perception unit 240. The map database 250 can provide a 3D map (e.g., a high definition (HD) map in the form of one or more electronic files, data objects, etc.) of an area in which the vehicle 110 is located, and the perception unit 240 can make observations of lane markings, traffic signs, and/or other visual features in the vehicle's surroundings. To do so, the perception unit 240 may comprise a feature-extraction engine that performs image processing and computer vision on images received from the camera(s) 210.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the apparatus of Chang in view of Kawasaki in view of Garimella in view of Kroepfl to include that the HD map is based on positioning sensor data obtained from one or more positioning sensors associated with the object, as taught by Kim as disclosed above, in order to ensure an accurate HD map for object localization (Kim Paragraph 4 “Techniques provided herein are directed toward accurately associating observed traffic signs from camera images to a counterpart traffic sign within a 3D map.”).
With respect to claim 4, and similarly claim 16, Chang in view of Kawasaki in view of Garimella in view of Kroepfl in view of Kim teach that the one or more positioning sensors comprises at least one of one or more satellite receivers or one or more inertial measurement units (IMUs) (See at least Kim Paragraph 8 “An example non-transitory computer-readable medium, according to this disclosure, has instructions stored thereby for estimating vehicle position based on an observed traffic sign and 3D map data for the observed traffic sign. The instructions, when executed by one or more processing units, cause the one or more processing units to, obtain location information comprising Global Navigation Satellite System (GNSS) information, Visual Inertial Odometry (VIO) information, or both, obtain observation data indicative of where the observed traffic sign is located within an image of the observed traffic sign taken from a vehicle, and obtain the 3D map data, wherein the 3D map data comprises a location, in a 3D frame, of each of one or more traffic signs in an area in which the vehicle is located.”).
Claims 9-10, 12, 21-22, 24, and 29-30 are rejected under 35 U.S.C. 103 as being unpatentable over Chang (US 20200134896 A1) (“Chang”) in view of Kawasaki (US 20210383213 A1) (“Kawasaki”) in view of Garimella (US 20220274625 A1) (“Garimella”) in view of Kroepfl (US 20210063200 A1) (“Kroepfl”) in view of Kim (US 20200217667 A1) (“Kim”) further in view of Vig (US 20220392235 A1) (“Vig”).
With respect to claim 9, and similarly claims 21 and 29, Chang in view of Kawasaki in view of Garimella in view of Kroepfl in view of Kim fail to explicitly disclose to determine a polyline score for each polyline of the plurality of predicted polylines, based on the respective node score for each pair of the one or more pairs of matched nodes that are associated with the plurality of predicted polylines; and determine the polyline score for each HD polyline of the plurality of HD polylines, based on the respective node score for each pair of the one or more pairs of matched nodes that are associated with the plurality of HD polylines.
Vig teaches to determine a polyline score for each polyline of the plurality of polylines, based on the respective node score for each pair of the one or more pairs of matched nodes that are associated with the plurality of polylines (See at least Vig Paragraph 141 “In some non-limiting embodiments or aspects, map generation system 102 filters a plurality of road edge polylines based on the prediction scores. In some examples, map generation system 102 filters a first road edge polyline generated incorrectly (e.g., a short polyline, a polyline based on a noisy input signal in the roadway, a polyline generated from points beyond the road boundary, etc.). For example, map generation system 102 performs filtering to remove polylines below a length threshold. In some non-limiting embodiments or aspects, map generation system 102 performs a smoothing and/or resampling step to smooth these points to the final polyline.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the apparatus of Chang in view of Kawasaki in view of Garimella in view of Kroepfl in view of Kim to include determining a polyline score for each polyline of the plurality of polylines, based on the respective node score for each pair of the one or more pairs of matched nodes that are associated with the plurality of polylines, as taught by Vig as disclosed above, such that the polyline score is determined for the predicted polylines and the HD polylines, in order to ensure accurate object localization (Vig Paragraph 5 “Provided are systems, devices, products, apparatuses, and/or methods for improving automated road edge boundary detection on a roadway of an autonomous vehicle, improving generation of an autonomous vehicle map to include locations of road edges, improving control of travel in a driving path of an autonomous vehicle based on automated road edge boundary detection, and/or the like”).
With respect to claim 10, and similarly claims 22 and 30, Chang in view of Kawasaki in view of Garimella in view of Kroepfl in view of Kim in view of Vig teach to determine the location of the object within the environment further based on the polyline score determined for each polyline of the plurality of predicted polylines and the polyline score determined for each HD polyline for the plurality of HD polylines (See at least Vig Paragraph 142 “In some non-limiting embodiments or aspects, map generation system 102 generates a road edge boundary of the roadway in the map based on the plurality of prediction scores and/or polylines. As an example, map generation system 102 outputs and/or writes the polylines into a map. In some non-limiting embodiments or aspects, map generation system 102 determines map data associated with a road edge boundary in the roadway (e.g., map data including an image, a map, and/or features including a ground truth representing the road edge boundary therein, etc.).”).
With respect to claim 12, and similarly claim 24, Chang in view of Kawasaki in view of Garimella in view of Kroepfl in view of Kim fail to explicitly disclose that the object is a vehicle.
Vig teaches to determine that the object is a vehicle (See at least Vig Paragraph 67 “In some non-limiting embodiments or aspects, perception system 228 detects and/or tracks objects (e.g., vehicles, pedestrians, bicycles, and/or the like) that are proximate to (e.g., in proximity to the surrounding environment of) autonomous vehicle 104 over a time period. In some non-limiting embodiments or aspects, perception system 228 can retrieve (e.g., obtain) map data from map database 214 that provides detailed information about the surrounding environment of autonomous vehicle 104.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the apparatus of Chang in view of Kawasaki in view of Garimella in view of Kroepfl in view of Kim to include that the object is a vehicle, as taught by Vig as disclosed above, in order to ensure accurate object localization (Vig Paragraph 5 “Provided are systems, devices, products, apparatuses, and/or methods for improving automated road edge boundary detection on a roadway of an autonomous vehicle, improving generation of an autonomous vehicle map to include locations of road edges, improving control of travel in a driving path of an autonomous vehicle based on automated road edge boundary detection, and/or the like”).
Claims 6 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Chang (US 20200134896 A1) (“Chang”) in view of Kawasaki (US 20210383213 A1) (“Kawasaki”) in view of Garimella (US 20220274625 A1) (“Garimella”) in view of Kroepfl (US 20210063200 A1) (“Kroepfl”)further in view of Ramezani (US 20220076432 A1) (“Ramezani”).
With respect to claim 6, and similarly claim 18, Chang in view of Kawasaki in view of Garimella in view of Kroepfl teach the use of a machine learning system (See at least Chang Paragraphs 143-145).
Chang in view of Kawasaki in view of Garimella in view of Kroepfl fail to explicitly disclose that the machine learning system is a graph neural network.
Ramezani teaches that the machine learning system is a graph neural network (See at least Ramezani Paragraph 8 “The multi-object tracking architecture is also configured to construct a message passing graph in which each of a multiplicity of layers corresponds to a respective one in the sequence of image by: generating, for each of the layers, a plurality of feature nodes to represent features detected in the corresponding image, and generating edges that interconnect at least some of the feature nodes across adjacent layers of the graph neural network to represent associations between the features. The multi-object tracking architecture is configured to track multiple features through the sequence of images, including passing messages in a forward direction and a backward direction through the message passing graph to share information across time.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the apparatus of Chang in view of Kawasaki in view of Garimella in view of Kroepfl to include that the machine learning system is a graph neural network, as taught by Ramezani as disclosed above, in order to ensure accurate node predictions (Ramezani Paragraph 25 “Generally speaking, the techniques of this disclosure allow a system to efficiently track multiple objects across sensor data collected at different times”).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to IBRAHIM ABDOALATIF ALSOMAIRY whose telephone number is (571)272-5653. The examiner can normally be reached M-F 7:30-5:30.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Faris Almatrahi can be reached at 313-446-4821. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/IBRAHIM ABDOALATIF ALSOMAIRY/ Examiner, Art Unit 3667 /KENNETH J MALKOWSKI/Primary Examiner, Art Unit 3667
Prosecution Insights