DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claims 1, 3-7, 9-14 and 16-23 have been examined.
Claims 2, 8 and 15 have been canceled.
Claims 20-23 have been added.
P = paragraph e.g. P[0001] = paragraph[0001]
Examiner’s Note:
The 07/23/2026 claim amendments have rendered moot the rejections under 35 U.S.C. 101, as Claims 1, 14, 18 and 19 have been amended to recite a step of controlling “based on the trajectory planning, automated driving of the vehicle”, which is not a step that can be performed mentally and which is a step that provides a practical application and that cannot be performed by a generic computer.
Response to Arguments
Applicant’s arguments filed 07/23/2026 have been considered but are moot in view of the new ground(s) of rejection.
Claim Interpretation
As per Claim 11, the limitations “for training a visual foundation model for generating a road map based on received image input data”, “for generating a road map based on received image input data” and “to generate, from the top view image, a generated machine-readable graph representation of the generated road map corresponding to the ground-truth machine-readable graph representation” are each directed to an intended use that does not further limit the claim.
As per Claim 16, the limitations “for training a visual foundation model for generating a road map based on received image input data”, “for generating a road map based on received image input data” and “to generate, from the top view image, a generated machine-readable graph representation of the generated road map corresponding to the ground-truth machine-readable graph representation” are each directed to an intended use that does not further limit the claim.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 11, 12, 14, 16, 17, 19, 20 and 23 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
As per Claim 11, it is unclear how or if the limitation “the training comprises training the graph generative model to generate, from the top view image, a generated machine-readable graph representation of the generated road map corresponding to the ground-truth machine-readable graph representation” is intended to further limit the limitation “training a visual foundation model comprising a graph generative model for generating a road map based on received image input data”, as the “training” step does not recite receiving any use of or receiving of a “top view image” and merely states “for generating a road map based on received image input data”, which is an intended use that is based on “received image input data”, not on a “top view image”. Therefore, it is unclear how or if “the training comprises training the graph generative model to generate, from the top view image, a generated machine-readable graph representation of the generated road map corresponding to the ground-truth machine-readable graph representation” further limits “training a visual foundation model comprising a graph generative model for generating a road map based on received image input data”, and it is unclear if “the training” refers to some additional step of training that is not claimed, or refers only to the intended use preamble limitation of “for training a visual foundation model for generating a road map based on received image input data”.
Therefore, the claim is unclear.
As per Claim 16, it is unclear how or if the limitation “the training comprises training the graph generative model to generate, from the top view image, a generated machine-readable graph representation of the generated road map corresponding to the ground-truth machine-readable graph representation” is intended to further limit the limitation “a visual foundation model training module configured to train a visual foundation model comprising a graph generative model for generating a road map based on received image input data”, as the “visual foundation model training module” is not recited as being configured to use or receive a “top view image” and merely states “for generating a road map based on received image input data”, which is an intended use that is based on “received image input data”, not on a “top view image”. Therefore, it is unclear how or if “the training comprises training the graph generative model to generate, from the top view image, a generated machine-readable graph representation of the generated road map corresponding to the ground-truth machine-readable graph representation” further limits “a visual foundation model training module configured to train a visual foundation model comprising a graph generative model for generating a road map based on received image input data”, and it is unclear if “the training” refers to some additional step of training that is not claimed, or refers only to the intended use preamble limitation of “for training a visual foundation model for generating a road map based on received image input data”.
Therefore, the claim is unclear.
As per Claim 19, the claim recites “a machine-readable graph representation that: is a nested graph including (1) a first graph level in which at least some nodes represent points on lane-boundary lines and at least some edges connect neighboring points on the lane-boundary lines, wherein the lane-boundary lines represented in the first graph level include a line element that separates neighboring lane pieces, and (2) a second graph level in which at least some of nodes represent pieces of lanes and at least some of edges connect the pieces of lanes”.
It is unclear how the limitation “lane-boundary lines” is or is not different from the limitations “lane pieces” and “pieces of lanes”, as a boundary of a lane can presumably be part of the lane itself, particularly when the claim does not clearly recite if a “lane-boundary” is or is not part of a lane or a “piece” of a lane.
Therefore, the claim is unclear.
As per Claim 20, the claim recites “wherein the machine-readable graph representation is a nested graph comprising: a first graph level in which nodes represent points on lane-boundary lines and edges connect neighboring points on the lane-boundary lines; and a second graph level in which nodes represent the pieces of lanes and edges connect the pieces of lanes”.
It is unclear how the limitation “points on lane-boundary lines” is or is not different from the limitation “pieces of lanes”, as a boundary of a lane can presumably be part of the lane itself, particularly when the claim does not clearly recite if a “lane-boundary” is or is not part of a lane or a “piece” of a lane.
Therefore, the claim is unclear.
Claim 14 recites the limitation "the at least one sensor system" in line 5. There is insufficient antecedent basis for this limitation in the claim.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 4, 7, 10, 13, 14, 18 and 22 are rejected under 35 U.S.C. 103 as being unpatentable over Khadem et al. (12,360,213) in view of Kapil et al. (2024/0263964).
Regarding Claim 1, Khadem et al. teaches the claimed computer-implemented method performed by a computing system that includes a computing device and a controller (see col.20, particularly lines 43-67 and col.21, particularly lines 1-3 and “The computing device 404 may include one or more processor(s) 416 and memory 418 communicatively coupled with the one or more processor(s) 416”, see col.21, particularly lines 4-21)
receiving, by one or more processors of the computing device, image input data, wherein the image input data the vehicle (“…the example process 100 may include a semantic classifier 128 configured to accept the 2D image 124 as input…”, see col.8, particularly lines 43-59 and FIG. 1);
generating, by the one or more processors, , and using a trained visual foundation model for road map generation (“…output a semantically labeled image 130, see col.8, particularly lines 43-59 and FIG. 1 and “…implementations of the semantic classifier 128 may comprise a previously-trained machine-learning (ML) model 132 for classifying pixels in images, such as the 2D image 124, with semantic labels corresponding to map elements associated with road network maps”, see col.8, particularly lines 60-67 and col.9, particularly lines 1-12 and FIG. 1), a road map that is usable automated driving (AD) of the vehicle (“…the road network map may include the class labels indicated in segments of the labeled image(s) 130, as illustrated in the example labeled image 134, and the spatial extents of the segments, in a common two-dimensional reference frame. For example, during an initial creation of the road network map (e.g., for a new environment that an autonomous vehicle needs to traverse), labeled image(s) 130, such as the example labeled image 134”, see col.10, particularly lines 51-67 and col.11, particularly lines 1-8 and “…transmitting the road network map to an autonomous vehicle configured to navigate the environment using the road network map”, see Claim 15), wherein:
…
the generated road map includes a road layout within the at least one area drivable by [[a]] the vehicle along with locally allocated contextual information in view of applicable traffic regulations (“…the labeled image 130 may include segment(s) corresponding to map elements of road network maps, indicating both semantic label(s) associated with the segment(s) and a spatial extent of the segment(s). For example, the map elements can include lane elements to indicate an extent of a lane or driving corridor, stop line elements to indicate a stop line at an intersection, crosswalk elements to indicate areas of pedestrian crossing, and the like”, see col.8, particularly lines 43-59 and FIG. 1);
…
providing, by the one or more processors, the generated road map including a machine-readable…representation to the controller;
performing, by the controller, trajectory planning of the vehicle using the machine-readable…representation, including using the indication of whether traversal between the connected lane pieces is possible and compliant with the one or more applicable traffic regulations (“The segmentation component 444 may include a machine-learned (ML) model 448 trained for segmentation and/or classification tasks in 2D or 3D. As an example, the ML model 448 may comprise a fully convolutional network (FCN) trained to identify class labels (e.g., map elements such as “driving lane,” “crosswalk,” “stop line,” and the like) associated with individual pixels of a 2D input image (e.g., a top-down view of a road network)”, see col.23, particularly lines 59-67 and col.24, particularly lines 1-27, and see col.20, particularly lines 43-67 and col.21, particularly lines 1-3); and
controlling, by the controller and based on the trajectory planning, automated driving of the vehicle (“The operation 514 may also include determining a trajectory for the vehicle or to otherwise control the vehicle to safely traverse the environment, based at least in part on map elements (e.g., driving lane, stop line, turn lane, and the like) indicated on the existing map. In examples, the operation 514 may be performed by the planning component 428”, see col.28, particularly lines 14-27).
Khadem et al. does not expressly recite the bolded portions of the claimed
the trained visual foundation model comprises a graph generative model;
the generating comprises generating, as an output of the trained visual foundation model, a machine-readable graph representation
and
the machine-readable graph representation comprises:
nodes and edges representing road-layout structures of the road layout, the edges connecting the nodes and representing relationships between the nodes; and
attributes assigned to at least a subset of the nodes or to at least a subset of the edges, the attributes representing the locally allocated contextual information; and
at least one of the edges of the machine-readable graph representation connects respective ones of the nodes that represent connected lane pieces in the machine-readable graph representation and indicates whether traversal between the connected lane pieces is possible and compliant with one or more of the applicable traffic regulations;
providing, by the one or more processors, the generated road map including the machine-readable graph representation to the controller;
performing, by the controller, trajectory planning of the vehicle using the machine-readable graph representation, including using the indication of whether traversal between the connected lane pieces is possible and compliant with the one or more applicable traffic regulations; and
controlling, by the controller and based on the trajectory planning, automated driving of the vehicle.
However, Kapil et al. (2024/0263964) teaches generating a machine-readable graph representation and the machine-readable graph representation comprises: nodes and edges representing road-layout structures of the road layout, the edges connecting the nodes and representing relationships between the nodes (Kapil et al.; see P[0033]); and
attributes assigned to at least a subset of the nodes or to at least a subset of the edges, the attributes representing the locally allocated contextual information (Kapil et al.; “…edge 434 of FIG. 4 has a 50 MPH speed limit, edge 448 has a corresponding 35 MPH speed limit, and edge 462 has a corresponding 20 MPH speed limit”, see P[0050]); and
at least one of the edges of the machine-readable graph representation connects respective ones of the nodes that represent connected lane pieces in the machine-readable graph representation and indicates whether traversal between the connected lane pieces is possible and compliant with one or more of the applicable traffic regulations (Kapil et al.; “…edge 434 of FIG. 4 has a 50 MPH speed limit, edge 448 has a corresponding 35 MPH speed limit, and edge 462 has a corresponding 20 MPH speed limit”, see P[0050]);
providing, by the one or more processors, the generated road map including the machine-readable graph representation to the controller and performing, by the controller, trajectory planning of the vehicle using the machine-readable graph representation, including using the indication of whether traversal between the connected lane pieces is possible and compliant with the one or more applicable traffic regulations, and controlling, by the controller and based on the trajectory planning, automated driving of the vehicle (Kapil et al.; “Once a road segment graph is created for a polygon or polygons of an ODD, a structured and strategic approach can be applied to create, schedule, optimize, and assign mapping routes to mapping vehicles in a fleet”, see P[0035] and “The control stack 1318 may manage the operation of the vehicle propulsion system 1330, the braking system 1332, the steering system 1334, the safety system 1336, and the cabin system 1338. The control stack 1318 may receive sensor signals from the sensor systems 1304-1308 as well as communicate with other stacks or components of the local computing device 1310 or a remote system (e.g., the data center 1350) to effectuate operation of the AV 1302)”, see P[0103] and “For example, the control stack 1318 may implement the final path or actions from the multiple paths or actions provided by the planning stack 1316. Implementation may involve turning the routes and decisions from the planning stack 1316 into commands for the actuators that control the AV's steering, throttle, brake, and drive unit”, see P[0104]).
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Khadem et al. with the teachings of Kapil et al., and the trained visual foundation model comprises a graph generative model, the generating comprises generating, as an output of the trained visual foundation model, a machine-readable graph representation, and the machine-readable graph representation comprises nodes and edges representing road-layout structures of the road layout, the edges connecting the nodes and representing relationships between the nodes, and attributes assigned to at least a subset of the nodes or to at least a subset of the edges, the attributes representing the locally allocated contextual information, and at least one of the edges of the machine-readable graph representation connects respective ones of the nodes that represent connected lane pieces in the machine-readable graph representation and indicates whether traversal between the connected lane pieces is possible and compliant with one or more of the applicable traffic regulations, providing, by the one or more processors, the generated road map including the machine-readable graph representation to the controller, performing, by the controller, trajectory planning of the vehicle using the machine-readable graph representation, including using the indication of whether traversal between the connected lane pieces is possible and compliant with the one or more applicable traffic regulations, and controlling, by the controller and based on the trajectory planning, automated driving of the vehicle, as rendered obvious by Kapil et al., in order to “improve mapping data collection” and in order to “improve route creation, assignment, monitoring, and validation” (Kapil et al.; see Abstract).
Regarding Claim 4, Khadem et al. teaches the claimed method according to claim 1, wherein:
the image input data include top view image data of an area including the at least one area which is drivable by [[a]] the vehicle (“…a top-down view…”, see col.10, particularly lines 8-2 and FIG. 1), and/or
the image input data are acquired using an optical sensor system, including a satellite imagery system and/or an airborne camera system via one or more drones.
Regarding Claim 7, Khadem et al. teaches the claimed method according to claim 1, wherein the generating of the machine-readable graph representation (“…spatial extent(s) of the portion(s) or segment(s) within the 2D image 320 (e.g., bounding box(es) encapsulating the two-dimensional segment(s), or splines fitted to linear segments)…”, see col.18, particularly lines 18-40).
Regarding Claim 10, Khadem et al. teaches the claimed method according to claim 1, wherein the trained visual foundation model is configured to perform a classification and/or a semantic segmentation of the image input data in connection with generating the machine-readable graph representation (“…the example process 100 may include a semantic classifier 128 configured to accept the 2D image 124 as input…”, see col.8, particularly lines 43-59 and FIG. 1).
Regarding Claim 13, Khadem et al. teaches the claimed method according to claim 1, wherein the performing of the trajectory planning of the includes using the machine-readable graph representation for or more of: trajectory prediction, (“…the road network map may include the class labels indicated in segments of the labeled image(s) 130, as illustrated in the example labeled image 134, and the spatial extents of the segments, in a common two-dimensional reference frame. For example, during an initial creation of the road network map (e.g., for a new environment that an autonomous vehicle needs to traverse), labeled image(s) 130, such as the example labeled image 134”, see col.10, particularly lines 51-67 and col.11, particularly lines 1-8 and “…transmitting the road network map to an autonomous vehicle configured to navigate the environment using the road network map”, see Claim 15).
Regarding Claim 14, Khadem et al. teaches the claimed computing device
one or more processors (“The computing device 404 may include one or more processor(s) 416 and memory 418 communicatively coupled with the one or more processor(s) 416”, see col.21, particularly lines 4-21), the one or more processors being configured to:
receive, from the at least one sensor system, image input data that includes acquired image data representing at least one area which is drivable by [[a]] the vehicle (“…the example process 100 may include a semantic classifier 128 configured to accept the 2D image 124 as input…”, see col.8, particularly lines 43-59 and FIG. 1);
generate, and using a road map that is usable in automated driving (AD) of the vehicle (“…output a semantically labeled image 130, see col.8, particularly lines 43-59 and FIG. 1 and “…implementations of the semantic classifier 128 may comprise a previously-trained machine-learning (ML) model 132 for classifying pixels in images, such as the 2D image 124, with semantic labels corresponding to map elements associated with road network maps”, see col.8, particularly lines 60-67 and col.9, particularly lines 1-12 and FIG. 1 and “…the road network map may include the class labels indicated in segments of the labeled image(s) 130, as illustrated in the example labeled image 134, and the spatial extents of the segments, in a common two-dimensional reference frame. For example, during an initial creation of the road network map (e.g., for a new environment that an autonomous vehicle needs to traverse), labeled image(s) 130, such as the example labeled image 134”, see col.10, particularly lines 51-67 and col.11, particularly lines 1-8 and “…transmitting the road network map to an autonomous vehicle configured to navigate the environment using the road network map”, see Claim 15), wherein:
…
the generated road map includes a road layout within the at least one area drivable by [[a]] the vehicle along with locally allocated contextual information in view of applicable traffic regulations (“…the labeled image 130 may include segment(s) corresponding to map elements of road network maps, indicating both semantic label(s) associated with the segment(s) and a spatial extent of the segment(s). For example, the map elements can include lane elements to indicate an extent of a lane or driving corridor, stop line elements to indicate a stop line at an intersection, crosswalk elements to indicate areas of pedestrian crossing, and the like”, see col.8, particularly lines 43-59 and FIG. 1);
…
provide the generated road map including a machine-readable…representation to a controller that is configured to:
perform trajectory planning of the vehicle using the machine-readable…representation, including using the indication of whether traversal between the connected lane pieces is possible and compliant with the one or more applicable traffic regulations (“The segmentation component 444 may include a machine-learned (ML) model 448 trained for segmentation and/or classification tasks in 2D or 3D. As an example, the ML model 448 may comprise a fully convolutional network (FCN) trained to identify class labels (e.g., map elements such as “driving lane,” “crosswalk,” “stop line,” and the like) associated with individual pixels of a 2D input image (e.g., a top-down view of a road network)”, see col.23, particularly lines 59-67 and col.24, particularly lines 1-27, and see col.20, particularly lines 43-67 and col.21, particularly lines 1-3); and
control, based on the trajectory planning, automated driving of the vehicle (“The operation 514 may also include determining a trajectory for the vehicle or to otherwise control the vehicle to safely traverse the environment, based at least in part on map elements (e.g., driving lane, stop line, turn lane, and the like) indicated on the existing map. In examples, the operation 514 may be performed by the planning component 428”, see col.28, particularly lines 14-27).
Khadem et al. does not expressly recite the bolded portions of the claimed
the trained visual foundation model comprises a graph generative model;
the generating comprises generating, as an output of the trained visual foundation model, a machine-readable graph representation
and
the machine-readable graph representation comprises:
nodes and edges representing road-layout structures of the road layout, the edges connecting the nodes and representing relationships between the nodes; and
attributes assigned to at least a subset of the nodes or to at least a subset of the edges, the attributes representing the locally allocated contextual information; and
at least one of the edges of the machine-readable graph representation connects respective ones of the nodes that represent connected lane pieces in the machine-readable graph representation and indicates whether traversal between the connected lane pieces is possible and compliant with one or more of the applicable traffic regulations; and
provide the generated road map including the machine-readable graph representation to a controller that is configured to:
perform trajectory planning of the vehicle using the machine-readable graph representation, including using the indication of whether traversal between the connected lane pieces is possible and compliant with the one or more applicable traffic regulations; and
control, based on the trajectory planning, automated driving of the vehicle.
However, Kapil et al. (2024/0263964) teaches generating, a machine-readable graph representation, the machine-readable graph representation comprises: nodes and edges representing road-layout structures of the road layout, the edges connecting the nodes and representing relationships between the nodes (Kapil et al.; see P[0033]), and
attributes assigned to at least a subset of the nodes or to at least a subset of the edges, the attributes representing the locally allocated contextual information (Kapil et al.; “…edge 434 of FIG. 4 has a 50 MPH speed limit, edge 448 has a corresponding 35 MPH speed limit, and edge 462 has a corresponding 20 MPH speed limit”, see P[0050]), and
at least one of the edges of the machine-readable graph representation connects respective ones of the nodes that represent connected lane pieces in the machine-readable graph representation and indicates whether traversal between the connected lane pieces is possible and compliant with one or more of the applicable traffic regulations (Kapil et al.; “…edge 434 of FIG. 4 has a 50 MPH speed limit, edge 448 has a corresponding 35 MPH speed limit, and edge 462 has a corresponding 20 MPH speed limit”, see P[0050]); and provide the generated road map including the machine-readable graph representation to a controller that is configured to perform trajectory planning of the vehicle using the machine-readable graph representation, including using the indication of whether traversal between the connected lane pieces is possible and compliant with the one or more applicable traffic regulations (Kapil et al.; “Once a road segment graph is created for a polygon or polygons of an ODD, a structured and strategic approach can be applied to create, schedule, optimize, and assign mapping routes to mapping vehicles in a fleet”, see P[0035]), and
control, based on the trajectory planning, automated driving of the vehicle (Kapil et al.; “Once a road segment graph is created for a polygon or polygons of an ODD, a structured and strategic approach can be applied to create, schedule, optimize, and assign mapping routes to mapping vehicles in a fleet”, see P[0035] and “The control stack 1318 may manage the operation of the vehicle propulsion system 1330, the braking system 1332, the steering system 1334, the safety system 1336, and the cabin system 1338. The control stack 1318 may receive sensor signals from the sensor systems 1304-1308 as well as communicate with other stacks or components of the local computing device 1310 or a remote system (e.g., the data center 1350) to effectuate operation of the AV 1302)”, see P[0103] and “For example, the control stack 1318 may implement the final path or actions from the multiple paths or actions provided by the planning stack 1316. Implementation may involve turning the routes and decisions from the planning stack 1316 into commands for the actuators that control the AV's steering, throttle, brake, and drive unit”, see P[0104]).
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Khadem et al. with the teachings of Kapil et al., and the trained visual foundation model comprises a graph generative model, the generating comprises generating, as an output of the trained visual foundation model, a machine-readable graph representation, and the machine-readable graph representation comprises nodes and edges representing road-layout structures of the road layout, the edges connecting the nodes and representing relationships between the nodes, and attributes assigned to at least a subset of the nodes or to at least a subset of the edges, the attributes representing the locally allocated contextual information, and at least one of the edges of the machine-readable graph representation connects respective ones of the nodes that represent connected lane pieces in the machine-readable graph representation and indicates whether traversal between the connected lane pieces is possible and compliant with one or more of the applicable traffic regulations, and provide the generated road map including the machine-readable graph representation to a controller that is configured to perform trajectory planning of the vehicle using the machine-readable graph representation, including using the indication of whether traversal between the connected lane pieces is possible and compliant with the one or more applicable traffic regulations, and control, based on the trajectory planning, automated driving of the vehicle, as rendered obvious by Kapil et al., in order to “improve mapping data collection” and in order to “improve route creation, assignment, monitoring, and validation” (Kapil et al.; see Abstract).
Regarding Claim 18, Khadem et al. teaches the claimed system
a controller (see col.20, particularly lines 43-67 and col.21, particularly lines 1-3 and “The computing device 404 may include one or more processor(s) 416 and memory 418 communicatively coupled with the one or more processor(s) 416”, see col.21, particularly lines 4-21);
at least one sensor system (see col.8, particularly lines 10-42); and
a computing device including one or more processors (“The computing device 404 may include one or more processor(s) 416 and memory 418 communicatively coupled with the one or more processor(s) 416”, see col.21, particularly lines 4-21), the one or more processors being configured to:
receive, from the at least one sensor system, image input data that includes acquired image data representing at least one area which is drivable by [[a]] the vehicle (see col.8, particularly lines 10-42 and “…the example process 100 may include a semantic classifier 128 configured to accept the 2D image 124 as input…”, see col.8, particularly lines 43-59 and FIG. 1);
generate, a road map that is usable in automated driving (AD) of the vehicle (“…output a semantically labeled image 130, see col.8, particularly lines 43-59 and FIG. 1 and “…implementations of the semantic classifier 128 may comprise a previously-trained machine-learning (ML) model 132 for classifying pixels in images, such as the 2D image 124, with semantic labels corresponding to map elements associated with road network maps”, see col.8, particularly lines 60-67 and col.9, particularly lines 1-12 and FIG. 1 and “…the road network map may include the class labels indicated in segments of the labeled image(s) 130, as illustrated in the example labeled image 134, and the spatial extents of the segments, in a common two-dimensional reference frame. For example, during an initial creation of the road network map (e.g., for a new environment that an autonomous vehicle needs to traverse), labeled image(s) 130, such as the example labeled image 134”, see col.10, particularly lines 51-67 and col.11, particularly lines 1-8 and “…transmitting the road network map to an autonomous vehicle configured to navigate the environment using the road network map”, see Claim 15), wherein:
…
the generated road map includes a road layout within the at least one area drivable by [[a]] the vehicle along with locally allocated contextual information in view of applicable traffic regulations (“…the labeled image 130 may include segment(s) corresponding to map elements of road network maps, indicating both semantic label(s) associated with the segment(s) and a spatial extent of the segment(s). For example, the map elements can include lane elements to indicate an extent of a lane or driving corridor, stop line elements to indicate a stop line at an intersection, crosswalk elements to indicate areas of pedestrian crossing, and the like”, see col.8, particularly lines 43-59 and FIG. 1);
…
provide the generated road map including a machine-readable…representation to the controller;
wherein the controller is configured to:
perform trajectory planning of the vehicle using the machine-readable…representation, including using the indication of whether traversal between the connected lane pieces is possible and compliant with the one or more applicable traffic regulations (“The segmentation component 444 may include a machine-learned (ML) model 448 trained for segmentation and/or classification tasks in 2D or 3D. As an example, the ML model 448 may comprise a fully convolutional network (FCN) trained to identify class labels (e.g., map elements such as “driving lane,” “crosswalk,” “stop line,” and the like) associated with individual pixels of a 2D input image (e.g., a top-down view of a road network)”, see col.23, particularly lines 59-67 and col.24, particularly lines 1-27, and see col.20, particularly lines 43-67 and col.21, particularly lines 1-3); and
control, based on the trajectory planning, automated driving of the vehicle (“The operation 514 may also include determining a trajectory for the vehicle or to otherwise control the vehicle to safely traverse the environment, based at least in part on map elements (e.g., driving lane, stop line, turn lane, and the like) indicated on the existing map. In examples, the operation 514 may be performed by the planning component 428”, see col.28, particularly lines 14-27).
Khadem et al. does not expressly recite the bolded portions of the claimed
the trained visual foundation model comprises a graph generative model;
the generating comprises generating, as an output of the trained visual foundation model, a machine-readable graph representation;
and
the machine-readable graph representation comprises:
nodes and edges representing road-layout structures of the road layout, the edges connecting the nodes and representing relationships between the nodes; and
attributes assigned to at least a subset of the nodes or to at least a subset of the edges, the attributes representing the locally allocated contextual information; and
at least one of the edges of the machine-readable graph representation connects respective ones of the nodes that represent connected lane pieces in the machine-readable graph representation and indicates whether traversal between the connected lane pieces is possible and compliant with one or more of the applicable traffic regulations; and
provide the generated road map including the machine-readable graph representation to the controller;
wherein the controller is configured to:
perform trajectory planning of the vehicle using the machine-readable graph representation, including using the indication of whether traversal between the connected lane pieces is possible and compliant with the one or more applicable traffic regulations; and
control, based on the trajectory planning, automated driving of the vehicle.
However, Kapil et al. (2024/0263964) generating a machine-readable graph representation, and the machine-readable graph representation comprises nodes and edges representing road-layout structures of the road layout, the edges connecting the nodes and representing relationships between the nodes (Kapil et al.; see P[0033]), and
attributes assigned to at least a subset of the nodes or to at least a subset of the edges, the attributes representing the locally allocated contextual information (Kapil et al.; “…edge 434 of FIG. 4 has a 50 MPH speed limit, edge 448 has a corresponding 35 MPH speed limit, and edge 462 has a corresponding 20 MPH speed limit”, see P[0050]), and
at least one of the edges of the machine-readable graph representation connects respective ones of the nodes that represent connected lane pieces in the machine-readable graph representation and indicates whether traversal between the connected lane pieces is possible and compliant with one or more of the applicable traffic regulations (Kapil et al.; “…edge 434 of FIG. 4 has a 50 MPH speed limit, edge 448 has a corresponding 35 MPH speed limit, and edge 462 has a corresponding 20 MPH speed limit”, see P[0050]); and provide the generated road map including the machine-readable graph representation to a controller that is configured to perform trajectory planning of the vehicle using the machine-readable graph representation, including using the indication of whether traversal between the connected lane pieces is possible and compliant with the one or more applicable traffic regulations (Kapil et al.; “Once a road segment graph is created for a polygon or polygons of an ODD, a structured and strategic approach can be applied to create, schedule, optimize, and assign mapping routes to mapping vehicles in a fleet”, see P[0035]), and
provide the generated road map including the machine-readable graph representation to the controller, wherein the controller is configured to perform trajectory planning of the vehicle using the machine-readable graph representation, including using the indication of whether traversal between the connected lane pieces is possible and compliant with the one or more applicable traffic regulations; and control, based on the trajectory planning, automated driving of the vehicle (Kapil et al.; “Once a road segment graph is created for a polygon or polygons of an ODD, a structured and strategic approach can be applied to create, schedule, optimize, and assign mapping routes to mapping vehicles in a fleet”, see P[0035] and “The control stack 1318 may manage the operation of the vehicle propulsion system 1330, the braking system 1332, the steering system 1334, the safety system 1336, and the cabin system 1338. The control stack 1318 may receive sensor signals from the sensor systems 1304-1308 as well as communicate with other stacks or components of the local computing device 1310 or a remote system (e.g., the data center 1350) to effectuate operation of the AV 1302)”, see P[0103] and “For example, the control stack 1318 may implement the final path or actions from the multiple paths or actions provided by the planning stack 1316. Implementation may involve turning the routes and decisions from the planning stack 1316 into commands for the actuators that control the AV's steering, throttle, brake, and drive unit”, see P[0104]).
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Khadem et al. with the teachings of Kapil et al., and the trained visual foundation model comprises a graph generative model, the generating comprises generating, as an output of the trained visual foundation model, a machine-readable graph representation, and
As per Claim 22, Khadem et al. does not expressly recite the claimed method according to claim 1, wherein at least one of the attributes represents, for a road or lane section represented by a corresponding node or edge, a speed limitation, a one-way traffic regulation, a lane-turn permission, a lane-change permission, or a traffic-sign-related regulation.
However, Kapil et al. (2024/0263964) teaches wherein at least one of the attributes represents, for a road or lane section represented by a corresponding node or edge, a speed limitation, a one-way traffic regulation, a lane-turn permission, a lane-change permission, or a traffic-sign-related regulation (Kapil et al.; “…edge 434 of FIG. 4 has a 50 MPH speed limit, edge 448 has a corresponding 35 MPH speed limit, and edge 462 has a corresponding 20 MPH speed limit”, see P[0050]).
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Khadem et al. with the teachings of Kapil et al., and wherein at least one of the attributes represents, for a road or lane section represented by a corresponding node or edge, a speed limitation, a one-way traffic regulation, a lane-turn permission, a lane-change permission, or a traffic-sign-related regulation, as rendered obvious by Kapil et al., in order to “improve route creation, assignment, monitoring, and validation” (Kapil et al.; see Abstract).
Claim 3 is rejected under 35 U.S.C. 103 as being unpatentable over Khadem et al. (12,360,213) in view of Kapil et al. (2024/0263964) further in view of Muzina et al. (2011/0172917).
Regarding Claim 3, Khadem et al. teaches the claimed method according to claim 1, wherein the road layout of the generated road map is adjustable in a granularity, for rendering and/or depending on the locally allocated contextual information.
However, Muzina et al. (2011/0172917) teaches displaying a map at various levels of detail (Muzina et al.; see P[0017]).
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Khadem et al. with the teachings of Muzina et al., and wherein the road layout of the generated road map is adjustable in a granularity, for rendering and/or depending on the locally allocated contextual information, as rendered obvious by Muzina et al., in order to provide for “adjusting a level of map detail” (Muzina et al.; see Abstract).
Claims 5 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Khadem et al. (12,360,213) in view of Kapil et al. (2024/0263964) further in view of Khan (2026/0235415).
Regarding Claim 5, Khadem et al. does not expressly recite the claimed method according to claim 1, wherein:
the machine-readable graph representation comprises the nodes and the edges;
the edges connect subsets of the nodes; [[,]] and
the .
However, Khan (2026/0235415) teaches wherein the machine-readable graph representation comprises a deep and/or nested graph including the nodes and the edges; the edges connect subsets of the nodes, and the attributes are assigned to nodes and/or the edge (Khan; see P[0055]-P[0056]).
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Khadem et al. with the teachings of Khan, and wherein the machine-readable graph representation comprises a deep and/or nested graph including the nodes and the edges; the edges connect subsets of the nodes, and the attributes are assigned to nodes and/or the edge, as rendered obvious by Khan, in order to provide “support tools for autonomous vehicles” and to “facilitate real-time planning, prediction and/or other online functions” (Khan; see P[0001]).
As per Claim 20, Khadem et al. does not expressly recite the claimed method according to claim 1, wherein the machine-readable graph representation is a nested graph comprising:
a first graph level in which nodes represent points on lane-boundary lines and edges connect neighboring points on the lane-boundary lines; and
a second graph level in which nodes represent the pieces of lanes and edges connect the pieces of lanes.
However, Khan (2026/0235415) teaches wherein the machine-readable graph representation is a nested graph comprising: a first graph level in which nodes represent points on lane-boundary lines and edges connect neighboring points on the lane-boundary lines, and a second graph level in which nodes represent the pieces of lanes and edges connect the pieces of lanes (Khan; “A predetermined type of road structure (such as a roundabout) is detected by searching for any subgraph within the driving topology graph satisfying one or more criteria (such as a one-way loop). A roundabout detection method based on one-way driving loops is described. The principles can be extended to other forms of road structure. For example, another form of search may be implemented as a graph isomorphism (locating a subgraph that is isomorphic to some indicated graph structure)”, see P[0055] and “A lane graph is described, which is a graph encoding a road topology at the level of individual lanes (or ‘side-lanes’, as referred to herein), where each node represents a side-lane and edges in the graph represent links between side-lanes. One implementation of the map annotation tool operates on the lane graph, which is appropriate for identifying more ‘granular’ road structures defined in terms of side-lane topology. Another implementation uses a ‘coarser’ road topology graph, e.g., where nodes represent roads and edges represent links between road, which is appropriate for less granular road structure, e.g., roundabouts defined at the level of road (rather than lane) interconnections”, see P[0056]).
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Khadem et al. with the teachings of Khan, and wherein the machine-readable graph representation is a nested graph comprising a first graph level in which nodes represent points on lane-boundary lines and edges connect neighboring points on the lane-boundary lines, and a second graph level in which nodes represent the pieces of lanes and edges connect the pieces of lanes, as rendered obvious by Khan, in order to provide “support tools for autonomous vehicles” and to “facilitate real-time planning, prediction and/or other online functions” (Khan; see P[0001]).
Claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over Khadem et al. (12,360,213) in view of Kapil et al. (2024/0263964) further in view of Pung et al. (2017/0356742).
Regarding Claim 6, Khadem et al. does not expressly recite the claimed method according to claim 5, wherein the machine-readable graph representation includes or is encoded as a scalable vector graphic (SVG).
However, Pung et al. (2017/0356742) teaches “map presentation data encoded using vector graphics (e.g., svg or drw data)” (Pung et al.; see P[0084]).
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Khadem et al. with the teachings of Pung et al., and wherein the machine-readable graph representation includes or is encoded as a scalable vector graphic (SVG), as rendered obvious by Pung et al., in order to “provide vector-based map data” (Pung et al.; see P[0084]).
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over Khadem et al. (12,360,213) in view of Kapil et al. (2024/0263964) further in view of Levihn et al. (11,555,706) further in view of Doersch (“Tutorial on Variational Autoencoders”, arXiv:1606.05908v1).
Regarding Claim 9, Khadem et al. does not expressly recite the claimed method according to claim [[8]] 1, wherein the graph generative model includes at least one of:
an autoregressive model, a variational autoencoder, a normalizing flow, a generative adversarial network, a diffusion model.
However, Levihn et al. (11,555,706) teaches a deep neural network based on model (Levihn et al.; “In the depicted embodiment, a deep neural network (DNN) based model 350 may be trained to reason over graph representations of tactical maps similar to homogenized graph representation 310”, see col.11, particularly lines 57-67 and col.12, particularly lines 1-3).
Furthermore, a “variational autoencoder” is simply a known type of neural network, as seen in Doersch (“Tutorial on Variational Autoencoders”, arXiv:1606.05908v1) (Tutorial on Variational Autoencoders; “A training-time variational autoencoder implemented as a feed forward neural network”, see Figure 4, page 10).
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Khadem et al. with the teachings of Levihn et al. and Doersch, and wherein the graph generative model includes at least one of: an autoregressive model, a variational autoencoder, a normalizing flow, a generative adversarial network, a diffusion model, as rendered obvious by Doersch, in order to provide “unsupervised learning of complicated distributions” (Doersch; see Abstract).
Claims 11, 12, 16 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Khadem et al. (12,360,213) in view of Konno et al. (2025/0292506) further in view of Kapil et al. (2024/0263964).
Regarding Claim 11, Khadem et al. teaches the claimed computer-implemented method for training a visual foundation model for generating a road map based on received image input data, the method comprising the following steps:
receiving a training dataset including or more training samples, each training sample including:
a top view image acquired using an optical sensor system…, wherein the top view image includes image data representing at least one area which is drivable by a vehicle (“…the trained ML model 132 may be previously trained using training data comprising images of top-down views of road networks where pixels have been labeled with a class label (e.g., manually, by class labels provided by a third-party map provider, and/or by auto-generated and human-verified class labels) from among a specified set of class labels”, see col.8, particularly lines 60-67 and col.9, particularly lines 1-12, and see col.8, particularly lines 10-42);
and
a ground-truth road map for the top view image…(“…the trained ML model 132 may be previously trained using training data comprising images of top-down views of road networks where pixels have been labeled with a class label (e.g., manually, by class labels provided by a third-party map provider, and/or by auto-generated and human-verified class labels) from among a specified set of class labels”, see col.8, particularly lines 60-67 and col.9, particularly lines 1-12, and see col.8, particularly lines 10-42);
…
; and
indicates whether traversal between the connected lane pieces is possible and compliant with one or more of the applicable traffic regulations (“The segmentation component 444 may include a machine-learned (ML) model 448 trained for segmentation and/or classification tasks in 2D or 3D. As an example, the ML model 448 may comprise a fully convolutional network (FCN) trained to identify class labels (e.g., map elements such as “driving lane,” “crosswalk,” “stop line,” and the like) associated with individual pixels of a 2D input image (e.g., a top-down view of a road network)”, see col.23, particularly lines 59-67 and col.24, particularly lines 1-27, and see col.20, particularly lines 43-67 and col.21, particularly lines 1-3); and
training a visual foundation model comprising a…generative model for generating a road map based on received image input data (“…the trained ML model 132 may be previously trained using training data comprising images of top-down views of road networks where pixels have been labeled with a class label (e.g., manually, by class labels provided by a third-party map provider, and/or by auto-generated and human-verified class labels) from among a specified set of class labels”, see col.8, particularly lines 60-67 and col.9, particularly lines 1-12), wherein:
the generated road map includes a road layout along with locally allocated contextual information in view of applicable traffic regulations (“…the labeled image 130 may include segment(s) corresponding to map elements of road network maps, indicating both semantic label(s) associated with the segment(s) and a spatial extent of the segment(s). For example, the map elements can include lane elements to indicate an extent of a lane or driving corridor, stop line elements to indicate a stop line at an intersection, crosswalk elements to indicate areas of pedestrian crossing, and the like”, see col.8, particularly lines 43-59 and FIG. 1); and
the training comprises training the…generative model to generate, from the top view image, a generated machine-readable…representation of the generated road map corresponding to the ground-truth machine-readable…representation (“…the labeled image 130 may include segment(s) corresponding to map elements of road network maps, indicating both semantic label(s) associated with the segment(s) and a spatial extent of the segment(s). For example, the map elements can include lane elements to indicate an extent of a lane or driving corridor, stop line elements to indicate a stop line at an intersection, crosswalk elements to indicate areas of pedestrian crossing, and the like”, see col.8, particularly lines 43-59 and FIG. 1).
Khadem et al. does not expressly recite the bolded portions of the claimed
a top view image acquired using an optical sensor system, including by a satellite imaging system and/or by one or more drones
and
wherein:
the ground-truth road map comprises a ground-truth machine-readable graph representation that includes:
nodes and edges representing road-layout structures of a road layout within the at least one area drivable by the vehicle, the edges connecting the nodes and representing relationships between the nodes; and
attributes assigned to at least a subset of the nodes or to at least a subset of the edges, the attributes representing locally allocated contextual information in view of applicable traffic regulations; [[,]] and at least one of the attributes:
is assigned to an edge of the ground-truth machine-readable graph representation that connects respective ones of the nodes representing connected lane pieces in the ground-truth machine-readable graph representation; and
indicates whether traversal between the connected lane pieces is possible and compliant with one or more of the applicable traffic regulations; and
training a visual foundation model comprising a graph generative model for generating a road map based on received image input data, wherein:
the generated road map includes a road layout along with locally allocated contextual information in view of applicable traffic regulations; and
the training comprises training the graph generative model to generate, from the top view image, a generated machine-readable graph representation of the generated road map corresponding to the ground-truth machine-readable graph representation.
However, Konno et al. (2025/0292506) teaches performing semantic segmentation on an image of a satellite photo (Konno et al.; see P[0126])
Furthermore, Kapil et al. (2024/0263964) teaches the ground-truth road map comprises a ground-truth machine-readable graph representation that includes nodes and edges representing road-layout structures of a road layout within the at least one area drivable by the vehicle, the edges connecting the nodes and representing relationships between the nodes (Kapil et al.; see P[0033]), and
attributes assigned to at least a subset of the nodes or to at least a subset of the edges, the attributes representing locally allocated contextual information in view of applicable traffic regulations (Kapil et al.; “…edge 434 of FIG. 4 has a 50 MPH speed limit, edge 448 has a corresponding 35 MPH speed limit, and edge 462 has a corresponding 20 MPH speed limit”, see P[0050]),
at least one of the attributes is assigned to an edge of the ground-truth machine-readable graph representation that connects respective ones of the nodes representing
and generate a generated machine-readable graph representation of the generated road map corresponding to the ground-truth machine-readable graph representation (Kapil et al.; “Once a road segment graph is created for a polygon or polygons of an ODD, a structured and strategic approach can be applied to create, schedule, optimize, and assign mapping routes to mapping vehicles in a fleet”, see P[0035] and “The control stack 1318 may manage the operation of the vehicle propulsion system 1330, the braking system 1332, the steering system 1334, the safety system 1336, and the cabin system 1338. The control stack 1318 may receive sensor signals from the sensor systems 1304-1308 as well as communicate with other stacks or components of the local computing device 1310 or a remote system (e.g., the data center 1350) to effectuate operation of the AV 1302)”, see P[0103] and “For example, the control stack 1318 may implement the final path or actions from the multiple paths or actions provided by the planning stack 1316. Implementation may involve turning the routes and decisions from the planning stack 1316 into commands for the actuators that control the AV's steering, throttle, brake, and drive unit”, see P[0104]).
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Khadem et al. with the teachings of Konno et al. and Kapil et al., and to provide a top view image acquired using an optical sensor system, including by a satellite imaging system and/or by one or more drones and wherein the ground-truth road map comprises a ground-truth machine-readable graph representation that includes nodes and edges representing road-layout structures of a road layout within the at least one area drivable by the vehicle, the edges connecting the nodes and representing relationships between the nodes, and attributes assigned to at least a subset of the nodes or to at least a subset of the edges, the attributes representing locally allocated contextual information in view of applicable traffic regulations; and at least one of the attributes is assigned to an edge of the ground-truth machine-readable graph representation that connects respective ones of the nodes representing connected lane pieces in the ground-truth machine-readable graph representation, and indicates whether traversal between the connected lane pieces is possible and compliant with one or more of the applicable traffic regulations, and training a visual foundation model comprising a graph generative model for generating a road map based on received image input data, wherein the generated road map includes a road layout along with locally allocated contextual information in view of applicable traffic regulations, and the training comprises training the graph generative model to generate, from the top view image, a generated machine-readable graph representation of the generated road map corresponding to the ground-truth machine-readable graph representation, as rendered obvious by Konno et al. and Kapil et al., in order to obtain “the position and outline of the objects and the meta information of the objects” (Konno et al.; see P[0125]), and in order to “improve mapping data collection” and in order to “improve route creation, assignment, monitoring, and validation” (Kapil et al.; see Abstract).
Regarding Claim 12, Khadem et al. teaches the claimed method according to claim 11, wherein the training is self-supervised and/or based on a reconstruction loss between [[a]] the ground-truth machine-readable graph representation and the generated machine-readable graph representation graph generative (“…supervised learning…”, see col.25, particularly lines 50-67 and col.26, particularly lines 1-22).
Regarding Claim 16, Khadem et al. teaches the claimed computing device for training a visual foundation model for generating a road map based on received image input data, comprising:
a training data receiving interface configured to receive a training dataset including or more training samples, each training sample including:
a top view image acquired using an optical sensor system…, wherein the top view image includes image data representing at least one area which is drivable by a vehicle (“…the trained ML model 132 may be previously trained using training data comprising images of top-down views of road networks where pixels have been labeled with a class label (e.g., manually, by class labels provided by a third-party map provider, and/or by auto-generated and human-verified class labels) from among a specified set of class labels”, see col.8, particularly lines 60-67 and col.9, particularly lines 1-12);
and
a ground-truth road map for the top view image…(“…the trained ML model 132 may be previously trained using training data comprising images of top-down views of road networks where pixels have been labeled with a class label (e.g., manually, by class labels provided by a third-party map provider, and/or by auto-generated and human-verified class labels) from among a specified set of class labels”, see col.8, particularly lines 60-67 and col.9, particularly lines 1-12, and see col.8, particularly lines 10-42); and
indicates whether traversal between the connected lane pieces is possible and compliant with one or more of the applicable traffic regulations (“The segmentation component 444 may include a machine-learned (ML) model 448 trained for segmentation and/or classification tasks in 2D or 3D. As an example, the ML model 448 may comprise a fully convolutional network (FCN) trained to identify class labels (e.g., map elements such as “driving lane,” “crosswalk,” “stop line,” and the like) associated with individual pixels of a 2D input image (e.g., a top-down view of a road network)”, see col.23, particularly lines 59-67 and col.24, particularly lines 1-27, and see col.20, particularly lines 43-67 and col.21, particularly lines 1-3); and
a visual foundation model training module configured to train a visual foundation model comprising a…generative model for generating a road map based on received image input data (“…the trained ML model 132 may be previously trained using training data comprising images of top-down views of road networks where pixels have been labeled with a class label (e.g., manually, by class labels provided by a third-party map provider, and/or by auto-generated and human-verified class labels) from among a specified set of class labels”, see col.8, particularly lines 60-67 and col.9, particularly lines 1-12), wherein:
the generated road map includes a road layout along with locally allocated contextual information in view of applicable traffic regulations (“The segmentation component 444 may include a machine-learned (ML) model 448 trained for segmentation and/or classification tasks in 2D or 3D. As an example, the ML model 448 may comprise a fully convolutional network (FCN) trained to identify class labels (e.g., map elements such as “driving lane,” “crosswalk,” “stop line,” and the like) associated with individual pixels of a 2D input image (e.g., a top-down view of a road network)”, see col.23, particularly lines 59-67 and col.24, particularly lines 1-27, and see col.20, particularly lines 43-67 and col.21, particularly lines 1-3); and
the training comprises training the…generative model to generate, from the top view image, a generated machine-readable…representation of the generated road map corresponding to the ground-truth machine-readable…representation (“…the labeled image 130 may include segment(s) corresponding to map elements of road network maps, indicating both semantic label(s) associated with the segment(s) and a spatial extent of the segment(s). For example, the map elements can include lane elements to indicate an extent of a lane or driving corridor, stop line elements to indicate a stop line at an intersection, crosswalk elements to indicate areas of pedestrian crossing, and the like”, see col.8, particularly lines 43-59 and FIG. 1).
Khadem et al. does not expressly recite the bolded portions of the claimed
a top view image acquired using an optical sensor system, including by a satellite imaging system and/or by one or more drones
and
wherein:
the ground-truth road map comprises a ground-truth machine-readable graph representation that includes:
nodes and edges representing road-layout structures of a road layout within the at least one area drivable by the vehicle, the edges connecting the nodes and representing relationships between the nodes;
and
attributes assigned to at least a subset of the nodes or to at least a subset of the edges, the attributes representing locally allocated contextual information in view of applicable traffic regulations; [[,]] and at least one of the attributes:
is assigned to an edge of the ground-truth machine-readable graph representation that connects respective ones of the nodes representing connected lane pieces in the ground-truth machine-readable graph representation; and
indicates whether traversal between the connected lane pieces is possible and compliant with one or more of the applicable traffic regulations; and
a visual foundation model training module configured to train a visual foundation model comprising a graph generative model for generating a road map based on received image input data, wherein:
the generated road map includes a road layout along with locally allocated contextual information in view of applicable traffic regulations; and
the training comprises training the graph generative model to generate, from the top view image, a generated machine-readable graph representation of the generated road map corresponding to the ground-truth machine-readable graph representation.
However, Konno et al. (2025/0292506) teaches performing semantic segmentation on an image of a satellite photo (Konno et al.; see P[0126])
Furthermore, Kapil et al. (2024/0263964) teaches a machine-readable graph representation that includes nodes and edges representing road-layout structures of a road layout within the at least one area drivable by the vehicle, the edges connecting the nodes and representing relationships between the nodes (Kapil et al.; see P[0033]), and
attributes assigned to at least a subset of the nodes or to at least a subset of the edges, the attributes representing locally allocated contextual information in view of applicable traffic regulations (Kapil et al.; “…edge 434 of FIG. 4 has a 50 MPH speed limit, edge 448 has a corresponding 35 MPH speed limit, and edge 462 has a corresponding 20 MPH speed limit”, see P[0050]); at least one of the attributes:
is assigned to an edge of the ground-truth machine-readable graph representation that connects respective ones of the nodes representing connected lane pieces in the ground-truth machine-readable graph representation (Kapil et al.; “…edge 434 of FIG. 4 has a 50 MPH speed limit, edge 448 has a corresponding 35 MPH speed limit, and edge 462 has a corresponding 20 MPH speed limit”, see P[0050]),
and generate a machine-readable graph representation of the generated road map corresponding to the machine-readable graph representation (Kapil et al.; “Once a road segment graph is created for a polygon or polygons of an ODD, a structured and strategic approach can be applied to create, schedule, optimize, and assign mapping routes to mapping vehicles in a fleet”, see P[0035] and “The control stack 1318 may manage the operation of the vehicle propulsion system 1330, the braking system 1332, the steering system 1334, the safety system 1336, and the cabin system 1338. The control stack 1318 may receive sensor signals from the sensor systems 1304-1308 as well as communicate with other stacks or components of the local computing device 1310 or a remote system (e.g., the data center 1350) to effectuate operation of the AV 1302)”, see P[0103] and “For example, the control stack 1318 may implement the final path or actions from the multiple paths or actions provided by the planning stack 1316. Implementation may involve turning the routes and decisions from the planning stack 1316 into commands for the actuators that control the AV's steering, throttle, brake, and drive unit”, see P[0104]).
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Khadem et al. with the teachings of Konno et al. and Kapil et al., and to provide a top view image acquired using an optical sensor system, including by a satellite imaging system and/or by one or more drones, and wherein the ground-truth road map comprises a ground-truth machine-readable graph representation that includes nodes and edges representing road-layout structures of a road layout within the at least one area drivable by the vehicle, the edges connecting the nodes and representing relationships between the nodes, and attributes assigned to at least a subset of the nodes or to at least a subset of the edges, the attributes representing locally allocated contextual information in view of applicable traffic regulations, and at least one of the attributes is assigned to an edge of the ground-truth machine-readable graph representation that connects respective ones of the nodes representing connected lane pieces in the ground-truth machine-readable graph representation, and indicates whether traversal between the connected lane pieces is possible and compliant with one or more of the applicable traffic regulations, and a visual foundation model training module configured to train a visual foundation model comprising a graph generative model for generating a road map based on received image input data, wherein the generated road map includes a road layout along with locally allocated contextual information in view of applicable traffic regulations, and the training comprises training the graph generative model to generate, from the top view image, a generated machine-readable graph representation of the generated road map corresponding to the ground-truth machine-readable graph representation, as rendered obvious by Konno et al. and Kapil et al., in order to obtain “the position and outline of the objects and the meta information of the objects” (Konno et al.; see P[0125]), and in order to “improve mapping data collection” and in order to “improve route creation, assignment, monitoring, and validation” (Kapil et al.; see Abstract).
Regarding Claim 17, Khadem et al. teaches the claimed computing device according to claim 16, wherein the training is self-supervised and/or based on a reconstruction loss between the ground-truth machine-readable graph representation and the generated machine-readable graph representation of the generated road map generated by the graph generative model (“…supervised learning…”, see col.25, particularly lines 5067 and col.26, particularly lines 1-22).
Claim 19 is rejected under 35 U.S.C. 103 as being unpatentable over Khadem et al. (12,360,213) in view of Khan (2026/0235415) further in view of Kapil et al. (2024/0263964).
Regarding Claim 19, Khadem et al. teaches the claimed controller for a vehicle for automated driving (AD) comprising:
a reception interface configured to receive a comprising a machine-readable…representation…(“…the example process 100 may include a semantic classifier 128 configured to accept the 2D image 124 as input…”, see col.8, particularly lines 43-59 and FIG. 1)
; and
includes:
…
…locally allocated contextual information that is in view of applicable traffic regulations…(“…the example process 100 may include a semantic classifier 128 configured to accept the 2D image 124 as input…”, see col.8, particularly lines 43-59 and FIG. 1);
…and indicates, based at least in part on a characteristic of the line element, whether traversal between the connected lane pieces is possible and compliant with one or more of the applicable traffic regulations (“The segmentation component 444 may include a machine-learned (ML) model 448 trained for segmentation and/or classification tasks in 2D or 3D. As an example, the ML model 448 may comprise a fully convolutional network (FCN) trained to identify class labels (e.g., map elements such as “driving lane,” “crosswalk,” “stop line,” and the like) associated with individual pixels of a 2D input image (e.g., a top-down view of a road network)”, see col.23, particularly lines 59-67 and col.24, particularly lines 1-27, and see col.20, particularly lines 43-67 and col.21, particularly lines 1-3); and
a processing unit configured to:
perform the vehicle using the machine-readable…representation, the trajectory planning including evaluating the at least one attribute to determine whether a planned trajectory can include traversal between the connected lane pieces (“The segmentation component 444 may include a machine-learned (ML) model 448 trained for segmentation and/or classification tasks in 2D or 3D. As an example, the ML model 448 may comprise a fully convolutional network (FCN) trained to identify class labels (e.g., map elements such as “driving lane,” “crosswalk,” “stop line,” and the like) associated with individual pixels of a 2D input image (e.g., a top-down view of a road network)”, see col.23, particularly lines 59-67 and col.24, particularly lines 1-27, and see col.20, particularly lines 43-67 and col.21, particularly lines 1-3); and
control, based on the trajectory planning, automated driving of the vehicle along the planned trajectory (“The operation 514 may also include determining a trajectory for the vehicle or to otherwise control the vehicle to safely traverse the environment, based at least in part on map elements (e.g., driving lane, stop line, turn lane, and the like) indicated on the existing map. In examples, the operation 514 may be performed by the planning component 428”, see col.28, particularly lines 14-27).
Khadem et al. does not expressly recite the bolded portions of the claimed
a reception interface configured to receive a road map comprising a machine-readable graph representation that:
is a nested graph including (1) a first graph level in which at least some nodes represent points on lane-boundary lines and at least some edges connect neighboring points on the lane-boundary lines, wherein the lane-boundary lines represented in the first graph level include a line element that separates neighboring lane pieces, and (2) a second graph level in which at least some of nodes represent pieces of lanes and at least some of edges connect the pieces of lanes; and
includes:
nodes and edges representing road-layout structures of a road layout within at least one area drivable by [[a]] the vehicle, wherein the edges connect the nodes and represent relationships between the nodes; and
attributes assigned to at least a subset of the nodes or to at least a subset of the edges, the attributes representing locally allocated contextual information that is in view of applicable traffic regulations, wherein at least one of the attributes:
is assigned to an edge of the machine-readable graph representation that connects respective ones of the nodes representing connected lane pieces in the second graph level of the nested graph, wherein the connected lane pieces are the neighboring lane pieces separated by the line element represented in the first graph level; and
indicates, based at least in part on a characteristic of the line element, whether traversal between the connected lane pieces is possible and compliant with one or more of the applicable traffic regulations; and
a processing unit configured to:
perform trajectory planning of the vehicle using the machine-readable graph representation, the trajectory planning including evaluating the at least one attribute to determine whether a planned trajectory can include traversal between the connected lane pieces; and
control, based on the trajectory planning, automated driving of the vehicle along the planned trajectory.
However, Khan (2026/0235415) teaches a machine-readable graph representation that: is a nested graph including (1) a first graph level in which at least some nodes represent points on lane-boundary lines and at least some edges connect neighboring points on the lane-boundary lines, wherein the lane-boundary lines represented in the first graph level include a line element that separates neighboring lane pieces, and (2) a second graph level in which at least some of nodes represent pieces of lanes and at least some of edges connect the pieces of lanes (Khan; “A predetermined type of road structure (such as a roundabout) is detected by searching for any subgraph within the driving topology graph satisfying one or more criteria (such as a one-way loop). A roundabout detection method based on one-way driving loops is described. The principles can be extended to other forms of road structure. For example, another form of search may be implemented as a graph isomorphism (locating a subgraph that is isomorphic to some indicated graph structure)”, see P[0055] and “A lane graph is described, which is a graph encoding a road topology at the level of individual lanes (or ‘side-lanes’, as referred to herein), where each node represents a side-lane and edges in the graph represent links between side-lanes. One implementation of the map annotation tool operates on the lane graph, which is appropriate for identifying more ‘granular’ road structures defined in terms of side-lane topology. Another implementation uses a ‘coarser’ road topology graph, e.g., where nodes represent roads and edges represent links between road, which is appropriate for less granular road structure, e.g., roundabouts defined at the level of road (rather than lane) interconnections”, see P[0056]).
Furthermore, Kapil et al. (2024/0263964) teaches nodes and edges representing road-layout structures of a road layout within at least one area drivable by the vehicle, wherein the edges connect the nodes and represent relationships between the nodes (Kapil et al.; see P[0033]), and
attributes assigned to at least a subset of the nodes or to at least a subset of the edges, the attributes representing locally allocated contextual information that is in view of applicable traffic regulations, wherein at least one of the attributes: is assigned to an edge of the machine-readable graph representation that connects respective ones of the nodes representing connected lane pieces in the second graph level of the nested graph, wherein the connected lane pieces are the neighboring lane pieces separated by the line element represented in the first graph level; and indicates, based at least in part on a characteristic of the line element, whether traversal between the connected lane pieces is possible and compliant with one or more of the applicable traffic regulations (Kapil et al.; “…edge 434 of FIG. 4 has a 50 MPH speed limit, edge 448 has a corresponding 35 MPH speed limit, and edge 462 has a corresponding 20 MPH speed limit”, see P[0050]), and
perform trajectory planning of the vehicle using the machine-readable graph representation, the trajectory planning including evaluating the at least one attribute to determine whether a planned trajectory can include traversal between the connected lane pieces, and control, based on the trajectory planning, automated driving of the vehicle along the planned trajectory (Kapil et al.; “Once a road segment graph is created for a polygon or polygons of an ODD, a structured and strategic approach can be applied to create, schedule, optimize, and assign mapping routes to mapping vehicles in a fleet”, see P[0035] and “The control stack 1318 may manage the operation of the vehicle propulsion system 1330, the braking system 1332, the steering system 1334, the safety system 1336, and the cabin system 1338. The control stack 1318 may receive sensor signals from the sensor systems 1304-1308 as well as communicate with other stacks or components of the local computing device 1310 or a remote system (e.g., the data center 1350) to effectuate operation of the AV 1302)”, see P[0103] and “For example, the control stack 1318 may implement the final path or actions from the multiple paths or actions provided by the planning stack 1316. Implementation may involve turning the routes and decisions from the planning stack 1316 into commands for the actuators that control the AV's steering, throttle, brake, and drive unit”, see P[0104]).
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Khadem et al. with the teachings of Khan and Kapil et al., and to provide a reception interface configured to receive a road map comprising a machine-readable graph representation that is a nested graph including (1) a first graph level in which at least some nodes represent points on lane-boundary lines and at least some edges connect neighboring points on the lane-boundary lines, wherein the lane-boundary lines represented in the first graph level include a line element that separates neighboring lane pieces, and (2) a second graph level in which at least some of nodes represent pieces of lanes and at least some of edges connect the pieces of lanes, and includes nodes and edges representing road-layout structures of a road layout within at least one area drivable by the vehicle, wherein the edges connect the nodes and represent relationships between the nodes. and attributes assigned to at least a subset of the nodes or to at least a subset of the edges, the attributes representing locally allocated contextual information that is in view of applicable traffic regulations, wherein at least one of the attributes is assigned to an edge of the machine-readable graph representation that connects respective ones of the nodes representing connected lane pieces in the second graph level of the nested graph, wherein the connected lane pieces are the neighboring lane pieces separated by the line element represented in the first graph level, and indicates, based at least in part on a characteristic of the line element, whether traversal between the connected lane pieces is possible and compliant with one or more of the applicable traffic regulations, and a processing unit configured to perform trajectory planning of the vehicle using the machine-readable graph representation, the trajectory planning including evaluating the at least one attribute to determine whether a planned trajectory can include traversal between the connected lane pieces, and control, based on the trajectory planning, automated driving of the vehicle along the planned trajectory, as rendered obvious by Khan and Kapil et al., in order to provide “support tools for autonomous vehicles” and to “facilitate real-time planning, prediction and/or other online functions” (Khan; see P[0001]), and in order to “improve mapping data collection” and in order to “improve route creation, assignment, monitoring, and validation” (Kapil et al.; see Abstract).
Claim 21 is rejected under 35 U.S.C. 103 as being unpatentable over Khadem et al. (12,360,213) in view of Kapil et al. (2024/0263964) further in view of Pronovost et al. (12,662,135).
As per Claim 21, Khadem et al. does not expressly recite the claimed method according to claim 1, wherein the trained visual foundation model comprises an encoder configured to map the image input data to a latent representation and a decoder configured to decode the latent representation into the machine-readable graph representation.
However, Pronovost et al. (12,662,135) teaches wherein the trained visual foundation model comprises an encoder configured to map the image input data to a latent representation and a decoder configured to decode the latent representation into the machine-readable graph representation (Pronovost et al.; “…the variable autoencoder may be configured to receive any other type of driving scene representation for the encoder input and/or to generate any other type of driving scene representation as the decoder output, including but not limited to top-down image/sensor data, vehicle perspective image/sensor data, top-down multi-channel representations, bounding box (or bounding contour) representations, etc.” and “When training the diffusion model to generate a synthetic driving scene, a model training component may receive a latent embedding representing a ground truth driving scene including a construction zone, that has been encoded by the variable autoencoder” and “After the diffusion model has generated a synthetic driving scene including a construction zone (e.g., as a de-noised latent embedding), the driving scene generator may decode the latent embedding (e.g., using the decoder of the variable autoencoder) into a driving scene that can be used by various downstream components”, see col.9, particularly lines 19-67 and col.10, particularly lines 1-33).
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Khadem et al. with the teachings of Pronovost et al., and wherein the trained visual foundation model comprises an encoder configured to map the image input data to a latent representation and a decoder configured to decode the latent representation into the machine-readable graph representation, as rendered obvious by Pronovost et al., in order to provide for generating “synthetic driving scenes including traffic control zones” (Pronovost et al.; see Abstract) and “generating synthetic driving scenes for use by various other components, including on-vehicle components used to control an autonomous vehicle within a real-world driving environment” (Pronovost et al.; see col.10, lines 34-41).
Claim 23 is rejected under 35 U.S.C. 103 as being unpatentable over Khadem et al. (12,360,213) in view of Konno et al. (2025/0292506) further in view of Kapil et al. (2024/0263964), further in view of Pronovost et al. (12,662,135).
As per Claim 23, Khadem et al. does not expressly recite the claimed method according to claim 11, wherein the visual foundation model includes an encoder-decoder architecture, wherein an encoder is trained to map the top view image to a latent representation, and wherein a decoder is trained to map the latent representation to the generated machine-readable graph representation.
However, Pronovost et al. (12,662,135) teaches wherein the visual foundation model includes an encoder-decoder architecture, wherein an encoder is trained to map the top view image to a latent representation, and wherein a decoder is trained to map the latent representation to the generated machine-readable graph representation (Pronovost et al.; “…the variable autoencoder may be configured to receive any other type of driving scene representation for the encoder input and/or to generate any other type of driving scene representation as the decoder output, including but not limited to top-down image/sensor data, vehicle perspective image/sensor data, top-down multi-channel representations, bounding box (or bounding contour) representations, etc.” and “When training the diffusion model to generate a synthetic driving scene, a model training component may receive a latent embedding representing a ground truth driving scene including a construction zone, that has been encoded by the variable autoencoder” and “After the diffusion model has generated a synthetic driving scene including a construction zone (e.g., as a de-noised latent embedding), the driving scene generator may decode the latent embedding (e.g., using the decoder of the variable autoencoder) into a driving scene that can be used by various downstream components”, see col.9, particularly lines 19-67 and col.10, particularly lines 1-33).
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings of Khadem et al. with the teachings of Pronovost et al., and wherein the visual foundation model includes an encoder-decoder architecture, wherein an encoder is trained to map the top view image to a latent representation, and wherein a decoder is trained to map the latent representation to the generated machine-readable graph representation, as rendered obvious by Pronovost et al., in order to provide for generating “synthetic driving scenes including traffic control zones” (Pronovost et al.; see Abstract) and “generating synthetic driving scenes for use by various other components, including on-vehicle components used to control an autonomous vehicle within a real-world driving environment” (Pronovost et al.; see col.10, lines 34-41).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ISAAC G SMITH whose telephone number is (571)272-9593. The examiner can normally be reached Monday-Thursday, 8AM-5PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, ANISS CHAD can be reached at 571-270-3832. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ISAAC G SMITH/ Primary Examiner, Art Unit 3662
Prosecution Insights