Detailed Office Action
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This is a non-final Office Action on the merits. Claims 1-2 and 5-13 are currently pending and are addressed below.
Priority
Acknowledgment is made of applicant's claim priority for DE 10 2022 212 571.1 filed November 24, 2022.
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 02/20/2026 has been entered.
Reply to Applicant’s Remarks
Applicant’s remarks filed 01/27/2025 have been fully considered and are addressed as follows:
Claim Rejections Under 35 U.S.C. 101:
Applicant’s amendments to the claims filed 01/27/2025 have not overcome the 35 U.S.C 101 rejections previously set forth. Regarding the Applicant’s argument that “the amended claims integrate the alleged abstract ideas into the practical application of controlling driving of a vehicle”, however, the Examiner respectfully disagrees.
As drafted, the claim limitations recite that an output of the model is merely used for controlling the driving of the vehicle, thus the invention itself is not doing the controlling and therefore cannot integrate the abstract idea into a practical application. The examiner points to pg.3 of the Final Rejection office action of 12/03/2025 with details on how to remedy the amendment to overcome the current rejection:
“However, if the applicant amends to specify that the “controlling the function of the vehicle” is specifically the driving of the vehicle (i.e. controlling the driving, steering, acceleration, braking, etc.) the claim may overcome the current 101 rejection.
Further, it may be advised to more directly claim the controlling. The current amendment of “wherein an output is used for controlling…” is directed to the model outputting an output which is then secondarily used to control the vehicle, which as claimed, may or may not be performed by the invention. Therefore, using language such as “controlling the driving of the vehicle using an output of the model” may help to overcome the current 101 rejection.”
Alternatively to the suggested amendment above, amending the limitation to recite the causing of the controlling of the driving may also be effective in positively claiming the controlling of the vehicle and therefore integrating the abstract idea into a practical application, such as: “…and wherein an output of the model is used for and causes the controlling of the driving of the vehicle”
Further, the applicant argues “claim 1 integrates any alleged abstract ideas into a practical application and improves lane representations of lane markings and control of vehicles and/or the field of autonomous driving” and “the novel parametric representation and training method enable the model to advantageously "directly output the parameters of the line model," unlike conventional techniques, thereby improving how the machine learning model itself operates”, however, the examiner respectfully disagrees.
Because the claims only recite mental processes and insignificant extra solution activities, there are no additional elements that can integrate the abstract idea into a practical application. Further, the claim cannot provide an improvement to the technology as an improved abstract idea is still an abstract idea. (see MPEP 2106.05(a) Section II, “However, it is important to keep in mind that an improvement in the abstract idea…is not an improvement in technology”).
See below for detailed rejection.
Claim Rejections Under 35 U.S.C. 102 and 103:
Applicant’s amendments and/or arguments with respect to the rejection of claims 1-2 and 7-13 under 35 USC 102 and Claims 5-6 under 35 USC 103 as set forth in the office action of 12/03/2025 have been considered but are moot because the new ground(s) of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-2 and 5-13 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The analysis of the claims’ subject matter eligibility will follow the 2019 Revised Patent Subject Matter Eligibility Guidance, 84 Fed. Reg. 50-57 (January 7, 2019) (“2019 PEG”).
101 Analysis - With respect to Claim 1
Claims 1, 7, 10, 12 and 13 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
101 Analysis - Step 1:
Claim 1 is directed towards a method for training which is directed to the statutory category of a process. Claim 7 is directed towards a method for detecting lane lines which is directed to the statutory category of a process. Claim 10 is directed towards a model for detecting lane markings which is directed towards the statutory category of a machine. Claim 12 is directed towards a non-transitory computer readable medium which is directed towards the statutory category of a manufacture. Claim 13 is directed towards a device which is directed towards the statutory category of a machine. Therefore Claims 1, 7, 10, 12, and 13 are within at least one of the four statutory categories.
101 Analysis- Step 2A Prong One:
Regarding Prong One of the Step 2A analysis in the 2019 PEG, the claims are to be analyzed to determine whether they recite subject matter that falls within one of the following groups of abstract ideas: a) mathematical concepts, b) certain methods of organizing human activity, and/or c) mental process.
Independent claims 1, 7, and 10 include limitations that recite an abstract idea (emphasized below) and will be used as a representative claim for the remainder of the 101 rejection.
Claims 1, 12, and 13 recite, inter alai:
“A method for training a model to detect lane markings, comprising the following steps:
providing images that are specific to a recording by at least one sensor of a vehicle, and in which the lane markings are mapped;
providing ground truth data specific to a geometry of the lane markings in the provided images; and
training the model based on the provided images and the provided ground truth data for a three-dimensional modeling of the geometry of the lane markings based on a parameterization of a continuous curve,
wherein the three-dimensional modeling is based on the parameterization of the continuous curve by using at least two parameters of a line model to generate the continuous curve, including a first parameter indicating a first control point defining a lateral component of the continuous curve and a second parameter indicating a second control point defining a vertical component of the continuous curve, wherein the model is trained to output the at least two parameters, a probability value for a presence of the lane markings, and a start point and end point for a calculation of the continuous curve in the modeling, and wherein an output of the model is used for controlling driving of the vehicle”
Claim 7 recites, inter alai:
“A method for detecting lane markings, comprising the following steps:
recording images resulting from a recording by at least one sensor of a vehicle and in which the lane markings are mapped; and
detecting the lane markings in the images by applying a model trained for a three- dimensional modeling of a geometry of the lane markings based on a parameterization of a continuous curve,
wherein the three-dimensional modeling based on the parameterization of the continuous curve uses at least two parameters of a line model to generate the continuous curve, including a first parameter indicating a first control point defining a lateral component of the continuous curve and a second parameter indicating a second control point defining a vertical component of the continuous curve, wherein the model is trained to output the at least two parameters, a probability value for a presence of the lane markings, and a start point and end point for a calculation of the continuous curve in the modeling,
and wherein an output of the model is used for controlling driving of the vehicle.”
Claim 10 recites, inter alai:
“A model for detecting lane markings and for three-dimensional modeling of a geometry of the lane markings based on a parameterization of a continuous curve, wherein the model comprises a detection head having an output layer, wherein the output layer is configured to output a plurality of parameters for detecting and generating the continuous curve
wherein the three-dimensional modeling is based on the parameterization of the continuous curve by using at least two parameters of a line model to generate the continuous curve, including a first parameter indicating a first control point defining a lateral component of the continuous curve and a second parameter indicating a second control point defining a vertical component of the continuous curve, wherein the plurality of parameters includes the at least two parameters, a probability value for a presence of the lane markings, and a start point and end point for a calculation of the continuous curve in the modeling, and wherein an output of the model is used for controlling driving of a vehicle”
The examiner submits that the foregoing bolded limitation(s) constitute a “mental process” because under its broadest reasonable interpretation, the claim covers performance of the limitation in the human mind.
For example, “training”, “detecting”, and “modeling” in the context of these claims, all encompass a person looking at available data and forming a simple judgement (determination, analysis, comparison, etc.) either manually or using a pen and paper. Accordingly, the claims recite at least one abstract idea. The examiner notes that under MPEP 2106.04(a)(2)(III), the courts consider a mental process (thinking) that "can be performed in the human mind, or by a human using a pen and paper" to be an abstract idea. CyberSource Corp. v. Retail Decisions, Inc., 654 F.3d 1366, 1372, 99 USPQ2d 1690, 1695 (Fed. Cir. 2011). As the Federal Circuit explained, "methods which can be performed mentally, or which are the equivalent of human mental work, are unpatentable abstract ideas the ‘basic tools of scientific and technological work’ that are open to all.’" 654 F.3d at 1371, 99 USPQ2d at 1694 (citing Gottschalk v. Benson, 409 U.S. 63, 175 USPQ 673 (1972)). See also Mayo Collaborative Servs. v. Prometheus Labs. Inc., 566 U.S. 66, 71, 101 USPQ2d 1961, 1965 ("‘[M]ental processes[] and abstract intellectual concepts are not patentable, as they are the basic tools of scientific and technological work’" (quoting Benson, 409 U.S. at 67, 175 USPQ at 675)); Parker v. Flook, 437 U.S. 584, 589, 198 USPQ 193, 197 (1978) (same).
As drafted, the above claims, under their broadest reasonable interpretation, cover mental processes performed in the human mind (including an observation, evaluation, judgement, opinion), that are merely completed via generic computer components. Accordingly, the claims recite an abstract idea.
Step 2A Prong Two Analysis:
Regarding Prong Two of the Step 2A analysis in the 2019 PEG, the claims are to be analyzed to determine whether the claim, as a whole, integrates the abstract idea into a practical application. As noted in the 2019 PEG, it must be determined whether any additional elements in the claim beyond the abstract idea integrate the exception into a practical application in a manner that imposes a meaningful limit on the judicial exception. The courts have indicated that additional elements merely using a computer to implement an abstract idea, adding insignificant extra solution activity, or generally linking use of a judicial exception to a particular technological environment or field of use do not integrate a judicial exception into a “practical application”.
In the present case, the additional limitations beyond the above-noted abstract idea are as follows (where the underlined portions are the “additional limitations” while the bolded portions continue to represent the “abstract idea”):
Claims 1, 12, and 13 recite, inter alai:
“A method for training a model to detect lane markings, comprising the following steps:
providing images that are specific to a recording by at least one sensor of a vehicle, and in which the lane markings are mapped;
providing ground truth data specific to a geometry of the lane markings in the provided images; and
training the model based on the provided images and the provided ground truth data for a three-dimensional modeling of the geometry of the lane markings based on a parameterization of a continuous curve,
wherein the three-dimensional modeling is based on the parameterization of the continuous curve by using at least two parameters of a line model to generate the continuous curve, including a first parameter indicating a first control point defining a lateral component of the continuous curve and a second parameter indicating a second control point defining a vertical component of the continuous curve, wherein the model is trained to output the at least two parameters, a probability value for a presence of the lane markings, and a start point and end point for a calculation of the continuous curve in the modeling, and wherein an output of the model is used for controlling driving of the vehicle”
Claim 7 recites, inter alai:
“A method for detecting lane markings, comprising the following steps:
recording images resulting from a recording by at least one sensor of a vehicle and in which the lane markings are mapped; and
detecting the lane markings in the images by applying a model trained for a three- dimensional modeling of a geometry of the lane markings based on a parameterization of a continuous curve,
wherein the three-dimensional modeling based on the parameterization of the continuous curve uses at least two parameters of a line model to generate the continuous curve, including a first parameter indicating a first control point defining a lateral component of the continuous curve and a second parameter indicating a second control point defining a vertical component of the continuous curve, wherein the model is trained to output the at least two parameters, a probability value for a presence of the lane markings, and a start point and end point for a calculation of the continuous curve in the modeling,
and wherein an output of the model is used for controlling driving of the vehicle.”
Claim 10 recites, inter alai:
“A model for detecting lane markings and for three-dimensional modeling of a geometry of the lane markings based on a parameterization of a continuous curve, wherein the model comprises a detection head having an output layer, wherein the output layer is configured to output a plurality of parameters for detecting and generating the continuous curve
wherein the three-dimensional modeling is based on the parameterization of the continuous curve by using at least two parameters of a line model to generate the continuous curve, including a first parameter indicating a first control point defining a lateral component of the continuous curve and a second parameter indicating a second control point defining a vertical component of the continuous curve, wherein the plurality of parameters includes the at least two parameters, a probability value for a presence of the lane markings, and a start point and end point for a calculation of the continuous curve in the modeling, and wherein an output of the model is used for controlling driving of a vehicle”
For the following reason(s), the examiner submits that the above identified additional limitations do not integrate the above-noted abstract idea into a practical application.
Regarding the additional limitation of “wherein the three-dimensional modeling is based on …”, this limitation merely describes how to generally “apply” the otherwise mental judgements in a generic or general purpose vehicle control environment. See Alice Corp. Pty. Ltd. v. CLS Bank Int'l, 573 U.S. at 223 (“[T]he mere recitation of a generic computer cannot transform a patent-ineligible abstract idea into a patent-eligible invention.”). The device(s) and processor(s) are recited at a high level of generality and merely automates the steps.
Regarding the additional limitations of “wherein an output of the model…”, “outputting…” providing…”, and “recording” these limitation merely describes the sending, receiving, and output of data which are insignificant extra solution activities. See MPEP § 2106.05(g).
Thus, taken alone, the additional elements do not integrate the abstract idea into a practical application. Further, looking at the additional limitation(s) as an ordered combination or as a whole, the limitation(s) add nothing that is not already present when looking at the elements taken individually. Accordingly, the additional limitation(s) do/does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea.
Step 2B Analysis:
The claims do not include additional elements (considered both individually and as an ordered combination) that are sufficient to amount to significantly more than the judicial exception for the same reasons to those discussed above with respect to determining that the claim does not integrate the abstract idea into a practical application. As discussed above with respect to integration of the abstract idea into a practical application, the additional element of using generic computer components to perform the abstract idea amounts to no more than mere instructions to apply the exception using a generic computer component. Mere instructions to apply an exception using a generic computer component cannot provide an inventive concept. Further, the act of collecting data and displaying data amounts to no more than merely storing and displaying information of the exception and thus is an extra-solution activity. The claims are not patent eligible.
Regarding dependent claims 2, 5-6, 8-9 and 11 no claim further adds a limitation that introduces any practical applications to the claimed invention, the dependent claims merely add more mental process, mathematical concepts, and post-solution activities and are thus not patent eligible.
Therefore, Claims 1-2 and 5-13 are ineligible under 35 USC §101.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-2 and 7-13 is rejected under 35 U.S.C. 103 as being unpatentable over Elluswamy et al (US 20200249685 A1) in view of Maheshwari et al (US 20210209941 A1). Hereafter referred to as Elluswamy and Maheshwari respectively.
Regarding Claim 1, Elluswamy teaches a method for training a model to detect lane markings (see at least Elluswamy [¶ 11] A machine learning training technique for generating highly accurate machine learning results is disclosed. Using data captured by sensors on a vehicle to capture the environment of the vehicle and vehicle operating parameters, a training data set is created. For example, sensors affixed to a vehicle capture data such as image data of the road and the surrounding environment a vehicle is driving on. The sensor data may capture vehicle lane lines, vehicle lanes, other vehicle traffic, obstacles, traffic control signs, etc)
comprising the following steps:
providing images that are specific to a recording by at least one sensor of a vehicle, and in which the lane markings are mapped (see at least Elluswamy [¶ 7-8, FIG 5-6] FIG. 5 is a diagram illustrating an example of an image captured from a vehicle sensor...FIG. 6 is a diagram illustrating an example of an image captured from a vehicle sensor with predicted three-dimensional trajectories of lane lines)
providing ground truth data specific to a geometry of the lane markings in the provided images (see at least Elluswamy [¶11-12] a ground truth is determined based on a group of time series elements and is associated with a single element from the group…a series of images for a time period, such as 30 seconds, is used to determine the actual path of a vehicle lane line over the time period the vehicle travels. The vehicle lane line is determined by using the most accurate images of the vehicle lane over the time period....a three-dimensional representation of a feature, such as a lane line, is created from the group of time series elements that corresponds to the ground truth…Although the ground truth is determined based on the group of images, the selected first frame and the ground truth are used to create a training data)
training the model based on the provided images and the provided ground truth data for a three-dimensional modeling of the geometry of the lane markings based on a parameterization of a continuous curve (see at least Elluswamy [¶ 27, 31, 39] At 201, training data is prepared…sensor data including image data and odometry data is received to create a training data set. The sensor data may include still images and/or video from one or more cameras….the sensor data is a time series of elements and is used to determine a ground truth. The ground truth of the group is then associated with a subset of the time series, such as the first frame of image data… the ground truth are used to prepare the training data…the training data is prepared to train a machine learning model to only identify features from sensor data such as lane lines...by applying the trained machine learning model, three-dimensional representations of features, such as lane lines, are identified and/or predicted... the ground truth is represented as a three-dimensional representation such as a three-dimensional trajectory…the ground truth associated with a lane line may be represented as a three-dimensional parameterized spline or curve).
However, while Elluswamy teaches three-dimensional modeling of the geometry of lane markings based on a parameterization of a continuous curve, it does not explicitly teach wherein the three-dimensional modeling is based on the parameterization of the continuous curve by using at least two parameters of a line model to generate the continuous curve, including a first parameter indicating a first control point defining a lateral component of the continuous curve and a second parameter indicating a second control point defining a vertical component of the continuous curve
wherein the model is trained to output:
the at least two parameters
a probability value for a presence of the lane markings
a start point and end point for a calculation of the continuous curve in the modeling, and
wherein an output of the model is used for controlling driving of the vehicle.
Maheswari, in the same field as the endeavor, teaches wherein the three-dimensional modeling is based on the parameterization of the continuous curve by using at least two parameters of a line model to generate the continuous curve, including a first parameter indicating a first control point defining a lateral component of the continuous curve and a second parameter indicating a second control point defining a vertical component of the continuous curve (see at least Maheswari [¶ 32, 34, 109-110, 79] Each group of linked pixels includes two endpoints, also referenced herein as “control points.”…The principle component may have an associated direction, such as parallel to the direction of travel of the vehicle, perpendicular to the direction of travel of the vehicle, or any other potential direction...The system may construct a graph by first assigning each of the control points as nodes of the graph. The system may then connect each node to every other node of the graph by fitting a spline between each pair of nodes. It should be appreciated that the system may connect each pair of nodes using any suitable spline, such as a line, a cubic spiral, etc...As illustrated in FIG. 6H, the lane detection module 104 may analyze the control points 677a-677f to fit a spline 678 representative of the lane markings indicated by the linked subsections (e.g., subsection 648)…the lane detection module 104 may use any suitable splining method to generate the spline...A full point cloud may include a three-dimensional rendering of a scene captured by a lidar system, and a classification operation may classify each pixel of the full point cloud as a particular type of pixel (e.g., road surface, lane marking, lane boundary, vehicle, bike, sign, etc.). The pixels 601 in FIG. 6A may be selected to include only the pixels from the full point cloud that are relevant to lane detection...FIG. 5 depicts an example real-world driving environment 380, and FIG. 6A depicts an example point cloud 600 that is generated by a lidar system scanning the environment 380) The disclosure of Maheswari teaches using a plurality of control points to define a curve in both the directions of parallel and perpendicular to the travel of the vehicle, which is analogous to two parameters that define lateral and vertical components of a continuous curve.
wherein the model is trained to output:
the at least two parameters (see at least Maheshwari [¶ 32, 109] The principle component may have an associated direction, such as parallel to the direction of travel of the vehicle, perpendicular to the direction of travel of the vehicle, or any other potential direction…the lane detection module 104 may analyze the control points 677a-677f to fit a spline 678 representative of the lane markings)
a probability value for a presence of the lane markings (see at least Maheshwari [¶ 22, 38] FIG. 7B illustrates an example likelihood evaluation technique to determine how well the extended spline from FIG. 7A fits the actual lane markings...In any event, the system may generate a spline to fit the predicted lane marking locations. The system may then receive sensor data indicating the true locations of the lane markings. Using the sensor data, the system may determine an accuracy of the predicted lane marking locations, referenced in the present disclosure as a “likelihood score” (“L”). The system may generate numerous predictions, and determine a likelihood score L for each prediction)
a start point and end point for a calculation of the continuous curve in the modeling (see at least Maheshwari [¶ 147] The partitioning algorithm may include constructing a line between a start pixel of the set of pixels and an end pixel of the set of pixels (e.g., as illustrated in FIG. 6E). The lane marking detection module 104 may then determine a distance of each pixel between the start pixel and end pixel from the line. Based on the distance corresponding to each pixel, the lane marking detection module 104 may partition each pixel between the start pixel and end pixel into one group of the plurality of groups), and
wherein an output of the model is used for controlling driving of the vehicle (see at least Maheshwari [¶ 26] Software-based techniques of this disclosure are used to detect and track lane markings on a roadway, such that the lane markings may inform control operations of an autonomous vehicle. The vehicle may be a fully self-driving or “autonomous” vehicle).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention to have modified the system set forth in Elluswamy to contain a system for wherein the three-dimensional modeling is based on the parameterization of the continuous curve by using at least two parameters of a line model to generate the continuous curve, including a first parameter indicating a first control point defining a lateral component of the continuous curve and a second parameter indicating a second control point defining a vertical component of the continuous curve and wherein the model is trained to output: the at least two parameters, a probability value for a presence of the lane markings, a start point and end point for a calculation of the continuous curve in the modeling, and wherein an output of the model is used for controlling driving of the vehicle with reasonable expectation of success. One of ordinary skill in the art would have been motivated to make such a modification for benefit of improving the detection of lane markings by reducing computational resources needed and thus improving the safety of the autonomous driving vehicle as discussed in Maheshwari (see at least Maheshwari [¶ 26, 35] the disclosed techniques may be used to capture vehicle environment information to improve the safety/performance of an autonomous vehicle…the system implementing the methods of the present disclosure achieves a notable technological improvement over conventional systems by reducing the computational resources required to identify lane markings).
Regarding Claim 2, Elluswamy in view of Maheshwari teaches all limitations of Claim 1 as set forth above. Elluswamy further teaches wherein the model is implemented as a machine learning model that includes at least one artificial neural network with a detection head (see at least Elluswamy [¶ 15] Using the trained machine learning model, a neural network can infer features associated with autonomous driving such as vehicle lanes)
wherein the model is trained for the three-dimensional modeling of the geometry of the lane markings in that the continuous curve is generated based on an output of the detection head of the artificial neural network for representing the three-dimensional modeling of the geometry of the lane markings wherein the continuous curve is implemented as a three-dimensional curve (see at least Elluswamy [¶ 16] the image data is used as an input to a neural network trained to predict vehicle lanes. The machine learning model infers a three-dimensional trajectory for a detected lane. Instead of segmenting the image into lanes and non-lane segments of a two-dimensional image, a three-dimensional representation is inferred. In some embodiments, the three-dimensional representation is a spline, a parametric curve, or another representation capable of describing curves in three-dimensions).
Regarding Claim 7, Elluswamy teaches a method for detecting lane markings (see at least Elluswamy [¶ 11] A machine learning training technique for generating highly accurate machine learning results is disclosed. Using data captured by sensors on a vehicle to capture the environment of the vehicle and vehicle operating parameters, a training data set is created. For example, sensors affixed to a vehicle capture data such as image data of the road and the surrounding environment a vehicle is driving on. The sensor data may capture vehicle lane lines, vehicle lanes, other vehicle traffic, obstacles, traffic control signs, etc)
comprising the following steps:
recording images resulting from a recording by at least one sensor of a vehicle and in which the lane markings are mapped (see at least Elluswamy [¶ 7-8, FIG 5-6] FIG. 5 is a diagram illustrating an example of an image captured from a vehicle sensor...FIG. 6 is a diagram illustrating an example of an image captured from a vehicle sensor with predicted three-dimensional trajectories of lane lines)
detecting the lane markings in the images by applying a model trained a three-dimensional modeling of a geometry of the lane markings based on a parameterization of a continuous curve (see at least Elluswamy [¶ 27, 31, 39] At 201, training data is prepared…sensor data including image data and odometry data is received to create a training data set. The sensor data may include still images and/or video from one or more cameras….the sensor data is a time series of elements and is used to determine a ground truth. The ground truth of the group is then associated with a subset of the time series, such as the first frame of image data… the ground truth are used to prepare the training data…the training data is prepared to train a machine learning model to only identify features from sensor data such as lane lines...by applying the trained machine learning model, three-dimensional representations of features, such as lane lines, are identified and/or predicted... the ground truth is represented as a three-dimensional representation such as a three-dimensional trajectory…the ground truth associated with a lane line may be represented as a three-dimensional parameterized spline or curve)
However, while Elluswamy teaches three-dimensional modeling of the geometry of lane markings based on a parameterization of a continuous curve, it does not explicitly teach wherein the three-dimensional modeling is based on the parameterization of the continuous curve by using at least two parameters of a line model to generate the continuous curve, including a first parameter indicating a first control point defining a lateral component of the continuous curve and a second parameter indicating a second control point defining a vertical component of the continuous curve
wherein the model is trained to output:
the at least two parameters
a probability value for a presence of the lane markings
a start point and end point for a calculation of the continuous curve in the modeling, and
wherein an output of the model is used for controlling driving of the vehicle.
Maheswari, in the same field as the endeavor, teaches wherein the three-dimensional modeling is based on the parameterization of the continuous curve by using at least two parameters of a line model to generate the continuous curve, including a first parameter indicating a first control point defining a lateral component of the continuous curve and a second parameter indicating a second control point defining a vertical component of the continuous curve (see at least Maheswari [¶ 32, 34, 109-110, 79] Each group of linked pixels includes two endpoints, also referenced herein as “control points.”…The principle component may have an associated direction, such as parallel to the direction of travel of the vehicle, perpendicular to the direction of travel of the vehicle, or any other potential direction...The system may construct a graph by first assigning each of the control points as nodes of the graph. The system may then connect each node to every other node of the graph by fitting a spline between each pair of nodes. It should be appreciated that the system may connect each pair of nodes using any suitable spline, such as a line, a cubic spiral, etc...As illustrated in FIG. 6H, the lane detection module 104 may analyze the control points 677a-677f to fit a spline 678 representative of the lane markings indicated by the linked subsections (e.g., subsection 648)…the lane detection module 104 may use any suitable splining method to generate the spline...A full point cloud may include a three-dimensional rendering of a scene captured by a lidar system, and a classification operation may classify each pixel of the full point cloud as a particular type of pixel (e.g., road surface, lane marking, lane boundary, vehicle, bike, sign, etc.). The pixels 601 in FIG. 6A may be selected to include only the pixels from the full point cloud that are relevant to lane detection...FIG. 5 depicts an example real-world driving environment 380, and FIG. 6A depicts an example point cloud 600 that is generated by a lidar system scanning the environment 380) The disclosure of Maheswari teaches using a plurality of control points to define a curve in both the directions of parallel and perpendicular to the travel of the vehicle, which is analogous to two parameters that define lateral and vertical components of a continuous curve.
wherein the model is trained to output:
the at least two parameters (see at least Maheshwari [¶ 32, 109] The principle component may have an associated direction, such as parallel to the direction of travel of the vehicle, perpendicular to the direction of travel of the vehicle, or any other potential direction…the lane detection module 104 may analyze the control points 677a-677f to fit a spline 678 representative of the lane markings)
a probability value for a presence of the lane markings (see at least Maheshwari [¶ 22, 38] FIG. 7B illustrates an example likelihood evaluation technique to determine how well the extended spline from FIG. 7A fits the actual lane markings...In any event, the system may generate a spline to fit the predicted lane marking locations. The system may then receive sensor data indicating the true locations of the lane markings. Using the sensor data, the system may determine an accuracy of the predicted lane marking locations, referenced in the present disclosure as a “likelihood score” (“L”). The system may generate numerous predictions, and determine a likelihood score L for each prediction)
a start point and end point for a calculation of the continuous curve in the modeling (see at least Maheshwari [¶ 147] The partitioning algorithm may include constructing a line between a start pixel of the set of pixels and an end pixel of the set of pixels (e.g., as illustrated in FIG. 6E). The lane marking detection module 104 may then determine a distance of each pixel between the start pixel and end pixel from the line. Based on the distance corresponding to each pixel, the lane marking detection module 104 may partition each pixel between the start pixel and end pixel into one group of the plurality of groups), and
wherein an output of the model is used for controlling driving of the vehicle (see at least Maheshwari [¶ 26] Software-based techniques of this disclosure are used to detect and track lane markings on a roadway, such that the lane markings may inform control operations of an autonomous vehicle. The vehicle may be a fully self-driving or “autonomous” vehicle).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention to have modified the system set forth in Elluswamy to contain a system for wherein the three-dimensional modeling is based on the parameterization of the continuous curve by using at least two parameters of a line model to generate the continuous curve, including a first parameter indicating a first control point defining a lateral component of the continuous curve and a second parameter indicating a second control point defining a vertical component of the continuous curve and wherein the model is trained to output: the at least two parameters, a probability value for a presence of the lane markings, a start point and end point for a calculation of the continuous curve in the modeling, and wherein an output of the model is used for controlling driving of the vehicle with reasonable expectation of success. One of ordinary skill in the art would have been motivated to make such a modification for benefit of improving the detection of lane markings by reducing computational resources needed and thus improving the safety of the autonomous driving vehicle as discussed in Maheshwari (see at least Maheshwari [¶ 26, 35] the disclosed techniques may be used to capture vehicle environment information to improve the safety/performance of an autonomous vehicle…the system implementing the methods of the present disclosure achieves a notable technological improvement over conventional systems by reducing the computational resources required to identify lane markings).
Regarding Claim 8, Elluswamy in view of Maheshwari teaches all limitations of Claim 7 as set forth above. Elluswamy further teaches wherein the images are repeatedly recorded to continuously detect the lane markings in a vicinity of the vehicle during a trip (see at least Elluswamy [¶ 15] The sensor data may include an image (such as video and/or still images), radar, audio, lidar, inertia, odometry, location, and/or other forms of sensor data. The sensor data includes a group of time series elements. For example, a group of time series elements may include a group of images captured from a camera sensor of a vehicle over a time period)
wherein the following steps are carried out:
determining lane information based on the output of the model (see at least Elluswamy [¶ 21] deep learning network 105 is a deep learning network used for determining vehicle control parameters including analyzing the driving environment to determine lane markers, lanes, drivable space, obstacles, and/or potential vehicle paths, etc…deep learning network 105 may be an artificial neural network such as a convolutional neural network (CNN) that is trained on input such as sensor data and its output is provided to vehicle control module 109…the output may include at least a three-dimensional representation of lane markers)
evaluating the determined lane information by an autonomous driving function of the vehicle (see at least Elluswamy [¶ 18, 21] The output of deep learning network 105 running on AI processor 107 is fed to vehicle control module 109…vehicle control module 109 is connected to and controls the operation of the vehicle such as the speed, braking, and/or steering, etc. of the vehicle...deep learning network 105 is a deep learning network used for determining vehicle control parameters including analyzing the driving environment to determine lane markers, lanes, drivable space, obstacles, and/or potential vehicle paths, etc…deep learning network 105 may be an artificial neural network such as a convolutional neural network (CNN) that is trained on input such as sensor data and its output is provided to vehicle control module 109
initiating a control of the vehicle based on the evaluation (see at least Elluswamy [¶ 23] vehicle control module 109 is utilized to process the output of artificial intelligence (AI) processor 107 and to translate the output into a vehicle control operation…vehicle control module 109 is utilized to control the vehicle for autonomous driving…vehicle control module 109 can adjust speed, acceleration, steering, braking, etc. of the vehicle. For example, in some embodiments, vehicle control module 109 is used to control the vehicle to maintain the vehicle's position within a lane, to merge the vehicle into another lane, to adjust the vehicle's speed and lane positioning to account for merging vehicles, etc.).
Regarding Claim 9, Elluswamy in view of Maheshwari teaches all limitations of Claim 7 as set forth above. Elluswamy further teaches wherein the model is trained by:
providing images that are specific to a recording by at least one sensor of a vehicle, and in which the lane markings are mapped (see at least Elluswamy [¶ 7-8, FIG 5-6] FIG. 5 is a diagram illustrating an example of an image captured from a vehicle sensor...FIG. 6 is a diagram illustrating an example of an image captured from a vehicle sensor with predicted three-dimensional trajectories of lane lines)
providing ground truth data specific to a geometry of the lane markings in the provided images (see at least Elluswamy [¶11-12] a ground truth is determined based on a group of time series elements and is associated with a single element from the group…a series of images for a time period, such as 30 seconds, is used to determine the actual path of a vehicle lane line over the time period the vehicle travels. The vehicle lane line is determined by using the most accurate images of the vehicle lane over the time period....a three-dimensional representation of a feature, such as a lane line, is created from the group of time series elements that corresponds to the ground truth…Although the ground truth is determined based on the group of images, the selected first frame and the ground truth are used to create a training data)
training the model based on the provided images and the provided ground truth data for a three-dimensional modeling of the geometry of the lane markings based on a parameterization of a continuous curve (see at least Elluswamy [¶ 27, 31, 39] At 201, training data is prepared…sensor data including image data and odometry data is received to create a training data set. The sensor data may include still images and/or video from one or more cameras….the sensor data is a time series of elements and is used to determine a ground truth. The ground truth of the group is then associated with a subset of the time series, such as the first frame of image data… the ground truth are used to prepare the training data…the training data is prepared to train a machine learning model to only identify features from sensor data such as lane lines...by applying the trained machine learning model, three-dimensional representations of features, such as lane lines, are identified and/or predicted... the ground truth is represented as a three-dimensional representation such as a three-dimensional trajectory…the ground truth associated with a lane line may be represented as a three-dimensional parameterized spline or curve).
Regarding Claim 10, Elluswamy teaches a model for detecting lane markings and for three- dimensional modeling of a geometry of the lane markings based on a parameterization of a continuous curve (see at least Elluswamy [¶ 16] the image data is used as an input to a neural network trained to predict vehicle lanes. The machine learning model infers a three-dimensional trajectory for a detected lane. Instead of segmenting the image into lanes and non-lane segments of a two-dimensional image, a three-dimensional representation is inferred… the three-dimensional representation is a spline, a parametric curve, or another representation capable of describing curves in three-dimensions)
wherein the model comprises a detection head having an output layer, wherein the output layer is configured to output a plurality of parameters for detecting and generating the continuous curve (see at least Elluswamy [¶ 49] the ground truth is determined to detect objects such as lane lines, drivable space, traffic controls, vehicles, etc. in three dimensions…the ground truth is determined to detect objects such as lane lines, drivable space, traffic controls, vehicles, etc. in three dimensions)
However, while Elluswamy teaches three-dimensional modeling of the geometry of lane markings based on a parameterization of a continuous curve, it does not explicitly teach wherein the three-dimensional modeling is based on the parameterization of the continuous curve by using at least two parameters of a line model to generate the continuous curve, including a first parameter indicating a first control point defining a lateral component of the continuous curve and a second parameter indicating a second control point defining a vertical component of the continuous curve
wherein the plurality of parameters includes
the at least two parameters
a probability value for a presence of the lane markings
a start point and end point for a calculation of the continuous curve in the modeling, and
wherein an output of the model is used for controlling driving of the vehicle.
Maheswari, in the same field as the endeavor, teaches wherein the three-dimensional modeling is based on the parameterization of the continuous curve by using at least two parameters of a line model to generate the continuous curve, including a first parameter indicating a first control point defining a lateral component of the continuous curve and a second parameter indicating a second control point defining a vertical component of the continuous curve (see at least Maheswari [¶ 32, 34, 109-110, 79] Each group of linked pixels includes two endpoints, also referenced herein as “control points.”…The principle component may have an associated direction, such as parallel to the direction of travel of the vehicle, perpendicular to the direction of travel of the vehicle, or any other potential direction...The system may construct a graph by first assigning each of the control points as nodes of the graph. The system may then connect each node to every other node of the graph by fitting a spline between each pair of nodes. It should be appreciated that the system may connect each pair of nodes using any suitable spline, such as a line, a cubic spiral, etc...As illustrated in FIG. 6H, the lane detection module 104 may analyze the control points 677a-677f to fit a spline 678 representative of the lane markings indicated by the linked subsections (e.g., subsection 648)…the lane detection module 104 may use any suitable splining method to generate the spline...A full point cloud may include a three-dimensional rendering of a scene captured by a lidar system, and a classification operation may classify each pixel of the full point cloud as a particular type of pixel (e.g., road surface, lane marking, lane boundary, vehicle, bike, sign, etc.). The pixels 601 in FIG. 6A may be selected to include only the pixels from the full point cloud that are relevant to lane detection...FIG. 5 depicts an example real-world driving environment 380, and FIG. 6A depicts an example point cloud 600 that is generated by a lidar system scanning the environment 380) The disclosure of Maheswari teaches using a plurality of control points to define a curve in both the directions of parallel and perpendicular to the travel of the vehicle, which is analogous to two parameters that define lateral and vertical components of a continuous curve.
wherein the plurality of parameters includes
the at least two parameters (see at least Maheshwari [¶ 32, 109] The principle component may have an associated direction, such as parallel to the direction of travel of the vehicle, perpendicular to the direction of travel of the vehicle, or any other potential direction…the lane detection module 104 may analyze the control points 677a-677f to fit a spline 678 representative of the lane markings)
a probability value for a presence of the lane markings (see at least Maheshwari [¶ 22, 38] FIG. 7B illustrates an example likelihood evaluation technique to determine how well the extended spline from FIG. 7A fits the actual lane markings...In any event, the system may generate a spline to fit the predicted lane marking locations. The system may then receive sensor data indicating the true locations of the lane markings. Using the sensor data, the system may determine an accuracy of the predicted lane marking locations, referenced in the present disclosure as a “likelihood score” (“L”). The system may generate numerous predictions, and determine a likelihood score L for each prediction)
a start point and end point for a calculation of the continuous curve in the modeling (see at least Maheshwari [¶ 147] The partitioning algorithm may include constructing a line between a start pixel of the set of pixels and an end pixel of the set of pixels (e.g., as illustrated in FIG. 6E). The lane marking detection module 104 may then determine a distance of each pixel between the start pixel and end pixel from the line. Based on the distance corresponding to each pixel, the lane marking detection module 104 may partition each pixel between the start pixel and end pixel into one group of the plurality of groups), and
wherein an output of the model is used for controlling driving of the vehicle (see at least Maheshwari [¶ 26] Software-based techniques of this disclosure are used to detect and track lane markings on a roadway, such that the lane markings may inform control operations of an autonomous vehicle. The vehicle may be a fully self-driving or “autonomous” vehicle).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention to have modified the system set forth in Elluswamy to contain a system for wherein the three-dimensional modeling is based on the parameterization of the continuous curve by using at least two parameters of a line model to generate the continuous curve, including a first parameter indicating a first control point defining a lateral component of the continuous curve and a second parameter indicating a second control point defining a vertical component of the continuous curve and wherein the model is trained to output: the at least two parameters, a probability value for a presence of the lane markings, a start point and end point for a calculation of the continuous curve in the modeling, and wherein an output of the model is used for controlling driving of the vehicle with reasonable expectation of success. One of ordinary skill in the art would have been motivated to make such a modification for benefit of improving the detection of lane markings by reducing computational resources needed and thus improving the safety of the autonomous driving vehicle as discussed in Maheshwari (see at least Maheshwari [¶ 26, 35] the disclosed techniques may be used to capture vehicle environment information to improve the safety/performance of an autonomous vehicle…the system implementing the methods of the present disclosure achieves a notable technological improvement over conventional systems by reducing the computational resources required to identify lane markings).
Regarding Claim 11, Elluswamy in view of Maheshwari teaches all limitations of Claim 10 as set forth above. Elluswamy further teaches wherein the model is trained for detecting lane markings by:
providing images that are specific to a recording by at least one sensor of a vehicle, and in which the lane markings are mapped (see at least Elluswamy [¶ 7-8, FIG 5-6] FIG. 5 is a diagram illustrating an example of an image captured from a vehicle sensor...FIG. 6 is a diagram illustrating an example of an image captured from a vehicle sensor with predicted three-dimensional trajectories of lane lines)
providing ground truth data specific to a geometry of the lane markings in the provided images (see at least Elluswamy [¶11-12] a ground truth is determined based on a group of time series elements and is associated with a single element from the group…a series of images for a time period, such as 30 seconds, is used to determine the actual path of a vehicle lane line over the time period the vehicle travels. The vehicle lane line is determined by using the most accurate images of the vehicle lane over the time period....a three-dimensional representation of a feature, such as a lane line, is created from the group of time series elements that corresponds to the ground truth…Although the ground truth is determined based on the group of images, the selected first frame and the ground truth are used to create a training data)
training the model based on the provided images and the provided ground truth data for a three-dimensional modeling of the geometry of the lane markings based on a parameterization of a continuous curve (see at least Elluswamy [¶ 27, 31, 39] At 201, training data is prepared…sensor data including image data and odometry data is received to create a training data set. The sensor data may include still images and/or video from one or more cameras….the sensor data is a time series of elements and is used to determine a ground truth. The ground truth of the group is then associated with a subset of the time series, such as the first frame of image data… the ground truth are used to prepare the training data…the training data is prepared to train a machine learning model to only identify features from sensor data such as lane lines...by applying the trained machine learning model, three-dimensional representations of features, such as lane lines, are identified and/or predicted... the ground truth is represented as a three-dimensional representation such as a three-dimensional trajectory…the ground truth associated with a lane line may be represented as a three-dimensional parameterized spline or curve).
Regarding Claim 12, Elluswamy teaches a non-transitory computer-readable medium on which is stored a computer program including instructions for training a model to detect lane markings (see at least Elluswamy [Claim 19] a computer program product, the computer program product being embodied in a non-transitory computer readable storage medium and comprising computer instructions for: receiving image data based on an image captured by a camera of a vehicle; using the image data as a basis of an input to a trained machine learning model trained to predict a three-dimensional trajectory of a vehicle lane; and providing the three-dimensional trajectory of the vehicle lane in automatically controlling the vehicle)
the instructions, when executed by a computer, causing the computer to perform the following steps:
providing images that are specific to a recording by at least one sensor of a vehicle, and in which the lane markings are mapped (see at least Elluswamy [¶ 7-8, FIG 5-6] FIG. 5 is a diagram illustrating an example of an image captured from a vehicle sensor...FIG. 6 is a diagram illustrating an example of an image captured from a vehicle sensor with predicted three-dimensional trajectories of lane lines)
providing ground truth data specific to a geometry of the lane markings in the provided images (see at least Elluswamy [¶11-12] a ground truth is determined based on a group of time series elements and is associated with a single element from the group…a series of images for a time period, such as 30 seconds, is used to determine the actual path of a vehicle lane line over the time period the vehicle travels. The vehicle lane line is determined by using the most accurate images of the vehicle lane over the time period....a three-dimensional representation of a feature, such as a lane line, is created from the group of time series elements that corresponds to the ground truth…Although the ground truth is determined based on the group of images, the selected first frame and the ground truth are used to create a training data)
training the model based on the provided images and the provided ground truth data for a three-dimensional modeling of the geometry of the lane markings based on a parameterization of a continuous curve (see at least Elluswamy [¶ 27, 31, 39] At 201, training data is prepared…sensor data including image data and odometry data is received to create a training data set. The sensor data may include still images and/or video from one or more cameras….the sensor data is a time series of elements and is used to determine a ground truth. The ground truth of the group is then associated with a subset of the time series, such as the first frame of image data… the ground truth are used to prepare the training data…the training data is prepared to train a machine learning model to only identify features from sensor data such as lane lines...by applying the trained machine learning model, three-dimensional representations of features, such as lane lines, are identified and/or predicted... the ground truth is represented as a three-dimensional representation such as a three-dimensional trajectory…the ground truth associated with a lane line may be represented as a three-dimensional parameterized spline or curve)
However, while Elluswamy teaches three-dimensional modeling of the geometry of lane markings based on a parameterization of a continuous curve, it does not explicitly teach wherein the three-dimensional modeling is based on the parameterization of the continuous curve by using at least two parameters of a line model to generate the continuous curve, including a first parameter indicating a first control point defining a lateral component of the continuous curve and a second parameter indicating a second control point defining a vertical component of the continuous curve
wherein the model is trained to output:
the at least two parameters
a probability value for a presence of the lane markings
a start point and end point for a calculation of the continuous curve in the modeling, and
wherein an output of the model is used for controlling driving of the vehicle.
Maheswari, in the same field as the endeavor, teaches wherein the three-dimensional modeling is based on the parameterization of the continuous curve by using at least two parameters of a line model to generate the continuous curve, including a first parameter indicating a first control point defining a lateral component of the continuous curve and a second parameter indicating a second control point defining a vertical component of the continuous curve (see at least Maheswari [¶ 32, 34, 109-110, 79] Each group of linked pixels includes two endpoints, also referenced herein as “control points.”…The principle component may have an associated direction, such as parallel to the direction of travel of the vehicle, perpendicular to the direction of travel of the vehicle, or any other potential direction...The system may construct a graph by first assigning each of the control points as nodes of the graph. The system may then connect each node to every other node of the graph by fitting a spline between each pair of nodes. It should be appreciated that the system may connect each pair of nodes using any suitable spline, such as a line, a cubic spiral, etc...As illustrated in FIG. 6H, the lane detection module 104 may analyze the control points 677a-677f to fit a spline 678 representative of the lane markings indicated by the linked subsections (e.g., subsection 648)…the lane detection module 104 may use any suitable splining method to generate the spline...A full point cloud may include a three-dimensional rendering of a scene captured by a lidar system, and a classification operation may classify each pixel of the full point cloud as a particular type of pixel (e.g., road surface, lane marking, lane boundary, vehicle, bike, sign, etc.). The pixels 601 in FIG. 6A may be selected to include only the pixels from the full point cloud that are relevant to lane detection...FIG. 5 depicts an example real-world driving environment 380, and FIG. 6A depicts an example point cloud 600 that is generated by a lidar system scanning the environment 380) The disclosure of Maheswari teaches using a plurality of control points to define a curve in both the directions of parallel and perpendicular to the travel of the vehicle, which is analogous to two parameters that define lateral and vertical components of a continuous curve.
wherein the model is trained to output:
the at least two parameters (see at least Maheshwari [¶ 32, 109] The principle component may have an associated direction, such as parallel to the direction of travel of the vehicle, perpendicular to the direction of travel of the vehicle, or any other potential direction…the lane detection module 104 may analyze the control points 677a-677f to fit a spline 678 representative of the lane markings)
a probability value for a presence of the lane markings (see at least Maheshwari [¶ 22, 38] FIG. 7B illustrates an example likelihood evaluation technique to determine how well the extended spline from FIG. 7A fits the actual lane markings...In any event, the system may generate a spline to fit the predicted lane marking locations. The system may then receive sensor data indicating the true locations of the lane markings. Using the sensor data, the system may determine an accuracy of the predicted lane marking locations, referenced in the present disclosure as a “likelihood score” (“L”). The system may generate numerous predictions, and determine a likelihood score L for each prediction)
a start point and end point for a calculation of the continuous curve in the modeling (see at least Maheshwari [¶ 147] The partitioning algorithm may include constructing a line between a start pixel of the set of pixels and an end pixel of the set of pixels (e.g., as illustrated in FIG. 6E). The lane marking detection module 104 may then determine a distance of each pixel between the start pixel and end pixel from the line. Based on the distance corresponding to each pixel, the lane marking detection module 104 may partition each pixel between the start pixel and end pixel into one group of the plurality of groups), and
wherein an output of the model is used for controlling driving of the vehicle (see at least Maheshwari [¶ 26] Software-based techniques of this disclosure are used to detect and track lane markings on a roadway, such that the lane markings may inform control operations of an autonomous vehicle. The vehicle may be a fully self-driving or “autonomous” vehicle).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention to have modified the system set forth in Elluswamy to contain a system for wherein the three-dimensional modeling is based on the parameterization of the continuous curve by using at least two parameters of a line model to generate the continuous curve, including a first parameter indicating a first control point defining a lateral component of the continuous curve and a second parameter indicating a second control point defining a vertical component of the continuous curve and wherein the model is trained to output: the at least two parameters, a probability value for a presence of the lane markings, a start point and end point for a calculation of the continuous curve in the modeling, and wherein an output of the model is used for controlling driving of the vehicle with reasonable expectation of success. One of ordinary skill in the art would have been motivated to make such a modification for benefit of improving the detection of lane markings by reducing computational resources needed and thus improving the safety of the autonomous driving vehicle as discussed in Maheshwari (see at least Maheshwari [¶ 26, 35] the disclosed techniques may be used to capture vehicle environment information to improve the safety/performance of an autonomous vehicle…the system implementing the methods of the present disclosure achieves a notable technological improvement over conventional systems by reducing the computational resources required to identify lane markings).
Regarding Claim 13, Elluswamy teaches a device for data processing that is configured to train a model to detect lane markings (see at least Elluswamy [Abstract] A processor coupled to memory is configured to receive image data based on an image captured by a camera of a vehicle. The image data is used as a basis of an input to a trained machine learning model trained to predict a three-dimensional trajectory of a machine learning feature. The three-dimensional trajectory of the machine learning feature is provided for automatically controlling the vehicle)
the device configured to:
provide images that are specific to a recording by at least one sensor of a vehicle, and in which the lane markings are mapped (see at least Elluswamy [¶ 7-8, FIG 5-6] FIG. 5 is a diagram illustrating an example of an image captured from a vehicle sensor...FIG. 6 is a diagram illustrating an example of an image captured from a vehicle sensor with predicted three-dimensional trajectories of lane lines)
provide ground truth data specific to a geometry of the lane markings in the provided images (see at least Elluswamy [¶11-12] a ground truth is determined based on a group of time series elements and is associated with a single element from the group…a series of images for a time period, such as 30 seconds, is used to determine the actual path of a vehicle lane line over the time period the vehicle travels. The vehicle lane line is determined by using the most accurate images of the vehicle lane over the time period....a three-dimensional representation of a feature, such as a lane line, is created from the group of time series elements that corresponds to the ground truth…Although the ground truth is determined based on the group of images, the selected first frame and the ground truth are used to create a training data)
train the model based on the provided images and the provided ground truth data for a three-dimensional modeling of the geometry of the lane markings based on a parameterization of a continuous curve (see at least Elluswamy [¶ 27, 31, 39] At 201, training data is prepared…sensor data including image data and odometry data is received to create a training data set. The sensor data may include still images and/or video from one or more cameras….the sensor data is a time series of elements and is used to determine a ground truth. The ground truth of the group is then associated with a subset of the time series, such as the first frame of image data… the ground truth are used to prepare the training data…the training data is prepared to train a machine learning model to only identify features from sensor data such as lane lines...by applying the trained machine learning model, three-dimensional representations of features, such as lane lines, are identified and/or predicted... the ground truth is represented as a three-dimensional representation such as a three-dimensional trajectory…the ground truth associated with a lane line may be represented as a three-dimensional parameterized spline or curve)
However, while Elluswamy teaches three-dimensional modeling of the geometry of lane markings based on a parameterization of a continuous curve, it does not explicitly teach wherein the three-dimensional modeling is based on the parameterization of the continuous curve by using at least two parameters of a line model to generate the continuous curve, including a first parameter indicating a first control point defining a lateral component of the continuous curve and a second parameter indicating a second control point defining a vertical component of the continuous curve
wherein the model is trained to output:
the at least two parameters
a probability value for a presence of the lane markings
a start point and end point for a calculation of the continuous curve in the modeling, and
wherein an output of the model is used for controlling driving of the vehicle.
Maheswari, in the same field as the endeavor, teaches wherein the three-dimensional modeling is based on the parameterization of the continuous curve by using at least two parameters of a line model to generate the continuous curve, including a first parameter indicating a first control point defining a lateral component of the continuous curve and a second parameter indicating a second control point defining a vertical component of the continuous curve (see at least Maheswari [¶ 32, 34, 109-110, 79] Each group of linked pixels includes two endpoints, also referenced herein as “control points.”…The principle component may have an associated direction, such as parallel to the direction of travel of the vehicle, perpendicular to the direction of travel of the vehicle, or any other potential direction...The system may construct a graph by first assigning each of the control points as nodes of the graph. The system may then connect each node to every other node of the graph by fitting a spline between each pair of nodes. It should be appreciated that the system may connect each pair of nodes using any suitable spline, such as a line, a cubic spiral, etc...As illustrated in FIG. 6H, the lane detection module 104 may analyze the control points 677a-677f to fit a spline 678 representative of the lane markings indicated by the linked subsections (e.g., subsection 648)…the lane detection module 104 may use any suitable splining method to generate the spline...A full point cloud may include a three-dimensional rendering of a scene captured by a lidar system, and a classification operation may classify each pixel of the full point cloud as a particular type of pixel (e.g., road surface, lane marking, lane boundary, vehicle, bike, sign, etc.). The pixels 601 in FIG. 6A may be selected to include only the pixels from the full point cloud that are relevant to lane detection...FIG. 5 depicts an example real-world driving environment 380, and FIG. 6A depicts an example point cloud 600 that is generated by a lidar system scanning the environment 380) The disclosure of Maheswari teaches using a plurality of control points to define a curve in both the directions of parallel and perpendicular to the travel of the vehicle, which is analogous to two parameters that define lateral and vertical components of a continuous curve.
wherein the model is trained to output:
the at least two parameters (see at least Maheshwari [¶ 32, 109] The principle component may have an associated direction, such as parallel to the direction of travel of the vehicle, perpendicular to the direction of travel of the vehicle, or any other potential direction…the lane detection module 104 may analyze the control points 677a-677f to fit a spline 678 representative of the lane markings)
a probability value for a presence of the lane markings (see at least Maheshwari [¶ 22, 38] FIG. 7B illustrates an example likelihood evaluation technique to determine how well the extended spline from FIG. 7A fits the actual lane markings...In any event, the system may generate a spline to fit the predicted lane marking locations. The system may then receive sensor data indicating the true locations of the lane markings. Using the sensor data, the system may determine an accuracy of the predicted lane marking locations, referenced in the present disclosure as a “likelihood score” (“L”). The system may generate numerous predictions, and determine a likelihood score L for each prediction)
a start point and end point for a calculation of the continuous curve in the modeling (see at least Maheshwari [¶ 147] The partitioning algorithm may include constructing a line between a start pixel of the set of pixels and an end pixel of the set of pixels (e.g., as illustrated in FIG. 6E). The lane marking detection module 104 may then determine a distance of each pixel between the start pixel and end pixel from the line. Based on the distance corresponding to each pixel, the lane marking detection module 104 may partition each pixel between the start pixel and end pixel into one group of the plurality of groups), and
wherein an output of the model is used for controlling driving of the vehicle (see at least Maheshwari [¶ 26] Software-based techniques of this disclosure are used to detect and track lane markings on a roadway, such that the lane markings may inform control operations of an autonomous vehicle. The vehicle may be a fully self-driving or “autonomous” vehicle).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention to have modified the system set forth in Elluswamy to contain a system for wherein the three-dimensional modeling is based on the parameterization of the continuous curve by using at least two parameters of a line model to generate the continuous curve, including a first parameter indicating a first control point defining a lateral component of the continuous curve and a second parameter indicating a second control point defining a vertical component of the continuous curve and wherein the model is trained to output: the at least two parameters, a probability value for a presence of the lane markings, a start point and end point for a calculation of the continuous curve in the modeling, and wherein an output of the model is used for controlling driving of the vehicle with reasonable expectation of success. One of ordinary skill in the art would have been motivated to make such a modification for benefit of improving the detection of lane markings by reducing computational resources needed and thus improving the safety of the autonomous driving vehicle as discussed in Maheshwari (see at least Maheshwari [¶ 26, 35] the disclosed techniques may be used to capture vehicle environment information to improve the safety/performance of an autonomous vehicle…the system implementing the methods of the present disclosure achieves a notable technological improvement over conventional systems by reducing the computational resources required to identify lane markings).
Claims 5 and 6 are rejected under 35 U.S.C. 103 as being unpatentable over Elluswamy et al (US 20200249685 A1) in view of Maheshwari et al (US 20210209941 A1) and Liu et al (CN 108573242 A). Hereafter referred to as Elluswamy, Maheshwari, and Liu respectively.
Regarding Claim 5, Elluswamy in view of Maheswari teaches all limitations of Claim 1 as set forth above. Elluswamy further teaches wherein the model is implemented as a machine learning model that includes an artificial neural network and wherein the at least two parameters are determined based on an output of the artificial neural network (see at least Elluswamy [¶ 15, 47-49] Using the trained machine learning model, a neural network can infer features associated with autonomous driving such as vehicle lanes…the deep learning analysis is performed using a neural network such as a convolutional neural network (CNN)...the various outputs of deep learning are used to construct a three-dimensional representation of the vehicle's environment for autonomous driving which includes predicted paths of vehicles, identified obstacles, identified traffic control signals including speed limits, etc).
However, Elluswamy does not explicitly teach wherein the first control point and the second control point are B-spline control points that form a B-spline curve.
Maheshwari, in the same field as the endeavor teaches wherein the first control point and the second control point are B-spline control points that form a B-spline curve (see at least Maheshwari [¶ 34-35] The system may construct a graph by first assigning each of the control points as nodes of the graph. The system may then connect each node to every other node of the graph by fitting a spline between each pair of nodes. It should be appreciated that the system may connect each pair of nodes using any suitable spline…the system may utilize any suitable spline technique).
Further, while Maheshwari teaches that any suitable spline technique may be used, which includes B-splines, we can further see under the context of Liu that the use of B-splines for lane marking detection is known in the art (see at least Liu [English Translation, Abstract] The invention claims a lane detecting method and device. the lane detecting method comprising: using lane characteristic point model obtained from the camera of the original lane image detection lane characteristic point; the detected lane feature points to refine the lane for fine positioning characteristic point after refining to obtain the lane line fine location result, the lane line obtained fine location result for B-spline curve fitting to obtain the lane curve).
Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention to have modified the system set forth in Elluswamy to contain a system for wherein the first control point and the second control point are B-spline control points that form a B-spline curve with reasonable expectation of success. One of ordinary skill in the art would have been motivated to make such a modification for benefit of improving the detection of lane markings by reducing computational resources needed and thus improving the safety of the autonomous driving vehicle as discussed in Maheshwari (see at least Maheshwari [¶ 26, 35] the disclosed techniques may be used to capture vehicle environment information to improve the safety/performance of an autonomous vehicle…the system implementing the methods of the present disclosure achieves a notable technological improvement over conventional systems by reducing the computational resources required to identify lane markings).
Regarding Claim 6, Elluswamy in view of Maheshwari and Liu teaches all limitations of Claim 5 as set forth above. Elluswamy further teaches wherein the images are based on the recording by the at least one sensor of the vehicle (see at least Elluswamy [¶ 44] sensor data is received…a vehicle equipped with sensors captures sensor data and provides the sensor data to a neural network running on the vehicle. In some embodiments, the sensor data may be vision data, ultrasonic data, LiDAR data, or other appropriate sensor data. For example, an image is captured from a high dynamic range forward-facing camera)
in which the images map a lane in a direction of travel, and wherein the ground truth data specify the geometry of the lane markings on the lane by three-dimensional coordinates (see at least [¶ 7-8, 39] FIG. 5 is a diagram illustrating an example of an image captured from a vehicle sensor…FIG. 6 is a diagram illustrating an example of an image captured from a vehicle sensor with predicted three-dimensional trajectories of lane lines….the ground truth associated with a lane line may be represented as a three-dimensional parameterized spline or curve).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JOSEPH A YANOSKA whose telephone number is (703)756-5891. The examiner can normally be reached M-F 9:00am to 5:00pm (Pacific Time).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Rachid Bendidi can be reached on (571) 272-4896. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JOSEPH ANDERSON YANOSKA/Examiner, Art Unit 3664
/RACHID BENDIDI/Supervisory Patent Examiner, Art Unit 3664
Prosecution Insights