DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 05/13/2025 was considered by the examiner.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-30 are rejected under 35 U.S.C. 103 as being unpatentable over Rodriguez Hervas et al. (U.S. Patent App. Pub No. 2022/0379913 A1, hereafter referred as Rodriguez Hervas) in view of Kocamaz et al. (U.S. Patent App. Pub No. 2023/0099494 A1, hereafter referred as Kocamaz).
Regarding Claim 1:
Rodriguez Hervas teaches an apparatus for determining lane information (Rodriguez Hervas: Par. [0004]; the systems and methods of the present disclosure provide for using perception systems of machines (e.g., vehicles, robots, etc.) to detect and/or interpret signs and lanes—such as to associate signs with lanes), the apparatus comprising: at least one memory; and at least one processor coupled to the at least one memory and configured to (Rodriguez Hervas: Par. [0037]; various functions may be carried out by a processor executing instructions stored in memory): obtain an image representative of one or more lanes of a road and an object (Rodriguez Hervas: Par. [0057-0058]) and Fig. 4; depicts examples of lanes, signs, parameters, and attributes that may be derived from input data; the input data may correspond to an input frame 400, the input frame 400 (and/or other images and/or other sensor data representations (e.g., point clouds, projection images, etc.) generated using one or more sensors of the vehicle or machine 1000)), wherein the object is adjacent to the road (Rodriguez Hervas: Par. [0058] and Fig. 4; depicts signs 404A-C being adjacent to lanes 402A-C).
Rodriguez Hervas fails to teach and determine coordinates of object-to-lane association points of at least one lane of the one or more lanes of the road, wherein the coordinates are associated with the object.
Kocamaz, like Rodriguez Hervas, is directed to determining relationships between signs adjacent to the road and lanes of the road. Kocamaz in combination with Rodriguez Hervas does teach and determine coordinates of object-to-lane association points of at least one lane of the one or more lanes of the road (Rodriguez Hervas: Par. [0059] and Fig. 4; the geometry associator 126 may determine a segment, for example, based at least on determining a location associated with a sign along a lane and defining the segment of the lane using the location), wherein the coordinates are associated with the object (Kocamaz: Par. [0055-0056] and Fig. 3; determining at least a subset of pixels of the image corresponds to the at least one bounding shape, assigning an object class and a lane identifier corresponding to the respective combination of object class and lane identifier to at least the subset of pixels).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Rodriguez Hervas to utilize associating the pixels with the object, as taught by Kocamaz, to arrive at the claimed invention discussed above. Such a modification is the result of combining prior art elements according to known methods to yield predictable results. As taught by Kocamaz, the proposed modification would better allow for the association of objects to lanes with reduced processing time (Kocamaz: Par. [0005])
In regards to Claim 2, Rodriguez Hervas as modified by Kocamaz further teaches the apparatus of claim 1, wherein the coordinates comprise image coordinates (Kocamaz: Par. [0055-0056] and Fig. 3; determining at least a subset of pixels of the image corresponds to the at least one bounding shape, assigning an object class and a lane identifier corresponding to the respective combination of object class and lane identifier to at least the subset of pixels).
In regards to Claim 3, Rodriguez Hervas as modified by Kocamaz further teaches the apparatus of claim 1, wherein the coordinates comprise three-dimensional coordinates (Kocamaz: Par. [0021]; in other embodiments, the output masks may be in three-dimensional (3D) world space, such that the input data is from a top-down or bird's eye view representation of an environment around the ego-vehicle; as such, the pixel coordinates may correspond to (x, y) locations in 3D world space relative to an origin of the ego-vehicle).
In regards to Claim 4, Rodriguez Hervas as modified by Kocamaz further teaches the apparatus of claim 3, wherein the three-dimensional coordinates are relative to a camera which captured the image (Kocamaz: Par. [0021]; in other embodiments, the output masks may be in three-dimensional (3D) world space, such that the input data is from a top-down or bird's eye view representation of an environment around the ego-vehicle; as such, the pixel coordinates may correspond to (x, y) locations in 3D world space relative to an origin of the ego-vehicle with the image sensor).
In regards to Claim 5, Rodriguez Hervas as modified by Kocamaz further teaches the apparatus of claim 3, wherein the three-dimensional coordinates are relative to a reference coordinate system (Kocamaz: Par. [0021]; as such, the pixel coordinates may correspond to (x, y) locations in 3D world space relative to an origin of the ego-vehicle with the image sensor).
In regards to Claim 6, Rodriguez Hervas as modified by Kocamaz further teaches the apparatus of claim 1, wherein the object comprises a sign that relates to the at least one lane (Rodriguez Hervas: Par. [0059] and Fig. 4; the geometry associator 126 may determine a segment, for example, based at least on determining a location associated with a sign along a lane and defining the segment of the lane using the location).
In regards to Claim 7, Rodriguez Hervas as modified by Kocamaz further teaches the apparatus of claim 1, wherein the object comprises a road sign providing information that pertains to the at least one lane (Rodriguez Hervas: Par. [0059] and Fig. 4; the geometry associator 126 may determine a segment, for example, based at least on determining a location associated with a sign along a lane and defining the segment of the lane using the location).
In regards to Claim 8, Rodriguez Hervas as modified by Kocamaz further teaches the apparatus of claim 1, wherein the coordinates are indicative of the at least one lane to which the object relates (Rodriguez Hervas: Par. [0059] and Fig. 4; the geometry associator 126 may determine a segment, for example, based at least on determining a location associated with a sign along a lane and defining the segment of the lane using the location).
In regards to Claim 9, Rodriguez Hervas as modified by Kocamaz further teaches the apparatus of claim 1, wherein the coordinates comprise image coordinates that are laterally offset in the image from the object in the image (Kocamaz: Fig. 5A; showcases determining pixels that are laterally offset from the object).
In regards to Claim 10, Rodriguez Hervas as modified by Kocamaz further teaches the apparatus of claim 1, wherein the coordinates comprise image coordinates that are at a level of the road in the image (Kocamaz: Fig. 5A; showcases determining pixels that are level to the road).
In regards to Claim 11, Rodriguez Hervas as modified by Kocamaz further teaches the apparatus of claim 1, wherein the coordinates comprise image coordinates that are lower in the image than the object (Kocamaz: Fig. 5A; showcases determining pixels that are lower than the object).
In regards to Claim 12, Rodriguez Hervas as modified by Kocamaz further teaches the apparatus of claim 1, wherein the coordinates comprise image coordinates and wherein a line between the image coordinates is substantially perpendicular to a direction of travel of the at least one lane (Kocamaz: Fig. 5A; 530 showcases determining pixels within the lane and being able to produce a line perpendicular to travel direction).
In regards to Claim 13, Rodriguez Hervas as modified by Kocamaz further teaches the apparatus of claim 1, wherein, to determine the coordinates, the at least one processor is configured to: provide the image to a neural network trained to determine coordinates representative of object-to-lane association points associated with objects; and obtain the coordinates from the neural network (Kocamaz: Par. [0070-0074] and Fig. 6; computing, using a neural network and based at least in part on sensor data generated using one or more sensors, one or more output masks, each output mask of the one or more output masks corresponding to an object class and a lane identifier; includes assigning an object class and a lane identifier to the object based at least in part on the at least one object class label and the at least one lane identifier label associated with the one or more points).
In regards to Claim 14, Rodriguez Hervas as modified by Kocamaz further teaches the apparatus of claim 13, wherein the coordinates comprise image coordinates and wherein the neural network is trained to determine image coordinates of object-to-lane association points (Kocamaz: Par. [0070-0074] and Fig. 6; computing, using a neural network and based at least in part on sensor data generated using one or more sensors, one or more output masks, each output mask of the one or more output masks corresponding to an object class and a lane identifier; includes assigning an object class and a lane identifier to the object based at least in part on the at least one object class label and the at least one lane identifier label associated with the one or more points).
In regards to Claim 15, Rodriguez Hervas as modified by Kocamaz further teaches the apparatus of claim 13, wherein the coordinates comprise three-dimensional coordinates and wherein the neural network is trained to determine three-dimensional coordinates of object-to-lane association points (Kocamaz: Par. [0021]; in other embodiments, the output masks may be in three-dimensional (3D) world space, such that the input data is from a top-down or bird's eye view representation of an environment around the ego-vehicle; as such, the pixel coordinates may correspond to (x, y) locations in 3D world space relative to an origin of the ego-vehicle).
In regards to Claim 16, Rodriguez Hervas as modified by Kocamaz further teaches the apparatus of claim 1, wherein the at least one processor is further configured to: obtain lane boundaries related to the image; and associate the lane boundaries with the object based on the coordinates (Rodriguez Hervas: Par. [0025]; the lanes and/or the attributes may be derived from sensor data generated using at least one sensor of a machine (e.g., a vehicle), for example, using one or more machine learning models (MLMs), such as Convolutional Neural Networks (CNNs); examples of the lane attributes are those that represent geometry of lanes (e.g., curvature, shape, coordinates, lane boundaries)).
In regards to Claim 17, Rodriguez Hervas as modified by Kocamaz further teaches the apparatus of claim 16, wherein, to obtain the lane boundaries, the at least one processor is configured to: provide the image to a neural network trained to determine lane boundaries based on images; and obtain the lane boundaries from the neural network (Rodriguez Hervas: Par. [0025]; the lanes and/or the attributes may be derived from sensor data generated using at least one sensor of a machine (e.g., a vehicle), for example, using one or more machine learning models (MLMs), such as Convolutional Neural Networks (CNNs); examples of the lane attributes are those that represent geometry of lanes (e.g., curvature, shape, coordinates, lane boundaries)).
In regards to Claim 18, Rodriguez Hervas as modified by Kocamaz further teaches the apparatus of claim 16, wherein the lane boundaries are based on map information (Rodriguez Hervas: Par. [0023]; sign-to-path relevance information provided by a live perception system in accordance with various embodiments of the disclosure can be fused with information from a map (e.g., an HD map) to further enhance robustness and provide coverage in a wide range of real-world scenarios).
In regards to Claim 19, Rodriguez Hervas as modified by Kocamaz further teaches the apparatus of claim 1, wherein the at least one processor is further configured to: provide the image to a neural network trained to determine coordinates representative of lane edges associated with objects and lane boundaries; obtain the coordinates from the neural network; and obtain lane boundaries from the neural network (Rodriguez Hervas: Par. [0025]; the lanes and/or the attributes may be derived from sensor data generated using at least one sensor of a machine (e.g., a vehicle), for example, using one or more machine learning models (MLMs), such as Convolutional Neural Networks (CNNs); examples of the lane attributes are those that represent geometry of lanes (e.g., curvature, shape, coordinates, lane boundaries)).
In regards to Claim 20, Rodriguez Hervas as modified by Kocamaz further teaches the apparatus of claim 1, wherein the at least one processor is further configured to: provide the image to a neural network trained to determine bounding boxes; and obtain a bounding box related to the object from the neural network (Kocamaz: Par. [0070-0074] and Fig. 6; computing, using a neural network and based at least in part on sensor data generated using one or more sensors, one or more output masks, each output mask of the one or more output masks corresponding to an object class and a lane identifier; determining a bounding shape corresponding to at least one object based at least in part on the sensor data; includes assigning an object class and a lane identifier to the object based at least in part on the at least one object class label and the at least one lane identifier label associated with the one or more points).
In regards to Claim 21, Rodriguez Hervas as modified by Kocamaz further teaches the apparatus of claim 20, wherein the coordinates are determined based on the bounding box (Kocamaz: Par. [0070-0074] and Fig. 6; computing, using a neural network and based at least in part on sensor data generated using one or more sensors, one or more output masks, each output mask of the one or more output masks corresponding to an object class and a lane identifier; determining a bounding shape corresponding to at least one object based at least in part on the sensor data; includes assigning an object class and a lane identifier to the object based at least in part on the at least one object class label and the at least one lane identifier label associated with the one or more points).
In regards to Claim 22, Rodriguez Hervas as modified by Kocamaz further teaches the apparatus of claim 1, wherein the at least one processor is further configured to determine bird’s-eye-view coordinates corresponding to the object-to-lane association points of the at least one lane of the one or more lanes of the road based on the coordinates (Kocamaz: Par. [0021]; in other embodiments, the output masks may be in three-dimensional (3D) world space, such that the input data is from a top-down or bird's eye view representation of an environment around the ego-vehicle; as such, the pixel coordinates may correspond to (x, y) locations in 3D world space relative to an origin of the ego-vehicle).
In regards to Claim 23, Rodriguez Hervas as modified by Kocamaz further teaches the apparatus of claim 22, wherein the at least one processor is further configured to track the bird’s-eye-view coordinates based on successive images (Kocamaz: Par. [0021]; in other embodiments, the output masks may be in three-dimensional (3D) world space, such that the input data is from a top-down or bird's eye view representation of an environment around the ego-vehicle; as such, the pixel coordinates may correspond to (x, y) locations in 3D world space relative to an origin of the ego-vehicle; obvious that the input data could be successive images).
In regards to Claim 24, Rodriguez Hervas as modified by Kocamaz further teaches the apparatus of claim 1, wherein the at least one processor is further configured to determine three-dimensional coordinates corresponding to the object-to-lane association points of the at least one lane of the one or more lanes of the road based on the coordinates (Kocamaz: Par. [0021]; in other embodiments, the output masks may be in three-dimensional (3D) world space, such that the input data is from a top-down or bird's eye view representation of an environment around the ego-vehicle; as such, the pixel coordinates may correspond to (x, y) locations in 3D world space relative to an origin of the ego-vehicle).
In regards to Claim 25, Rodriguez Hervas as modified by Kocamaz further teaches the apparatus of claim 24, wherein the at least one processor is further configured to track the three-dimensional coordinates based on successive images (Kocamaz: Par. [0021]; in other embodiments, the output masks may be in three-dimensional (3D) world space, such that the input data is from a top-down or bird's eye view representation of an environment around the ego-vehicle; as such, the pixel coordinates may correspond to (x, y) locations in 3D world space relative to an origin of the ego-vehicle; obvious that the input data could be successive images).
In regards to Claim 26, Rodriguez Hervas as modified by Kocamaz further teaches the apparatus of claim 1, wherein the at least one processor is further configured to control a vehicle based on the coordinates (Rodriguez Hervas: Par. [0036]; disclosed embodiments may be comprised in a variety of different systems such as automotive systems (e.g., a control system for an autonomous or semi-autonomous machine, a perception system for an autonomous or semi-autonomous machine)).
In regards to Claim 27, Rodriguez Hervas as modified by Kocamaz further teaches the apparatus of claim 1, wherein the at least one processor is further configured to provide information to a driver of a vehicle based on the coordinates (Rodriguez Hervas: Par. [0089]; to enable autonomous driving and/or to assist a human driver in driving the vehicle 1000).
Regarding Claim 28:
Rodriguez Hervas as modified by Kocamaz further teaches a method for determining lane information (Rodriguez Hervas: Par. [0004]; the systems and methods of the present disclosure provide for using perception systems of machines (e.g., vehicles, robots, etc.) to detect and/or interpret signs and lanes—such as to associate signs with lanes), the method comprising: obtaining an image representative of one or more lanes of a road and an object (Rodriguez Hervas: Par. [0057-0058]) and Fig. 4; depicts examples of lanes, signs, parameters, and attributes that may be derived from input data; the input data may correspond to an input frame 400, the input frame 400 (and/or other images and/or other sensor data representations (e.g., point clouds, projection images, etc.) generated using one or more sensors of the vehicle or machine 1000)), wherein the object is adjacent to the road (Rodriguez Hervas: Par. [0058] and Fig. 4; depicts signs 404A-C being adjacent to lanes 402A-C); and determining coordinates of object-to-lane association points of at least one lane of the one or more lanes of the road (Rodriguez Hervas: Par. [0059] and Fig. 4; the geometry associator 126 may determine a segment, for example, based at least on determining a location associated with a sign along a lane and defining the segment of the lane using the location), wherein the coordinates are associated with the object (Kocamaz: Par. [0055-0056] and Fig. 3; determining at least a subset of pixels of the image corresponds to the at least one bounding shape, assigning an object class and a lane identifier corresponding to the respective combination of object class and lane identifier to at least the subset of pixels).
In regards to Claim 29, Rodriguez Hervas as modified by Kocamaz further teaches the method of claim 28, wherein the coordinates comprise image coordinates (Kocamaz: Par. [0055-0056] and Fig. 3; determining at least a subset of pixels of the image corresponds to the at least one bounding shape, assigning an object class and a lane identifier corresponding to the respective combination of object class and lane identifier to at least the subset of pixels).
In regards to Claim 30, Rodriguez Hervas as modified by Kocamaz further teaches the method of claim 28, wherein the coordinates comprise three-dimensional coordinates (Kocamaz: Par. [0021]; in other embodiments, the output masks may be in three-dimensional (3D) world space, such that the input data is from a top-down or bird's eye view representation of an environment around the ego-vehicle; as such, the pixel coordinates may correspond to (x, y) locations in 3D world space relative to an origin of the ego-vehicle).
Pertinent Art
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Tsai et al. (U.S. Patent App. Pub No. 2021/0110210 A1) teaches lane marking and road sign recognition.
Xu et al. (U.S. Patent App. Pub No. 2023/0267701 A1) teaches to compute a segmentation mask representative of portions of the image corresponding to lane markings of the driving surface of the vehicle.
Jain et al. (U.S. Patent App. Pub No. 2023/0206651 A1) teaches that lane location criteria and object class criteria may be used to determine a set of objects in an environment to track.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to RENAE BITOR whose telephone number is (703)756-5563. The examiner can normally be reached Monday to Friday: 8:00 - 5:30 but off the 1st Friday of the biweek.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, GREG MORSE can be reached on (571)272-3838. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/RENAE A BITOR/Examiner, Art Unit 2663
/GREGORY A MORSE/Supervisory Patent Examiner, Art Unit 2698