Detailed Action
This Office Action is taken in response to Applicant’s Amendment and Remarks filed on 4/13/2026 regarding Application No. 18/178,725 originally filed on 03/06/2023. Claims 1-2, 4-9, 11-16, 18-20 as filed are currently pending and have been considered as follows:
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
The applicant argues that “Ferguson does not assign a first label indicating an object type and then, based on both the factor features and that first label, assign a separate, different second label” and that Ferguson uses a “single-stage classification system” [Remarks, pp. 11-12]. The examiner respectfully disagrees.
The present rejection does not rely on Ferguson alone for the first label or the overall sensor-fusion framework. Liao teaches the LiDAR-camera fusion framework and the first label indicating object type. (as per “The output of each detection box includes a center position of the target X, Y, the width and height of the detection target, and the category information label of the detection target” in C4L35-45). Ferguson then teaches using object type and sensor characteristics to determine a different drivability classification. (as per “The type may then be fed into the classifier to determine the drivable or not drivable classification of the object” in ¶29, as per “sensor information defining characteristics of an object, such as the shape, height or other dimensions, location, speed, color, object type, etc.” in ¶30). Thus, Liao teaches the first object-type label, and Ferguson teaches a different second label, such as drivable, drivable if straddled, not drivable, or not drivable but likely to move away, based at least in part on object type and object characteristics.
The applicant argues that Ferguson’s confidence value is merely “an accuracy estimate for the actual classification” and does not indicate whether colliding with the obstacle would constitute safe driving [Remarks, p. 12]. The examiner respectfully disagrees.
Ferguson’s confidence value is assigned to the drivability classification, and the drivability classification itself determines whether the vehicle can safely drive over the object. Ferguson discloses that “each classification may be provided with a confidence value indicative of the likelihood that the object is safe for the vehicle to drive over or not” in ¶21 and that the classification and associated confidence value are used “to make a determination as to whether it is safe or not for the vehicle to drive over the object in real time” in ¶60. Therefore, even if the confidence value is an accuracy estimate for the classification, it is an accuracy estimate for a classification directed to whether drive-over or contact with the object is safe.
The applicant argues that Ferguson filters out vehicles, bicyclists, and pedestrians and is limited to debris-type objects [Remarks, pp. 11-12]. The examiner respectfully disagrees.
Ferguson is not relied upon as the sole obstacle-detection system. Liao supplies the general detection and object-type labeling framework for the detected obstacles. Ferguson is relied upon for the known technique of assigning a different drivability label with confidence to an object for vehicle control. Ferguson also teaches that object type may be used in determining the drivability classification. (as per ¶29-¶30). Accordingly, the fact that Ferguson describes filtering in one embodiment does not negate its teaching of assigning a different drivability classification and confidence value based on object characteristics and object type.
Applicant’s other arguments with respect to the claim(s) have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1-5, 7-12, 14-18, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Liao (US Pat. No. 11403860) in view of Ferguson (US Pub. No. 20180032078) in view of Prediger (US Pub. No. 20240190422).
As per Claim 1, Liao discloses of a multi-sensor object detection fusion system and method using point cloud projection, comprising:
generating one or more data points from one or more sensors coupled to a vehicle, (as per “the system 10 includes a LiDAR assembly 100, a camera assembly 200, and a controller 300” in C2L10-20, as per “the system 10 acquires point cloud data using the LiDAR assembly 100 and image data using the camera assembly 200 of one or more objects. It is preferable that the point cloud data and the image data of the one or more objects are taken from the same angle of view (AOV)” in C2L45-55) wherein:
the one or more sensors comprise:
a Light Detection and Ranging (LiDAR) sensor; and a camera, and (as per “the system 10 includes a LiDAR assembly 100, a camera assembly 200, and a controller 300” in C2L10-20)
the one or more data points comprise: a LiDAR point cloud generated by the LiDAR sensor; and an image captured by the camera; (as per “the system 10 acquires point cloud data using the LiDAR assembly 100 and image data using the camera assembly 200 of one or more objects. It is preferable that the point cloud data and the image data of the one or more objects are taken from the same angle of view (AOV). In doing so, the system 10 detects the one or more objects using the LiDAR assembly 100 and/or the camera assembly 200. The system 10 then classifies the one or more objects using the point cloud data and/or the image data” in C2L45-60)
detecting one or more obstacles within the LiDAR point cloud; (as per “The system 10 then classifies the one or more objects using the point cloud data and/or the image data. The system 10 may utilize an object detection algorithm” in C2L50-65, as per “the system 10 processes the point cloud data to obtain a LiDAR 3D object bounding box, and the image data to obtain a camera 2D object bounding box” in C2L55-65)
generating at least one patch in the LiDAR point cloud indicative of the one or more obstacles; (as per “the system 10 (i) segments the point cloud data at a non-ground point, (ii) clusters the point cloud data of an object with an adaptive threshold Euclidean clustering algorithm, and (iii) outputs the corresponding 3D object bounding box of the object” in C4L5-20)
projecting the at least one patch of the LiDAR point cloud onto the image to obtain combined data, wherein said at least one patch designates a region of the image that is to be analyzed; (as per “the system 10 processes the 3D object bounding box resulting from 700 by projecting, according to the external parameter matrix determined in 500, the 3D object bounding box to a pixel coordinate system where the 2D object bounding box is located. The system 10 then acquires, in the pixel coordinate system, corresponding corner coordinate(s) and a center coordinate of the 2D object detection box output in 700” in C4L20-30, as per “This includes transforming a point cloud data coordinate from a LiDAR coordinate system to a camera coordinate system and transforming the point cloud coordinate from the camera coordinate system to the pixel coordinate system…” in C5L55-65, as per “The point cloud data transformed into the pixel coordinate system determines the rectangular area according to its size in the u and v directions in the pixel coordinate system” in C4L50-60)
performing a factor query using the combined data to query at least one factor feature for points of the at least one patch associated with each of the one or more obstacles; (as per “The output of the 2D target detection box includes a rectangular detection box. The output of each detection box includes a center position of the target X, Y, the width and height of the detection target, and the category information label of the detection target. In some embodiments, the LiDAR point cloud is processed through original data analysis, LiDAR data preprocessing and point cloud segmentation” in C4L35-45, as per “The point cloud data transformed into the pixel coordinate system determines the rectangular area according to its size in the u and v directions in the pixel coordinate system” in C4L50-60, as per “The target after clustering preferably includes a 3D point cloud group. The 3D point cloud points are then transformed into the camera coordinate system through the external parameter transformation matrix determined at 500. At this point, the transformation from 3D point cloud data to 2D pixel points in the camera coordinate system may be completed” in C4L40-55)
for each obstacle of the one or more obstacles, based on the at least one factor feature and a first label indicating an object type of a plurality of object types, (as per “The output of the 2D target detection box includes a rectangular detection box. The output of each detection box includes a center position of the target X, Y, the width and height of the detection target, and the category information label of the detection target. In some embodiments, the LiDAR point cloud is processed through original data analysis, LiDAR data preprocessing and point cloud segmentation” in C4L35-45)
Liao fails to expressly disclose:
labeling the obstacle with a second label that is different from the first label, the second label being assigned a confidence value
wherein the confidence value indicates whether colliding with the obstacle would constitute safe driving; and
based on the first and second labels of the one or more obstacles, planning a trajectory of the vehicle,
the planning the trajectory comprising:
comparing the confidence value to a threshold value;
accepting one or more plans that would collide with the one or more obstacles, when the confidence value is above the threshold value; and
not accepting one or more plans that would collide with the one or more obstacles, when the confidence value is below the threshold value.
Ferguson discloses of determining drivability of objects for autonomous vehicles, comprising:
labeling the obstacle with a second label that is different from the first label, the second label being assigned a confidence value, (as per “. The type may then be fed into the classifier to determine the drivable or not drivable classification of the object. Alternatively, sub-classifications within these classifications may also be made by the classifier corresponding to the type of the object. For instance, in the case of an object classified as drivable, the object may be further classified by the type of drivable object such as paper, plastic bag, leaves, etc” in ¶29)
wherein the confidence value indicates whether colliding with the obstacle would constitute safe driving; (as per “The classifier may then classify the drivability of a particular object, or rather, a determination of whether the object is drivable safe for the vehicle to drive over or not in real time. Each classification may be provided with a confidence value indicative of the likelihood that the object is safe for the vehicle to drive over or not. This confidence value may be compared with a threshold value to determine whether the object is to be classified as drivable or not drivable” in ¶21, as per “computing device 110 may then use the classification (or classifications) and associated confidence value (or values) to make a determination as to whether it is safe or not for the vehicle to drive over the object in real time, for instance, as the vehicle approaches the object” in ¶60)
based on the first and second labels of the one or more obstacles, planning a trajectory of the vehicle, (as per “If object 686 is classified as not drivable, the computing device 110 may cause the vehicle to stop or maneuver around the object. In addition, if object 686 is classified as not drivable but likely to move away on its own, the computing device 110 may cause the vehicle to slow down as the vehicle approaches the object, in order to give the object a greater amount of time to move out of the expected future path of the vehicle before the vehicle reaches the object. Of course, if the object does not move out of the expected future path of the vehicle, the computing device 110 may cause the vehicle to come to a complete stop or maneuver the vehicle around the object” in ¶65)
In this way, Ferguson operates to determine drivability of an object by associating a confidence value with whether an object is safe for the vehicle to drive over or not and using the classification and associated confidence value “to make a determination as to whether it is safe or not for the vehicle to drive over the object in real time” (¶60). Like Liao, Ferguson is concerned with vehicle systems.
It would have been obvious for one of ordinary skill in the art before the effective filing date to have modified the multi-sensor object detection fusion system by Liao with the obstacle drivability confidence determination as taught by Ferguson to enable another standard means determining whether colliding with or driving over an obstacle would constitute safe driving (¶21, ¶60). Such modification also allows the system to differentiate between objects which the vehicle must drive around to avoid an accident versus objects which the vehicle can drive over, thereby avoiding unnecessary maneuvering or stopping for objects that are safe to drive over and improving overall safety (¶16).
Liao and Ferguson fail to expressly disclose:
the planning the trajectory comprising:
comparing the confidence value to a threshold value;
accepting one or more plans that would collide with the one or more obstacles, when the confidence value is above the threshold value; and
not accepting one or more plans that would collide with the one or more obstacles, when the confidence value is below the threshold value.
Prediger discloses of drivable path determination for a vehicle, comprising:
the planning the trajectory comprising:
comparing the confidence value to a threshold value; (as per “determining a drivable path for a vehicle during a ride comprises the steps of: obtaining sensor data indicating a perceived obstacle associated with a geographical location; retrieving, from a data storage, data associated with the perceived obstacle, the data comprising a confidence score; and if the confidence score is above a threshold, determining the perceived obstacle as overdrivable for the vehicle” in ¶9, as per “When the vehicle overdrives the perceived obstacle, which is possible if the obstacle is just a bump or a crack in a road, the confidence score is increased to express the fact that the obstacle is actually overdrivable. Similarly, the confidence score may be increased when the sensor data allows to draw the conclusion that the perceived obstacle can be overdriven. Hence, it is possible to increase the confidence score each time the vehicle passes or drives over the perceived obstacle. With this, it can be learned that the obstacle is in fact overdrivable. Therefore, if the vehicle once again comes across the perceived obstacle during another ride or pass-by, it may determine, based on the confidence score, that the perceived obstacle is actually overdrivable and may thus include the location of the obstacle in its path planning.” in ¶12)
accepting one or more plans that would collide with the one or more obstacles, when the confidence value is above the threshold value; (as per “Then, in step S103, the value of the confidence score may be determined. If the confidence score is above a threshold, the perceived obstacle may be considered as overdrivable, e.g., as drivable area for the vehicle (S104)” in ¶61, as per “If the perceived obstacle is considered overdrivable, which may be the case if the confidence score is above a threshold, then the geographical location associated with the perceived obstacle may be included in the drivable path. In contrast, if the perceived obstacle is considered non-overdrivable, then the geographical location associated with the perceived obstacle is not included in the path. This ensures that only overdrivable objects are part of the planned path” in ¶22)
not accepting one or more plans that would collide with the one or more obstacles, when the confidence value is below the threshold value. (as per “if the confidence score is below the threshold, determining the perceived obstacle as non-overdrivable for the vehicle.” in ¶13, as per “If the perceived obstacle is considered overdrivable, which may be the case if the confidence score is above a threshold, then the geographical location associated with the perceived obstacle may be included in the drivable path. In contrast, if the perceived obstacle is considered non-overdrivable, then the geographical location associated with the perceived obstacle is not included in the path. This ensures that only overdrivable objects are part of the planned path” in ¶22)
In this way, Prediger operates to determine a drivable path for a vehicle by comparing a confidence score associated with a perceived obstacle to a threshold and determining the perceived obstacle as overdrivable or non-overdrivable based on the comparison (¶9, ¶13). Like Liao and Ferguson, Prediger is concerned with vehicle systems.
It would have been obvious for one of ordinary skill in the art before the effective filing date to have modified the multi-sensor object detection fusion system by Liao and the obstacle drivability confidence determination by Ferguson with the drivable path determination as taught by Prediger to enable another standard means determining whether to include or exclude an obstacle location from a drivable path based on an obstacle-associated confidence score (¶21-¶22). Such modification also allows the system to include the geographical location associated with a perceived obstacle in the drivable path when the perceived obstacle is considered overdrivable and exclude the geographical location associated with the perceived obstacle from the drivable path when the perceived obstacle is considered non-overdrivable, thereby ensuring that only overdrivable objects are part of the planned path (¶22).
As per Claim 2, the combination of Liao, Ferguson, and Prediger teaches or suggests all limitations of Claim 1. Liao fails to expressly disclose wherein the plurality of object types comprise:
a piece of vegetation;
a pedestrian;
not a pedestrian; and/or a vehicle.
See Claim 1 for teachings of Ferguson. Ferguson further discloses wherein the plurality of object types comprise:
a piece of vegetation; (as per “As an example, these maps may identify the shape and elevation of roadways, lane markers, intersections, crosswalks, speed limits, traffic signal lights, buildings, signs, real time traffic information, vegetation, or other such objects and information” in ¶40)
a pedestrian; (as per “objects such as vehicles, pedestrians and bicyclists may be readily identifiable from their visual characteristics (using image recognition techniques), physical characteristics (size, shape, etc.), speed (relative to the vehicle 100 or actual speed), and location (in a lane, in a crosswalk, on a sidewalk, etc.) captured by lasers or camera sensors of the perception system. Of course, the same may not be true for road debris, small animals, or other such items which can appear in the roadway. This sensor information may be sent to and received by the computing device 110. In this regard, the sensor information may include raw sensor data and/or other information describing the characteristics extracted from the raw sensor data such as a descriptive function or vector” in ¶36)
not a pedestrian; and/or a vehicle. (as per “As shown in the example of FIG. 6, the perception system 172 has detected and identified a vehicle 680, a bicyclist 682, and a pedestrian 684. In addition, the perception system has detected objects 686 and 688 that do not correspond to vehicles, bicyclists or pedestrians” in ¶53)
In this way, Ferguson operates to determine drivability of an object by associating a confidence value with whether an object is safe for the vehicle to drive over or not and using the classification and associated confidence value “to make a determination as to whether it is safe or not for the vehicle to drive over the object in real time” (¶60). Like Liao and Prediger, Ferguson is concerned with vehicle systems.
It would have been obvious for one of ordinary skill in the art before the effective filing date to have modified the multi-sensor object detection fusion system by Liao and the drivable path determination of Prediger with the obstacle drivability confidence determination as taught by Ferguson to enable another standard means determining whether colliding with or driving over an obstacle would constitute safe driving (¶21, ¶60). Such modification also allows the system to differentiate between objects which the vehicle must drive around to avoid an accident versus objects which the vehicle can drive over, thereby avoiding unnecessary maneuvering or stopping for objects that are safe to drive over and improving overall safety (¶16).
As per Claim 4, the combination of Liao, Ferguson, and Prediger teaches or suggests all limitations of Claim 1. Liao fails to expressly disclose wherein the planning the trajectory comprises using the processor:
for each of the one or more obstacles, based on the second label of the obstacle,
determining one or more vehicle actions for the vehicle to perform; and
causing the vehicle to perform the one or more actions.
See Claim 1 for teachings of Ferguson. Ferguson further discloses wherein the planning the trajectory comprises using the processor:
for each of the one or more obstacles, based on the second label of the obstacle, determining one or more vehicle actions for the vehicle to perform; and causing the vehicle to perform the one or more actions. (as per “The classification may then be used to control the vehicle. For instance, as an example, if an object is identified as drivable, the vehicle may proceed to drive over the object. Alternatively, if an object is classified as not drivable, the vehicle may stop or maneuver around the object. As noted above, by classifying the drivability of objects, unnecessary maneuvering or stopping, for instance to avoid a plastic bag or other similar debris, can be avoided” in ¶22, as per “For instance, if object 686 is determined to be drivable, the computing device 110 may cause the vehicle to proceed to drive over the object. If object 686 is classified as drivable if straddled, the computing device 110 may cause the vehicle maneuver to drive over the object such that the object is positioned between the wheels (i.e. driver and passenger side wheels) of the vehicle. If object 686 is classified as not drivable, the computing device 110 may cause the vehicle to stop or maneuver around the object. In addition, if object 686 is classified as not drivable but likely to move away on its own, the computing device 110 may cause the vehicle to slow down as the vehicle approaches the object, in order to give the object a greater amount of time to move out of the expected future path of the vehicle before the vehicle reaches the object. Of course, if the object does not move out of the expected future path of the vehicle, the computing device 110 may cause the vehicle to come to a complete stop or maneuver the vehicle around the object” in ¶65)
In this way, Ferguson operates to determine drivability of an object by associating a confidence value with whether an object is safe for the vehicle to drive over or not and using the classification and associated confidence value “to make a determination as to whether it is safe or not for the vehicle to drive over the object in real time” (¶60). Like Liao and Prediger, Ferguson is concerned with vehicle systems.
It would have been obvious for one of ordinary skill in the art before the effective filing date to have modified the multi-sensor object detection fusion system by Liao and the drivable path determination of Prediger with the obstacle drivability confidence determination as taught by Ferguson to enable another standard means determining whether colliding with or driving over an obstacle would constitute safe driving (¶21, ¶60). Such modification also allows the system to differentiate between objects which the vehicle must drive around to avoid an accident versus objects which the vehicle can drive over, thereby avoiding unnecessary maneuvering or stopping for objects that are safe to drive over and improving overall safety (¶16).
As per Claim 5, the combination of Liao, Ferguson, and Prediger teaches or suggests all limitations of Claim 4. Liao fails to expressly disclose wherein the one or more actions comprises one or more of:
planning a path of the vehicle;
increasing a speed of the vehicle;
decreasing a speed of the vehicle;
stopping the vehicle; and
adjusting a trajectory of the vehicle.
See Claim 4 for teachings of Ferguson. Ferguson further discloses wherein the one or more actions comprises one or more of:
planning a path of the vehicle; (as per “the navigation system 168 may use the map information of data 134 to determine a route or path to the destination location that follows a set of connected rails of map information 200. The computing device 110 may then maneuver the vehicle autonomously (or in an autonomous driving mode) as described above along the route towards the destination. In order to do so, the vehicle's computing device 110 may create a plan identifying the locations, speeds and orientations of the vehicles along the route. Together, these locations, speeds and orientations define an expected future path of the vehicle.” in ¶50)
increasing a speed of the vehicle; (as per “The computing device 110 may use the positioning system 170 to determine the vehicle's location and perception system 172 to detect and respond to objects when needed to reach the location safely. In order to do so, computer 110 may cause the vehicle to accelerate (e.g., by increasing fuel or other energy provided to the engine by acceleration system 162), decelerate (e.g., by decreasing the fuel supplied to the engine, changing gears, and/or by applying brakes by deceleration system 160), change direction (e.g., by turning the front or rear wheels of vehicle 100 by steering system 164), and signal such changes (e.g., by lighting turn signals of signaling system 166)” in ¶38)
decreasing a speed of the vehicle; (as per “The computing device 110 may use the positioning system 170 to determine the vehicle's location and perception system 172 to detect and respond to objects when needed to reach the location safely. In order to do so, computer 110 may cause the vehicle to accelerate (e.g., by increasing fuel or other energy provided to the engine by acceleration system 162), decelerate (e.g., by decreasing the fuel supplied to the engine, changing gears, and/or by applying brakes by deceleration system 160), change direction (e.g., by turning the front or rear wheels of vehicle 100 by steering system 164), and signal such changes (e.g., by lighting turn signals of signaling system 166)” in ¶38)
stopping the vehicle; and
adjusting a trajectory of the vehicle. (as per “wherein when the classification is not drivable, maneuvering the vehicle includes altering the expected future path of the vehicle to avoid driving over the object.” in Claim 4)
In this way, Ferguson operates to determine drivability of an object by associating a confidence value with whether an object is safe for the vehicle to drive over or not and using the classification and associated confidence value “to make a determination as to whether it is safe or not for the vehicle to drive over the object in real time” (¶60). Like Liao and Prediger, Ferguson is concerned with vehicle systems.
It would have been obvious for one of ordinary skill in the art before the effective filing date to have modified the multi-sensor object detection fusion system by Liao and the drivable path determination of Prediger with the obstacle drivability confidence determination as taught by Ferguson to enable another standard means determining whether colliding with or driving over an obstacle would constitute safe driving (¶21, ¶60). Such modification also allows the system to differentiate between objects which the vehicle must drive around to avoid an accident versus objects which the vehicle can drive over, thereby avoiding unnecessary maneuvering or stopping for objects that are safe to drive over and improving overall safety (¶16).
As per Claim 7, the combination of Liao, Ferguson, and Prediger teaches or suggests all limitations of Claim 1. Liao further discloses wherein the performing the factor query comprises:
performing a color query on the image for each of the one or more obstacles; (as per “through the inverse matrix operation of the camera's internal and external parameter matrices, the position of this point in the LiDAR coordinate system may be calculated, and at the same time, the R, G, B channels of the corresponding location pixel values may be assigned to the point cloud data” in C8L10-20, as per “{Then, at 906, the system 10 interpolates RGB image information into the point cloud data, and augments point cloud data having a format of (x, y, z, i, r, g, b), thereby augmenting a channel value of the point cloud data” in C8L20-35)
Liao fails to expressly disclose:
performing a shape query on the image for each of the one or more obstacles; and
performing a movement query on the image for each of the one or more obstacles.
See Claim 4 for teachings of Ferguson. Ferguson further discloses
performing a shape query on the image for each of the one or more obstacles; (as per “objects such as vehicles, pedestrians and bicyclists may be readily identifiable from their visual characteristics (using image recognition techniques), physical characteristics (size, shape, etc.), speed (relative to the vehicle 100 or actual speed), and location (in a lane, in a crosswalk, on a sidewalk, etc.) captured by lasers or camera sensors of the perception system” in ¶36, as per “Other characteristics defined in the sensor information for an object may be used to identify and/or filter the set of objects relevant for classification. For example, the computing device 110 and/or the perception system 172 may filter for objects for which drivability is not clear based on geometry (size or shape).” in ¶56)
performing a movement query on the image for each of the one or more obstacles. (as per “the vehicle's perception system may use various sensors, such as LIDAR, sonar, radar, cameras, etc. to detect objects and their characteristics. These characteristics may include, for example, location, dimensions, direction of motion, velocity, shape, density, reflectivity, intensity, texture, etc” in ¶17, as per “For instance, objects such as vehicles, pedestrians and bicyclists may be readily identifiable from their visual characteristics (using image recognition techniques), physical characteristics (size, shape, etc.), speed (relative to the vehicle 100 or actual speed), and location (in a lane, in a crosswalk, on a sidewalk, etc.) captured by lasers or camera sensors of the perception system” in ¶36, as per ¶57)
In this way, Ferguson operates to determine drivability of an object by associating a confidence value with whether an object is safe for the vehicle to drive over or not and using the classification and associated confidence value “to make a determination as to whether it is safe or not for the vehicle to drive over the object in real time” (¶60). Like Liao and Prediger, Ferguson is concerned with vehicle systems.
It would have been obvious for one of ordinary skill in the art before the effective filing date to have modified the multi-sensor object detection fusion system by Liao and the drivable path determination of Prediger with the obstacle drivability confidence determination as taught by Ferguson to enable another standard means determining whether colliding with or driving over an obstacle would constitute safe driving (¶21, ¶60). Such modification also allows the system to differentiate between objects which the vehicle must drive around to avoid an accident versus objects which the vehicle can drive over, thereby avoiding unnecessary maneuvering or stopping for objects that are safe to drive over and improving overall safety (¶16).
Claims 8 and 15 are rejected using the same rationale, mutatis mutandis, applied to Claim 1 above, respectively.
Claims 9 and 16 are rejected using the same rationale, mutatis mutandis, applied to Claim 2 above, respectively.
Claims 10 and 17 are rejected using the same rationale, mutatis mutandis, applied to Claim 3 above, respectively.
Claim 11 is rejected using the same rationale, mutatis mutandis, applied to Claim 4 above, respectively.
Claim 12 is rejected using the same rationale, mutatis mutandis, applied to Claim 5 above, respectively.
Claims 14 and 20 are rejected using the same rationale, mutatis mutandis, applied to Claim 7 above, respectively.
As per Claim 18, the combination of Liao, Ferguson, and Prediger teaches or suggests all limitations of Claim 15. Liao fails to expressly disclose wherein the planning the trajectory comprises:
for each of the one or more obstacles, based on the second label of the obstacle, determining one or more vehicle actions for the vehicle to perform; and
causing the vehicle to perform the one or more actions, and the one or more actions comprises one or more of:
planning a path of the vehicle;
increasing a speed of the vehicle;
decreasing a speed of the vehicle; stopping the vehicle; and
adjusting a trajectory of the vehicle.
See Claim 15 for teachings of Ferguson. Ferguson further discloses wherein the planning the trajectory comprises:
for each of the one or more obstacles, based on the second label of the obstacle, determining one or more vehicle actions for the vehicle to perform; and causing the vehicle to perform the one or more actions, (as per “The classification may then be used to control the vehicle. For instance, as an example, if an object is identified as drivable, the vehicle may proceed to drive over the object. Alternatively, if an object is classified as not drivable, the vehicle may stop or maneuver around the object. As noted above, by classifying the drivability of objects, unnecessary maneuvering or stopping, for instance to avoid a plastic bag or other similar debris, can be avoided” in ¶22, as per “For instance, if object 686 is determined to be drivable, the computing device 110 may cause the vehicle to proceed to drive over the object. If object 686 is classified as drivable if straddled, the computing device 110 may cause the vehicle maneuver to drive over the object such that the object is positioned between the wheels (i.e. driver and passenger side wheels) of the vehicle. If object 686 is classified as not drivable, the computing device 110 may cause the vehicle to stop or maneuver around the object. In addition, if object 686 is classified as not drivable but likely to move away on its own, the computing device 110 may cause the vehicle to slow down as the vehicle approaches the object, in order to give the object a greater amount of time to move out of the expected future path of the vehicle before the vehicle reaches the object. Of course, if the object does not move out of the expected future path of the vehicle, the computing device 110 may cause the vehicle to come to a complete stop or maneuver the vehicle around the object” in ¶65)
and the one or more actions comprises one or more of:
planning a path of the vehicle; (as per “the navigation system 168 may use the map information of data 134 to determine a route or path to the destination location that follows a set of connected rails of map information 200. The computing device 110 may then maneuver the vehicle autonomously (or in an autonomous driving mode) as described above along the route towards the destination. In order to do so, the vehicle's computing device 110 may create a plan identifying the locations, speeds and orientations of the vehicles along the route. Together, these locations, speeds and orientations define an expected future path of the vehicle.” in ¶50)
increasing a speed of the vehicle; (as per “The computing device 110 may use the positioning system 170 to determine the vehicle's location and perception system 172 to detect and respond to objects when needed to reach the location safely. In order to do so, computer 110 may cause the vehicle to accelerate (e.g., by increasing fuel or other energy provided to the engine by acceleration system 162), decelerate (e.g., by decreasing the fuel supplied to the engine, changing gears, and/or by applying brakes by deceleration system 160), change direction (e.g., by turning the front or rear wheels of vehicle 100 by steering system 164), and signal such changes (e.g., by lighting turn signals of signaling system 166)” in ¶38)
decreasing a speed of the vehicle; (as per “The computing device 110 may use the positioning system 170 to determine the vehicle's location and perception system 172 to detect and respond to objects when needed to reach the location safely. In order to do so, computer 110 may cause the vehicle to accelerate (e.g., by increasing fuel or other energy provided to the engine by acceleration system 162), decelerate (e.g., by decreasing the fuel supplied to the engine, changing gears, and/or by applying brakes by deceleration system 160), change direction (e.g., by turning the front or rear wheels of vehicle 100 by steering system 164), and signal such changes (e.g., by lighting turn signals of signaling system 166)” in ¶38)
stopping the vehicle; and
adjusting a trajectory of the vehicle. (as per “wherein when the classification is not drivable, maneuvering the vehicle includes altering the expected future path of the vehicle to avoid driving over the object.” in Claim 4)
In this way, Ferguson operates to determine drivability of an object by associating a confidence value with whether an object is safe for the vehicle to drive over or not and using the classification and associated confidence value “to make a determination as to whether it is safe or not for the vehicle to drive over the object in real time” (¶60). Like Liao and Prediger, Ferguson is concerned with vehicle systems.
It would have been obvious for one of ordinary skill in the art before the effective filing date to have modified the multi-sensor object detection fusion system by Liao and the drivable path determination of Prediger with the obstacle drivability confidence determination as taught by Ferguson to enable another standard means determining whether colliding with or driving over an obstacle would constitute safe driving (¶21, ¶60). Such modification also allows the system to differentiate between objects which the vehicle must drive around to avoid an accident versus objects which the vehicle can drive over, thereby avoiding unnecessary maneuvering or stopping for objects that are safe to drive over and improving overall safety (¶16).
Claim(s) 6, 13, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Liao (US Pat. No. 11403860) in view of Ferguson (US Pub. No. 20180032078) in view of Prediger (US Pub. No. 20240190422) in further view of Singh (US Pub. No. 20220388535).
As per Claim 6, the combination of Liao, Ferguson, and Prediger teaches or suggests all limitations of Claim 1. Liao further discloses wherein each patch forms a bounding box on the image (“A 3D bounding box is determined for the point cloud data and a 2D bounding box is determined for the image data. The LiDAR and camera are jointly calibrated, and the 3D object bounding box of the point cloud data is mapped to the corresponding 2D object bounding box of the image data. The system performs decision-level fusion on the point cloud data and the image data followed by a pixel-level fusion of the 3D object bounding box and the 2D object bounding box” in Abstract) and performing the factor query comprises performing the factor query on the resized image. (as per “The output of the 2D target detection box includes a rectangular detection box. The output of each detection box includes a center position of the target X, Y, the width and height of the detection target, and the category information label of the detection target. In some embodiments, the LiDAR point cloud is processed through original data analysis, LiDAR data preprocessing and point cloud segmentation” in C4L35-45, as per “The point cloud data transformed into the pixel coordinate system determines the rectangular area according to its size in the u and v directions in the pixel coordinate system” in C4L50-60, as per “The target after clustering preferably includes a 3D point cloud group. The 3D point cloud points are then transformed into the camera coordinate system through the external parameter transformation matrix determined at 500. At this point, the transformation from 3D point cloud data to 2D pixel points in the camera coordinate system may be completed” in C4L40-55)
Liao, Ferguson, and Prediger fail to expressly disclose cropping the region of the image within the bounding box, forming a cropped image; and resizing the cropped image, forming a resized image.
Singh discloses of image annotation for deep neural networks, further comprising cropping the region of the image within the bounding box, forming a cropped image; (as per "An image 600 can be cropped by starting with an image 500 from FIG. 5 that includes a bounding box 508 and object 502, for example. All the image data outside of the bounding box 508 is deleted, leaving only image 600 including object 502. The cropped image 600 can be input to a DNN 200 to detect the object 502 included in the cropped image." in ¶45) and resizing the cropped image, forming a resized image, wherein performing the factor query comprises performing the factor query on the resized image. (as per "The cropped second image can be transformed to a higher spatial resolution by super resolution. The cropped second image can be transformed to include motion blurring. The cropped second image can be transformed by zooming the cropped second image. The cropped second image can be transformed to include hierarchical pyramid processing to obtain image data including the second object at a plurality of spatial resolutions." in ¶17, as per "At block 720 the cropped image 600 can be modified using super resolution, blurring, zooming, or hierarchical pyramid processing." in ¶59)
In this way, Singh operates to crop an image region within a bounding box and modify the cropped image using super resolution, blurring, zooming, or hierarchical pyramid processing (¶45, ¶59). Like Liao, Ferguson, and Prediger, Singh is concerned with processing image data using neural networks.
It would have been obvious for one of ordinary skill in the art before the effective filing date to have modified the multi-sensor object detection fusion system by Liao, the obstacle drivability confidence determination by Ferguson, and the drivable path determination by Prediger with the image cropping and resizing/modification technique as taught by Singh to enable another standard means of cropping the region of the image within the bounding box and resizing or otherwise modifying the cropped image for further image analysis (¶45, ¶59). Such modification also improves the operation of neural networks by providing modified cropped images, including images transformed using super resolution, blurring, zooming, or hierarchical pyramid processing, for improved object detection and/or training of deep neural networks (¶15, ¶17, ¶40, ¶59).
Claims 13 and 19 are rejected using the same rationale, mutatis mutandis, applied to Claim 6 above, respectively.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to TYLER R ROBARGE whose telephone number is (703)756-5872. The examiner can normally be reached Monday - Friday, 8:00 am - 5:00 pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Ramón Mercado can be reached at (571) 270-5744. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/T.R.R./Examiner, Art Unit 3658
/Ramon A. Mercado/Supervisory Patent Examiner, Art Unit 3658
Prosecution Insights