DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on April 7, 2026 has been entered.
Response to Amendment
The amendment filed April 7, 2026 has been entered. Claims 1, 3-5, 7, 9-11, 13, 15-20, and 22-26 remain pending in the application.
Response to Arguments
Applicant’s arguments, see Page 9, filed April 7, 2026, with respect to the rejection(s) of claims 1, 3-5, 7, 9-11, 13, 15-20, and 22 under 25 U.S.C. 103 have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of Tariq et al. (US 20190392242 A1) and Rodriguez et al. (US 20140347511 A1).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 7, 11, 20, and 25 are rejected under 35 U.S.C. 103 as being unpatentable over Tariq et al. (US 20190392242 A1) in view of Tran (US 10928830 B1) and Rodriguez et al. (US 20140347511 A1), hereinafter Tariq, Tran, and Rodriguez respectively.
Regarding claim 1, Tariq teaches a method (Paragraph 0022 – “The techniques discussed herein may include providing an image to an ML model and receiving, from the ML model, multiple regions of interest (ROIs) for different portions of an image. These ROIs may be any form of identifying what the ML model believes to be the existence of an object in the image”; Note: the techniques describe a method) comprising:
generating a bounding area around an object identified in a two-dimensional image captured by one or more sensors of a vehicle (Paragraph 0045, 0052, 0054 – “The autonomous vehicle 104 may receive sensor data from one or more sensors of the autonomous vehicle 104. The autonomous vehicle 104 may use this sensor data to determine a trajectory for controlling motion of the autonomous vehicle. The sensor data may include an image such as, for example, example image 100… the ML model may be configured to receive the image and output one or more ROIs… FIG. 2B illustrates example portions of the image (i.e., cells in the image) and their associated regions of interest (ROIs) with respect to a classification of “car.” FIG. 2B illustrates the ROIs as bounding boxes”; Note: the ROI is a bounding area, and the object is a car. Fig. 9A below shows that the image is 2D);
determining whether the object satisfies a confidence criterion (Paragraph 0053-0054 – “each output cell may correspond with a ROI indicated by a center position (e.g., a <u, v> image coordinate position), extents (e.g., a width and/or height), and/or a confidence level for any one or more classifications. As a non-limiting example, each cell may be associated with a center, extents, and confidence for each of a car, pedestrian, bicyclist, truck/bus, traffic light, and/or stop sign classification…the ML model may determine a first confidence score in association with ROI 204′…A confidence score may indicate a probability that the associated ROI accurately represents a region of the image that represents an object (here, a car)”; Note: it is determined whether the object in the ROI satisfies a confidence criterion that it matches a classification);
in response to the confidence criterion being satisfied: performing segmentation of the two-dimensional image based on the bounding area to differentiate between the object and a traversable space (Paragraph 0042, 0102, 0106 – “the techniques for instance segmentation (e.g., segmenting an image on a pixel by pixel basis) may include receiving, at a ML model, an image; and determining an ROI for a portion of the image (e.g., a pixel, a cluster of pixels) and a confidence score associated therewith, until the ML model has determined a plurality of ROIs and a plurality of confidence scores associated therewith for a plurality of portions of the image…To segment the region of the image that represents the object that the output ROI identifies, the techniques may include determining that a subset of the plurality of ROIs associated with confidence scores meet or exceed a confidence score threshold, substantially overlap with the ROI associated with the maximum confidence score, and/or are within a threshold confidence of the maximum confidence score; and concatenating the portions from which the ROIs were determined to an image segmentation… FIG. 9A illustrates an example ROI 900 determined by an ML model for a portion 902 of an example image 904… FIG. 9C illustrates two representations (910 & 912) of an example instance segmentation (i.e., a mask in the depicted example) identifying an object”; Note: when the confidence score of an ROI (bounding box) related to the object is above a threshold, segmentation occurs for the corresponding regions. Fig. 9A shows the object, bounded by a box, in the 2D image. Fig. 9C shows the segmentation of the object, which differentiates the object from the traversable space/road);
PNG
media_image1.png
344
410
media_image1.png
Greyscale
Screenshot of Fig. 9A (taken from Tariq)
PNG
media_image2.png
319
454
media_image2.png
Greyscale
Screenshot of Fig. 9C (taken from Tariq)
and if the confidence criterion is satisfied based on data, performing segmentation based on the data (Paragraph 0102, 0106 – “To segment the region of the image that represents the object that the output ROI identifies, the techniques may include determining that a subset of the plurality of ROIs associated with confidence scores meet or exceed a confidence score threshold, substantially overlap with the ROI associated with the maximum confidence score, and/or are within a threshold confidence of the maximum confidence score; and concatenating the portions from which the ROIs were determined to an image segmentation… FIG. 9A illustrates an example ROI 900 determined by an ML model for a portion 902 of an example image 904… FIG. 9C illustrates two representations (910 & 912) of an example instance segmentation (i.e., a mask in the depicted example) identifying an object”; Note: when the confidence score of an ROI (bounding box) related to the object is above a threshold, segmentation occurs for the corresponding regions).
Tariq does not teach performing semantic segmentation nor “generating a three-dimensional model of an environment comprised of the object and the traversable space based on the semantic segmentation, wherein the three-dimensional model is used for one or more of processing or transmitting instructions useable by one or more driver assistance features of the vehicle”. Tariq teaches instance segmentation instead of semantic segmentation. However, Tran teaches performing semantic segmentation (Col. 14 lines 20-22 and 31-34, Col. 18 lines 61-67, Col. 25 lines 30-45 – “The voxelized geometric map is produced by segmenting the point cloud into voxels…The semantic map layer builds on the geometric map layer by adding semantic objects such as traffic 2D and 3D objects, lane boundaries, intersections, crosswalks, parking spots, stop signs, traffic lights, etc. that are used for driving…The HD map represents portions of the lanes as lane elements. A lane element specifies the boundaries of the lane and various constraints including the legal direction in which a vehicle can travel within the lane element, the speed with which the vehicle can drive within the lane element, whether the lane element is for left turn only, or right turn only, and so on…A lane cut is generated where there is a topological change in the road network (e.g., an intersection, a split or a merge of lanes) or where there is a semantic change in the lane (e.g., a change in speed limit)…Lane elements, also referred to as cells or LaneEls, have left and right edges that are defined by lane lines or navigable boundaries”; Note: the geometric map layer and semantic map layer together are generated from a semantic segmentation process. An example is demonstrated by lane elements, which are defined semantically and geometrically). Tran also teaches generating a three-dimensional model of an environment comprised of the object and the traversable space based on the semantic segmentation (Col. 5 lines 2-3, Col. 14 lines 31-34, Col. 25 lines 30-45, Col. 26 lines 19-23, 64-67 - "The sensors can generate a 3D model of an environment. The 3D model can be a high definition map... The semantic map layer builds on the geometric map layer by adding semantic objects such as traffic 2D and 3D objects, lane boundaries, intersections, crosswalks, parking spots, stop signs, traffic lights, etc. that are used for driving…A lane cut is generated where there is a topological change in the road network (e.g., an intersection, a split or a merge of lanes) or where there is a semantic change in the lane (e.g., a change in speed limit). A lane cut goes through the width of the road, cutting it into adjacent segments. A lane cut ends at a navigable boundary. The lane lines and navigable boundaries may be generated from received image frames from an imaging system mounted on a vehicle. Lane elements, also referred to as cells or LaneEls, have left and right edges that are defined by lane lines or navigable boundaries. Lane elements have a bottom and a top edge defined by lane cut segments. Lane elements have 0 or 1 left and right neighbors and 0 or more predecessor and successor neighbors. Each lane elements can be associated with features that only affect local lane elements (e.g., stop sign, yield sign, or traffic light)…lane cut lines and navigable boundaries are generated from a plurality of received image frames from an imaging system mounted on a vehicle. Lane cuts are converted into lane cut segments across a single lane... A high definition map of the local area can then be generated including the lane element graph for use in driving by one or more autonomous vehicles."; Note: semantic segmentation is performed on image frames to generate lane cut segments and elements. The lane cuts and elements are then used to generate a high definition map, which is a 3D model. The map also includes objects, like stop signs, and the lane (traversable space)), wherein the three-dimensional model is used for one or more of processing or transmitting instructions useable by one or more driver assistance features of the vehicle (Col. 45 lines 21-44 – “After the adjusting, aggregating, by a processor, the plurality of 3D models to generate a comprehensive 3D model; combining the comprehensive 3D model with detailed map information; and using the combined comprehensive 3D model with detailed map information to maneuver the vehicle”). A person of ordinary skill in the art before the effective filing date of the claimed invention would have recognized that the instance segmentation of Tariq could have been substituted for the semantic segmentation of Tran because both the instance segmentation of Tariq and semantic segmentation of Tran serve the purpose of labeling and creating a mask for an identified object. Furthermore, a person of ordinary skill in the art would have been able to carry out the substitution. Finally, the substitution achieves the predictable result of identifying an object and differentiating it from a traversable space, and it would achieve the benefit of faster computation and simpler results. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to substitute the instance segmentation of Tariq for the semantic segmentation of Tran according to known methods to yield the predictable result of identifying an object and differentiating it from a traversable space. Moreover, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Tariq to incorporate the teachings of Tran to generate a 3D model of the object and traversable space based on the semantic segmentation because semantic segmentation provides precise identification of objects and their location, which enhances the 3D model for navigation and awareness. It also would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Tariq to incorporate the teachings of Tran to use the 3D model for driver assistance because the 3D model can provide a better view of road and indicate obstacles for the vehicle to avoid.
Tariq modified by Tran still does not teach in response to the confidence criterion being unsatisfied: accessing data related to the object from an additional sensor of the vehicle that is configured to sense an area corresponding to the two-dimensional image, wherein the additional sensor is separate from the one or more sensors; determining whether the confidence criterion for the object is satisfied based on the data. However, Rodriguez teaches in response to the confidence criterion being unsatisfied: accessing data related to the object from an additional sensor of the vehicle that is configured to sense an area corresponding to the two-dimensional image, wherein the additional sensor is separate from the one or more sensors (Fig. 5B, Paragraph 0051, 0053, 0055 – “if in 320 the confidence score is less than or, in some embodiments, equal to the threshold score, the computing device can continue to 330. In 330, the computing device can trigger an image re-capture and receive a subsequent image captured using an alternative mode…the subsequent image can be captured using an alternative camera, and the alternative camera may be a comparably costly camera compared to the camera used to capture the first image or images from previous iterations (e.g. a high-definition camera) or a camera that includes additional equipment (e.g. includes a flash bulb)… if an object was detected in the first image but a confidence score for a recognition task performed on the object was low, the computing device can transmit a command to the alternative camera to pan, tilt, and/or zoom to capture as the subsequent image a specific area within the wide field-of-view captured in the first image”; Note: when the confidence score is less than a threshold (unsatisfied), then a subsequent image is accessed. The subsequent image is data related to the object captured by an additional camera. The camera captures a 2D image in the same area as the previous image, as shown in screenshot of Fig. 5B below); determining whether the confidence criterion for the object is satisfied based on the data (Paragraph 0056, 0058 – “In 340, the computing device can process the subsequent image… In 350, the computing device can compare the confidence score from 340 to the threshold score. The threshold score in 340 can be the same threshold score as in 320 or can be a different threshold score. If the confidence score is greater than or, in some embodiments, equal to the threshold score in 350, the computing device can use the result of processing the first image for an intended task”; Note: it is determined whether or not the subsequent image satisfies the confidence score threshold/criterion).
PNG
media_image3.png
658
294
media_image3.png
Greyscale
Screenshot of Fig. 5B (taken from Rodriguez)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Tariq to incorporate the teachings of Rodriguez to obtain additional data from an additional sensor if the confidence criterion is not satisfied for the benefit of ensuring that the first sensor did not make a mistake in capturing the object. The additional data verifies whether the object truly exists in the location or not, and if it does, that it is properly captured. Proper capture of objects enhances safety for autonomous driving by assisting in environment awareness and avoiding collisions.
Regarding claim 7, Tariq in view of Tran and Rodriguez teaches the method of claim 1. Tariq further teaches wherein the bounding area is generated in response to identifying a predefined object in the two-dimensional image (Paragraph 0054 – “FIG. 2B illustrates example portions of the image (i.e., cells in the image) and their associated regions of interest (ROIs) with respect to a classification of “car.” FIG. 2B illustrates the ROIs as bounding boxes…An ML model, as discussed herein, may determine ROI 204′ for portion 204 (e.g., cell 204)…the ML model may determine a first confidence score in association with ROI 204′…A confidence score may indicate a probability that the associated ROI accurately represents a region of the image that represents an object (here, a car)…ROI 204′ may be considered a “detection,” by the ML model, of vehicle 106 in the image 100”; Note: an ROI (bounding box) is generated due to a detection of a car (predefined object) in the image).
Regarding claim 11, Tariq teaches a system (Paragraph 0115 – “vehicle system 1002”) comprising:
a monocular camera of a vehicle (Paragraph 0115 – “The example vehicle system 1002 may include sensor(s) 1012…The sensor(s) 1012 may include, for example, one or more LIDAR sensors, one or more cameras (e.g., RGB-cameras”); Note: the RGB camera is a monocular camera);
processing circuitry, communicatively coupled to the monocular camera (Paragraph 0115, 0117 – “The example vehicle system 1002 may include sensor(s) 1012…The sensor(s) 1012 may include, for example, one or more LIDAR sensors, one or more cameras (e.g., RGB-cameras…The perception engine 1026 may include instructions stored on memory 1006 that, when executed by the processor(s) 1004, configure the processor(s) 1004 to receive sensor data from the sensor(s) 1012 as input”; Note: the processor is coupled to the sensors (monocular camera) since it receives sensor data from the sensors), configured to:
generate a bounding area around an object identified in a two-dimensional image captured by the monocular camera (Paragraph 0045, 0052, 0054, 0115 – “The autonomous vehicle 104 may receive sensor data from one or more sensors of the autonomous vehicle 104. The autonomous vehicle 104 may use this sensor data to determine a trajectory for controlling motion of the autonomous vehicle. The sensor data may include an image such as, for example, example image 100… the ML model may be configured to receive the image and output one or more ROIs… FIG. 2B illustrates example portions of the image (i.e., cells in the image) and their associated regions of interest (ROIs) with respect to a classification of “car.” FIG. 2B illustrates the ROIs as bounding boxes… The sensor(s) 1012 may include, for example, one or more LIDAR sensors, one or more cameras (e.g., RGB-cameras”; Note: the ROI is a bounding area, and the object is a car. Fig. 9A above shows that the image is 2D);
determine whether the object satisfies a confidence criterion (Paragraph 0053-0054 – “each output cell may correspond with a ROI indicated by a center position (e.g., a <u, v> image coordinate position), extents (e.g., a width and/or height), and/or a confidence level for any one or more classifications. As a non-limiting example, each cell may be associated with a center, extents, and confidence for each of a car, pedestrian, bicyclist, truck/bus, traffic light, and/or stop sign classification…the ML model may determine a first confidence score in association with ROI 204′…A confidence score may indicate a probability that the associated ROI accurately represents a region of the image that represents an object (here, a car)”; Note: it is determined whether the object in the ROI satisfies a confidence criterion that it matches a classification);
in response to the confidence criterion being satisfied: perform segmentation of the two-dimensional image based on the bounding area to differentiate between the object and a traversable space (Paragraph 0042, 0102, 0106 – “the techniques for instance segmentation (e.g., segmenting an image on a pixel by pixel basis) may include receiving, at a ML model, an image; and determining an ROI for a portion of the image (e.g., a pixel, a cluster of pixels) and a confidence score associated therewith, until the ML model has determined a plurality of ROIs and a plurality of confidence scores associated therewith for a plurality of portions of the image…To segment the region of the image that represents the object that the output ROI identifies, the techniques may include determining that a subset of the plurality of ROIs associated with confidence scores meet or exceed a confidence score threshold, substantially overlap with the ROI associated with the maximum confidence score, and/or are within a threshold confidence of the maximum confidence score; and concatenating the portions from which the ROIs were determined to an image segmentation… FIG. 9A illustrates an example ROI 900 determined by an ML model for a portion 902 of an example image 904… FIG. 9C illustrates two representations (910 & 912) of an example instance segmentation (i.e., a mask in the depicted example) identifying an object”; Note: when the confidence score of an ROI (bounding box) related to the object is above a threshold, segmentation occurs for the corresponding regions. Fig. 9A shows the object, bounded by a box, in the 2D image. Fig. 9C shows the segmentation of the object, which differentiates the object from the traversable space/road);
and if the confidence criterion is satisfied based on data, perform segmentation based on the data (Paragraph 0102, 0106 – “To segment the region of the image that represents the object that the output ROI identifies, the techniques may include determining that a subset of the plurality of ROIs associated with confidence scores meet or exceed a confidence score threshold, substantially overlap with the ROI associated with the maximum confidence score, and/or are within a threshold confidence of the maximum confidence score; and concatenating the portions from which the ROIs were determined to an image segmentation… FIG. 9A illustrates an example ROI 900 determined by an ML model for a portion 902 of an example image 904… FIG. 9C illustrates two representations (910 & 912) of an example instance segmentation (i.e., a mask in the depicted example) identifying an object”; Note: when the confidence score of an ROI (bounding box) related to the object is above a threshold, segmentation occurs for the corresponding regions).
Tariq does not teach performing semantic segmentation nor “generating a three-dimensional model of an environment comprised of the object and the traversable space based on the semantic segmentation, wherein the three-dimensional model is used for one or more of processing or transmitting instructions useable by one or more driver assistance features of the vehicle”. Tariq teaches instance segmentation instead of semantic segmentation. However, Tran teaches performing semantic segmentation (Col. 14 lines 20-22 and 31-34, Col. 18 lines 61-67, Col. 25 lines 30-45 – “The voxelized geometric map is produced by segmenting the point cloud into voxels…The semantic map layer builds on the geometric map layer by adding semantic objects such as traffic 2D and 3D objects, lane boundaries, intersections, crosswalks, parking spots, stop signs, traffic lights, etc. that are used for driving…The HD map represents portions of the lanes as lane elements. A lane element specifies the boundaries of the lane and various constraints including the legal direction in which a vehicle can travel within the lane element, the speed with which the vehicle can drive within the lane element, whether the lane element is for left turn only, or right turn only, and so on…A lane cut is generated where there is a topological change in the road network (e.g., an intersection, a split or a merge of lanes) or where there is a semantic change in the lane (e.g., a change in speed limit)…Lane elements, also referred to as cells or LaneEls, have left and right edges that are defined by lane lines or navigable boundaries”; Note: the geometric map layer and semantic map layer together are generated from a semantic segmentation process. An example is demonstrated by lane elements, which are defined semantically and geometrically). Tran also teaches generating a three-dimensional model of an environment comprised of the object and the traversable space based on the semantic segmentation (Col. 5 lines 2-3, Col. 14 lines 31-34, Col. 25 lines 30-45, Col. 26 lines 19-23, 64-67 - "The sensors can generate a 3D model of an environment. The 3D model can be a high definition map... The semantic map layer builds on the geometric map layer by adding semantic objects such as traffic 2D and 3D objects, lane boundaries, intersections, crosswalks, parking spots, stop signs, traffic lights, etc. that are used for driving…A lane cut is generated where there is a topological change in the road network (e.g., an intersection, a split or a merge of lanes) or where there is a semantic change in the lane (e.g., a change in speed limit). A lane cut goes through the width of the road, cutting it into adjacent segments. A lane cut ends at a navigable boundary. The lane lines and navigable boundaries may be generated from received image frames from an imaging system mounted on a vehicle. Lane elements, also referred to as cells or LaneEls, have left and right edges that are defined by lane lines or navigable boundaries. Lane elements have a bottom and a top edge defined by lane cut segments. Lane elements have 0 or 1 left and right neighbors and 0 or more predecessor and successor neighbors. Each lane elements can be associated with features that only affect local lane elements (e.g., stop sign, yield sign, or traffic light)…lane cut lines and navigable boundaries are generated from a plurality of received image frames from an imaging system mounted on a vehicle. Lane cuts are converted into lane cut segments across a single lane... A high definition map of the local area can then be generated including the lane element graph for use in driving by one or more autonomous vehicles."; Note: semantic segmentation is performed on image frames to generate lane cut segments and elements. The lane cuts and elements are then used to generate a high definition map, which is a 3D model. The map also includes objects, like stop signs, and the lane (traversable space)), wherein the three-dimensional model is used for one or more of processing or transmitting instructions useable by one or more driver assistance features of the vehicle (Col. 45 lines 21-44 – “After the adjusting, aggregating, by a processor, the plurality of 3D models to generate a comprehensive 3D model; combining the comprehensive 3D model with detailed map information; and using the combined comprehensive 3D model with detailed map information to maneuver the vehicle”). A person of ordinary skill in the art before the effective filing date of the claimed invention would have recognized that the instance segmentation of Tariq could have been substituted for the semantic segmentation of Tran because both the instance segmentation of Tariq and semantic segmentation of Tran serve the purpose of labeling and creating a mask for an identified object. Furthermore, a person of ordinary skill in the art would have been able to carry out the substitution. Finally, the substitution achieves the predictable result of identifying an object and differentiating it from a traversable space, and it would achieve the benefit of faster computation and simpler results. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to substitute the instance segmentation of Tariq for the semantic segmentation of Tran according to known methods to yield the predictable result of identifying an object and differentiating it from a traversable space. Moreover, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Tariq to incorporate the teachings of Tran to generate a 3D model of the object and traversable space based on the semantic segmentation because semantic segmentation provides precise identification of objects and their location, which enhances the 3D model for navigation and awareness. It also would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Tariq to incorporate the teachings of Tran to use the 3D model for driver assistance because the 3D model can provide a better view of road and indicate obstacles for the vehicle to avoid.
Tariq modified by Tran still does not teach in response to the confidence criterion being unsatisfied: accessing data related to the object from an additional sensor of the vehicle that is configured to sense an area corresponding to the two-dimensional image, wherein the additional sensor is separate from the monocular camera; determining whether the confidence criterion for the object is satisfied based on the data. However, Rodriguez teaches in response to the confidence criterion being unsatisfied: accessing data related to the object from an additional sensor of the vehicle that is configured to sense an area corresponding to the two-dimensional image, wherein the additional sensor is separate from the monocular camera (Fig. 5B, Paragraph 0051, 0053, 0055 – “if in 320 the confidence score is less than or, in some embodiments, equal to the threshold score, the computing device can continue to 330. In 330, the computing device can trigger an image re-capture and receive a subsequent image captured using an alternative mode…the subsequent image can be captured using an alternative camera, and the alternative camera may be a comparably costly camera compared to the camera used to capture the first image or images from previous iterations (e.g. a high-definition camera) or a camera that includes additional equipment (e.g. includes a flash bulb)… if an object was detected in the first image but a confidence score for a recognition task performed on the object was low, the computing device can transmit a command to the alternative camera to pan, tilt, and/or zoom to capture as the subsequent image a specific area within the wide field-of-view captured in the first image”; Note: when the confidence score is less than a threshold (unsatisfied), then a subsequent image is accessed. The subsequent image is data related to the object captured by an additional camera, separate from a previous camera. The camera captures a 2D image in the same area as the previous image, as shown in screenshot of Fig. 5B above); determining whether the confidence criterion for the object is satisfied based on the data (Paragraph 0056, 0058 – “In 340, the computing device can process the subsequent image… In 350, the computing device can compare the confidence score from 340 to the threshold score. The threshold score in 340 can be the same threshold score as in 320 or can be a different threshold score. If the confidence score is greater than or, in some embodiments, equal to the threshold score in 350, the computing device can use the result of processing the first image for an intended task”; Note: it is determined whether or not the subsequent image satisfies the confidence score threshold/criterion). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Tariq to incorporate the teachings of Rodriguez to obtain additional data from an additional sensor if the confidence criterion is not satisfied for the benefit of ensuring that the first sensor did not make a mistake in capturing the object. The additional data verifies whether the object truly exists in the location or not, and if it does, that it is properly captured. Proper capture of objects enhances safety for autonomous driving by assisting in environment awareness and avoiding collisions.
Regarding claim 20, Tariq teaches a non-transitory computer readable medium comprising computer readable instructions which, when processed by processing circuitry (Paragraph 0110-0111, 0155 – “The processor(s) 1004 may be any suitable processor capable of executing instructions…The example vehicle system 1002 may include memory 1006. In some instances, the memory 1006 may include a non-transitory computer readable media configured to store executable instructions/modules, data, and/or data items accessible by the processor(s)… A non-transitory computer-readable medium having a set of instructions that, when executed, cause one or more processors to perform operations”), cause the processing circuitry to:
generate a bounding area around an object identified in a two-dimensional image captured by one or more sensors of a vehicle (Paragraph 0045, 0052, 0054 – “The autonomous vehicle 104 may receive sensor data from one or more sensors of the autonomous vehicle 104. The autonomous vehicle 104 may use this sensor data to determine a trajectory for controlling motion of the autonomous vehicle. The sensor data may include an image such as, for example, example image 100… the ML model may be configured to receive the image and output one or more ROIs… FIG. 2B illustrates example portions of the image (i.e., cells in the image) and their associated regions of interest (ROIs) with respect to a classification of “car.” FIG. 2B illustrates the ROIs as bounding boxes”; Note: the ROI is a bounding area, and the object is a car. Fig. 9A above shows that the image is 2D);
determine whether the object satisfies a confidence criterion (Paragraph 0053-0054 – “each output cell may correspond with a ROI indicated by a center position (e.g., a <u, v> image coordinate position), extents (e.g., a width and/or height), and/or a confidence level for any one or more classifications. As a non-limiting example, each cell may be associated with a center, extents, and confidence for each of a car, pedestrian, bicyclist, truck/bus, traffic light, and/or stop sign classification…the ML model may determine a first confidence score in association with ROI 204′…A confidence score may indicate a probability that the associated ROI accurately represents a region of the image that represents an object (here, a car)”; Note: it is determined whether the object in the ROI satisfies a confidence criterion that it matches a classification);
in response to the confidence criterion being satisfied: perform segmentation of the two-dimensional image based on the bounding area to differentiate between the object and a traversable space (Paragraph 0042, 0102, 0106 – “the techniques for instance segmentation (e.g., segmenting an image on a pixel by pixel basis) may include receiving, at a ML model, an image; and determining an ROI for a portion of the image (e.g., a pixel, a cluster of pixels) and a confidence score associated therewith, until the ML model has determined a plurality of ROIs and a plurality of confidence scores associated therewith for a plurality of portions of the image…To segment the region of the image that represents the object that the output ROI identifies, the techniques may include determining that a subset of the plurality of ROIs associated with confidence scores meet or exceed a confidence score threshold, substantially overlap with the ROI associated with the maximum confidence score, and/or are within a threshold confidence of the maximum confidence score; and concatenating the portions from which the ROIs were determined to an image segmentation… FIG. 9A illustrates an example ROI 900 determined by an ML model for a portion 902 of an example image 904… FIG. 9C illustrates two representations (910 & 912) of an example instance segmentation (i.e., a mask in the depicted example) identifying an object”; Note: when the confidence score of an ROI (bounding box) related to the object is above a threshold, segmentation occurs for the corresponding regions. Fig. 9A shows the object, bounded by a box, in the 2D image. Fig. 9C shows the segmentation of the object, which differentiates the object from the traversable space/road);
and if the confidence criterion is satisfied based on data, perform segmentation based on the data (Paragraph 0102, 0106 – “To segment the region of the image that represents the object that the output ROI identifies, the techniques may include determining that a subset of the plurality of ROIs associated with confidence scores meet or exceed a confidence score threshold, substantially overlap with the ROI associated with the maximum confidence score, and/or are within a threshold confidence of the maximum confidence score; and concatenating the portions from which the ROIs were determined to an image segmentation… FIG. 9A illustrates an example ROI 900 determined by an ML model for a portion 902 of an example image 904… FIG. 9C illustrates two representations (910 & 912) of an example instance segmentation (i.e., a mask in the depicted example) identifying an object”; Note: when the confidence score of an ROI (bounding box) related to the object is above a threshold, segmentation occurs for the corresponding regions).
Tariq does not teach performing semantic segmentation nor “generating a three-dimensional model of an environment comprised of the object and the traversable space based on the semantic segmentation, wherein the three-dimensional model is used for one or more of processing or transmitting instructions useable by one or more driver assistance features of the vehicle”. Tariq teaches instance segmentation instead of semantic segmentation. However, Tran teaches performing semantic segmentation (Col. 14 lines 20-22 and 31-34, Col. 18 lines 61-67, Col. 25 lines 30-45 – “The voxelized geometric map is produced by segmenting the point cloud into voxels…The semantic map layer builds on the geometric map layer by adding semantic objects such as traffic 2D and 3D objects, lane boundaries, intersections, crosswalks, parking spots, stop signs, traffic lights, etc. that are used for driving…The HD map represents portions of the lanes as lane elements. A lane element specifies the boundaries of the lane and various constraints including the legal direction in which a vehicle can travel within the lane element, the speed with which the vehicle can drive within the lane element, whether the lane element is for left turn only, or right turn only, and so on…A lane cut is generated where there is a topological change in the road network (e.g., an intersection, a split or a merge of lanes) or where there is a semantic change in the lane (e.g., a change in speed limit)…Lane elements, also referred to as cells or LaneEls, have left and right edges that are defined by lane lines or navigable boundaries”; Note: the geometric map layer and semantic map layer together are generated from a semantic segmentation process. An example is demonstrated by lane elements, which are defined semantically and geometrically). Tran also teaches generating a three-dimensional model of an environment comprised of the object and the traversable space based on the semantic segmentation (Col. 5 lines 2-3, Col. 14 lines 31-34, Col. 25 lines 30-45, Col. 26 lines 19-23, 64-67 - "The sensors can generate a 3D model of an environment. The 3D model can be a high definition map... The semantic map layer builds on the geometric map layer by adding semantic objects such as traffic 2D and 3D objects, lane boundaries, intersections, crosswalks, parking spots, stop signs, traffic lights, etc. that are used for driving…A lane cut is generated where there is a topological change in the road network (e.g., an intersection, a split or a merge of lanes) or where there is a semantic change in the lane (e.g., a change in speed limit). A lane cut goes through the width of the road, cutting it into adjacent segments. A lane cut ends at a navigable boundary. The lane lines and navigable boundaries may be generated from received image frames from an imaging system mounted on a vehicle. Lane elements, also referred to as cells or LaneEls, have left and right edges that are defined by lane lines or navigable boundaries. Lane elements have a bottom and a top edge defined by lane cut segments. Lane elements have 0 or 1 left and right neighbors and 0 or more predecessor and successor neighbors. Each lane elements can be associated with features that only affect local lane elements (e.g., stop sign, yield sign, or traffic light)…lane cut lines and navigable boundaries are generated from a plurality of received image frames from an imaging system mounted on a vehicle. Lane cuts are converted into lane cut segments across a single lane... A high definition map of the local area can then be generated including the lane element graph for use in driving by one or more autonomous vehicles."; Note: semantic segmentation is performed on image frames to generate lane cut segments and elements. The lane cuts and elements are then used to generate a high definition map, which is a 3D model. The map also includes objects, like stop signs, and the lane (traversable space)), wherein the three-dimensional model is used for one or more of processing or transmitting instructions useable by one or more driver assistance features of the vehicle (Col. 45 lines 21-44 – “After the adjusting, aggregating, by a processor, the plurality of 3D models to generate a comprehensive 3D model; combining the comprehensive 3D model with detailed map information; and using the combined comprehensive 3D model with detailed map information to maneuver the vehicle”). A person of ordinary skill in the art before the effective filing date of the claimed invention would have recognized that the instance segmentation of Tariq could have been substituted for the semantic segmentation of Tran because both the instance segmentation of Tariq and semantic segmentation of Tran serve the purpose of labeling and creating a mask for an identified object. Furthermore, a person of ordinary skill in the art would have been able to carry out the substitution. Finally, the substitution achieves the predictable result of identifying an object and differentiating it from a traversable space, and it would achieve the benefit of faster computation and simpler results. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to substitute the instance segmentation of Tariq for the semantic segmentation of Tran according to known methods to yield the predictable result of identifying an object and differentiating it from a traversable space. Moreover, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Tariq to incorporate the teachings of Tran to generate a 3D model of the object and traversable space based on the semantic segmentation because semantic segmentation provides precise identification of objects and their location, which enhances the 3D model for navigation and awareness. It also would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Tariq to incorporate the teachings of Tran to use the 3D model for driver assistance because the 3D model can provide a better view of road and indicate obstacles for the vehicle to avoid.
Tariq modified by Tran still does not teach in response to the confidence criterion being unsatisfied: accessing data related to the object from an additional sensor of the vehicle that is configured to sense an area corresponding to the two-dimensional image, wherein the additional sensor is separate from the one or more sensors; determining whether the confidence criterion for the object is satisfied based on the data. However, Rodriguez teaches in response to the confidence criterion being unsatisfied: accessing data related to the object from an additional sensor of the vehicle that is configured to sense an area corresponding to the two-dimensional image, wherein the additional sensor is separate from the one or more sensors (Fig. 5B, Paragraph 0051, 0053, 0055 – “if in 320 the confidence score is less than or, in some embodiments, equal to the threshold score, the computing device can continue to 330. In 330, the computing device can trigger an image re-capture and receive a subsequent image captured using an alternative mode…the subsequent image can be captured using an alternative camera, and the alternative camera may be a comparably costly camera compared to the camera used to capture the first image or images from previous iterations (e.g. a high-definition camera) or a camera that includes additional equipment (e.g. includes a flash bulb)… if an object was detected in the first image but a confidence score for a recognition task performed on the object was low, the computing device can transmit a command to the alternative camera to pan, tilt, and/or zoom to capture as the subsequent image a specific area within the wide field-of-view captured in the first image”; Note: when the confidence score is less than a threshold (unsatisfied), then a subsequent image is accessed. The subsequent image is data related to the object captured by an additional camera. The camera captures a 2D image in the same area as the previous image, as shown in screenshot of Fig. 5B above); determining whether the confidence criterion for the object is satisfied based on the data (Paragraph 0056, 0058 – “In 340, the computing device can process the subsequent image… In 350, the computing device can compare the confidence score from 340 to the threshold score. The threshold score in 340 can be the same threshold score as in 320 or can be a different threshold score. If the confidence score is greater than or, in some embodiments, equal to the threshold score in 350, the computing device can use the result of processing the first image for an intended task”; Note: it is determined whether or not the subsequent image satisfies the confidence score threshold/criterion). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Tariq to incorporate the teachings of Rodriguez to obtain additional data from an additional sensor if the confidence criterion is not satisfied for the benefit of ensuring that the first sensor did not make a mistake in capturing the object. The additional data verifies whether the object truly exists in the location or not, and if it does, that it is properly captured. Proper capture of objects enhances safety for autonomous driving by assisting in environment awareness and avoiding collisions.
Regarding claim 25, Tariq in view of Tran and Rodriguez teaches the method of claim 1. Tariq further teaches wherein the two-dimensional image comprises a plurality of pixels, further comprising: processing the two-dimensional image to generate a plurality of block groups having at least one related pixel value (Fig. 2B, Paragraph 0051, 0053, 0066 – “FIG. 2A illustrates example image 100 and an example output grid 200, where each cell of the output grid 200 identifies a portion of the image 100…a “portion of the image” may include a single pixel of the image and/or a collection of pixels of the image…each output cell may correspond with a ROI indicated by a center position (e.g., a <u, v> image coordinate position), extents (e.g., a width and/or height), and/or a confidence level for any one or more classifications. As a non-limiting example, each cell may be associated with a center, extents, and confidence for each of a car, pedestrian, bicyclist, truck/bus, traffic light, and/or stop sign classification… the ML model may determine a cluster of ROIs that overlap (e.g., that have degrees of alignment with respect to each other that meet or exceed a threshold degree of alignment), thereby indicating a likelihood that an object is represented in the image somewhere in the image around that cluster of ROIs”; Note: the image, which is 2D as shown in Fig. 2B below, is processed to be a grid of cells of pixels. The pixels are grouped as ROIs, which overlap so that they share some related pixels); generating a pixel group stack based on the plurality of block groups and on respective locations of the plurality of pixels within the two-dimensional image (Fig. 6A, Paragraph 0066 – “the ML model may determine a cluster of ROIs that overlap (e.g., that have degrees of alignment with respect to each other that meet or exceed a threshold degree of alignment), thereby indicating a likelihood that an object is represented in the image somewhere in the image around that cluster of ROIs”; Note: the cluster of overlapping ROIs is equivalent to the pixel group stack. The ROIs are equivalent to block groups. The ROIs overlap, meaning the cluster is dependent on the locations of the pixels in the ROIs. 612 in Fig. 6A below shows an example of a pixel group stack); providing the pixel group stack to a first processing unit to perform the semantic segmentation (Paragraph 0066, 0102, 0104-0105 – “the ML model may determine a cluster of ROIs that overlap (e.g., that have degrees of alignment with respect to each other that meet or exceed a threshold degree of alignment), thereby indicating a likelihood that an object is represented in the image somewhere in the image around that cluster of ROIs, and may NMS the cluster of ROIs to determine an output ROI for the object…unlike the models trained in the above examples, the ML model illustrated in FIG. 9A may be trained, instead of using ROIs associated with a central 30%, but those ROIs associated with pixels in a mask of an object. Specifically, an image mask and a corresponding ROI for an object in an image may be used as ground truth when training the ML model. Such a training region of interest may be determined, for example, based on an associated ROI with the mask (e.g., a bounding box based at least in part on the pixels of the mask)… the ML model may determine a single ROI to be associated with the object (e.g., object 110) out of the plurality of ROIs determined. In at least one example, such a detection may be accomplished using NMS… The ML model may substantially simultaneously determine an ROI to associate with vehicle 110 and a mask that identifies particular portions of the image that are associated with the object (e.g., pixels that represent the object) based at least in part on the determination of the ROI for output. In at least one instance, such an ML model may retain an indication of the pixels and/or regions which were suppressed during the NMS. All such pixels and/or portions of the image may be associated with a mask (instance segmentation) of the object”; Note: The ML model generates a mask (performs segmentation) and uses NMS to do so, which involves the use of the cluster of overlapping ROIs. The cluster of overlapping ROIs is equivalent to the pixel group stack. The processing unit that runs the ML model that generates the mask is equivalent to the first processing unit); and providing the pixel group stack to a second processing unit to generate the bounding area (Paragraph 0066 – “the ML model may determine a cluster of ROIs that overlap (e.g., that have degrees of alignment with respect to each other that meet or exceed a threshold degree of alignment), thereby indicating a likelihood that an object is represented in the image somewhere in the image around that cluster of ROIs, and may NMS the cluster of ROIs to determine an output ROI for the object. In other words, outputting an ROI in association with a single object may be based on determining an ROI associated with a maximum confidence score, of the multiple ROIs that overlap. The ML model may repeat identifying clusters and performing NMS with respect to the clusters until all objects have been identified and subjected to NMS. This may be done for each object classification for which an ROI is detected”; Note: the cluster of overlapping ROIs is equivalent to the pixel group stack, and it is provided to an ML model. The ML model generates an output ROI, which is equivalent to the bounding area, and the processing unit that runs the ML model that generates the output ROI is equivalent to the second processing unit).
PNG
media_image4.png
354
490
media_image4.png
Greyscale
Screenshot of Fig. 2B (taken from Tariq)
PNG
media_image5.png
175
249
media_image5.png
Greyscale
Screenshot of Fig. 6A (taken from Tariq)
Claims 3 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Tariq in view of Tran, Rodriguez, Zheng et al. (Exploring OpenStreetMap Capability for Road Perception), and Dodson et al. (US 20210149408 A1), hereinafter Zheng and Dodson respectively.
Regarding claim 3, Tariq in view of Tran and Rodriguez teaches the method of claim 1. Tariq further teaches using a vehicle display (Paragraph 0113 – “the input/output (“I/O”) interface 1008 may be configured to coordinate I/O traffic between the processor(s) 1004, the memory 1006, the network interface 1010, sensor(s) 1012, I/O devices 1014, drive system 1016, and/or any other hardware of the vehicle system 1002. In some instances, the I/O devices 1014 may include external and/or internal speaker(s), display(s)”). Tariq does not teach generating for display, a processed image by modifying the two- dimensional image to differentiate between the object and the traversable space by incorporating in the processed image one or more of a change in a color of pixels comprising one or more of the object or the traversable space or a label corresponding to a predefined classification of pixels comprising one or more of the object or the traversable space; and assigning values to pixels corresponding to the object in the processed image, wherein the values correspond to one or more of a heading, a depth within a three-dimensional space, or a regression value. However, Zheng teaches generating for display, a processed image by modifying the two- dimensional image to differentiate between the object and the traversable space (Paragraph 4 in 1st Col. of Page 2, Paragraph 1 in 2nd Col. of Page 2 – “The OSM2World toolkit [18] can be used to create 3D models from the OSM XML data. It renders a 3D virtual world from the bird-eye view, and generates a shape object (e.g. road, building, etc.) description file. When the vehicle’s GPS coordinates and its heading direction is given, we can set it as a viewpoint and project the 3D virtual world into the driver’s perspective”; Note: the projected 3D virtual world is equivalent to the processed image based on a 3D model, and it is rendered, implying that it is generated for display) by incorporating in the processed image one or more of a change in a color of pixels comprising one or more of the object or the traversable space or a label corresponding to a predefined classification of pixels comprising one or more of the object or the traversable space (Fig. 3, Paragraph 3 in 2nd Col. of Page 2 – “One label refinement approach is to look the image in super-pixels [5]. The processing steps are shown in Fig. 4. The super-pixel segmentation is based on the real camera image, using the K-means clustering method. Specific, an image size is 375*1242 pixels, and we select K=800 to generate 800 super-pixel segments. Initial pixel-wise labels from the OSM road mask are overlaid onto the real image and assigned to all the super-pixel segments. If a segment contains more than 50% initial road labels, it should be relabeled as a road super-pixel, and non-road otherwise”; Note: Fig. 3 shows how the image is processed to have a change of color of pixels for the road (traversable area); see screenshot of Fig. 3 below. The road has a different color from the objects and the rest of the scene. The different colors correspond to the labeling/classification of pixels). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Tariq to incorporate the teachings of Zheng to display a processed image with a change in color for the traversable space based on a classification for the benefit of enhancing the user’s environment awareness. For example, having a different color for the road and other objects would help the user see the road clearer, remain in their lane, and avoid collisions.
PNG
media_image6.png
414
859
media_image6.png
Greyscale
Screenshot of Fig. 3 (taken from Zheng)
Tariq modified by Zheng still does not teach assigning values to pixels corresponding to the object in the processed image, wherein the values correspond to one or more of a heading, a depth within a three-dimensional space, or a regression value. However, Dodson teaches assigning values to pixels corresponding to the object in the processed image, wherein the values correspond to a depth within a three-dimensional space (Paragraph 0043 – “the on-board neural network subsystem 134 can predict the depth value output of a non-static object from the depth map 165 generated from camera images. For example based on the depth value 165 of the road areas generated from camera images, the on-board neural network subsystem 134 can assign, to a car in the environment, the depth value for a portion of the road that is next to the car. Even though the car is outside the range measurable from lasers and radars, the depth value of the car can still be estimated based on the camera image”; Note: depth values are assigned to pixels corresponding to the car/object). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Tariq to incorporate the teachings of Dodson to assign depth values to the pixels of the object because “When a planning subsystem 136 receives the depth map 165, the planning subsystem 136 can use the depth map 165 to make fully-autonomous or semi-autonomous driving decisions. For example, the planning subsystem 136 can generate a fully-autonomous plan to navigate on a highway or other road by querying the depth map 165 to identify distances to static surfaces in the vicinity of the car and to identify areas where there are occlusions, i.e., areas where depth information is not present in the depth map 165. By identifying occlusions through the depth map, during a turn operation, the vehicle can perform a necessary yield operation to a potential object which cannot be seen because the object is occluded by a building, a car, or a tree, etc” (Dodson: Paragraph 0044). In other words, the depth values help differentiate and locate the road and the objects, which then help avoid collisions.
Regarding claim 13, Tariq in view of Tran and Rodriguez teaches the system of claim 11. Tariq further teaches a display (Paragraph 0113 – “the input/output (“I/O”) interface 1008 may be configured to coordinate I/O traffic between the processor(s) 1004, the memory 1006, the network interface 1010, sensor(s) 1012, I/O devices 1014, drive system 1016, and/or any other hardware of the vehicle system 1002. In some instances, the I/O devices 1014 may include external and/or internal speaker(s), display(s)”). Tariq does not teach modifying the two-dimensional image to generate a processed image that visually differentiates between the object and the traversable space by incorporating one or more of a change in a color of pixels corresponding to one or more of the object or the traversable space or a label corresponding to a predefined classification of pixels comprising one or more of the object or the traversable space; and assigning values to pixels corresponding to the object, wherein the values correspond to one or more of a heading, a depth within a three-dimensional space, or a regression value. However, Zheng teaches modifying the two-dimensional image to generate a processed image that visually differentiates between the object and the traversable space (Fig. 6, Paragraph 1 in 2nd Col. of Page 3 – “Fig. 6 shows an example how the GrabCut algorithm is utilized for the road detection. In our case, the top-bar is pre-selected as the certain non-road background, and the mid-bottom area is pre-selected as the certain road foreground. The GrabCut algorithm will start processing on these two rectangles, and segment the entire image into road and non-road regions.”; Note: Fig. 6 below shows how a 2D image is modified to show the difference between objects and the road (traversable space)) by incorporating one or more of a change in a color of pixels corresponding to one or more of the object or the traversable space or a label corresponding to a predefined classification of pixels comprising one or more of the object or the traversable space (Paragraph 3 in 2nd Col. of Page 2 – “One label refinement approach is to look the image in super-pixels [5]. The processing steps are shown in Fig. 4. The super-pixel segmentation is based on the real camera image, using the K-means clustering method. Specific, an image size is 375*1242 pixels, and we select K=800 to generate 800 super-pixel segments. Initial pixel-wise labels from the OSM road mask are overlaid onto the real image and assigned to all the super-pixel segments. If a segment contains more than 50% initial road labels, it should be relabeled as a road super-pixel, and non-road otherwise”; Note: Fig. 6 shows how the image is processed to have a change of color of pixels for the road (traversable area); see screenshot of Fig. 6 below. The road has a different color from the objects and the rest of the scene. The different color correspond to the labeling/classification of pixels). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Tariq to incorporate the teachings of Zheng to display a processed image with a change in color for the traversable space based on a classification for the benefit of enhancing the user’s environment awareness. For example, having a different color for the road and other objects would help the user see the road clearer, remain in their lane, and avoid collisions.
PNG
media_image7.png
410
591
media_image7.png
Greyscale
Screenshot of Fig. 6 (taken from Zheng)
Tariq modified by Zheng still does not teach assigning values to pixels corresponding to the object, wherein the values correspond to one or more of a heading, a depth within a three-dimensional space, or a regression value. However, Dodson teaches assigning values to pixels corresponding to the object, wherein the values correspond to a depth within a three-dimensional space (Paragraph 0043 – “the on-board neural network subsystem 134 can predict the depth value output of a non-static object from the depth map 165 generated from camera images. For example based on the depth value 165 of the road areas generated from camera images, the on-board neural network subsystem 134 can assign, to a car in the environment, the depth value for a portion of the road that is next to the car. Even though the car is outside the range measurable from lasers and radars, the depth value of the car can still be estimated based on the camera image”; Note: depth values are assigned to pixels corresponding to the car/object). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Tariq to incorporate the teachings of Dodson to assign depth values to the pixels of the object because “When a planning subsystem 136 receives the depth map 165, the planning subsystem 136 can use the depth map 165 to make fully-autonomous or semi-autonomous driving decisions. For example, the planning subsystem 136 can generate a fully-autonomous plan to navigate on a highway or other road by querying the depth map 165 to identify distances to static surfaces in the vicinity of the car and to identify areas where there are occlusions, i.e., areas where depth information is not present in the depth map 165. By identifying occlusions through the depth map, during a turn operation, the vehicle can perform a necessary yield operation to a potential object which cannot be seen because the object is occluded by a building, a car, or a tree, etc” (Dodson: Paragraph 0044). In other words, the depth values help differentiate and locate the road and the objects, which then help avoid collisions.
Claim 4 is rejected under 35 U.S.C. 103 as being unpatentable over Tariq in view of Tran, Rodriguez, and Zheng.
Regarding claim 4, Tariq in view of Tran and Rodriguez teaches the method of claim 1. Tariq further teaches using a vehicle display (Paragraph 0113 – “the input/output (“I/O”) interface 1008 may be configured to coordinate I/O traffic between the processor(s) 1004, the memory 1006, the network interface 1010, sensor(s) 1012, I/O devices 1014, drive system 1016, and/or any other hardware of the vehicle system 1002. In some instances, the I/O devices 1014 may include external and/or internal speaker(s), display(s)”). Tariq does not teach generating for display, a processed image based on the three-dimensional model, wherein the processed image comprises a change in a color of pixels corresponding to the object or the traversable space to differentiate between the object, the traversable space, and non-traversable space. However, Zheng teaches generating for display, a processed image based on the three-dimensional model (Paragraph 4 in 1st Col. of Page 2, Paragraph 1 in 2nd Col. of Page 2 – “The OSM2World toolkit [18] can be used to create 3D models from the OSM XML data. It renders a 3D virtual world from the bird-eye view, and generates a shape object (e.g. road, building, etc.) description file. When the vehicle’s GPS coordinates and its heading direction is given, we can set it as a viewpoint and project the 3D virtual world into the driver’s perspective”; Note: the projected 3D virtual world is equivalent to the processed image based on a 3D model, and it is rendered, implying that it is generated for display), wherein the processed image comprises a change in a color of pixels corresponding to the object or the traversable space to differentiate between the object, the traversable space, and non-traversable space (Fig. 3 – The figure shows how the image is processed to have a change of color of pixels for the road (traversable area); see screenshot of Fig. 3 above. The road has a different color from the objects and the rest of the scene). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Tariq to incorporate the teachings of Zheng to display a processed image based on a 3D model with a change in color for the traversable space for the benefit of highlighting parts of the image that the user should pay more attention to. For example, in the street view in Fig. 3, the road is easier to see due to the change in color, which would make it safer for the user to navigate and be aware of while driving or monitoring an autonomous vehicle.
Claims 5 and 15-16 are rejected under 35 U.S.C. 103 as being unpatentable over Tariq in view of Tran, Rodriguez, and He et al. (Nonparametric Semantic Segmentation for 3D Street Scenes), hereinafter He.
Regarding claim 5, Tariq in view of Tran and Rodriguez teaches the method of claim 1. Tariq further teaches using a vehicle display (Paragraph 0113 – “the input/output (“I/O”) interface 1008 may be configured to coordinate I/O traffic between the processor(s) 1004, the memory 1006, the network interface 1010, sensor(s) 1012, I/O devices 1014, drive system 1016, and/or any other hardware of the vehicle system 1002. In some instances, the I/O devices 1014 may include external and/or internal speaker(s), display(s)”). Tariq does not teach generating for display, a processed image based on the three-dimensional model, wherein: the processed image comprises a three-dimensional bounding area around one or more of the object or the traversable space to differentiate between the object identified, the traversable space, and non-traversable space; and the three-dimensional bounding area modifies one or more of the object or the traversable space in the processed image to include one or more of a color-based demarcation or a text label. However, He teaches generating for display, a processed image based on the three-dimensional model, wherein: the processed image comprises a three-dimensional bounding area around one or more of the object or the traversable space to differentiate between the object identified, the traversable space, and non-traversable space (Paragraph 2 in 2nd Col. of Page 4, Paragraph 1 in 1st Col. of Page 5 – “we employ a simple yet effective method to filter the 3D map. Once we obtain the 3D semantic map and camera trajectory, we argue that the region where the car can drive through should be free 3D space. Therefore, the region covered by the camera trajectory is traversable. In Fig. 5, we know the width (wcar) of the car on which the stereo rig is mounted. In addition, there are free regions (ws) between the car and other obstacles for safety purposes. Any occupied voxels within this bounding box defined by wcar are removed if their semantic labels are not road. As expected, it causes a significant number of holes in the 3D model, however, we know they are likely to be road. We use the geometric information from the remaining voxels (most of them should be road) within the bounding box defined by wcar+ws to generate new road voxels to fill the holes. In particular, we adjust the ws ∈[0.3m 0.5m] (reasonable safe distance between cars) to achieve the smallest standard deviation along z-axis (i.e., altitude above the sea level). The resultant 3D semantic map is shown in Fig. 6(b) and 6(d)”; Note: there is a 3D bounding box around the car that distinguishes it from the road. The bounding box is 3D since it is comprised of voxels. It is obvious that the 3D semantic map is generated for display because it visualizes the 3D scene) and the three-dimensional bounding area modifies one or more of the object or the traversable space in the processed image to include one or more of a color-based demarcation or a text label (Fig. 6C, 6D, and 7B – Figure 6 shows how the model is filtered semantically by the 3D bounding area, and Figure 7 shows how the processed image includes different colors to distinguish the objects and traversable space).
PNG
media_image8.png
484
989
media_image8.png
Greyscale
Screenshot of Fig. 6C and 6D (taken from He)
PNG
media_image9.png
650
1004
media_image9.png
Greyscale
Screenshot of Fig. 7B (taken from He)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Tariq to incorporate the teachings of He to display a processed image of a 3D bounding area based on a 3D model with a change in color for the traversable space for the benefit of highlighting parts of the image that the user should pay more attention to. For example, in the segmented images in Fig. 7B, the road is easier to see due to certain areas being bounded and the changes in color, which would make it safer for the user to navigate and be aware of while driving or monitoring an autonomous vehicle.
Regarding claim 15, Tariq in view of Tran and Rodriguez teaches the system of claim 11. Tariq further teaches a display (Paragraph 0113 – “the input/output (“I/O”) interface 1008 may be configured to coordinate I/O traffic between the processor(s) 1004, the memory 1006, the network interface 1010, sensor(s) 1012, I/O devices 1014, drive system 1016, and/or any other hardware of the vehicle system 1002. In some instances, the I/O devices 1014 may include external and/or internal speaker(s), display(s)”). Tariq does not teach generating a processed image on the display, and wherein the processed image comprises a three- dimensional bounding area around one or more of the object or the traversable space. However, He teaches generating a processed image on the display, and wherein the processed image comprises a three- dimensional bounding area around one or more of the object or the traversable space (Paragraph 2 in 2nd Col. of Page 4, Paragraph 1 in 1st Col. of Page 5 – “we employ a simple yet effective method to filter the 3D map. Once we obtain the 3D semantic map and camera trajectory, we argue that the region where the car can drive through should be free 3D space. Therefore, the region covered by the camera trajectory is traversable. In Fig. 5, we know the width (wcar) of the car on which the stereo rig is mounted. In addition, there are free regions (ws) between the car and other obstacles for safety purposes. Any occupied voxels within this bounding box defined by wcar are removed if their semantic labels are not road. As expected, it causes a significant number of holes in the 3D model, however, we know they are likely to be road. We use the geometric information from the remaining voxels (most of them should be road) within the bounding box defined by wcar+ws to generate new road voxels to fill the holes. In particular, we adjust the ws ∈[0.3m 0.5m] (reasonable safe distance between cars) to achieve the smallest standard deviation along z-axis (i.e., altitude above the sea level). The resultant 3D semantic map is shown in Fig. 6(b) and 6(d)”; Note: there is a 3D bounding box around the car that distinguishes it from the road. The bounding box is 3D since it is comprised of voxels. It is obvious that the 3D semantic map is generated for display because it visualizes the 3D scene. Fig. 6C and 6D above show how the bounding area distinguishes the objects and the road). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Tariq to incorporate the teachings of He to display a processed image of a 3D bounding area for the object or traversable space for the benefit of highlighting parts of the image that the user should pay more attention to. For example, in the segmented images in Fig. 7B, the road is easier to see due to certain areas being bounded and the changes in color, which would make it safer for the user to navigate and be aware of while driving or monitoring an autonomous vehicle.
Regarding claim 16, Tariq in view of Tran, Rodriguez, and He teaches the system of claim 15. Tariq does not teach wherein the three-dimensional bounding area modifies the processed image such that one or more of the object or the traversable space includes one or more of a color-based demarcation or a text label. However, He teaches wherein the three-dimensional bounding area modifies the processed image such that one or more of the object or the traversable space includes one or more of a color-based demarcation or a text label (Fig. 6C, 6D, and 7B; Paragraph 2 in 2nd Col. of Page 4, Paragraph 1 in 1st Col. of Page 5 – “we employ a simple yet effective method to filter the 3D map. Once we obtain the 3D semantic map and camera trajectory, we argue that the region where the car can drive through should be free 3D space. Therefore, the region covered by the camera trajectory is traversable. In Fig. 5, we know the width (wcar) of the car on which the stereo rig is mounted. In addition, there are free regions (ws) between the car and other obstacles for safety purposes. Any occupied voxels within this bounding box defined by wcar are removed if their semantic labels are not road. As expected, it causes a significant number of holes in the 3D model, however, we know they are likely to be road. We use the geometric information from the remaining voxels (most of them should be road) within the bounding box defined by wcar+ws to generate new road voxels to fill the holes. In particular, we adjust the ws ∈[0.3m 0.5m] (reasonable safe distance between cars) to achieve the smallest standard deviation along z-axis (i.e., altitude above the sea level). The resultant 3D semantic map is shown in Fig. 6(b) and 6(d)”; Note: Figure 6 above shows how the model is filtered semantically by the 3D bounding area, and Figure 7 above shows how the processed image includes different colors to distinguish the objects and traversable space). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Tariq to incorporate the teachings of He to display a processed image of a 3D bounding area that differentiates the object and traversable space for the benefit of highlighting parts of the image that the user should pay more attention to. For example, in the segmented images in Fig. 7B, the road is easier to see due to certain areas being bounded and the changes in color, which would make it safer for the user to navigate and be aware of while driving or monitoring an autonomous vehicle.
Claims 17-18, and 22 are rejected under 35 U.S.C. 103 as being unpatentable over Tariq in view of Tran, Rodriguez, and Guo et al. (US 20190303686 A1), hereinafter Guo.
Regarding claim 17, Tariq in view of Tran and Rodriguez teaches the system of claim 11. Tariq further teaches wherein the processing circuitry is further configured to: identify one or more objects in the two-dimensional image (Paragraph 0054 – “FIG. 2B illustrates example portions of the image (i.e., cells in the image) and their associated regions of interest (ROIs) with respect to a classification of “car.” … ROI 204′ may be considered a “detection,” by the ML model, of vehicle 106 in the image 100”; Note: a car in the image is identified). Tariq does not teach wherein the processing circuitry is further configured to: compare the one or more objects to predefined objects stored in memory to determine a confidence factor that corresponds to a likelihood the object corresponds to one or more of the predefined objects; identify the one or more objects as respective predefined objects and in response to the confidence factor being greater than or equal to a threshold, generate one or more respective bounding areas around the respective one or more predefined objects; or in response to the confidence factor being less than the threshold, generating the bounding area around the object. However, Guo teaches comparing the one or more objects to predefined objects stored in memory to determine a confidence factor that corresponds to a likelihood the object corresponds to one or more of the predefined objects (Paragraph 0061– “the object detector 204 may determine whether the object confidence score of the candidate situation object satisfies a predefined object confidence score threshold (e.g., below 20%). If in block 616, the object detector 204 determines that the object confidence score of the candidate situation object satisfies the predefined object confidence score threshold, the method 600 proceeds to block 618. In block 618, the object detector 204 may determine that the candidate situation object is a situation object matching the object type associated with the object detector 204”; Note: the candidate situation object is compared to an object type, which corresponds to a predefined object. The confidence score is equivalent to the confidence factor); identifying the one or more objects as respective predefined objects and in response to the confidence factor being greater than or equal to a threshold, generating one or more respective bounding areas around the respective one or more predefined objects (Paragraph 0058, 0061-0062 – “the part filter 252 may determine that the first part matching score of the candidate first object satisfies the predefined part matching score threshold (e.g., above 70%), and thus the top portion of the situation object matching the first object type “traffic cone” is present in the candidate first object…the object detector 204 may determine whether the object confidence score of the candidate situation object satisfies a predefined object confidence score threshold (e.g., below 20%). If in block 616, the object detector 204 determines that the object confidence score of the candidate situation object satisfies the predefined object confidence score threshold, the method 600 proceeds to block 618. In block 618, the object detector 204 may determine that the candidate situation object is a situation object matching the object type associated with the object detector 204… if the candidate situation object is determined to be a situation object matching the object type associated with the object detector 204, the object detector 204 may determine the bounding box indicating the situation object in the ROI… the object detector 204 may output the bounding box indicating the situation object in the ROI”; Note: the candidate situation object is compared to an object type, which corresponds to a predefined object. The confidence score is equivalent to the confidence factor. When the confidence score threshold is satisfied, a bounding box is determined for the object); or in response to the confidence factor being less than the threshold, generating the bounding area around the object (Paragraph 0061-0062 – “If in block 616, the object detector 204 determines that the object confidence score of the candidate situation object does not satisfy the predefined object confidence score threshold, the method 600 proceeds to block 620. In block 620, the object detector 204 may determine that the candidate situation object is not a situation object matching the object type associated with the object detector 204… object detector 204 may output the bounding box and the object confidence score of all candidate situation objects regardless of whether the object confidence score of these candidate situation objects satisfies the predefined object confidence score threshold”; Note: the candidate situation object is compared to an object type, which corresponds to a predefined object. The confidence score is equivalent to the confidence factor. When the confidence score threshold is not satisfied, a bounding box is still determined for the object). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Tariq to incorporate the teachings of Guo to compare the object to predefined objects and determine a confidence factor for the benefit of quick identification of objects in the image based on the classifications, which would help with a user or vehicle being able to quickly understand and react to the environment. The confidence factor assists in determining which object type the object most resembles. Since Tariq already teaches generating bounding boxes (Paragraph 0052, 0054 – “the ML model may be configured to receive the image and output one or more ROIs… FIG. 2B illustrates example portions of the image (i.e., cells in the image) and their associated regions of interest (ROIs) with respect to a classification of “car.” FIG. 2B illustrates the ROIs as bounding boxes”), it also would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Tariq to incorporate the teachings of Guo to generate bounding areas around either the object or predefined objects for the benefit of making it easier to see and know where the objects are in the environment, regardless of whether the object type is known or unknown.
Regarding claim 18, Tariq in view of Tran, Rodriguez, and Guo teaches the system of claim 17. Tariq further teaches wherein the predefined object is one of a vehicle, a pedestrian, a structure, a driving lane indicator, or a solid object impeding travel along a trajectory from a current vehicle position (Paragraph 0053 – “each cell may be associated with a center, extents, and confidence for each of a car, pedestrian, bicyclist, truck/bus, traffic light, and/or stop sign classification”; Note: the classification represents a predefined object).
Regarding claim 22, Tariq in view of Tran and Rodriguez teaches the method of claim 1. Tariq further teaches wherein generating the bounding area further comprises: detecting the object in the two-dimensional image (Paragraph 0054 – “FIG. 2B illustrates example portions of the image (i.e., cells in the image) and their associated regions of interest (ROIs) with respect to a classification of “car.” FIG. 2B illustrates the ROIs as bounding boxes… ROI 204′ may be considered a “detection,” by the ML model, of vehicle 106 in the image 100”; Note: a car in the image is detected and an ROI (bounding box) is generated for the car). Tariq does not teach comparing the object to one or more predefined objects to determine a confidence factor that corresponds to a likelihood the object corresponds to a predefined object, and in response to the confidence factor being less than a threshold, generating the bounding area around the object; or in response to the confidence factor being greater than or equal to the threshold, generating the bounding area around the predefined object. However, Guo teaches comparing the object to one or more predefined objects to determine a confidence factor that corresponds to a likelihood the object corresponds to a predefined object (Paragraph 0061– “the object detector 204 may determine whether the object confidence score of the candidate situation object satisfies a predefined object confidence score threshold (e.g., below 20%). If in block 616, the object detector 204 determines that the object confidence score of the candidate situation object satisfies the predefined object confidence score threshold, the method 600 proceeds to block 618. In block 618, the object detector 204 may determine that the candidate situation object is a situation object matching the object type associated with the object detector 204”; Note: the candidate situation object is compared to an object type, which corresponds to a predefined object. The confidence score is equivalent to the confidence factor), and in response to the confidence factor being less than a threshold, generating the bounding area around the object (Paragraph 0061-0062 – “If in block 616, the object detector 204 determines that the object confidence score of the candidate situation object does not satisfy the predefined object confidence score threshold, the method 600 proceeds to block 620. In block 620, the object detector 204 may determine that the candidate situation object is not a situation object matching the object type associated with the object detector 204… object detector 204 may output the bounding box and the object confidence score of all candidate situation objects regardless of whether the object confidence score of these candidate situation objects satisfies the predefined object confidence score threshold”; Note: the candidate situation object is compared to an object type, which corresponds to a predefined object. The confidence score is equivalent to the confidence factor. When the confidence score threshold is not satisfied, a bounding box is still determined for the object); or in response to the confidence factor being greater than or equal to the threshold, generating the bounding area around the predefined object (Paragraph 0058, 0061-0062 – “the part filter 252 may determine that the first part matching score of the candidate first object satisfies the predefined part matching score threshold (e.g., above 70%), and thus the top portion of the situation object matching the first object type “traffic cone” is present in the candidate first object…the object detector 204 may determine whether the object confidence score of the candidate situation object satisfies a predefined object confidence score threshold (e.g., below 20%). If in block 616, the object detector 204 determines that the object confidence score of the candidate situation object satisfies the predefined object confidence score threshold, the method 600 proceeds to block 618. In block 618, the object detector 204 may determine that the candidate situation object is a situation object matching the object type associated with the object detector 204… if the candidate situation object is determined to be a situation object matching the object type associated with the object detector 204, the object detector 204 may determine the bounding box indicating the situation object in the ROI… the object detector 204 may output the bounding box indicating the situation object in the ROI”; Note: the candidate situation object is compared to an object type, which corresponds to a predefined object. The confidence score is equivalent to the confidence factor. When the confidence score threshold is satisfied, a bounding box is determined for the object It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Tariq to incorporate the teachings of Guo to compare the object to predefined objects and determine a confidence factor for the benefit of quick identification of objects in the image based on the classifications, which would help with a user or vehicle being able to quickly understand and react to the environment. The confidence factor assists in determining which object type the object most resembles. Since Tariq already teaches generating bounding boxes (Paragraph 0052, 0054 – “the ML model may be configured to receive the image and output one or more ROIs… FIG. 2B illustrates example portions of the image (i.e., cells in the image) and their associated regions of interest (ROIs) with respect to a classification of “car.” FIG. 2B illustrates the ROIs as bounding boxes”), it also would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Tariq to incorporate the teachings of Guo to generate bounding areas around either the object or predefined objects for the benefit of making it easier to see and know where the objects are in the environment, regardless of whether the object type is known or unknown.
Claims 9 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Tariq in view of Tran, Rodriguez, and Hirano et al. (CN 112319466 A), hereinafter Hirano.
Regarding claim 9, Tariq in view of Tran and Rodriguez teaches the method of claim 1. Tariq does not teach wherein the three-dimensional model comprises a characterization of movement of the object relative to the vehicle and the traversable space based on one or more values assigned to pixels corresponding to the object in the two-dimensional image, wherein the one or more values correspond to one or more of a heading, a depth within a three-dimensional space around the vehicle, or a regression value. However, Hirano teaches wherein the three-dimensional model comprises a characterization of movement of the object relative to the vehicle and the traversable space based on one or more values assigned to pixels corresponding to the object in the two-dimensional image (Fig. 36B-36C, Paragraph 0100 – “FIG. 36C provides two sets of three-dimensional shapes with varying colors and gradients to represent predicted trajectories”; Note: color is a type of value assigned to pixels. Additionally, trajectory represents movement), wherein the one or more values correspond to one or more of a heading, a depth within a three-dimensional space around the vehicle, or a regression value (Paragraph 0099 – “Three-dimensional shapes can similarly be designed with different gradients or colors to better indicate direction of travel, speed”; Note: Direction is the equivalent to heading. The assigned color values of pixels are dependent on direction). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Tariq to incorporate the teachings of Hirano, wherein the three-dimensional model comprises a characterization of movement, for the benefit of showing the user the possible trajectory of the object in order to help the user or vehicle prevent collisions with the object (Hirano: Paragraph 0006).
Regarding claim 19, Tariq in view of Tran and Rodriguez teaches the method of claim 11. Tariq does not teach wherein the three-dimensional model comprises a characterization of movement of the object relative to the vehicle and the traversable space based on one or more values assigned to pixels corresponding to the object in the two-dimensional image, wherein the one or more values correspond to one or more of a heading, a depth within a three-dimensional space around the vehicle, or a regression value. However, Hirano teaches wherein the three-dimensional model comprises a characterization of movement of the object relative to the vehicle and the traversable space based on one or more values assigned to pixels corresponding to the object in the two-dimensional image (Fig. 36B-36C, Paragraph 0100 – “FIG. 36C provides two sets of three-dimensional shapes with varying colors and gradients to represent predicted trajectories”; Note: color is a type of value assigned to pixels. Additionally, trajectory represents movement), wherein the one or more values correspond to one or more of a heading, a depth within a three-dimensional space around the vehicle, or a regression value (Paragraph 0099 – “Three-dimensional shapes can similarly be designed with different gradients or colors to better indicate direction of travel, speed”; Note: Direction is the equivalent to heading. The assigned color values of pixels are dependent on direction). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Tariq to incorporate the teachings of Hirano, wherein the three-dimensional model comprises a characterization of movement, for the benefit of showing the user the possible trajectory of the object in order to help the user or vehicle prevent collisions with the object (Hirano: Paragraph 0006).
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Tariq in view of Tran, Rodriguez, Shin et al. (US 20230386043 A1), and Wang (CN 110910453 B), hereinafter Shin and Wang respectively.
Regarding claim 10, Tariq in view of Tran and Rodriguez teaches the method of claim 1. Tariq further teaches generating a first bounding area around an object for a first two-dimensional image captured by a first monocular camera (Paragraph 0045, 0052, 0054, 0115 – “The autonomous vehicle 104 may receive sensor data from one or more sensors of the autonomous vehicle 104. The autonomous vehicle 104 may use this sensor data to determine a trajectory for controlling motion of the autonomous vehicle. The sensor data may include an image such as, for example, example image 100… the ML model may be configured to receive the image and output one or more ROIs… FIG. 2B illustrates example portions of the image (i.e., cells in the image) and their associated regions of interest (ROIs) with respect to a classification of “car.” FIG. 2B illustrates the ROIs as bounding boxes…The sensor(s) 1012 may include, for example, one or more LIDAR sensors, one or more cameras (e.g., RGB-cameras”; Note: the sensor is an RGB camera, which is a monocular camera, that captures an image. A bounding box ROI is generated around the car object); processing data corresponding to pixels within the first bounding area to generate object characterization data (Paragraph 0052-0053 – “an output grid 200 may be discretized into m/4 by n/4 cells, according to an image of m by n pixels. In some instances, the cells may be 4 pixels by 4 pixels…each output cell may correspond with a ROI indicated by a center position (e.g., a <u, v> image coordinate position), extents (e.g., a width and/or height), and/or a confidence level for any one or more classifications. As a non-limiting example, each cell may be associated with a center, extents, and confidence for each of a car, pedestrian, bicyclist, truck/bus, traffic light, and/or stop sign classification”; Note: each cell (group of pixels) within an ROI (bounding area) are classified to generate object characterization data, such as car, pedestrian, etc.). Tariq does not teach wherein the bounding area is a second bounding area, wherein the two-dimensional image is a second two-dimensional image, and wherein generating the second bounding area comprises: generating the second bounding area around the object identified in the second two-dimensional image captured by a second monocular camera based on the object characterization data. However, Shin teaches wherein the bounding area is a second bounding area, wherein the two-dimensional image is a second two-dimensional image (Paragraph 0052 – “the object detection apparatus may generate a virtual bounding box of the first object in the second frame image”; Note: The generated bounding box can be considered a second bounding area, as it is generated in the second frame image. Additionally, a frame image is a 2D image), and wherein generating the second bounding area comprises: generating a first bounding area around an object for a first two-dimensional image captured by a first camera (Paragraph 0047-0048 – “the object detection apparatus may acquire a plurality of frame images…the object detection apparatus may detect a first bounding box of a first object… from the first frame image”); processing data corresponding to pixels within the first bounding area to generate object characterization data (Paragraph 0051, 0075-0077 – “the object detection apparatus may assign a first identification value to the first bounding box…When bounding boxes corresponding to the detected objects are generated for each of the plurality of frame images as described above, … data 303 of the bounding box may be generated with respect to an identification value of each object”; Note: After generating a first bounding box, an identification value is obtained, which can be considered equivalent to characterization data); and generating the second bounding area around the object identified in the second two-dimensional image captured by a second camera based on the object characterization data (Paragraph 0052-0054 – “the object detection apparatus may generate a virtual bounding box of the first object in the second frame image…a third bounding box detected from the second frame image is mapped to the virtual bounding box… when the size of the area where the third bounding box overlaps the virtual bounding box is not less than a reference value, the object detection apparatus may assign the first identification value to the third bounding box”; Note: although the prior art refers to a “third” bounding box, it can be considered equivalent to a second bounding box since it corresponds to a bounding box in the second frame image. Additionally, the identification value can be considered equivalent to the characterization data of the object). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Tariq to incorporate the teachings of Shin to have a second bounding area for the benefit of increasing the accuracy of identifying an object (Shin: Paragraph 0003-0004). Furthermore, Tariq modified by Shin still does not teach a second monocular camera capturing a second two-dimensional image. However, Wang teaches capturing two-dimensional images by a first and second monocular camera (Paragraph 0016 – “synchronously collecting environmental images by using multiple vehicle-mounted monocular cameras”; Note: There are multiple monocular cameras so it is implied that there is at least a first and second monocular camera. Additionally, monocular cameras can only capture two-dimensional images). Since Tariq already teaches a first monocular camera, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Tariq to incorporate a second monocular camera because monocular cameras are cheaper and require less processing power than other types of cameras, such as binocular ones (Wang: Paragraph 0011). Thus, using additional monocular cameras would make the method more efficient and cost less.
Claim 23 is rejected under 35 U.S.C. 103 as being unpatentable over Tariq in view of Tran, Rodriguez, and Ueda (JP 2021181915 A), hereinafter Ueda.
Regarding claim 23, Tariq in view of Tran and Rodriguez teaches the method of claim 1. Tariq does not teach generating for display, using a vehicle display, an overhead view based on the three-dimensional model, wherein the overhead view comprises the vehicle and the object. However, Ueda teaches generating for display, using a vehicle display, an overhead view based on the three-dimensional model (Paragraph 0020-0021 – “The map display control unit 21b is a program module that causes the control unit 20 to execute a function to display a map including the vehicle's current location in an overhead view on the display unit of the user I/F unit 44… the overhead view is drawn as a map showing an overhead view of a 3D model in a virtual 3D space”), wherein the overhead view comprises the vehicle and the object (Fig. 4, Paragraph 0021 – “the control unit 20 virtually places a 3D model consisting of roads and objects existing around the vehicle's current location into a virtual 3D space. The control unit 20 then draws an overhead view by projecting roads and objects that can be seen from the viewpoint onto a projection area U on the plane E when the three-dimensional model is viewed from above from the viewpoint”; Note: Fig. 4 shows the overhead view of the vehicle C and box objects).
PNG
media_image10.png
418
552
media_image10.png
Greyscale
Screenshot of Fig. 4 (taken from Ueda)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Tariq to incorporate the teachings of Ueda to display an overhead view based on the 3D model of the vehicle and object for the benefit of providing the user a clear view of their vehicle location relative to the road and environment, which would make it easier to stay in the lane, avoid obstacles, and be aware of objects in their surroundings.
Claim 24 is rejected under 35 U.S.C. 103 as being unpatentable over Tariq in view of Tran, Rodriguez, Ueda, and Zheng.
Regarding claim 24, Tariq in view of Tran, Rodriguez, and Ueda teaches the method of claim 23. Tariq does not teach generating for display, using another vehicle display, a processed image based on the three-dimensional model, wherein the processed image comprises a change in a color of pixels comprising the object or the traversable space to differentiate between the object, the traversable space, and non-traversable space. However, Zheng teaches generating for display, a processed image based on the three-dimensional model (Paragraph 4 in 1st Col. of Page 2, Paragraph 1 in 2nd Col. of Page 2 – “The OSM2World toolkit [18] can be used to create 3D models from the OSM XML data. It renders a 3D virtual world from the bird-eye view, and generates a shape object (e.g. road, building, etc.) description file. When the vehicle’s GPS coordinates and its heading direction is given, we can set it as a viewpoint and project the 3D virtual world into the driver’s perspective”; Note: the projected 3D virtual world is equivalent to the processed image based on a 3D model, and it is rendered, implying that it is generated for display), wherein the processed image comprises a change in a color of pixels comprising the object or the traversable space to differentiate between the object, the traversable space, and non-traversable space (Fig. 3 – The figure shows how the image is processed to have a change of color of pixels for the road (traversable area); see screenshot of Fig. 3 above. The road has a different color from the rest of the scene). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Tariq to incorporate the teachings of Zheng to display a processed image based on a 3D model with a change in color for the traversable space for the benefit of highlighting parts of the image that the user should pay more attention to. For example, in the street view in Fig. 3, the road is easier to see due to the change in color, which would make it safer for the user to navigate and be aware of while driving or monitoring an autonomous vehicle.
Furthermore, Tariq modified by Zheng still does not teach generating for display using another vehicle display. However, Tran teaches having and using another vehicle display (Col. 58 lines 24-26 – “presenting the change can include displaying the modified cost of the insurance policy in a dashboard-mounted display and/or a heads-up display”; Note: there are multiple vehicle displays). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Tariq to incorporate the teachings of Tran to use another vehicle display for the benefit of being able to show multiple images or views at the same time so that the user does not have to switch views and instead can see them together.
Claim 26 is rejected under 35 U.S.C. 103 as being unpatentable over Tariq in view of Tran, Rodriguez, Noh et al. (Learning Deconvolution Network for Semantic Segmentation), and Wang et al. (Road Damage Detection and Classification with Faster R-CNN), hereinafter Noh and Wang 2 respectively.
Regarding claim 26, Tariq in view of Tran and Rodriguez teaches the method of claim 1. Tariq further teaches wherein the two-dimensional image comprises a plurality of pixels, further comprising determining a grouping of the plurality of pixels based on at least one related pixel value (Fig. 2B, Paragraph 0051, 0053, 0066 – “FIG. 2A illustrates example image 100 and an example output grid 200, where each cell of the output grid 200 identifies a portion of the image 100…a “portion of the image” may include a single pixel of the image and/or a collection of pixels of the image…each output cell may correspond with a ROI indicated by a center position (e.g., a <u, v> image coordinate position), extents (e.g., a width and/or height), and/or a confidence level for any one or more classifications. As a non-limiting example, each cell may be associated with a center, extents, and confidence for each of a car, pedestrian, bicyclist, truck/bus, traffic light, and/or stop sign classification… the ML model may determine a cluster of ROIs that overlap (e.g., that have degrees of alignment with respect to each other that meet or exceed a threshold degree of alignment), thereby indicating a likelihood that an object is represented in the image somewhere in the image around that cluster of ROIs”; Note: the image, which is 2D as shown in Fig. 2B above, is processed to be a grid of cells of pixels. The pixels are grouped as ROIs, which overlap so that they share some related pixels). Tariq does not teach wherein the two-dimensional image comprises a plurality of pixels, further comprising determining a grouping of the plurality of pixels based on at least one related pixel value, wherein: performing the semantic segmentation of the two-dimensional image comprises performing a deconvolution, using a first processing unit, based on the grouping to identify first pixels associated with the object and second pixels associated with the traversable space; and generating the bounding area around the object comprises performing a convolution, using a second processing unit, based on the grouping. However, Noh teaches wherein the two-dimensional image comprises a plurality of pixels, further comprising determining a grouping of the plurality of pixels based on at least one related pixel value (Fig. 2 and 3 – The figures show how the input 2D image is pooled, which forms pixel groups based on the max value. The max value is the related pixel value in this case. See screenshots of Fig. 2 and 3 below), wherein: performing the semantic segmentation of the two-dimensional image comprises performing a deconvolution, using a first processing unit, based on the grouping to identify first pixels associated with the object and second pixels associated with the traversable space (Fig. 7A, Paragraph 4 in 1st Col. of Page 3, Paragraph 2 in 1st Col. of Page 4, Paragraph 3 in 2nd Col. of Page 4 – “we employ unpooling layers in deconvolution network, which perform the reverse operation of pooling and reconstruct the original size of activations as illustrated in Figure 3…our algorithm generates object segmentation masks using deep deconvolution network, where a dense pixel-wise class probability map is obtained by successive operations of unpooling, deconvolution, and rectification… Given our network, semantic segmentation on a whole image is obtained by applying the network to each candidate proposals extracted from the image and aggregating outputs of all proposals to the original image space”; Note: the processing unit that executes the deep deconvolution network is equivalent to the first processing unit. Fig. 7a below shows how the output identifies the pixels of the object and the pixels with the traversable space). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Tariq to incorporate the teachings of Noh to perform deconvolution to perform semantic segmentation based on the grouping because deconvolution allows for being able to learn with weights and producing more accurate results, and pooling/grouping helps lower computational costs.
PNG
media_image11.png
309
1174
media_image11.png
Greyscale
Screenshot of Fig. 2 (taken from Noh)
PNG
media_image12.png
385
571
media_image12.png
Greyscale
Screenshot of Fig. 3 (taken from Noh)
PNG
media_image13.png
247
1160
media_image13.png
Greyscale
Screenshot of Fig. 7A (taken from Noh)
Tariq modified by Noh still does not teach generating the bounding area around the object comprises performing a convolution, using a second processing unit, based on the grouping. However, Wang 2 teaches generating the bounding area around the object comprises performing a convolution, using a second processing unit, based on the grouping (Paragraph 4-5 in 2nd Col. of Page 1 – “In the first stage, the pre-processed input images are processed using a feature extractor. Then the Region Proposal Network (RPN) will use the feature maps as input and outputs a set of rectangular object proposals with their scores. The second stage is the Fast R-CNN detector [12], [13]. For each object proposal, the Region of Interest (RoI) pooling layer will extract a fixed-length feature vector from the feature maps. Then each feature vector will be fed into a sequence of fully connected layers to predict the class label and refine the bounding box”; Note: a bounding box is generated around an object by using a convolutional neural network (fast R-CNN), which performs convolution, based on ROI pooling/grouping. The processing unit that executes the CNN is equivalent to the second processing unit). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Tariq to incorporate the teachings of Wang 2 to perform convolution to generate a bounding area based on the grouping because convolution allows for efficient object detection and boundary generation, and pooling/grouping helps lower computational costs.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Gummadi et al. (US 20220129685 A1) teaches a method for detecting an object around an autonomous vehicle.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MICHELLE HAU MA whose telephone number is (571)272-2187. The examiner can normally be reached M-Th 7-5:30.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, King Poon can be reached on (571) 270-0728. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MICHELLE HAU MA/Examiner, Art Unit 2617 /KING Y POON/Supervisory Patent Examiner, Art Unit 2617
Prosecution Insights