DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
The Amendment filed 10/03/2025 has been entered. Claims 1-20 are pending in this application.
Claims 1, 8- 11, and 19- 20 have been amended. Claims 6- 7, and 18 are cancelled.
Response to Arguments
Applicant's arguments filed 10/03/2025 have been fully considered but they are not persuasive
In response to applicant’s argument that there is no teaching, suggestion, or motivation to combine the references, the examiner recognizes that obviousness may be established by combining or modifying the teachings of the prior art to produce the claimed invention where there is some teaching, suggestion, or motivation to do so found either in the references themselves or in the knowledge generally available to one of ordinary skill in the art. See In re Fine, 837 F.2d 1071, 5 USPQ2d 1596 (Fed. Cir. 1988), In re Jones, 958 F.2d 347, 21 USPQ2d 1941 (Fed. Cir. 1992), and KSR International Co. v. Teleflex, Inc., 550 U.S. 398, 82 USPQ2d 1385 (2007).
Argument A (pages 7- 10) under Independent Claim 1:
The applicant contends that the prior art of record Das fails to disclosed the limitations of amended claim 1. The Applicant asserts that Das fails to teach the limitation of determining a camera class age and a lidar or radar class age. The examiner respectfully disagrees. Das teaches determining the track time as part of the modality specific feature data. Das teaches the track time as the time period indicating how long the object has been tracked using the modality (Das [0018]). The track time is a part of the feature data for the radar, lidar, and vision systems. Furthermore, the prior art of record Castor teaches the age of a tracklet is an indication of how long a valid object has been detected (Castor [0015]), and further adds determining the tracking age by determining how long the feature has been tracked or detected (Castor [0056]).
The Applicant further asserts that Das fails to teach comparing the camera class age and the lidar/radar class age, Das teaches a confidence component that receives confidence levels for multiple modalities and determines which modality’s confidence/output to use based on comparative evaluation. The comparison includes comparison to a threshold, where determining to output vision confidence when fused confidence is below the threshold and the vision confidence is above the threshold. The examiner respectfully disagrees. Das determines modality confidence levels using modality feature data that include the track time. Das’s comparative evaluation of modality confidence levels includes comparing modality specific track data that includes the track time factors in selecting which modality output to use
The Applicant further asserts that Das alone or in combination with the prior art of reference fail to teach updating the classification information of the sensor fusion track the determination results. The Examiner respectfully disagrees. The prior art of record Zhong teaches generating visual object classification data for tracked objects. Das further teaches outputting track data and updating track data based on the intermediate output depending on which modality confidence output is selected. Prior art of record Lee further teach determining association at overlapping points of sensing regions and outputs a sensor fusion track based on the determined association. Therefore, it would have been obvious to the person having ordinary skill in the art to combine the teachings of Zhong, Lee, and Das to update the classification information based on the determination result and the modality comparison outcome.
Argument B (page 10) under Independent Claim 11:
Applicant contends that although independent claim 11 is different is scope, that it has been amended to include the features discusses in Argument A under Independent Claim 1. See response to argument under Argument A as stated above.
Argument C (pages 10- 11) under Dependent Claims 2- 5, 8- 10, 12- 17, 19 and 20:
Applicant contends that claims 2- 5, and 8- 10 depend on claim 1 and therefor follow the same rational as independent claim 1. The Applicant further contends that Claims 12- 17, 19 and 20 depend on claim 11 and therefor follow the same rational as independent claim 11. See response to argument under Argument A as stated above.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1- 2, 4- 5, 8, 11- 14, 16- 17 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Ziguo Zhong (US 20190120955 A1) (hereinafter Zhong) in view of Hoon Lee (US 20200174113 A1) (hereinafter Lee) in view Marcos Paul Gerardo Castro (US 20180137374 A1) (hereinafter Castro) further in view of Subhasis Das (US 20230192145 A1) (hereinafter Das):
Regarding Claim 1, Zhong teaches a sensor information fusion device (“system and method for camera radar fusion.” [0002]), comprising:
a camera sensor (“a video camera”[0006]; “Two types of sensors which may be deployed in automobiles are video sensors and radar sensors. Video sensors are passive sensors that obtain data using video cameras.” [0023]); and
a processor configured to fuse sensor information obtained from the camera sensor (“camera radar fusion to improve sensing compared to separate camera and radar sensing” [0025]; “ one or more processors are configured to perform camera radar fusion on the aligned radar object detection data, the vision motion estimation vector, and the visual object classification data, to generate camera radar fusion data.” [0006]) and recognize an object (“The video processing may include vision motion estimation, image processing, and object recognition” [0040]; “The processed image data from the block 178 is output to the block 214 for image recognition. In an embodiment, the image recognition is performed by a DSP” [0058]),
wherein the processor (“performed by a DSP.” [0054]) is configured to:
update classification information of the sensor fusion track based on i) a result of the determining (“configured to perform image processing, filtering, and object classification and tracking based on the video data,” [0006]; “The DSP determines visual object classification data, for example bounding 214 es for the objects being tracked.” [0079]).
Zhong does not explicitly teach the following limitations; however, in an analogous art, Lee teaches determine whether a sensor fusion track is located in a region of interest (ROI) of the camera sensor and whether camera data obtained by the camera sensor is associated with the sensor fusion track (“When the generated sensor track is located at an overlapping point between the sensing regions of the sensors, the system may determine the association between the sensor fusion track at a previous time point and the sensor track at a current time point, may change the sensor track information in response to the determined association, and may output a sensor fusion track.” [0071]; “determine, when the generated sensor track is located at an overlapping point of sensing regions of sensors” [0009]; “the system may generate a sensor fusion track based on the association of a sensor track located at an overlapping point of the sensing regions of the sensors” [0066]);
determining whether a sensor fusion track is located in the ROI of the camera sensor and whether camera data obtained by the camera sensor is associated with the sensor fusion track (“When the generated sensor track is located at an overlapping point between the sensing regions of the sensors, the system may determine the association between the sensor fusion track at a previous time point and the sensor track at a current time point, may change the sensor track information in response to the determined association, and may output a sensor fusion track.” [0071]; “determine, when the generated sensor track is located at an overlapping point of sensing regions of sensors” [0009]; “the system may generate a sensor fusion track based on the association of a sensor track located at an overlapping point of the sensing regions of the sensors” [0066]).
It would have been obvious to the person having ordinary skill in the art before the effective filling date of the claimed invention to modify the camera radar data fusion as disclosed by Zhong to add the teachings of Lee as disclosed above to increase the reliability of sensor fusion (Lee [0007]).
Lee does not explicitly teach the following limitations; however, in an analogous art, Castro teaches determine a camera class age (“Data obtained from range sensors or (or other sensor data) along with the methods, algorithms, systems, and devices discussed herein allow for the study of three aspects of a tracklet, namely age, consistency, and variability. The age of a tracklet is an indication of how long a valid object has been detected by the tracking algorithm” [0015]; “camera data may be processed to generate range data for one or more points within a field of view of a camera.” [0028]) and a lidar or radar class age (“Data obtained from range sensors or (or other sensor data) along with the methods, algorithms, systems, and devices discussed herein allow for the study of three aspects of a tracklet, namely age, consistency, and variability. The age of a tracklet is an indication of how long a valid object has been detected by the tracking algorithm” [0015]; “the range data may include data from an ultrasound sensor, radar system, LIDAR system, or the like.” [0028]).
It would have been obvious to the person having ordinary skill in the art before the effective filling date of the claimed invention to modify the camera radar data fusion as disclosed by Zhong in view of Lee to further add the teachings of Castro as disclosed above to improve the classification of objects (Castro [0002]).
Castro does not explicitly teach the following limitations; however, in an analogous art, Das teaches compare the camera class age and the lidar or radar class age ([0022] teaches modality specific confidence level that re determined from the modality features that include track time that indicates how long the object has been tracked. Comparison between modality outputs that includes threshold comparison to select a modality out that is a comparison that uses per modality age features),
ii) a result of comparing the camera class age and the lidar or radar class age ([0018]- [0020], [0022], and [0087] teaches comparing the camera modality’s track time and lidar/radar modality track time and update the classification based on the results).
It would have been obvious to the person having ordinary skill in the art before the effective filling date of the claimed invention to modify the camera radar data fusion as disclosed by Zhong in view of Lee and Castro to further add the teachings of Das as disclosed above to improve object detection (Das [0015]).
Regarding Claim 2, Zhong in view of Lee, Castro, and Das teach the sensor information fusion device of claim 1. Lee further teaches the processor (“processing unit“ [0009]) comprises:
an ROI checking unit configured to output a first check signal when the sensor fusion track overlaps within the ROI of the camera sensor (“the system may determine or confirm whether the generated sensor track is located at an overlapping point of the sensing regions of the one or more sensors. “ [0135]; “a sensor track association determination unit configured to determine, when the generated sensor track is located at an overlapping point of sensing regions of sensors,” [0161]; “the sensor track moves from a first overlapping point between the sensing region of the lateral radar CR and the sensing region of the forward camera FC to a second overlapping point among the sensing region of the lateral radar CR, the sensing region of the forward camera FC and the sensing region of the forward radar FR” [0098]);
a sensor association unit configured to output a first association signal when the first check signal is output and the camera data is associated with the sensor fusion track [“changing the sensor track information, if the association is such that the sensor track moves from the sensing region of the forward radar to an overlapping point between the sensing region of the forward radar and the sensing region of the forward camera, the sensor track association determination unit 120 may change the sensor track information.” [0053]].
It would have been obvious to the person having ordinary skill in the art before the effective filling date of the claimed invention to modify the camera radar data fusion as disclosed by Zhong to add the teachings of Lee as disclosed above to increase the reliability of sensor fusion (Lee [0007]).
Lee does not explicitly teach the following limitations; however, in an analogous art, Castro teaches a class update unit configured to:
determine a class age when the sensor fusion track is associated with the camera data (“The tracking parameters component 306 is configured to determine tracking parameters for one or more features in the range data. The features may include points or tracklets that are or were tracked by the tracking component 304. For example, the tracking component 304 may generate and store information about a position or location of each feature over time. Based on this data, the tracking parameters component 306 may determine one or more characteristics or parameters about the movement, positions, or other details of the feature. In one embodiment, the tracking parameters component 306 determines one or more of a tracking age, a detection consistency, and a position variability of a feature.” [0030]; “the tracking parameters component 306 may determine the tracking age for a specific feature by determining how long the specific feature has been tracked or detected.” [0031]; “the classification component 308 may classify a feature as foliage or a solid object based on values for the age” [0037]); and
update the classification information of the sensor fusion track based on the class age (“the classification component 308 is configured to classify a feature of the one or more features as corresponding to foliage based on the tracking parameters. For example, the classification component 308 may classify a feature as foliage or a solid object based on values for the age … the classification component 308 may classify an object as foliage if its corresponding features or tracklets have a low age, low consistency, and/or high variability.” [0037]);
a class fixing unit configured to determine whether a class is in a fixed state based on the class age (“the classification component 308 may classify a feature as foliage or a solid object based on values for the age” [0037]); and
a class conversion unit configured to, when the fixed state of the class is canceled, apply the updated classification information through current class conversion (“the classification component 308 may classify an object as foliage if its corresponding features or tracklets have a low age, low consistency, and/or high variability. For example, if the tracking age falls below an age threshold, the detection consistency falls below a consistency threshold, and/or the position variability exceeds a variability threshold, a feature may be identified as foliage.” [0037]).
It would have been obvious to the person having ordinary skill in the art before the effective filling date of the claimed invention to modify the camera radar data fusion as disclosed by Zhong in view of Lee to further add the teachings of Castro as disclosed above to improve the classification of objects (Castro [0002]).
Regarding Claim 4, Zhong in view of Lee, Castro, and Das teach the sensor information fusion device of claim 2.. Zhong further teaches determine the class age based on a point in time at which the camera data is updated (“These timestamps are used to order the data. Additionally, the timestamps may be used for aging the kinematic model. ” [0028]; “a timestamp indicating the time instance of the source data capture from the corresponding sensor.” [0030]; “in the kinematic model for time aware camera radar fusion processing, the states of object tracking are timestamped, and inputs are time-gated. The timestamp associated with the kinematic model of an individual object is updated in response to a new input, when the timestamp associated with the new input is more recent than the timestamp for the kinematic model.” [0031]; “timestamped data flow, object models are processed with an aging mechanism, in which the uncertainty associated with estimated kinematic models degrade proportionally to the elapsed time. The elapsed time is the interval between the current time and the time instance of the latest acquisition for data and most recent processing.” [0032]).
Regarding Claim 5, Zhong in view of Lee, Castro, and Das teach the sensor information fusion device of claim 2. Castro further teaches when the determined class age is greater than or equal to a preset reference parameter, set a class at a current time to be in the fixed state (“if the tracking age falls below an age threshold, the detection consistency falls below a consistency threshold, and/or the position variability exceeds a variability threshold, a feature may be identified as foliage. Similarly, if the tracking age exceeds an age threshold, the detection consistency exceeds a consistency threshold, and/or the position variability falls below a variability threshold, a feature may be identified as a solid object” [0037]).
It would have been obvious to the person having ordinary skill in the art before the effective filling date of the claimed invention to modify the camera radar data fusion as disclosed by Zhong in view of Lee to further add the teachings of Castro as disclosed above to improve the classification of objects (Castro [0002]).
Regarding Claim 8, Zhong in view of Lee, Castro, and Das teach the sensor information fusion device of claim 1. Das further teaches when the camera class age is less than the lidar/radar class age as a result of the comparing and analyzing, update a lidar/radar class (“In some examples, the process can include a confidence component 130 receiving the radar confidence level 108, the lidar confidence level 116, the vision confidence level 124, and the fused confidence level 128. In some examples, the confidence component 130 can determine the confidence level to output based on whether the fused confidence level 128 is higher than the radar confidence level 108, the lidar confidence level 116, and the vision confidence level 124 or whether at least one of the radar confidence level 108, the lidar confidence level 116, or the vision confidence level 124 is higher than the fused confidence level 128. In some examples, this determination may be made by comparing at least one of the radar confidence level 108, the lidar confidence level 116, or the vision confidence level 124 and the fused confidence level 128 with a threshold. For example, when the fused confidence level 128 is below the threshold and the vision confidence level 124 is above the threshold, then the confidence component 130 can determine to output the vision confidence level 124” [0022]).
It would have been obvious to the person having ordinary skill in the art before the effective filling date of the claimed invention to modify the camera radar data fusion as disclosed by Zhong in view of Lee and Castro to further add the teachings of Das as disclosed above to improve object detection (Das [0015]).
Regarding Claim 11, Zhong teaches a sensor information fusion method performed using a sensor information fusion device (“system and method for camera radar fusion.” [0002]) comprising a processor (“ performed by a DSP.” [0054]), the sensor information fusion method comprising:
determining, using the processor (“performed by a DSP.” [0054]) …,
updating classification information of the sensor fusion track based i) a result of the determining (“configured to perform image processing, filtering, and object classification and tracking based on the video data,” [0006]; “The DSP determines visual object classification data, for example bounding 214 es for the objects being tracked.” [0079]).
Zhong does not explicitly teach the following limitations; however, in an analogous art, Lee teaches whether a sensor fusion track is located in a region of interest (ROI) of a camera sensor (“When the generated sensor track is located at an overlapping point between the sensing regions of the sensors, the system may determine the association between the sensor fusion track at a previous time point and the sensor track at a current time point, may change the sensor track information in response to the determined association, and may output a sensor fusion track.” [0071]; “determine, when the generated sensor track is located at an overlapping point of sensing regions of sensors” [0009]; “the system may generate a sensor fusion track based on the association of a sensor track located at an overlapping point of the sensing regions of the sensors” [0066]);
when the sensor fusion track is located in the ROI of the camera sensor, determining whether camera data obtained by the camera sensor is associated with the sensor fusion track (“When the generated sensor track is located at an overlapping point between the sensing regions of the sensors, the system may determine the association between the sensor fusion track at a previous time point and the sensor track at a current time point, may change the sensor track information in response to the determined association, and may output a sensor fusion track.” [0071]; “determine, when the generated sensor track is located at an overlapping point of sensing regions of sensors” [0009]; “the system may generate a sensor fusion track based on the association of a sensor track located at an overlapping point of the sensing regions of the sensors” [0066]);
whether a sensor fusion track is located in the ROI of the camera sensor and whether camera data obtained by the camera sensor is associated with the sensor fusion track (“When the generated sensor track is located at an overlapping point between the sensing regions of the sensors, the system may determine the association between the sensor fusion track at a previous time point and the sensor track at a current time point, may change the sensor track information in response to the determined association, and may output a sensor fusion track.” [0071]; “determine, when the generated sensor track is located at an overlapping point of sensing regions of sensors” [0009]; “the system may generate a sensor fusion track based on the association of a sensor track located at an overlapping point of the sensing regions of the sensors” [0066]).
It would have been obvious to the person having ordinary skill in the art before the effective filling date of the claimed invention to modify the camera radar data fusion as disclosed by Zhong to add the teachings of Lee as disclosed above to increase the reliability of sensor fusion (Lee [0007]).
Lee does not explicitly teach the following limitations; however, in an analogous art, Castro teaches determine a camera class age (“Data obtained from range sensors or (or other sensor data) along with the methods, algorithms, systems, and devices discussed herein allow for the study of three aspects of a tracklet, namely age, consistency, and variability. The age of a tracklet is an indication of how long a valid object has been detected by the tracking algorithm” [0015]; “camera data may be processed to generate range data for one or more points within a field of view of a camera.” [0028]) and a lidar or radar class age (“Data obtained from range sensors or (or other sensor data) along with the methods, algorithms, systems, and devices discussed herein allow for the study of three aspects of a tracklet, namely age, consistency, and variability. The age of a tracklet is an indication of how long a valid object has been detected by the tracking algorithm” [0015]; “the range data may include data from an ultrasound sensor, radar system, LIDAR system, or the like.” [0028]);
It would have been obvious to the person having ordinary skill in the art before the effective filling date of the claimed invention to modify the camera radar data fusion as disclosed by Zhong in view of Lee to further add the teachings of Castro as disclosed above to improve the classification of objects (Castro [0002]).
Castro does not explicitly teach the following limitations; however, in an analogous art, Das teaches comparing the camera class age and the lidar or radar class age ([0022] teaches modality specific confidence level that re determined from the modality features that include track time that indicates how long the object has been tracked. Comparison between modality outputs that includes threshold comparison to select a modality out that is a comparison that uses per modality age features);
ii) a result of comparing the camera class age and the lidar or radar class age.([0018]- [0020], [0022], and [0087] teaches comparing the camera modality’s track time and lidar/radar modality track time and update the classification based on the results).
It would have been obvious to the person having ordinary skill in the art before the effective filling date of the claimed invention to modify the camera radar data fusion as disclosed by Zhong in view of Lee and Castro to further add the teachings of Das as disclosed above to improve object detection (Das [0015]).
Regarding Claim 12, Zhong in view of Lee, Castro, and Das teach the sensor information fusion method of claim 11. Lee further teaches wherein determining whether the sensor fusion track is located in the ROI of the camera sensor .(“the system may determine or confirm whether the generated sensor track is located at an overlapping point of the sensing regions of the one or more sensors. “ [0135]) includes:
outputting a first check signal when the sensor fusion track overlaps the ROI of the camera sensor.(“the system may determine or confirm whether the generated sensor track is located at an overlapping point of the sensing regions of the one or more sensors. “ [0135]; “a sensor track association determination unit configured to determine, when the generated sensor track is located at an overlapping point of sensing regions of sensors,” [0161]; “the sensor track moves from a first overlapping point between the sensing region of the lateral radar CR and the sensing region of the forward camera FC to a second overlapping point among the sensing region of the lateral radar CR, the sensing region of the forward camera FC and the sensing region of the forward radar FR” [0098]);
It would have been obvious to the person having ordinary skill in the art before the effective filling date of the claimed invention to modify the camera radar data fusion as disclosed by Zhong to add the teachings of Lee as disclosed above to increase the reliability of sensor fusion (Lee [0007]).
Regarding Claim 13, Zhong in view of Lee, Castro, and Das teach the sensor information fusion method of claim 12. Lee further teaches wherein determining whether the camera data is associated with the sensor fusion track .(“the system may determine or confirm whether the generated sensor track is located at an overlapping point of the sensing regions of the one or more sensors. “ [0135]) includes:
when the first check signal is output and the camera data is associated with the sensor fusion track, outputting a first association signal (“changing the sensor track information, if the association is such that the sensor track moves from the sensing region of the forward radar to an overlapping point between the sensing region of the forward radar and the sensing region of the forward camera, the sensor track association determination unit 120 may change the sensor track information.” [0053])
It would have been obvious to the person having ordinary skill in the art before the effective filling date of the claimed invention to modify the camera radar data fusion as disclosed by Zhong to add the teachings of Lee as disclosed above to increase the reliability of sensor fusion (Lee [0007]).
Regarding Claim 14, Zhong in view of Lee, Castro, and teach the sensor information fusion method of claim 13. Castro further teaches when the sensor fusion track is associated with the camera data, determining a class age and updating the classification information of the sensor fusion track based on the class age (“The tracking parameters component 306 is configured to determine tracking parameters for one or more features in the range data. The features may include points or tracklets that are or were tracked by the tracking component 304. For example, the tracking component 304 may generate and store information about a position or location of each feature over time. Based on this data, the tracking parameters component 306 may determine one or more characteristics or parameters about the movement, positions, or other details of the feature. In one embodiment, the tracking parameters component 306 determines one or more of a tracking age, a detection consistency, and a position variability of a feature.” [0030]; “the tracking parameters component 306 may determine the tracking age for a specific feature by determining how long the specific feature has been tracked or detected.” [0031]; “the classification component 308 may classify a feature as foliage or a solid object based on values for the age” [0037]); and
determining whether a class is in a fixed state based on the class age (“the classification component 308 may classify a feature as foliage or a solid object based on values for the age” [0037]); and
when the fixed state of the class is canceled, applying the updated classification information (“the classification component 308 may classify an object as foliage if its corresponding features or tracklets have a low age, low consistency, and/or high variability. For example, if the tracking age falls below an age threshold, the detection consistency falls below a consistency threshold, and/or the position variability exceeds a variability threshold, a feature may be identified as foliage.” [0037]).
It would have been obvious to the person having ordinary skill in the art before the effective filling date of the claimed invention to modify the camera radar data fusion as disclosed by Zhong in view of Lee to further add the teachings of Castro as disclosed above to improve the classification of objects (Castro [0002]).
Regarding Claim 16, Zhong in view of Lee, Castro, and teach the sensor information fusion method of claim 14. Zhong further teaches determining the class age based on a point in time at which the camera data is updated (“These timestamps are used to order the data. Additionally, the timestamps may be used for aging the kinematic model. ” [0028]; “a timestamp indicating the time instance of the source data capture from the corresponding sensor.” [0030]; “in the kinematic model for time aware camera radar fusion processing, the states of object tracking are timestamped, and inputs are time-gated. The timestamp associated with the kinematic model of an individual object is updated in response to a new input, when the timestamp associated with the new input is more recent than the timestamp for the kinematic model.” [0031]; “timestamped data flow, object models are processed with an aging mechanism, in which the uncertainty associated with estimated kinematic models degrade proportionally to the elapsed time. The elapsed time is the interval between the current time and the time instance of the latest acquisition for data and most recent processing.” [0032]).
Regarding Claim 17, Zhong in view of Lee, Castro, and teach the sensor information fusion method of claim 14. Castro further teaches when the class age is greater than or equal to a preset reference parameter, setting a class at a current time to be in the fixed state (“if the tracking age falls below an age threshold, the detection consistency falls below a consistency threshold, and/or the position variability exceeds a variability threshold, a feature may be identified as foliage. Similarly, if the tracking age exceeds an age threshold, the detection consistency exceeds a consistency threshold, and/or the position variability falls below a variability threshold, a feature may be identified as a solid object” [0037]).
It would have been obvious to the person having ordinary skill in the art before the effective filling date of the claimed invention to modify the camera radar data fusion as disclosed by Zhong in view of Lee to further add the teachings of Castro as disclosed above to improve the classification of objects (Castro [0002]).
Regarding Claim 19, Zhong in view of Lee, Castro, and teach the sensor information fusion method of claim 11. Das further teaches when the camera class age is greater than the lidar/radar class age, updating a camera class (“the process can include a confidence component 130 receiving the radar confidence level 108, the lidar confidence level 116, the vision confidence level 124, and the fused confidence level 128. … In some examples, when the fused confidence level 128 is below the second threshold and the vision confidence level 124 is above the threshold, the confidence component 130 can determine to output the vision confidence level 124.” [0022]) ; or
when the camera class age is less than the lidar/radar class age, updating a lidar/radar class (“In some examples, the process can include a confidence component 130 receiving the radar confidence level 108, the lidar confidence level 116, the vision confidence level 124, and the fused confidence level 128. In some examples, the confidence component 130 can determine the confidence level to output based on whether the fused confidence level 128 is higher than the radar confidence level 108, the lidar confidence level 116, and the vision confidence level 124 or whether at least one of the radar confidence level 108, the lidar confidence level 116, or the vision confidence level 124 is higher than the fused confidence level 128. In some examples, this determination may be made by comparing at least one of the radar confidence level 108, the lidar confidence level 116, or the vision confidence level 124 and the fused confidence level 128 with a threshold. For example, when the fused confidence level 128 is below the threshold and the vision confidence level 124 is above the threshold, then the confidence component 130 can determine to output the vision confidence level 124” [0022]).
It would have been obvious to the person having ordinary skill in the art before the effective filling date of the claimed invention to modify the camera radar data fusion as disclosed by Zhong in view of Lee and Castro to further add the teachings of Das as disclosed above to improve object detection (Das [0015]).
Claims 3, 9- 10, 15 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Ziguo Zhong (US 20190120955 A1) (hereinafter Zhong) in view of Hoon Lee (US 20200174113 A1) (hereinafter Lee) in view Marcos Paul Gerardo Castro (US 20180137374 A1) (hereinafter Castro) in view of Subhasis Das (US 20230192145 A1) (hereinafter Das) further in view of Michael J. Delp (US 20170075356 A1) (hereinafter Delp):
Regarding Claim 3, Zhong in view of Lee, Castro, and Das teach the sensor information fusion device of claim 2; however do not appear to explicitly teach the class age has a weight that varies according to a position or performance of the camera sensor.
However, in an analogous art, Delp teaches the class age has a weight that varies according to a position or performance of the camera sensor (“the weighing factor αt for a given cluster-based classifier may increase with increasing amounts of information on the cluster of 3D points from which it is identified according to … where nt is the number of 3D points in the cluster at time t and nα is a parameter controlling how quickly α grows with the number of 3D points. In FIG. 8, nα=250, and it can be seen that 0≦α≦1 and α=0.5 when nt=nα.” [0063]; “begin with, however, assuming this conditional independence, to identify the log-odds L(ω,z1:T) given all local features for a track's clusters of 3D points and the global features for a track over all timesteps (from 1 to T) for the track, and using Bayes rule: … This has the effect of placing unequal weight on the contribution of the track-based classifier, depending on the length of the track. Adding normalization term:” [0061]; “it has been found that the predictive accuracy of the cluster-based classifiers significantly increases with increasing amounts of information on the cluster of 3D points from which they are identified” [0062]; “For the track, there is a single global, or holistic, feature set ω. With both the cluster feature set z1:T and the single holistic feature set ω, the feature set for the track at T is xT=z1:T, ω.” [0045]).
It would have been obvious to the person having ordinary skill in the art before the effective filling date of the claimed invention to modify the camera radar data fusion as disclosed by Zhong in view of Lee, Castro, and Das to further add the teachings of Delp as disclosed above to improve the classification of objects as the object is tracked over time (Delp [0004]).
Regarding Claim 9, Zhong in view of Lee, Castro, and Das teach the sensor information fusion device of claim 1; however do not appear to explicitly teach update current classification information in a pre-class info matrix to compare histories of a camera class or a lidar/radar class.
However, in an analogous art, Delp teaches update current classification information in a pre-class info matrix to compare histories of a camera class or a lidar/radar class (“A given cluster of 3D points at one timestep representing an object in the environment surrounding the vehicle 10 is associated to clusters of 3D points at previous timesteps” [0034]; “FIG. 9 shows the classifier confidence over time for an example track in a cases where the object represented by the track's clusters of 3D points is a bicycle. The solid lines show the confidence for the combined results from the classifier, while the dashed lines show the confidence for the cluster-based classifiers and the dashed-dot lines show the confidence for the track-based classifiers, for each of the pedestrian (cp), bicycle (cb) vehicle (cv) and background (cbg) object classes.” [0067]; “, with a computing device, identifying, from the 3D points, a temporal series of clusters of 3D points representing the same object in the environment surrounding the vehicle as a track, identifying cluster-based classifiers for the object based on identified local features for the clusters in the track, identifying track-based classifiers for the object based on identified global features for the track, combining the cluster-based classifiers and the track-based classifiers to classify the object, with the cluster-based classifiers being weighted based on an amount of information on the clusters from which they are identified, and with the weight increasing with increasing amounts of information, and driving the vehicle along a route based on the object's classification.” [0004]).
It would have been obvious to the person having ordinary skill in the art before the effective filling date of the claimed invention to modify the camera radar data fusion as disclosed by Zhong in view of Lee, Castro, and Das to further add the teachings of Delp as disclosed above to improve the classification of objects as the object is tracked over time (Delp [0004]).
Regarding Claim 10, Zhong in view of Lee, Castro, and Das teach the sensor information fusion device of claim 9; however do not appear to explicitly teach update the current classification information by applying the camera class age or the lidar/radar class age to classification information stored at a previous time to compare the histories of the camera class or the lidar/radar class.
However, in an analogous art, Delp teaches update the current classification information by applying the camera class age or the lidar/radar class age to classification information stored at a previous time to compare the histories of the camera class or the lidar/radar class (“FIG. 9 shows the classifier confidence over time for an example track in a cases where the object represented by the track's clusters of 3D points is a bicycle. The solid lines show the confidence for the combined results from the classifier, while the dashed lines show the confidence for the cluster-based classifiers and the dashed-dot lines show the confidence for the track-based classifiers, for each of the pedestrian (cp), bicycle (cb) vehicle (cv) and background (cbg) object classes. For the first 120 timesteps, only the track-based classifier contributes to the classification of the object because there are too few 3D points in the clusters (i.e., fewer than 25). The object is initially classified as bicycle, for the first 40 timesteps, but then is misclassified as car at a distance of 82 meters, for the next 80 timesteps, until there are enough 3D points to use the cluster-based classifiers at a distance of 40 meters, at which point the bicycle (cb) object class quickly wins out and remains represented in the combined results of the classifier despite several cluster misclassifications later. “ [0067]; “, with a computing device, identifying, from the 3D points, a temporal series of clusters of 3D points representing the same object in the environment surrounding the vehicle as a track, identifying cluster-based classifiers for the object based on identified local features for the clusters in the track, identifying track-based classifiers for the object based on identified global features for the track, combining the cluster-based classifiers and the track-based classifiers to classify the object, with the cluster-based classifiers being weighted based on an amount of information on the clusters from which they are identified, and with the weight increasing with increasing amounts of information, and driving the vehicle along a route based on the object's classification.” [0004]).
It would have been obvious to the person having ordinary skill in the art before the effective filling date of the claimed invention to modify the camera radar data fusion as disclosed by Zhong in view of Lee, Castro, and Das to further add the teachings of Delp as disclosed above to improve the classification of objects as the object is tracked over time (Delp [0004]).
Regarding Claim 15 Zhong in view of Lee, Castro, and Das teach the sensor information fusion method of claim 14; however do not appear to explicitly teach the class age has a weight that varies according to a position or performance of the camera sensor.
However, in an analogous art, Delp teaches the class age has a weight that varies according to a position or performance of the camera sensor (“the weighing factor αt for a given cluster-based classifier may increase with increasing amounts of information on the cluster of 3D points from which it is identified according to … where nt is the number of 3D points in the cluster at time t and nα is a parameter controlling how quickly α grows with the number of 3D points. In FIG. 8, nα=250, and it can be seen that 0≦α≦1 and α=0.5 when nt=nα.” [0063]; “begin with, however, assuming this conditional independence, to identify the log-odds L(ω,z1:T) given all local features for a track's clusters of 3D points and the global features for a track over all timesteps (from 1 to T) for the track, and using Bayes rule: … This has the effect of placing unequal weight on the contribution of the track-based classifier, depending on the length of the track. Adding normalization term:” [0061]; “it has been found that the predictive accuracy of the cluster-based classifiers significantly increases with increasing amounts of information on the cluster of 3D points from which they are identified” [0062]; “For the track, there is a single global, or holistic, feature set ω. With both the cluster feature set z1:T and the single holistic feature set ω, the feature set for the track at T is xT=z1:T, ω.” [0045]).
It would have been obvious to the person having ordinary skill in the art before the effective filling date of the claimed invention to modify the camera radar data fusion as disclosed by Zhong in view of Lee, Castro, and Das to further add the teachings of Delp as disclosed above to improve the classification of objects as the object is tracked over time (Delp [0004]).
Regarding Claim 20, Zhong in view of Lee, Castro, and teach the sensor information fusion method of claim 11; however do not appear to explicitly teach updating current classification information in a pre-class info matrix to compare histories of a camera class or a lidar/radar class; or
updating the current classification information by applying the camera class age or the lidar/radar class age to classification information stored at a previous time to compare the histories of the camera class or the lidar/radar class.
However, in an analogous art, Delp teaches updating current classification information in a pre-class info matrix to compare histories of a camera class or a lidar/radar class (“A given cluster of 3D points at one timestep representing an object in the environment surrounding the vehicle 10 is associated to clusters of 3D points at previous timesteps” [0034]; “FIG. 9 shows the classifier confidence over time for an example track in a cases where the object represented by the track's clusters of 3D points is a bicycle. The solid lines show the confidence for the combined results from the classifier, while the dashed lines show the confidence for the cluster-based classifiers and the dashed-dot lines show the confidence for the track-based classifiers, for each of the pedestrian (cp), bicycle (cb) vehicle (cv) and background (cbg) object classes.” [0067]; “, with a computing device, identifying, from the 3D points, a temporal series of clusters of 3D points representing the same object in the environment surrounding the vehicle as a track, identifying cluster-based classifiers for the object based on identified local features for the clusters in the track, identifying track-based classifiers for the object based on identified global features for the track, combining the cluster-based classifiers and the track-based classifiers to classify the object, with the cluster-based classifiers being weighted based on an amount of information on the clusters from which they are identified, and with the weight increasing with increasing amounts of information, and driving the vehicle along a route based on the object's classification.” [0004]) ; or
updating the current classification information by applying the camera class age or the lidar/radar class age to classification information stored at a previous time to compare the histories of the camera class or the lidar/radar class.(“FIG. 9 shows the classifier confidence over time for an example track in a cases where the object represented by the track's clusters of 3D points is a bicycle. The solid lines show the confidence for the combined results from the classifier, while the dashed lines show the confidence for the cluster-based classifiers and the dashed-dot lines show the confidence for the track-based classifiers, for each of the pedestrian (cp), bicycle (cb) vehicle (cv) and background (cbg) object classes. For the first 120 timesteps, only the track-based classifier contributes to the classification of the object because there are too few 3D points in the clusters (i.e., fewer than 25). The object is initially classified as bicycle, for the first 40 timesteps, but then is misclassified as car at a distance of 82 meters, for the next 80 timesteps, until there are enough 3D points to use the cluster-based classifiers at a distance of 40 meters, at which point the bicycle (cb) object class quickly wins out and remains represented in the combined results of the classifier despite several cluster misclassifications later. “ [0067]; “, with a computing device, identifying, from the 3D points, a temporal series of clusters of 3D points representing the same object in the environment surrounding the vehicle as a track, identifying cluster-based classifiers for the object based on identified local features for the clusters in the track, identifying track-based classifiers for the object based on identified global features for the track, combining the cluster-based classifiers and the track-based classifiers to classify the object, with the cluster-based classifiers being weighted based on an amount of information on the clusters from which they are identified, and with the weight increasing with increasing amounts of information, and driving the vehicle along a route based on the object's classification.” [0004]).
It would have been obvious to the person having ordinary skill in the art before the effective filling date of the claimed invention to modify the camera radar data fusion as disclosed by Zhong in view of Lee, Castro, and Das to further add the teachings of Delp as disclosed above to improve the classification of objects as the object is tracked over time (Delp [0004]).
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MAHMOUD KAMAL ABOUZAHRA whose telephone number is (703)756-1694. The examiner can normally be reached M-F 7:00 AM to 5:00 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jamie Atala can be reached at (571) 272-7384. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MAHMOUD KAMAL ABOUZAHRA/Examiner, Art Unit 2486 /JAMIE J ATALA/Supervisory Patent Examiner, Art Unit 2486