DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Receipt is acknowledged of certified copies of papers submitted under 35 U.S.C. 119(a)-(d), which papers have been placed of record in the file.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 11/19/2024 has been considered by the examiner.
Claim Objections
Claims 1 and 5-7 and are objected to because of the following informalities:
In claim 1, line 4, “comprising the following steps:” should read “comprising: .
In claim 5, lines 6-27, “collect first data… collecting frame, collect second data… collecting frame, determine… first data, determine… second data, update… after the first data collecting frame, fuse… based on the collected first and second data, and apply… about a classification of the detected object.” should read “collect first data… collecting frame[[,]] ; collect second data… collecting frame[[,]] ; determine… first data[[,]] ; determine… second data[[,]] ; update… after the first data collecting frame[[,]] ; fuse… based on the collected first and second data[[,]] ; and apply… about a classification of the detected object.” in order to clearly separate claim limitations using semicolons instead of commas.
In claim 6, line 6, “including the following steps:” should read “including: .
In claim 6, lines 7-28, “collecting first data… collecting frame, collecting second data… collecting frame, determining… first data, determining… second data, updating… after the first data collecting frame, fusing… based on the collected first and second data, and applying… about a classification of the detected object.” should read “collecting first data… collecting frame[[,]] ; collecting second data… collecting frame[[,]] ; determining… first data[[,]] ; determining… second data[[,]] ; updating… after the first data collecting frame[[,]] ; fusing… based on the collected first and second data[[,]] ; and applying… about a classification of the detected object.” in order to clearly separate claim limitations using semicolons instead of commas.
In claim 7, line 6, “causing the computer to perform the following steps:” should read “causing the computer to perform: .
Appropriate correction is required.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed
invention is not identically disclosed as set forth in section 102 of this title, if the
differences between the claimed invention and the prior art are such that the claimed
invention as a whole would have been obvious before the effective filing date of the
claimed invention to a person having ordinary skill in the art to which the claimed
invention pertains. Patentability shall not be negated by the manner in which the
invention was made.
Claims 1 and 3-7 are rejected under 35 U.S.C. 103 as being unpatentable over PANAS (US 20200132845 A1), hereinafter referenced as PANAS in view of HWANG (US 20230213643 A1), hereinafter referenced as HWANG.
Regarding claim 1, PANAS teaches a computer-implemented method for classification of at least one object in an environment of a vehicle (Fig. 5, Paragraph [0055] - PANAS discloses mapping system 200 may include an electronic processor (e.g., computer) 204. Paragraph [0056] - PANAS further discloses referring to Fig. 5, a flowchart 300 illustrates a plurality of high level operations performed by the system 200 in creating a highly detailed environmental map of a scene around a vehicle. Paragraph [0054] - PANAS discloses the mapping system 200 first separates the field of view into primitive objects (objects for which insufficient information is currently available to identify exactly what they are), then begins to refine the data around the shapes and areas of interest to classify these abstracted objects (i.e., objects that can be definitively classified, for example as a bicycle, trash can, etc.).)
using a sensor fusion-based approach (Fig. 1, Paragraph [0050] - PANAS discloses the Lidar control software 16 shown in FIG. 1 may be used in whole or in part to implement a multi-process, “sensor fusion” environmental mapping system 200 (hereinafter simply “mapping system 200”), in accordance with the present disclosure.)
and a data-driven model (Fig. 4, #212d called internal model, Paragraph [0010] - PANAS discloses the system may also include an internal model having an image recognition algorithm that receives the information and uses the image recognition algorithm to initially identify a plurality of objects in the environmental scene.), comprising the following steps:
collecting first data from a first sensor within a first data collecting frame (Fig. 5, Paragraph 0057] - PANAS discloses identification and intention mapping phase 310 involves taking additional measurements (i.e., a second round of measurements) of specific objects identified in the scene (both primitive and abstract) to obtain a better understanding of what the object is and its possible importance/relevance in the environmental scene. Paragraph [0060] - PANAS further discloses the second phase 310, that is, the identification and intention mapping phase, which is implemented using the IIM subsystem 212 bin FIG. 4, adaptive, high spatial resolution sensors.);
collecting second data from at least a second sensor within a second data collecting frame (Fig. 5, Paragraph [0057] - PANAS discloses operations performed at operation 312 involve gathering a 2d image of specific objects identified in the scene in order to apply surface mapping (e.g., shading, contour lines, color, etc.) to the objects in accordance with the priority list established at operation 308. Paragraph [0092] - PANAS further discloses the third category of sensors is 2d image sensors, which is focused on a sensor that can provide color images of surfaces on at close to moderate (up to perhaps 50-100 m) range. These sensors are used may be used for the third phase surface mapping measurement.);
determining a first object representation using the first data (Fig. 5, Paragraph [0060] - PANAS discloses in the second phase 310, that is, the identification and intention mapping phase, which is implemented using the IIM subsystem 212b in FIG. 4, adaptive, high spatial resolution sensors, for example those forming the micromirrors within the digital micromirror assembly 32 of FIG. 1, may be used to generate detailed 3D maps of the objects (both primitive and abstract objects).);
determining a second object representation using the second data (Fig. 5, Paragraph [0061] - PANAS discloses in the third phase 312, which is the surface mapping phase using the surface mapping subsystem 212 c, surface mapping on one or more objects in the internal model 212 d may be performed. PANAS further discloses The 2d image sensors associated with the surface mapping subsystem 212 c which may provide color discernment, may be used to generate detailed surface maps of the objects identified for study, taking into account the color of each object under study.);
updating the first object representation and/or the second object representation (Fig. 5, Paragraph [0058] - PANAS discloses operation 314 involves updating the internal map 212 d with the results of the second and third phase operations, and then re-running the first phase measurements (i.e., repeating operation 302) as indicated by loop back line 316. PANAS further discloses the method 300 is repeated and continuously updates the internal model 212 d with new primitive and abstract objects.),
depending on an arrival of third data from the at least second sensor that has been collected in a third data collecting frame after the first data collecting frame (Fig. 6, Paragraph [0069] - PANAS discloses after each first phase 302 refresh, valuation algorithms (306 a and 308 a) choose where to distribute the focus of the second phase 310 and third phase 310 sensing operations among the objects in the internal model 212 d. The order of measurement for the second and third phase measurements, which are contained in the priority lists 306 and 308, are both updated after each first phase refresh. This allows the second phase 310 and third phase 312 measurements to operate out of sync from the first phase 302 refresh rate.);
Although PANAS further teaches fusing the first object representation and the at least second object representation to determine an updated representation of the object based on the collected first and second data (Fig. 5, Paragraph [0058] - PANAS discloses at operation 322 the updated internal map may be transmitted to the vehicle's autonomous operation algorithm 218. Paragraph [0086] - PANAS discloses mapping system 200 would then be abstracting and classifying multi-sensor fused data into an object-based environmental map. Paragraph [0097] - PANAS discloses the proposed phases of measurement intelligently fuse the data from the multiple kinds of sensors in a way that maximizes the quality of the captured data, thus providing a full environmental awareness with only a small fraction of the data rate of the conventional technique of simple data overlap.);
PANAS fails to explicitly teach applying the updated representation for training the data-driven model as input data for the data-driven model to obtain output data containing an information about a classification of the detected object.
However, HWANG explicitly teaches applying the updated representation for training the data-driven model as input data for the data-driven model (Fig. 5, Paragraph [0065] - HWANG discloses the system obtains and processes image data 510 and radar data 520 that characterize the same scene of an environment to generate a fused point cloud 530 which is a three-dimensional representation of the scene of the environment. Fig. 7, Paragraph [0082] - HWANG further discloses the system processes the masked fused point cloud using the output neural network in accordance with current values of output network parameters to generate a training network output for a given machine learning task (step 708). In the implementations where the output neural network is configured to perform an object detection task, the training network output can be an output including 3-D bounding box data that identifies objects that are characterized by the fused point cloud.)
to obtain output data containing an information about a classification of the detected object (Fig. 5, Paragraph [0065] - HWANG discloses the system uses an output neural network configured as an object detection neural network to process the fused point cloud 530 to generate an object detection output that identifies locations of multiple 3-D bounding boxes in the fused point cloud 530. As shown in the circles, one of the 3-D bounding boxes defined with reference to the fused point cloud 530 identifies a vehicle that is present in the scene and that corresponds to the vehicle characterized in the image data 510 and radar data 520, respectively. See also Fig. 7, Paragraph [0082].).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date the claimed invention was made to combine the teachings of PANAS of having a computer-implemented method for classification of at least one object in an environment of a vehicle using a sensor fusion-based approach and a data-driven model, comprising the following steps: collecting first data from a first sensor within a first data collecting frame; collecting second data from at least a second sensor within a second data collecting frame; determining a first object representation using the first data; determining a second object representation using the second data; updating the first object representation and/or the second object representation, depending on an arrival of third data from the at least second sensor that has been collected in a third data collecting frame after the first data collecting frame; fusing the first object representation and the at least second object representation to determine an updated representation of the object based on the collected first and second data; with the teachings of HWANG of having applying the updated representation for training the data-driven model as input data for the data-driven model to obtain output data containing an information about a classification of the detected object.
Wherein PANAS’s computer-implemented method wherein having applying the updated representation for training the data-driven model as input data for the data-driven model to obtain output data containing an information about a classification of the detected object.
The motivation behind this modification would have been to provide an enhanced method of object detection/classification by improving detection capability and accuracy, since both PANAS and HWANG relate to methods and systems for processing sensor data, wherein PANAS relates to environmental and object mapping systems used in connection with autonomous systems, and more particularly to a mapping system which is able to apply a multi-modal sensing approach to the environment which is able to use the gathered knowledge about the surrounding objects to adaptively maximize the value of the captured data, to provide the best possible detection capability, and deliver the best possible, fine resolution 3D mapping capability to support the object detection; the proposed phases of measurement intelligently fuse the data from the multiple kinds of sensors in a way that maximizes the quality of the captured data, thus providing a full environmental awareness with only a small fraction of the data rate of the conventional technique of simple data overlap, and HWANG relates to processing sensor data, e.g., camera sensor data or radar sensor data, using neural networks; the use of attention mechanism during sensor fusion allows for the described techniques to effectively make use of radar features to more accurately predict pixel depths when generating a fused point cloud that will be processed using a neural network. Please see PANAS (US 20200132845 A1), Paragraph [0003, 0096], and HWANG (US 20230213643 A1), Paragraph [0002, 0018].
Regarding claim 3, PANAS in view of HWANG teach the computer-implemented method according to claim 1,
PANAS further teaches wherein the updating of the first object representation and/or the second object representation includes updating a state information of a current first object representation and/or current second object representation at a time t (Fig. 6, Paragraph [0053] - PANAS discloses eventual operation of the sensors and the image recognition algorithms result in an abstracted, object-oriented environmental map. The abstracted objects are objects that may have started out as primitive objects (i.e., not showing sufficient detail to make a definite determination as to exactly what they are), but through two or more repeated scanning operations, have been sufficiently updated with new information to form abstracted objects, where their identity is now known and classified. Paragraph [0070] - PANAS further discloses one possible general instantiation for the prioritization algorithms which produce priority lists 306 and 308 would be to generate a value for each object in the field of view and distribute the second phase 310 and third phase 312 detailed mapping sensing to the highest value objects at any time. The value of each object could be set to grow at a certain rate (in time or per frame) to encourage the algorithm to get around to measuring it again periodically. Unknown objects in the internal map would be given a high priority value.).
Regarding claim 4, PANAS in view of HWANG teach the computer-implemented method according to claim 1,
PANAS further teaches wherein during the step of updating the first object representation and/or the second object representation at time t (Fig. 5, Paragraph [0058] - PANAS discloses operation 314 involves updating the internal map 212 d with the results of the second and third phase operations, and then re-running the first phase measurements (i.e., repeating operation 302) as indicated by loop back line 316. PANAS further discloses the internal model may be updated with information confirming an identity of one or more of the various objects in the internal model, adding a metadata tag (e.g., information on now known dimensions and/or location in 3D space, etc.) and subsuming more primitive objects or abstracted objects into a new object when these component objects have been positively discerned/identified to make up a single larger object.),
a step of collecting a state information of at least a potential second object at time t is performed (Fig. 5, Paragraph [0058] - PANAS discloses operation 322 may be repeated continuously, for example at 10-30 hz as is typical for autonomous algorithms, or whenever required by the algorithm, as the method 300 is repeated and continuously updates the internal model 212d with new primitive and abstract objects.).
Regarding claim 5, PANAS teaches a vehicle (Fig. 4, #202 called vehicle, Paragraph [0055] - PANAS discloses mapping system 200 may output real time data to the vehicle's autonomous operation algorithm 218 for use in controlling navigation of the vehicle 202.), comprising:
a system (Fig. 4, #200 called mapping system, Paragraph [0050]) for classification of at least one object in an environment of a vehicle (Fig. 4, Paragraph [0050] - PANAS discloses FIG. 4 shows the mapping system 200 being used with a car 202, although it will be appreciated that the mapping system 200 is not limited to use with just automobiles such as cars, trucks, vans, SUVs, light pickups, etc., but rather may be used with virtually any type of vehicle such as drones, autonomous land vehicles used in mining, agriculture, earth moving, as well as fixed wing aircraft, and even marine vessels. Paragraph [0054] - PANAS discloses the mapping system 200 first separates the field of view into primitive objects (objects for which insufficient information is currently available to identify exactly what they are), then begins to refine the data around the shapes and areas of interest to classify these abstracted objects (i.e., objects that can be definitively classified, for example as a bicycle, trash can, etc.)
using a sensor fusion-based approach (Fig. 1, Paragraph [0050] - PANAS discloses the Lidar control software 16 shown in FIG. 1 may be used in whole or in part to implement a multi-process, “sensor fusion” environmental mapping system 200 (hereinafter simply “mapping system 200”), in accordance with the present disclosure.)
and a data-driven model (Fig. 4, #212d called internal model, Paragraph [0010] - PANAS discloses the system may also include an internal model having an image recognition algorithm that receives the information and uses the image recognition algorithm to initially identify a plurality of objects in the environmental scene.),
the system (Fig. 4, #200 called mapping system, Paragraph [0050]) configured to:
collect first data from a first sensor within a first data collecting frame (Fig. 5, Paragraph 0057] - PANAS discloses identification and intention mapping phase 310 involves taking additional measurements (i.e., a second round of measurements) of specific objects identified in the scene (both primitive and abstract) to obtain a better understanding of what the object is and its possible importance/relevance in the environmental scene. Paragraph [0060] - PANAS further discloses the second phase 310, that is, the identification and intention mapping phase, which is implemented using the IIM subsystem 212 bin FIG. 4, adaptive, high spatial resolution sensors.),
collect second data from at least a second sensor within a second data collecting frame (Fig. 5, Paragraph [0057] - PANAS discloses operations performed at operation 312 involve gathering a 2d image of specific objects identified in the scene in order to apply surface mapping (e.g., shading, contour lines, color, etc.) to the objects in accordance with the priority list established at operation 308. Paragraph [0092] - PANAS further discloses the third category of sensors is 2d image sensors, which is focused on a sensor that can provide color images of surfaces on at close to moderate (up to perhaps 50-100 m) range. These sensors are used may be used for the third phase surface mapping measurement.),
determine a first object representation using the first data (Fig. 5, Paragraph [0060] - PANAS discloses in the second phase 310, that is, the identification and intention mapping phase, which is implemented using the IIM subsystem 212b in FIG. 4, adaptive, high spatial resolution sensors, for example those forming the micromirrors within the digital micromirror assembly 32 of FIG. 1, may be used to generate detailed 3D maps of the objects (both primitive and abstract objects).),
determine a second object representation using the second data (Fig. 5, Paragraph [0061] - PANAS discloses in the third phase 312, which is the surface mapping phase using the surface mapping subsystem 212 c, surface mapping on one or more objects in the internal model 212 d may be performed. PANAS further discloses The 2d image sensors associated with the surface mapping subsystem 212 c which may provide color discernment, may be used to generate detailed surface maps of the objects identified for study, taking into account the color of each object under study.),
update the first object representation and/or the second object representation (Fig. 5, Paragraph [0058] - PANAS discloses operation 314 involves updating the internal map 212 d with the results of the second and third phase operations, and then re-running the first phase measurements (i.e., repeating operation 302) as indicated by loop back line 316. PANAS further discloses the method 300 is repeated and continuously updates the internal model 212 d with new primitive and abstract objects.),
depending on an arrival of third data from the at least second sensor that has been collected in a third data collecting frame after the first data collecting frame (Fig. 6, Paragraph [0069] - PANAS discloses after each first phase 302 refresh, valuation algorithms (306 a and 308 a) choose where to distribute the focus of the second phase 310 and third phase 310 sensing operations among the objects in the internal model 212 d. The order of measurement for the second and third phase measurements, which are contained in the priority lists 306 and 308, are both updated after each first phase refresh. This allows the second phase 310 and third phase 312 measurements to operate out of sync from the first phase 302 refresh rate.),
Although PANAS further teaches fuse the first object representation and the at least second object representation to determine an updated representation of the object based on the collected first and second data (Fig. 5, Paragraph [0058] - PANAS discloses at operation 322 the updated internal map may be transmitted to the vehicle's autonomous operation algorithm 218. Paragraph [0086] - PANAS discloses mapping system 200 would then be abstracting and classifying multi-sensor fused data into an object-based environmental map. Paragraph [0097] - PANAS discloses the proposed phases of measurement intelligently fuse the data from the multiple kinds of sensors in a way that maximizes the quality of the captured data, thus providing a full environmental awareness with only a small fraction of the data rate of the conventional technique of simple data overlap.),
PANAS fails to explicitly teach and apply the updated representation for training the data-driven model as input data for the data- driven model to obtain output data containing an information about a classification of the detected object.
However, HWANG explicitly teaches and apply the updated representation for training the data-driven model as input data for the data- driven model (Fig. 5, Paragraph [0065] - HWANG discloses the system obtains and processes image data 510 and radar data 520 that characterize the same scene of an environment to generate a fused point cloud 530 which is a three-dimensional representation of the scene of the environment. Fig. 7, Paragraph [0082] - HWANG further discloses the system processes the masked fused point cloud using the output neural network in accordance with current values of output network parameters to generate a training network output for a given machine learning task (step 708). In the implementations where the output neural network is configured to perform an object detection task, the training network output can be an output including 3-D bounding box data that identifies objects that are characterized by the fused point cloud.)
to obtain output data containing an information about a classification of the detected object (Fig. 5, Paragraph [0065] - HWANG discloses the system uses an output neural network configured as an object detection neural network to process the fused point cloud 530 to generate an object detection output that identifies locations of multiple 3-D bounding boxes in the fused point cloud 530. As shown in the circles, one of the 3-D bounding boxes defined with reference to the fused point cloud 530 identifies a vehicle that is present in the scene and that corresponds to the vehicle characterized in the image data 510 and radar data 520, respectively. See also Fig. 7, Paragraph [0082].).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date the claimed invention was made to combine the teachings of PANAS of having a vehicle, comprising: a system for classification of at least one object in an environment of a vehicle using a sensor fusion-based approach and a data-driven model, the system configured to: collect first data from a first sensor within a first data collecting frame, collect second data from at least a second sensor within a second data collecting frame, determine a first object representation using the first data, determine a second object representation using the second data, update the first object representation and/or the second object representation, depending on an arrival of third data from the at least second sensor that has been collected in a third data collecting frame after the first data collecting frame, fuse the first object representation and the at least second object representation to determine an updated representation of the object based on the collected first and second data, with the teachings of HWANG of having and apply the updated representation for training the data-driven model as input data for the data-driven model to obtain output data containing an information about a classification of the detected object.
Wherein PANAS’s vehicle wherein having and apply the updated representation for training the data-driven model as input data for the data-driven model to obtain output data containing an information about a classification of the detected object.
The motivation behind this modification would have been to provide an enhanced method of object detection/classification by improving detection capability and accuracy, since both PANAS and HWANG relate to methods and systems for processing sensor data, wherein PANAS relates to environmental and object mapping systems used in connection with autonomous systems, and more particularly to a mapping system which is able to apply a multi-modal sensing approach to the environment which is able to use the gathered knowledge about the surrounding objects to adaptively maximize the value of the captured data, to provide the best possible detection capability, and deliver the best possible, fine resolution 3D mapping capability to support the object detection; the proposed phases of measurement intelligently fuse the data from the multiple kinds of sensors in a way that maximizes the quality of the captured data, thus providing a full environmental awareness with only a small fraction of the data rate of the conventional technique of simple data overlap, and HWANG relates to processing sensor data, e.g., camera sensor data or radar sensor data, using neural networks; the use of attention mechanism during sensor fusion allows for the described techniques to effectively make use of radar features to more accurately predict pixel depths when generating a fused point cloud that will be processed using a neural network. Please see PANAS (US 20200132845 A1), Paragraph [0003, 0096], and HWANG (US 20230213643 A1), Paragraph [0002, 0018].
Regarding claim 6, PANAS teaches a computer (Fig. 4, Paragraph [0055] - PANAS discloses mapping system 200 may include an electronic processor (e.g., computer) 204.), comprising:
a processor (Fig. 4, #204 called electronic processor, Paragraph [0055] - PANAS discloses mapping system 200 may include an electronic processor (e.g., computer) 204.) configured to perform a computer-implemented method for classification of at least one object in an environment of a vehicle (Fig. 5, Paragraph [0055] - PANAS discloses mapping system 200 may include an electronic processor (e.g., computer) 204. Paragraph [0056] - PANAS further discloses referring to Fig. 5, a flowchart 300 illustrates a plurality of high level operations performed by the system 200 in creating a highly detailed environmental map of a scene around a vehicle. Paragraph [0054] - PANAS discloses the mapping system 200 first separates the field of view into primitive objects (objects for which insufficient information is currently available to identify exactly what they are), then begins to refine the data around the shapes and areas of interest to classify these abstracted objects (i.e., objects that can be definitively classified, for example as a bicycle, trash can, etc.).)
using a sensor fusion-based approach (Fig. 1, Paragraph [0050] - PANAS discloses the Lidar control software 16 shown in FIG. 1 may be used in whole or in part to implement a multi-process, “sensor fusion” environmental mapping system 200 (hereinafter simply “mapping system 200”), in accordance with the present disclosure.)
and a data-driven model (Fig. 4, #212d called internal model, Paragraph [0010] - PANAS discloses the system may also include an internal model having an image recognition algorithm that receives the information and uses the image recognition algorithm to initially identify a plurality of objects in the environmental scene.), including the following steps:
collecting first data from a first sensor within a first data collecting frame (Fig. 5, Paragraph 0057] - PANAS discloses identification and intention mapping phase 310 involves taking additional measurements (i.e., a second round of measurements) of specific objects identified in the scene (both primitive and abstract) to obtain a better understanding of what the object is and its possible importance/relevance in the environmental scene. Paragraph [0060] - PANAS further discloses the second phase 310, that is, the identification and intention mapping phase, which is implemented using the IIM subsystem 212 bin FIG. 4, adaptive, high spatial resolution sensors.),
collecting second data from at least a second sensor within a second data collecting frame (Fig. 5, Paragraph [0057] - PANAS discloses operations performed at operation 312 involve gathering a 2d image of specific objects identified in the scene in order to apply surface mapping (e.g., shading, contour lines, color, etc.) to the objects in accordance with the priority list established at operation 308. Paragraph [0092] - PANAS further discloses the third category of sensors is 2d image sensors, which is focused on a sensor that can provide color images of surfaces on at close to moderate (up to perhaps 50-100 m) range. These sensors are used may be used for the third phase surface mapping measurement.),
determining a first object representation using the first data (Fig. 5, Paragraph [0060] - PANAS discloses in the second phase 310, that is, the identification and intention mapping phase, which is implemented using the IIM subsystem 212b in FIG. 4, adaptive, high spatial resolution sensors, for example those forming the micromirrors within the digital micromirror assembly 32 of FIG. 1, may be used to generate detailed 3D maps of the objects (both primitive and abstract objects).),
determining a second object representation using the second data (Fig. 5, Paragraph [0061] - PANAS discloses in the third phase 312, which is the surface mapping phase using the surface mapping subsystem 212 c, surface mapping on one or more objects in the internal model 212 d may be performed. PANAS further discloses The 2d image sensors associated with the surface mapping subsystem 212 c which may provide color discernment, may be used to generate detailed surface maps of the objects identified for study, taking into account the color of each object under study.),
updating the first object representation and/or the second object representation (Fig. 5, Paragraph [0058] - PANAS discloses operation 314 involves updating the internal map 212 d with the results of the second and third phase operations, and then re-running the first phase measurements (i.e., repeating operation 302) as indicated by loop back line 316. PANAS further discloses the method 300 is repeated and continuously updates the internal model 212 d with new primitive and abstract objects.),
depending on an arrival of third data from the at least second sensor that has been collected in a third data collecting frame after the first data collecting frame (Fig. 6, Paragraph [0069] - PANAS discloses after each first phase 302 refresh, valuation algorithms (306 a and 308 a) choose where to distribute the focus of the second phase 310 and third phase 310 sensing operations among the objects in the internal model 212 d. The order of measurement for the second and third phase measurements, which are contained in the priority lists 306 and 308, are both updated after each first phase refresh. This allows the second phase 310 and third phase 312 measurements to operate out of sync from the first phase 302 refresh rate.),
fusing the first object representation and the at least second object representation to determine an updated representation of the object based on the collected first and second data (Fig. 5, Paragraph [0058] - PANAS discloses at operation 322 the updated internal map may be transmitted to the vehicle's autonomous operation algorithm 218. Paragraph [0086] - PANAS discloses mapping system 200 would then be abstracting and classifying multi-sensor fused data into an object-based environmental map. Paragraph [0097] - PANAS discloses the proposed phases of measurement intelligently fuse the data from the multiple kinds of sensors in a way that maximizes the quality of the captured data, thus providing a full environmental awareness with only a small fraction of the data rate of the conventional technique of simple data overlap.),
PANAS fails to explicitly teach applying the updated representation for training the data-driven model as input data for the data- driven model to obtain output data containing an information about a classification of the detected object.
However, HWANG explicitly teaches applying the updated representation for training the data-driven model as input data for the data- driven model (Fig. 5, Paragraph [0065] - HWANG discloses the system obtains and processes image data 510 and radar data 520 that characterize the same scene of an environment to generate a fused point cloud 530 which is a three-dimensional representation of the scene of the environment. Fig. 7, Paragraph [0082] - HWANG further discloses the system processes the masked fused point cloud using the output neural network in accordance with current values of output network parameters to generate a training network output for a given machine learning task (step 708). In the implementations where the output neural network is configured to perform an object detection task, the training network output can be an output including 3-D bounding box data that identifies objects that are characterized by the fused point cloud.)
to obtain output data containing an information about a classification of the detected object (Fig. 5, Paragraph [0065] - HWANG discloses the system uses an output neural network configured as an object detection neural network to process the fused point cloud 530 to generate an object detection output that identifies locations of multiple 3-D bounding boxes in the fused point cloud 530. As shown in the circles, one of the 3-D bounding boxes defined with reference to the fused point cloud 530 identifies a vehicle that is present in the scene and that corresponds to the vehicle characterized in the image data 510 and radar data 520, respectively. See also Fig. 7, Paragraph [0082].).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date the claimed invention was made to combine the teachings of PANAS of having a computer, comprising: a processor configured to perform a computer- implemented method for classification of at least one object in an environment of a vehicle using a sensor fusion-based approach and a data-driven model, including the following steps: collecting first data from a first sensor within a first data collecting frame, collecting second data from at least a second sensor within a second data collecting frame, determining a first object representation using the first data, determining a second object representation using the second data, updating the first object representation and/or the second object representation, depending on an arrival of third data from the at least second sensor that has been collected in a third data collecting frame after the first data collecting frame, fusing the first object representation and the at least second object representation to determine an updated representation of the object based on the collected first and second data, with the teachings of HWANG of having applying the updated representation for training the data-driven model as input data for the data-driven model to obtain output data containing an information about a classification of the detected object.
Wherein PANAS’s computer wherein having applying the updated representation for training the data-driven model as input data for the data-driven model to obtain output data containing an information about a classification of the detected object.
The motivation behind this modification would have been to provide an enhanced method of object detection/classification by improving detection capability and accuracy, since both PANAS and HWANG relate to methods and systems for processing sensor data, wherein PANAS relates to environmental and object mapping systems used in connection with autonomous systems, and more particularly to a mapping system which is able to apply a multi-modal sensing approach to the environment which is able to use the gathered knowledge about the surrounding objects to adaptively maximize the value of the captured data, to provide the best possible detection capability, and deliver the best possible, fine resolution 3D mapping capability to support the object detection; the proposed phases of measurement intelligently fuse the data from the multiple kinds of sensors in a way that maximizes the quality of the captured data, thus providing a full environmental awareness with only a small fraction of the data rate of the conventional technique of simple data overlap, and HWANG relates to processing sensor data, e.g., camera sensor data or radar sensor data, using neural networks; the use of attention mechanism during sensor fusion allows for the described techniques to effectively make use of radar features to more accurately predict pixel depths when generating a fused point cloud that will be processed using a neural network. Please see PANAS (US 20200132845 A1), Paragraph [0003, 0096], and HWANG (US 20230213643 A1), Paragraph [0002, 0018].
Regarding claim 7, PANAS teaches a non-transitory machine-readable data medium on which is stored a computer program (Fig. 4, #206 called non-volatile memory, Paragraph [0055] - PANAS discloses mapping system 200 may include an electronic processor (e.g., computer) 204, a non-volatile memory 206 used to store one or more algorithms 208 as well as data collected by the system 200, and an interface 210 for communicating with a bus of the vehicle 202 (e.g., typically a CAN bus in automotive applications). The mapping system 200 may also include an imaging subsystem 212 which is controlled by the processor 204, and which uses the memory 206 to temporarily store data collected during use of the mapping system 200.)
for classification of at least one object in an environment of a vehicle (Fig. 5, Paragraph [0056] - PANAS discloses referring to Fig. 5, a flowchart 300 illustrates a plurality of high level operations performed by the system 200 in creating a highly detailed environmental map of a scene around a vehicle. Paragraph [0054] - PANAS further discloses the mapping system 200 first separates the field of view into primitive objects (objects for which insufficient information is currently available to identify exactly what they are), then begins to refine the data around the shapes and areas of interest to classify these abstracted objects (i.e., objects that can be definitively classified, for example as a bicycle, trash can, etc.).)
using a sensor fusion-based approach (Fig. 1, Paragraph [0050] - PANAS discloses the Lidar control software 16 shown in FIG. 1 may be used in whole or in part to implement a multi-process, “sensor fusion” environmental mapping system 200 (hereinafter simply “mapping system 200”), in accordance with the present disclosure.)
and a data-driven model (Fig. 4, #212d called internal model, Paragraph [0010] - PANAS discloses the system may also include an internal model having an image recognition algorithm that receives the information and uses the image recognition algorithm to initially identify a plurality of objects in the environmental scene.),
the computer program, when executed by a computer (Fig. 4, #206 called non-volatile memory, Paragraph [0055] - PANAS discloses mapping system 200 may include an electronic processor (e.g., computer) 204, a non-volatile memory 206 used to store one or more algorithms 208 as well as data collected by the system 200, and an interface 210 for communicating with a bus of the vehicle 202 (e.g., typically a CAN bus in automotive applications).), causing the computer to perform the following steps:
collecting first data from a first sensor within a first data collecting frame (Fig. 5, Paragraph 0057] - PANAS discloses identification and intention mapping phase 310 involves taking additional measurements (i.e., a second round of measurements) of specific objects identified in the scene (both primitive and abstract) to obtain a better understanding of what the object is and its possible importance/relevance in the environmental scene. Paragraph [0060] - PANAS further discloses the second phase 310, that is, the identification and intention mapping phase, which is implemented using the IIM subsystem 212 bin FIG. 4, adaptive, high spatial resolution sensors.);
collecting second data from at least a second sensor within a second data collecting frame (Fig. 5, Paragraph [0057] - PANAS discloses operations performed at operation 312 involve gathering a 2d image of specific objects identified in the scene in order to apply surface mapping (e.g., shading, contour lines, color, etc.) to the objects in accordance with the priority list established at operation 308. Paragraph [0092] - PANAS further discloses the third category of sensors is 2d image sensors, which is focused on a sensor that can provide color images of surfaces on at close to moderate (up to perhaps 50-100 m) range. These sensors are used may be used for the third phase surface mapping measurement.);
determining a first object representation using the first data (Fig. 5, Paragraph [0060] - PANAS discloses in the second phase 310, that is, the identification and intention mapping phase, which is implemented using the IIM subsystem 212b in FIG. 4, adaptive, high spatial resolution sensors, for example those forming the micromirrors within the digital micromirror assembly 32 of FIG. 1, may be used to generate detailed 3D maps of the objects (both primitive and abstract objects).);
determining a second object representation using the second data (Fig. 5, Paragraph [0061] - PANAS discloses in the third phase 312, which is the surface mapping phase using the surface mapping subsystem 212 c, surface mapping on one or more objects in the internal model 212 d may be performed. PANAS further discloses The 2d image sensors associated with the surface mapping subsystem 212 c which may provide color discernment, may be used to generate detailed surface maps of the objects identified for study, taking into account the color of each object under study.);
updating the first object representation and/or the second object representation (Fig. 5, Paragraph [0058] - PANAS discloses operation 314 involves updating the internal map 212 d with the results of the second and third phase operations, and then re-running the first phase measurements (i.e., repeating operation 302) as indicated by loop back line 316. PANAS further discloses the method 300 is repeated and continuously updates the internal model 212 d with new primitive and abstract objects.),
depending on an arrival of third data from the at least second sensor that has been collected in a third data collecting frame after the first data collecting frame (Fig. 6, Paragraph [0069] - PANAS discloses after each first phase 302 refresh, valuation algorithms (306 a and 308 a) choose where to distribute the focus of the second phase 310 and third phase 310 sensing operations among the objects in the internal model 212 d. The order of measurement for the second and third phase measurements, which are contained in the priority lists 306 and 308, are both updated after each first phase refresh. This allows the second phase 310 and third phase 312 measurements to operate out of sync from the first phase 302 refresh rate.);
Although PANAS further teaches fusing the first object representation and the at least second object representation to determine an updated representation of the object based on the collected first and second data (Fig. 5, Paragraph [0058] - PANAS discloses at operation 322 the updated internal map may be transmitted to the vehicle's autonomous operation algorithm 218. Paragraph [0086] - PANAS discloses mapping system 200 would then be abstracting and classifying multi-sensor fused data into an object-based environmental map. Paragraph [0097] - PANAS discloses the proposed phases of measurement intelligently fuse the data from the multiple kinds of sensors in a way that maximizes the quality of the captured data, thus providing a full environmental awareness with only a small fraction of the data rate of the conventional technique of simple data overlap.);
PANAS fails to explicitly teach applying the updated representation for training the data-driven model as input data for the data-driven model to obtain output data containing an information about a classification of the detected object.
However, HWANG explicitly teaches applying the updated representation for training the data-driven model as input data for the data-driven model (Fig. 5, Paragraph [0065] - HWANG discloses the system obtains and processes image data 510 and radar data 520 that characterize the same scene of an environment to generate a fused point cloud 530 which is a three-dimensional representation of the scene of the environment. Fig. 7, Paragraph [0082] - HWANG further discloses the system processes the masked fused point cloud using the output neural network in accordance with current values of output network parameters to generate a training network output for a given machine learning task (step 708). In the implementations where the output neural network is configured to perform an object detection task, the training network output can be an output including 3-D bounding box data that identifies objects that are characterized by the fused point cloud.)
to obtain output data containing an information about a classification of the detected object (Fig. 5, Paragraph [0065] - HWANG discloses the system uses an output neural network configured as an object detection neural network to process the fused point cloud 530 to generate an object detection output that identifies locations of multiple 3-D bounding boxes in the fused point cloud 530. As shown in the circles, one of the 3-D bounding boxes defined with reference to the fused point cloud 530 identifies a vehicle that is present in the scene and that corresponds to the vehicle characterized in the image data 510 and radar data 520, respectively. See also Fig. 7, Paragraph [0082].).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date the claimed invention was made to combine the teachings of PANAS of having a non-transitory machine-readable data medium on which is stored a computer program for classification of at least one object in an environment of a vehicle using a sensor fusion- based approach and a data-driven model, the computer program, when executed by a computer, causing the computer to perform the following steps: collecting first data from a first sensor within a first data collecting frame; collecting second data from at least a second sensor within a second data collecting frame; determining a first object representation using the first data; determining a second object representation using the second data; updating the first object representation and/or the second object representation, depending on an arrival of third data from the at least second sensor that has been collected in a third data collecting frame after the first data collecting frame; fusing the first object representation and the at least second object representation to determine an updated representation of the object based on the collected first and second data; with the teachings of HWANG of having applying the updated representation for training the data-driven model as input data for the data-driven model to obtain output data containing an information about a classification of the detected object.
Wherein PANAS’s non-transitory machine-readable data medium wherein having applying the updated representation for training the data-driven model as input data for the data-driven model to obtain output data containing an information about a classification of the detected object.
The motivation behind this modification would have been to provide an enhanced method of object detection/classification by improving detection capability and accuracy, since both PANAS and HWANG relate to methods and systems for processing sensor data, wherein PANAS relates to environmental and object mapping systems used in connection with autonomous systems, and more particularly to a mapping system which is able to apply a multi-modal sensing approach to the environment which is able to use the gathered knowledge about the surrounding objects to adaptively maximize the value of the captured data, to provide the best possible detection capability, and deliver the best possible, fine resolution 3D mapping capability to support the object detection; the proposed phases of measurement intelligently fuse the data from the multiple kinds of sensors in a way that maximizes the quality of the captured data, thus providing a full environmental awareness with only a small fraction of the data rate of the conventional technique of simple data overlap, and HWANG relates to processing sensor data, e.g., camera sensor data or radar sensor data, using neural networks; the use of attention mechanism during sensor fusion allows for the described techniques to effectively make use of radar features to more accurately predict pixel depths when generating a fused point cloud that will be processed using a neural network. Please see PANAS (US 20200132845 A1), Paragraph [0003, 0096], and HWANG (US 20230213643 A1), Paragraph [0002, 0018].
Claim 2 is rejected under 35 U.S.C. 103 as being unpatentable over PANAS (US 20200132845 A1), hereinafter referenced as PANAS in view of HWANG (US 20230213643 A1), hereinafter referenced as HWANG, in further view of ION (US 20220076402 A1), hereinafter referenced as ION.
Regarding claim 2, PANAS in view of HWANG teach the computer-implemented method according to claim 1,
PANAS in view of HWANG fail to explicitly teach wherein the first data collecting frame and/or the second data collecting frame is represented by a data collecting window of a fixed length and within a defined time interval during which data collection of the first sensor and/or the at least second sensor is performed.
However, ION explicitly teaches wherein the first data collecting frame and/or the second data collecting frame (Fig. 6, Paragraph [0073] - ION discloses a third row 630 that indicates example timeframes.)
is represented by a data collecting window of a fixed length (Fig. 6, Paragraph [0074] - ION discloses diagram 600 shows how a frame, also referred to as image, may be captured by an image sensor every one hundred milliseconds for ten milliseconds [wherein ten milliseconds is a window of a fixed length] and the transferred frame is then transferred for twenty milliseconds.)
and within a defined time interval during which data collection of the first sensor and/or the at least second sensor is performed (Fig. 6, Paragraph [0074] - ION discloses diagram 600 shows how a frame, also referred to as image, may be captured by an image sensor every one hundred milliseconds [wherein every one hundred milliseconds is a defined time interval] for ten milliseconds and the transferred frame is then transferred for twenty milliseconds.).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date the claimed invention was made to combine the teachings of PANAS in view of HWANG of having a computer-implemented method for classification of at least one object in an environment of a vehicle using a sensor fusion-based approach and a data-driven model, comprising the following steps: collecting first data from a first sensor within a first data collecting frame; collecting second data from at least a second sensor within a second data collecting frame; with the teachings of ION of having wherein the first data collecting frame and/or the second data collecting frame is represented by a data collecting window of a fixed length and within a defined time interval during which data collection of the first sensor and/or the at least second sensor is performed.
Wherein PANAS’s computer-implemented method wherein the first data collecting frame and/or the second data collecting frame is represented by a data collecting window of a fixed length and within a defined time interval during which data collection of the first sensor and/or the at least second sensor is performed.
The motivation behind this modification would have been to provide an enhanced method of object detection/classification by improving detection capability and simplifying processing pipeline architecture, since both PANAS and ION relate to methods and systems for processing sensor data, wherein PANAS relates to environmental and object mapping systems used in connection with autonomous systems, and more particularly to a mapping system which is able to apply a multi-modal sensing approach to the environment which is able to use the gathered knowledge about the surrounding objects to adaptively maximize the value of the captured data, to provide the best possible detection capability, and deliver the best possible, fine resolution 3D mapping capability to support the object detection; the proposed phases of measurement intelligently fuse the data from the multiple kinds of sensors in a way that maximizes the quality of the captured data, thus providing a full environmental awareness with only a small fraction of the data rate of the conventional technique of simple data overlap, and ION describes technologies relating to an image sensor that captures images and performs processing on the image sensor to detect objects; having the image sensor 100 perform processing may simplify a processing pipeline architecture, provide higher bandwidth and lower latency, allow for selective frame rate operations, reduce costs with the stacked architecture, provide higher system reliability as an integrated circuit may have fewer potential points of failure, and provide significant cost and power savings on computational resources. Please see PANAS (US 20200132845 A1), Paragraph [0003, 0096], and ION (US 20220076402 A1), Paragraph [0003, 0054].
Conclusion
Listed below are the prior arts made of record and not relied upon but are considered pertinent to applicant’s disclosure.
CHAO et al. (US 20260035006 A1) - A computer implemented method is provided. The method, comprises: receiving from an autonomous vehicle sensor data, wherein the sensor data is indicative of an environment in which the vehicle is currently located or was previously located. The method further comprises receiving or determining additional data associated with at least one of: the sensor data, the vehicle, or the environment, wherein the additional data is different from the sensor data. The method further comprises displaying, on a display, an output comprising a representation of the sensor data. The method further comprises determining, based at least in part on the additional data, a reliability metric, the reliability metric being indicative of how reliable the sensor data is at representing the environment in which the vehicle is located at a current time. The method further comprises causing the output on the display to be based at least in part on the reliability metric...… Fig. 1, Abstract.
BLAES et al. (US 12221115 B1) - Techniques for updating data operations in a perception system are discussed herein. A vehicle may use a perception system to capture data about an environment proximate to the vehicle. The perception system may output the data about the environment to a system configured to determine positions of objects relative to the perception system over time. The positions of the objects may be used to estimate an object velocity and may be compared against machine learning model outputs in a self-supervised manner to train the machine learning model to output object velocities based on inputs from the perception system. The output of the machine learning model may include a two-dimensional velocity for objects in the environment. The two-dimensional velocity may be used for a vehicle system such that the vehicle can make environmentally aware operational decisions, which may improve reaction time(s) and/or safety outcomes of the vehicle...… Fig. 1, Abstract.
BURLINA et al. (US 20240353231 A1) - A transformer-based machine-learned model may use cross-attention between map data and various sensor data and/or perception data, such as an object detection, to augment perception tasks. In particular, the transformer-based machine-learned model may comprise two or more encoders, one of which may determine a first embedding from map data and a second encoder that may determine a second embedding from sensor data and/or perception data. An encoder may determine a score that may be used to determine various outputs that may improve partially occluded object detection, ground plane classification, static object detection, and suppress false positive object detections....… Fig. 1, Abstract.
DAS et al. (US 20230192145 A1) - Techniques for determining an output from a plurality of sensor modalities are discussed herein. Features from a radar sensor, a lidar sensor, and an image sensor may be input into respective models to determine respective intermediate outputs associated with a tracks associated with an object and associated confidence levels. The Intermediate outputs from a radar model, a lidar model, and an vision model may be input into a fused model to determine a fused confidence level and fused output associated with the track. The fused confidence level and the individual confidence levels are compared to a threshold to generate the track to transmit to a planning system or prediction system of an autonomous vehicle. Additionally, a vehicle controller can control the autonomous vehicle based on the track and/or on the confidence level(s)....… Fig. 1, Abstract.
GOEDDEL et al. (US 20220194436 A1) - The method for dynamically updating an environmental representation of an autonomous agent can include: receiving a set of inputs S210; generating an environmental representation S220; and updating the environmental representation S230. Additionally or alternatively, the method S200 can include providing the environmental representation to a planning module S240 and/or any other suitable processes. The method S200 functions to generate and/or dynamically update an environmental representation to facilitate control of an autonomous agent.....… Fig. 1, Abstract.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to BEZAWIT N SHIMELES whose telephone number is (571)272-7663. The examiner can normally be reached M-F 7:30am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chineyere Wills-Burns can be reached at (571) 272-9752. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/BEZAWIT NOLAWI SHIMELES/Examiner, Art Unit 2673
/CHINEYERE WILLS-BURNS/Supervisory Patent Examiner, Art Unit 2673
Prosecution Insights