DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 4, 6, 12-14, 17, 19, and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Neumann et al. (US 20220108152 A1) in view of Wang et al. (US 20230135088 A1).
Regarding claim 1, Neumann et al. disclose a cooperative perception system comprising: a plurality of imaging sensors each of the imaging sensors connected to provide output images to one of one or more machine learning (ML) systems (machine learning system, a first output signal characterizing a classification and/or a regression of a first input signal, abstract, imaging sensor such as a camera sensor, which may also be provided as a plurality of sensors such as a stereo camera, separate sensor signal in each case if multiple sensors are involved—of the sensor is forwarded to the control system, [0096], motor vehicle may include multiple sensors such as sensors of a different type, e.g., LIDAR sensors, camera sensors and/or ultrasonic sensors, [0105]), the one or more ML systems trained to process the output images to yield hypotheses, each of the hypotheses comprising: one or more objects and, for each of the objects, values for each of a plurality of regressed parameters (output signal including a first representation that characterizes an expected value of the classification and/or the regression, [0008], characterize an expected value as well as a variance of a regression of at least one real value, For instance, the first representation of the first output signal may include a vector that includes a multiplicity of logit values for a corresponding multiplicity of classes, [0021], first representation being used as an expected value of the probability density function, [0066], The prediction may characterize a classification of the first input signal (x). Alternatively or additionally, it is possible that the prediction characterizes a regression of a real value, a real vector, a real matrix or a real tensor, [0090]) and variation data indicating uncertainty of the value for each of the plurality of regressed parameters (variance of the classification and/or the regression, [0008], “For instance, the first representation of the first output signal may include a vector that includes a multiplicity of logit values for a corresponding multiplicity of classes. The second representation may include a vector of real values, each of these real values characterizing a variance, i.e. an uncertainty, of one of the logit values”, [0021], second representation being used as a variance or a covariance matrix of the probability density function, [0066]); and a processor connected to receive the hypotheses produced by the ML systems and to fuse the hypotheses using the variation data to yield a fused hypothesis (In an advantageous manner, latent representations that characterize a high variance and thus a high uncertainty are thereby able not to be taken into account in the accumulation or to be considered only at a low weighting, [0030] In this case, the machine learning system (60) may be understood to be carrying out a fusion of the sensor signals (S), the result of the fusion including an uncertainty, [0107]).
To the extent Neumann et al. discloses regression and logit values but not the term regressed parameters explicitly, another reference is added herein.
Wang et al. teach a plurality of imaging sensors each of the imaging sensors connected to provide output images (Additionally or alternatively, training data may be generated from real-world data. For example, one or more vehicles may collect sensor data from an equipped sensor, such as one or more cameras and LiDAR [0046]) to one of one or more machine learning (ML) systems (deep learning model(s), [0076]), the one or more ML systems trained to process the output images to yield hypotheses (predict a particular type of information about the 3D surface structure of interest [0078]), each of the hypotheses comprising: one or more objects and, for each of the objects, values for each of a plurality of regressed parameters (regression head may regress a particular type of information about the 3D surface structure, regressed value predicted by the regression head [0036]) and variation data indicating uncertainty of the value for each of the plurality of regressed parameters (confidence head may predict a confidence map with values representing the confidence of a corresponding regressed value predicted by the regression head, [0036]); and a processor connected to receive the hypotheses produced by the ML systems and to fuse the hypotheses using the variation data to yield a fused hypothesis (the DNN may be used to perform multi-modal learning by fusing information from different sources for better prediction, [0038], Including one or more recurrent layers may allow the DNN to leverage information from previous time slices, resulting in better predictions and more stable densification results over time, [0039], LiDAR data that is triggered and/or captured from the same time slice by multiple LiDAR sensors may be accumulated in order to densify the collected data, [0046], The regression head 545 may include any number of layers (e.g., convolutions, pooling, classifiers such as softmax, and/or other types of operations, etc.) that predict a particular type of information about the 3D surface structure of interest, [0078]).
Neumann et al. and Wang et al. are in the same art of regression (Neumann et al., abstract; Wang et al., [0036]). The combination of Wang et al. with Neumann et al. will enable the use of regressed parameters. It would have been obvious at the time of filing to one of ordinary skill in the art to combine the regressed parameters of Wang et al. with the invention of Neumann et al. as this was known at the time of filing, the combination would have predictable results, and as Wang et al. indicate “Embodiments of the present disclosure relate to 3D surface estimation. In some embodiments, a 3D surface structure such as the 3D surface structure of a road (3D road surface) may be observed and estimated to generate a 3D point cloud or other representation of the 3D surface structure. Since the representation may be sparse, one or more densification techniques may be applied to generate a dense representation of the 3D surface structure, which may be provided to an autonomous vehicle drive stack to enable safe and comfortable planning and control of the autonomous vehicle” ([0004]) thereby providing a consumer safety benefit in autonomous vehicles that would result from combining the inventions.
Regarding claim 4, Neumann et al. and Wang et al. disclose the cooperative perception system according to claim 1. Neumann et al. further indicate each of the one or more ML systems is configured to output a precision matrix or covariance matrix that includes the variation data (second output being used as a variance or a covariance matrix of the probability density function, [0132]).
Regarding claim 6, Neumann et al. and Wang et al. disclose the cooperative perception system according to claim 1. Neumann et al. and Wang et al. further indicate the ML systems are configured to classify the one or more objects into each of a plurality of classes and to output the values for each of a plurality of regressed parameters and variation data for each of the plurality of classes for each of the one or more objects (Neumann et al., In accordance with an example embodiment of the present invention, the first output signal may characterize an expected value as well as a variance of a regression of at least one real value. As an alternative, it is possible that the output signal is also able to characterize a classification as well as an uncertainty inherent in the classification. For instance, the first representation of the first output signal may include a vector that includes a multiplicity of logit values for a corresponding multiplicity of classes. The second representation may include a vector of real values, each of these real values characterizing a variance, i.e. an uncertainty, of one of the logit values, [0021]; Wang et al., predict a confidence map with values representing the confidence of a corresponding regressed value predicted by the regression head, [0036], More specifically, in some embodiments, object detection, free space estimation, and/or image segmentation may be applied (e.g., by the 3D estimator 105 or some other component) to classify, segment, and/or predict regions (e.g., pixels) of the image data 102 that are part of a desired class. For example, one more deep learning models (e.g., a convolutional neural network) may be trained to predict one or more segmentation masks and/or confidence maps representing pixels that belong to a drivable road surface or other navigable space, other environmental parts (e.g., sidewalks, buildings), animate objects, and/or other classes, [0059], As such, the road surface point selector 240 may use the segmentation mask or other classification data to elect points from the estimated 3D structure that belong to the class represented by the segmentation mask or other classification data, [0060], labeling tool may accept inputs specifying ground truth annotations identifying points on a surface of interest (e.g., 3D points, boundaries, enclosed regions, class labels), [0133]).
Regarding claim 12, Neumann et al. and Wang et al. disclose the cooperative perception system according to claim 1. Wang et al. further indicate the variation data comprises a multivariate probability distribution and, in fusing the hypotheses, the processor is configured to compute products of the multivariate probability distributions of the hypotheses (Assuming each node in the graph corresponds to a random variable, the Markov random field for the graph may model or otherwise represent a joint probability distribution of the random variables corresponding to the nodes in the graph. Knowing the joint probability distribution and a set of observed values (from the sparse projection image), values for the dense projection image (e.g., a height es ate for each pixel of the dense 2D height map) may be estimated using any known MAP inference algorithm, such as Iterative Conditional Mode, Gaussian Belief Propagation, or others. Thus, a Markov random field may be used to densify the representation of the 3D surface structure, [0035], With the joint probability distribution P (g, o) and a set of observed values of the sparse height map o (or other sparse detection data 110), the Markov random field surface estimator 310 may predict a value (e.g., a height estimate) for each pixel in the dense height map g (or other dense detection data 120) using any known MAP inference algorithm, such as Iterative Conditional Mode, Gaussian Belief Propagation, or others, Generally, the Markov random field surface estimator 310 may estimate a dense representation g of a 3D surface structure from a sparse representation o (e.g., a noisy and partial observation) of the 3D surface structure, [0070]) [Maximum a Posterior (MAP) inference is a type of multivariate probability distributions].
Regarding claim 13, Neumann et al. and Wang et al. disclose the cooperative perception system according to claim 1. Wang et al. further indicate the one or more ML system is trained to, for each of the objects, output a likelihood that the object belongs to each of a plurality of classes (More specifically, in some embodiments, object detection, free space estimation, and/or image segmentation may be applied (e.g., by the 3D estimator 105 or some other component) to classify, segment, and/or predict regions (e.g., pixels) of the image data 102 that are part of a desired class. For example, one more deep learning models (e.g., a convolutional neural network) may be trained to predict one or more segmentation masks and/or confidence maps representing pixels that belong to a drivable road surface or other navigable space, other environmental parts (e.g., sidewalks, buildings), animate objects, and/or other classes. In some embodiments, an individual image (e.g., an RBG image) captured by a single camera may be segmented and/or classified. In some cases, a composite image (e g., an RBG image) may be generated by stitching together images captured by multiple cameras, and the composite image may be segmented and/or classified. As such, a segmentation mask or other classification data delineating or representing the road or drivable space (or some other desired surface) may be obtained and/or generated (e.g., from the predicted masks or confidence maps), [0059] For example, one more machine learning models (e.g., a convolutional neural network) may be trained to predict one or more segmentation mask(s) 1230 and/or confidence maps representing pixels that belong to a drivable road surface or other navigable space, other environmental parts (e.g., sidewalks, buildings), animate objects, and/or other classes [0134]).
Regarding claim 14, Neumann et al. and Wang et al. disclose the cooperative perception system according to claim 1. Wang et al. further indicate the output images include a first set of one or more of the output images that are 2D images and a second set of the output images that are volumetric images (In various examples, a 3D surface structure such as the 3D surface structure of a road (3D road surface) may be observed and estimated to generate a 3D point cloud or other representation of the 3D surface structure, abstract, any suitable 3D structure estimation technique may be applied to generate a representation of a 3D surface structure of interest, such a 3D road surface, [0005], Systems and methods relating to three-dimensional (3D) surface estimation are disclosed. For example, the present disclosure describes systems and methods of reconstructing a 3D surface structure of a road or other component of an environment, for use by autonomous vehicles, semi-autonomous vehicles, robots, and/or other object types, [0031] At a high level, the pipeline 100 may estimate and generate a representation of an observed 3D surface structure, such as that of a 3D road surface or other environmental part, based on image data 102 of a three-dimensional (3D) environment, [0051] resulting representation of the estimated 3D surface structure (e.g., sparse detection data 110) may take any form, such as a 3D point cloud, [0060]).
Regarding claim 17, Neumann et al. and Wang et al. disclose the cooperative perception system according to claim 1. Neumann et al. and Wang et al. further indicate the regressed parameters for the one or more objects comprise localization parameters that estimate a position of the object (Neumann et al., Machine learning systems are typically designed to ascertain a prediction with regard to the input image such as the type and position of an object or a distance to the object on the basis of an input signal, [0005] characterize positions of objects, [0106]; Wang et al., location of other vehicles (e.g., an occupancy grid), information about objects and status of objects as perceived by the controller(s) 1736, etc., [0162], positions and extents of objects, [0198], 3D location estimates of the object obtained from the neural network, [0202]) and one or more object size parameters that estimate a size of the object (Wang et al., 3D structure height maps, [0005], [0051], positions and extents of objects, [0198]).
Regarding claim 19, Neumann et al. and Wang et al. disclose the cooperative perception system according to claim 1. Neumann et al. and Wang et al. further indicate the regressed parameters of the one or more objects comprise one or more object size parameters that estimate size of the object in two or more dimensions (Wang et al., 3D structure height maps, [0005], [0051], regression head may regress a particular type of information about the 3D surface structure, regressed value predicted by the regression head, [0036], positions and extents of objects, [0198]).
Regarding claim 20, Neumann et al. and Wang et al. disclose the cooperative perception system according to claim 1. Neumann et al. and Wang et al. further indicate the processor is configured to filter the hypotheses to remove any of the hypotheses that have a confidence value below a confidence threshold before fusing the hypotheses (Neumann et al., latent representations that characterize a high variance and thus a high uncertainty are thereby able not to be taken into account in the accumulation or to be considered only at a low weighting, [0030] In this case, the machine learning system (60) may be understood to be carrying out a fusion of the sensor signals (S), the result of the fusion including an uncertainty, [0107]; Wang et al., outliers are removed using a statistical or clustering technique, [0033]).
Claim(s) 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Neumann et al. (US 20220108152 A1) and Wang et al. (US 20230135088 A1) as applied to claim 14 above, further in view of Ambrus et al. (US 20230177850 A1).
Regarding claim 15, Neumann et al. and Wang et al. disclose the cooperative perception system according to claim 14. Wang et al. further indicate the ML systems connected to receive the first set of the output images comprise a depth channel and the regressed parameters include a depth estimate output by the depth channel (3D point cloud or other representation of 3D structure may be generated by unprojecting range values from the corresponding depth map into the 3D environment using the location and orientation of the virtual sensor, [0043], regression head 545 and/or the confidence head 550, [0077]) however another reference is added to make this more explicit.
Ambrus et al. teach the ML systems connected to receive the first set of the output images comprise a depth channel and the regressed parameters include a depth estimate output by the depth channel (A method for 3D object detection is described. The method includes predicting, using a trained monocular depth network, an estimated monocular input depth map of a monocular image of a video stream and an estimated depth uncertainty map associated with the estimated monocular input depth map. The method also includes feeding back a depth uncertainty regression loss associated with the estimated monocular input depth map during training of the trained monocular depth network to update the estimated monocular input depth map. The method further includes detecting 3D objects from a 3D point cloud computed from the estimated monocular input depth map based on seed positions selected from the 3D point cloud and the estimated depth uncertainty map. The method also includes selecting 3D bounding boxes of the 3D objects detected from the 3D point cloud based on the seed positions and an aggregated depth uncertainty, abstract, [0008], Aspects of the present disclosure are directed to a new method that is a key issue in the 3D monocular object detection setting. In particular, conventional 3D object detection methods are designed to work with relatively accurate depth sensors (e.g., LIDAR and/or RGB depth (RGB-D) cameras), with well understood noise characteristics and patterns. Nevertheless, a depth regressed by monocular depth estimation networks may exhibit significantly different statistics, leading to a gap in performance when directly applying methods designed to work with data of a different level of quality. To compensate for this deficiency, aspects of the present disclosure provide a modified depth prediction network to learn a per-pixel uncertainty value associated with the depth prediction, [0026], In this configuration, the first sensor 304 captures monocular (single camera) 2D RGB images from which a monocular depth map is predicted by the monocular depth network 312 using the depth uncertainty regression module 314, [0055]).
Neumann et al. and Wang et al. and Ambrus et al. are in the same art of regression (Neumann et al., abstract; Wang et al., [0036]; Ambrus et al., abstract). The combination of Ambrus et al. with Neumann et al. and Wang et al. will enable the use of regressed parameters. It would have been obvious at the time of filing to one of ordinary skill in the art to combine the regressed parameters of Ambrus et al. with the invention of Neumann et al. and Wang et al. as this was known at the time of filing, the combination would have predictable results, and as Ambrus et al. indicate “Aspects of the present disclosure are directed to a new method that is a key issue in the 3D monocular object detection setting. In particular, conventional 3D object detection methods are designed to work with relatively accurate depth sensors (e.g., LIDAR and/or RGB depth (RGB-D) cameras), with well understood noise characteristics and patterns. Nevertheless, a depth regressed by monocular depth estimation networks may exhibit significantly different statistics, leading to a gap in performance when directly applying methods designed to work with data of a different level of quality. To compensate for this deficiency, aspects of the present disclosure provide a modified depth prediction network to learn a per-pixel uncertainty value associated with the depth prediction. Subsequently, the depth prediction is combined with a 3D object detection framework that relies on a voting scheme that operates directly with the 3D points, which avoids any discretization operation. In these aspects of the present disclosure, a 3D object detection framework is modified to incorporate an uncertainty associated with a predicted monocular depth” ([0026]) thereby providing a consumer safety benefit in autonomous vehicles that would result from combining the inventions.
Allowable Subject Matter
Claims 26-27, 29-31, 38, and 41 are allowed. While IDS documents are relevant, in particular “A Survey and Framework of Cooperative Perception: From Heterogeneous Singleton to Hierarchical Cooperation” which teaches fused feature maps, this reference would not be fairly combined with the references cited to reject claim 1.
Claims 7, 8, 22, 25 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is deemed relevant to allowed subject matter but not sufficient alone or in combination to disclose, teach, or fairly suggest the subject matter of the allowed claims:
US 20220075994 A1: For example, graph convolution network 415 may aggregate weighted image patches corresponding to facial landmarks based on a two-dimensional graph such that the weighted image patches are aggregated according to their correspondence with the two-dimensional graph (e.g., as further described herein, for example, with reference to FIG. 7), [0067] “FIG. 8 shows an example of regression error estimation according to aspects of the present disclosure. For instance, FIG. 8 illustrates facial landmarks 800 detected from example images 805-820 via a neural network object detection model described herein (e.g., where the estimated error of the detection of facial landmarks 800 varies across example images 805-820)”, [0078], The facial landmark detection model (e.g., a neural network model described herein, for example, with reference to FIGS. 1, 2, and 4) learns to estimate regression error by designating a neuron to determine the error, [0079], For instance, in some cases, inaccurate facial detection (e.g., in accurate determination of a facial bounding box) may cause facial landmark regression errors. If the estimated error after the first iteration is high, the first iteration may be used again with a different facial detection initialization, [0080], estimating a regression error associated with the detected plurality of objects [0099]
“A novel divergence measure in Dempster–Shafer evidence theory based on pignistic probability transform and its application in multi-sensor data fusion”: “Based on the PPT divergence and the Deng entropy, a new multi-sensor data fusion method is presented. This method takes advantage of PPT divergence to measure the discrepancy
between evidences in order to obtain the credibility weights, and the Deng entropy to measure the uncertainty of the evidences in order to obtain the information volume weights, which can fully mine the potential information between evidences. Then, the credibility weights and
information volume weights are integrated to generate an appropriate weighted average evidence before using Dempster’s combination rule. Two application cases are provided to illustrate the superiority of our method”, p2
PNG
media_image1.png
442
536
media_image1.png
Greyscale
“MHEntropy: Entropy Meets Multiple Hypotheses for Pose and Shape Recovery”:
PNG
media_image2.png
260
362
media_image2.png
Greyscale
PNG
media_image3.png
106
338
media_image3.png
Greyscale
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MICHELLE M ENTEZARI HAUSMANN whose telephone number is (571)270-5084. The examiner can normally be reached 10-7 M-F.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Vincent M Rudolph can be reached at (571) 272-8243. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MICHELLE M ENTEZARI HAUSMANN/Primary Examiner, Art Unit 2671 vc c c c c c
Prosecution Insights