Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over Qian (Chinese Patent Pub. No.: CN 115862000 A), hereinafter Qian, in view of Fu (FEATUP: A MODEL-AGNOSTIC FRAMEWORK FOR FEATURES AT ANY RESOLUTION, International Conference on Learning Representations. Vol. 2024. May, 2024), hereinafter Fu, further in view of Qin (Chinese Patent Pub. No.: CN 118736198 A), hereinafter Qin.
Regarding claim 1, Qian teaches a method for saliency-driven refinement of object detection proposals comprising: obtaining image data generated by one or more sensors of a vehicle (The point cloud detection dataset can be collected by the LiDAR installed on the vehicle, including scene data under different weather conditions, time periods, and operating conditions, without specific limitations. [n0057]); extracting, from the image data (Step 2.1: Truncate the data within the specified ROI region of the stitched point cloud, according to the coordinate system of the LiDAR radar [n0104]), a plurality of features to generate a plurality of feature maps (Step 3: Construct a sparse convolutional network module, feed the voxelized initial features obtained in the previous step into the network to obtain a 3D feature map [n0107]); projecting the plurality of upsampled feature maps (performing convolutional upsampling and deconvolutional upsampling on the high-level semantic feature map after cross-channel feature aggregation to obtain an upsampling result and an enlarged semantic feature map; concatenating the first upsampling result with the shallow spatial feature map after cross-channel feature aggregation to obtain a semantic spatial feature map. [n0085]) onto a plurality of Bird’s Eye View (BEV) feature maps (performing pixel-by-pixel weighting on the cross-channel feature aggregation semantic spatial feature map and the enlarged semantic feature map to obtain the final bird's-eye view feature map. [n0085]); generating, based on the plurality of BEV feature maps, a plurality of object detection proposals for the image data (inputting the bird's-eye view feature map into a confidence correction module, outputting corrected 3D detection boxes, and using the corrected 3D detection boxes to detect targets. Abstract); and applying a confidence threshold (The IoU threshold between the 3D candidate boxes and the ground truth is iou_thresh, and the threshold of the sum of the products of the predicted IoU value and the actual IoU value of the ground truth is cnt_thresh. [n0131]) to the plurality of object detection proposals to generate refined object detection proposals (inputting the bird's-eye view feature map into a confidence correction module, outputting corrected 3D detection boxes, and using the corrected 3D detection boxes to detect targets. Abstract).
Qian does not teach the following limitations as further recited, but Fu further teaches upsampling the plurality of feature maps to generate a plurality of upsampled feature maps, wherein the upsampling increases a spatial resolution of the plurality of feature maps (FeatUp: a new method to significantly improve the spatial resolution of any model’s features, parametrized as either a fast feedforward upsampling network or an implicit network. Page 2 3rd paragraph); generating one or more saliency maps for the image data (In short, this component uses a 1x1 convolution to predict a saliency map from the high-resolution features. Page 5 4th paragraph).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Qian to incorporate the teachings of Fu to upsample the plurality of feature maps to generate a plurality of upsampled feature maps, wherein the upsampling increases a spatial resolution of the plurality of feature maps, and generate saliency maps for the image data in order to improve the spatial resolution of the model’s features and improve the model prediction performance.
The combination of Qian and Fu does not teach the following limitations as further recited, but Qin further teaches applying a confidence threshold, based on the one or more saliency maps (The saliency score may be binarized according to a preset threshold, the portion above the threshold is set to 1, the portion below the threshold is set to 0, and the generated binarized image is used as a mask, so that the salient portion in the image can be identified, and the method is used for tasks such as object segmentation and target detection. Page 10 4th paragraph).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Qin to apply a confidence threshold, based on the saliency maps, in order to obtain finer and complete significance prediction and ensure uniformity and consistency of prediction.
Regarding claim 2, Qin in the combination teaches the method of claim 1, further comprising: determine a confidence threshold based on the one or more saliency maps (The saliency score may be binarized according to a preset threshold, the portion above the threshold is set to 1, the portion below the threshold is set to 0, and the generated binarized image is used as a mask, so that the salient portion in the image can be identified, and the method is used for tasks such as object segmentation and target detection. Page 10 4th paragraph).
Qian in the combination further teaches adjusting the confidence threshold (correcting the confidence (which reads on “adjusting the confidence threshold”) of the 3D detection box classification based on the IOU regression feature map and the autoregression feature map, and obtaining the execution degree of the corrected 3D detection box based on the positive/negative classification feature map and the directional classification feature map. [n0088]).
Regarding claim 3, Qian in the combination teaches the method of claim 1, wherein the plurality of object detection proposals include one or more suppressed true positives (correcting the confidence (In other words, if the confidence is too low and not corrected, it may result in a suppressed true positive.) of the 3D detection box classification based on the IOU regression feature map and the autoregression feature map, and obtaining the execution degree of the corrected 3D detection box based on the positive and negative classification feature map and the orientation classification feature map. [0013]).
Regarding claim 4, Qian in the combination teaches the method of claim 1, further comprising: incorporating positional and color components of the image data (Among them, the features extracted by the shallow spatial feature map are closer to the input feature map, containing more information about pixels, such as color, texture, edges and corners, and can capture more details with higher resolution. [n0080]) into the plurality of BEV feature maps (extracting the shallow spatial feature map and the high-level semantic feature map of the bird's-eye view feature map; and aggregating the shallow spatial feature map and the high-level semantic feature map to obtain the final bird's-eye view feature map. [n0079]).
Regarding claim 5, Fu in the combination teaches the method of claim 4, wherein the positional and color components include information about spatial frequency content of a corresponding feature map (We follow the guidance of prior works (Mildenhall et al., 2020; Sitzmann et al., 2020a; Tancik et al., 2020) and use Fourier features to improve the spatial resolution of our implicit representations. Page 7 1st paragraph).
Qian in the combination further teaches information about spatial content (Among them, the features extracted by the shallow spatial feature map are closer to the input feature map, containing more information about pixels, such as color, texture, edges and corners, and can capture more details with higher resolution. [n0080]) of a corresponding BEV feature map (extracting the shallow spatial feature map and the high-level semantic feature map of the bird's-eye view feature map; and aggregating the shallow spatial feature map and the high-level semantic feature map to obtain the final bird's-eye view feature map. [n0079]).
Regarding claim 6, Fu in the combination teaches the method of claim 4, wherein the positional and color components include Fourier features (We follow the guidance of prior works (Mildenhall et al., 2020; Sitzmann et al., 2020a; Tancik et al., 2020) and use Fourier features to improve the spatial resolution of our implicit representations. Page 7 1st paragraph) and wherein incorporating the positional and color components further comprises: incorporating the positional and color components of the image data (In addition to standard Fourier positional features, we show that adding Fourier color features allows the network to use high-frequency color information from the original image. Page 7 1st paragraph) into the plurality of feature maps using a multi-layer perceptron (MLP) (Our MLP is a small 3-layer ReLU (Glorot et al., 2011) network with dropout (Srivastava et al., 2014)(p = .1) and layer normalization (Ba et al., 2016). Page 7 2nd paragraph), wherein the Fourier features comprise an input into the MLP (
PNG
media_image1.png
192
1154
media_image1.png
Greyscale
. Page 7 2nd paragraph).
Qian in the combination further teaches incorporating the positional and color components of the image data (wherein the characteristic of shallow space characteristic pattern can be extracted is close to the input characteristic pattern, comprising more pixel point information, such as color, texture, edge and corner information, capable of capturing more details, the resolution is higher. Page 9 last paragraph) into the plurality of BEV feature maps (extracting the shallow spatial feature map and the high-level semantic feature map of the bird's-eye view feature map; and aggregating the shallow spatial feature map and the high-level semantic feature map to obtain the final bird's-eye view feature map. [n0079]).
Regarding claim 7, Fu in the combination teaches the method of claim 1, wherein generating the one or more saliency maps for the image data further comprises: generating the one or more saliency maps using a pre-trained Contrastive Language-Image Pre-training (CLIP) model (All backbones (DINO, DINOv2, ViT, ResNet-50, CLIP, and DeepLabV3) used to train FeatUp are frozen, pre-trained models obtained from the community. Page 27 1st paragraph).
Regarding claim 8, Qian in the combination teaches the method of claim 1, wherein projecting the plurality of upsampled feature maps onto the plurality of BEV feature maps further comprises: generating a BEV grid representing an environment (Step 2: Preprocess the stitched point cloud data, extract the point cloud data of the specified ROI (region of interest), voxelize the point cloud data along the xyz axis, and calculate the geometric mean of the points in each grid as the initial feature. [n0103]) surrounding the vehicle (Step 1.2: Project the left and right radars onto the vehicle coordinate system. [n0102]. It should be noted that the x-axis represents the vehicle's forward direction, the y-axis represents the left direction of the vehicle's forward direction, and the z-axis represents the vertical direction, i.e., a right-handed coordinate system. The region of interest (ROI) is {0≤x≤1,-w≤y≤w, -h1≤z≤h2}. [n0105]).
Regarding claim 9, Qian in the combination teaches the method of claim 1, further comprising operating an Advanced Driver Assistance System (ADAS) (In autonomous driving, the perception of the vehicle's surrounding environment is crucial. Compared to traditional vision sensors, LiDAR is more reliable in dealing with complex road conditions and scenarios, making it essential to use LiDAR for target detection. [n0002]) based on the refined object detection proposals (Two-stage detectors, based on the proposals generated by the one-stage network, use the proposals as input to the two-stage network to refine their positions, thereby obtaining higher regression classification accuracy. [n0003]).
Apparatus claims 10-18 are drawn to the apparatus corresponding to the method of using same as claimed in claims 1-9. Therefore apparatus claims 10-18 correspond to method claims 1-9, and are rejected for the same reasons of obviousness as used above.
Claims 19-20 are drawn to a non-transitory computer-readable storage medium having executable instructions stored for carrying out the method of claims 1-2. Therefore, claims 19-20 correspond to method claims 1-2, and are rejected for the same reasons of obviousness as used above.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to LEI ZHAO whose telephone number is (703)756-1922. The examiner can normally be reached Monday - Friday 8:00 am - 5:00 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, VU LE can be reached at (571)272-7332. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/LEI ZHAO/Examiner, Art Unit 2668
/VU LE/Supervisory Patent Examiner, Art Unit 2668
Prosecution Insights