DETAILED ACTION
This action is in response to Applicant’s response filed on 08/18/2026. The amendments to Claims 7, 19-20, 27, 34, 38, and 43, filed on 08/18/2026 are entered. Claims 1-6, 18, and 33 are canceled. Claims 7-17, 19-32, and 34-45 remain pending. The Declaration of Attribution Under 37 C.F.R § 1.130(A) filed on 08/18/2026 has been entered. This Action is made FINAL.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
The Declaration of Attribution Under 37 C.F.R § 1.130(A) filed on 08/18/2026 disqualifies the previously relied upon Chen et al. (“FocalFormer3D: Focusing on Hard Instance for 3D Object Detection”; hereinafter “Chen”) reference as prior art. Accordingly, the 35 U.S.C. 102 rejection relying on Chen et al., and previously set forth in the Office Action Mailed on 05/20/2026, is withdrawn.
Applicant has amended Claims 7, 19-20, 27, 34, 38, and 43. Applicant’s amendments to independent claims 7, 27, and 38 clarify that the recited masking includes removing the prior detected 3D objects to form the masked 3D scene representation, which was previously provided by Canceled Claims 18 and 33. Support is found in at least paragraphs [0023] and [0037] of the instant specification.
Applicant’s arguments received on 08/18/2026 focus on the prior art cited in the Office Action mailed on 05/20/2026, specifically Ambrus et al. (US 20240135721 A1) and Martinson et al. (US 20160180195 A1), allegedly failing to teach the amended independent claims. Applicant’s arguments have been fully considered but are not persuasive for the reasons found below.
Applicant argues that Ambrus merely recognizes objects and manipulates properties and magnitudes of those objects for use in training a downstream 3D object detection network, rather than removing previously detected 3D objects.
Examiner response: Although Ambrus generally describes manipulating properties and magnitudes of detected objects, its disclosed manipulation expressly includes removal. In particular, Ambrus teaches that a 3D object detector detects vehicles in input image 405 and then generates augmentation image 460 by “removing the detected vehicle objects” represented by labels 420b and 420c (Ambrus paragraphs [0057]-[0058]; FIG. 4). Thus, Applicant’s characterization of Ambrus as merely manipulating object properties does not account for Ambrus’s express removal of previously detected 3D objects.
Applicant further argues that, according to Ambrus paragraph [0056], Ambrus’s “image masks” merely refer to 3D object labels and therefore do not teach the claimed removal and formation of a masked 3D scene representation.
Examiner response: Applicant’s argument regarding the image masks of Ambrus paragraph [0056] has been considered. However, in view of the amended claim language reciting “removing” prior detected 3D objects, the rejection relies on Ambrus’s separate express teachings that detected vehicle objects are removed to generate augmentation image 460, which is subsequently provided to downstream 3D object detection network 480 (Ambrus paragraphs [0057]-[0059]; FIG. 4). The instant specification expressly describes masking prior detected 3D objects as including removing those objects from the 3D scene (Instant application specification paragraph [0023]). Thus, once the previously detected objects are removed from the scene representation, the resulting representation is a “masked 3D scene representation” within the meaning of the claim, regardless of whether Ambrus itself uses the term “masked” to describe that representation. Accordingly, the image masks described in Ambrus paragraph [0056] are not required to satisfy the claimed masking or removal, because Ambrus separately teaches removal of previously detected objects to form the resulting scene representation.
Applicant further argues that Ambrus does not teach or suggest removing previously detected 3D objects from the 3D scene representation to form a masked 3D scene representation and then detecting additional 3D objects from that masked 3D scene representation, and that the Martinson reference does not cure this deficiency.
Examiner response: Ambrus expressly teaches detecting 3D objects, removing selected previously detected objects to generate a resulting augmentation image, and then providing that resulting image to downstream 3D object detection network 480 for subsequent 3D object detection processing (Ambrus paragraphs [0057]-[0059]; FIG. 4; see also [0020]). Thus, Ambrus teaches the formation of a modified scene representation after removal of previously detected 3D objects, contrary to Applicant’s argument.
The subsequent detection of additional 3D objects from said masked 3D scene representation is further deemed obvious based on the incorporation of the teachings of Martinson, as seen in the amended rejection below. Specifically, Martinson teaches object recognition processing in which an image representation is filtered or pre-filtered before subsequent CNN classification (Martinson paragraphs [0128]-[0130], [0140]; FIGS. 14A-B, and 18). As set forth more fully in the rejection below, Ambrus and Martinson are relied upon collectively for processing the resulting object-removed scene representation in a subsequent detection stage to detect additional objects.
Accordingly, Applicant’s arguments do not overcome the rejection for the reasons discussed above and in the amended rejection set forth below. The rejection has been updated to address the amended language and to clarify the teachings relied upon while maintaining the same Ambrus and Martinson combination relied upon in the independent claims and the same combinations previously relied upon for the dependent claims.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 7-10, 13, 15-16, 22, 27-30, and 38-40 are rejected under 35 U.S.C. 103 as being obvious over Ambrus et al. (US 20240135721 A1; hereinafter “Ambrus”) in view of Martinson and Yalla (US 20160180195 A1; hereinafter “Martinson”).
Regarding claim 7,
Ambrus teaches: A method (Abstract “A method for improving 3D object detection…”; [0004]), comprising:
at a device ([0031] “The 3D object detection system 300 may be a component of a vehicle, a robotic device, or other device.”; see FIG. 1 and FIG. 3), training a three-dimensional (3D) object detector to detect 3D objects from a given 3D scene representation, wherein the training includes detecting 3D objects over at least two stages ([0004] “A method for improving 3D object detection via object-level augmentations is described. The method includes recognizing…an object in an image of a scene… The method also includes training a downstream task network based on a set of training data generated based on the set of properties”; FIG. 3: 3D Training Data Module 318; [0020] teaches that a first pass through a 3D object detection network identifies individual objects in a scene, after which the identified 3D objects may be decomposed from the scene and manipulated; see als [0057]-[0059] and FIG. 4, discussed below).), including:
in a first stage of the at least two stages, detecting, by the 3D object detector, 3D objects from a 3D scene representation ([0020] teaches that a first pass identifies individual objects in a scene using a 3D object detection network model; [0053] “…neural scene rendering for 3D object detection including recognizing, by the image recognition model 312, an object in an image of a scene captured”; [0057] further teaches by example that a 3D object detector detects three vehicles in input image 405);
in at least one subsequent stage of the at least two stages, removing prior detected 3D objects from the 3D scene representation to form a masked 3D scene representation ([0058] teaches that object level augmentation model 450 generates augmentation image 460 by removing detected vehicle objects represented by 3D labels 420b and 420c; [0059] teaches that the resulting augmentation image 460 is thereafter provided to 3D object detection network 480 for training; see FIG. 4. Under the broadest reasonable interpretation, removal of the previously detected objects forms the claimed masked 3D scene representation. This interpretation is consistent with paragraph [0023] of the specification, which expressly provides that masking prior detected 3D objects includes removing the prior detected 3D objects from the 3D scene representation.)
determining a loss based on the 3D objects detected
and updating the 3D object detector based on the loss (Ambrus teaches determining a loss associated with a downstream 3D object detection model based on manipulated 3D object representations and updating the training process based on the determined loss ([0026] “manipulate a random property of the object by a random magnitude at an object level to determine a set of properties and a set of magnitudes of an object manipulation that maximizes a loss function of the image recognition model,” and further teaches to “train a downstream task network based on a set of training data generated based on the set of properties and the set of magnitudes of the object manipulation, such that a loss function is minimized.”; [0084] “propagating, using the 3D object detection network, downstream gradients to the object level augmentation model.”). A person of ordinary skill in the art would understand Ambrus’s teachings of loss maximization/minimization and downstream gradient propagation to teach gradient-based optimization and parameter updating based on the determined loss.).
Ambrus therefore expressly teaches detecting 3D objects, removing previously detected 3D objects to form a resulting masked 3D scene representation, and providing the resulting representation to a 3D object detection network.
Ambrus fails to explicitly disclose: detecting, by the 3D object detector, additional 3D objects from the masked 3D scene representation and therefore further fails to explicitly disclose determining losses associated with object detections over multiple detection stages.
In a related art, Martinson teaches: creating a masked representation of a scene/object region (Abstract; [0015]; [0061]) as a step in the process of recognizing image objects ([0061]; See [0128]-[0130] and FIGs 14A-B, an exemplary flow chart method of recognizing image object, including “Create mask image by segmenting depth image 1406”.). Martinson further teaches pre-filtering the mask image based on earlier object classification processing and then using the pre-filtered mask image with a CNN classifier for subsequent object classification (Abstract; [0129]-[0130]; FIG. 18; [0140] “…generates an image mask that is then pre-filtered and sent to the CNN classifier node 1806 for classification using the deep convolutional neural network”). Thus, Martinson teaches performing subsequent object-recognition processing on a filtered scene representation following an earlier object-recognition stage, corresponding to the claimed subsequent-stage processing of the masked 3D scene representation.
Both inventions lie in the field of endeavor of image analysis with an aim at object detection using images of scenes.
Therefore, it would have been obvious to a person of ordinary skill in the art, prior to the effective filing date of the claimed invention, to have modified the teachings of Ambrus to incorporate Martinson’s staged filtering and subsequent object-recognition processing, such that Ambrus’s masked 3D scene representation is processed in a subsequent detection stage after the previously detected objects have been removed. Because the previously detected objects are no longer present in the representation, objects detected during the subsequent stage would be additional to those detected in the prior stage. Such a modification would have predictably focused subsequent object detection processing on the remaining scene information while reducing unnecessary processing, consistent with Martinson’s teachings and stated need to reduce classification errors without increasing processing time and computational difficulty (Martinson [0014]), and furthering Ambrus’s goal of solving difficult downstream detection cases and improving the robustness of the resulting 3D object detection model (Ambrus [0022]).
Regarding claim 27,
Ambrus teaches: A system, comprising: a non-transitory memory storage comprising instructions; and one or more processors in communication with the memory, wherein the one or more processors execute the instructions to train a three-dimensional (3D) object detector to detect 3D objects from a given 3D scene representation ([0005]-[0006]; FIG. 1 and FIG. 4).
The remaining limitations found in claim 27 equally mirror limitation(s) taught by Ambrus, in view of Martinson, in claim 7. For sake of brevity, refer to back to claim 7’s 103 rejection found above for corresponding teachings. Thus, claim 27 is rejected based on the prior art of Ambrus and Martinson in claim 7 of the present office action. Accordingly, the corresponding motivation to combine, found in claim 7 rejection above, is also applicable.
Regarding claim 38,
Ambros teaches: A non-transitory computer-readable media storing computer instructions which when executed by one or more processors of a device cause the device to train a three-dimensional (3D) object detector ([0005]-[0006]; FIG. 1 and FIG. 4).
The remaining limitations found in claim 38 equally mirror limitation(s) taught by Ambrus, in view of Martinson, in claim 7. For sake of brevity, refer back to claim 7’s 103 rejection above for corresponding teachings. Thus, claim 38 is rejected based on the prior art of Ambrus and Martinson in claim 7 of the present office action. Accordingly, the corresponding motivation to combine, found in claim 7 rejection above, is also applicable.
Regarding claim 8, 28, and 39,
Ambros and Martinson teach: the method of claim 7, the system of claim 27, and the non-transitory computer-readable media of claim 38.
Ambros further teaches: wherein the 3D scene representation is a heatmap (Ambros discloses “…generating M attention heatmaps per object” ([0065]). A person of ordinary skill in the art would understand the disclosed “M attention heatmaps per object” to constitute 3D scene representation because the heatmaps spatially encode regions corresponding to detected 3D objects for subsequent processing and downstream detector training.).
Regarding claims 9, 29, and 40
Ambros and Martinson teach: the method of claim 7, the system of claim 27, and the non-transitory computer-readable media of claim 38.
Ambros teaches: wherein the 3D scene representation is generated from a feature map (Ambros teaches a detection transformer block that “extracts a set of features from the RGB image 505 and an encoder-decoder (ED) transformer…that constructs attention maps between features in the set of extracted features…the ED transformer neural network 520 applies a self-attention mechanism…For example, the notion of object queries is used to retrieve output features for each of the detected objects, which are then used as input to the object reasoning block 530 and 3D reasoning block 560. More particularly, the ED transformer neural network 520 outputs a latent vector 525… for each detected object. Each latent vector 525 for the detected objects becomes an input to object reasoning block 530” ([0063]), then Ambros goes on to teach the output features corresponding to detected objects from the aforementioned object reasoning block 530 are put through an optimization process and pose estimation to “form a recovered 3D scene” ([0070]). Thus, under the broadest reasonable interpretation of the claim, Ambros teaches 3D scene representation is generated from a feature map when disclosing feature-map-based attention processing to generate a recovered 3D scene.
Examiner notes, art not relied upon for this claim, “BSH-Det3D: Improving 3D object Detection with BEV Shape Heatmap” by Yang et al., and used in subsequent claims (e.g. claim 11), further exemplifies that 3D scene representation being generated from at least one feature map is a well-known technique in the art (Yang teaches LiDar-based 3D object detection that focuses on enhancing special features (Yang, Abstract). See feature map used in Fig. 3, backbone network to extract features from raw points used in subsequent hybrid-attention based strategies to fuse features and generate 3D proposals, which is further refined for ground truth scene representation (p. 5732, FIG. 3).
Regarding claims 10 and 30,
Ambrus and Martinson teach the method of claim 9 and the system of claim 29.
Ambrus further teaches: the feature map is generated from at least one input that captures a 3D scene (Ambrus teaches the first (vision) sensor used may be a red-blue (RGB) camera for capturing 2D RGB images (see [0035]), the first sensor is used to capture 3D object information ([0036]), an input image may be an RGB image ([0056]), and the feature map discussed in claims 9, 29, and 40 (found above) is extracted from RGB images ([0063]). Thus, Ambrus teaches RGB images from a camera sensor used to capture 3D object information (i.e. information found in a 3D scene) and subsequent feature maps are derived from RGB image data derived from said camera sensor input data.).
Regarding claim 13,
Ambrus and Martinson teach the method of claim 10.
Ambrus further teaches: wherein the input includes an image captured by a camera (Ambrus teaches using images from a video, taken by a camera (e.g. first sensor to capture RGB image data), and inputting the image into the object detection model for further processing (see FIG. 3 and [0031]-[036]; [0028]).
Regarding claim 15,
Ambrus and Martinson teach the method of claim 7.
Ambrus and Martinson further teaches: wherein detecting the 3D objects includes detecting a location of the 3D objects (Ambrus teaches recovering partial 3D shapes of detected objects includes the 3D reasoning block 560, found on FIG. 3, which determines the pose estimation using coordinates of an object detected from image (([0066]-[0068]) “Because they encode 3D coordinates via RGB, visualizing each 3D color component in 3D space enables 3D reasoning block 560 to recover a partial 3D shape of the object in its normalized coordinate space (NOCS)…Given estimated object masks and NOCS maps, the 3D reasoning block 560 recovers three things that enable a pose estimation 565: (1) 2D coordinates of the object in the RGB image 505, (2) 3D points of the partial 3D object shape in the canonical frame of reference”). To a person of ordinary skill in the art, coordinates are a type of location metrics; Martinson [0065] “In some embodiments, the image processor 202 may extract blobs of objects depicted by the image. In some embodiments, the sensor data may include depth image data describing the position of the objects relative to a point of reference.”).
Regarding claim 16,
Ambrus and Martinson teach the method of claim 7.
Ambrus further teaches: wherein detecting the 3D objects includes detecting a point on the 3D objects (Ambrus teaches the 3D object detection model includes detecting LiDAR data points of objects using a second LiDAR sensor ([0051])).
Regarding claim 22,
Ambrus and Martinson teach the method of claim 7.
Ambrus further teaches: wherein an encoder of the 3D object detector detects the 3D objects over the at least two stages (Ambrus teaches detecting objects over at least two stages using an encoder-decoder transformer network that “constructs attention maps between features,” where “object queries is used to retrieve output features for each of the detected objects,” and further “transformer neural network 520 outputs a latent vector 525 … for each detected object,” where the latent vector for the detected objects becomes “an input to object reasoning block 530 and 3D reasoning block 560” ([0063]). Ambrus further teaches the architecture is suitable for “duplicate suppression” ([0063]). A person of ordinary skill in the art would understand these teachings to constitute encoder-based staged object detection and subsequent object reasoning processing).
Claim(s) 11-12, 14, 17, 19, 31-32, and 41-42 are rejected under 35 U.S.C. 103 as being obvious over Ambrus et al. (US 20240135721 A1; hereinafter “Ambrus”) in view of Martinson and Yalla (US 20160180195 A1; hereinafter “Martinson”), and in further view of Shen et al. (“BSH-Det3D: Improving 3D Object Detection with BEV Shape Heatmap”; hereinafter “Shen”).
Regarding claims 11,
Ambrus and Martinson teach the method of claim 10, including a feature map generated from an input that captures a 3D scene (see claim 10’s 103 rejection above).
Ambrus also teaches its object detection method includes LiDar sensors for capturing and processing image data (e.g. capturing LiDar data points and 3D auto labeling of data) processing (See Ambrus [0035], [0050]-[0051]).
However, Ambrus and Martinson fail to explicitly disclose the feature map combines feature maps generated from a plurality of inputs that capture the 3D scene.
In a related art, Shen teaches: LiDar-based 3D object detection that focuses on enhancing special features (Yang, Abstract). Shen further teaches a BEV-based method that encodes points into pseudo-images to extract features (see Shen p. 5730 right column, last paragraph, - p. 5731 left column, first line). Shen further discloses a set of feature maps is used during the feature extraction technique (See feature maps used in “Feature Extract BackBone” in top of Fig. 3), including a process where feature maps are merged by summation for further processing (p. 5733, left column, second paragraph). Shen’s backbone network to extract features from raw points derived from point cloud inputs and used in subsequent hybrid-attention based strategies to fuse features and generate 3D proposals, which is further refined for ground truth scene representation (p. 5732, FIG. 3; p. 5732 left column first paragraph). Under the broadest reasonable interpretation of the claim, examiner interprets Shen’s teaching of extracting a plurality of feature maps from a plurality of point clouds, based on 3D objects found in a scene, and further summation of feature maps (resulting in one feature map) to be used in subsequent object detection processing to constitute “the feature map combines feature maps generated from a plurality of inputs that capture the 3D scene”.
It would have been obvious to a person of ordinary skill in the art, prior to the effective filing date of the claimed invention, to have modified Ambrus and Martinson’s teachings of generating feature maps from an input that captures a 3D scene to incorporate the teachings of Shen’s multi-input, multi-feature map, and feature map summation techniques in order to provide a more robust object detection method that relies on an increased amount of image data for detecting objects, thereby increasing the accuracy of object detection by decreasing errors that may result from relying on only one input feature map. All referenced prior art references, and the present application, lie in the field of image analysis with an aim at object detection. While Ambrus and Martin’s previously modified feature map method relied on RGB image data, Ambrus does teach collecting LiDAR image data (e.g. points) using LiDAR sensors (See Ambrus [0035], [0050]-[0051]), and Shen’s 3D object detection method is based on LiDAR image data (see Shen Abstract), Shen further indicates LiDAR and RGB based methodologies for object detection may improve fundamental challenges in 3D object detection, stating “we demonstrate that shape deterioration is a fundamental challenge in 3D object detection. To tackle this, we present the novel BSH-Det3D exploiting the potential of BEV shapes to improve detection...In future work, we plan to integrate the BEV shape completion module with RGB images to further improve the performance” (p. 5736 right column, last paragraph – p. 5737, left column first paragraph).
Regarding claim 12,
Ambrus and Martinson teach the method of claim 10.
Ambrus further teaches: LiDAR sensors collect image data (e.g. points) ([0035]; [0050]-[0051]) and ground-truth point clouds and associated surface normals represent rendered objects used in subsequent training of a signed-distance field module ([0073]).
Ambrus and Martinson fail to disclose: wherein the input includes a lidar point cloud.
In a related art, Shen teaches: a method that includes at least one point cloud input being input into a pipeline for 3D object detection, including inputting the point cloud(s) into a module for generating a BEV shaped heatmap and a module for feature extraction (see p. 5732, FIG. 3; p. 5731 right column first paragraph; p. 5732 left column first paragraph).
It would have been obvious to a person of ordinary skill in the art to modify the teachings of Ambrus and Martinson to incorporate the LiDAR point cloud structured input of Shen in order provide geometric, shape, and scale information when processing images for object detection, thereby increasing the accuracy of the representation of an object found in a scene. Shen states “While 2D images have valuable information, 3D point clouds can provide more geometric, shape and scale information [8], significantly improving scene perception capability” (p. 5730, section 1, paragraph 1).
Regarding claim 14,
Ambrus and Martinson teach the method of claim 10.
Ambros further teaches: wherein the input includes ([0035]; [0050]-[0051]) and at least one image captured by a camera (see FIG. 3 and [0031]-[036]; [0028]).
Ambrus and Martinson fail to explicitly disclose: wherein the input includes a lidar point cloud.
In a related art, Shen teaches: 3D object detection using LiDar point clouds as input (Abstract; p. 5730 left column first paragraph; Fig. 3).
It would have been obvious to a person of ordinary skill in the art, prior to the effective filing date of the claimed invention, to have modified Ambrus and Martinson’s model of inputting LiDar point data and images captured by an image to incorporate 3D point cloud structures taught by Shen for inputting into a model for 3D detection in order to increase the robustness of information collected for processing and increase the accuracy of the 3D object detection method. As Shen explains, “While 2D images have valuable information, 3D point clouds can provide more geometric, shape and scale information [8], significantly improving scene perception capability” (Shen p. 5730 left column, section 1, first paragraph).
Regarding claim 17,
Ambrus and Martinson teach the method of claim 7.
Ambrus teaches generating a heatmap (refer back to claim 8; ([0065])) and using a LiDAR sensor to detect data points of objects ([0051]). Mapping the entirety of an object would necessitate the detection of all spatial points, including a center point.
However, Ambrus fails to explicitly disclose: wherein detecting the 3D objects includes detecting a center point on the 3D objects.
In a related art, Shen teaches: utilizing center coordinates during RPN proposal generation from BEV shape heatmaps (p. 5733 right column second paragraph titled “One-stage”), which are spatial information corresponding to detected 3D objects (see FIG. 3).
It would have been obvious to one of ordinary skill in the art, prior to the effective filing date of the claimed invention, to have applied the feature-associated BEV shape heatmaps and center-coordinate proposal generation of Shen to the 3D object detection teachings of Ambrus and Martinson because all references aim to improve object detection using feature derived scene representation, and the combination would predictably improve object localization and difficult object detection performance.
Regarding claim 19
Ambrus and Martinson teach the method of claim 7.
Ambrus further teaches: detecting 3D objects during “a first pass of the input data through a network…” (Ambrus [0020]), that “identifies individual objects in a scene using a 3D object detection network model” ([0020]), where the “identified 3D objects may then be decomposed from the scene and randomly manipulated” ([0020]). As seen with respect to claim 7, Ambrus teaches removing prior detected 3D objects from a 3D scene representation to form a masked 3D representation (refer back to claim 7 for further clarity; also see Ambrus [0057]-[0059], and FIG. 4). Ambrus further teaches generating “bounding boxes and image mask” ([0056]) and generating object-associated attention heatmaps for downstream 3D object detection processing. ([0065]; refer back to claim 8 for further explanation of Ambrus’ heatmap teachings).
Ambrus and Martinson fail to explicitly disclose progressively refining masked 3D scene representation utilized for subsequent stage 3D object detection processing in which additional and further 3D objects are detected from progressively masked scene representation, over the course of three stages.
In a related art, Shen teaches: generating BEV shape heatmaps from extracted scene features, adaptively associating features with the heatmaps and raw points, and utilizing a center-coordinate, bounding box, and classification confidence-based region proposal network (RPN) processing to localize 3D objects from progressively refined BEV heatmap scene representation (see FIG. 3; Abstract; p. 5731 right column “A. Model Overview” subsection through end of section on p. 5732; p. 5733 right column “One-Stage” subsection). A person of ordinary skill in the art would understand the disclosed BEV shape heatmaps to constitute feature map 3D scene representations utilized for subsequent stage 3D object detection processing.
It would have been obvious to a person of ordinary skill in the art, prior to the effective filing date of the claimed invention, to have incorporated the progressively refined BEV heatmap scene representations and RPN proposal of Shen into the object masking and downstream 3D object detection framework of Ambrus and Martinson because all reference aim to improve 3D object detection, while Ambrus and Shen both aim to improve localization using object feature map and/or heatmap representations. The combination of references would predictably improve difficult object localization tasks and downstream 3D object detection robustness and accuracy by enabling subsequent stages of the object detection processing to focus on progressively refined and relevant mask scene representations.
A person of ordinary skill in the art would understand the combined teachings of Ambrus, Martinson, and Shen to teach, as claimed, at least one subsequent stage includes at least:
a second stage in which 3D objects detected in the first stage are removed from the 3D scene representation to form a first masked 3D scene representation and in which the 3D object detector detects additional 3D objects from the first masked 3D scene representation;
and a third stage in which the 3D objects detected in the first stage and the additional 3D objects detected in the second stage are removed from the 3D scene representation to form a second masked 3D scene representation and in which the 3D object detector detects further 3D objects from the second masked 3D scene representation,
because the combined references disclose generating object associated masks and attention heatmaps corresponding to detected 3D objects and subsequently performing additional object localization from progressively refined BEV heatmap scene representation.
Regarding claim 31,
Ambrus and Martinson teach the system of claim 30.
The remaining limitations found in claim 31 mirror limitation(s) taught by Ambrus, in view of Martinson and Shen, in claim 14, except claim 31 does not require both a lidar point cloud and an image captured by a camera. For sake of brevity, refer back to claim 14’s 103 rejection above for corresponding teachings. Thus, claim 31 is rejected based on the prior art of Ambrus, Martinson, and Shen in claim 14 of the present office action. Accordingly, the corresponding motivation to combine, found in claim 14 rejection above, is also applicable.
Regarding claim 32 and 42,
Ambrus and Martinson teach the system of claim 27 and the non-transitory computer-readable media of claim 38.
The remaining limitation(s) found in claims 32 and 42 combine mirrored limitations found in claim 15 (location of the 3D objects), claim 16 (a point on the 3D objects), and claim 17 (a center point on the 3D objects) taught by Ambrus in view of Martinson and Shen. Accordingly, claims 32 and 42 are rejected based on the prior art corresponding to these respective claims and the motivation to combine is also applicable.
Regarding claim 41,
Ambrus and Martinson teach non-transitory computer-readable media of claim 40.
The remaining limitation(s) found in claim 41 combine mirrored limitations found in claim 10 (feature map is generated from at least one input that captures a 3D scene), claim 14 (wherein the input includes at least one of a lidar point cloud and an image captured by a camera) taught by Ambrus, Martinson, and Shen. Examiner notes claim 41 requires only either at least one of a lidar point cloud or an image captured by a camera as an input, while claim 14 required both. Therefore, the scope of the limitation found in claim 14 includes the scope of the corresponding limitation in claim 41. Accordingly, claim 41 is rejected based on the prior art taught in claims 10 and 14 and the motivation to combine is also applicable.
Claim(s) 20, 23, 26, 34-35, and 43-44 are rejected under 35 U.S.C. 103 as being obvious over Ambrus et al. (US 20240135721 A1; hereinafter “Ambrus”) in view of Martinson and Yalla (US 20160180195 A1; hereinafter “Martinson”), and in further view Koivisto et al. (US 20190258878 A1; hereinafter “Koivisto”).
Regarding claim 20, 34, and 43,
Ambrus and Martinson teach: the method of claim 7, the system of claim 27, and the non-transitory computer-readable media of claim 38, including removing prior detected 3D objects from the 3D scene representation (for clarity, refer back to the 103 rejections for claims 7, 27, and 38).
Ambrus further teaches: Training a downstream 3D object detection network such that “a loss function is minimized” ([0026]), and “propagating, using the 3D object detection network, downstream gradients to the object level augmentation model” ([0084]), which an ordinary person of skill in the art would understand teaches loss-based optimization and updating of the downstream 3D object detector.
Ambrus and Martinson fail to explicitly disclose: wherein removing the prior detected 3D objects from the 3D scene representation prevents a subsequent stage from applying a loss to those prior detected 3D objects.
In a related art, Koivisto teaches: refining subsequent stage processing-based object detections, corresponding prior detection confidence, and false-positive object identification ([0037]; [0080]; [0113]).
It would have been obvious to a person of ordinary skill in the art, prior to the effective filing date of the claimed invention, to apply the prior-detection-based refinement of Koivisto to the object removal and downstream detector optimization of Ambrus, previously modified by Martinson, so that previously detected objects are removed from subsequent stage detector processing, thereby preventing subsequent-stage detector loss from being applied to the previously detected objects and instead focusing on later stage loss processing of remaining objects. Doing so would focus more processing resources on hard to detect objects, thereby increasing the model’s accuracy. All references aim to improve object detection techniques.
Regarding claims 23, 35, and 44,
Ambrus and Martinson teach the method of claim 7, the system of claim 27, and the non-transitory computer-readable media of claim 28, including determining a loss based on the 3D objects detected over the at least two stages.
Ambrus further teaches: (FIG 4 and [0056] “As shown in FIG. 4, an input image 405 may be an RGB image including 3D object labels, such as 3D bounding boxes and image masks…the bounding boxes and image mask may be determined from ground truth.”).
While Ambrus and Martinson teach: determining a loss based on the 3D objects detected over the at least two stages (see claim 7, 27, and 38 rejections), Ambrus and Martinson fail to explicitly disclose: wherein the loss is determined between the 3D objects detected over the at least two stages.
In a related art, Koivisto teaches: refining subsequent stage object processing based on prior object detections and corresponding confidence determinations, and false-positive object identification ([0037]; [0080]; [0113]).
It would have been obvious to a person of ordinary skill in the art, prior to the effective filing date of the claimed invention, to have applied the prior object detection and confidence determination techniques taught by Koivisto to Ambrus and Martinson’s model for determining a loss based on 3D objects detected over the at least two stages in order predictably improve subsequent stage object detection accuracy and downstream detector optimization by refining later stage processing based on relationships between prior stage and subsequent stage object detections.
Regarding claim 26,
Ambrus, Martinson, and Koivisto teach the method of claim 23.
Ambrus further teaches: wherein a decoder of the 3D object detector determines the loss and updates the 3D object detector (Ambrus teaches an encoder-decoder (ED) transformer neural network that constructs attention maps between features, retrieves output features for detected objects using object queries, and outputs latent vectors for the detected objects that are provided to downstream object reasoning and 3D reasoning stages ([0063]). Shen further teaches minimizing a loss function and propagating downstream gradients during downstream 3D object detector training ([0026]; [0084). A person of ordinary skill in the art would understand these teachings to disclose encoder-decoder transformer processing applied during loss-based updating of the 3D object detector.).
Claim(s) 21 is rejected under 35 U.S.C. 103 as being obvious over Ambrus et al. (US 20240135721 A1; hereinafter “Ambrus”) in view of Martinson and Yalla (US 20160180195 A1; hereinafter “Martinson”), and in further view Brown et al. (US 20120170805 A1; hereinafter “Brown”).
Regarding claim 21,
Ambrus and Martinson teach the method of claim 7, including wherein training the 3D object detector includes detecting 3D objects over at least two stages.
Ambrus and Martinson fail to explicitly disclose: wherein training the 3D object detector further includes: accumulating 3D objects detected over the at least two stages.
In a related art, Brown teaches: object detection and generating progressively modified object representations (Abstract). During the processing, the modified object representation is used as training data for subsequent detector training stages ([0011]-[0014]; [0030]-[0033]). A person of ordinary skill in the art would understand the disclosed iterative processing to teach maintaining or accumulating and further utilizing previously detected object-associated information across multiple training stages during subsequent refinement.
It would have obvious to a person of ordinary skill in the art, prior to the effective filing date of the claimed invention, to have modified the downstream loss-based 3D object framework of Ambrus and Martinson to incorporate the iterative object occlusion and staged detector refinement teachings of Brown in order to improve difficult to identify object localization and downstream detector refinement by utilizing previously detected object information across multiple detection and training stages. All three inventions aim to improve object detector robustness using progressively modified object scene representations during detector training.
Claim(s) 24-25, 36-37, and 45 are rejected under 35 U.S.C. 103 as being obvious over Ambrus et al. (US 20240135721 A1; hereinafter “Ambrus”) in view of Martinson and Yalla (US 20160180195 A1; hereinafter “Martinson”), in further view Koivisto et al. (US 20190258878 A1; hereinafter “Koivisto”), and in further view of Shen et al. (“BSH-Det3D: Improving 3D Object Detection with BEV Shape Heatmap”; hereinafter “Shen”).
Regarding claims 24 and 36,
Ambrus, Martinson, and Koivisto teach the method of claim 23 and system of claim 35.
Ambrus, Martinson, and Koivisto fail to explicitly disclose: wherein the loss is a Gaussian focal loss.
In a related art, Shen teaches: Gausssian based BEV shape heatmap labels and focal loss-based BEV heatmap estimation (see Shen p. 5732 left column subsection “Generationo f ground truth labels and right column subsection “Estimation of BEV shape heatmap”; FIG. 4 on p. 5733), thereby teaching Gaussian focal loss-based supervision for 3D object detection.
It would have been obvious to a person of ordinary skill in the art, prior to the effective filing date of the claimed invention, to have applied the Gaussian focal loss 3D object detection teachings of Shen to the loss teachings of Ambrus, Martinson, and Koivisto in order to improve difficult object detection by emphasizing object associated heatmap areas while reducing the influence of background regions during detector training. All inventions aim to improve object detection.
Regarding claim 25, 37, and 45,
Ambrus, Martinson, and Koivisto teach he method of claim 23, system of claim 35, and the non-transitory computer-readable media of claim 44.
Ambrus further teaches: detecting 3D objects and generating corresponding “bounding boxes and image mask” associated with detected or ground truth labels ([0056]-[0057]).
Ambrus Martinson, and Koivisto fail to explicitly disclose: predicting bounding boxes for the 3D objects detected over the at least two stages;
wherein the loss is determined between the bounding boxes predicted for the 3D objects detected over the at least two stages and bounding boxes of the 3D objects labeled in the ground truth given for the 3D scene representation.
In a related art, Shen teaches: a “Two-stage” 3D object detection framework including stage-one proposal generation and stage-two proposal refinement, where “residuals between the 3D proposal and ground truth bounding box for two-stage results” are determined and “smooth-L1 loss is used to anchor box regression with the predicted residual and regression target” (see Shen p. 5733 right column second paragraph titled “Two-stage” through p. 5732 left column line 9). Shen further teaches focal loss-based anchor classification and “IoU-guided confidence prediction losses” (see same section from Shen).
It would have been obvious to one of ordinary skill in the art, prior to the effective filing date of the claimed invention, to have applied the two-stage proposal refinement and ground truth-based loss optimization of Shen to the 3D object detection framework of Ambrus, Martinson, and Koivisto in order to improve 3D localization and downstream detector refinement using progressively refined bounding box predictions and ground truth-based loss optimization over multiple detection stages.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SAMUEL DAVID BAYNES whose telephone number is (571)272-0607. The examiner can normally be reached Monday - Friday 8:00 am - 5:00 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Stephen R Koziol can be reached at (408)918-7630. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SDB/
Samuel Baynes
Examiner, Art Unit 2665
/Stephen R Koziol/Supervisory Patent Examiner, Art Unit 2665
