Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement filed 29 August, 2024 fails to comply with 37 CFR 1.98(a)(2), which requires a legible copy of each cited foreign patent document; each non-patent literature publication or that portion which caused it to be listed; and all other information or that portion which caused it to be listed. It has been placed in the application file, but the information referred to therein has not been considered.
Specification
The disclosure is objected to because of the following informalities:
¶ 0044, 0047, 0048, 0051, and 0054 contain a character in formulas which appears to be an error in the printing of the specification:
PNG
media_image1.png
265
688
media_image1.png
Greyscale
Each instance of this character should be corrected as necessary to correct the recitation of the formulas throughout these paragraphs.
Appropriate correction is required.
Drawings
The drawings are objected to because:
Figure 4: 402A and 402C reference labels are misaligned and overlap with the areas of the image beneath them making it appear as if there are multiple blocks labelled 402.
PNG
media_image2.png
471
681
media_image2.png
Greyscale
Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. The figure or figure number of an amended drawing should not be labeled as “amended.” If a drawing figure is to be canceled, the appropriate figure must be removed from the replacement sheet, and where necessary, the remaining figures must be renumbered and appropriate changes made to the brief description of the several views of the drawings for consistency. Additional replacement sheets may be necessary to show the renumbering of the remaining figures. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Claim Objections
Claims 7 and 17 are objected to because of the following informalities:
Line 1 states “by performing local transformation within each…”, the examiner believes this should say “by performing a location transformation within each…”.
Line 2 – 3 states “results of the performing local transformation”, the examiner believes this should be revised to say something similar to “results of performing each of the local transformations” as the “performing local transformation” of line 1 is performed on each detected bounding box. This should be plural as well unless this claim intends to only refer to a single bounding box.
Claims 8 and 18 are objected to because of the following informalities:
Line 2 states “padding with zero in case that a number of points within a bounding box…”, the examiner believes this should be revised to say something similar to “padding the first set with zeros in the case that a number of points within a bounding box…”. As currently written this claim is grammatically confusing and needs revision.
Claims 9 and 19 are objected to because of the following informalities:
Line 2 states “the 3D decode”, this should be “the 3D decoder”.
Claim 15 is objected to because of the following informalities:
Claim 15 states “the apparatus of any claim 11…”, this should be rewritten to correctly indicate claim 15 depends from claim 11. Additionally, the claim dependency of claims 15 – 18 does not match that of 5 – 8, however the claims are identical in their limitations. If applicant intends for these to share the same chain of dependency this should be amended.
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 3, 5, 7, 8, 10, 13, 15, 17, 18 and 20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claims 3 and 13 recite the limitation "the one or more keypoints" in line 6. There is insufficient antecedent basis for these limitations in the claims.
Claims 5 and 15 recite the limitation "the at least three features" in line 1. There is insufficient antecedent basis for these limitations in the claims. Examiner believes this is intended to reference the “at least three feature sets” of claim 1 and 11.
Claims 7 and 17 recite the limitation "a first set" in line 1. There is insufficient antecedent basis for these limitations in the claims. Examiner is unsure if this is the same “first set” of claim 1 or intended to be a new and different “first set”.
Additionally, claim 17 states “within each detected bounding box”, however the step of detecting bounding boxes is performed in claim 16. Claim 17 depends from claim 15.
Claims 8 and 18 recite the limitation "the first set" in lines 1 and 2. Due to the antecedent basis rejection of claims 7 and 17, the examiner is unsure if this is the same “first set” of claim 1 or intended to be a new and different “first set” being claimed in claim 7 and 17.
Regarding claims 8 and 18 the term “extra points” in is a relative term which renders the claim indefinite. The term “extra points” is not defined by the claim, the specification does not provide a standard for ascertaining the requisite degree, and one of ordinary skill in the art would not be reasonably apprised of the scope of the invention.
The applicant’s specification filed 29 August, 2024 does not disclose what points are considered “extra points” nor does claim 8 or any claims from which it depends. As such it is not possible to establish the scope of how many points are to be removed from the first set of features in response to the number of points in a bounding box being below a threshold.
Claims 10 and 20 recite the limitation "for each bounding box" in line 1. There is insufficient antecedent basis for these limitations in the claims. The step of detecting one or more bounding boxes is performed in claim 6 and 16, however claim 10 depends from claim 5 and claim 20 depends from claim 15.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 11, and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Zanfir et al (Zanfir, Andrei, et al. "Hum3dil: Semi-supervised multi-modal 3d human pose estimation for autonomous driving." Conference on Robot Learning. PMLR, 2023., hereinafter “Zanfir”) in view of Wekel et al (U.S. Patent Publication No. 2021/0063578 A1, hereinafter “Wekel”).
Regarding claim 1, Zanfir teaches an image processing method, comprising:
estimating, by performing a two-stage analysis, a pose of a human object in a point cloud received from a light detection and ranging (lidar) sensor (Page 2, ¶ 2: We propose HUM3DIL, a light-weight 3D human joints prediction network, that leverages RGB information with LiDAR points, in a novel fashion, by computing pixel-aligned [12] multi-modal features with the 3D positions of the LiDAR signal.), wherein the two-stage analysis includes:
a first stage in which (Figure 1 including: Multi-modal features, Camera intrinsics, random Fourier embedded point cloud; Figure 1, Projected LiDAR depth; Page 3, Section 2.2 “Enriching LiDAR points with image evidence”: Thus, we simultaneously inform the convolutional layers of the regions of interest in the image, i.e. sparse locations on the silhouette of the person, and make depth information available from the start… depth map channel will have non zero values only for a person of interest.), wherein the (Figure 1, Random Fourier Embedding, Multi-modal features, Camera Intrinsics; Page 1, Section: “Enriching LiDAR Points with image evidence”: Thus, we simultaneously inform the convolutional layers of the regions of interest in the image, i.e. sparse locations on the silhouette of the person, and make depth information available from the start… depth map channel will have non zero values only for a person of interest.; Page 4, Figure 1: We encode LiDAR points P through a Random Fourier Embedding [35] to produce
P
~
… We first use an image feature extractor (U-Net) that will act on the concatenation of D and I. The projected LiDAR points will further read features from the produced map F… Each token, in the beginning, will have information relating to the image features, camera intrinsics and Random Fourier Embeddings.; Examiner’s note: Examiner is interpreting “local point information” to mean any data relating to position, and as such the Random Fourier Embedded point cloud information.); and
a second stage that generates one or more human pose keypoints for the human object in the point cloud by analyzing the at least three feature sets using an attention mechanism (Figure 1, Predicted Joints; Page 4, Figure 1: We read the final Nj tokens and regress the 3D joints through an MLP.; Page 4, Section “Transforming the LiDAR Points”: Similar to [9], we use a cascaded block of L Transformer encoder layers, and collect the predicted 3D keypoints
Y
~
from an MLP applied on the transformed joints tokens:
PNG
media_image3.png
121
475
media_image3.png
Greyscale
).
Zanfir does not explicitly teach in which at least three feature sets are generated from the point cloud by performing a three-dimensional (3D) object detection and a 3D semantic segmentation on the point cloud
However, Wekel does teach a first stage in which at least three feature sets are generated from the point cloud by performing a three-dimensional (3D) object detection and a 3D semantic segmentation on the point cloud (Figure 2A, 7; ¶ 0026: The DNN(s) may process the LiDAR data to compute outputs corresponding to instance segmentation masks, per-class semantic segmentations masks, and/or bounding shapes (e.g., two-dimensional (2D) range image bounding boxes). These outputs may be processed into 2D bounding boxes ( e.g., corresponding to the LiDAR range image) and/or three-dimensional (3D) bounding boxes ( e.g., corresponding to a LiDAR point cloud used to generate the LiDAR range image) and class labels for the detected objects.), wherein the at least three feature sets include a first set that includes local point information (¶ 0042: Once transferred or unprojected, the labels in the LiDAR point cloud may maintain their associated classification information, instance information, etc., but the location information may be updated to reflect their relative location in the LiDAR domain.).
Wekel is considered to be analogous art it pertains to point cloud object detection. Therefore, it would have been obvious to one of ordinary skill in the art to combine the system for semi-supervised multi-modal 3D human pose estimation for autonomous driving (as taught by Zanfir) and the object detection and classification system (as taught by Wekel) before the effective filing date of the claimed invention. The motivation for this combination of references would be the system of Wekel utilizes increases training data by creating a series of virtual LiDAR sensors with varying virtual sensory fields. This results in a reduction of time and resources to generate a large enough training set for the DNN by repurposing existing ground truth data using the virtual LiDAR sensors. (See ¶ 0005).
This motivation for the combination of Zanfir and Wekel is supported by KSR exemplary rationale (G) Some teaching, suggestion, or motivation in the prior art that would have led one of ordinary skill to modify the prior art reference or to combine prior art reference teachings to arrive at the claimed invention. MPEP 2141 (III).
Regarding claim 11, claim 11 has been analyzed with regard to claim 1 and is rejected for the same reasons of obviousness as used above as well as in accordance with Zanfir’s further teaching on:
An apparatus for image processing, comprising one or more processors, wherein the one or more processors are configured to perform a method comprising (Page 6, ¶ 1: All of our networks were trained on a single V100 GPU with 16GB of memory.)…:
Regarding claim 16, the Zanfir and Wekel combination teaches the apparatus of claim 11.
Additionally, Wekel teaches wherein the performing the 3D object detection and the 3D semantic segmentation comprises detecting one or more bounding boxes for human objects in the point cloud (¶ 0024: For example, LiDAR range image DNN based processing may be executed in the form of a combined point cloud segmentation and bounding box regression network (PCSNet).; ¶ 0026: As such, in a non-limiting example, the DNN(s) may be used to predict one or more bounding boxes for each detected object on the road or sidewalk…; ¶ 0035: For example, with reference to FIG. 2A, image labels 106 (e.g., annotations) corresponding to an image 104Amay include bounding shapes 202A and 202B corresponding to car 204 and pedestrian 206, respectively. (emphasis added)).
Wekel is considered to be analogous art as it pertains to point cloud object detection. Therefore, it would have been obvious to one of ordinary skill in the art to combine the system for semi-supervised multi-modal 3D human pose estimation for autonomous driving (as taught by Zanfir) and the object detection and classification system (as taught by Wekel) before the effective filing date of the claimed invention. The motivation for this combination of references would be the system of Wekel utilizes spatial and temporal noise reduction to compensate for motion error in the capturing of data. (See ¶ 0149).
Claims 2 and 12 are rejected under 35 U.S.C. 103 as being unpatentable over Zanfir et al (Zanfir, Andrei, et al. "Hum3dil: Semi-supervised multi-modal 3d human pose estimation for autonomous driving." Conference on Robot Learning. PMLR, 2023., hereinafter “Zanfir”) in view of Wekel et al (U.S. Patent Publication No. 2021/0063578 A1, hereinafter “Wekel”) and further in view of Shi et al (Shi, Shaoshuai, et al. "From points to parts: 3d object detection from point cloud with part-aware and part-aggregation network." IEEE transactions on pattern analysis and machine intelligence 43.8 (2020): 2647-2664, hereinafter “Shi”).
Regarding claim 2, the Zanfir and Wekel combination teaches the method of claim 1.
Zanfir does not explicitly teach wherein the at least three feature sets include a second set that includes semantic voxel-wise point features and a third set that includes box-wise features.
However, Shi does teach wherein the at least three feature sets include a second set that includes semantic voxel-wise point features (Page 4, Col. 1, Section 3: The predicted 3D intra-object part locations and point-wise 3D features within each 3D proposal are then aggregated in the second stage to score the boxes and refine their locations.; Page 4, Col. 2, Section 3.1.1: Specifically, we voxelize the 3D space into regular voxels and extract the voxel-wise features of each non-empty voxel by stacking sparse convolutions and sparse deconvolutions, where the initial feature of each voxel is simply calculated as the mean values of point coordinates within each voxel in the LiDAR coordinate system.) and a third set that includes box-wise features (Page 4, Col. 1, Section 3: The relative locations of foreground points provide strong cues for box scoring and localization. We name the relative locations of the 3D foreground points w.r.t. to their corresponding boxes the intra-object part locations.).
Shi is considered to be analogous art as it pertains to point cloud object detection. Therefore, it would have been obvious to one of ordinary skill in the art to combine the system for semi-supervised multi-modal 3D human pose estimation for autonomous driving as taught by Zanfir) and the system for 3D object detection from point clouds (as taught by Shi) before the effective filing date of the claimed invention. The motivation for this combination of references would be the system of Shi utilizes a canonical coordinate system which eliminates rotation and location variations of different 3D proposals and improves the efficiency of feature learning for later box location refinement (See Page 7, Col. 2, ¶ 1).
This motivation for the combination of Zanfir, Wekel, and Shi is supported by KSR exemplary rationale (G) Some teaching, suggestion, or motivation in the prior art that would have led one of ordinary skill to modify the prior art reference or to combine prior art reference teachings to arrive at the claimed invention. MPEP 2141 (III).
Regarding claim 12, claim 12 has been analyzed with regard to respective claim 2 and is rejected for the same reasons of obviousness as used above.
Claims 3 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Zanfir et al (Zanfir, Andrei, et al. "Hum3dil: Semi-supervised multi-modal 3d human pose estimation for autonomous driving." Conference on Robot Learning. PMLR, 2023., hereinafter “Zanfir”) in view of Wekel et al (U.S. Patent Publication No. 2021/0063578 A1, hereinafter “Wekel”) and further in view of Shi et al (Shi, Shaoshuai, et al. "From points to parts: 3d object detection from point cloud with part-aware and part-aggregation network." IEEE transactions on pattern analysis and machine intelligence 43.8 (2020): 2647-2664, hereinafter “Shi”) and Ni et al (U.S. Patent Publication No. 2022/0101603 A1, hereinafter “Ni”) and Guo et al (U.S. Patent Publication No. 2026/0045091 A1, hereinafter “Guo”).
Regarding claim 3, the Zanfir, Wekel, and Shi combination teaches the method of claim 2.
Additionally, Zanfir teaches wherein the one or more human keypoints are generated by:
generating a fused feature set by combining a compressed feature set that is generated from the third set with the first set and the second set (Figure 1, including the Random Fourier Embedded point cloud, camera intrinsics, and multi-modal features being input into the Transformer Encoder as a single input matrix; Page 4, Section “Transforming the LiDAR points”: The complete input sequence is
M
ϵ
R
p
N
m
a
x
x
D
.
This sequence is at first linearly embedded by using a learnable matrix
E
ϵ
R
D
x
D
0
.);
learning internal keypoints in the point cloud by applying a keypoint transformer to the fused feature set and a learnable 3D keypoint query (Figure 1, including
R
N
j
x
D
0
input into Transformer Encoder; Page 4, Section “Transforming the LiDAR Points”: We additionally concatenate learnable joints tokens…);
Zanfir does not explicitly teach determining keypoint offsets for the one or more keypoints along X, Y and Z axes based on the learning the internal keypoints; determining 3D keypoint visibilities of the one or more keypoints along the Y axis; and estimating the pose of the human object based on the keypoint offsets and the 3D keypoint visibilities.
However, Ni teaches determining keypoint offsets for the one or more keypoints along X, Y and Z axes based on the learning the internal keypoints (¶ 0073: In step 530, the processor 210 estimates the joint coordinates. In certain embodiments, the joint coordinates can be estimated using a volumetric based pose estimator such as a 3D Convolutional Neural Net (CNN).; ¶ 0077: In step 550, processor 210 optimizes the joint locations. For example, the estimated parametric body model parameters are then used as the initialization of the optimization based post processing. The post processing fine tunes the predicted parameters ( of step 540) to minimize the modulated joint distance., including estimating the joint coordinates and optimizing the joint locations.);
estimating the pose of the human object based on the keypoint offsets (¶ 0077: In step 550, processor 210 optimizes the joint locations. For example, the estimated parametric body model parameters are then used as the initialization of the optimization based post processing. The post processing fine tunes the predicted parameters ( of step 540) to minimize the modulated joint distance., including estimating the joint coordinates and optimizing the joint locations.)
Ni is considered to be analogous art as it pertains to human pose estimation using input 3D data. Therefore, it would have been obvious to one of ordinary skill in the art to combine the system for semi-supervised multi-modal 3D human pose estimation for autonomous driving (as taught by Zanfir) and the body shape and pose estimation system (as taught by Ni) before the effective filing date of the claimed invention. The motivation for this combination of references would be the system of Ni unifies different 3D scan inputs to a voxel grid of a fixed size, this can reduce the size of a 3D scan to a predefined resolution thus reducing the complexity and processing requirements of the system (See ¶ 0072).
Additionally, Guo teaches determining 3D keypoint visibilities of the one or more keypoints along the Y axis (¶ 0161: The landmark visibility module 370 produces a visibility tensor 375 of shape (h1, w1, 28), representing visibility confidence scores (e.g., probabilities) for each of the 14 predicted posture points.); and
estimating the pose of the human object based on the keypoint offsets and the 3D keypoint visibilities (¶ 0164: The output of the landmark visibility module 370 is used in conjunction with the outputs of the landmark module 360 to inform downstream components such as tracking, pose smoothing, and behavior recognition. By providing visibility information, the module enables tracking systems to intelligently compensate for temporarily occluded keypoints using motion models or prior frame data, thereby improving the overall robustness and continuity of human pose estimation under real-world conditions.).
Guo is considered to be analogous art as it pertains to human pose estimation. Therefore, it would have been obvious to one of ordinary skill in the art to combine the system for semi-supervised multi-modal 3D human pose estimation for autonomous driving (as taught by Zanfir) and the human subject tracking (as taught by Guo) before the effective filing date of the claimed invention. The motivation for this combination of references would be the system of Guo is able to compensate for occluded keypoints thereby improving the overall robustness and continuity of human pose estimation. (See ¶ 0164).
This motivation for the combination of Zanfir, Wekel, Shi, Ni, and Guo is supported by KSR exemplary rationale (G) Some teaching, suggestion, or motivation in the prior art that would have led one of ordinary skill to modify the prior art reference or to combine prior art reference teachings to arrive at the claimed invention. MPEP 2141 (III).
Regarding claim 13, claim 13 has been analyzed with regard to respective claim 3 and is rejected for the same reasons of obviousness as used above.
Claims 5, 6, and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Zanfir et al (Zanfir, Andrei, et al. "Hum3dil: Semi-supervised multi-modal 3d human pose estimation for autonomous driving." Conference on Robot Learning. PMLR, 2023., hereinafter “Zanfir”) in view of Wekel et al (U.S. Patent Publication No. 2021/0063578 A1, hereinafter “Wekel”) and further in view of Ni et al (U.S. Patent Publication No. 2022/0101603 A1, hereinafter “Ni”)
Regarding claim 5, the Zanfir and Wekel combination teaches the method of claim 1.
Zanfir does not explicitly teach wherein the at least three features are generated by the first stage by processing the point cloud through a 3D encoder followed by a global context pooling module followed by a 3D decoder.
However, Ni does teach wherein the at least three features are generated by the first stage by processing the point cloud through a 3D encoder followed by a global context pooling module followed by a 3D decoder (¶ 0093: As illustrated, the volumetric based pose estimator 530a includes several residually connected 3D convolutional layers with one 3D pooling layer in between, followed by an encoder-decoder built by residual 3D CNN along with 3D max-pooling and deconvolutional up sampling layers. (emphasis added)).
Ni is considered to be analogous art as it pertains to human pose estimation using input 3D data. Therefore, it would have been obvious to one of ordinary skill in the art to combine the system for semi-supervised multi-modal 3D human pose estimation for autonomous driving (as taught by Zanfir) and the body shape and pose estimation system (as taught by Ni) before the effective filing date of the claimed invention. The motivation for this combination of references would be the system of Ni unifies different 3D scan inputs to a voxel grid of a fixed size, this can reduce the size of a 3D scan to a predefined resolution thus reducing the complexity and processing requirements of the system (See ¶ 0072).
This motivation for the combination of Zanfir, Wekel, and Ni is supported by KSR exemplary rationale (G) Some teaching, suggestion, or motivation in the prior art that would have led one of ordinary skill to modify the prior art reference or to combine prior art reference teachings to arrive at the claimed invention. MPEP 2141 (III).
Regarding claim 6, the Zanfir, Wekel, and Ni combination teaches the method of claim 5.
Additionally, Wekel teaches wherein the performing the 3D object detection and the 3D semantic segmentation comprises detecting one or more bounding boxes for human objects in the point cloud (¶ 0024: For example, LiDAR range image DNN based processing may be executed in the form of a combined point cloud segmentation and bounding box regression network (PCSNet).; ¶ 0026: As such, in a non-limiting example, the DNN(s) may be used to predict one or more bounding boxes for each detected object on the road or sidewalk…; ¶ 0035: For example, with reference to FIG. 2A, image labels 106 (e.g., annotations) corresponding to an image 104A may include bounding shapes 202A and 202B corresponding to car 204 and pedestrian 206, respectively. (emphasis added)).
Wekel is considered to be analogous art as it pertains to point cloud object detection. Therefore, it would have been obvious to one of ordinary skill in the art to combine the system for semi-supervised multi-modal 3D human pose estimation for autonomous driving (as taught by Zanfir) and the object detection and classification system (as taught by Wekel) before the effective filing date of the claimed invention. The motivation for this combination of references would be the system of Wekel utilizes spatial and temporal noise reduction to compensate for motion error in the capturing of data. (See ¶ 0149).
Regarding claim 15, claim 15 has been analyzed with regard to respective claim 5 and is rejected for the same reasons of obviousness as used above.
Claims 7, 9, 10, 17, 19, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Zanfir et al (Zanfir, Andrei, et al. "Hum3dil: Semi-supervised multi-modal 3d human pose estimation for autonomous driving." Conference on Robot Learning. PMLR, 2023., hereinafter “Zanfir”) in view of Wekel et al (U.S. Patent Publication No. 2021/0063578 A1, hereinafter “Wekel”) and further in view of Ni et al (U.S. patent publication No. 2022/0101603 A1, hereinafter “Ni”) and Shi et al (Shi, Shaoshuai, et al. "From points to parts: 3d object detection from point cloud with part-aware and part-aggregation network." IEEE transactions on pattern analysis and machine intelligence 43.8 (2020): 2647-2664, hereinafter “Shi”).
Regarding claim 7, the Zanfir, Wekel, and Ni combination teaches the method of claim 6.
Zanfir does not explicitly teach wherein a first set is generated by performing local transformation within each detected bounding box in the point cloud and concatenating results of the performing local transformation with corresponding original features.
However, Shi does teach wherein a first set is generated by performing local transformation within each detected bounding box in the point cloud and concatenating results of the performing local transformation with corresponding original features (Figure 4; Page 5, Col. 1, Section: Formulation of intra-object part location: As shown in Fig. 4, we formulate intra-object part location of each foreground point as its relative location in the 3D ground-truth bounding box that it belongs to. We denote three continuous values (x(part), y(part), z(part)) as the target intra-object part location of the foreground point (x(p), y(p), z(p)), which can be calculated as follows
PNG
media_image4.png
117
373
media_image4.png
Greyscale
where (x(c), y(c), z(c)) is the box center, (h, w, l) is the box size (height, width, length) and θ is the box orientation in bird-view; Examiner’s note: In view of the 112(b) rejection of claim 7, examiner is interpreting “a first set” to be the same first set of features of claim 1 which includes local point information.).
Shi is considered to be analogous art as it pertains to point cloud object detection. Therefore, it would have been obvious to one of ordinary skill in the art to combine the system for semi-supervised multi-modal 3D human pose estimation for autonomous driving (as taught by Zanfir) and the system for 3D object detection from point clouds (as taught by Shi) before the effective filing date of the claimed invention. The motivation for this combination of references would be the system of Shi utilizes a canonical coordinate system which eliminates rotation and location variations of different 3D proposals and improves the efficiency of feature learning for later box location refinement (See Page 7, Col. 2, ¶ 1).
This motivation for the combination of Zanfir, Wekel, Ni, and Shi is supported by KSR exemplary rationale (G) Some teaching, suggestion, or motivation in the prior art that would have led one of ordinary skill to modify the prior art reference or to combine prior art reference teachings to arrive at the claimed invention. MPEP 2141 (III).
Regarding claim 9, the Zanfir, Wekel, and Ni combination teaches the method of claim 5.
Additionally, Shi teaches wherein a second set is generated by gathering 3D sparse features from an output of the 3D decode based on corresponding voxelization indexes (Figure 2: SparseConv Encoder, SparseConv Decoder, SparseConv 3x3x3 within Voxel-Wise Part Feature Fusion; Page 4, Section 3.1.1: we propose to utilize an encoder-decoder network with sparse convolution and deconvolution [30], [31] to learn discriminative point-wise features for foreground point segmentation and intra-object part location estimation… Our sparse convolution based backbone is designed based on the encoder-decoder architecture. The spatial resolution of input feature volumes is 8 times downsampled by a series of sparse convolution layers with stride 2, and is then gradually upsampled to the original resolution by the sparse deconvolutions for the voxel-wise feature learning.).
Zanfir and Shi are considered to be analogous art as both pertain to point cloud object detection. Therefore, it would have been obvious to one of ordinary skill in the art to combine the system for semi-supervised multi-modal 3D human pose estimation for autonomous driving (as taught by Zanfir) and the system for 3D object detection from point clouds (as taught by Shi) before the effective filing date of the claimed invention. The motivation for this combination of references would be the system of Shi utilizes a canonical coordinate system which eliminates rotation and location variations of different 3D proposals and improves the efficiency of feature learning for later box location refinement (See Page 7, Col. 2, ¶ 1).
Regarding claim 10, the Zanfir, Wekel, and Ni combination teaches the method of claim 5.
Additionally, Shi teaches wherein a third set is generated by selecting, for each bounding box, a bird’s eye view (BEV) feature at a center thereof and centers of edges in a 2D BEV feature map thereof as box features (Figure 5, object center, foreground points; Page 6, Col. 1, ¶ 1: As shown in Fig. 5, we split the surrounding bird-view area of each foreground point into a series of discrete bins along the X and Y axes by dividing a search range S of each axis into bins of uniform length δ, which represents different object centers (x, y) on the X-Y plane…
PNG
media_image5.png
122
376
media_image5.png
Greyscale
S is the search range, (x(p); y(p); z(p)) is the coordinates of a foreground point of interest, (x(c); y(c); z(c)) is the center coordinates of its corresponding object…)
Zanfir and Shi are considered to be analogous art as both pertain to point cloud object detection. Therefore, it would have been obvious to one of ordinary skill in the art to combine the system for semi-supervised multi-modal 3D human pose estimation for autonomous driving (as taught by Zanfir) and the system for 3D object detection from point clouds (as taught by Shi) before the effective filing date of the claimed invention. The motivation for this combination of references would be the system of Shi utilizes bin-based classification with cross entropy loss for the X and Y axes, resulting in more accurate and robust center localization (See Page 6, Col. 1, ¶ 1).
Regarding claim 17, claim 17 has been analyzed with regard to respective claim 7 and is rejected for the same reasons of obviousness as used above.
Regarding claim 19, claim 19 has been analyzed with regard to respective claim 9 and is rejected for the same reasons of obviousness as used above.
Regarding claim 20, claim 20 has been analyzed with regard to respective claim 10 and is rejected for the same reasons of obviousness as used above.
Allowable Subject Matter
Claims 4 and 14 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Regarding claims 4 and 14 neither Zanfir, alone or in combination with other prior art discloses wherein the compressed feature set is generated by compressing a dimension of the third set using a multilayer perceptron.
Zanfir teaches inputting the third feature set into a multilayer perceptron, but there is no disclosure of compression on the third set of features by the multilayer perceptron.
Claims 8 and 18 are potentially allowable and would be allowable over prior art if the 35 U.S.C. 112(b) rejection on claims 8 and 18 is overcome.
Regarding claims 8 and 18, Zanfir teaches further including, removing extra points from the first set by randomly shuffling points in the first set and padding with zero in case that a number of points (Page 4, Section “Transforming the LiDAR points”; We shuffle and trim excess points, and pad with zeros if we have a fewer number of points.).
The distinction between Zanfir and the instant application claim 8 is that Zanfir performs this reduction and padding on the token input into the Transformer Encoder which comprises the first, second, and third feature set. The threshold is based on “a fewer number of points” however, not specifically a threshold number of points within a bounding box.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ANDREW JONES whose telephone number is (703)756-4573. The examiner can normally be reached Monday - Friday 8:00-5:00 EST, off Every Other Friday.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matthew Bella can be reached at (571) 272-7778. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ANDREW B. JONES/Examiner, Art Unit 2667
/MATTHEW C BELLA/Supervisory Patent Examiner, Art Unit 2667
Prosecution Insights