DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
The amendment filed August 19, 2026 has been entered. Claims 1 and 3–21 are pending. Claims 1, 3, and 4 have been amended, claim 2 has been canceled, and claim 21 has been added. Claims 7–20 remain withdrawn from consideration.
New claim 21 adds post-estimation noise reduction based on anchor voxels and neighboring extracted voxels. This added subject matter corresponds to nonelected Invention Group II. Claim 21 also incorporates all limitations of elected claim 1 and therefore presents a combination of the elected method and the additional noise-reduction operation. No additional restriction is imposed on that combination in this action, and claim 21 is examined on the merits. Examination of claim 21 does not reinstate withdrawn claims 7–20.
Response to Arguments
Applicant’s amendment and arguments, concerning the rejections under 35 U.S.C. §101, the rejection under 35 U.S.C. § 112(b), and the previous rejections under 35 U.S.C. § 102 have been considered. Accordingly, the previously stated rejections under 35 U.S.C. §101, 35 U.S.C. § 112(b) and 35 U.S.C. § 102 are withdrawn.
However, new grounds of rejection under 35 U.S.C. §103 are made in this Office action. Claims 1 and 3–6 are newly rejected under §103 over Wang in view of Luo. These substantive limitations narrow the claimed combination beyond the subject matter addressed by the previous action. Claim 21 is newly rejected under §103 over Wang in view of Luo and further in view of Hoy. Because Applicant’s substantive amendments necessitated the new grounds of rejection presented in this Office action, this action is properly made final in accordance with MPEP §706.07(a).
Based on these facts, this action is made FINAL.
Claim Objections
Claims 1 and 3 are objected to because of the following informalities:
In claim 1, the principal projection clause recites projecting world voxels “to image coordinates associated with the camera”, whereas the following “wherein projecting” clause refers to projecting the world voxels “to the camera”. The latter phrase should be made consistent with the expressly identified image-coordinate destination, for example by replacing that occurrence of “to the camera” with “to image coordinates associated with the camera”.
In claim 3, the “wherein projecting” clause likewise refers to projecting world voxels “to the camera”. This phrase should be made consistent with the corresponding projection limitation of claim 1, for example by replacing it with “to image coordinates associated with the camera”.
Appropriate correction is required.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1, and 3–6 are rejected under 35 U.S.C. §103 as being unpatentable over Wang (Wang et al, “PANOOCC: Unified Occupancy Representation for Camera-based 3D panoptic segmentation”. arXiv.org., 2023) in view of Luo (Luo et al (2022). DETR4D: Direct Multi-View 3D Object Detection with Sparse Attention (2212.07849v1). arXiv.).
Regarding claim 1, Wang teaches a computer-implemented method comprising:
receiving a first image captured by a camera of a vehicle at a first time t;
receiving a second image captured by the camera of the vehicle at a second time t-1;
( [Sec. 3.1-3.2], [Fig. 2]: Wang teaches taking current multi-view images
I
t
=
{
I
t
1
,
I
t
2
,
.
.
.
,
I
t
n
}
and previous frames
I
t
-
1
,
.
.
.
,
I
t
-
k
as input, where n denotes a camera-view index and k denotes the number of history frames; wherein the current multi-view images
I
t
correspond to the first image captured at the first time t, and the previous frames
I
t
-
1
correspond to the second image captured at the second time t−1, both captured by cameras of the ego-vehicle in the autonomous driving context. Fig. 2 depicts current frame T and previous frames including T−1. Wang also teaches use of the nuScenes autonomous-driving dataset, where each sample includes RGB images from six cameras. )
projecting each of a plurality of world voxels to the camera at the first time t and the second time t-1;
( [Sec. 3.3-3.4], [Eq. 1–3], [Eq. 6–7]: Wang teaches defining 3D-grid-shape voxel queries Q ∈
R
H
×
W
×
Z
×
D
, where each voxel query at position (i, j, k) is responsible for a corresponding 3D voxel-grid cell region, and each grid cell corresponds to a real-world size. Wang further teaches voxel cross-attention in which πn(Refm i,j,k) denotes a projected reference point in the n-th camera view, projected by projection matrix πn from the voxel grid located at (i, j, k). Wang also teaches the projection between a real 3D reference point in the ego-vehicle frame and a corresponding 2D reference point on the n-th image view using projection matrix Pn. Wang further teaches temporal voxel alignment between the current frame and previous frames using transformation matrix
T
t
→
t
-
k
, including previous frame t−1. )
projecting each of a plurality of world voxels to image coordinates associated with the camera at the first time t and the second time t-1,
( [Sec. 3.3-3.4], [Eq. 1–3], [Eq. 6–7]: Wang teaches defining 3D-grid-shape voxel queries Q ∈
R
H
×
W
×
Z
×
D
, where each voxel query at position (i, j, k) is responsible for a corresponding 3D voxel-grid cell region, and each grid cell corresponds to a real-world size. Wang further teaches voxel cross-attention in which πn(Refm i,j,k) denotes a projected reference point in the n-th camera view, projected by projection matrix πn from the voxel grid located at (i, j, k). Wang also teaches the projection between a real 3D reference point in the ego-vehicle frame and a corresponding 2D reference point on the n-th image view using projection matrix Pn. Wang further teaches temporal voxel alignment between the current frame and previous frames using transformation matrix
T
t
→
t
-
k
, including previous frame t−1. )
wherein projecting each of a plurality of world voxels to the camera at the first time t and the second time t-1 comprises:
extracting a first feature from the first image captured at the first time t;
( [Sec. 3.2], [Sec. 3.4], [Fig. 2], [Sec. 4.2]: PanoOcc takes multi-frame multi-view images as input and first extracts multi-scale features using an image backbone; wherein the image backbone (ResNet50, ResNet101-DCN, or InternImage) extracts multi-scale image features
F
from the current multi-view images
I
t
at the first time t. )
projecting a voxel grid definition in a local coordinate system at the first time t to the first image; and performing feature sampling for the first image based at least in part on results of the extracting and results of the projecting,
( [Sec. 3.3-3.4], [Eq. 1–3], [Fig. 2]: Wang teaches defining 3D-grid-shape voxel queries Q ∈
R
H
×
W
×
Z
×
D
, wherein the voxel queries are positioned at grid coordinates (i, j, k), the local coordinate system at the first time t. Wang further discloses the projection of this voxel grid definition to the first image via Eq. 2–3, where
P
n
is the projection matrix of the n-th camera. Feature sampling is then performed at the resulting projected 2D reference points using a voxel cross-attention mechanism via Eq. 1, using the projected reference point in the n-th camera view projected by projection matrix
π
n
from the voxel grid located at (i,j,k) and the extracted image features Fn. )
wherein performing the feature sampling comprises projecting a 3D location of each of the plurality of world voxels to a 2D coordinate location in the first image using extrinsic calibration and intrinsic calibration of the camera,
( [Secs. 3.1–3.4], [Eqs. (1)–(3)], [Fig. 2]: Wang teaches voxel queries corresponding to physical 3D voxel-grid cells, each having a real-world size. For each voxel query, Wang identifies a reference point having a physical three-dimensional location in the ego-vehicle coordinate frame and projects that location to corresponding two-dimensional coordinates in a camera image using the camera’s projection matrix. For the current-frame image, corresponding to the first image, Wang uses the projected reference points together with extracted image features in voxel cross-attention. Under the standard projection, the projection accounts for the camera’s rotation and translation relative to the physical three-dimensional coordinate frame, corresponding to extrinsic calibration, and the camera’s focal distance and principal point, corresponding to intrinsic calibration. )
and interpolating image features at locations obtained by adding offsets to the projected 2D coordinates, wherein the extrinsic calibration comprises rotation and translation with respect to the world coordinate system, and wherein intrinsic calibration comprises focal distance and principal point;
( [Secs. 3.3–3.4], [Eqs. (1)–(3)], [Fig. 2]: Wang teaches projecting physical 3D voxel reference points to corresponding 2D camera-image coordinates and using the projected reference points together with extracted image features in voxel cross-attention. Wang’s deformable-attention operation obtains interpolated image features at sampling locations determined by adding learned two-dimensional offsets to the projected reference points. The standard calibrated-camera of Wang’s projection matrix, the projection accounts for the camera’s rotation and translation relative to the physical three-dimensional coordinate frame of the voxel grid, corresponding to extrinsic calibration, and the camera’s focal distance and principal point, corresponding to intrinsic calibration. )
aggregating voxel features for the plurality of world voxels for the first image and the second image; and
( [Sec. 3.2], [Sec. 3.4], [Fig. 2]: Wang teaches using voxel queries to aggregate spatiotemporal information. Wang further teaches that an image backbone extracts multi-scale features, a View Encoder uses voxel queries to learn voxel features, and a Temporal Encoder aligns and fuses previous voxel features with the current frame. Wang also teaches that previous aligned voxel queries [world voxels for the second image at time t-1] are concatenated with current voxel queries [world voxels for the first image at time t] and fused to output fused-voxel queries Qf. )
training an occupancy classifier using the aggregated voxel features.
( [Sec. 3.1], [Sec. 3.5-3.6], [Sec. 4.2], [Eq. 8–9]: Wang teaches camera-based 3D semantic occupancy prediction for predicting semantic voxel volume and determining whether a voxel grid is empty or occupied. Wang further teaches converting fused voxel features Qf into occupancy features O, training the model end-to-end based on the unified occupancy representation, and using a lightweight MLP segmentation head based on occupancy features to query voxel-grid status at arbitrary positions. Wang also teaches supervising voxel prediction using segmentation losses during training. )
generating, using the occupancy classifier, a plurality of occupancy estimations for voxels of new images.
( [Secs. 3.1, 3.6, and 4.1–4.3], [Fig. 2], [Table 2]: Wang teaches a trained occupancy-prediction model having a multilayer-perceptron segmentation head that classifies voxel-grid locations using occupancy features. The model produces a semantic voxel volume distinguishing the empty class from occupied semantic classes, thereby generating a plurality of voxel occupancy estimations. Wang further describes separate training and validation datasets, including 700 training scenes and 150 validation scenes for Occ3D-nuScenes, and reports occupancy-prediction results on the validation set in Table 2. Accordingly, the trained model, including its segmentation head corresponding to the occupancy classifier, generates occupancy estimations for voxels represented by camera images from validation scenes distinct from the training scenes, corresponding to the claimed new images. )
Wang teaches projecting 3D voxel reference points to 2D image coordinates and using those projected reference points in deformable attention. As discussed above, the sampling locations in that operation are obtained by adding learned offsets (
p
+
Δ
p
) to the projected reference points. However, requires interpolating image features at the 2D coordinate (
p
) resulting from the recited projection of the voxel’s 3D location. Wang’s offset-based sampling (
p
+
Δ
p
) disclosure does not expressly establish interpolation at that particular projected coordinate (
p
). Luo teaches sampling image features directly at projected three-dimensional grid locations, as follows:
interpolating image features at the 2D coordinate location,
( Luo > [Sec. 3.2, p. 4], [Eq. (1)], [Wang, Sec. 3.4, Eqs. (2)–(3)]: Luo teaches extracting image features from input camera images, forming a three-dimensional sampling grid corresponding to the scene, and projecting the grid’s sampling locations into the images using the known camera transformation. Luo then obtains volumetric features by bilinearly sampling the image features at the projected grid locations and combining the results across valid camera views. Accordingly, Luo teaches interpolating image features at the 2D coordinate location obtained by projecting the corresponding 3D grid location. )
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to substitute Luo’s projected-grid bilinear sampling for Wang’s offset-based image-feature retrieval when constructing Wang’s voxel representation. This substitution would provide a nonparametric means of obtaining image features at geometrically determined grid projections, avoiding the need to learn additional image-plane sampling offsets for that retrieval step. Both methods obtain three-dimensional representations from camera-image features, and Luo’s sampler accepts projected grid locations and image-feature maps and produces volumetric features suitable for subsequent voxel processing. A skilled artisan therefore would have reasonably expected to adapt Luo’s sampler to Wang’s voxel grid and use the resulting features in Wang’s downstream feature aggregation and occupancy classification. The proposed modification substitutes a known feature-retrieval technique performing its established function, with the predictable result of obtaining image-derived voxel features at the projected grid locations.
Regarding claim 3, Wang [as modified by Luo] teaches the computer-implemented method of claim 1, wherein projecting each of a plurality of world voxels to the camera at the first time t and the second time t-1 further comprises:
extracting a second the feature from the second image captured at the second time t-1;
( Wang > [Sec. 3.2], [Sec. 3.4], [Fig. 2], [Sec. 4.2]: Wang takes current and previous camera images as inputs and extracts multi-scale image features using an image backbone. The same backbone discussed for claim 1 processes the previous-frame images, including the image captured at time t−1, thereby extracting the second feature from the second image. Wang subsequently uses the image-derived voxel features from previous frames in its temporal processing. )
transforming the voxel grid definition to the local coordinate system at the second time t-I to generate a transformed voxel grid definition:
( Wang > [Sec. 3.4, p.5: “Temporal Encoder”], [Eq. (6)], [Fig. 2]: Wang expressly teaches transforming the current-frame voxel grid into the coordinate frame of a previous time. Equation (6) applies a transformation matrix that maps points at the current time to a previous frame. For the immediately preceding frame, this transforms the voxel grid at time t into the same grid expressed in the coordinate frame at time t−1. Wang identifies the resulting grid as the current-frame grid represented at the previous frame. This corresponds to transforming the voxel grid definition into the previous local coordinate system to generate a transformed voxel grid definition. )
projecting the transformed voxel grid definition in the local coordinate system at the second time t-1 to the second image; and performing feature sampling for the second image based at least in part on results of the extracting and results of the projecting.
( Wang > [Sec. 3.4, Eqs. (2)–(3) and (6)], Luo > [Sec. 3.2, Eq. (1), Sec. 3.4, Eqs. (6)–(7)]: Wang supplies the physical voxel-grid coordinates, the transformation into the previous coordinate frame, and the camera projection operation. Luo teaches transforming a current object-query location to the previous timestamp to compensate for ego-motion and then querying the previous image features using that transformed location. Luo separately teaches projecting three-dimensional grid locations into camera images and bilinearly sampling the image features at the projected locations. In the Wang [as modified by Luo]'s system, Luo’s motion-compensated image-query approach and projected-grid sampler are applied across Wang’s transformed voxel grid: the transformed grid locations are projected into the second image, and features extracted from that image are sampled at the resulting image coordinates. The sampling therefore depends on both the extracted second-image features and the projection of the transformed grid. )
Regarding claim 4, Wang [as modified by Luo] teaches the computer-implemented method of claim 3, wherein aggregating voxel features comprises:
performing voxel feature aggregation based at least in part on results of the feature sampling for the first image and results of the feature sampling for the second image; and
( Wang > [Sec. 3.2], [Sec. 3.4-3.5], [Fig. 2]: Wang teaches using voxel queries to aggregate spatiotemporal information from multi-frame and multi-view images. Wang specifically teaches after aligning previous voxel queries to the current frame, concatenating previous aligned voxel queries with current voxel queries, and using residual 3D convolution to fuse the queries and output fused voxel queries Qf; wherein the fused voxel queries
Q
f
∈
R
H
×
W
×
Z
×
D
represent voxel feature aggregation performed based on the feature sampling results for the first image at time t (
Q
t
from VCA, Eq. 1) and the feature sampling results for the second image at time t−1 (
Q
t
-
1
→
t
from GridSample, Eq. 7). )
wherein training the occupancy classifier comprises generating an occupancy estimation network based at least in part on the voxel feature aggregation.
( Wang > [Sec. 3.2], [Sec. 3.5-3.6], [Fig. 2]: Wang teaches an Occupancy Encoder that outputs fused voxel features Qf, an Occupancy Decoder that converts the fused voxel feature Qf into fine-grained occupancy features O, and a Task Head including a segmentation head. Wang further teaches that the model is trained end-to-end based on the unified occupancy representation, and that the segmentation head is a lightweight MLP based on occupancy features O or Osparse that queries voxel-grid status at arbitrary positions. )
Regarding claim 5, Wang [as modified by Luo] teaches the computer-implemented method of claim 4, wherein the occupancy estimation network is generated using voxel grid features.
( Wang > [Sec. 3.3-3.6], [Sec. 4.1]: Wang teaches voxel queries arranged in a 3D grid shape
Q
∈
R
H
×
W
×
Z
×
D
, , with each voxel query corresponding to a 3D voxel-grid cell region. Wang further teaches that, given voxel queries Q and extracted image features F, the Occupancy Encoder outputs fused voxel features
Q
f
∈
R
H
×
W
×
Z
×
D
. Wang then teaches converting the fused voxel feature Qf into occupancy features O, and using an MLP segmentation head based on occupancy features to query voxel-grid status [corresponding to generating the occupancy-estimation network using voxel-grid features]. )
Regarding claim 6, Wang [as modified by Luo] teaches the computer-implemented method of claim 4, wherein the voxel feature aggregation is performed using at least one of an average, a weighted average, or a deformable attention.
( Wang > [Sec. 3.4], [Eq. 1], [Table 8]: Wang explicitly states "we draw inspiration from the querying paradigm in recent BEV-based methods and adopt efficient deformable attention for voxel cross-attention and voxel self-attention", directly naming deformable attention as the aggregation mechanism. Wang’s voxel cross-attention is formulated using
D
A
(
q
,
π
n
(
R
e
f
i
,
j
,
k
m
)
,
F
n
)
, where DA represents deformable attention applied between voxel queries, projected reference points, and image features. Wang further confirms at Table 8 ablation: "DA means deformable attention" thereby teaching voxel feature aggregation performed using at least one of an average, a weighted average, or a deformable attention.)
Claim 21 is rejected under 35 U.S.C. §103 as being unpatentable over Wang [as modified by Luo] in view of Hoy (Hoy et al, US 2023/0324919 A1, 2023).
Regarding claim 21, Wang [as modified by Luo] teaches the computer-implemented method of claim 1, further comprising:
Wang [as modified by Luo] teaches generating occupancy estimations for voxels of new images using the trained occupancy classifier. However, the cited teachings do not establish subsequently reducing noise in those occupancy estimations based on anchor voxels and extracted voxels within a threshold distance of the anchor voxels. Hoy teaches this feature:
subsequent to generating, using the occupancy classifier, the plurality of occupancy estimations for voxels of new images, performing noise reduction aggregation on the plurality of occupancy estimations to generate a noise reduced occupancy estimation, wherein the noise reduction aggregation is based on anchor voxels and extracted voxels within a threshold distance of the anchor voxels.
( Hoy > [0042–0045], [0048–0057], [0063], [Figs. 7–9]: Hoy selects high-confidence static reference voxels, corresponding to anchor voxels, within a threshold distance of uncertain occupied regions. The nearby uncertain occupied voxels are extracted relative to those static references using the same spatial-distance relationship. Hoy combines neighboring connection-score contributions and uses the resulting information to retain or clear occupancy. Paragraph [0063] specifically removes stale occupancy associated with a departed person while retaining the static shelf. This reference-based removal of erroneous occupancy is relied upon as noise reduction aggregation, producing a corrected occupancy map. )
It would have been obvious to a person of ordinary skill in the art, before the effective filing date of the claimed invention, to apply Hoy’s correction to Wang [as modified by Luo]’s method, to reduce falsely retained occupancy that can cause unnecessary waiting or detours ( Hoy > [0026]–[0029] ). The modification would retain Hoy’s observation history and reference categories and select nearby uncertain voxels from each static reference using Hoy’s distance criterion. Because both methods use spatial voxel occupancy representations, applying this known correction after classifier inference would predictably remove erroneous occupancy while preserving reliable static occupancy. The combination applies this correction after Wang [as modified by Luo]’s classifier, generates the new-image occupancy estimations.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KEN KUDO whose telephone number is (571)272-4498. The examiner can normally be reached M-F 8am - 5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Vincent Rudolph can be reached at 571-272-8243. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
KEN KUDO
Examiner
Art Unit 2671
/KEN KUDO/Examiner, Art Unit 2671
/VINCENT RUDOLPH/Supervisory Patent Examiner, Art Unit 2671
