Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 10/02/2024 and 1/06/2026 are in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 1-4, 9, 11-15, and 19 is/are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Schramm et al. (US 2025/0277665; hereinafter Schramm).
Regarding claim 1:
Schramm discloses an autonomy computing system of an autonomous vehicle for object detection by the autonomous vehicle in adverse environmental conditions (see Figs. 1 and 5), the autonomy computing system comprising at least one processor in communication with at least one memory device (see Fig. 1; controller 106 and memory 160), and the at least one processor programmed to:
receive sensor data of an environment in which the autonomous vehicle is operating, the sensor data detected from one or more sensors of a plurality of modalities, the plurality of modalities including a first modality and a second modality, the sensor data including first sensor data from one or more sensors of the first modality and second sensor data from one or more sensors of the second modality, the second sensor data being in a bird’s eye view (BEV) (see Fig. 5 and paragraph [0031]);
extract first features in the environment based on the first sensor data and second features in the environment based on the second sensor data (see Fig. 1; camera 104 is a first sensor, and radar system 102 is a second sensor; also see Fig. 5; output of the encoder 502 is first features, and output of the encoder 504 is second features);
fuse, in the BEV, the first features and the second features into first enriched features of the first modality and second enriched features of the second modality by:
representing the first features in the BEV to derive first BEV features, based on depth information of the first features (see Fig. 5 and paragraph [0071]; “Query initializer 506 may lift the image features output by encoder 502 into BEV space to generate values for the attention mechanism that utilizes radar to query from these lifted image features”);
fusing the first features with the second features corresponding to the first BEV features to derive the first enriched features (see Fig. 5, the output of the lifter 510 is a first enriched features; also see Fig. 7 and paragraph [0083]); and
fusing the second features with the first features corresponding to the second features to derive the second enriched features (see Fig. 5; the output of radar query 508 is a second enriched features; also see paragraphs [0054]-[0055]);
detect object proposals based on the first enriched features and the second enriched features (see Fig. 5; output of 516 and paragraph [0091]);
predict objects in the environment based on the object proposals (see paragraph [0062]; “By fusing the radar and camera information in a shared BEV feature space, architecture 500 may make dense semantic predictions on other vehicles and the map surrounding the ego vehicle”); and
control operation of the autonomous vehicle based on predicted objects (see Fig. 9, step 908).
Regarding claim 2:
Schramm discloses the autonomy computing system of claim 1, wherein the at least one processor is further programmed to:
fuse the first features and the second features by:
integrating cross-modal attention between the first modality and the second modality in fusion (see paragraphs [0075]-[0076]).
Regarding claim 3:
Schramm discloses the autonomy computing system of claim 2, wherein the at least one processor is further programmed to:
fuse the first features and the second features by integrating intra-modal attention of at least one of the first modality or the second modality in the fusion (see paragraph [0075]; “The transformer block of fuser 512 may include three main parts, a self-attention layer, a cross-attention layer, as well as a fully connected layer at the end”; the fuser 512 includes self-attention layer which is an intra-modal attention).
Regarding claim 4:
Schramm discloses the autonomy computing system of claim 1, wherein the at least one processor is further programmed to:
detect the object proposals by:
generating initial object proposals based on the first enriched features and the second enriched features (see Fig. 5; output of encoder 514 is generated initial object proposals); and
detecting, using a transformer decoder, the object proposals in the first enriched features and the second enriched features based on the initial object proposals (see Fig. 5; output of decoder 516 is object proposals).
Regarding claim 9:
Schramm discloses the autonomy computing system of claim 1, wherein the plurality of modalities include three or more modalities (see paragraphs [0030]-[0031]).
Regarding claim 11:
Schramm discloses the autonomy computing system of claim 1, wherein the plurality of modalities include radio detection and ranging (radar) (see paragraphs [0024] and [0026]).
Regarding claims 12-15 and 19:
Claims 12-15 and 19 recite similar limitations as in claims 1-4 and 9. Hence, claims 12-15 and 19 are rejected under the same reasons as discussed above in claims 1-4 and 9, respectively.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 10 and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Schramm.
Regarding claim 10:
Schramm discloses all the features in claim 1. Schramm does not expressly disclose the autonomy computing system, wherein the plurality of modalities include a gated camera.
However, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to use any kind of camera in order to obtain an image of an object and/or environment condition, where the claimed differences involved to the substitution of interchangeable or replaceable equivalents and the reason for the selection of one equivalent for another was not to solve an existent problem, such substitution has been judicially determined to have been obvious. In re Ruff, 118, USPQ, 343 (CCPA 1958). This supporting is based on a recognition that the claimed difference exist not a result of an attempt by applicant to solve a problem but merely amounts to selection of expedients known to the artisan of ordinary skill as design choices.
Regarding claim 20:
Claim 20 recites similar limitations as in claim 10. Hence, claims 20 is rejected under the same reason as discussed above in claim 10.
Allowable Subject Matter
Claims 5-8 and 16-18 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
In regards to claim 5, none of the reference of record alone or in combination discloses or suggests the autonomy computing system of claim 4, wherein the plurality of modalities further include a third modality, the sensor data including third sensor data from one or more sensors of the third modality, the at least one processor further programmed to: extract third features based on the third sensor data; fuse the first features, the second features, and the third features into the first enriched features, the second enriched features, and third enriched features; represent the first enriched features in the BEV to derive first BEV enriched features; fuse the first BEV enriched features, the second enriched features, and the third enriched features into a fused feature map; and compute the initial object proposals based on the fused feature map.
In regard to claim 6, none of the reference of record alone or in combination discloses or suggests the autonomy computing system of claim 5, wherein the second modality has a different range from the third modality, the at least one processor further programmed to: fuse the first enriched features, the second enriched features, and the third enriched features by: combining the second enriched features weighted by a first weighting and the third enriched features weighted by a second weighting, the first weighting and the second weighting being dependent on a distance of a feature point from the autonomous vehicle.
In regards to claim 7, none of the reference of record alone or in combination discloses or suggests the autonomy computing system of claim 1, wherein the plurality of modalities include a first camera modality and a second camera modality, the at least one processor further programmed to: extract first camera features based on sensor data from one or more sensors of the first camera modality, and second camera features based on sensor data from one or more sensors of the second camera modality; blend the second features corresponding to first BEV camera features and the second features corresponding to second BEV camera features to derive composite paired second features, the first BEV camera features being the first camera features represented in the BEV, the second BEV camera features being the second camera features represented in the BEV; fuse the composite paired second features with the first camera features to derive first enriched camera features; and fuse the composite paired second features with the second camera features to derive second enriched camera features.
In regards to claim 8, none of the reference of record alone or in combination discloses or suggests the autonomy computing system of claim 1, wherein the plurality of modalities include a first camera modality and a second camera modality, the at least one processor further programmed to: extract first camera features based on sensor data from one or more sensors of the first camera modality; extract second camera features based on sensor data from one or more sensors of the second camera modality; blend the first camera features corresponding to the second features and the second camera features corresponding to the second features to derive composite paired camera features; and fuse the second features with the composite paired camera features to derive the second enriched features.
In regards to claim 16, none of the reference of record alone or in combination discloses or suggests the method of claim 15, wherein the plurality of modalities further include a third modality, the sensor data including third sensor data from one or more sensors of the third modality, the method further comprising: extracting third features based on the third sensor data; fusing the first features, the second features, and the third features into the first enriched features, the second enriched features, and third enriched features; representing the first enriched features in the BEV to derive first BEV enriched features; fusing the first BEV enriched features, the second enriched features, and the third enriched features into a fused feature map; and computing the initial object proposals based on the fused feature map.
In regards to claim 17, none of the reference of record alone or in combination discloses or suggests the method of claim 16, wherein the second modality has a different range from the third modality, fusing the first enriched features, the second enriched features, and the third enriched features further comprising: combining the second enriched features weighted by a first weighting and the third enriched features weighted by a second weighting, the first weighting and the second weighting being dependent on a distance of a feature point from the autonomous vehicle.
In regards to claim 18, none of the reference of record alone or in combination discloses or suggests the method of claim 12, wherein the plurality of modalities include a first camera modality and a second camera modality, the method further comprising: extracting first camera features based on sensor data from one or more sensors of the first camera modality; extracting second camera features based on sensor data from one or more sensors of the second camera modality; blending the second features corresponding to first BEV camera features and the second features corresponding to second BEV camera features to derive composite paired second features, the first BEV camera features being the first camera features represented in the BEV, the second BEV camera features being the second camera features represented in the BEV; fusing the composite paired second features with the first camera features to derive first enriched camera features; and fusing the composite paired second features with the second camera features to derive second enriched camera features.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Ravi Kumar et al. (US 2025/0239061) discloses a learnable sensor signatures to incorporate modality specific information into joint representation for multi-modal fusion.
Hao et al. (US 2025/0086953) discloses a device comprising a multi-modal BEV feature fusion module.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to LIXI CHOW SIMPSON whose telephone number is (571)272-7571. The examiner can normally be reached Mon-Fri 7:00am-3:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, William Boddie can be reached at 517-272-0666. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/LIXI C SIMPSON/Primary Examiner, Art Unit 2625
Prosecution Insights