DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Claims
This Office Action is in response to Application No. 18/825,120, filed September 5, 2024. Claims 1-20 are presently pending and are presented for examination.
Election
In response to the Restriction/Election Requirement dated January 28, 2026, Applicant elected, without traverse, Group 1, currently including claims 1-20.
Priority
Applicant’s claim for the benefit of a prior-filed application under 35 U.S.C. 119(e) or under 35 U.S.C. 120, 121, 365(c), or 386(c) for U.S. Provisional Patent Application No. 63/631,449, filed on April 8, 2024, is acknowledged and accepted.
Information Disclosure Statement
The information disclosure statements (IDS) submitted on September 5, 2024 is in compliance with the provisions of 37 CFT 1.97. Accordingly, the information disclosure statement has been considered by the examiner.
Drawing Objections
The drawings are objected to as failing to comply with 37 CFR 1.84, as indicated below.
The drawings are objected to because Figs. 1 and 3 (e.g. image data 101, and lidar data 121) and Figs. 5-8, are blurry and/or comprise photos which are difficult to understand and not adapted for reproduction. Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. The figure or figure number of an amended drawing should not be labeled as “amended.” If a drawing figure is to be canceled, the appropriate figure must be removed from the replacement sheet, and where necessary, the remaining figures must be renumbered and appropriate changes made to the brief description of the several views of the drawings for consistency. Additional replacement sheets may be necessary to show the renumbering of the remaining figures. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Claim Objections
Claim 9 is objected to because it appears to omit “configure” after “further.”.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an Abstract idea without significantly more.
Regarding Prong I of the Step 2A analysis in the 2019 PEG, the claims are to be analyzed to determine whether they recite subject matter that falls within one of the follow groups of abstract ideas: a) mathematical concepts, b) certain methods of organizing human activity, and/or c) mental processes.
With respect to claim 1, and similarly with respect to claims 11 and 12, the claim recites:
Claim 1: One or more processors comprising processing circuitry to:
A) detect, based at least on one or more neural networks (NNs) comprising one or more transformers processing a representation of image data and LiDAR data corresponding to an environment of an ego-machine, one or more hazards in the environment; and
B) control one or more operations of the ego-machine based at least on the one or more hazards.
The examiner submits that the foregoing bolded limitation(s) constitute “mental processes” or “computational method” because under its broadest reasonable interpretation, the claims cover detecting, within one’s mind or through mathematical algorithms, hazards in an environment based upon collected data including camera data.
Regarding Prong II of the Step 2A analysis in the 2019 PEG, the claims are to be analyzed to determine whether the claim, as a whole, integrates the abstract into a practical application. As noted in the 2019 PEG, it must be determined whether any additional elements in the claim beyond the abstract idea integrate the exception into a practical application in a manner that imposes a meaningful limit on the judicial exception. The courts have indicated that additional elements merely using a computer to implement an abstract idea, adding insignificant extra solution activity, or generally linking use of a judicial exception to a particular technological environment or field of use do not integrate a judicial exception into a “practical application.”
In the present case, the additional limitations beyond the above-noted abstract idea are as follows (where the underlined portions are the “additional limitations” while the bolded portions continue to represent the “abstract idea”).
Claim 1: One or more processors comprising processing circuitry to:
A) detect, based at least on one or more neural networks (NNs) comprising one or more transformers processing a representation of image data and LiDAR data corresponding to an environment of an ego-machine, one or more hazards in the environment; and
B) control one or more operations of the ego-machine based at least on the one or more hazards.
For the following reason(s), the examiner submits that the above identified additional limitations, underlined, do not integrate the above-noted abstract idea into a practical application.
Regarding the additional limitations of one or more processors, neural networks and transformers, the examiner submits that these features are part of a generic computer configured to merely execute instructions to apply an exception, per 2106.05(f). Accordingly, this comprises an extra solution activity that is well-understood, routine and/or conventional activities in the field of the particular claim. See MPEP 2106.05(d).
Regarding the additional limitation of “controlling one or more operations of the ego-machines,” the examiner submits that given the breadth of this limitation, this may merely comprise storing or transmitting data, which would include extra solution activities. Alternatively, the recitation of claim limitations that attempt to cover any solution to an identified problem with no restriction on how the result is accomplished and no description of the mechanism for accomplishing the result, does not integrate a judicial exception into a practical application or provide significantly more because this type of recitation is equivalent to the words “apply it”. See Electric Power Group, LLC v. Alstom, S.A., 830 F.3d 1350, 1356, 119 USPQ2d 1739, 1743-44 (Fed. Cir. 2016); Intellectual Ventures I v. Symantec, 838 F.3d 1307, 1327, 120 USPQ2d 1353, 1366 (Fed. Cir. 2016); Internet Patents Corp. v. Active Network, Inc., 790 F.3d 1343, 1348, 115 USPQ2d 1414, 1417 (Fed. Cir. 2015).
Dependent claims 2-8, 10 and 13-20 do not include any feature other than a continued description of the abstract concepts or otherwise an extra solution activity is well-understood, routine and/or conventional activities in the field of the particular claim. See MPEP 2106.05(d).
With respect to claim 9, this claim appears to indicate that the processing circuitry is configured to navigate the ego-machine, however, the step is not actually performed.
Thus, taken alone, the additional elements do not integrate the abstract idea into a practical application. Further, looking at the additional limitation(s) as an ordered combination or as a whole, the limitation(s) add nothing that is not already present when looking at the elements taken individually. For instance, there is no indication that the additional elements, when considered as a whole, reflect an improvement in the functioning of a computer or an improvement to another technology or technical field, apply or use the above-noted judicial exception to effect a particular treatment or prophylaxis for a disease or medical condition, implement/use the above-noted judicial exception with a particular machine or manufacture that is integral to the claim, effect a transformation or reduction of a particular article to a different state or thing, or apply or use the judicial exception in some other meaningful way beyond generally linking the use of the judicial exception to a particular technological environment, such that the claim as a whole is not more than a drafting effort designed to monopolize the exception (MPEP § 2106.05). Accordingly, the additional limitation(s) do not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea.
Regarding Step 2B of the Revised Guidance, representative independent claims do not include additional elements (considered both individually and as an ordered combination) that are sufficient to amount to significantly more than the judicial exception for the same reasons to those discussed above with respect to determining that the claim does not integrate the abstract idea into a practical application. Additionally, as discussed above with respect to integration of the abstract idea into a practical application, the additional limitations of one or more processors, neural networks and transformers, the examiner submits that reciting a generic computer, which comprises a CPU and memory (see para 0088), comprises mere instructions to apply an exception, per 2106.05(f). Additionally, with respect to the identified extra-solution activity, the Symantec, TLI, OIP Techs. and buySAFE court decisions cited in MPEP 2106.05(d)(II) indicate that mere receiving or transmitting data over a network is a well‐understood, routine, conventional function when it is claimed in a merely generic manner (as it is here). Furthermore, the Versata and OIP Techs court decisions cited in MPEP 2106.05(d)(II) indicate that storing and retrieving data in memory is a well‐understood, routine, conventional function when it is claimed in a merely generic manner (as it is here).
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
Claims 1-20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor, or for pre-AIA the applicant regards as the invention.
The term “hazard” in claims 1, 11 and 12 is a relative term which renders the claim indefinite. The term “hazard” is not defined by the claim, the specification does not provide a standard for ascertaining the requisite degree, and one of ordinary skill in the art would not be reasonably apprised of the scope of the invention.
Dependent claims 2-10 and 13-20 are rejected for the same reasons as they are dependent upon either claim 1 or claim 12.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-20 are rejected under 35 U.S.C. 102(a)(2) as being anticipated by U.S. Patent Publication No. 2025/0022143, to Li et al. (hereinafter Li).
As per claim 1, and similarly with respect to claims 11 and 12, Li discloses one or more processors comprising processing circuitry to: detect, based at least on one or more neural networks (NNs) comprising one or more transformers processing a representation of image data and LiDAR data corresponding to an environment of an ego-machine (e.g. see Figs. 1 and 3, and paras 0030-0050, wherein an autonomous vehicle (AV) (i.e. ego-machine) is provided including a sensing system 110 having LiDAR 112 and cameras 118 transmitting data (LiDAR and image data) to a perception system 130 including an object tracking pipeline (OTP) 132, wherein the OTP includes machine learning models 220-240 comprising deep neural networks having various neural networks and transformers), one or more hazards in the environment (e.g. see Fig. 4, para 0047, wherein the AV 402, via LiDAR sensor 404 and camera 406, captures potential hazardous objects in an environment such as other vehicles 410, 420 and street signs 414, 424); and control one or more operations of the ego-machine based at least on the one or more hazards (e.g. see para 0039, wherein based upon the monitoring and prediction component the AV is controlled by the AV control system 140).
As per claim 2, and similarly with respect to claim 13, Li discloses the features of claims 1 and 12, respectively, and further discloses wherein the processing the representation of the image data comprises generating one or more three-dimensional transformer queries based at least on one or more two-dimensional candidate bounding shapes extracted from the image data using the one or more NNs, and applying one or more sampled two-dimensional image features extracted from the image data using the one or more NNs to the one or more transformers (e.g. see Fig. 4 and paras 0019 and 0047-0050, wherein bounding boxes 412, 422 of two-dimensional camera images are converted, via a lifting transform (i.e. one or more NNs), to three-dimension representations (i.e. three dimensional transformer queries) of the driving environment).
As per claim 3, and similarly with respect to claim 14, Li discloses the features of claims 1 and 12, respectively, and further discloses wherein the processing the representation of the LiDAR data comprises generating one or more three-dimensional transformer queries based at least on one or more two-dimensional candidate bounding shapes extracted from the LiDAR data using the one or more NNs, and applying one or more sampled two-dimensional LiDAR features extracted from the LiDAR data using the one or more NNs to the one or more transformers (e.g. see Fig. 4 and paras 0019 and 0047-0050, wherein bounding boxes 412, 422 of two-dimensional camera images are converted, via a lifting transform (i.e. one or more NNs), to three-dimension representations (i.e. three dimensional transformer queries) of the driving environment; the Office further notes, as disclosed in para 0047, that images used to form the three-dimensional shapes are based upon cropped images formed by a cropping module 330, which utilizes both the camera and LiDAR images).
As per claim 4, and similarly with respect to claim 15, Li discloses the features of claims 1 and 12, respectively, and further discloses wherein the processing the representation of the image data and the LiDAR data comprises fusing one or more sampled two- dimensional image features and one or more sampled two-dimensional LiDAR features using one or more cross-attention layers of the one or more transformers (e.g. see Figs. 3 and 6, wherein camera and LiDAR data are combined (i.e. fused), which would include two-dimensional image features of both since three-dimensional aspects could not be rendered in two-dimensional form, using models (i.e. transformers, which include cross-attention layers)).
As per claim 5, and similarly with respect to claim 16, Li discloses the features of claims 1 and 12, respectively, and further discloses wherein the processing the representation of the image data and the LiDAR data comprises projecting one or more keypoints associated with one or more reference three-dimensional positions corresponding to one or more transformer queries into extracted image features and extracted LiDAR features (e.g. see Fig. 4, wherein the transformation, based upon camera and LiDAR data, is based upon keypoints, shown as dots on objects detected by the LiDAR 404)).
As per claim 6, and similarly with respect to claim 17, Li discloses the features of claims 1 and 12, respectively, and further discloses wherein the processing circuitry is further to generate a plurality of transformer queries based at least on one or more candidate bounding shapes extracted from the image data or the LiDAR data (e.g. see Fig. 4 and paras 0047-0051, wherein the transformation, based upon camera and LiDAR data, is based upon the formation of bounding boxes 412, 422), one or more randomly initialized three-dimensional positions (e.g. see Fig. 4, wherein position dots are identified by LiDAR 404) and one or more ego-motion compensated transformer predictions (e.g. see para 0038, wherein the AV includes prediction component 134).
As per claim 7, and similarly with respect to claim 18, Li discloses the features of claims 1 and 12, respectively, and further discloses wherein the processing circuitry is further to detect the one or more hazards based at least on the one or more transformers (e.g. see para 0033, wherein the perception system detects the presence of road objects, which would include perceptible hazards): generating classification data representing whether there is road debris predicted at a three- dimensional location corresponding to each transformer query of one or more transformer queries (e.g. see para 0037, wherein the objects are identified and classified), and regressing a representation of a bounding shape of the road debris at the three-dimensional location (e.g. see Fig. 4, the Office further notes that while vehicle and sign objects are identified and included within bounding shapes, given the object identification and classification other objects would also be identified and bounded within a shape).
As per claim 8, and similarly with respect to claim 19, Li discloses the features of claims 1 and 12, respectively, and further discloses wherein the processing circuitry is further to update the one or more transformers based at least on omitting transformer queries representing reference three-dimensional locations outside a ground truth navigable space (e.g. see Fig. 3 and para 0043-0046, wherein the models 220, 230, 240, which include one or more transformers, are trained with historic data (i.e. omitted transformer queries) from various location).
As per claim 9, Li discloses the features of claim 1, and further discloses wherein the processing circuitry is further to navigate the ego-machine based at least on avoiding or compensating for the one or more hazards (e.g. see Fig. 3 and rejection of claim 1, wherein an AVCS 140 is provided for controlling the vehicle based upon detected objects).
As per claim 10, and similarly with respect to claim 20, Li discloses the features of claims 1 and 12, respectively, and further discloses wherein the one or more processors are comprised in at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine (e.g. see Fig. 1 and para 0037, wherein prediction model 134 is part of a perception system of the AV); a system for performing simulation operations; a system for performing digital twin operations; a system for performing light transport simulation; a system for performing collaborative content creation for 3D assets; a system for performing deep learning operations; a system for performing remote operations; a system for performing real-time streaming; a system for generating or presenting one or more of augmented reality content, virtual reality content, or mixed reality content; a system implemented using an edge device; a system implemented using a robot; a system for performing conversational AI operations; a system implementing one or more language models; a system implementing one or more large language models (LLMs); a system implementing one or more vision language models (VLMs); a system implementing one or more multi-modal language models; a system for generating synthetic data; a system for generating synthetic data using AI; a system incorporating one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources (e.g. see Fig. 3 and para 0043-0046, wherein the models 220, 230, 240, which include one or more transformers, are trained with historic data (i.e. omitted transformer queries) from various location).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to James M. McPherson whose telephone number is (313) 446-6543. The examiner can normally be reached on 7:30 AM - 5PM Mon-Fri Eastern Alt Fri. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Abby Flynn can be reached on 571 272-9855. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JAMES M MCPHERSON/Primary Examiner, Art Unit 3663B
Prosecution Insights