DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statements (IDS) submitted on 02/17/2026 have been considered by the examiner.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-2, 5-14, and 19-20 are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Pronovost (US Pub. No. 2024/0101150 A1).
Regarding claim 1, Pronovost discloses, a computer-implemented method comprising: receiving sensor data of an environment having one or more agents; decomposing the sensor data into a plurality of scene elements in the environment; (See Pronovost ¶36, “In various examples, the vehicle computing device can receive the sensor data and can semantically classify the detected objects (e.g., determine an object type), such as, for example, whether the object is a pedestrian, such as object 108, a vehicle such as object 110, a building, a truck, a motorcycle, a moped, or the like. The objects may include static objects (e.g., buildings, bridges, signs, etc.) and dynamic objects such as other vehicles, pedestrians, bicyclists, or the like. In some examples, a classification may include another vehicle (e.g., a car, a pick-up truck, a semi-trailer truck, a tractor, a bus, a train, etc.), a pedestrian, a child, a bicyclist, a skateboarder, an equestrian, an animal, or the like.”)
generating a plurality of tokens including a respective token for each respective scene element of the plurality of scene elements in the environment; (See Pronovost ¶54, “In various examples, the computing device generating feature vectors based at least in part on state data associated with a vehicle and/or object(s). The state data can include data describing an object (e.g., the pedestrian 108, the vehicle 110 in FIG. 1) and/or a vehicle (e.g., vehicle 102) in an environment, such as in example environment 100.”
Further see Pronovost ¶55, “In some examples, sensor data or processed sensor data (e.g., a top-down representation) may be input into a machine learned model (e.g., a convolutional neural network (CNN), a Recurrent Neural Network (RNN), a graph neural network (GNN), etc.), which can determine a feature vector for processing by a machine learned model.”
Further see Pronovost ¶59, “In some examples, the codebook 308 can receive the discretized feature vectors 312 and determine the token sequence 316 by mapping index values of the discretized feature vectors 312 to corresponding index values of tokens. … In some examples, the quantizer 310 is not used, such as when the machine learned model 314 exchanges data with the codebook 308 to determine the token sequence 316.”)
and processing the plurality of tokens using a decoder model to generate a respective predicted trajectory for the one or more agents in the environment. (See Pronovost ¶62, “In various examples, a decoder 318 can receive the token sequence 316 and determine the output data 320. The decoder 318 can represent a machine learned model such as a GNN, a GAN, an RNN, another Transformer model, etc. The output data 320 can, for example, be similar to the output data 106 or the output data 210 and represent an object trajectory, scene data, simulation data, and so on.”)
Regarding claim 2, Pronovost discloses, the method of claim 1, wherein decomposing the sensor data into the plurality of scene elements comprises distinguishing an agent in the environment from a ground region in the environment, (See Pronovost ¶65, “As shown in FIG. 3, the environment may be represented by the vector representation 322 comprising vectors to represent … an attribute of the autonomous vehicle 324 (e.g., velocity, position, etc.), and/or features of the environment (e.g., roadway boundary, roadway centerline, crosswalk permission, traffic light permission, and the like).”)
and wherein generating the plurality of tokens comprises generating a first token for the agent and a different second token for the ground region. (See Pronovost ¶82, “For example, a first token can represent a vehicle traveling in a first direction at a particular velocity and a second token can represent an object facing a second direction and not moving. A token may also or instead represent a stop sign, crosswalk, a roadway, or other environmental feature.”)
Regarding claim 5, Pronovost discloses, the method of claim 1, wherein generating a token for a scene element in the environment comprises encoding image features (See Pronovost ¶35, “In some examples, the sensors may include sensors mounted on the vehicle 102, and include, without limitation, ultrasonic sensors, radar sensors, light detection and ranging (lidar) sensors, cameras,”
Further see Pronovost ¶55, “In some examples, sensor data or processed sensor data (e.g., a top-down representation) may be input into a machine learned model (e.g., a convolutional neural network (CNN), a Recurrent Neural Network (RNN), a graph neural network (GNN), etc.), which can determine a feature vector for processing by a machine learned model.”)
and geometry features of an entity in the environment. (See Pronovost ¶64, “The vector representation 322 may, in some examples, be determined based on a polyline (e.g., a set of line segments) representing one or more map elements. For instance, the graph neural network can encode and aggregate the polyline into a node data structure representing with the map element(s). For example, an object or feature of the environment can be represented by polylines (e.g., a lane can be segmented into a number of smaller line segments whose length, location, orientation angle (e.g., yaw), and directionality, when aggregated, define the lane). Similarly, a crosswalk may be defined by four connected line segments, and a roadway edge or roadway centerline may be multiple connected line segments.”)
Regarding claim 6, Pronovost discloses, the method of claim 1, wherein generating the plurality of tokens is performed by a multi-modality scene encoder configured to tokenize scene elements in a scene as well as perception outputs representing sensor information of a vehicle. (See Pronovost ¶65, “As shown in FIG. 3, the environment may be represented by the vector representation 322 comprising vectors to represent objects and/or features of the environment including one or more of: an attribute (e.g., position, velocity, acceleration, yaw, etc.) of the object 326, history of the object 326 (e.g., location history, velocity history, etc.), an attribute of the autonomous vehicle 324 (e.g., velocity, position, etc.), and/or features of the environment (e.g., roadway boundary, roadway centerline, crosswalk permission, traffic light permission, and the like).”
Further see Pronovost ¶44. “For example, a first token can represent an object state, a second token can represent a vehicle state, and a third token can represent environment features (e.g., a traffic signals, crosswalk, weather, etc.).”)
Regarding claim 7, Pronovost discloses, the method of claim 1, wherein generating the plurality of tokens comprises applying the operations of a plurality of self-attention layers. (See Pronovost ¶58, “A machine learned model 314 (e.g., a Transformer model) can receive tokens, discrete feature vectors, and/or continuous feature vectors from the codebook and arrange the tokens and associated feature vectors into a token sequence 316. The token sequence 316 can represent tokens arranged or clustered in a particular order. The machine learned model 314 can include one or more self-attention layers to cause the tokens to be arranged with attention to features of another token.”)
Regarding claim 8, Pronovost discloses, the method of claim 7, wherein the plurality of self-attention layers fuse information extracted from tokens for the scene elements and tokens generated from perception systems of a vehicle. (See Pronovost ¶44. “For example, a first token can represent an object state, a second token can represent a vehicle state, and a third token can represent environment features (e.g., a traffic signals, crosswalk, weather, etc.).”
Further see Pronovost ¶58, “A machine learned model 314 (e.g., a Transformer model) can receive tokens, discrete feature vectors, and/or continuous feature vectors from the codebook and arrange the tokens and associated feature vectors into a token sequence 316. The token sequence 316 can represent tokens arranged or clustered in a particular order. The machine learned model 314 can include one or more self-attention layers to cause the tokens to be arranged with attention to features of another token.”)
Regarding claim 9, Pronovost discloses the method of claim 1, wherein the decoder model has a Transformer-based architecture. (See Pronovost ¶62, “In various examples, a decoder 318 can receive the token sequence 316 and determine the output data 320. The decoder 318 can represent a machine learned model such as a GNN, a GAN, an RNN, another Transformer model, etc.”)
Regarding claim 10, Pronovost discloses the method of claim 1, wherein generating the plurality of tokens comprises generating a token representing one or more features of a road graph for the environment. (See Pronovost ¶64, “The vector representation 322 may, in some examples, be determined based on a polyline (e.g., a set of line segments) representing one or more map elements. For instance, the graph neural network can encode and aggregate the polyline into a node data structure representing with the map element(s). For example, an object or feature of the environment can be represented by polylines (e.g., a lane can be segmented into a number of smaller line segments whose length, location, orientation angle (e.g., yaw), and directionality, when aggregated, define the lane). Similarly, a crosswalk may be defined by four connected line segments, and a roadway edge or roadway centerline may be multiple connected line segments.”
Further see Pronovost ¶59, “In some examples, the codebook 308 can receive the discretized feature vectors 312 and determine the token sequence 316 by mapping index values of the discretized feature vectors 312 to corresponding index values of tokens.”
Further see Pronovost ¶44. “For example, a first token can represent an object state, a second token can represent a vehicle state, and a third token can represent environment features (e.g., a traffic signals, crosswalk, weather, etc.).”)
Regarding claim 11, Pronovost discloses, the method of claim 1, generating the plurality of tokens comprises generating a token representing one or more motion-based characteristics of an agent in the environment. (See Pronovost ¶65, “As shown in FIG. 3, the environment may be represented by the vector representation 322 comprising vectors to represent objects and/or features of the environment including one or more of: an attribute (e.g., position, velocity, acceleration, yaw, etc.) of the object 326, history of the object 326 (e.g., location history, velocity history, etc.), an attribute of the autonomous vehicle 324 (e.g., velocity, position, etc.).”
Further see Pronovost ¶59, “In some examples, the codebook 308 can receive the discretized feature vectors 312 and determine the token sequence 316 by mapping index values of the discretized feature vectors 312 to corresponding index values of tokens.”
Regarding claim 12, Pronovost discloses the method of claim 11, wherein the motion-based characteristics include an agent velocity or a motion history. (See Pronovost ¶65, “As shown in FIG. 3, the environment may be represented by the vector representation 322 comprising vectors to represent objects and/or features of the environment including one or more of: an attribute (e.g., position, velocity, acceleration, yaw, etc.) of the object 326, history of the object 326 (e.g., location history, velocity history, etc.), an attribute of the autonomous vehicle 324 (e.g., velocity, position, etc.).”)
Regarding claim 13, Pronovost discloses, the method of claim 1, wherein generating the plurality of tokens comprises generating a token representing traffic light information for a traffic light within the environment. (See Pronovost ¶44. “For example, a first token can represent an object state, a second token can represent a vehicle state, and a third token can represent environment features (e.g., a traffic signals, crosswalk, weather, etc.).”
Further see Pronovost ¶65, “As shown in FIG. 3, the environment may be represented by the vector representation 322 comprising vectors to represent objects and/or features of the environment including one or more of: … features of the environment (e.g., roadway boundary, roadway centerline, crosswalk permission, traffic, light permission, and the like).”)
Regarding claim 14, Pronovost discloses, the method of claim 1, further comprising adjusting a trajectory of a vehicle based on the predicted trajectory of the scene element in the environment. (See Pronovost ¶142, “For example, an output from the model component 830 can be sent to the perception component 822 or the planning component 824, just to name a few. In various examples, the vehicle computing device may control operation of the vehicle, such as the planning component 824. The vehicle computing device may determine a vehicle trajectory based at least in part on the object trajectory(ies) thereby improving vehicle safety by planning for the possibility that the object may intersect with the vehicle in the future.”)
Regarding claim 15, Pronovost discloses, the method of claim 1, wherein the sensor data represents sensor observations over a plurality of history frames. (See Pronovost ¶119, “Additionally, or in the alternative, the sensor system(s) 806 may send sensor data, via the one or more networks 840, to the one or more computing device(s) 834 at a particular frequency, after a lapse of a predetermined period of time, in near real-time, etc.”)
Regarding claim 19, Pronovost discloses, a system comprising one or more computers and one or more storage devices storing instructions that are operable, when executed by the one or more computers, to cause the one or more computers to perform operations comprising: (See Pronovost ¶150, “In the context of software, the operations represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the recited operations.”)
receiving sensor data of an environment having one or more agents; decomposing the sensor data into a plurality of scene elements in the environment; (See Pronovost ¶36, “In various examples, the vehicle computing device can receive the sensor data and can semantically classify the detected objects (e.g., determine an object type), such as, for example, whether the object is a pedestrian, such as object 108, a vehicle such as object 110, a building, a truck, a motorcycle, a moped, or the like. The objects may include static objects (e.g., buildings, bridges, signs, etc.) and dynamic objects such as other vehicles, pedestrians, bicyclists, or the like. In some examples, a classification may include another vehicle (e.g., a car, a pick-up truck, a semi-trailer truck, a tractor, a bus, a train, etc.), a pedestrian, a child, a bicyclist, a skateboarder, an equestrian, an animal, or the like.”)
generating a plurality of tokens including a respective token for each respective scene element of the plurality of scene elements in the environment; (See Pronovost ¶54, “In various examples, the computing device generating feature vectors based at least in part on state data associated with a vehicle and/or object(s). The state data can include data describing an object (e.g., the pedestrian 108, the vehicle 110 in FIG. 1) and/or a vehicle (e.g., vehicle 102) in an environment, such as in example environment 100.”
Further see Pronovost ¶55, “In some examples, sensor data or processed sensor data (e.g., a top-down representation) may be input into a machine learned model (e.g., a convolutional neural network (CNN), a Recurrent Neural Network (RNN), a graph neural network (GNN), etc.), which can determine a feature vector for processing by a machine learned model.”
Further see Pronovost ¶59, “In some examples, the codebook 308 can receive the discretized feature vectors 312 and determine the token sequence 316 by mapping index values of the discretized feature vectors 312 to corresponding index values of tokens. … In some examples, the quantizer 310 is not used, such as when the machine learned model 314 exchanges data with the codebook 308 to determine the token sequence 316.”)
and processing the plurality of tokens using a decoder model to generate a respective predicted trajectory for the one or more agents in the environment. (See Pronovost ¶62, “In various examples, a decoder 318 can receive the token sequence 316 and determine the output data 320. The decoder 318 can represent a machine learned model such as a GNN, a GAN, an RNN, another Transformer model, etc. The output data 320 can, for example, be similar to the output data 106 or the output data 210 and represent an object trajectory, scene data, simulation data, and so on.”)
Regarding claim 20, Pronovost discloses, one or more non-transitory computer storage media encoded with computer program instructions that when executed by a plurality of computers cause the plurality of computers to perform operations comprising: (See Pronovost ¶150, “In the context of software, the operations represent computer-executable instructions stored on one or more computer-readable storage media that, when executed by one or more processors, perform the recited operations.”)
receiving sensor data of an environment having one or more agents; decomposing the sensor data into a plurality of scene elements in the environment; (See Pronovost ¶36, “In various examples, the vehicle computing device can receive the sensor data and can semantically classify the detected objects (e.g., determine an object type), such as, for example, whether the object is a pedestrian, such as object 108, a vehicle such as object 110, a building, a truck, a motorcycle, a moped, or the like. The objects may include static objects (e.g., buildings, bridges, signs, etc.) and dynamic objects such as other vehicles, pedestrians, bicyclists, or the like. In some examples, a classification may include another vehicle (e.g., a car, a pick-up truck, a semi-trailer truck, a tractor, a bus, a train, etc.), a pedestrian, a child, a bicyclist, a skateboarder, an equestrian, an animal, or the like.”)
generating a plurality of tokens including a respective token for each respective scene element of the plurality of scene elements in the environment; (See Pronovost ¶54, “In various examples, the computing device generating feature vectors based at least in part on state data associated with a vehicle and/or object(s). The state data can include data describing an object (e.g., the pedestrian 108, the vehicle 110 in FIG. 1) and/or a vehicle (e.g., vehicle 102) in an environment, such as in example environment 100.”
Further see Pronovost ¶55, “In some examples, sensor data or processed sensor data (e.g., a top-down representation) may be input into a machine learned model (e.g., a convolutional neural network (CNN), a Recurrent Neural Network (RNN), a graph neural network (GNN), etc.), which can determine a feature vector for processing by a machine learned model.”
Further see Pronovost ¶59, “In some examples, the codebook 308 can receive the discretized feature vectors 312 and determine the token sequence 316 by mapping index values of the discretized feature vectors 312 to corresponding index values of tokens. … In some examples, the quantizer 310 is not used, such as when the machine learned model 314 exchanges data with the codebook 308 to determine the token sequence 316.”)
and processing the plurality of tokens using a decoder model to generate a respective predicted trajectory for the one or more agents in the environment. (See Pronovost ¶62, “In various examples, a decoder 318 can receive the token sequence 316 and determine the output data 320. The decoder 318 can represent a machine learned model such as a GNN, a GAN, an RNN, another Transformer model, etc. The output data 320 can, for example, be similar to the output data 106 or the output data 210 and represent an object trajectory, scene data, simulation data, and so on.”)
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 16-17 are rejected under 35 U.S.C. 103 as being unpatentable over Pronovost (US Pub. No. 2024/0101150 A1) in view of Saranin et al. (US Pub. No. 2022/0129684 A1).
Regarding claim 16, Pronovost discloses, the method of claim 15, but he fails to disclose, further comprising collapsing scene element representations that appear over the plurality of history frames into a single scene element representation.
However, Saranin discloses, further comprising collapsing scene element representations that appear over the plurality of history frames into a single scene element representation. (See Saranin ¶114, “As shown in FIG. 8, method 800 begins with 802 and continues with optional 804 where the LiDAR dataset is downsampled based on a planned trajectory of an AV. For example, downsampling is performed for LiDAR data points corresponding to a region of interest along a planned trajectory of the AV at a lower rate than the LiDAR data points corresponding to other regions that are not along the planned trajectory of the AV.”)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to include the downsampling of LIDAR points clouds of a scene prior to performing other operations as suggested by Saranin to Pronovost’s predicting the trajectories of vehicles, pedestrians, and other objects. This can be done using known engineering techniques, with a reasonable expectation of success. The motivation for doing so is faster inference. Downsampling reduces the input size, allowing a machine learning model to make predictions within milliseconds, which is a necessity for autonomous driving safety.
Regarding claim 17, Pronovost and Saranin disclose, the method of claim 15, wherein collapsing the scene element representations comprises downsampling LiDAR data. (See Saranin ¶114, “As shown in FIG. 8, method 800 begins with 802 and continues with optional 804 where the LiDAR dataset is downsampled based on a planned trajectory of an AV. For example, downsampling is performed for LiDAR data points corresponding to a region of interest along a planned trajectory of the AV at a lower rate than the LiDAR data points corresponding to other regions that are not along the planned trajectory of the AV.”)
Allowable Subject Matter
Claims 3, 4 and 18 are objected to as being dependent upon a rejected base claim but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Regarding claim 3, the method of claim 2, wherein generating the first token comprises using a first model and wherein generating the second token comprises using a different second model. (The disclosed prior art of record fails to disclose the limitations of this claim.)
Regarding claim 4, the method of claim 3, wherein decomposing the sensor data into the plurality of scene elements comprises generating a different third token for an object in the environment. (This claim is objected to since it depends from objected to claim 3.)
Regarding claim 18, the method of claim 17, wherein downsampling the LiDAR data comprises performing a first downsampling process for ground scene elements and a different second downsampling process for other scene elements. (The disclosed prior art of record fails to disclose the limitations of this claim.)
Conclusion
Listed below are the prior arts made of record and not relied upon but are considered pertinent to applicant’s disclosure.
Bauer et al. (“Slim: Self-supervised lidar scene flow and motion segmentation”)
We learn our model end-to-end by backpropagating gradients through Kabsch’s algorithm and demonstrate that this leads to accurate ego motion which in turn improves the scene flow estimate. Using our method, we show state-of-the-art results across multiple scene flow metrics for different real-world datasets, showcasing the robustness and generalizability of this approach. We further analyze the performance gain when performing joint motion segmentation and scene flow in an ablation study. We also present a novel network architecture for 3D LiDAR scene flow which is capable of handling an order of magnitude more points during training than previously possible.
Ngiam et al. (Scene transformer: A Unified architecture for predicting multiple agent trajectories”) Our model architecture employs attention to combine features across road elements, agent interactions, and time steps. We evaluate our approach on autonomous driving datasets for both marginal and joint motion prediction and achieve state of the art performance across two popular datasets. Through combining a scene-centric approach, agent permutation equivariant model, and a sequence masking strategy, we show that our model can unify a variety of motion prediction tasks from joint motion predictions to conditioned prediction.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DAVID PERLMAN whose telephone number is (571) 270-1417.
The examiner can normally be reached on Monday - Friday; 10:00am -6:30pm.
Examiner interviews are available via telephone and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chineyere Wills-Burns can be reached at (571) 272-9752. The fax phone number for the organization where this application or proceeding is assigned is
(571) 273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at (866) 217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call (800) 786-9199 (IN USA OR CANADA) or (571) 272-1000.
/DAVID PERLMAN/Primary Examiner, Art Unit 2673
Prosecution Insights