Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
RESPONSE TO ARGUMENTS
35 USC 101 Alice
The examiner acknowledges the amendment of claims 1, 3, 5-7, 14, 16 & 18-20, addition of claims 21-24 and the cancellation of claims 9-13 filed 05/18/2026. After carefully reviewing applicant amendments, 35 USC 101 guidance and claim limitations, claim limitations are sufficient to overcome grounds of rejection.
Prior Art Rejection
Argument 1
Applicant submits current references do not disclose amended limitations requiring one or more features to be combined with camera data associated with the first sensor image and the second sensor image to obtain an input, wherein camera data includes at least one of a camera center or one or more vectors of one or more rays from the camera center to the one or more features.
Applicant argument has been considered but is not persuasive. Emmons discloses image based and camera data aspects that Kim does not expressly disclose. Emmons discloses obtaining images from multiple image sensors positioned around a vehicle and processing those images using a vision based machine learning model. See Emmons [0030-0031, 0037, 0041-0044, 0051]. Emmon [0051, 0056] further discloses that backbone networks receive respective images as input, process raw pixels included in the images, and output feature maps or tensors. Emmons [0055, 0064] discloses transformations may be based on camera parameters associated with the image sensors, such as extrinsic and/or intrinsic camera parameters, and that a lookup table may be used in combination with extrinsic and intrinsic camera parameters associated with the image sensors. Emmons [0056-0057, 0063-0065, 0079] discloses output tensors from the backbone networks may be combined or fused together into respective virtual camera spaces, such as vector spaces, via the VRU and non-VRU networks. Examiner concludes under the broadest reasonable interpretation that Emmons discloses combining image derived features with camera data associated with sensor images to obtain an input.
In view of above arguments, examiner submits rejection is sufficient and respectfully maintained.
Argument 2
Applicant submits Kim is silent to receiving first and second sensor images, determining image based features from those sensor images, combining those features with camera data associated with the sensor images, or using camera center/ray vector information as part of the input for determining a grid.
In response examiner submits applicant is arguing Kim alone wherein the rejection on above limitations is based on Kim in view of Emmons.
Kim’s [0010-0011, 0043, 0059] discloses processing an input using a neural network to generate a top-down scene representation/grid. Kim discloses the system processes an encoder input using a point cloud encoder neural network to generate a global feature map including respective features for each of a plurality of locations in a top-down representation of the environment. Kim’s [0023, 0060, 0063-0064, 0096-0097] discloses identifying a grid representation of the top-down representation that discretizes the top-down representation into a plurality of pillars, with each point assigned to a respective pillar. Therefore, Kim discloses determining a grid/top-down scene representation using a neural network. Emmons [0031, 0037, 0051, 0056, 0090] discloses the sensor image, image feature, camera data that Kim does not expressly disclose. Emmon’s [0031, 0037, 0044, 0051, 0056, 0090] discloses obtaining images from image sensors, processing those images using a vision-based machine leaning model, and producing feature maps/tensors from those images. In view of above claim mapping, examiner submit rejection is sufficient and respectfully maintained.
Argument 3
Applicant submits Emmon’s does not teach the specific claimed operation of combining the one or more features with camera data associated with the first sensor image and the second sensor image to obtain an input that is then used by the neural network to determine a grid of the scene. Applicant submits Emmons use of camera parameters for rectification or projection is different from the claimed use of camera data as context for neural network processing of the first and second sensor images.
In response, examiner submits Emmons discloses transformations may be based on camera parameters in isolation. Emmons [0055] discloses transformations may be based on camera parameters associated with the image sensors, including extrinsic and intrinsic parameters. [0055] discloses rectification may optionally represent one or more layers of the backbone networks, in which values for the transformation are learned based on training data. [0056-0057] discloses backbone networks output features maps or tensors, and that those output tensors may be combined or fused into respective virtual camera spaces. [0063-0065] discloses projecting information into virtual camera space using a fixed a projection engine and a lookup table used with extrinsic and intrinsic camera parameters. Examiner concludes that Emmons discloses using camera geometry information together with image derived features during neural network image processing. Kim’s [0011, 0023, 0059-0064, 0096-0097] discloses grid/top down scene representation based on neural network processing. It would have been obvious to use Emmon’s camera data conditioned image feature processing with Kim’s grid/top-down neural network representation in order to use temporally separated camera frames and camera geometry to form or update a scene representation for autonomous vehicle perception.
In view of above arguments, examiner submits rejection is sufficient and respectfully maintained.
Argument 4
Applicant submits Emmons does not disclose conditioning the neural network by providing context during processing of the first sensor image and the second sensor image. Applicant submits amended claim limitations require more than merely projecting image information into a virtual camera space; rather, the claims require camera data, including camera center or ray vector information, to be combined with the determined features and used as contextual input for the neural network when determining the grid of the scene.
In response, examiner submits Emmon’s [0055] discloses camera parameters-based transformations may be used during image processing and that rectification may optionally represent one or more layers of the backbone networks. [0063-0065] discloses projecting information into virtual camera space using intrinsic/extrinsic camera parameters and ray geometry. [0065] discloses each pixel may represent a ray out of an image, with the ray extending in virtual camera space, and that the processor system may form the virtual camera space based on combinations of these rays for the pixels of the images. Because claim recites “at least one of” a camera center or one or more vectors of one or more rays, Emmon’s disclosure of ray based virtual camera geometry is sufficient to teach the limitation. [0055, 0064-0065, 0079, 0090-0092] discloses camera parameter and ray based processing provides camera geometry context during neural network image processing, which corresponds to conditioning the neural network by providing context during processing of the first and second sensor images.
In view of above arguments, examiner submits rejection is sufficient and respectfully maintained.
Prior Art Rejection Conclusion
The examiner acknowledges the amendment of claims 1, 3, 5-7, 14, 16 & 18-20, addition of claims 21-24 and the cancellation of claims 9-13 filed 05/18/2026. Applicants arguments filed on (05/18/2026) have been fully considered but are deemed moot in view of new grounds of rejection. Due to the variation in claim scope via amendments a new ground of rejection is proper.
CLAIM REJECTIONS - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 3, 5-8, 14, 16 & 18-24 are rejected under 35 U.S.C. 103 as being unpatentable over
Kim et al. (U.S. Publication 2022/0155096) in view of Emmons et al. (U.S. Publication 2023/0057509)
As to claims 1, 14 & 24, Kim discloses determining, using the one or more processors and the neural network, and based at on an input, a grid of the scene in which the one or more features are respectively assigned to respective cells of the grid ([0011] discloses generating a global feature map including respective features for each of a plurality of locations in the top-down representation. [0023] discloses identifying a grid representation discretizes into a plurality of pillars, with each points being assigned. [0059] discloses identifying a grid representation of the top down representation that discretizes of the top down representation into a plurality of pillars, with each of the plurality of points being assigned to a respective one of the pillars. [0060] discloses identifying a grid representation of the top down representation that discretizes the top down representation into a plurality of pillars, with each of the plurality of points being assigned to a respective one of the pillars. [0064] discloses discretizing the point set into an evenly spaced grid of shape MxN in the x-y plane, creating a set of MN pillars.).
Kim is silent to receiving, using one or more processors, a first sensor image detected at a first time point and a second sensor image detected at a second time point; determining, using the one or more processors and a neural network, and based at least on the first sensor image and the second sensor image, one or more features represented by the first sensor image and the second sensor image; combine the one or more features with camera data associated with the first sensor image and the second sensor image to obtain an input, the camera data (i) including at least one of a camera center or one or more vectors of one or more rays from the camera center to the one or more features and (ii) conditioning the neural network by providing context during processing of the first sensor image and the second sensor image;
and at least one of (i) assigning, using the one or more processors, the grid to a map data structure or (ii) presenting, using the one or more processors and a display device the grid.
However, Emmon’s discloses receiving, using one or more processors, a first sensor image detected at a first time point and a second sensor image detected at a second time point. ([0031] discloses image sensors obtain images used by the processor system to determine information associated with objects positioned proximate to a vehicle. [0067] discloses vision based machine learning model can utilize a multitude of frames during a forward pass through the model. [0068] discloses the frame selector engine may obtain frames (e.g. vision information associated with 12-, 14- or 16-time stamps at which images were taken) spread over previous 3, 5, 7, 9 seconds. [0082] discloses the selected vision information may represent frames spread apart in time from within a threshold period of time. [0090] discloses at block 602, the system obtains images from multitude of image sensors positioned about a vehicle. )
determining, using the one or more processors and a neural network, and based at least on the first sensor image and the second sensor image, one or more features represented by the first sensor image and the second sensor image ([0051] discloses the vision based machine learning model includes backbone network 200 which receive respective image as input. Thus, the backbone network 200 process the raw pixels included in the images 202A-202H. [0056] discloses the backbone networks 200 may thus output feature maps (e.g. tensors) which are used by VRU network 210 and non-VRU network 230. [0090] discloses at block 604, the system computes a forward pass through backbone networks. The backbone networks may represent convolutional neural networks which optionally pre-process the images (e.g. rectify );
Combining the one or more features with camera data associated with the first sensor image and second sensor image to obtain an input. ([0055] discloses rectification may be performed via the backbone networks 200 to address these differences. For example, a transformation (e.g. an affine transformation) may be applied to the images 202A-202H, or a portion thereof, to normalize the images. In this example, the transformation may be based on camera parameters associated with the image sensors (e.g. image sensors 102A-102F), such as extrinsic and/or intrinsic parameters. [0056] discloses the backbone networks 200 may thus output feature maps (e.g. tensors) which are used by VRU network 210 and non-VRU network 230. In some embodiments, the output from the backbone networks 200 may be combined into a matrix or tensor. In some embodiments, the output may be provided as a multitude of tensors (e.g. 8 tensors in the illustrated example, the output is referred to as vision information 204 which is input into the networks 210, 230. [0057] discloses output tensor from the backbone networks 200 may be combined (e.g. fused) together into respective virtual camera spaces (e.g. a vector space) via the VRU 210 and non-VRU network 230. [0063] discloses vision information 204 from the backbone networks (e.g. networks 200) is provided as input into a fixed projection engine 302.)
the camera data (i) including at least one of a camera center or one or more vectors of one or more rays from the camera center to the one or more features. ([0064] discloses the fixed projection engine 302 may project information into a virtual camera space associated with a virtual camera. As described above, the virtual camera may be positioned at 1 meter, 1.5 meters, 2.5 meters, and so on, above an autonomous vehicle executing the vision based machine learning model. Without being constrained by way of theory, it may be appreciated that pixels of input images may be mapped into the virtual camera space. For example, a lookup table may be used in combination with extrinsic and intrinsic camera parameters associated with the image sensors (e.g. image sensor 102A-102F.) Also see with respect to the ray, the fixed projection engine 302 may identify two different depths along the ray from the given pixel. The processor system 120 may then form the virtual camera space based on combinations of these rays for the pixels of the images. Because the claim recites “at least one of” a camera center or one or more vectors of one or more rays, Emmons discloses ray based virtual camera geometry discloses claimed limitations.)
(ii) conditioning the neural network by providing context during processing of the first sensor image and the second sensor image; ([0055] discloses rectification may be performed via the backbone networks 200 to address these differences. For example, a transformation (e.g. an affine transformation) may be applied to the images 202A-202H, or a portion thereof, to normalize the images. Transformation may be based on camera parameters associated with the image sensors (e.g. image sensors 102A-102F), such as extrinsic and/or intrinsic parameters. The rectification may optionally represent one or more layers of the backbone networks 200, in which values for the transformation are learned based on training data. [0064] discloses a lookup table may be used in combination with extrinsic and intrinsic camera parameters associated with the image sensor (e.g. image sensors 102A-102F). [0065] discloses each pixel may represent a ray out of an image, with the ray extending in the virtual camera space.
and at least one of (i) assigning, using the one or more processors, the grid to a map data structure or (ii) presenting, using the one or more processors and a display device the grid.
([0093] discloses the information (e.g. the outputs described herein) determined by the machine learning model described herein may be presented in a display of the vehicle. The information may be used to inform autonomous driving (e.g. used by a planning and/or navigation engine) and optionally be presented as a visualization for a driver or passenger to view. [0096] discloses the display may present graphical depictions of objects (e.g. VRU and/or non-VRU objects positioned about the vehicle 700).)
It would have been obvious to one of ordinary skill in the art at the time of effective filing to modify Kim’s disclosure to include the above limitations in order to leverage temporally separated camera frames and camera parameter/ray based virual camera geometry to form/update the scene representation and present that representation to a driver/passenger.
As to claims 3 & 16, Kim in view of Emmons discloses everything as disclosed in claim 1. In addition, Emmons discloses wherein the one or more circuits are to provide, as input to the neural network, a position representation of the at least one of the camera center or the one or more vectors as of at least one of a camera center of the first sensor image, a camera center of the second sensor image, a vector of a ray to a feature of the first sensor image, or a vector of a ray to a feature of the second sensor image. ([0064-0065] discloses a lookup table used with extrinsic and intrinsic camera parameters. [0065] discloses each pixel represents a ray out of an image wherein the ray extends in the virtual camera space.)
As to claims 5 & 18, Kim in view of Emmons discloses everything as disclosed in claim 1. In addition, Kim discloses wherein the grid comprises a two- dimensional representation of the scene in a top-down frame of reference, and the one or more circuits are to determine, for each feature of the one or more features, a polyline representing the feature, the polyline comprising a plurality of points indicating a plurality of line segments, wherein the one or more circuits are to assign the feature to the respective cell of the feature by assigning the polyline to the respective cell of the feature. ([0043] discloses in the specification, a top down representation of an environment is a representation from a top down view of the environment e.g. centered at the current position of the autonomous vehicle. [0053] discloses discretize into an orthogonal top down 2D grid map. [0055] discloses road elements annotated in the form of continuous curves. Also, see wherein we uniformly sample the road segment represented as an unordered set of points. [0060] discloses the system 100 identifies a grid representation of the top down representation that discretizes the top down representation into a plurality of plurality, with each of the plurality of points being assigned to a respective one of the pillars. [0064] discloses the system 100 discretizes the point set into an evenly spaced grid of shape MxN in the xy plane creating a set of MN pillars. )
As to claims 6 & 19, Kim in view of Emmons discloses everything as disclosed in claim 1. In addition, Kim discloses wherein the one or more circuits are to assign at least one of a height of the feature or a class of the feature to the respective cell of the feature. ([0047] discloses the map features can include features for one or more road elements such as solid double yellow lanes, dotted lanes, crosswalks, speed bumps, stop/yield signs, parking lines, solid single/double lanes, road edge boundaries as one hot vector. [0055] discloses type vector q encodes the road element type (e.g. lane, road, boundary).)
As to claims 7 & 20, Kim in view of Emmons discloses everything as disclosed in claim 1. In addition, Emmons discloses wherein the first sensor image and the second sensor image comprise camera images. ([0030-0031, 0051] discloses the image sensors 102A-102F may include cameras which are positioned about the vehicle 100. Image data may include images captured by any type of image sensor (e.g. camera).)
As to claim 8, Kim in view of Emmons discloses everything as disclosed in claim 1. In addition, Kim discloses wherein the processor is comprised in at least one of: a control system for an autonomous or semi-autonomous machine; a perception system for an autonomous or semi-autonomous machine; a system for performing simulation operations; a system for performing digital twin operations; a system for performing light transport simulation; a system for performing collaborative content creation for 3D assets; a system for performing deep learning operations; a system implemented using an edge device; a system implemented using a robot; a system for performing conversational Al operations; a system for generating synthetic data; a system incorporating one or more virtual machines (VMs); a system implemented at least partially in a data center; or a system implemented at least partially using cloud computing resources. ([0039] discloses the autonomous driving system may include an onboard computer, a planning system, generates a driving route and vehicle controls actuate steering braking throttle.)
As to claim 21, Kim in view of Emmons discloses everything as disclosed in claim 14. In addition, Emmons discloses conditioning the neural network by using the camera data to provide context during processing of the first sensor image and the second sensor image. ([0055, 0064-0065])
As to claim 22, Kim in view of Emmons discloses everything as disclosed in claim 1. In addition, Kim discloses use an encoder of neural network to receive the input. ([0011, 0059])
Emmons discloses wherein the one or more circuits are to use a featurizer of the neural network to determine the one or more features. ([0051, 0056])
As to claim 23, Kim in view of Emmons discloses everything as disclosed in claim 14. In addition, Emmons discloses using the camera data as an inductive prior for determining one or more locations of the one or more features. ([0055, 0064-0065])
Claims 2 & 15 are rejected under 35 U.S.C. 103 as being unpatentable over
Kim et al. (U.S. Publication 2022/0155096) in view of Emmons et al. (U.S. Publication 2023/0057509) as applied in claims 1 & 14 above further in view Musk et al. (U.S. Publication 2020/0257317)
As to claims 2 & 15, Kim in view of Emmons discloses everything as disclosed in claims 1 & 14 but is silent to determining, using the one or more processors, the one or more features using at least one of radio detection and ranging (RADAR) data, light detection and ranging (LIDAR) data, or ultrasound data corresponding to at least one of the first sensor image or the second sensor image.
However, Musk discloses determining, using the one or more processors, the one or more features using at least one of radio detection and ranging (RADAR) data, light detection and ranging (LIDAR) data, or ultrasound data corresponding to at least one of the first sensor image or the second sensor image. ([0036] discloses vision data supplemented with additional sensor data. [0079] discloses the model input includes camera sensor data and auxiliary sensor data, such as ultrasonic sensor data. [0053] discloses image data may be augmented based on auxiliary data (including radar, ultrasonic)
It would have been obvious to one of ordinary skill in the art at the time of effective filing to modify Kim in view of Emmons’s disclosure to include the above limitations in order to improve robustness/accuracy of feature determination when image cues are degraded (e.g. occlusion/low visibility) in order to improve the reliability of the resulting scene/grid representation.
Claims 4 & 17 are rejected under 35 U.S.C. 103 as being unpatentable over
Kim et al. (U.S. Publication 2022/0155096) in view of Emmons et al. (U.S. Publication 2023/0057509) as applied in claims 1 & 14 above further in view JAEGLE et al. (WO 2022/248727)
As to claims 4 & 17, Kim in view of Emmons discloses everything as disclosed in claims 1 & 14 but is silent to wherein the neural network comprises: a featurizer to convert image data of the first sensor image and the second sensor image into a plurality of tokens in a latent data space; an encoder cross-attention processor to process the plurality of tokens and a latent data representation maintained by one or more self-attention modules; and a decoder cross-attention processor to process an intermediate output of the neural network and the latent data representation to determine the grid of the scene.
However, JAEGLE discloses wherein the neural network comprises: a featurizer to convert image data of the first sensor image and the second sensor image into a plurality of tokens in a latent data space (Pages 9-10 disclose converting image inputs into embeddings, including per pixel embeddings, and forming latent embeddings. See For Example if the network input includes an image the data element embeddings can correspond to each pixel. The encoder block can generate a representation as a set of latent embeddings in a latent space.); an encoder cross-attention processor to process the plurality of tokens and a latent data representation maintained by one or more self-attention modules (Pages 10-11 discloses cross attention between input embeddings and latent embeddings then self attention blocks operating over the latent embeddings. ); and a decoder cross-attention processor to process an intermediate output of the neural network and the latent data representation to determine the grid of the scene (Page 10 discloses each dimension of the network output generated through cross attention query embedding over the set of latent embeddings.).
It would have been obvious to one of ordinary skill in the art at the time of effective filing to modify Kim in view of Emmons’s disclosure to include the above limitations in order to (A) improve scalability and efficiency (B) improve representation capacity (C) provide flexible decoding mechanism.
CONCLUSION
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Stephen P Coleman whose telephone number is (571)270-5931. The examiner can normally be reached Monday-Thursday 8AM-5PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Moyer can be reached at (571) 272-9523. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
Stephen P. Coleman
Primary Examiner
Art Unit 2675
/STEPHEN P COLEMAN/Primary Examiner, Art Unit 2675
Prosecution Insights