DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1-4, 9-14, 19, and 20 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Huang et al. (WO 2024237939), hereinafter Huang.
Regarding claim 1, Huang discloses a processor-implemented method with three-dimensional (3D) occupancy prediction learning (Paragraph [0008]: a method may comprise inputting, by a processor using one or more cameras of an ego object, image data of a space around the ego object into an artificial intelligence model; predicting, by the processor executing the artificial intelligence model, a surface attribute of one or more surfaces of the space around the ego object; and generating, by the processor, a dataset based on the one or more surfaces and their corresponding surface attribute; Paragraph [0104]: the Al model may perform the featurization discussed herein. In some other embodiments, a convolutional neural network may be used to featurize the image data. In one non-limiting example, as depicted, a RegNet (Regularized Neural Networks) may be used to transform the data into a BiFPN (Bi-directional Feature Pyramid Network). However, other protocols may also be used. In some other embodiments, a transformer may be used to featurize the image data), the method comprising: extracting multi-scale image feature vectors from received two-dimensional (2D) image data (Paragraphs [0125]-[0127]: The images received from the ego’s cameras may include a 2D representation of the ego’s surroundings. This representation is sometimes referred to as a 2D or flat lattice. The flat lattice may be transformed into different nodes having particular X-axis and Y-axis coordinate values. Using the method 201, the Al model may predict a Z-axis coordinate value for each node within the flat lattice. Specifically, using the method 201, the Al model may predict a feature vector for each point within image data having distinct X-axis and Y-axis coordinate values. As used herein, Z-axis coordinate values for each point or node may represent that point’s elevation relative to a flat surface having a 0 elevation in the world…Using the feature vectors for each node, the Al model may generate a mesh representation that corresponds to the ego’s surroundings. A mesh, as used herein, may refer to a series of interconnected nodes representing the ego’s surroundings where each node includes X, Y, and Z-axis coordinate values. Each node may also include data indicating its attributes and categories; Paragraph [0134]: the Al model may be configured to ingest image data and generate a lattice having various nodes where each node has a respective feature vector including X and Y-axis coordinate values (identified via the image data) and a Z-axis coordinate value predicted by the Al model. The Al model may also predict one or more attributes for each node. For instance, a particular node may include a feature vector that includes a predicted elevation (e.g., 1 meter above ego). Additionally, the Al model may predict that the node is a road node (because the corresponding pixel is predicted to be a driving surface) and the node has paint on it and the paint is yellow); generating a local cluster feature vector by clustering the extracted multi-scale image feature vectors (Paragraphs [0131]-[0134]: Using the semantic segmentation model, the analytics server may filter down the points and cluster them into their respective category (e.g., pixels that represent a sidewalk, pixels that represent a dirt road or an asphalt road). The analytics server may analyze different image data at different timestamps…After executing the semantic segmentation model, the point cloud may be segmented in accordance with their corresponding image data and/or their attributes (as predicted by the semantic segmentation model). As a result, points relevant to a particular surface and the image data relevant to the same surface surrounding the ego may be identified and isolated. The analytics server may then fit a mesh surface on the isolated data points. This may be because the Al model may execute more efficiently using a smoothed surface, which may be more indicative of the reality. Effectively, the mesh fitting may de-noise the data and provide a more realistic representation of the surfaces surrounding the ego. The fitted surface may be used as ground truth for training purposes… the Al model may be configured to ingest image data and generate a lattice having various nodes where each node has a respective feature vector including X and Y-axis coordinate values (identified via the image data) and a Z-axis coordinate value predicted by the Al model. The Al model may also predict one or more attributes for each node. For instance, a particular node may include a feature vector that includes a predicted elevation (e.g., 1 meter above ego). Additionally, the Al model may predict that the node is a road node (because the corresponding pixel is predicted to be a driving surface) and the node has paint on it and the paint is yellow); mapping the local cluster feature vector to a 3D space through an attention operation using a learnable voxel query (Paragraphs [0106]-[0117]: transformer may ingest three separate inputs: image key, image value, and 3D queries. The image key and image value may refer to attributes associated with the 2D image data received from the ego. For instance, these values may be outputted via image featurization (step 220). The transformer may also use an image query from the 3D space. The depicted spatial attention module may use a 3D query to analyze the 2D image key and image value. As depicted, the BiFPNs generated in the step 220 may be aggregated into a multi-camera query embedding and may be used to perform 3D spatial queries. In some embodiments, each voxel may have its own query. Using the 3D spatial query, the analytics server may identify a region within the 2D featurized image corresponding to a particular portion of the 3D representation. The identified region within the featurized image may then be analyzed to transform the multicamera image data into a 3D representation of each voxel, which may produce a 3D representation of the ego’s surroundings. Accordingly, the depicted spatial attention module may output a single 3D vector space representing the ego’s surroundings. This, in effect, moves all the image data generated by all camera feeds into a top-down space or a 3D space representation of the ego’s surroundings…or alternatively, the analytics server may decode a sub-voxel value to identify the shape of the sub-voxels (inside of an occupied voxel). For instance, if a voxel is half occupied, the analytics server may define a set of sub-voxels and use the methods discussed herein to identify volume outputs for the sub-voxels. When the sub-voxels are aggregated (back into the original voxel), the analytics server may determine a shape for the voxel. For instance, each voxel may have eight vertices. In some embodiments, each vertex can be analyzed separately and have its embeddings. As a result, any point within each vertex of the voxel can be queried separately. Therefore, in this “continuous resolution” approach, the analytics server may not define a size for the sub-voxel. In some embodiments, the analytics server may use a multi-variant interpolation (e.g., trilinear interpolation) protocol to estimate the occupancy status of each sub-voxel and/or any point within each vertex…Additionally or alternatively, the analytics server may generate a map corresponding to the predicted occupancy status of different voxels. In a non-limiting example, the analytics server may use a multi-view 3D reconstruction protocol to visualize each voxel and its occupancy status. A non-limiting example of the map or occupancy map is presented in FIGS. 3A-B (e.g., a simulation 350). In some embodiments, the simulation 350 may be displayed on a user interface of an ego. The simulation 350 may illustrate camera feeds 300 depicted in FIG. 3A. The camera feeds 300 represent image data received from eight different cameras of an ego (whether in real-time or near real-time). Specifically, the camera feed 300 may include camera feeds 310a-c received from three different front-facing cameras of the ego; camera feeds 320a-b received from two different right-side-facing cameras of the ego; camera feeds 330a-b received from two different left-side-facing cameras of the ego; and camera feed 340 received from a rear-facing camera of the ego); decoding a 3D voxel query generated according to the mapping result (Paragraph [0112]: the analytics server may decode a sub-voxel value to identify the shape of the sub-voxels (inside of an occupied voxel). For instance, if a voxel is half occupied, the analytics server may define a set of sub-voxels and use the methods discussed herein to identify volume outputs for the sub-voxels. When the sub-voxels are aggregated (back into the original voxel), the analytics server may determine a shape for the voxel. For instance, each voxel may have eight vertices. In some embodiments, each vertex can be analyzed separately and have its embeddings. As a result, any point within each vertex of the voxel can be queried separately. Therefore, in this “continuous resolution” approach, the analytics server may not define a size for the sub-voxel. In some embodiments, the analytics server may use a multi-variant interpolation (e.g., trilinear interpolation) protocol to estimate the occupancy status of each sub-voxel and/or any point within each vertex); and predicting a 3D occupancy state and a semantic class for a space, based on the decoding result (Paragraphs [0117]-[0119]: the analytics server may generate a map corresponding to the predicted occupancy status of different voxels. In a non-limiting example, the analytics server may use a multi-view 3D reconstruction protocol to visualize each voxel and its occupancy status. A non-limiting example of the map or occupancy map is presented in FIGS. 3A-B (e.g., a simulation 350). In some embodiments, the simulation 350 may be displayed on a user interface of an ego. The simulation 350 may illustrate camera feeds 300 depicted in FIG. 3A…simulated mass 370b (e.g., a vehicle) may have a second color indicating that it is a parked or stationary vehicle. In contrast, simulated mass 370a (e.g., another vehicle) may have a third color and/or other visual characteristics indicating that it is predicted to be moving. [0120] Additionally or alternatively, the analytics server may transmit the generated map to a downstream software application or another server. The predicted results may be further analyzed and used in various models and/or algorithms to perform various actions. For instance, a software model or a processor associated with the autonomous navigation system of the ego may receive the occupancy data predicted by the trained Al model, according to which navigational decisions may be made).
Regarding claim 2, Huang discloses the method of claim 1, wherein the attention operation reflects clustered information in the learnable voxel query by performing aggregate and dispatch (Paragraph [0106]: transformer may ingest three separate inputs: image key, image value, and 3D queries. The image key and image value may refer to attributes associated with the 2D image data received from the ego. For instance, these values may be outputted via image featurization (step 220). The transformer may also use an image query from the 3D space. The depicted spatial attention module may use a 3D query to analyze the 2D image key and image value. As depicted, the BiFPNs generated in the step 220 may be aggregated into a multi-camera query embedding and may be used to perform 3D spatial queries. In some embodiments, each voxel may have its own query. Using the 3D spatial query, the analytics server may identify a region within the 2D featurized image corresponding to a particular portion of the 3D representation. The identified region within the featurized image may then be analyzed to transform the multicamera image data into a 3D representation of each voxel, which may produce a 3D representation of the ego’s surroundings. Accordingly, the depicted spatial attention module may output a single 3D vector space representing the ego’s surroundings. This, in effect, moves all the image data generated by all camera feeds into a top-down space or a 3D space representation of the ego’s surroundings; Paragraph [0112]: the analytics server may decode a sub-voxel value to identify the shape of the sub-voxels (inside of an occupied voxel). For instance, if a voxel is half occupied, the analytics server may define a set of sub-voxels and use the methods discussed herein to identify volume outputs for the sub-voxels. When the sub-voxels are aggregated (back into the original voxel), the analytics server may determine a shape for the voxel. For instance, each voxel may have eight vertices. In some embodiments, each vertex can be analyzed separately and have its embeddings. As a result, any point within each vertex of the voxel can be queried separately. Therefore, in this “continuous resolution” approach, the analytics server may not define a size for the sub-voxel. In some embodiments, the analytics server may use a multi-variant interpolation (e.g., trilinear interpolation) protocol to estimate the occupancy status of each sub-voxel and/or any point within each vertex).
Regarding claim 3, Huang discloses the method of claim 1, further comprising training networks for 3D occupancy prediction learning by using the 3D voxel query in 2D image segmentation supervised learning (Paragraphs [0087]-[0088]: the analytics server 110a may use a supervised method of training. For instance, using the ground truth and the visual data received, the Al model(s) 110c may train itself, such that it can predict an occupancy status for a voxel using only an image of that voxel. As a result, when trained, the Al model(s) 110c may receive a camera feed, analyze the camera feed, and determine an occupancy status for each voxel within the camera feed… analytics server 110a may feed the series of training datasets to the Al model(s) 110c and obtain a set of predicted outputs (e.g., predicted occupancy status). The analytics server 110a may then compare the predicted data with the ground truth data to determine a difference and train the Al model(s) 110c by adjusting the Al model’s 110c internal weights and parameters proportional to the determined difference according to a loss function. The analytics server 110a may train the Al model(s) 110c in a similar manner until the trained Al model’s 110c prediction is accurate to a certain threshold; Paragraphs [0131]-[0133]: Using the semantic segmentation model, the analytics server may filter down the points and cluster them into their respective category (e.g., pixels that represent a sidewalk, pixels that represent a dirt road or an asphalt road). The analytics server may analyze different image data at different timestamps. [0132] After executing the semantic segmentation model, the point cloud may be segmented in accordance with their corresponding image data and/or their attributes (as predicted by the semantic segmentation model). As a result, points relevant to a particular surface and the image data relevant to the same surface surrounding the ego may be identified and isolated. The analytics server may then fit a mesh surface on the isolated data points. This may be because the Al model may execute more efficiently using a smoothed surface, which may be more indicative of the reality. Effectively, the mesh fitting may de-noise the data and provide a more realistic representation of the surfaces surrounding the ego. The fitted surface may be used as ground truth for training purposes…Al model may be trained using the image data received from the egos and the ground truth, such that, when trained, the Al model may not need any sensor data to analyze the image data received from an ego. Effectively, using this particular training paradigm, the Al model may correlate how pixels associated with a particular surface having particular attributes (e.g., uphill dirt road having white paint) are represented. Therefore, the Al model (at inference time) may only utilize image data and not need other sensor data).
Regarding claim 4, Huang discloses the method of claim 3, wherein the training of the networks comprises: obtaining an encoded 3D voxel query from the 3D voxel query and the extracted multi-scale image feature vectors (Paragraphs [0105]-[0106]: After the image data is encoded/featurized, a transformer may be used to change the image data from 2D images into 3D images (step 230). As discussed herein, in an example configuration, there may be eight distinct cameras in communication with the ego. As a result, the image data may include eight distinct camera feeds (one feed corresponding to each camera or other sensor) and may include overlapping views. The transformer may aggregate these separate camera feeds and generate one or more 3D representations using the received camera feeds…identified region within the featurized image may then be analyzed to transform the multicamera image data into a 3D representation of each voxel, which may produce a 3D representation of the ego’s surroundings. Accordingly, the depicted spatial attention module may output a single 3D vector space representing the ego’s surroundings. This, in effect, moves all the image data generated by all camera feeds into a top-down space or a 3D space representation of the ego’s surroundings; Paragraph [0125]: images received from the ego’s cameras may include a 2D representation of the ego’s surroundings. This representation is sometimes referred to as a 2D or flat lattice. The flat lattice may be transformed into different nodes having particular X-axis and Y-axis coordinate values. Using the method 201, the Al model may predict a Z-axis coordinate value for each node within the flat lattice. Specifically, using the method 201, the Al model may predict a feature vector for each point within image data having distinct X-axis and Y-axis coordinate values. As used herein, Z-axis coordinate values for each point or node may represent that point’s elevation relative to a flat surface having a 0 elevation in the world); and outputting an attention segmentation map based on a deformable attention map derived from the encoded 3D voxel query (Paragraphs [0106]-[0117]: transformer may ingest three separate inputs: image key, image value, and 3D queries. The image key and image value may refer to attributes associated with the 2D image data received from the ego. For instance, these values may be outputted via image featurization (step 220). The transformer may also use an image query from the 3D space. The depicted spatial attention module may use a 3D query to analyze the 2D image key and image value. As depicted, the BiFPNs generated in the step 220 may be aggregated into a multi-camera query embedding and may be used to perform 3D spatial queries. In some embodiments, each voxel may have its own query. Using the 3D spatial query, the analytics server may identify a region within the 2D featurized image corresponding to a particular portion of the 3D representation. The identified region within the featurized image may then be analyzed to transform the multicamera image data into a 3D representation of each voxel, which may produce a 3D representation of the ego’s surroundings. Accordingly, the depicted spatial attention module may output a single 3D vector space representing the ego’s surroundings. This, in effect, moves all the image data generated by all camera feeds into a top-down space or a 3D space representation of the ego’s surroundings…or alternatively, the analytics server may decode a sub-voxel value to identify the shape of the sub-voxels (inside of an occupied voxel). For instance, if a voxel is half occupied, the analytics server may define a set of sub-voxels and use the methods discussed herein to identify volume outputs for the sub-voxels. When the sub-voxels are aggregated (back into the original voxel), the analytics server may determine a shape for the voxel. For instance, each voxel may have eight vertices. In some embodiments, each vertex can be analyzed separately and have its embeddings. As a result, any point within each vertex of the voxel can be queried separately. Therefore, in this “continuous resolution” approach, the analytics server may not define a size for the sub-voxel. In some embodiments, the analytics server may use a multi-variant interpolation (e.g., trilinear interpolation) protocol to estimate the occupancy status of each sub-voxel and/or any point within each vertex…Additionally or alternatively, the analytics server may generate a map corresponding to the predicted occupancy status of different voxels. In a non-limiting example, the analytics server may use a multi-view 3D reconstruction protocol to visualize each voxel and its occupancy status. A non-limiting example of the map or occupancy map is presented in FIGS. 3A-B (e.g., a simulation 350). In some embodiments, the simulation 350 may be displayed on a user interface of an ego. The simulation 350 may illustrate camera feeds 300 depicted in FIG. 3A. The camera feeds 300 represent image data received from eight different cameras of an ego (whether in real-time or near real-time). Specifically, the camera feed 300 may include camera feeds 310a-c received from three different front-facing cameras of the ego; camera feeds 320a-b received from two different right-side-facing cameras of the ego; camera feeds 330a-b received from two different left-side-facing cameras of the ego; and camera feed 340 received from a rear-facing camera of the ego).
Regarding claim 9, Huang discloses the method of claim 1, wherein the 2D image data comprises image data obtained from a multi-view camera (Paragraph [0105]: there may be eight distinct cameras in communication with the ego. As a result, the image data may include eight distinct camera feeds (one feed corresponding to each camera or other sensor) and may include overlapping views. The transformer may aggregate these separate camera feeds and generate one or more 3D representations using the received camera feeds).
Regarding claim 10, Huang discloses a non-transitory computer-readable storage medium storing instructions that, when executed by one or more processors, configure the one or more processors to perform the method of claim 1 (See claim 1; Paragraph [0154]: the functions may be stored as one or more instructions or code on a non-transitory, computer-readable, or processor-readable storage medium. The steps of a method or algorithm disclosed herein may be embodied in a processor-executable software module, which may reside on a computer-readable or processor-readable storage medium. A non-transitory computer-readable or processor-readable media includes both computer storage media and tangible storage media that facilitates the transfer of a computer program from one place to another. A non-transitory, processor-readable storage media may be any available media that can be accessed by a computer. By way of example, and not limitation, such non-transitory, processor-readable media may comprise RAM, ROM, EEPROM, CD-ROM or other optical disk storage, magnetic disk storage or other magnetic storage devices, or any other tangible storage medium that may be used to store desired program code in the form of instructions or data structures and that can be accessed by a computer or processor).
Regarding claim 11, the limitations of this claim substantially correspond to the limitations of claim 1; thus they are rejected on similar grounds.
Regarding claim 12, the limitations of this claim substantially correspond to the limitations of claim 2; thus they are rejected on similar grounds.
Regarding claim 13, the limitations of this claim substantially correspond to the limitations of claim 3; thus they are rejected on similar grounds.
Regarding claim 14, the limitations of this claim substantially correspond to the limitations of claim 4; thus they are rejected on similar grounds.
Regarding claim 19, the limitations of this claim substantially correspond to the limitations of claim 9; thus they are rejected on similar grounds.
Regarding claim 20, Huang discloses a vehicle (Fig. 1A) comprising: one or more processors (Fig. 1A) configured to: drive a three-dimensional (3D) voxel query decoder trained in a 3D occupancy prediction learning process (Paragraphs [0112]-[0113]: the analytics server may decode a sub-voxel value to identify the shape of the sub-voxels (inside of an occupied voxel). For instance, if a voxel is half occupied, the analytics server may define a set of sub-voxels and use the methods discussed herein to identify volume outputs for the sub-voxels. When the sub-voxels are aggregated (back into the original voxel), the analytics server may determine a shape for the voxel. For instance, each voxel may have eight vertices. In some embodiments, each vertex can be analyzed separately and have its embeddings. As a result, any point within each vertex of the voxel can be queried separately. Therefore, in this “continuous resolution” approach, the analytics server may not define a size for the sub-voxel. In some embodiments, the analytics server may use a multi-variant interpolation (e.g., trilinear interpolation) protocol to estimate the occupancy status of each sub-voxel and/or any point within each vertex…volume output may also include 3D semantic data indicating the object occupying the voxel (or a group of voxels). The 3D semantic may indicate whether the voxel and/or a group of nearby voxels are occupied by a car, street curb, building, or other objects. The 3D semantic may also indicate whether the voxel is occupied by a static or moving mass. The 3D semantic data may be identified using various temporal attributes of the voxel. For instance, if a group of voxels is identified to be occupied by a mass, the collective shape of the voxels may indicate that the voxels belong to a vehicle. If, at a previous timestamp, the identified group of voxels (now known to be a vehicle) was identified as moving, then the group of voxels may have a 3D semantic indicating that the group of voxels belongs to a moving vehicle); and drive a 3D voxel decoder configured to predict a 3D occupancy state and a semantic class for a space from a two-dimensional (2D) image received from a camera included in the vehicle (Paragraph [0112]: the analytics server may decode a sub-voxel value to identify the shape of the sub-voxels (inside of an occupied voxel). For instance, if a voxel is half occupied, the analytics server may define a set of sub-voxels and use the methods discussed herein to identify volume outputs for the sub-voxels. When the sub-voxels are aggregated (back into the original voxel), the analytics server may determine a shape for the voxel. For instance, each voxel may have eight vertices. In some embodiments, each vertex can be analyzed separately and have its embeddings. As a result, any point within each vertex of the voxel can be queried separately. Therefore, in this “continuous resolution” approach, the analytics server may not define a size for the sub-voxel. In some embodiments, the analytics server may use a multi-variant interpolation (e.g., trilinear interpolation) protocol to estimate the occupancy status of each sub-voxel and/or any point within each vertex; Paragraphs [0125]-[0127]: The images received from the ego’s cameras may include a 2D representation of the ego’s surroundings. This representation is sometimes referred to as a 2D or flat lattice. The flat lattice may be transformed into different nodes having particular X-axis and Y-axis coordinate values. Using the method 201, the Al model may predict a Z-axis coordinate value for each node within the flat lattice. Specifically, using the method 201, the Al model may predict a feature vector for each point within image data having distinct X-axis and Y-axis coordinate values. As used herein, Z-axis coordinate values for each point or node may represent that point’s elevation relative to a flat surface having a 0 elevation in the world…Using the feature vectors for each node, the Al model may generate a mesh representation that corresponds to the ego’s surroundings. A mesh, as used herein, may refer to a series of interconnected nodes representing the ego’s surroundings where each node includes X, Y, and Z-axis coordinate values. Each node may also include data indicating its attributes and categories; Paragraph [0134]: the Al model may be configured to ingest image data and generate a lattice having various nodes where each node has a respective feature vector including X and Y-axis coordinate values (identified via the image data) and a Z-axis coordinate value predicted by the Al model. The Al model may also predict one or more attributes for each node. For instance, a particular node may include a feature vector that includes a predicted elevation (e.g., 1 meter above ego). Additionally, the Al model may predict that the node is a road node (because the corresponding pixel is predicted to be a driving surface) and the node has paint on it and the paint is yellow), wherein the training of the 3D voxel query decoder in the 3D occupancy prediction learning process comprises: extracting multi-scale image feature vectors from received 2D image data (Paragraphs [0125]-[0127]: The images received from the ego’s cameras may include a 2D representation of the ego’s surroundings. This representation is sometimes referred to as a 2D or flat lattice. The flat lattice may be transformed into different nodes having particular X-axis and Y-axis coordinate values. Using the method 201, the Al model may predict a Z-axis coordinate value for each node within the flat lattice. Specifically, using the method 201, the Al model may predict a feature vector for each point within image data having distinct X-axis and Y-axis coordinate values. As used herein, Z-axis coordinate values for each point or node may represent that point’s elevation relative to a flat surface having a 0 elevation in the world…Using the feature vectors for each node, the Al model may generate a mesh representation that corresponds to the ego’s surroundings. A mesh, as used herein, may refer to a series of interconnected nodes representing the ego’s surroundings where each node includes X, Y, and Z-axis coordinate values. Each node may also include data indicating its attributes and categories; Paragraph [0134]: the Al model may be configured to ingest image data and generate a lattice having various nodes where each node has a respective feature vector including X and Y-axis coordinate values (identified via the image data) and a Z-axis coordinate value predicted by the Al model. The Al model may also predict one or more attributes for each node. For instance, a particular node may include a feature vector that includes a predicted elevation (e.g., 1 meter above ego). Additionally, the Al model may predict that the node is a road node (because the corresponding pixel is predicted to be a driving surface) and the node has paint on it and the paint is yellow); generating a local cluster feature vector by clustering the extracted multi-scale image feature vectors (Paragraphs [0131]-[0134]: Using the semantic segmentation model, the analytics server may filter down the points and cluster them into their respective category (e.g., pixels that represent a sidewalk, pixels that represent a dirt road or an asphalt road). The analytics server may analyze different image data at different timestamps…After executing the semantic segmentation model, the point cloud may be segmented in accordance with their corresponding image data and/or their attributes (as predicted by the semantic segmentation model). As a result, points relevant to a particular surface and the image data relevant to the same surface surrounding the ego may be identified and isolated. The analytics server may then fit a mesh surface on the isolated data points. This may be because the Al model may execute more efficiently using a smoothed surface, which may be more indicative of the reality. Effectively, the mesh fitting may de-noise the data and provide a more realistic representation of the surfaces surrounding the ego. The fitted surface may be used as ground truth for training purposes… the Al model may be configured to ingest image data and generate a lattice having various nodes where each node has a respective feature vector including X and Y-axis coordinate values (identified via the image data) and a Z-axis coordinate value predicted by the Al model. The Al model may also predict one or more attributes for each node. For instance, a particular node may include a feature vector that includes a predicted elevation (e.g., 1 meter above ego). Additionally, the Al model may predict that the node is a road node (because the corresponding pixel is predicted to be a driving surface) and the node has paint on it and the paint is yellow); mapping the local cluster feature vector to a 3D space through an attention operation using a learnable voxel query (Paragraphs [0106]-[0117]: transformer may ingest three separate inputs: image key, image value, and 3D queries. The image key and image value may refer to attributes associated with the 2D image data received from the ego. For instance, these values may be outputted via image featurization (step 220). The transformer may also use an image query from the 3D space. The depicted spatial attention module may use a 3D query to analyze the 2D image key and image value. As depicted, the BiFPNs generated in the step 220 may be aggregated into a multi-camera query embedding and may be used to perform 3D spatial queries. In some embodiments, each voxel may have its own query. Using the 3D spatial query, the analytics server may identify a region within the 2D featurized image corresponding to a particular portion of the 3D representation. The identified region within the featurized image may then be analyzed to transform the multicamera image data into a 3D representation of each voxel, which may produce a 3D representation of the ego’s surroundings. Accordingly, the depicted spatial attention module may output a single 3D vector space representing the ego’s surroundings. This, in effect, moves all the image data generated by all camera feeds into a top-down space or a 3D space representation of the ego’s surroundings…or alternatively, the analytics server may decode a sub-voxel value to identify the shape of the sub-voxels (inside of an occupied voxel). For instance, if a voxel is half occupied, the analytics server may define a set of sub-voxels and use the methods discussed herein to identify volume outputs for the sub-voxels. When the sub-voxels are aggregated (back into the original voxel), the analytics server may determine a shape for the voxel. For instance, each voxel may have eight vertices. In some embodiments, each vertex can be analyzed separately and have its embeddings. As a result, any point within each vertex of the voxel can be queried separately. Therefore, in this “continuous resolution” approach, the analytics server may not define a size for the sub-voxel. In some embodiments, the analytics server may use a multi-variant interpolation (e.g., trilinear interpolation) protocol to estimate the occupancy status of each sub-voxel and/or any point within each vertex…Additionally or alternatively, the analytics server may generate a map corresponding to the predicted occupancy status of different voxels. In a non-limiting example, the analytics server may use a multi-view 3D reconstruction protocol to visualize each voxel and its occupancy status. A non-limiting example of the map or occupancy map is presented in FIGS. 3A-B (e.g., a simulation 350). In some embodiments, the simulation 350 may be displayed on a user interface of an ego. The simulation 350 may illustrate camera feeds 300 depicted in FIG. 3A. The camera feeds 300 represent image data received from eight different cameras of an ego (whether in real-time or near real-time). Specifically, the camera feed 300 may include camera feeds 310a-c received from three different front-facing cameras of the ego; camera feeds 320a-b received from two different right-side-facing cameras of the ego; camera feeds 330a-b received from two different left-side-facing cameras of the ego; and camera feed 340 received from a rear-facing camera of the ego); decoding a 3D voxel query generated according to the mapping result; and training the 3D voxel query decoder by predicting a 3D occupancy state and a semantic class for a space, based on the decoding result (Paragraph [0112]: the analytics server may decode a sub-voxel value to identify the shape of the sub-voxels (inside of an occupied voxel). For instance, if a voxel is half occupied, the analytics server may define a set of sub-voxels and use the methods discussed herein to identify volume outputs for the sub-voxels. When the sub-voxels are aggregated (back into the original voxel), the analytics server may determine a shape for the voxel. For instance, each voxel may have eight vertices. In some embodiments, each vertex can be analyzed separately and have its embeddings. As a result, any point within each vertex of the voxel can be queried separately. Therefore, in this “continuous resolution” approach, the analytics server may not define a size for the sub-voxel. In some embodiments, the analytics server may use a multi-variant interpolation (e.g., trilinear interpolation) protocol to estimate the occupancy status of each sub-voxel and/or any point within each vertex); and predicting a 3D occupancy state and a semantic class for a space, based on the decoding result (Paragraphs [0117]-[0119]: the analytics server may generate a map corresponding to the predicted occupancy status of different voxels. In a non-limiting example, the analytics server may use a multi-view 3D reconstruction protocol to visualize each voxel and its occupancy status. A non-limiting example of the map or occupancy map is presented in FIGS. 3A-B (e.g., a simulation 350). In some embodiments, the simulation 350 may be displayed on a user interface of an ego. The simulation 350 may illustrate camera feeds 300 depicted in FIG. 3A…simulated mass 370b (e.g., a vehicle) may have a second color indicating that it is a parked or stationary vehicle. In contrast, simulated mass 370a (e.g., another vehicle) may have a third color and/or other visual characteristics indicating that it is predicted to be moving. [0120] Additionally or alternatively, the analytics server may transmit the generated map to a downstream software application or another server. The predicted results may be further analyzed and used in various models and/or algorithms to perform various actions. For instance, a software model or a processor associated with the autonomous navigation system of the ego may receive the occupancy data predicted by the trained Al model, according to which navigational decisions may be made).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 5 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Huang, in view of Radhakrishnan et al. (US Pub. 2022/0292306), hereinafter Radhakrishnan.
Regarding claim 5, Huang discloses the method of claim 4
Huang does not explicitly disclose further comprising performing contrastive learning using the attention segmentation map and a pseudo mask.
However, Radhakrishnan teaches occupancy prediction (Paragraph [0141]), further comprising performing contrastive learning using the attention segmentation map and a pseudo mask (Fig. 5; Paragraphs [0081]-[0082]: FIG. 5 is a flow diagram showing a method 500 for generating a trained neural network that is robust to rotation, occlusion, truncation, other forms of obstruction, or other environmental factors in images or videos, in accordance with some embodiments of the present disclosure. In various embodiments, the method 500 for training the neural network, includes two training stages, a first training stage 502 and a second training stage 504. Furthermore, in one example, the first training stage 502 includes the method 100 as described above and the second training stage includes the method 200 as described above. During the first training stage 502, in block 506, the system performing the method 500 utilizes a neural network to perform inferencing on a dataset. As described above, the neural network may include a Mask R-CNN trained based at least in part on a curated dataset (which may be different from the dataset used to perform inferencing). Furthermore, in various embodiments, in block 508, pseudo-mask segmentation is generated as a result of performing inferencing on the dataset by the neural network. For example, the pseudo-mask segmentation generated by the neural network may include preliminary mask annotations as described above…in the second training stage 504 at block 510, a second neural network is trained based at least in part on the pseudo-mask segmentation and the dataset. As described above, the dataset, in various embodiments, is augmented or otherwise expanded by modifying the dataset (e.g., images or videos included in the dataset). For example, images of the dataset are modified and segmentation data (e.g., a mask) is generated for the modified dataset, which is then used to train the neural network. At block 508, the trained neural network is used to generate finer pseudo-mask segmentation. In an embodiment, the finer pseudo-mask segmentation includes the finer mask annotations described above in connection with FIG. 4. Furthermore, the finer pseudo-mask segmentation, in various embodiments, is used to train additional models that are robust to various environmental factors captured by at least modifying the dataset as described above). Radhakrishnan teaches that this will improve efficiency and accuracy of such processing (Paragraph [0495]). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Huang with the features of above as taught by Radhakrishnan so as to improve efficiency and accuracy as presented by Radhakrishnan.
Regarding claim 15, the limitations of this claim substantially correspond to the limitations of claim 5; thus they are rejected on similar grounds.
Claims 6-8 and 16-18 are rejected under 35 U.S.C. 103 as being unpatentable over Huang, in view of Pang et al. (KR Pub. 20250096744; translation attached), hereinafter Pang.
Regarding claim 6, Huang discloses the method of claim 1,
Huang does not explicitly disclose wherein the decoding of the 3D voxel query comprises performing voxel upsampling of the 3D voxel query by reflecting permutation invariance of a 3D space.
However, Pang teaches occupancy prediction (Abstract), further comprising wherein the decoding of the 3D voxel query comprises performing voxel upsampling of the 3D voxel query by reflecting permutation invariance of a 3D space (Fig. 6B; Paragraph [0167]: the context information may be any part, combination, and/or permutation of the exemplary context features mentioned above; Paragraph [0256]: Fig. 24 is a flowchart illustrating an exemplary process of context-aware voxel-based upsampling using pruning according to some embodiments. In some embodiments, the exemplary process (2400) may include the step (2402) of upsampling a first point cloud using initial upsampling to obtain a second point cloud. In some embodiments, the exemplary process (2400) may further include the step (2404) of associating features of the second point cloud with voxel-unit context information to obtain a third point cloud. In some embodiments, the exemplary process (2400) may further include the step (2406) of predicting the occupancy status of at least one voxel of the third point cloud. In some embodiments, the exemplary process (2400) may further include the step (2408) of generating a pruned point cloud by removing voxels of a third point cloud classified as empty according to a predicted occupancy state. In some embodiments, the initial upsampling may include nearest neighbor upsampling. In some embodiments, associating features may include connecting features). Pang teaches that this will allow for reduced computational cost (Paragraph [0246]). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Huang with the features of above as taught by Pang so as to allow for reduced computational cost as presented by Pang.
Regarding claim 7, Huang, in view of Pang teaches the method of claim 6, Huang discloses wherein the performing of the voxel upsampling comprises generating augmented 3D voxel queries by transforming the 3D voxel query into a plurality of viewpoints (Paragraph [0105]: there may be eight distinct cameras in communication with the ego. As a result, the image data may include eight distinct camera feeds (one feed corresponding to each camera or other sensor) and may include overlapping views. The transformer may aggregate these separate camera feeds and generate one or more 3D representations using the received camera feeds).
Regarding claim 8, Huang, in view of Pang teaches the method of claim 7, Huang discloses further comprising applying a consistency regularization technique via a transposed convolutional network to the augmented 3D voxel queries (Paragraph [0104]: the Al model may perform the featurization discussed herein. In some other embodiments, a convolutional neural network may be used to featurize the image data. In one non-limiting example, as depicted, a RegNet (Regularized Neural Networks) may be used to transform the data into a BiFPN (Bi-directional Feature Pyramid Network). However, other protocols may also be used. In some other embodiments, a transformer may be used to featurize the image data; Paragraph [0112]: the analytics server may decode a sub-voxel value to identify the shape of the sub-voxels (inside of an occupied voxel). For instance, if a voxel is half occupied, the analytics server may define a set of sub-voxels and use the methods discussed herein to identify volume outputs for the sub-voxels. When the sub-voxels are aggregated (back into the original voxel), the analytics server may determine a shape for the voxel. For instance, each voxel may have eight vertices. In some embodiments, each vertex can be analyzed separately and have its embeddings. As a result, any point within each vertex of the voxel can be queried separately. Therefore, in this “continuous resolution” approach, the analytics server may not define a size for the sub-voxel. In some embodiments, the analytics server may use a multi-variant interpolation (e.g., trilinear interpolation) protocol to estimate the occupancy status of each sub-voxel and/or any point within each vertex).
Regarding claim 16, the limitations of this claim substantially correspond to the limitations of claim 6; thus they are rejected on similar grounds.
Regarding claim 17, the limitations of this claim substantially correspond to the limitations of claim 7; thus they are rejected on similar grounds.
Regarding claim 18, the limitations of this claim substantially correspond to the limitations of claim 8; thus they are rejected on similar grounds.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Danielczuk et al. (US Pub. 2022/0152826) teaches voxel queries and occupancy determination.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MATTHEW D SALVUCCI whose telephone number is (571)270-5748. The examiner can normally be reached M-F: 7:30-4:00PT.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, XIAO WU can be reached at (571) 272-7761. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MATTHEW SALVUCCI/Primary Examiner, Art Unit 2613
Prosecution Insights