DETAILED ACTION
This Office action is in response to the Application filed on September 6, 2024. An action on the merits follows. Claims 1-20 are pending on the application.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Specification
The title of the invention is not descriptive. A new title is required that is clearly indicative of the invention to which the claims are directed.
Claim Objections
Claim 17 is objected to because of the following informalities: line 2 of claim 17 recites “radar sensor”. However, the acronym radar is undefined in the claims. To clarify that the acronym means radio detection and ranging, examiner suggests amending “radar sensor” in line 2 of claim 17 to “radio detection and ranging (radar) sensor”. Appropriate correction is required.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-7 and 9-20 are rejected under 35 U.S.C. 103 as being unpatentable over KEHL et al. (US PG Publication No. US 2021/0005018 A1), hereafter referred to as KEHL, in view of Hu et al. (“Where2comm: Communication-Efficient Collaborative Perception via Spatial Confidence Maps”), Applicant cited prior art furnished via IDS.
Regarding claim 1, KEHL discloses an apparatus for vehicle communications at a first vehicle (Par. [0002-4]: system and method for inferring a location of a three-dimensional (3D) object with respect to a location of an autonomous agent… Autonomous agents (e.g., vehicles, robots, drones, etc.) and semi-autonomous agents use machine vision for analyzing areas of interest in a surrounding environment… autonomous agents may rely on a trained convolutional neural network (CNN) to identify objects within areas of interest in an image of a surrounding environment. For example, a CNN may be trained to identify and track objects captured by one or more sensors… The sensors may be coupled to, or in communication with, a device, such as the autonomous agent; an apparatus for vehicle communications at a first vehicle (e.g. system and method for inferring a location of a three-dimensional (3D) object with respect to a location of an autonomous agent, including an autonomous vehicle (i.e. a first vehicle), for example, include a convolutional neural network (CNN) trained to identify and track objects captured by one or more sensors that are coupled to, or in communication with, the autonomous agent (i.e. an apparatus for vehicle communications at a first vehicle), as indicated above), for example), the apparatus comprising:
at least one memory (Par. [0009]: apparatus having a processor, a memory coupled with the processor, and instructions stored in the memory); and
at least one processor coupled to the at least one memory and configured (Par. [0008-9]: a non-transitory computer-readable medium with non-transitory program code recorded thereon is disclosed. The program code is for inferring a location of an object. The program code is executed by a processor and includes program code to extract features from sensor data obtained from a number of sensors of an autonomous vehicle… apparatus for inferring a location of an object. The apparatus having a processor, a memory coupled with the processor, and instructions stored in the memory. The instructions are operable, when executed by the processor, to cause the apparatus to extract features from sensor data obtained from a number of sensors of an autonomous vehicle) to:
generate, using one or more encoders, one or more first tensors based on first sensor data obtained from one or more first sensors of the first vehicle;
obtain one or more second tensors, wherein the one or more second tensors are associated with second sensor data obtained from one or more second sensors (Par. [0007-9]: extracting features from sensor data obtained from a number of sensors of an autonomous vehicle… encoding the features to a number of sensor space representations, each sensor space representation corresponding to a sensor feature space of one of the number of sensors… extract features from sensor data obtained from a number of sensors of an autonomous vehicle… encode the features to a number of sensor space representations, each sensor space representation corresponding to a sensor feature space of one of the number of sensors; Par. [0024-32]: agent may detect objects from outputs of various sensors… objects are localized based on features extracted from different sensor outputs… features are extracted from different sensor outputs. The extracted features may be matched, reshaped, and mapped to elements of a data structure. The data structure, such as a vector, corresponds to a feature space of a 3D representation of a spatial environment… elements of a vector may be processed to determine whether a cell contains features for object recognition… Using a trained artificial neural network, features of the non-empty cells are analyzed to identify one or more of a class of an object, orientation of the object, and/or distance to the object… 3D inference system of the agent 100 may extract features from objects of interest… The features may be encoded in a data structure, such as a tensor, as explained in detail herein. The reshape module may reshape the features obtained by the tensors and map the reshaped tensors to a vector corresponding to a feature space. The feature space represents the layout of cells of a 3D grid. As previously discussed, the 3D grid represents a spatial area within a vicinity of the agent 100; Par. [0038-46]: cells 204, 206 of each 3D grid 200, 202 may be represented as elements in a vector 220, 220… As is known to those of skill in the art, features extracted from an image are encoded in a 3D tensor. FIG. 2B illustrates an example of a 3D tensor 250 and a 2D image 252 according to aspects of the present disclosure. The 3D tensor 250 encodes both the spatial structure and features of the 2D image 252. The first two dimensions of the 3D tensor 250 are a height (H) and a width (W) that correspond to the height (H) and the width (W) of the 2D image 252. The third dimension (D) encodes a number of feature channels, one for each spatial location (e.g., pixel)… the 3D grids 200, 202 represent a different feature space in comparison to a feature space of a sensor output… The sensor outputs 302, 304, 306 are received at a feature extraction module 308. The feature extraction module 308 may be a convolutional neural network or another type of artificial neural network, such as a support vector machine (SVM), for extracting features from each sensor output 302, 304, 306. For example, the features from the first sensor output 302 are encoded in an RGB image tensor (FRGB). The features from the second sensor output 304 may be encoded in a bird's eye view (BEV) tensor (FBEV). Finally, the features from the third sensor output 306 may be encoded in a forward facing view (FFV) tensor (FFFV). Each tensor FRGB, FBEV, FFFV may have three dimensions (Height×Width×Depth)… As shown in FIG. 3, each sensor output 302, 304, 306 provides a different perspective of the agent's environment. As such, each tensor FRGB, FBEV, FFFV represents a different feature space; Par. [0071-72]: feature extractor 510 extracts (e.g., identifies) areas of interest from each frame of the data stream. For example, the feature extractor 510 may be trained to extract features of 3D objects… the feature exactor 510 identifies areas of attention based on the training… feature extractor 510 may encode each set of features in a multi-dimensional tensor. The reshape module 512 receives the multi-dimensional tensors from the feature extractor 510 to reshape the multi-dimensional tensor to a vector corresponding to a layout of a 3D grid. The reshape module 512 may be a fully connected layer of the CNN that has been trained to match, reshape, and combine the features from the multi-dimensional tensors to a single vector. The reshape module 512 outputs the vector corresponding to cells of the 3D grid; generate, using one or more encoders, one or more first tensors based on first sensor data obtained from one or more first sensors of the first vehicle;
obtain one or more second tensors, wherein the one or more second tensors are associated with second sensor data obtained from one or more second sensors (e.g. system and method for inferring a location of a 3D object with respect to a location of an autonomous agent, including an autonomous vehicle (i.e. a first vehicle), for example, include extracting features from sensor data obtained from a number of sensors of the autonomous vehicle (i.e. first, second, third... Nth sensor data obtained from one or more first, second, third... Nth sensors of the first vehicle), respectively, in which extracted features are encoded in data structures, such as 3D tensors (i.e. feature representations, vectors, voxels, etc.), for example, by using a convolutional neural network, or another type of artificial neural network (i.e. using one or more encoders), for extracting features from each sensor output, in which extracted features from a first sensor output are encoded in an RGB image tensor (i.e. generate, using one or more encoders, one or more first tensors based on first sensor data obtained from one or more first sensors of the first vehicle) and extracted features from a second s from a second sensor output are encoded in a bird's eye view (BEV) tensor, and extracted features from a third sensor output are encoded in a forward facing view (FFV) tensor, respectively (i.e. obtain one or more second tensors, wherein the one or more second tensors are associated with second sensor data obtained from one or more second sensors), as indicated above), for example);
generate an output based on the one or more first tensors and the one or more second tensors (Par. [0030-32]: extracted features may be transmitted to a reshape module of the 3D inference system… extracted features may be transmitted to the reshape module of the 3D inference system… The features may be encoded in a data structure, such as a tensor, as explained in detail herein. The reshape module may reshape the features obtained by the tensors and map the reshaped tensors to a vector corresponding to a feature space. The feature space represents the layout of cells of a 3D grid. As previously discussed, the 3D grid represents a spatial area within a vicinity of the agent 100; Par. [0041-48]: provide merge and reshape features from different feature spaces. The reshaped features may then be mapped to a feature space of a vector that represents a layout of a 3D grid. As such, aspects of the present disclosure improve 3D inference by providing a 3D inference system that infers 3D objects based on the output of multiple sensors regardless of the layout of a 3D grid… 3D inference system is trained to map features extracted from sensor outputs to a representation (e.g., vector) of a 3D grid. In one configuration, during an offline training step, the 3D inference system acquires features extracted from different sensor outputs (e.g., LIDAR output and 2D camera output). Furthermore, the 3D inference system is trained to generate merged features by matching, reshaping, and combining the features based on a layout of the 3D grid. The merged features are then mapped to cells of the 3D grid that are defined as elements of a vector. After mapping the merged features, the mapped features are classified by a trained neural network. The classified features may be used to infer 3D objects in a spatial area… the reshape module 310 is a fully connected layer that receives the tensors FRGB, FBEV, FFFV corresponding to each sensor output 302, 304, 306. The reshape module 310 performs a reshape (e.g., matrix multiplication) to output a 3D grid vector (V3D) corresponding to a layout of the 3D grid 360, 366. That is, the reshape module 310 includes a reshape function F( ) that receives tensors (FRGB, FBEV, FFFV) as an input x to generate an output y (V3D); generate an output based on the one or more first tensors and the one or more second tensors (e.g. system and method for inferring a location of a 3D object with respect to a location of an autonomous agent, including an autonomous vehicle, include extracting features from sensor data obtained from a number of sensors of the autonomous vehicle (i.e. first, second, third... Nth sensor data obtained from one or more first, second, third... Nth sensors of the first vehicle), respectively, in which extracted features are encoded in data structures, such as 3D tensors (i.e. feature representations, vectors, voxels, etc.), for example, by using a convolutional neural network, or another type of artificial neural network, for extracting features from each sensor output, in which extracted features from a first sensor output are encoded in an RGB image tensor and extracted features from a second sensor output are encoded in a bird's eye view (BEV) tensor, for example, and are used to merge and reshape the features obtained by the tensors and map the reshaped tensors to a vector corresponding to a feature space, for example, including a reshape function that receives encoded tensors, such as the RGB image tensor and the bird's eye view (BEV) tensor, as an input x to generate an output y (i.e. generate an output based on the one or more first tensors and the one or more second tensors), as indicated above), for example), but fails to teach the following as further recited in claim 1.
However, Hu teaches obtain one or more second tensors from a second vehicle, wherein the one or more second tensors are associated with second sensor data obtained from one or more second sensors of the second vehicle, the second sensor data comprising information associated with one or more regions obscured from the first sensor data (Pg. 1-2: Collaborative perception enables multiple agents to share complementary perceptual information with each other, promoting more holistic perception. It provides a new direction to fundamentally overcome a number of inevitable limitations of single-agent perception, such as occlusion and long-range issues. Related methods and systems are desperately needed in a broad range of real-world applications, such as vehicle-to-everything-communication-aided autonomous driving [1–3], multirobot warehouse automation system [4, 5] and multi-UAVs (unmanned aerial vehicles) for search and rescue [6–8] … spatial-confidence-aware communication strategy. The core idea is to enable a spatial confidence map for each agent, where each element reflects the perceptually critical level of a corresponding spatial area. Based on this map, agents decide which spatial area (where) to communicate about. That is, each agent offers spatially sparse, yet critical features to support other agents, and meanwhile requests complementary information from others through multi-round communication to perform efficient and mutually beneficial collaboration… a spatial confidence-aware message fusion module, which uses novel confidence-aware multi-head attention to fuse all messages received from other agents, upgrading the feature map for each agent; Pg. 4-5: observation encoder extracts feature maps from the sensor data. Where2comm accepts single/multimodality inputs, such as RGB images and 3D point clouds. This work adopts the feature representations in bird’s eye view (BEV), where all agents project their individual perceptual information to the same global coordinate system… All agents share the same BEV coordinate system. For the image input, Φenc(·) is followed by a warping function that transforms the extracted feature from front-view to BEV. For 3D point cloud input, we discretize 3D points as a BEV map and Φenc(·) extracts features in BEV. The extracted feature map is output to the spatial confidence generator and the message fusion module… The spatial confidence generator generates a spatial confidence map from the feature map of each agent. The spatial confidence map reflects the perceptually critical level of various spatial areas… During collaboration, areas with objects could help recover the miss-detected objects due to the limited view… Where2comm iteratively updates the feature map by aggregating information from other agents… Message packing determines what information should be included in the to-be sent message. The proposed message includes: i) a request map that indicates at which spatial areas the agent needs to know more; and ii) a spatially sparse, yet perceptually critical feature map… The request map of the ith agent is… negatively correlated with the spatial confidence map. The intuition is, for the locations with low confidence score, an agent is hard to tell if there is really no objects or it is just caused by the limited information (e.g. occlusion). Thus, the low confidence score indicates there could be missing information at that location. Requesting information at these locations from other agents could improve the current agent’s detection accuracy. The spatially sparse feature map are selected based on each agent’s spatial confidence map and the received request maps from others; Pg. 9: proposed spatial confidence map. In the scene, Drone 1’s view is occluded by a tall building. With Drone 2’s help, Drone 1 is able to detect through occlusion. Fig. 6 (a-d) shows Drone 1’s observation, spatial confidence map (1), binary selection matrix (2), and ego attention weight (3)… We see that the proposed spatial confidence map contributes to spatially sparse, yet perceptually critical message, which effectively helps Drone 1 detect occluded objects; obtain one or more second tensors from a second vehicle, wherein the one or more second tensors are associated with second sensor data obtained from one or more second sensors of the second vehicle, the second sensor data comprising information associated with one or more regions obscured from the first sensor data (e.g. spatial-confidence-aware communication strategy includes a spatial confidence map for each (first, second, third... Nth) agent that requests complementary information from other agents (i.e. from a second vehicle) through multi-round communication, such as vehicle-to-everything-communication-aided autonomous driving, to perform collaboration, for example, including overcoming a number of limitations of single-agent perception, such as occlusion limitations (i.e. one or more regions obscured from a first sensor data), for example, including an observation encoder that extracts feature maps (i.e. tensors, feature representations, etc.) from sensor data for each agent (one or more first, second, third... Nth tensors from a first, second, third... Nth vehicle), including feature representations in bird’s eye view (BEV), for example, where all agents project their individual perceptual information to the same global coordinate system to generate an extracted feature map (i.e. obtain one or more first, second, third... Nth tensors are associated with first, second, third... Nth sensor data obtained from one or more second sensors of a first, second, third... Nth vehicle), and during collaboration, help recover miss-detected objects due to a limited view (i.e. one or more regions obscured from the first, second, third... Nth sensor data), by requesting a map that indicates which spatial areas an agent needs to know more (i.e. obtain one or more second tensors from a second vehicle, wherein the one or more second tensors are associated with second sensor data obtained from one or more second sensors of the second vehicle, the second sensor data comprising information associated with one or more regions obscured from the first sensor data), as indicated above), for example).
KEHL and Hu are considered to be analogous art because they pertain to image processing applications related to autonomous driving. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify the system and method for inferring a location of a three-dimensional (3D) object with respect to a location of an autonomous agent by extracting features from sensor data obtained from a number of sensors of an autonomous agent, which are encoded in data structures, such as 3D tensors (as disclosed by KEHL), with obtain one or more second tensors from a second vehicle, wherein the one or more second tensors are associated with second sensor data obtained from one or more second sensors of the second vehicle, the second sensor data comprising information associated with one or more regions obscured from the first sensor data (as taught by Hu, Pg. 1-2, 4-5, 9) to overcome a number of limitations of single-agent perception, such as occlusion and long-range issues, to perform efficient and mutually beneficial collaboration, to fuse all messages received from other agents, upgrading the feature map for each agent, to help recover miss-detected objects due to limited view, and to help agents detect occluded objects (Hu, Pg. 1-2, 4-5, 9).
Regarding claim 2, claim 1 is incorporated and the combination of KEHL and Hu, as a whole, teaches the apparatus (KEHL, Par. [0002-4]), wherein the at least one processor is configured to determine a region of interest based on a view of at least one first sensor of the one or more first sensors (KEHL, Par. [0002-4]: system and method for inferring a location of a three-dimensional (3D) object with respect to a location of an autonomous agent… Autonomous agents (e.g., vehicles, robots, drones, etc.) and semi-autonomous agents use machine vision for analyzing areas of interest in a surrounding environment… autonomous agents may rely on a trained convolutional neural network (CNN) to identify objects within areas of interest in an image of a surrounding environment. For example, a CNN may be trained to identify and track objects captured by one or more sensors; Par. [0029-30]: 2D camera 108 captures a 2D image 120 that includes objects in the 2D camera's 108 field of view 114. The LIDAR sensor 106 may generate one or more output streams 122, 124. The first output stream 122 may include a 3D cloud point of objects in a first field of view, such as a 360° field of view 112 (e.g., bird's eye view). The second output stream 124 may include a 3D cloud point of objects in a second field of view, such as a forward facing field of view 126… The 2D image 120 captured by the 2D camera includes a 2D image of the first vehicle 104, as the first vehicle 104 is in the 2D camera's 108 field of view 114. The 3D inference system of the agent 100 may extract features from objects of interest in the 2D image 120. For example, an artificial neural network, such as a convolutional neural network, of the 3D inference system extracts features of the first vehicle 104; Par. [0071-72]: feature extractor 510 extracts (e.g., identifies) areas of interest from each frame of the data stream. For example, the feature extractor 510 may be trained to extract features of 3D objects… the feature exactor 510 identifies areas of attention based on the training… feature extractor 510 may encode each set of features in a multi-dimensional tensor. The reshape module 512 receives the multi-dimensional tensors from the feature extractor 510 to reshape the multi-dimensional tensor to a vector corresponding to a layout of a 3D grid. The reshape module 512 may be a fully connected layer of the CNN that has been trained to match, reshape, and combine the features from the multi-dimensional tensors to a single vector. The reshape module 512 outputs the vector corresponding to cells of the 3D grid; wherein the at least one processor is configured to determine a region of interest based on a view of at least one first sensor of the one or more first sensors (e.g. system and method for inferring a location of a 3D object with respect to a location of an autonomous agent, including an autonomous vehicle, include extracting features from sensor data obtained from a number of sensors of the autonomous vehicle (i.e. first, second, third... Nth sensor data obtained from one or more first, second, third... Nth sensors of the first vehicle), respectively, for example, including a trained convolutional neural network (CNN) to identify objects within areas of interest in an image of a surrounding environment with respect to each sensor field of view (i.e. determine a region of interest based on a view of at least one first sensor of the one or more first sensors), as indicated above), for example).
Regarding claim 3, claim 2 is incorporated and the combination of KEHL and Hu, as a whole, teaches the apparatus (KEHL, Par. [0002-4]), wherein the at least one processor is configured to output, for transmission to the second vehicle, a request for a sensor listing comprising the one or more second sensors with a view of the region of interest (Hu, Pg. 1-2: Collaborative perception enables multiple agents to share complementary perceptual information with each other, promoting more holistic perception. It provides a new direction to fundamentally overcome a number of inevitable limitations of single-agent perception, such as occlusion and long-range issues. Related methods and systems are desperately needed in a broad range of real-world applications, such as vehicle-to-everything-communication-aided autonomous driving [1–3], multirobot warehouse automation system [4, 5] and multi-UAVs (unmanned aerial vehicles) for search and rescue [6–8] … spatial-confidence-aware communication strategy. The core idea is to enable a spatial confidence map for each agent, where each element reflects the perceptually critical level of a corresponding spatial area. Based on this map, agents decide which spatial area (where) to communicate about. That is, each agent offers spatially sparse, yet critical features to support other agents, and meanwhile requests complementary information from others through multi-round communication to perform efficient and mutually beneficial collaboration… a spatial confidence-aware message fusion module, which uses novel confidence-aware multi-head attention to fuse all messages received from other agents, upgrading the feature map for each agent; Pg. 4-5: observation encoder extracts feature maps from the sensor data. Where2comm accepts single/multimodality inputs, such as RGB images and 3D point clouds. This work adopts the feature representations in bird’s eye view (BEV), where all agents project their individual perceptual information to the same global coordinate system, avoiding complex coordinate transformations and supporting better shared cross-agent collaboration… spatial confidence generator generates a spatial confidence map from the feature map of each agent. The spatial confidence map reflects the perceptually critical level of various spatial areas. Intuitively, for object detection task, the areas that contain objects are more critical than background areas. During collaboration, areas with objects could help recover the miss-detected objects due to the limited view; and background areas could be omitted to save the precious bandwidth. So we represent the spatial confidence map with the detection confidence map, where the area with high perceptually critical level is the area that contains an object with a high confidence score. To implement, we use a detection decoder structure to produce the detection confidence map… With the guidance of spatial confidence maps, the proposed communication module packs compact messages with spatially sparse feature maps and transmits messages through a sparsely-connected communication graph… To reduce the communication bandwidth without affecting perception, we leverage the spatial confidence map to select the most informative spatial areas in the feature map (where to communicate) and decide the most beneficial collaboration partners (who to communicate)… Message packing determines what information should be included in the to-be sent message. The proposed message includes: i) a request map that indicates at which spatial areas the agent needs to know more; and ii) a spatially sparse, yet perceptually critical feature map… The request map of the ith agent is… negatively correlated with the spatial confidence map. The intuition is, for the locations with low confidence score, an agent is hard to tell if there is really no objects or it is just caused by the limited information (e.g. occlusion). Thus, the low confidence score indicates there could be missing information at that location. Requesting information at these locations from other agents could improve the current agent’s detection accuracy. The spatially sparse feature map are selected based on each agent’s spatial confidence map and the received request maps from others … Communication graph construction targets to identify when and who to communicate to avoid unnecessary communication that wastes the bandwidth… the necessity of communication between the ith and the jth agents is simply measured by the overlap between the information that the ith agent has and the information that the jth agent needs. With the help of the spatial confidence map and the request map, we construct a more interpretable communication graph; Pg. 17: Spatial confidence-aware message fusion targets to augment the feature of each agent by aggregating the received messages from the other agents. To achieve this, we adopt a transformer architecture, which leverages multi-head attention to fuse the corresponding features from multiple agents at each individual spatial location. The key technical design is to include the spatial confidence maps of all the agents to promote cross-agent attention learning; Pg. 14: fine-grained spatial-aware communication strategy, where each agent can decide where to communicate and pack messages only related to the most perceptually critical spatial areas. This strategy not only enables more precise support for other agents, but also more targeted request from other agents in multi-round communication; Pg. 21: multi-round communication employs a request map in the second and subsequent communication rounds to denote the spatial area where each agent needs more information, which enables more targeted and efficient communication; wherein the at least one processor is configured to output, for transmission to the second vehicle, a request for a sensor listing comprising the one or more second sensors with a view of the region of interest (e.g. spatial-confidence-aware communication strategy includes a spatial confidence map for each (first, second, third... Nth) agent that requests complementary information from other agents (i.e. from a second vehicle) through multi-round communication, such as vehicle-to-everything-communication-aided autonomous driving, to perform collaboration, for example, including a collaborative perception which enables multiple agents to share complementary perceptual information with each other to overcome a number of limitations of single-agent perception, such as occlusion limitations, for example, including an observation encoder that extracts feature maps (i.e. tensors, feature representations, etc.) from sensor data for each agent (i.e. a sensor listing comprising one or more first, second, third... Nth tensors from a first, second, third... Nth vehicle), for example, in order to select the most informative spatial areas (i.e. a view of a region of interest) in the feature maps (where to communicate) and decide the most beneficial collaboration partners (who to communicate) by requesting complementary information from others through multi-round communication to help recover miss-detected objects due to a limited view, for example, including message packing which determines what information should be included in the to-be sent message (i.e. output, for transmission to other vehicles), including a request map that indicates which spatial areas the agent needs to know more (i.e. output, for transmission to the second vehicle, a request for a sensor listing comprising the one or more second sensors with a view of the region of interest), for example, and requesting information at these locations from other agents improves the current (i.e. first vehicle) agent’s detection accuracy, as indicated above), for example).
The same motivation to combine above-mentioned teachings applies, as previously indicated in claim 1.
Regarding claim 4, claim 3 is incorporated and the combination of KEHL and Hu, as a whole, teaches the apparatus (KEHL, Par. [0002-4]), wherein the at least one processor is configured to:
receive, from the second vehicle, the sensor listing; and
output, for transmission to the second vehicle based on the sensor listing, a request to subscribe to the one or more second sensors in the sensor listing (Hu, Pg. 4-5: proposed communication module packs compact messages with spatially sparse feature maps and transmits messages through a sparsely-connected communication graph … Communication graph construction targets to identify when and who to communicate to avoid unnecessary communication that wastes the bandwidth… the necessity of communication between the ith and the jth agents is simply measured by the overlap between the information that the ith agent has and the information that the jth agent needs. With the help of the spatial confidence map and the request map, we construct a more interpretable communication graph… For the initial communication round, every agent in the system is not aware of other agents yet. To activate the collaboration, we construct a fully-connected communication graph. Every agent will broadcast its message to the rest of the system. For the subsequent communication rounds, we examine if the communication between agent i and agent j is necessary; receive, from the second vehicle, the sensor listing; and
output, for transmission to the second vehicle based on the sensor listing, a request to subscribe to the one or more second sensors in the sensor listing (e.g. spatial-confidence-aware communication strategy includes a spatial confidence map for each (first, second, third... Nth) agent that requests complementary information from other agents (i.e. from a second vehicle) through multi-round communication, such as vehicle-to-everything-communication-aided autonomous driving, to perform collaboration, for example, including a collaborative perception which enables multiple agents to share complementary perceptual information with each other to overcome a number of limitations of single-agent perception, such as occlusion limitations, for example, including an observation encoder that extracts feature maps (i.e. tensors, feature representations, etc.) from sensor data for each agent (i.e. a sensor listing comprising one or more first, second, third... Nth tensors from a first, second, third... Nth vehicle), for example, in order to select the most informative spatial areas (i.e. a view of a region of interest) in the feature maps (where to communicate) and decide the most beneficial collaboration partners (who to communicate) by requesting and receiving complementary information from others through multi-round communication (i.e. receive, from the second vehicle, the sensor listing of the one or more second sensors) to help recover miss-detected objects due to a limited view, for example, including a communication module which transmits (i.e. outputs) messages through a sparsely-connected (i.e. established, subscribed, etc.) communication graph to identify when and who to communicate (i.e. output, for transmission to the second vehicle based on the sensor listing, a request to subscribe to the one or more second sensors in the sensor listing) to avoid unnecessary communication that wastes the bandwidth in order to help recover miss-detected objects due to a limited view, for example, and requesting information at these locations from other agents improves the current (i.e. first vehicle) agent’s detection accuracy, as indicated above), for example).
The same motivation to combine above-mentioned teachings applies, as previously indicated in claim 1.
Regarding claim 5, claim 4 is incorporated and the combination of KEHL and Hu, as a whole, teaches the apparatus (KEHL, Par. [0002-4]), wherein the at least one processor is configured to output, for transmission to a cloud service (KEHL, Par. [0063]: 3D inference system 500 includes a transceiver 516 coupled to the processor 520, the sensor module 502, an artificial neural network module 508, the communication module 52… the transceiver 516 may transmit driving statistics and information from the artificial neural network module 508 to a server; Par. [0097]: modules and/or other appropriate means for performing the methods and techniques described herein can be downloaded and/or otherwise obtained by a user terminal and/or base station as applicable. For example, such a device can be coupled to a server to facilitate the transfer of means for performing the methods described herein), a request for a vehicle listing, the vehicle listing comprising a plurality of vehicles with sensors having views of the region of interest, wherein the plurality of vehicles of the vehicle listing comprises the second vehicle (Hu, Pg. 1-2: Collaborative perception enables multiple agents to share complementary perceptual information with each other, promoting more holistic perception. It provides a new direction to fundamentally overcome a number of inevitable limitations of single-agent perception, such as occlusion and long-range issues. Related methods and systems are desperately needed in a broad range of real-world applications, such as vehicle-to-everything-communication-aided autonomous driving [1–3], multirobot warehouse automation system [4, 5] and multi-UAVs (unmanned aerial vehicles) for search and rescue [6–8] … spatial-confidence-aware communication strategy. The core idea is to enable a spatial confidence map for each agent, where each element reflects the perceptually critical level of a corresponding spatial area. Based on this map, agents decide which spatial area (where) to communicate about. That is, each agent offers spatially sparse, yet critical features to support other agents, and meanwhile requests complementary information from others through multi-round communication to perform efficient and mutually beneficial collaboration… a spatial confidence-aware message fusion module, which uses novel confidence-aware multi-head attention to fuse all messages received from other agents, upgrading the feature map for each agent; Pg. 4-5: observation encoder extracts feature maps from the sensor data. Where2comm accepts single/multimodality inputs, such as RGB images and 3D point clouds. This work adopts the feature representations in bird’s eye view (BEV), where all agents project their individual perceptual information to the same global coordinate system, avoiding complex coordinate transformations and supporting better shared cross-agent collaboration… spatial confidence generator generates a spatial confidence map from the feature map of each agent. The spatial confidence map reflects the perceptually critical level of various spatial areas. Intuitively, for object detection task, the areas that contain objects are more critical than background areas. During collaboration, areas with objects could help recover the miss-detected objects due to the limited view; and background areas could be omitted to save the precious bandwidth. So we represent the spatial confidence map with the detection confidence map, where the area with high perceptually critical level is the area that contains an object with a high confidence score. To implement, we use a detection decoder structure to produce the detection confidence map… With the guidance of spatial confidence maps, the proposed communication module packs compact messages with spatially sparse feature maps and transmits messages through a sparsely-connected communication graph… To reduce the communication bandwidth without affecting perception, we leverage the spatial confidence map to select the most informative spatial areas in the feature map (where to communicate) and decide the most beneficial collaboration partners (who to communicate)… Message packing determines what information should be included in the to-be sent message. The proposed message includes: i) a request map that indicates at which spatial areas the agent needs to know more; and ii) a spatially sparse, yet perceptually critical feature map… The request map of the ith agent is… negatively correlated with the spatial confidence map. The intuition is, for the locations with low confidence score, an agent is hard to tell if there is really no objects or it is just caused by the limited information (e.g. occlusion). Thus, the low confidence score indicates there could be missing information at that location. Requesting information at these locations from other agents could improve the current agent’s detection accuracy. The spatially sparse feature map are selected based on each agent’s spatial confidence map and the received request maps from others … Communication graph construction targets to identify when and who to communicate to avoid unnecessary communication that wastes the bandwidth… the necessity of communication between the ith and the jth agents is simply measured by the overlap between the information that the ith agent has and the information that the jth agent needs. With the help of the spatial confidence map and the request map, we construct a more interpretable communication graph; Pg. 17: Spatial confidence-aware message fusion targets to augment the feature of each agent by aggregating the received messages from the other agents. To achieve this, we adopt a transformer architecture, which leverages multi-head attention to fuse the corresponding features from multiple agents at each individual spatial location. The key technical design is to include the spatial confidence maps of all the agents to promote cross-agent attention learning; Pg. 14: fine-grained spatial-aware communication strategy, where each agent can decide where to communicate and pack messages only related to the most perceptually critical spatial areas. This strategy not only enables more precise support for other agents, but also more targeted request from other agents in multi-round communication; Pg. 21: multi-round communication employs a request map in the second and subsequent communication rounds to denote the spatial area where each agent needs more information, which enables more targeted and efficient communication; a request for a vehicle listing, the vehicle listing comprising a plurality of vehicles with sensors having views of the region of interest, wherein the plurality of vehicles of the vehicle listing comprises the second vehicle (e.g. spatial-confidence-aware communication strategy includes a spatial confidence map for each (first, second, third... Nth) agent that requests complementary information from other agents (i.e. from a second vehicle) through multi-round communication, such as vehicle-to-everything-communication-aided autonomous driving, to perform collaboration, for example, including a collaborative perception which enables multiple agents to share complementary perceptual information with each other to overcome a number of limitations of single-agent perception, such as occlusion limitations, for example, including an observation encoder that extracts feature maps (i.e. tensors, feature representations, etc.) from sensor data for each agent (i.e. a sensor listing comprising one or more first, second, third... Nth tensors from a first, second, third... Nth vehicle), for example, in order to select the most informative spatial areas (i.e. a view of a region of interest) in the feature maps (where to communicate) and decide the most beneficial collaboration partners (who to communicate) by requesting complementary information from others through multi-round communication to help recover miss-detected objects due to a limited view, for example, including message packing which determines what information should be included in the to-be sent message (i.e. output, for transmission to other vehicles), including a request map that indicates which spatial areas the agent needs to know more (i.e. a request for a vehicle listing, the vehicle listing comprising a plurality of vehicles with sensors having views of the region of interest, wherein the plurality of vehicles of the vehicle listing comprises the second vehicle or other vehicles), for example, and requesting information at these locations from other agents improves the current (i.e. first vehicle) agent’s detection accuracy, as indicated above), for example).
The same motivation to combine above-mentioned teachings applies, as previously indicated in claim 1.
Regarding claim 6, claim 1 is incorporated and the combination of KEHL and Hu, as a whole, teaches the apparatus (KEHL, Par. [0002-4]), wherein the at least one processor is configured to:
output, for transmission to the second vehicle, a request for information associated with the second vehicle;
receive, from the second vehicle, the information based on the request; and
establish, based on the information, direct communications with the second vehicle (Hu, Pg. 4-5: proposed communication module packs compact messages with spatially sparse feature maps and transmits messages through a sparsely-connected communication graph … Communication graph construction targets to identify when and who to communicate to avoid unnecessary communication that wastes the bandwidth… the necessity of communication between the ith and the jth agents is simply measured by the overlap between the information that the ith agent has and the information that the jth agent needs. With the help of the spatial confidence map and the request map, we construct a more interpretable communication graph… For the initial communication round, every agent in the system is not aware of other agents yet. To activate the collaboration, we construct a fully-connected communication graph. Every agent will broadcast its message to the rest of the system. For the subsequent communication rounds, we examine if the communication between agent i and agent j is necessary; output, for transmission to the second vehicle, a request for information associated with the second vehicle;
receive, from the second vehicle, the information based on the request; and
establish, based on the information, direct communications with the second vehicle (e.g. spatial-confidence-aware communication strategy includes a spatial confidence map for each (first, second, third... Nth) agent that requests complementary information from other agents (i.e. from a second vehicle) through multi-round communication, such as vehicle-to-everything-communication-aided autonomous driving, to perform collaboration, for example, including a collaborative perception which enables multiple agents to share complementary perceptual information with each other to overcome a number of limitations of single-agent perception, such as occlusion limitations, for example, including an observation encoder that extracts feature maps (i.e. tensors, feature representations, etc.) from sensor data for each agent (i.e. a sensor listing comprising one or more first, second, third... Nth tensors from a first, second, third... Nth vehicle), for example, in order to select the most informative spatial areas (i.e. a view of a region of interest) in the feature maps (where to communicate) and decide the most beneficial collaboration partners (who to communicate) by requesting and receiving complementary information from others through multi-round communication (i.e. output, for transmission to the second vehicle, a request for information associated with the second vehicle and receive, from the second vehicle, the information based on the request) to help recover miss-detected objects due to a limited view, for example, including a communication module which transmits (i.e. outputs) messages through a sparsely-connected (i.e. established, subscribed, etc.) communication graph to identify when and who to communicate (i.e. output, for transmission to the second vehicle, a request for information associated with the second vehicle and receive, from the second vehicle, the information based on the request and establish, based on the information, direct communications with the second vehicle) to avoid unnecessary communication that wastes the bandwidth in order to help recover miss-detected objects due to a limited view, for example, and requesting information at these locations from other agents improves the current (i.e. first vehicle) agent’s detection accuracy, as indicated above), for example).
The same motivation to combine above-mentioned teachings applies, as previously indicated in claim 1.
Regarding claim 7, claim 6 is incorporated and the combination of KEHL and Hu, as a whole, teaches the apparatus (KEHL, Par. [0002-4]), wherein the information comprises at least one of a license plate number of the second vehicle, a color of the second vehicle, a position of the second vehicle, a make of the second vehicle, a model of the second vehicle, a physical identification (ID) of the second vehicle, or any combination thereof (Hu, Pg. 23: initial location and motion trajectory of each vehicle is determined by the map’s road layout.… During data collection, 3D bounding boxes of vehicles are recorded at the same moment with images, including location (x, y, z), rotation (w, x, y, z in quaternion) in the global coordinate and their length, width and height. The location (x, y, z) is the center of the bounding box; wherein the information comprises at least one of a license plate number of the second vehicle, a color of the second vehicle, a position of the second vehicle, a make of the second vehicle, a model of the second vehicle, a physical identification (ID) of the second vehicle, or any combination thereof (e.g. spatial-confidence-aware communication strategy includes a spatial confidence map for each (first, second, third... Nth) agent that requests complementary information from other agents (i.e. from a second vehicle) through multi-round communication, such as vehicle-to-everything-communication-aided autonomous driving, to perform collaboration, for example, including recorded 3D bounding boxes of vehicles, which include location (i.e. wherein the information comprises at least one of a position of the second vehicle), as indicated above), for example).
The same motivation to combine above-mentioned teachings applies, as previously indicated in claim 1.
Regarding claim 9, claim 1 is incorporated and the combination of KEHL and Hu, as a whole, teaches the apparatus (KEHL, Par. [0002-4]), wherein, to generate the output based on the one or more first tensors and the one or more second tensors, the at least one processor is configured to:
fuse the one or more first tensors and the one or more second tensors to generate a third tensor; and
process the third tensor to generate an output (KEHL, Par. [0030-42]: extracted features may be transmitted to a reshape module of the 3D inference system… extracted features may be transmitted to the reshape module of the 3D inference system… The features may be encoded in a data structure, such as a tensor, as explained in detail herein. The reshape module may reshape the features obtained by the tensors and map the reshaped tensors to a vector corresponding to a feature space. The feature space represents the layout of cells of a 3D grid. As previously discussed, the 3D grid represents a spatial area within a vicinity of the agent 100. In one example, the 3D grid may be the spatial area directly in front of the agent 100. As another example, the 3D grid may be a 360° area around the agent 100… 3D inference system is trained to map features extracted from sensor outputs to a representation (e.g., vector) of a 3D grid. In one configuration, during an offline training step, the 3D inference system acquires features extracted from different sensor outputs (e.g., LIDAR output and 2D camera output). Furthermore, the 3D inference system is trained to generate merged features by matching, reshaping, and combining the features based on a layout of the 3D grid. The merged features are then mapped to cells of the 3D grid that are defined as elements of a vector. After mapping the merged features, the mapped features are classified by a trained neural network. The classified features may be used to infer 3D objects in a spatial area; Par. [0041-48]: provide merge and reshape features from different feature spaces. The reshaped features may then be mapped to a feature space of a vector that represents a layout of a 3D grid. As such, aspects of the present disclosure improve 3D inference by providing a 3D inference system that infers 3D objects based on the output of multiple sensors regardless of the layout of a 3D grid… 3D inference system is trained to map features extracted from sensor outputs to a representation (e.g., vector) of a 3D grid. In one configuration, during an offline training step, the 3D inference system acquires features extracted from different sensor outputs (e.g., LIDAR output and 2D camera output). Furthermore, the 3D inference system is trained to generate merged features by matching, reshaping, and combining the features based on a layout of the 3D grid. The merged features are then mapped to cells of the 3D grid that are defined as elements of a vector. After mapping the merged features, the mapped features are classified by a trained neural network. The classified features may be used to infer 3D objects in a spatial area… the reshape module 310 is a fully connected layer that receives the tensors FRGB, FBEV, FFFV corresponding to each sensor output 302, 304, 306. The reshape module 310 performs a reshape (e.g., matrix multiplication) to output a 3D grid vector (V3D) corresponding to a layout of the 3D grid 360, 366. That is, the reshape module 310 includes a reshape function F( ) that receives tensors (FRGB, FBEV, FFFV) as an input x to generate an output y (V3D); Par. [0072]: feature extractor 510 may encode each set of features in a multi-dimensional tensor. The reshape module 512 receives the multi-dimensional tensors from the feature extractor 510 to reshape the multi-dimensional tensor to a vector corresponding to a layout of a 3D grid. The reshape module 512 may be a fully connected layer of the CNN that has been trained to match, reshape, and combine the features from the multi-dimensional tensors to a single vector; wherein, to generate the output based on the one or more first tensors and the one or more second tensors, the at least one processor is configured to:
fuse the one or more first tensors and the one or more second tensors to generate a third tensor; and
process the third tensor to generate an output (e.g. system and method for inferring a location of a 3D object with respect to a location of an autonomous agent, including an autonomous vehicle, include extracting features from sensor data obtained from a number of sensors of the autonomous vehicle (i.e. first, second, third... Nth sensor data obtained from one or more first, second, third... Nth sensors of the first vehicle), respectively, in which extracted features are encoded in data structures, such as 3D tensors (i.e. feature representations, vectors, voxels, etc.), for example, by using a convolutional neural network, or another type of artificial neural network, for extracting features from each sensor output, in which extracted features from a first sensor output are encoded in an RGB image tensor, extracted features from a second sensor output are encoded in a bird's eye view (BEV) tensor, and extracted features from a third sensor output are encoded in a forward facing view (FFV) tensor, respectively, for example, and are used to merge (i.e. fuse, combine, etc.) and reshape the features obtained by the first, second, third... Nth tensors and map the reshaped tensors to a vector corresponding to a feature space, for example, including a reshape function that receives first, second, third... Nth encoded tensors as an input x to generate an output y (i.e. process the third tensor to generate an output), as indicated above), for example).
Regarding claim 10, claim 9 is incorporated and the combination of KEHL and Hu, as a whole, teaches the apparatus (KEHL, Par. [0002-4]), wherein, to fuse the one or more first tensors and the one or more second tensors, the at least one processor is configured to project, using a view transform, the one or more first tensors and the one or more second tensors onto a top view representation of the first vehicle to generate the third tensor (Hu, Pg. 4: observation encoder extracts feature maps from the sensor data. Where2comm accepts single/multimodality inputs, such as RGB images and 3D point clouds. This work adopts the feature representations in bird’s eye view (BEV), where all agents project their individual perceptual information to the same global coordinate system, avoiding complex coordinate transformations and supporting better shared cross-agent collaboration… All agents share the same BEV coordinate system. For the image input… followed by a warping function that transforms the extracted feature from front-view to BEV. For 3D point cloud input, we discretize 3D points as a BEV map and… extracts features in BEV. The extracted feature map is output to the spatial confidence generator and the message fusion module; wherein, to fuse the one or more first tensors and the one or more second tensors, the at least one processor is configured to project, using a view transform, the one or more first tensors and the one or more second tensors onto a top view representation of the first vehicle to generate the third tensor (e.g. spatial-confidence-aware communication strategy includes a spatial confidence map for each (first, second, third... Nth) agent that requests complementary information from other agents (i.e. from a second vehicle) through multi-round communication, such as vehicle-to-everything-communication-aided autonomous driving, to perform collaboration, for example, including an observation encoder that extracts feature maps (i.e. tensors, feature representations, etc.) from sensor data for each agent (i.e. a sensor listing comprising one or more first, second, third... Nth tensors from a first, second, third... Nth vehicle), for example, in order to select the most informative spatial areas in the feature maps, for example, including feature representations in bird’s eye view (BEV) (i.e. a top view representation), where all agents project their individual perceptual information to the same global coordinate system, including a warping function that transforms the extracted features to BEV and extracts features in BEV (i.e. wherein, to fuse the one or more first tensors and the one or more second tensors, project, using a view transform, the one or more first tensors and the one or more second tensors onto a top view representation of the first vehicle to generate the third tensor), for example).
The same motivation to combine above-mentioned teachings applies, as previously indicated in claim 1.
Regarding claim 11, claim 9 is incorporated and the combination of KEHL and Hu, as a whole, teaches the apparatus (KEHL, Par. [0002-4]), wherein, to process the third tensor to generate the output, the at least one processor is configured to detect, based on the third tensor, one or more objects to generate an object detection output (KHEL, Par. [0024-33]: agent may detect objects from outputs of various sensors… objects are localized based on features extracted from different sensor outputs… features are extracted from different sensor outputs… 3D inference system of the agent 100 may extract features from objects of interest… The features may be encoded in a data structure, such as a tensor, as explained in detail herein. The reshape module may reshape the features obtained by the tensors and map the reshaped tensors to a vector corresponding to a feature space… the features mapped to the vector correspond to specific cells of the 3D grid. The features of the vector may be analyzed by a trained neural network to classify objects in the spatial area. The classification may include determining a class label of an object, the spatial dimensions of the object, an orientation of the object with respect to the agent, and/or other attributes. By classifying the features, aspects of the present disclosure may detect 3D objects; Par. [0071]: feature extractor 510 extracts (e.g., identifies) areas of interest from each frame of the data stream. For example, the feature extractor 510 may be trained to extract features of 3D objects. As another example, the feature extractor 510 may be trained to extract features of different terrains, such as roads, sidewalks, buildings, and background. That is, the feature exactor 510 identifies areas of attention based on the training. The artificial neural network 508 may include one or more feature extractors 510. For example, one feature extractor 510 may be configured to detect 3D objects; wherein, to process the third tensor to generate the output, the at least one processor is configured to detect, based on the third tensor, one or more objects to generate an object detection output (e.g. system and method for inferring a location of a 3D object with respect to a location of an autonomous agent, including an autonomous vehicle (i.e. a first vehicle), for example, include extracting features from sensor data obtained from a number of sensors of the autonomous vehicle (i.e. first, second, third... Nth sensor data obtained from one or more first, second, third... Nth sensors of the first vehicle), respectively, in which extracted features are encoded in data structures, such as 3D tensors (i.e. feature representations, vectors, voxels, etc.), for example, and detecting objects from outputs of various sensors (i.e. wherein, to process the third tensor to generate the output, detect, based on the third tensor, one or more objects to generate an object detection output), as indicated above), for example).
Regarding claim 12, claim 11 is incorporated and the combination of KEHL and Hu, as a whole, teaches the apparatus (KEHL, Par. [0002-4]), wherein, to process the third tensor, the at least one processor is configured to generate, using one or more bird's eye view (BEV) encoders and one or more decoders, one or more output tensors representing the object detection output (Hu, Pg. 4: observation encoder extracts feature maps from the sensor data. Where2comm accepts single/multimodality inputs, such as RGB images and 3D point clouds. This work adopts the feature representations in bird’s eye view (BEV), where all agents project their individual perceptual information to the same global coordinate system, avoiding complex coordinate transformations and supporting better shared cross-agent collaboration… All agents share the same BEV coordinate system. For the image input… followed by a warping function that transforms the extracted feature from front-view to BEV. For 3D point cloud input, we discretize 3D points as a BEV map and… extracts features in BEV. The extracted feature map is output to the spatial confidence generator and the message fusion module; wherein, to fuse the one or more first tensors and the one or more second tensors, the at least one processor is configured to project, using a view transform, the one or more first tensors and the one or more second tensors onto a top view representation of the first vehicle to generate the third tensor (e.g. spatial-confidence-aware communication strategy includes a spatial confidence map for each (first, second, third... Nth) agent that requests complementary information from other agents (i.e. from a second vehicle) through multi-round communication, such as vehicle-to-everything-communication-aided autonomous driving, to perform collaboration, for example, including an observation (i.e. detection, etc.) encoder that extracts feature maps (i.e. tensors, feature representations, etc.) from sensor data for each agent (i.e. a sensor listing comprising one or more first, second, third... Nth tensors from a first, second, third... Nth vehicle), for example, in order to select (i.e. detect, identify, etc.) the most informative spatial areas in the feature maps, for example, including feature representations in bird’s eye view (BEV), where all agents project their individual perceptual information to the same global coordinate system, including a warping function that transforms the extracted features to BEV and extracts features in BEV (i.e. wherein, to process the third tensor, generate, using one or more bird's eye view (BEV) encoders and one or more decoders, one or more output tensors representing the object detection output), for example).
The same motivation to combine above-mentioned teachings applies, as previously indicated in claim 1.
Regarding claim 13, claim 12 is incorporated and the combination of KEHL and Hu, as a whole, teaches the apparatus (KEHL, Par. [0002-4]), wherein each output tensor of the one or more output tensors represents at least one property of the one or more objects (KEHL, Par. [0030-32]: extracted features may be transmitted to a reshape module of the 3D inference system… extracted features may be transmitted to the reshape module of the 3D inference system… The features may be encoded in a data structure, such as a tensor, as explained in detail herein. The reshape module may reshape the features obtained by the tensors and map the reshaped tensors to a vector corresponding to a feature space. The feature space represents the layout of cells of a 3D grid. As previously discussed, the 3D grid represents a spatial area within a vicinity of the agent 100; Par. [0041-48]: provide merge and reshape features from different feature spaces. The reshaped features may then be mapped to a feature space of a vector that represents a layout of a 3D grid. As such, aspects of the present disclosure improve 3D inference by providing a 3D inference system that infers 3D objects based on the output of multiple sensors regardless of the layout of a 3D grid… 3D inference system is trained to map features extracted from sensor outputs to a representation (e.g., vector) of a 3D grid. In one configuration, during an offline training step, the 3D inference system acquires features extracted from different sensor outputs (e.g., LIDAR output and 2D camera output). Furthermore, the 3D inference system is trained to generate merged features by matching, reshaping, and combining the features based on a layout of the 3D grid. The merged features are then mapped to cells of the 3D grid that are defined as elements of a vector. After mapping the merged features, the mapped features are classified by a trained neural network. The classified features may be used to infer 3D objects in a spatial area… the reshape module 310 is a fully connected layer that receives the tensors FRGB, FBEV, FFFV corresponding to each sensor output 302, 304, 306. The reshape module 310 performs a reshape (e.g., matrix multiplication) to output a 3D grid vector (V3D) corresponding to a layout of the 3D grid 360, 366. That is, the reshape module 310 includes a reshape function F( ) that receives tensors (FRGB, FBEV, FFFV) as an input x to generate an output y (V3D); wherein each output tensor of the one or more output tensors represents at least one property of the one or more objects (e.g. system and method for inferring a location of a 3D object with respect to a location of an autonomous agent, including an autonomous vehicle (i.e. a first vehicle), for example, include extracting features (i.e. properties, attributes, characteristics, etc.) from sensor data obtained from a number of sensors of the autonomous vehicle (i.e. first, second, third... Nth sensor data obtained from one or more first, second, third... Nth sensors of the first vehicle), respectively, in which extracted features are encoded in data structures, such as 3D tensors (i.e. wherein each output tensor of the one or more output tensors represents at least one property of the one or more objects), as indicated above), for example).
Regarding claim 14, claim 13 is incorporated and the combination of KEHL and Hu, as a whole, teaches the apparatus (KEHL, Par. [0002-4]), wherein the at least one property includes at least one of a respective probability of each object being located at a respective location within an environment of the first vehicle, a respective orientation of each object, a respective class of each object, a respective size of each object, a respective velocity of each object, or any combination thereof (Hu, (Pg. 1-2: Collaborative perception enables multiple agents to share complementary perceptual information with each other, promoting more holistic perception. It provides a new direction to fundamentally overcome a number of inevitable limitations of single-agent perception, such as occlusion and long-range issues. Related methods and systems are desperately needed in a broad range of real-world applications, such as vehicle-to-everything-communication-aided autonomous driving [1–3], multirobot warehouse automation system [4, 5] and multi-UAVs (unmanned aerial vehicles) for search and rescue [6–8] … spatial-confidence-aware communication strategy. The core idea is to enable a spatial confidence map for each agent, where each element reflects the perceptually critical level of a corresponding spatial area. Based on this map, agents decide which spatial area (where) to communicate about. That is, each agent offers spatially sparse, yet critical features to support other agents, and meanwhile requests complementary information from others through multi-round communication to perform efficient and mutually beneficial collaboration… a spatial confidence-aware message fusion module, which uses novel confidence-aware multi-head attention to fuse all messages received from other agents, upgrading the feature map for each agent; Pg. 4-5: observation encoder extracts feature maps from the sensor data. Where2comm accepts single/multimodality inputs, such as RGB images and 3D point clouds. This work adopts the feature representations in bird’s eye view (BEV), where all agents project their individual perceptual information to the same global coordinate system… All agents share the same BEV coordinate system. For the image input, Φenc(·) is followed by a warping function that transforms the extracted feature from front-view to BEV. For 3D point cloud input, we discretize 3D points as a BEV map and Φenc(·) extracts features in BEV. The extracted feature map is output to the spatial confidence generator and the message fusion module… The spatial confidence generator generates a spatial confidence map from the feature map of each agent. The spatial confidence map reflects the perceptually critical level of various spatial areas… During collaboration, areas with objects could help recover the miss-detected objects due to the limited view… Where2comm iteratively updates the feature map by aggregating information from other agents… Message packing determines what information should be included in the to-be sent message. The proposed message includes: i) a request map that indicates at which spatial areas the agent needs to know more; and ii) a spatially sparse, yet perceptually critical feature map… The request map of the ith agent is… negatively correlated with the spatial confidence map. The intuition is, for the locations with low confidence score, an agent is hard to tell if there is really no objects or it is just caused by the limited information (e.g. occlusion). Thus, the low confidence score indicates there could be missing information at that location. Requesting information at these locations from other agents could improve the current agent’s detection accuracy. The spatially sparse feature map are selected based on each agent’s spatial confidence map and the received request maps from others; Pg. 9: proposed spatial confidence map. In the scene, Drone 1’s view is occluded by a tall building. With Drone 2’s help, Drone 1 is able to detect through occlusion. Fig. 6 (a-d) shows Drone 1’s observation, spatial confidence map (1), binary selection matrix (2), and ego attention weight (3). Fig. 6 (f-h) shows Drone 2’s observation and message sent to Drone 1, including the request map (opposite of confidence map) and the sparse feature map, achieving efficient communication. Fig. 6 (i) shows the attention weight for Drone 1 to fuse Drone 2’s messages, which is sparse, yet highlights the objects’ positions. Fig. 6 (e) and (j) compares the detection results before and after the collaboration with Drone 2. We see that the proposed spatial confidence map contributes to spatially sparse, yet perceptually critical message, which effectively helps Drone 1 detect occluded objects; wherein the at least one property includes at least one of a respective probability (i.e. confidence) of each object being located at a respective location within an environment of the first vehicle, a respective orientation of each object, a respective class of each object, a respective size of each object, a respective velocity of each object, or any combination thereof (e.g. e.g. spatial-confidence-aware communication strategy includes a spatial confidence (i.e. probability, likelihood, etc.) map for each (first, second, third... Nth) agent that reflects the perceptually critical level of various spatial areas including location information from other agents (i.e. wherein the at least one property includes at least one of a respective probability of each object being located at a respective location within an environment of the first vehicle), as indicated above), for example).
The same motivation to combine above-mentioned teachings applies, as previously indicated in claim 1.
Regarding claim 15, claim 9 is incorporated and the combination of KEHL and Hu, as a whole, teaches the apparatus (KEHL, Par. [0002-4]), wherein the third tensor represents a bird's eye view (BEV) of the environment of the first vehicle (Hu, Pg. 4: observation encoder extracts feature maps from the sensor data. Where2comm accepts single/multimodality inputs, such as RGB images and 3D point clouds. This work adopts the feature representations in bird’s eye view (BEV), where all agents project their individual perceptual information to the same global coordinate system, avoiding complex coordinate transformations and supporting better shared cross-agent collaboration… All agents share the same BEV coordinate system. For the image input… followed by a warping function that transforms the extracted feature from front-view to BEV. For 3D point cloud input, we discretize 3D points as a BEV map and… extracts features in BEV. The extracted feature map is output to the spatial confidence generator and the message fusion module; wherein, to fuse the one or more first tensors and the one or more second tensors, the at least one processor is configured to project, using a view transform, the one or more first tensors and the one or more second tensors onto a top view representation of the first vehicle to generate the third tensor (e.g. spatial-confidence-aware communication strategy includes a spatial confidence map for each (first, second, third... Nth) agent that requests complementary information from other agents (i.e. from a second vehicle) through multi-round communication, such as vehicle-to-everything-communication-aided autonomous driving, to perform collaboration, for example, including an observation (i.e. detection, etc.) encoder that extracts feature maps (i.e. tensors, feature representations, etc.) from sensor data for each agent (i.e. a sensor listing comprising one or more first, second, third... Nth tensors from a first, second, third... Nth vehicle), for example, in order to select (i.e. detect, identify, etc.) the most informative spatial areas in the feature maps, for example, including feature representations in bird’s eye view (BEV), where all agents project their individual perceptual information to the same global coordinate system, including a warping function that transforms the extracted features to BEV and extracts features in BEV (i.e. wherein the third tensor represents a bird's eye view (BEV) of the environment of the first vehicle), for example).
The same motivation to combine above-mentioned teachings applies, as previously indicated in claim 1.
Regarding claim 16, claim 1 is incorporated and the combination of KEHL and Hu, as a whole, teaches the apparatus (KEHL, Par. [0002-4]), wherein each tensor of the one or more first tensors and each tensor of the one or more second tensors represents a perspective view of an environment of the first vehicle (KEHL, Par. [0029-32]: 2D camera 108 captures a 2D image 120 that includes objects in the 2D camera's 108 field of view 114. The LIDAR sensor 106 may generate one or more output streams 122, 124. The first output stream 122 may include a 3D cloud point of objects in a first field of view, such as a 360° field of view 112 (e.g., bird's eye view). The second output stream 124 may include a 3D cloud point of objects in a second field of view, such as a forward facing field of view 126… The 2D image 120 captured by the 2D camera includes a 2D image of the first vehicle 104, as the first vehicle 104 is in the 2D camera's 108 field of view 114. The 3D inference system of the agent 100 may extract features from objects of interest in the 2D image 120. For example, an artificial neural network, such as a convolutional neural network, of the 3D inference system extracts features of the first vehicle 104… … The features may be encoded in a data structure, such as a tensor, as explained in detail herein. The reshape module may reshape the features obtained by the tensors and map the reshaped tensors to a vector corresponding to a feature space. The feature space represents the layout of cells of a 3D grid. As previously discussed, the 3D grid represents a spatial area within a vicinity of the agent 100; Par. [0038-46]: cells 204, 206 of each 3D grid 200, 202 may be represented as elements in a vector 220, 220… As is known to those of skill in the art, features extracted from an image are encoded in a 3D tensor. FIG. 2B illustrates an example of a 3D tensor 250 and a 2D image 252 according to aspects of the present disclosure. The 3D tensor 250 encodes both the spatial structure and features of the 2D image 252. The first two dimensions of the 3D tensor 250 are a height (H) and a width (W) that correspond to the height (H) and the width (W) of the 2D image 252. The third dimension (D) encodes a number of feature channels, one for each spatial location (e.g., pixel)… the 3D grids 200, 202 represent a different feature space in comparison to a feature space of a sensor output… The sensor outputs 302, 304, 306 are received at a feature extraction module 308. The feature extraction module 308 may be a convolutional neural network or another type of artificial neural network, such as a support vector machine (SVM), for extracting features from each sensor output 302, 304, 306. For example, the features from the first sensor output 302 are encoded in an RGB image tensor (FRGB). The features from the second sensor output 304 may be encoded in a bird's eye view (BEV) tensor (FBEV). Finally, the features from the third sensor output 306 may be encoded in a forward facing view (FFV) tensor (FFFV). Each tensor FRGB, FBEV, FFFV may have three dimensions (Height×Width×Depth)… As shown in FIG. 3, each sensor output 302, 304, 306 provides a different perspective of the agent's environment. As such, each tensor FRGB, FBEV, FFFV represents a different feature space; wherein each tensor of the one or more first tensors and each tensor of the one or more second tensors represents a perspective view of an environment of the first vehicle (e.g. system and method for inferring a location of a 3D object with respect to a location of an autonomous agent, including an autonomous vehicle, include extracting features from sensor data obtained from a number of sensors of the autonomous vehicle (i.e. first, second, third... Nth sensor data obtained from one or more first, second, third... Nth sensors of the first vehicle), respectively, in which extracted features are encoded in data structures, such as 3D tensors (i.e. feature representations, vectors, voxels, etc.), for example, by using a convolutional neural network, or another type of artificial neural network, for extracting features from each sensor output, in which extracted features from a first sensor output are encoded in an RGB image tensor and extracted features from a second sensor output are encoded in a bird's eye view (BEV) tensor , for example, to identify objects within areas of interest in an image of a surrounding environment with respect to each sensor field of view and each sensor provides a different perspective of the agent's environment (i.e. wherein each tensor of the one or more first tensors and each tensor of the one or more second tensors represents a perspective view of an environment of the first vehicle), as indicated above), for example).
Regarding claim 17, claim 1 is incorporated and the combination of KEHL and Hu, as a whole, teaches the apparatus (KEHL, Par. [0002-4]), wherein each sensor of the one or more first sensors and each sensor of the one or more second sensors is a respective image sensor, a respective radar sensor, or respective a light detection and ranging (Lidar) sensor (KEHL, Par. [0065]: sensor module 502 may be used to obtain measurements via different sensors, such as a first sensor 506 and a second sensor 504. The first sensor 506 may be a vision sensor, such as a stereoscopic camera or an RGB camera, for capturing 2D images. The second sensor 504 may be a ranging sensor, such as a light detection and ranging (LIDAR) sensor or a radio detection and ranging (RADAR) sensor).
Regarding claim 18, claim 1 is incorporated and the combination of KEHL and Hu, as a whole, teaches the apparatus (KEHL, Par. [0002-4]), wherein each encoder of the one or more encoders is included in a neural network of the first vehicle (KEHL, Par. [0004]: autonomous agents may rely on a trained convolutional neural network (CNN) to identify objects within areas of interest in an image of a surrounding environment. For example, a CNN may be trained to identify and track objects captured by one or more sensors, such as light detection and ranging (LIDAR) sensors, sonar sensors, red-green-blue (RGB) cameras, RGB-depth (RGB-D) cameras, and the like. The sensors may be coupled to, or in communication with, a device, such as the autonomous agent. Object detection applications for autonomous agents may analyze sensor image data for detecting objects (e.g., pedestrians, cyclists, other cars, etc.) in the surrounding scene from the autonomous agent; Par. [0044-49]: sensor outputs 302, 304, 306 are received at a feature extraction module 308. The feature extraction module 308 may be a convolutional neural network or another type of artificial neural network, such as a support vector machine (SVM), for extracting features from each sensor output 302, 304, 306. For example, the features from the first sensor output 302 are encoded in an RGB image tensor… features from the second sensor output 304 may be encoded in a bird's eye view (BEV) tensor… the features from the third sensor output 306 may be encoded in a forward facing view (FFV) tensor… 3D grid vector (V3D) may be interpreted by a classification module 322 to infer the 3D objects in a spatial area 370. The features encoded in the 3D grid vector (V3D) may be processed by a convolutional neural network, or other type of neural network; Par. [0071-72]: feature extractor 510 extracts (e.g., identifies) areas of interest from each frame of the data stream. For example, the feature extractor 510 may be trained to extract features of 3D objects… the feature exactor 510 identifies areas of attention based on the training… feature extractor 510 may encode each set of features in a multi-dimensional tensor. The reshape module 512 receives the multi-dimensional tensors from the feature extractor 510 to reshape the multi-dimensional tensor to a vector corresponding to a layout of a 3D grid. The reshape module 512 may be a fully connected layer of the CNN that has been trained to match, reshape, and combine the features from the multi-dimensional tensors to a single vector. The reshape module 512 outputs the vector corresponding to cells of the 3D grid; wherein each encoder of the one or more encoders is included in a neural network of the first vehicle (e.g. system and method for inferring a location of a 3D object with respect to a location of an autonomous agent, including an autonomous vehicle (i.e. a first vehicle), for example, include extracting features from sensor data obtained from a number of sensors of the autonomous vehicle (i.e. first, second, third... Nth sensor data obtained from one or more first, second, third... Nth sensors of the first vehicle), respectively, in which extracted features are encoded in data structures, such as 3D tensors (i.e. feature representations, vectors, voxels, etc.), for example, by using a convolutional neural network, or another type of artificial neural network (i.e. wherein each encoder of the one or more encoders is included in a neural network of the first vehicle), for extracting features from each sensor output, as indicated above), for example).
Regarding claim 19, KEHL discloses a method for vehicle communications at a first vehicle (Par. [0002-4]: system and method for inferring a location of a three-dimensional (3D) object with respect to a location of an autonomous agent… Autonomous agents (e.g., vehicles, robots, drones, etc.) and semi-autonomous agents use machine vision for analyzing areas of interest in a surrounding environment… autonomous agents may rely on a trained convolutional neural network (CNN) to identify objects within areas of interest in an image of a surrounding environment. For example, a CNN may be trained to identify and track objects captured by one or more sensors… The sensors may be coupled to, or in communication with, a device, such as the autonomous agent; an apparatus for vehicle communications at a first vehicle (e.g. system and method for inferring a location of a three-dimensional (3D) object with respect to a location of an autonomous agent, including an autonomous vehicle (i.e. a first vehicle), for example, include a convolutional neural network (CNN) trained to identify and track objects captured by one or more sensors that are coupled to, or in communication with, the autonomous agent (i.e. an apparatus for vehicle communications at a first vehicle), as indicated above), for example), the method comprising:
generating, by one or more encoders of the first vehicle (Par. [0061-64]: FIG. 5 is a diagram illustrating an example of a hardware implementation for a 3D inference system 500 according to aspects of the present disclosure. The 3D inference system 500 may be a component of a vehicle… The 3D inference system 500 includes a transceiver 516 coupled to the processor 520, the sensor module 502, an artificial neural network module 508… The 3D inference system 500 includes the processor 520 coupled to the computer-readable medium 514. The processor 520 performs processing, including the execution of software stored on the computer-readable medium 514 providing functionality according to the disclosure. The software, when executed by the processor 520, causes the 3D inference system 500 to perform the various functions described for a particular device, such as the car 528, or any of the modules 502, 508, 514, 516, 518, 520, 522, 524, 526; generating, by one or more encoders of the first vehicle (e.g. system and method for inferring a location of a 3D object with respect to a location of an autonomous agent, including an autonomous vehicle (i.e. a first vehicle), for example, including software that, when executed by a processor, causes 3D inference system to perform various functions described for a particular device, including encoding (i.e. generating) extracted features in data structures, such as 3D tensors (i.e. feature representations, vectors, voxels, etc.), for example, by using a convolutional neural network, or another type of artificial neural network (i.e. generating, by one or more encoders of the first vehicle), for extracting features from each sensor output, as indicated above.
The steps of the method further recited in claim 19 correspond to claim 1 when executed and are rejected as applied to apparatus claim 1 above.
Regarding claim 20, claim 19 is incorporated and the combination of KEHL and Hu, as a whole, teaches the method (KEHL, Par. [0002-4]), further comprising determining a region of interest based on a view of at least one first sensor of the one or more first sensors (KEHL, Par. [0002-4]: system and method for inferring a location of a three-dimensional (3D) object with respect to a location of an autonomous agent… Autonomous agents (e.g., vehicles, robots, drones, etc.) and semi-autonomous agents use machine vision for analyzing areas of interest in a surrounding environment… autonomous agents may rely on a trained convolutional neural network (CNN) to identify objects within areas of interest in an image of a surrounding environment. For example, a CNN may be trained to identify and track objects captured by one or more sensors; Par. [0029-30]: 2D camera 108 captures a 2D image 120 that includes objects in the 2D camera's 108 field of view 114. The LIDAR sensor 106 may generate one or more output streams 122, 124. The first output stream 122 may include a 3D cloud point of objects in a first field of view, such as a 360° field of view 112 (e.g., bird's eye view). The second output stream 124 may include a 3D cloud point of objects in a second field of view, such as a forward facing field of view 126… The 2D image 120 captured by the 2D camera includes a 2D image of the first vehicle 104, as the first vehicle 104 is in the 2D camera's 108 field of view 114. The 3D inference system of the agent 100 may extract features from objects of interest in the 2D image 120. For example, an artificial neural network, such as a convolutional neural network, of the 3D inference system extracts features of the first vehicle 104; Par. [0071-72]: feature extractor 510 extracts (e.g., identifies) areas of interest from each frame of the data stream. For example, the feature extractor 510 may be trained to extract features of 3D objects… the feature exactor 510 identifies areas of attention based on the training… feature extractor 510 may encode each set of features in a multi-dimensional tensor. The reshape module 512 receives the multi-dimensional tensors from the feature extractor 510 to reshape the multi-dimensional tensor to a vector corresponding to a layout of a 3D grid. The reshape module 512 may be a fully connected layer of the CNN that has been trained to match, reshape, and combine the features from the multi-dimensional tensors to a single vector. The reshape module 512 outputs the vector corresponding to cells of the 3D grid; wherein the at least one processor is configured to determine a region of interest based on a view of at least one first sensor of the one or more first sensors (e.g. system and method for inferring a location of a 3D object with respect to a location of an autonomous agent, including an autonomous vehicle, include extracting features from sensor data obtained from a number of sensors of the autonomous vehicle (i.e. first, second, third... Nth sensor data obtained from one or more first, second, third... Nth sensors of the first vehicle), respectively, for example, including a trained convolutional neural network (CNN) to identify objects within areas of interest in an image of a surrounding environment with respect to each sensor field of view (i.e. determine a region of interest based on a view of at least one first sensor of the one or more first sensors), as indicated above), for example).
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over KEHL, in view of Hu, as applied to claim 1 above, in further view of Huang et al. (US PG Publication No. US 2022/0188554 A1), hereafter referred to as Huang.
Regarding claim 8, claim 1 is incorporated and the combination of KEHL and Hu, as a whole, teaches the apparatus (KEHL, Par. [0002-4]), wherein the at least one processor is configured to receive, from vehicles comprising the second vehicle, one or more fourth tensors associated with third sensor data, wherein the third sensor data is associated with the one or more regions obscured within the first sensor data (Hu, Pg. 1-2: Collaborative perception enables multiple agents to share complementary perceptual information with each other, promoting more holistic perception. It provides a new direction to fundamentally overcome a number of inevitable limitations of single-agent perception, such as occlusion and long-range issues. Related methods and systems are desperately needed in a broad range of real-world applications, such as vehicle-to-everything-communication-aided autonomous driving [1–3], multirobot warehouse automation system [4, 5] and multi-UAVs (unmanned aerial vehicles) for search and rescue [6–8] … spatial-confidence-aware communication strategy. The core idea is to enable a spatial confidence map for each agent, where each element reflects the perceptually critical level of a corresponding spatial area. Based on this map, agents decide which spatial area (where) to communicate about. That is, each agent offers spatially sparse, yet critical features to support other agents, and meanwhile requests complementary information from others through multi-round communication to perform efficient and mutually beneficial collaboration… a spatial confidence-aware message fusion module, which uses novel confidence-aware multi-head attention to fuse all messages received from other agents, upgrading the feature map for each agent; Pg. 4-5: observation encoder extracts feature maps from the sensor data. Where2comm accepts single/multimodality inputs, such as RGB images and 3D point clouds. This work adopts the feature representations in bird’s eye view (BEV), where all agents project their individual perceptual information to the same global coordinate system, avoiding complex coordinate transformations and supporting better shared cross-agent collaboration… spatial confidence generator generates a spatial confidence map from the feature map of each agent. The spatial confidence map reflects the perceptually critical level of various spatial areas. Intuitively, for object detection task, the areas that contain objects are more critical than background areas. During collaboration, areas with objects could help recover the miss-detected objects due to the limited view; and background areas could be omitted to save the precious bandwidth. So we represent the spatial confidence map with the detection confidence map, where the area with high perceptually critical level is the area that contains an object with a high confidence score. To implement, we use a detection decoder structure to produce the detection confidence map… With the guidance of spatial confidence maps, the proposed communication module packs compact messages with spatially sparse feature maps and transmits messages through a sparsely-connected communication graph… To reduce the communication bandwidth without affecting perception, we leverage the spatial confidence map to select the most informative spatial areas in the feature map (where to communicate) and decide the most beneficial collaboration partners (who to communicate)… Message packing determines what information should be included in the to-be sent message. The proposed message includes: i) a request map that indicates at which spatial areas the agent needs to know more; and ii) a spatially sparse, yet perceptually critical feature map… The request map of the ith agent is… negatively correlated with the spatial confidence map. The intuition is, for the locations with low confidence score, an agent is hard to tell if there is really no objects or it is just caused by the limited information (e.g. occlusion). Thus, the low confidence score indicates there could be missing information at that location. Requesting information at these locations from other agents could improve the current agent’s detection accuracy. The spatially sparse feature map are selected based on each agent’s spatial confidence map and the received request maps from others … Communication graph construction targets to identify when and who to communicate to avoid unnecessary communication that wastes the bandwidth… the necessity of communication between the ith and the jth agents is simply measured by the overlap between the information that the ith agent has and the information that the jth agent needs. With the help of the spatial confidence map and the request map, we construct a more interpretable communication graph; Pg. 17: Spatial confidence-aware message fusion targets to augment the feature of each agent by aggregating the received messages from the other agents. To achieve this, we adopt a transformer architecture, which leverages multi-head attention to fuse the corresponding features from multiple agents at each individual spatial location. The key technical design is to include the spatial confidence maps of all the agents to promote cross-agent attention learning; Pg. 14: fine-grained spatial-aware communication strategy, where each agent can decide where to communicate and pack messages only related to the most perceptually critical spatial areas. This strategy not only enables more precise support for other agents, but also more targeted request from other agents in multi-round communication; Pg. 21: multi-round communication employs a request map in the second and subsequent communication rounds to denote the spatial area where each agent needs more information, which enables more targeted and efficient communication; receive, from vehicles comprising the second vehicle, one or more fourth tensors associated with third sensor data, wherein the third sensor data is associated with the one or more regions obscured within the first sensor data (e.g. spatial-confidence-aware communication strategy includes a spatial confidence map for each (first, second, third... Nth) agent that requests complementary information from other agents (i.e. from a second vehicle) through multi-round communication, such as vehicle-to-everything-communication-aided autonomous driving, to perform collaboration, for example, including a collaborative perception which enables multiple agents to share complementary perceptual information with each other to overcome a number of limitations of single-agent perception, such as occlusion limitations (i.e. wherein the third sensor data is associated with the one or more regions obscured within the first sensor data), for example, including an observation encoder that extracts feature maps (i.e. tensors, feature representations, etc.) from sensor data for each agent (i.e. receive, from vehicles comprising the second vehicle, one or more first, second, third, fourth... Nth tensors associated with third sensor data, wherein the first, second, third, fourth... Nth sensor data is associated with the one or more regions obscured within the first sensor data), for example, in order to select the most informative spatial areas (i.e. a view of a region of interest) in the feature maps by requesting complementary information from others through multi-round communication to help recover miss-detected objects due to a limited view, as indicated above), for example).
The same motivation to combine above-mentioned teachings applies, as previously indicated in claim 1.
However, Huang teaches vehicles within a fleet of vehicles (Par. [0072-84]: transportation management system 760, autonomous or semi-autonomous vehicle 740… may be communicatively connected or co-located with each other in whole or in part… the vehicle 740 may be an autonomous or semi-autonomous vehicle and equipped with an array of sensors 744, a navigation system 746, and a ride-service computing device 748… a fleet of autonomous or semi-autonomous vehicles 740 may be managed by the transportation management system 760… the transportation management system 760 may control the operations of the autonomous vehicles 740, including, e.g., dispatching select vehicles 740 to fulfill ride requests, instructing the vehicles 740 to perform select operations (e.g., head to a service center or charging/fueling station, pull over, stop immediately, self-diagnose, lock/unlock compartments, change music station, change temperature, and any other suitable operations), and instructing the vehicles 740 to enter select operation modes (e.g., operate normally, drive at a reduced speed, drive under the command of human operators, and any other suitable operational modes)).
KEHL, Hu, and Huang are considered to be analogous art because they pertain to image processing applications related to autonomous driving. Therefore, the combined teachings of KEHL, Hu, and Huang, as a whole, would have rendered obvious the invention recited in claim 8 with a reasonable expectation of success in order to the system and method for inferring a location of a three-dimensional (3D) object with respect to a location of an autonomous agent by extracting features from sensor data obtained from a number of sensors of an autonomous agent, which are encoded in data structures, such as 3D tensors (as disclosed by KEHL), with vehicles within a fleet of vehicles (as taught by Huang, Abstract, Par. [0072-84]) to provide techniques to improve object detection, to control operations of autonomous vehicles, to instruct vehicles to perform select operations, and to instruct vehicles to enter select operation modes (Huang, Abstract, Par. [001-2, 16, 72-84]).
Contact Information
Any inquiry concerning this communication or earlier communications from the examiner should be directed to GUILLERMO M RIVERA-MARTINEZ whose telephone number is (571) 272-4979. The examiner can normally be reached on 9 am to 5 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Bee can be reached on 571-270-5183. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see https://ppair-my.uspto.gov/pair/PrivatePair. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/GUILLERMO M RIVERA-MARTINEZ/ Primary Examiner, Art Unit 2677
Prosecution Insights