DETAILED ACTION
Notice of Pre-AIA or AIA Status
Claims 1-16 are pending in this application. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Specification
The title of the invention is not descriptive. A new title is required that is clearly indicative of the invention to which the claims are directed.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1, 15 and 16 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Boyraz et al. (US PGPub US20210405638 A1), hereby referred to as “Boyraz”.
Consider Claims 1, 15 and 16.
Boyraz teaches:
1. A method, comprising: / 15. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of an electronic device / 16. An electronic device, comprising: a first sensor; a second sensor that is different from the first sensor; (Boyraz: abstract, The present disclosure generally relates to a system of a delivery device for combining sensor data from various types of sensors to generate a map that enables the delivery device to navigate from a first location to a second location to deliver an item to the second location. The system obtains data from RGB, LIDAR, and depth sensors and combines this sensor data according to various algorithms to detect objects in an environment of the delivery device, generate point cloud and pose information associated with the detected objects, and generates object boundary data for the detected objects. The system further identifies object states for the detected object and generates the map for the environment based on the detected object, the generated object proposal data, the labeled point cloud data, and the object states. The generated map may be provided to other systems to navigate the delivery device. [0023]-[0032], Figures 1 and 2)
15. that is in communication with a first sensor and a second sensor different from the first sensor, the one or more programs including instructions for:/ 16. one or more processors; and memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for: (Bayroz: [0030] FIG. 2 is a flow diagram showing an illustrative perception system 200 used to acquire information for and perform one or more functions described herein for the robotic system 101 of FIG. 1 to autonomously travel from the first location to the second location. The perception system 200 may be a stack that includes sensors and processing components to reliably construct a local representation of an environment of the robotic system 101. The perception system 200 may enable detection of objects or obstacles in the environment of the robotic system 101 and allow the robotic system 101 to track or predict movement of the objects or obstacles. Such tracking or prediction of movement of the objects or obstacles enable the robotic system 101 to generate a path for the robotic system 101 to travel while avoiding (for example, travel around) the objects or obstacles. [0031] The perception system 200 may include, but is not limited to, one or more of a plurality of sensors 202, a localization module 204, an offline map server 206, an RGB depth network module 208, a geometric obstacle detection and terrain estimation module 210, a radar module 212, an object tracking and future state prediction module 214, a probabilistic three-dimensional (“3D”) map module 216, a birds-eye-view (BEV) map module 218, and a perception application programming interface (API) 220. The various modules of the perception system 200 may use data from the sensors 202 to generate a data flow and ultimately generate map information and future state predictions for use or analysis by the perception API 220.)
1. obtaining, via a first sensor, a first value corresponding to an object; / 15. obtaining, via the first sensor, a first value corresponding to an object; / 16. obtaining, via the first sensor, a first value corresponding to an object; (Boyraz: [0027] The perception system 200, as shown in further detail with respect to FIG. 2, may include a plurality of sensors, such as stereo depth sensors (for example, cameras or imaging sensors) for terrain and obstacle detection and RGB sensors (for example, light sensors or cameras that detect a color of a reflected surface and context information of neighboring pixels in a corresponding image) for semantic segmentation. In some embodiments, the sensors used in the robotic system 101 include active dense ranging sensors, such as a scanning or solid-state light detection and ranging (LIDAR) sensor, though other active ranging sensors are contemplated. While any one sensor or sensor type may be successful in detecting and capturing a subset of information, fusing data from various types of sensors may provide for a robust perception system. For example, fusing context rich data obtained from one or more RGB sensors with sparser but highly accurate range information obtained from one or more LIDAR sensors provides a more robust perception system than perception systems using only one of RGB/LIDAR sensors. [0028]-[0031])
1. obtaining, via a second sensor different from the first sensor, a second value corresponding to the object; / 15. obtaining, via the second sensor, a second value corresponding to the object; / 16. obtaining, via the second sensor, a second value corresponding to the object; (Boyraz: [0032] The sensors 202 may include a plurality of sensors, including a plurality of RGB sensors disposed on a front or front surface or a back or back surface of the robotic system 101, a plurality of ultrasonic sensors, a time-of-flight (TOF) sensor, stereo sensors, a LIDAR sensor, and so forth. The RGB sensors may include, but is not limited to, three RGB sensors with overlapping fields of view, where the three RGB sensors provide approximately a 180° horizontal field of view (each RGB sensors having approximately 70° horizontal field of view that overlaps with the neighboring RGB sensor(s)) and approximately a 110° vertical field of view in front of the robotic system 101. The RGB sensors may be used to generate RGB sensor data. The sensors 202 may also include an RGB sensor disposed at the back or rear of the robotic system 101. The sensors 202 also include a plurality of stereo depth sensors having similar fields of view or placements as the RGB sensors. The stereo depth sensors may be used to generate depth data. [0033] The sensors 202 further comprise a single LIDAR sensor mounted at the front of the robotic system 101 and configured to capture at least a fraction of the approximately 180° horizontal and 110° vertical fields of view of the RGB sensors. For example, the single LIDAR comprises a solid state LIDAR with 110° horizontal and 32° vertical fields of view. The sensors 202 may also include the TOF sensor mounted at the front of the robotic system 101 to provide up to one meter depth information coverage from the bottom of the robotic system 101. The sensors 202 may be disposed to identify a 360° environment of the robotic system 101. The sensors 202 provide image, environmental, and corresponding data to various modules of the perception system 200. [0034] The RGB depth network module 208 generates three (3) outputs based on the data received from the sensors 202 and the localization module 204. These outputs include: (1) 3D object proposal data 270 that is passed to an object tracking and future state prediction module 214; (2) labeled point cloud data 272 (for example, including terrain or obstacle data or semantic classes) to pass to the probabilistic 3D map module 216; and (3) curb traversability prediction data 274 passed to the BEV map module 218. In some embodiments, RGB depth network module 208 may combine data from one or more of the RGB sensors, LIDAR sensors, TOF sensors, or the stereo depth sensors to perform 3D object detection, semantic segmentation, terrain prediction, and depth prediction for partitions of the environment around the robotic system 101.)
1. in accordance with a determination that the object is classified as having a first level of criticalness, / 15. in accordance with a determination that the object is classified as having a first level of criticalness, / 16. in accordance with a determination that the object is classified as having a first level of criticalness, (Boyraz: [0065] Thus, cross features are used from cross components for the integration to take place in the sensor domain having the most information or most critical information. For example, for semantic segmentation, the increased context and information in the RGB sensor data as compared to the sparser LIDAR data suggests merging the LIDAR data or other depth data into the RGB sensor data and doing corresponding predictions/analysis with emphasis on the RGB sensor data. [0038] The 3D probabilistic map may represent the environment around the robotic system 101. The 3D probabilistic map may combine sensor data and other relevant information across time and from different sources…. Each grid cell (or partition) can include information about the environment, including:
A characterization of whether any cells identified as terrain can be further characterized into a set of classes of the terrain: sidewalk, road, lawn, driveway, etc.,
A characterization of whether any cells identified as obstacle/object can be further characterized into a set of classes the object (for example, person, car, pet, trash can, mailbox, bicycle, wheelchair, stroller, other) or unknown (obstacle), [0044] The perception system 200 may update states of each cell of the 3D probability map using a Bayesian model (such as a Kalman filter). The updated states for each cell may maintain probability distributions of each cell over class labels and every hypothesis to update state probability of the 3D cell. [0050] The RGB depth network 208 may combine the RGB data 252 and the combined dense point cloud and 3D pose data 262 to generate the 3D object proposal data 270, the labeled point cloud data 272 (for example, including terrain or obstacle data or semantic classes), and the curb traversability prediction data 274.)
1. fusing data associated with the first value and data associated with the second value; / 15. fusing data associated with the first value and data associated with the second value; / 16. fusing data associated with the first value and data associated with the second value; (Boyraz: [0050] The RGB depth network 208 may combine the RGB data 252 and the combined dense point cloud and 3D pose data 262 to generate the 3D object proposal data 270, the labeled point cloud data 272 (for example, including terrain or obstacle data or semantic classes), and the curb traversability prediction data 274. In some embodiments, the fusing involves information from various sensors and sensor types with different levels of accuracy and different types of acquired data (for example the context rich data obtained by the RGB sensors with point cloud data generated from an active ranging sensor). Such fusing of data from different sensors across time may relies on accurate extrinsic calibration, time synchronization, or 3D pose estimation information. The fusing may be performed by a localization stack or by the perception system 200 (for example, using 3D pose graph provided by the localization module 202). Additionally, the RGB depth module 208 may receive time-synchronized RGB images (for example, the RGB data 252 from the sensors 202 and aligned point cloud data (for example, the combined dense point cloud and 3D pose data 254 in the Odom frame (where the 3D pose of the robotic system 101 is at the time of the data capture by one or both of the RGB sensors and LIDAR sensors.)
1. and in accordance with a determination that the object is classified as having a second level of criticalness that is different from the first level of criticalness, / 15. and in accordance with a determination that the object is classified as having a second level of criticalness that is different from the first level of criticalness, / 16. and in accordance with a determination that the object is classified as having a second level of criticalness that is different from the first level of criticalness, (Boyraz: [0063] Effectively, the flow diagram of FIG. 3 shows that depth features (for example, from the LIDAR sensor) and other sensor features are projected into a viewpoint of the RGB sensor. When these features share the viewpoint of the RGB sensor, the backbone networks (for example, the image backbone network 306 or the depth backbone network 308) may perform feature extraction (extracting features in the data 356 and 358) which are then combined by the feature integration module 310 before the further analysis of the segmentation module 312, the depth completion module 314, and the 3D object detection module 316. More simply stated, features from the LIDAR data (and other sensors) are projected onto features of the RGB data. When the features overlap, the backbone networks may have more information for feature detection and when there is little or no overlap, the context information from the RGB data alone may be for feature detection. The feature integration is performed on any of the features detected and then the further processing by the modules 312-316 is completed. [0064]-[0065] For example, features are extracted from every RGB sensor viewpoint and every other sensor viewpoint and the information is combined using geometric warping or merging into or with data from other sensor types. For example, LIDAR features are obtained and geometrically warped into the data from the RGB sensors for the semantic segmentation (for example, by the segmentation module 312), the obstacle detection, and the depth completion (for example, by the depth completion module 314) while the features from the RGB sensor data are merged or applied to the LIDAR sensor data for 3D object detection, for example by the 3D object detection module 316. Effectively, when the processing (e.g., the segmentation, depth completion, or 3D object detection) is more efficient or effective in one sensor domain over another, the features from other sensors are warped into that one sensor domain for that particular processing. Thus, cross features are used from cross components for the integration to take place in the sensor domain having the most information or most critical information. For example, for semantic segmentation, the increased context and information in the RGB sensor data as compared to the sparser LIDAR data suggests merging the LIDAR data or other depth data into the RGB sensor data and doing corresponding predictions/analysis with emphasis on the RGB sensor data. For 3D object detection, where bounding box location and sizing is more important and more accurately determined from LIDAR sensor data, the RGB data is merged into the LIDAR sensor data to complete that analysis. As described herein, when data from a first sensor is merged with data from a second sensor, the data from the first sensor is used to complement the data from the second sensor. In some embodiments, one or more steps, routines, or processes described with relation to FIG. 3 may be implemented by one or more of the components described herein, for example one or more of the components of FIG. 2, such as the RGB Depth Fusion Network module 208.)
1. selecting the first value or the second value. / 15. selecting the first value or the second value. / 16. selecting the first value or the second value. (Boyraz: [0066] In some aspects, outputs of ML networks and components used herein, for example neural networks applied in backbone networks, may apply a softmax function to normalize object class scores for a pixel across all possible object classes so that the sum of all object class scores is 1. As such, the object class of all possible object classes having the highest value or score is selected to be the label for the pixel. As an example, an output of the segmentation module 312 includes a H×W×N channel matrix where H×W is an input RGB image size in pixels and N is a number of semantic labels or channels (such as road, sidewalk, and so forth). Each channel may correspond to a certain label, (for example, channel 0 corresponds to the sidewalk label, channel 1 corresponds to the road label, and so forth, and each channel may have a score that is representative of a likelihood that that pixel should be labeled according to that channel. For each pixel location in the H×W image, a sum of N label or channel scores is always 1. The segmentation module 312 may select the channel for each pixel that has a highest score to label the pixel and uses the score for that channel as the confidence score for that pixel. For the 3D object detection module 316, similarly the outputs of the corresponding ML networks and components used therein produce a 1×K normalized class score vector, where K is a number of types of objects (such as person, car, and so forth) that may be detected in the environment of the robotic system 101.)
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent may not be obtained though the invention is not identically disclosed or described as set forth in section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are such that the subject matter as a whole would have been obvious at the time the invention was made to a person having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the manner in which the invention was made.
Claims 1-16 are rejected under 35 U.S.C. 103 as being unpatentable over Boyraz et al. (US PGPub US20210405638 A1), hereby referred to as “Boyraz” in view of Woody et al. (US PGPub US20180262711 A1), hereby referred to as “Woody”.
Consider Claims 1, 15 and 16.
Boyraz teaches:
1. A method, comprising: / 15. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of an electronic device / 16. An electronic device, comprising: a first sensor; a second sensor that is different from the first sensor; (Boyraz: abstract, The present disclosure generally relates to a system of a delivery device for combining sensor data from various types of sensors to generate a map that enables the delivery device to navigate from a first location to a second location to deliver an item to the second location. The system obtains data from RGB, LIDAR, and depth sensors and combines this sensor data according to various algorithms to detect objects in an environment of the delivery device, generate point cloud and pose information associated with the detected objects, and generates object boundary data for the detected objects. The system further identifies object states for the detected object and generates the map for the environment based on the detected object, the generated object proposal data, the labeled point cloud data, and the object states. The generated map may be provided to other systems to navigate the delivery device. [0023]-[0032], Figures 1 and 2)
15. that is in communication with a first sensor and a second sensor different from the first sensor, the one or more programs including instructions for:/ 16. one or more processors; and memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for: (Bayroz: [0030] FIG. 2 is a flow diagram showing an illustrative perception system 200 used to acquire information for and perform one or more functions described herein for the robotic system 101 of FIG. 1 to autonomously travel from the first location to the second location. The perception system 200 may be a stack that includes sensors and processing components to reliably construct a local representation of an environment of the robotic system 101. The perception system 200 may enable detection of objects or obstacles in the environment of the robotic system 101 and allow the robotic system 101 to track or predict movement of the objects or obstacles. Such tracking or prediction of movement of the objects or obstacles enable the robotic system 101 to generate a path for the robotic system 101 to travel while avoiding (for example, travel around) the objects or obstacles. [0031] The perception system 200 may include, but is not limited to, one or more of a plurality of sensors 202, a localization module 204, an offline map server 206, an RGB depth network module 208, a geometric obstacle detection and terrain estimation module 210, a radar module 212, an object tracking and future state prediction module 214, a probabilistic three-dimensional (“3D”) map module 216, a birds-eye-view (BEV) map module 218, and a perception application programming interface (API) 220. The various modules of the perception system 200 may use data from the sensors 202 to generate a data flow and ultimately generate map information and future state predictions for use or analysis by the perception API 220.)
1. obtaining, via a first sensor, a first value corresponding to an object; / 15. obtaining, via the first sensor, a first value corresponding to an object; / 16. obtaining, via the first sensor, a first value corresponding to an object; (Boyraz: [0027] The perception system 200, as shown in further detail with respect to FIG. 2, may include a plurality of sensors, such as stereo depth sensors (for example, cameras or imaging sensors) for terrain and obstacle detection and RGB sensors (for example, light sensors or cameras that detect a color of a reflected surface and context information of neighboring pixels in a corresponding image) for semantic segmentation…. In some embodiments, the sensors used in the robotic system 101 include active dense ranging sensors, such as a scanning or solid-state light detection and ranging (LIDAR) sensor, though other active ranging sensors are contemplated. While any one sensor or sensor type may be successful in detecting and capturing a subset of information, fusing data from various types of sensors may provide for a robust perception system. For example, fusing context rich data obtained from one or more RGB sensors with sparser but highly accurate range information obtained from one or more LIDAR sensors provides a more robust perception system than perception systems using only one of RGB or LIDAR sensors. [0028]-[0031])
1. obtaining, via a second sensor different from the first sensor, a second value corresponding to the object; / 15. obtaining, via the second sensor, a second value corresponding to the object; / 16. obtaining, via the second sensor, a second value corresponding to the object; (Boyraz: [0032] The sensors 202 may include a plurality of sensors, including a plurality of RGB sensors disposed on a front or front surface or a back or back surface of the robotic system 101, a plurality of ultrasonic sensors, a time-of-flight (TOF) sensor, stereo sensors, a LIDAR sensor, and so forth. The RGB sensors may include, but is not limited to, three RGB sensors with overlapping fields of view, where the three RGB sensors provide approximately a 180° horizontal field of view (each RGB sensors having approximately 70° horizontal field of view that overlaps with the neighboring RGB sensor(s)) and approximately a 110° vertical field of view in front of the robotic system 101. The RGB sensors may be used to generate RGB sensor data. The sensors 202 may also include an RGB sensor disposed at the back or rear of the robotic system 101. The sensors 202 also include a plurality of stereo depth sensors having similar fields of view or placements as the RGB sensors. The stereo depth sensors may be used to generate depth data. [0033] The sensors 202 further comprise a single LIDAR sensor mounted at the front of the robotic system 101 and configured to capture at least a fraction of the approximately 180° horizontal and 110° vertical fields of view of the RGB sensors. For example, the single LIDAR comprises a solid state LIDAR with 110° horizontal and 32° vertical fields of view. The sensors 202 may also include the TOF sensor mounted at the front of the robotic system 101 to provide up to one meter depth information coverage from the bottom of the robotic system 101. The sensors 202 may be disposed to identify a 360° environment of the robotic system 101. The sensors 202 provide image, environmental, and corresponding data to various modules of the perception system 200. [0034] The RGB depth network module 208 generates three (3) outputs based on the data received from the sensors 202 and the localization module 204. These outputs include: (1) 3D object proposal data 270 that is passed to an object tracking and future state prediction module 214; (2) labeled point cloud data 272 (for example, including terrain or obstacle data or semantic classes) to pass to the probabilistic 3D map module 216; and (3) curb traversability prediction data 274 passed to the BEV map module 218. In some embodiments, RGB depth network module 208 may combine data from one or more of the RGB sensors, LIDAR sensors, TOF sensors, or the stereo depth sensors to perform 3D object detection, semantic segmentation, terrain prediction, and depth prediction for partitions of the environment around the robotic system 101.)
1. in accordance with a determination that the object is classified as having a first level of criticalness, / 15. in accordance with a determination that the object is classified as having a first level of criticalness, / 16. in accordance with a determination that the object is classified as having a first level of criticalness, (Boyraz: [0065] Thus, cross features are used from cross components for the integration to take place in the sensor domain having the most information or most critical information. For example, for semantic segmentation, the increased context and information in the RGB sensor data as compared to the sparser LIDAR data suggests merging the LIDAR data or other depth data into the RGB sensor data and doing corresponding predictions/analysis with emphasis on the RGB sensor data. [0038] The 3D probabilistic map may represent the environment around the robotic system 101. The 3D probabilistic map may combine sensor data and other relevant information across time and from different sources…. Each grid cell (or partition) can include information about the environment, including:
A characterization of whether any cells identified as terrain can be further characterized into a set of classes of the terrain: sidewalk, road, lawn, driveway, etc.,
A characterization of whether any cells identified as obstacle/object can be further characterized into a set of classes the object (for example, person, car, pet, trash can, mailbox, bicycle, wheelchair, stroller, other) or unknown (obstacle), [0044] The perception system 200 may update states of each cell of the 3D probability map using a Bayesian model (such as a Kalman filter). The updated states for each cell may maintain probability distributions of each cell over class labels and every hypothesis to update state probability of the 3D cell. [0050] The RGB depth network 208 may combine the RGB data 252 and the combined dense point cloud and 3D pose data 262 to generate the 3D object proposal data 270, the labeled point cloud data 272 (for example, including terrain or obstacle data or semantic classes), and the curb traversability prediction data 274.)
1. fusing data associated with the first value and data associated with the second value; / 15. fusing data associated with the first value and data associated with the second value; / 16. fusing data associated with the first value and data associated with the second value; (Boyraz: [0050] The RGB depth network 208 may combine the RGB data 252 and the combined dense point cloud and 3D pose data 262 to generate the 3D object proposal data 270, the labeled point cloud data 272 (for example, including terrain or obstacle data or semantic classes), and the curb traversability prediction data 274. In some embodiments, the fusing involves information from various sensors and sensor types with different levels of accuracy and different types of acquired data (for example the context rich data obtained by the RGB sensors with point cloud data generated from an active ranging sensor). Such fusing of data from different sensors across time may relies on accurate extrinsic calibration, time synchronization, or 3D pose estimation information. The fusing may be performed by a localization stack or by the perception system 200 (for example, using 3D pose graph provided by the localization module 202). Additionally, the RGB depth module 208 may receive time-synchronized RGB images (for example, the RGB data 252 from the sensors 202 and aligned point cloud data (for example, the combined dense point cloud and 3D pose data 254 in the Odom frame (where the 3D pose of the robotic system 101 is at the time of the data capture by one or both of the RGB sensors and LIDAR sensors.)
1. and in accordance with a determination that the object is classified as having a second level of criticalness that is different from the first level of criticalness, / 15. and in accordance with a determination that the object is classified as having a second level of criticalness that is different from the first level of criticalness, / 16. and in accordance with a determination that the object is classified as having a second level of criticalness that is different from the first level of criticalness, (Boyraz: [0063] Effectively, the flow diagram of FIG. 3 shows that depth features (for example, from the LIDAR sensor) and other sensor features are projected into a viewpoint of the RGB sensor. When these features share the viewpoint of the RGB sensor, the backbone networks (for example, the image backbone network 306 or the depth backbone network 308) may perform feature extraction (extracting features in the data 356 and 358) which are then combined by the feature integration module 310 before the further analysis of the segmentation module 312, the depth completion module 314, and the 3D object detection module 316. More simply stated, features from the LIDAR data (and other sensors) are projected onto features of the RGB data. When the features overlap, the backbone networks may have more information for feature detection and when there is little or no overlap, the context information from the RGB data alone may be for feature detection. The feature integration is performed on any of the features detected and then the further processing by the modules 312-316 is completed. [0064]-[0065] For example, features are extracted from every RGB sensor viewpoint and every other sensor viewpoint and the information is combined using geometric warping or merging into or with data from other sensor types. For example, LIDAR features are obtained and geometrically warped into the data from the RGB sensors for the semantic segmentation (for example, by the segmentation module 312), the obstacle detection, and the depth completion (for example, by the depth completion module 314) while the features from the RGB sensor data are merged or applied to the LIDAR sensor data for 3D object detection, for example by the 3D object detection module 316. Effectively, when the processing (e.g., the segmentation, depth completion, or 3D object detection) is more efficient or effective in one sensor domain over another, the features from other sensors are warped into that one sensor domain for that particular processing. Thus, cross features are used from cross components for the integration to take place in the sensor domain having the most information or most critical information. For example, for semantic segmentation, the increased context and information in the RGB sensor data as compared to the sparser LIDAR data suggests merging the LIDAR data or other depth data into the RGB sensor data and doing corresponding predictions/analysis with emphasis on the RGB sensor data. For 3D object detection, where bounding box location and sizing is more important and more accurately determined from LIDAR sensor data, the RGB data is merged into the LIDAR sensor data to complete that analysis. As described herein, when data from a first sensor is merged with data from a second sensor, the data from the first sensor is used to complement the data from the second sensor. In some embodiments, one or more steps, routines, or processes described with relation to FIG. 3 may be implemented by one or more of the components described herein, for example one or more of the components of FIG. 2, such as the RGB Depth Fusion Network module 208.)
1. selecting the first value or the second value. / 15. selecting the first value or the second value. / 16. selecting the first value or the second value. (Boyraz: [0066] In some aspects, outputs of ML networks and components used herein, for example neural networks applied in backbone networks, may apply a softmax function to normalize object class scores for a pixel across all possible object classes so that the sum of all object class scores is 1. As such, the object class of all possible object classes having the highest value or score is selected to be the label for the pixel. As an example, an output of the segmentation module 312 includes a H×W×N channel matrix where H×W is an input RGB image size in pixels and N is a number of semantic labels or channels (such as road, sidewalk, and so forth). Each channel may correspond to a certain label, (for example, channel 0 corresponds to the sidewalk label, channel 1 corresponds to the road label, and so forth, and each channel may have a score that is representative of a likelihood that that pixel should be labeled according to that channel. For each pixel location in the H×W image, a sum of N label or channel scores is always 1. The segmentation module 312 may select the channel for each pixel that has a highest score to label the pixel and uses the score for that channel as the confidence score for that pixel. For the 3D object detection module 316, similarly the outputs of the corresponding ML networks and components used therein produce a 1×K normalized class score vector, where K is a number of types of objects (such as person, car, and so forth) that may be detected in the environment of the robotic system 101.)
Even if Boyraz does not teach: “a determination that the object is classified as having a second level of criticalness that is different from the first level of criticalness”
Woody teaches:
1. A method, comprising: / 15. A non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of an electronic device / 16. An electronic device, comprising: a first sensor; a second sensor that is different from the first sensor; (Woody: abstract, A sensor data processing apparatus can be coupled to multiple image sensors of different types. The apparatus adjusts frame transmission rates based on the number of sensors and type of image data sourced by the sensors to optimize utilization of bandwidth on a number of transport channels. The apparatus can be configured to transport selected frames in the image data that are identified as critical frames at a higher rate than non-selected frames in the image data. [0033]-[0035], Figures 1-2)
15. that is in communication with a first sensor and a second sensor different from the first sensor, the one or more programs including instructions for:/ 16. one or more processors; and memory storing one or more programs configured to be executed by the one or more processors, the one or more programs including instructions for: (Woody: [0033] An illustrative embodiment of a UHD sensor data processing system according to an aspect of the present disclosure is described with reference to FIG. 1. The system 100 includes UHD segmentation circuitry 102 coupled to a UHD image sensor 104 via a raw UHD data input path 106. In the illustrative embodiment, the system 100 also includes video processing circuitry 108 coupled to the UHD segmentation circuitry 102 via a number of image data output paths 110 and one or more metadata paths 112. The data output paths 110 and the metadata paths 112 may coexist on the same conductive pathway or may be alternatively be configured on separate conductive pathways. [0034] In the illustrative embodiment the UHD segmentation circuitry 102 includes memory circuitry coupled to processor circuitry. The processor circuitry is configured to receive raw UHD data from the UHD image sensor 104, divide the raw UHD data into lossless segments and direct the lossless segments in parallel onto the image data output paths 110. In the illustrative embodiment, the processor circuitry is also configured to generate metadata including encoded information that facilitates reconstruction of the raw UHD data from the lossless segments, and to direct the metadata onto the metadata output paths 112.)
1. obtaining, via a first sensor, a first value corresponding to an object; / 15. obtaining, via the first sensor, a first value corresponding to an object; / 16. obtaining, via the first sensor, a first value corresponding to an object; (Woody: [0035] A method for processing UHD sensor data according to an aspect of the present disclosure is described with reference to FIG. 2. The method 200 includes receiving raw UHD data from a UHD sensor, such as a UHD image sensor 104 of FIG. 1, at block 202 and dividing the raw UHD data into lossless segments at block 204. In an illustrative embodiment the raw UHD data is divided by UHD segmentation circuitry 102, of FIG. 1 which may include a series of FPGA and processing systems, for example. In the illustrative embodiment the UHD segmentation circuitry 102 of FIG. 1 includes digital video processor (DVP) circuitry that receives the video from the UHD image sensor 104 and divides it up into multiple 720p images. [0036] In an illustrative embodiment, the UHD segmentation circuitry 102 of FIG. 1 includes SMPTE video processor (SVP) circuitry that receives the 720p images from the DVP, circuitry, divides them into appropriately formatted SMPTE 1080p video frames, and adds appropriately formatted SMPTE metadata to ancillary video space. The metadata includes packing details, such as pixel location of start of frame and end of frame, frame rate, bit depth, bit packing mode, etc. The same metadata space has provisions for giving line of sight, or pointing information indicating where the UHD image sensor 104 was pointed for each applicable frame so that this information can be used to add context to the UHD video frame captured by the UHD image sensor 104. [0052] In an illustrative embodiment, the sensor data processing apparatus 600 includes processing circuitry, a raw UHD video data input path coupled to the processing circuitry, and a number of image data output paths coupled in parallel to the processing circuitry. The sensor data processing apparatus 600 also includes one or more metadata output paths coupled to the processing circuitry in parallel with the image data output paths, and a bandwidth monitor module 602 coupled to the image data output paths.)
1. obtaining, via a second sensor different from the first sensor, a second value corresponding to the object; / 15. obtaining, via the second sensor, a second value corresponding to the object; / 16. obtaining, via the second sensor, a second value corresponding to the object; (Woody: [0058] Referring to FIG. 8, parallel video streams each include their own horizontal ancillary (HANC) metadata space 802 and VANC metadata space 804. According to an aspect of the present disclosure, unique time aligned packing and spreading information is included in each VANC metadata space 804 for each frame 806. [0068] Referring to FIG. 10, a bit rate streaming or frame per second (FPS) throttle (throttle module) 1002 is configured to detect multiple SMPTE connections between SMPTE video processor 312 and video processor 316. When multiple SMPTE connections are detected, the throttle module 1002 sends data from the multiple SMPTE connections along parallel channels. In an illustrative embodiment, the throttle module 1002 is used in conjunction with existing algorithms to dynamically adapt bit rates of the video to achieve a balance between latency and resolution without loss of video data. [0069] According to an aspect of the present disclosure, the throttle module 1002 first detects the number of physical connections between SMPTE video processor 312 and video processor 316. The throttle module 1002 may be configured to select compression techniques and data paths based on the number of physical connections between SMPTE video processor 312 and video processor 316. The compression techniques and data paths may be selected based on configurable parameters for compression options and/or predetermined timing constraints that may be programmed in software or firmware of the throttle module 1002, for example. In an illustrative embodiment, additional pixel packing can be performed to maximize use of the SMPTE pixel space that is defined according to SMPTE standards.)
1. in accordance with a determination that the object is classified as having a first level of criticalness, / 15. in accordance with a determination that the object is classified as having a first level of criticalness, / 16. in accordance with a determination that the object is classified as having a first level of criticalness, (Woody: [0070] According to another aspect of the present disclosure, the throttle module 1002 may be configured to identify user-defined critical regions of an image and transport data corresponding to the critical regions between SMPTE video processor 312 and video processor 316 at a higher rate than data is transferred for other areas of the image. In an illustrative embodiment, critical regions may be identified based on user input wherein the throttle module is in communication with a user interface to receive parameters defining the critical region from a user, for example. In an alternative embodiment, the throttle module may be configured to identify a predetermined area of each frame, such as a center area, for example. [0071] Referring to FIG. 11, in an illustrative embodiment, a user selects an HD image region by identifying it as critical region. The throttle module 1002 is configured to identify image data in the critical region and transport the data corresponding to the critical region at full rate along one channel 1102 to critical area memory space 1105 of a display 1106. In an illustrative embodiment, data for the critical region is transported as unpacked pixels to the display 1106 on every output cycle of the throttle module 1002 so that every frame in the critical region that is received from the image sensor 302 is transported to the display 1106.)
1. fusing data associated with the first value and data associated with the second value; / 15. fusing data associated with the first value and data associated with the second value; / 16. fusing data associated with the first value and data associated with the second value; (Woody: [0071] Referring to FIG. 11, in an illustrative embodiment, a user selects an HD image region by identifying it as critical region. The throttle module 1002 is configured to identify image data in the critical region and transport the data corresponding to the critical region at full rate along one channel 1102 to critical area memory space 1105 of a display 1106. In an illustrative embodiment, data for the critical region is transported as unpacked pixels to the display 1106 on every output cycle of the throttle module 1002 so that every frame in the critical region that is received from the image sensor 302 is transported to the display 1106. [0072] The throttle module 1002 allocates the remaining available connections to transport the remaining video and associated metadata as packed pixels (2 pixels for every 16 bits in the SMPTE stream). The packed pixels are unpacked based on the associated metadata and transported to non-critical area memory space 1107, 1109 the display 1106 along a number of parallel channels 1104 at less than the full rate. In the illustrative embodiment, the throttle module 1002 sends alternating portions of the data received from the image sensor 302 for areas outside of the critical area to the non-critical area of memory space 1107, 1109 of the display 1106 on every other output cycle of the throttle module 1002. For example, in this embodiment, the throttle module 1002 couples a first non-critical area of memory space 1107 to the parallel channels 1104 on even numbered (N) frame cycles of the throttle module 1002, and couples a second non-critical area 1109 of memory space to parallel channels 1104 on odd numbered (N+1) frame cycles of the throttle module 1002. In this embodiment, different non-critical regions of each image received from image sensor 302 are updated in the display 1106 every other cycle in an alternating sequence while critical regions of each image received from the image sensor 302 are updated on every cycle.)
1. and in accordance with a determination that the object is classified as having a second level of criticalness that is different from the first level of criticalness, / 15. and in accordance with a determination that the object is classified as having a second level of criticalness that is different from the first level of criticalness, / 16. and in accordance with a determination that the object is classified as having a second level of criticalness that is different from the first level of criticalness, (Woody: [0072] The throttle module 1002 allocates the remaining available connections to transport the remaining video and associated metadata as packed pixels (2 pixels for every 16 bits in the SMPTE stream). The packed pixels are unpacked based on the associated metadata and transported to non-critical area memory space 1107, 1109 the display 1106 along a number of parallel channels 1104 at less than the full rate. In the illustrative embodiment, the throttle module 1002 sends alternating portions of the data received from the image sensor 302 for areas outside of the critical area to the non-critical area of memory space 1107, 1109 of the display 1106 on every other output cycle of the throttle module 1002. For example, in this embodiment, the throttle module 1002 couples a first non-critical area of memory space 1107 to the parallel channels 1104 on even numbered (N) frame cycles of the throttle module 1002, and couples a second non-critical area 1109 of memory space to parallel channels 1104 on odd numbered (N+1) frame cycles of the throttle module 1002. In this embodiment, different non-critical regions of each image received from image sensor 302 are updated in the display 1106 every other cycle in an alternating sequence while critical regions of each image received from the image sensor 302 are updated on every cycle. [0073] Although FIG. 11 is described with respect to one critical region of an image and a number of non-critical regions of the image, it should be understood that alternative embodiments of the disclosed system and method could be implemented in which multiple critical areas are pre-determined or selected by a user. Persons skilled in the art should understand that various alternative multiplexing techniques may be used by the throttle module to transport a number of critical regions of images from the image sensor 302 to the display 1106 at a higher transport rate than non-critical regions of the images.)
1. selecting the first value or the second value. / 15. selecting the first value or the second value. / 16. selecting the first value or the second value. (Woody: [0073]-[0074] A method of transporting video data from a UHD image sensor 302 according to an aspect of the present disclosure is described with reference to FIG. 12. In an illustrative embodiment, one or more steps of the method 1200 may be performed by the throttle module 1002 of FIG. 10 and FIG. 11, for example. The method 1200 includes receiving from the UHD image sensor 302 a stream of images, wherein each of the images includes a number of UHD frames at step 1202 and identifying one or more of the UHD frames as selected frames