Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Acknowledgment is made of applicant’s claim for foreign priority under 35 U.S.C. 119 (a)-(d). The certified copy has been filed of DE Application No. 10 2023 209 728.1, filed on 10/05/2023.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 09/27/2024 is being considered by the examiner.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
Claims 1-13 are rejected under 35 U.S.C. 103 as being unpatentable over Liang (US PGPub 2020/0025931) in view of Hu (“Squeeze-and-Excitation Networks,” 2018).
Regarding claim 1, Liang discloses A method for aligning two map datasets for determining navigation information for a mobile device that is moving or is to move in an environment, the method comprising the following steps [Liang ¶ 0006 “ The method also includes detecting, by the computing system, based on the map-modified LIDAR data, three-dimensional objects of interest within the environment.”]:
providing two map datasets, each containing environmental information, wherein in the two map datasets the environmental information was collected from the mobile device and/or the environment by a sensor of the mobile device, and wherein at least one of the two map datasets is a sparse map dataset [Liang ¶ 0045 "In some implementations, the machine-learned neural network can include one or more fusion layers that are configured to fuse image features from image data (e.g., image data captured by a camera system within an autonomous vehicle) with LIDAR features from LIDAR point cloud data (e.g., LIDAR point cloud data captured by a LIDAR system within an autonomous vehicle)." Two map datasets = LIDAR point cloud data and camera image data];
providing the two map datasets as input feature data or determining the input feature data based on the two map datasets [Liang ¶ 0108 "In some implementations, the machine-learned sensor fusion model 300 can include one or more fusion layers 306 that are configured to fuse image features from image data (e.g., image data 304) with LIDAR features from LIDAR point cloud data (e.g., BEV LIDAR data 302)."];
carrying out an alignment of the two map datasets using a machine learning algorithm which includes a convolutional neural network based on sparse convolution, wherein output data are generated from the input feature data via intermediate feature data in one or more intermediate layers, wherein the output data include information about a transformative relation between the two map datasets, wherein the carrying out of the alignment includes one or more adjustment operations, each including [Liang ¶ 0108 "In some implementations, the machine-learned sensor fusion model 300 can include one or more fusion layers 306 that are configured to fuse image features from image data (e.g., image data 304) with LIDAR features from LIDAR point cloud data (e.g., BEV LIDAR data 302)." and ¶ 0110 "Example neural networks include feed-forward neural networks, (fully) convolutional neural networks, recurrent neural networks (e.g., long short-term memory recurrent neural networks), or other forms of neural networks." and ¶ 0173 "At 610, method 600 can include generating a feature map comprising the fused image features from the first data stream and the LIDAR features from the second data stream." output data = feature map]:
adjusting the intermediate feature data of the one or of another of the one or more intermediate layers based on the global map dataset [Liang ¶ 0060 " the means can be configured to receive a target data point associated with the image data, extract a plurality of source data points associated with the LIDAR point cloud data based on a distance of each source data point to the target data point (e.g., using a KNN pooling technique), and fuse information from the plurality of source data points in the one or more fusion layers to generate an output feature at the target data point (e.g., by concatenating a plurality of LIDAR features associated with the LIDAR point cloud data at the plurality of source data points)."]; and
providing the output data for use in determining the navigation information [Liang ¶ 0053 " computing system associated with an autonomous vehicle can implement additional autonomy processing functionality based on the output of the machine-learned detector model. For example, a motion planning system can determine a motion plan for the autonomous vehicle based at least in part on the detection output(s) and forecasting output(s)."].
Liang does not teach determining a global map dataset based on feature data, using a global pooling operation, wherein the feature data include the input feature data or the intermediate feature data of one of the one or more intermediate layers.
However, in a related field of invention, Hu does teach determining a global map dataset based on feature data, using a global pooling operation, wherein the feature data include the input feature data or the intermediate feature data of one of the one or more intermediate layers [Hu, page 7134 column 1 "To mitigate this problem, we propose to squeeze global spatial information into a channel descriptor. This is achieved by using global average pooling to generate channel-wise statistics."].
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention with a reasonable expectation of success to substitute the adjusting of intermediate layers based on a KNN pooling technique as taught by Liang with using a global pooling technique as taught by Hu in order to increase the accuracy of the convolutional neural network [Hu, page 7134, column 1].
Regarding claim 2, Liang as modified by Hu teach claim 1. Liang further teaches wherein the adjusting of the intermediate feature data includes at least one of the following procedures:
addition of the global map dataset and the intermediate feature data,
multiplication of the global map dataset by the intermediate feature data,
concatenation of the global map dataset and the intermediate feature data [Liang ¶ 0060 " the means can be configured to receive a target data point associated with the image data, extract a plurality of source data points associated with the LIDAR point cloud data based on a distance of each source data point to the target data point (e.g., using a KNN pooling technique), and fuse information from the plurality of source data points in the one or more fusion layers to generate an output feature at the target data point (e.g., by concatenating a plurality of LIDAR features associated with the LIDAR point cloud data at the plurality of source data points)."].
Regarding claim 3, Liang as modified by Hu teach claim 1. Liang further teaches wherein the adjusting of the intermediate feature data, when a number of channels of the global map dataset and the intermediate feature data differ from one another, includes an interposed convolution [Liang ¶ 0127 "To combine high-level context with low-level details, some embodiments combine the output features from each convolutional block via re-sizing and channel concatenation."].
Regarding claim 4, Liang as modified by Hu teach claim 1. Liang further teaches wherein the carrying out of the alignment includes multiple adjustment operations [Liang ¶ 0127-128 “batch normalization”, “MaxPool layer”, and “conv2D layer may be applied “and Figure 7].
Regarding claim 5, Liang as modified by Hu teach claim 4. Liang further teaches wherein different intermediate feature data are adjusted in different adjustment operations of the multiple adjustment operations [Liang ¶ 0127-128 “batch normalization”, “MaxPool layer”, and “conv2D layer may be applied “and Figure 7].
Regarding claim 6, Liang as modified by Hu teach claim 4. Hu further teaches wherein in different adjustment operations of the multiple adjustment operations, different global map datasets are determined based on different feature data [Hu, page 7134 column 2 "The activations act as channel weights adapted to the input-specific descriptor z. In this regard, SE blocks intrinsically introduce dynamics conditioned on the input, helping to boost feature discriminability."].
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention with a reasonable expectation of success to substitute the adjusting of intermediate layers based on a KNN pooling technique as taught by Liang with using a global pooling technique that change based on different feature data as taught by Hu in order to increase the accuracy of the convolutional neural network [Hu, page 7134, column 1].
Regarding claim 7, Liang as modified by Hu teach claim 1. Liang further teaches wherein the sensor of the mobile device includes one of the following sensors: a camera, a radar sensor, a lidar sensor, an ultrasonic sensor [Liang ¶ 0081 "The one or more autonomy system sensors 114 can include a Light Detection and Ranging (LIDAR) system, a Radio Detection and Ranging (RADAR) system, one or more cameras (e.g., visible spectrum cameras and/or infrared cameras), motion sensors, and/or other types of imaging capture devices and/or sensors."].
Regarding claim 8, Liang as modified by Hu teach claim 1. Liang further teaches wherein the environmental information of at least one of the two map datasets includes images and/or positions of at least one of the following objects: lane markings, road posts, traffic signs [Liang ¶ 0082 "The map data 122 can provide detailed information about the surrounding environment of the vehicle 102. For example, the map data 122 can provide information regarding: the identity and location of different roadways, road segments, buildings, or other items or objects (e.g., lampposts, crosswalks and/or curb); the location and directions of traffic lanes (e.g., the location and direction of a parking lane, a turning lane, a bicycle lane, or other lanes within a particular roadway or other travel way and/or one or more boundary markings associated therewith);"].
Regarding claim 9, Liang as modified by Hu teach claim 1. Liang further teaches further comprising:
determining the navigation information based on the output data, wherein the navigation information includes a map of the environment and/or a trajectory for the mobile device [Liang ¶ 0053-54 " computing system associated with an autonomous vehicle can implement additional autonomy processing functionality based on the output of the machine-learned detector model. For example, a motion planning system can determine a motion plan for the autonomous vehicle based at least in part on the detection output(s) and forecasting output(s)."].
Regarding claim 10, all limitations have been examined with respect to the method in claim 1. The method taught/disclosed in claim 1 can clearly perform on the system of claim 10. Therefore, claim 10 is rejected under the same rationale.
Regarding claim 11, Liang teaches A mobile device configured to obtain navigation information determined by [Liang ¶ 0070 "As illustrated, FIG. 1 shows a system 100 that can include a vehicle 102" and Figure 1]
providing two map datasets, each containing environmental information, wherein in the two map datasets the environmental information was collected from the mobile device and/or an environment of the mobile device by a sensor of the mobile device, and wherein at least one of the two map datasets is a sparse map dataset [Liang ¶ 0045 "In some implementations, the machine-learned neural network can include one or more fusion layers that are configured to fuse image features from image data (e.g., image data captured by a camera system within an autonomous vehicle) with LIDAR features from LIDAR point cloud data (e.g., LIDAR point cloud data captured by a LIDAR system within an autonomous vehicle)." Two map datasets = LIDAR point cloud data and camera image data];
providing the two map datasets as input feature data or determining the input feature data based on the two map datasets [Liang ¶ 0108 "In some implementations, the machine-learned sensor fusion model 300 can include one or more fusion layers 306 that are configured to fuse image features from image data (e.g., image data 304) with LIDAR features from LIDAR point cloud data (e.g., BEV LIDAR data 302)."];
carrying out an alignment of the two map datasets using a machine learning algorithm which includes a convolutional neural network based on sparse convolution, wherein output data are generated from the input feature data via intermediate feature data in one or more intermediate layers, wherein the output data include information about a transformative relation between the two map datasets, wherein the carrying out of the alignment includes one or more adjustment operations, each including [Liang ¶ 0108 "In some implementations, the machine-learned sensor fusion model 300 can include one or more fusion layers 306 that are configured to fuse image features from image data (e.g., image data 304) with LIDAR features from LIDAR point cloud data (e.g., BEV LIDAR data 302)." and ¶ 0110 "Example neural networks include feed-forward neural networks, (fully) convolutional neural networks, recurrent neural networks (e.g., long short-term memory recurrent neural networks), or other forms of neural networks." and ¶ 0173 "At 610, method 600 can include generating a feature map comprising the fused image features from the first data stream and the LIDAR features from the second data stream." output data = feature map]:
adjusting the intermediate feature data of the one or of another of the one or more intermediate layers based on the global map dataset [Liang ¶ 0060 " the means can be configured to receive a target data point associated with the image data, extract a plurality of source data points associated with the LIDAR point cloud data based on a distance of each source data point to the target data point (e.g., using a KNN pooling technique), and fuse information from the plurality of source data points in the one or more fusion layers to generate an output feature at the target data point (e.g., by concatenating a plurality of LIDAR features associated with the LIDAR point cloud data at the plurality of source data points)."];
providing the output data for use in determining the navigation information [Liang ¶ 0053 " computing system associated with an autonomous vehicle can implement additional autonomy processing functionality based on the output of the machine-learned detector model. For example, a motion planning system can determine a motion plan for the autonomous vehicle based at least in part on the detection output(s) and forecasting output(s)."].; and
determining the navigation information based on the output data, wherein the navigation information includes a map of the environment and/or a trajectory for the mobile device [Liang ¶ 0087 “motion planning system 128”];
wherein the mobile device has a sensor for detecting the environmental information and is configured to navigate based on the navigation information with a control or regulating unit and a drive unit for moving the mobile device according to the navigation information [Liang ¶ 0053-54 " computing system associated with an autonomous vehicle can implement additional autonomy processing functionality based on the output of the machine-learned detector model. For example, a motion planning system can determine a motion plan for the autonomous vehicle based at least in part on the detection output(s) and forecasting output(s)."].
Liang does not teach determining a global map dataset based on feature data, using a global pooling operation, wherein the feature data include the input feature data or the intermediate feature data of one of the one or more intermediate layers.
However, in a related field of invention, Hu does teach determining a global map dataset based on feature data, using a global pooling operation, wherein the feature data include the input feature data or the intermediate feature data of one of the one or more intermediate layers [Hu, page 7134 column 1 "To mitigate this problem, we propose to squeeze global spatial information into a channel descriptor. This is achieved by using global average pooling to generate channel-wise statistics."].
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention with a reasonable expectation of success to substitute the adjusting of intermediate layers based on a KNN pooling technique as taught by Liang with using a global pooling technique as taught by Hu in order to increase the accuracy of the convolutional neural network [Hu, page 7134, column 1].
Regarding claim 12, Liang as modified by Hu teaches claim 11. Liang further teaches wherein the which is a vehicle that moves in an at least partially automated manner, including: a passenger transport vehicle or a goods transport vehicle or a robot or a household robot or a cleaning robot or a floor-cleaning device or a street-cleaning device or a lawnmower robot or a drone [Liang ¶ 0077 "The vehicle 102 can be a ground-based vehicle (e.g., an automobile), an aircraft, a bike, a scooter and/or another type of vehicle or light electric vehicle."].
Regarding claim 13, all limitations have been examined with respect to the method in claim 1. The medium taught/disclosed in claim 13 can clearly perform the method of claim 1. Therefore, claim 13 is rejected under the same rationale.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JOSEPHINE RICH whose telephone number is (571)272-6384. The examiner can normally be reached M-F 8-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Scott Browne can be reached at (571) 270-0151. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/J.E.R./Examiner, Art Unit 3666
/SCOTT A BROWNE/Supervisory Patent Examiner, Art Unit 3666