DETAILED ACTION
Notice of Pre-AIA or AIA Status
1. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
2. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-13, 15, and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Chen (US 10,060,751), and further in view of Kwant (US 2018/0165831).
Regarding claim 1, Chen discloses a method (method and system for map matching using machine learning; Chen at abstract), comprising:
Performing one or more operations of a first machine using one or more neural networks, wherein at least one parameter of the one or more neural networks is updated using training data that is generated (probe vehicle uses neural network, sensor data, and data from trained map matching platform for localization in autonomous vehicle travel; Chen at col 6 lines 18-38, col 9 lines 8-67), at least, by:
Assigning a label to an image captured using a camera of a second machine while the second machine is disposed in an environment at a pose (multiple probe vehicles in area provide image data at a detected pose with marked attribute data; Chen at col 6 lines 38-53, col 13 lines 44-60), the label corresponding to a feature of the environment that is depicted in the image (static or dynamic features in the image are marked, e.g. stop signs, road features, other vehicles, buildings, etc; Chen at col 6 lines 18-54), the assigning of the label being based at least on:
Information corresponding to the feature as obtained from map data corresponding to the environment (known features in the environment used to map match data; Chen at abstract).
The image being captured while the second machine is disposed at the location at a given pose (probe vehicle data including image and pose in same location used to make prediction; Chen at col 9 lines 23-67).
While Chen uses predictions in training and localization, it cannot be ascertained from the disclosure of Chen a determination, based at least on the map data and the pose, that the feature is observable at the pose. Chen is also silent as to an alignment of the image with a map corresponding to the map data.
Kwant, in a similar invention in the same field of endeavor, teaches assigning of a label based on information corresponding to the feature as obtained from map data corresponding to the environment (reference map data prior observed by vehicle including static positions of objects and first vehicle pose; Kwant at 0005, 0006), a determination, based on map data and pose, that the feature is observable at the pose (expected position and view of static features in environment given second vehicle pose; Kwant at 0031, 0034, 0077), the image being captured while the second machine is disposed at the pose (second vehicle records second image and pose data; Kwant at 0048, 0077), and alignment of the image with a map corresponding to the map data (reference image taken by second vehicle compared to image taken by first vehicle for error determination and updated labeling on static objects in the environment; Kwant at claim 0076, 0077, claim 7).
Regarding claim 2, the combination teaches wherein the pose of the second machine is determined based at least on a localization determined based at least on the map data and sensor data captured by the second machine (correction to pose of second vehicle determined and pose and localization adjusted; Kwant at 0061, 0062).
Regarding claim 3, Chen discloses wherein the information about the feature as included in the map data is based at least on a plurality of other images captured by a plurality of other
machines different from the second machine and the first machine (any multitude of probe vehicles used to collect image and pose data at a given location; Chen at col 6 liens lines 18-54, col 11 lines 48-67, col 13 lines 34-60).
Regarding claim 4, Chen discloses wherein the image is captured with respect to one or more different conditions than at least one other image of the plurality of other images (e.g. time; Chen at col 1 lines 15-20, col 6 lines 18-53).
Regarding claim 5, Chen discloses wherein the one or more different conditions include one or more of: a time of day, a weather condition, a time of year, or a lighting condition (time; Chen at col 1 lines 15-20, col 6 lines 18-53).
Regarding claim 6, Chen discloses the map data includes a point cloud including one or more points corresponding to the feature; and the information about the feature is associated with the one or more points in the map data (map data includes lidar data; Chen at col 8 lines 55-67, col 13 lines 44-60).
Regarding claim 7, Chen discloses wherein the information about the feature as included in the map data is based at least on one or more respective projections to the point cloud of one or more other images that individually depict the feature (predictions on location of objects based on map data including prior recorded lidar data and image data; Chen at col 13 lines 44-67, col 14 lines 1-13, 62-67, col 15 lines 1-14)
Regarding claim 8, Chen discloses wherein the one or more neural networks are trained at least by computing an estimated classification corresponding to the feature using at least one neural network of the one or more neural networks, comparing the estimated classification with the label as assigned to the image and corresponding to the feature; and updating one or more parameters of the at least one neural network based at least on the comparing (updating map data based on confidence score.
Regarding claim 9, Chen discloses wherein the one or more neural networks are included in a machine learning model (machine learning classifier; Chen at abstract).
Regarding claim 10, the combination teaches wherein the alignment of the image with the map is based at least on the pose as imposed with respect to the map (reference image and captured image and their respective poses used to correct alignment and pose error; Kwant at 0061, 0062).
Regarding claim 11, Chen discloses a system (map matching system; Chen at Fig. 1, abstract) comprising:
One or more processors (one or more processors; Chen at col 1 lines 61-67, col 2 lines 1-30) to cause the system to perform operations comprising:
Assigning a label corresponding to a map portion of a map to a feature depicted in an image portion of an image that is aligned with the map portion based at least on a pose of a machine (multiple probe vehicles in area provide image data at a detected pose with marked attribute data; Chen at col 6 lines 38-53, col 13 lines 44-60).
Computing, using a learning model, an estimated classification corresponding to feature as depicted in the image portion (using machine learning on trained data to mark static or dynamic features in the image captured, e.g. stop signs, road features, other vehicles, buildings, etc; Chen at col 6 lines 18-54).
Comparing the estimated classification with the label as projected to feature as depicted in the image portion (map matching the image data to prestored features from a geographical database; Chen at abstract).
Updating one or more parameters of a learning model based at least on the comparing (map matching training data used to update the machine learning classifier; Chen at abstract).
While the machine learning of Chen does resemble a deep learning model, Chen does not explicitly disclose said facet of the invention.
Kwant, in a similar invention in the same field of endeavor, teaches a trained deep neural network configured to match features in a captured image at a sensed pose with known ground truth features for map matching and localization (Kwant at 0028, 0033, 0066).
It would be obvious to one of ordinary skill in the art before the time of the claimed invention to augment the machine learning of Chen with the deep learning neural network of Kwant. Doing so would provide for improvement in recognition of features in an image.
Regarding claim 12, Chen discloses wherein the map includes map data that corresponds to first sensor data obtained using a plurality of machines different from the machine (any multitude of probe vehicles used to collect image and pose data at a given location; Chen at col 6 liens lines 18-54, col 11 lines 48-67, col 13 lines 34-60).
Regarding claim 13, the combination teaches wherein the pose is determined at least by comparing the map data with second sensor data obtained using one or more sensors corresponding to the machine (pose error correction in vehicle via in vehicle sensors and map matching; Kwant at 0008).
Regarding claim 15, the combination teaches wherein the feature is selected for labeling based at least on a determination that the feature is observable from the pose of the machine at a time of capture of the image (expected position and view of static features in environment given second vehicle pose; Kwant at 0031, 0034, 0077).
Regarding claim 16, Chen discloses wherein the feature includes one or more of:
an object; a landmark; a structure; a road marking (static or dynamic features in the image are marked, e.g. stop signs, road features, other vehicles, buildings, etc; Chen at col 6 lines 18-54).
Allowable Subject Matter
3. Claims 17-20 are allowed.
Claim Objections
4. Claim 14 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JONATHAN M DAGER whose telephone number is (571)270-1332. The examiner can normally be reached on M-F 0830-1730.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Angela Ortiz can be reached on 571-272-1206. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JONATHAN M DAGER/Primary Examiner, Art Unit 3663
Prosecution Insights