DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Claim 34 is objected to because of the following informalities: It recites: “The non-transitory computer-readable medium of claim 3,”. The examiner believes this was intended to be “…of claim 33,”. For examination purposes, that is how it will be interpreted. Appropriate correction is respectfully required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 8 and 9 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor, or for pre-AIA the applicant regards as the invention.
Claim 8 recites
The system of claim 1, wherein
the indicators are inputted into one or more trained neural networks, and the one or more trained neural networks are updated based on the aggregated indicators collected from a plurality of vehicles traversing a road segment.
Yet claim 1 already recited “a plurality of vehicles” and “a road segment”. For examination purposes, these phrase will be interpreted as if they recited “the” instead of “a”. Claim 9 is rejected due to its dependency.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-7, 10, 11, 14-18, 23-26, 28, 29, 32, 33, and 36 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Fridman (US2018/0025235).
Regarding claim 1, Fridman discloses:
A system for generating map information for use in navigating a host vehicle relative to a road segment, the system comprising (see paragraph 0011):
at least one processor comprising circuitry and a memory, wherein the memory includes instructions that when executed by the circuitry cause the at least one processor to (see paragraph 0092):
receive drive information from each of a plurality of vehicles that traversed a road segment [in the past], wherein the drive information includes indicators representative of road topography features associated with the road segment (see Fig. 18. Note this figure in Fridman is identical to the figure in the present disclosure. See the title of the disclosure of Fridman and paragraph 0207 for crowdsourcing drive information of a road segment. See Fig. 19 for receiving drive information from a plurality of vehicles along a road segment.);
aggregate the indicators representative of road topography features and generate an image representation of road topography of the road segment based on the aggregated indicators (see paragraph 0241. See paragraph 0244 for the target trajectory being generated based on “an average of first, second, and third trajectories” the target trajectory being “an aggregation…of two or more reconstructed trajectories.” See also paragraph 0240 for landmarks also being generated via an aggregation process, including rejecting landmarks when a ratio of images containing them is too low.);
provide the image representation of road topography of the road segment as input to at least one trained model configured to generate, in response to the provided input, an output including at least one target trajectory for the road segment (in the present filed specification, paragraph 0301 teaches that the “target trajectories” are trajectories that vehicles have driven at that lane segment before. According to paragraph 0300 these can be clusters of trajectories.
With that in mind, see Fridman, paragraph 00379 for teaching that “applications processor 180 and/or image processor 190 may execute the instructions stored in any of the modules 2502, 2504, 2506, and 2508. Although paragraph 0384 leaves module 2508 off of the list of the other three modules that may use “a trained system (such as a neural network…)” paragraph 0097 teaches that “processors 180 and/or image processor 190” together with the memory and instructions with which they run, including “various databases and image processing software, as well as a trained system, such as a neural network, or a deep neural network”. So according to paragraph 0097, items 180 and 190 are running a trained neural network, which paragraph 00379 says executes module 2508. See paragraph 0383 for the “navigational response module 2508” which “may store software executable by processing unit 110 to determine a desired navigational response based on data derived from execution of sparse map module 2502, image analysis module 2504 and/or road surface features module 2506.” See paragraph 0379 for “processing unit 110 may refer to applications processor 180 and image processor 190 individually or collectively.”);
store the at least one target trajectory in a map (see paragraph 0372. See also paragraph 0336 for a “target trajectory stored”. See paragraph 0102 for a “map database 160 that may store…target trajectories”); and
provide the map to at least one host vehicle navigation system for use in navigating the host vehicle relative to the at least one target trajectory of the road segment (see paragraph 0375 for “determining a target trajectory along common road segment”).
Regarding claim 2, Fridman teaches the system of claim 1.
Fridman further teaches:
The system of claim 1, wherein
the indicators representative of road topography features identify a feature type and a position associated with each of the road topography features (see paragraph 0148 for identifying lane marks, pedestrians, etc. See paragraph 0158 for also obtaining data of a “position” and state data of the “nearby vehicle, pedestrian, or road object, position information for vehicle 200 relative to lane markings of the road, and the like.” See Fig. 17 and paragraph 0294. See also paragraph 0307-0310 for locating landmarks and other objects in the local and world coordinate frames.).
Regarding claim 3, Fridman teaches the system of claim 2.
Fridman further teaches:
The system of claim 2, wherein
the feature type includes a lane marking (see paragraphs 0295 and 0302).
Regarding claim 4, Fridman teaches the system of claim 2.
Fridman further teaches:
The system of claim 2, wherein
the feature type includes a road edge (see paragraph 0302).
Regarding claim 5, Fridman teaches the system of claim 2.
Fridman further teaches:
The system of claim 2, wherein
the feature type includes at least one of a traffic sign, a traffic light, a lamp post, a building, a road barrier, or a speed bump (see paragraph 0298).
Regarding claim 6, Fridman teaches the system of claim 2.
Fridman further teaches:
The system of claim 2, wherein
the position is identified as a three-dimensional, real-world position (see paragraph 0307 and 0170. See paragraph 0173 for determining a “real-world position” of objects.).
Regarding claim 7, Fridman teaches the system of claim 2.
Fridman further teaches:
The system of claim 2, wherein
the position is identified as a two-dimensional position relative to an image frame (see paragraph 0316 for aligning coordinates to the camera and first map. See paragraph 0310 for using a “body reference frame”. See paragraph 0361).
Regarding claim 10, Fridman teaches the system of claim 1.
Fridman further teaches:
The system of claim 1, wherein
the image representation is generated from the output of one or more trained neural networks by inputting feature information of the aggregated indicators into the one or more trained neural networks (see Fridman, paragraph 00379 for teaching that “applications processor 180 and/or image processor 190 may execute the instructions stored in any of the modules 2502, 2504, 2506, and 2508. Although paragraph 0384 leaves module 2508 off of the list of the other three modules that may use “a trained system (such as a neural network…)” paragraph 0097 teaches that “processors 180 and/or image processor 190” together with the memory and instructions with which they run, including “various databases and image processing software, as well as a trained system, such as a neural network, or a deep neural network”. So according to paragraph 0097, items 180 and 190 are running a trained neural network, which paragraph 00379 says executes module 2508. See paragraph 0383 for the “navigational response module 2508” which “may store software executable by processing unit 110 to determine a desired navigational response based on data derived from execution of sparse map module 2502, image analysis module 2504 and/or road surface features module 2506.” See paragraph 0379 for “processing unit 110 may refer to applications processor 180 and image processor 190 individually or collectively.” See paragraph 0379 for item 2506 being a “road surface feature module”. The features are aggregated and then used by module 2506 as an input to the neural network.).
Regarding claim 11, Fridman teaches the system of claim 10.
Fridman further teaches:
The system of claim 10, wherein
the feature information includes one or more of type, size, shape, color, position in time, position in space, occlusions, relative position to other indicators, and neighborhood information (see paragraphs 0162 and 0166-0167).
Regarding claim 14, Fridman teaches the system of claim 1.
Fridman further teaches:
The system of claim 1, wherein
the at least one trained model includes one or more trained neural networks (see paragraph 0097 and 0379).
Regarding claim 15, Fridman teaches the system of claim 1.
Fridman further teaches:
The system of claim 1, wherein
the road segment is an arbitrary road segment through which the host vehicle has not driven (nothing in the disclosure of Fridman suggests that the host vehicle has to have driven through a road segment in the past for the other vehicles that form the crowdsource to collect information about that road segment for the next time the host vehicle traverses that road segment. The conspicuous absence of such a teaching clearly means that Fridman teaches this limitation. The examiner believes any other interpretation of Fridman is unreasonable making Fridman’s disclosure meet this limitation above a preponderance of the evidence. See also Fig. 24. First navigation information is crowdsourced, then it is distributed to vehicles traversing that road segment.).
Regarding claim 16, Fridman teaches the system of claim 1.
Fridman further teaches:
The system of claim 1, wherein
the road segment is an arbitrary road segment through which the host vehicle previously did not have the target trajectory generated by the at least one trained model (nothing in the disclosure of Fridman suggests that the host vehicle has to have driven through a road segment in the past for the other vehicles that form the crowdsource to collect information about that road segment for the next time the host vehicle traverses that road segment. The conspicuous absence of such a teaching clearly means that Fridman teaches this limitation. The examiner believes any other interpretation of Fridman is unreasonable making Fridman’s disclosure meet this limitation above a preponderance of the evidence. Since the host vehicle had not traversed the road segment before it would not have a target trajectory generated for it. See also Fig. 24. First navigation information is crowdsourced, then it is distributed to vehicles traversing that road segment.).
Regarding claim 17, Fridman teaches the system of claim 1.
Fridman further teaches:
The system of claim 1, wherein
the at least one target trajectory includes a plurality of target trajectories (see paragraph 0215, especially the first sentence),
wherein each of the plurality of target trajectories is associated with a different lane of travel of the road segment (see paragraph 0215, especially the middle of the paragraph).
Regarding claim 18, Fridman teaches the system of claim 17.
Fridman further teaches:
The system of claim 17, wherein
the plurality of target trajectories are representative of all lanes of travel associated with the road segment (see paragraph 0215, especially the middle of the paragraph).
Regarding claim 23, Fridman teaches the system of claim 1.
Fridman further teaches:
The system of claim 1, wherein
the aggregation of the indicators includes alignment of the received drive information received from the plurality of vehicles (see paragraph 0254).
Regarding claim 24, Fridman teaches the system of claim 1.
Fridman further teaches:
The system of claim 1, wherein
the aggregation of the indicators includes determining refined positions associated with each of the road topography features (see paragraph 0286, especially the last sentence.).
Regarding claim 25, Fridman teaches the system of claim 1.
Fridman further teaches:
The system of claim 1, wherein
the image representation of road topography of the road segment includes a two-dimensional top view of the road segment and the road topography of the road segment (see paragraphs 0047 and 0221-0222 and Fig. 9A).
Regarding claim 26, Fridman teaches the system of claim 1.
Fridman further teaches:
The system of claim 1, wherein
the at least one target trajectory is represented as a three- dimensional spline (see paragraph 0054 and Fig. 13).
Regarding claim 28, Fridman teaches the system of claim 1.
Fridman further teaches:
The system of claim 1, wherein
the at least one target trajectory is associated with at least one of a highway exit lane, a highway entrance lane, or a parking lot (see paragraph 0156-0157).
Regarding claim 29, Fridman discloses:
A method for generating map information for use in navigating a host vehicle relative to a road segment, the method comprising (see paragraph 0002):
receiving drive information from each of a plurality of vehicles that traversed a road segment, wherein the drive information includes indicators representative of road topography features associated with the road segment (for the remainder of the rejection, see the analogous bullets points in claim 1 which is substantially similar);
aggregating the indicators representative of road topography features and generate an image representation of road topography of the road segment based on the aggregated indicators;
providing the image representation of road topography of the road segment as input to at least one trained model configured to generate, in response to the provided input, an output including at least one target trajectory for the road segment;
storing the at least one target trajectory in a map; and
providing the map to at least one host vehicle navigation system for use in navigating the host vehicle relative to the at least one target trajectory of the road segment.
Regarding claim 32, the claim is substantially similar to claim 10. See the rejection for that claim.
The method of claim 29, wherein
the image representation is generated from the output of one or more trained neural networks by inputting feature information of the aggregated indicators into the one or more trained neural networks.
Regarding claim 33, Fridman discloses:
A non-transitory computer-readable medium storing instructions for generating map information for use in navigating a host vehicle relative to a road segment according to a method, the method comprising (see paragraph 0014):
receiving drive information from each of a plurality of vehicles that traversed a road segment, wherein the drive information includes indicators representative of road topography features associated with the road segment (for the remainder of the rejection, see the analogous bullets points in claim 1 which is substantially similar);
aggregating the indicators representative of road topography features and generate an image representation of road topography of the road segment based on the aggregated indicators;
providing the image representation of road topography of the road segment as input to at least one trained model configured to generate, in response to the provided input, an output including at least one target trajectory for the road segment;
storing the at least one target trajectory in a map; and
providing the map to at least one host vehicle navigation system for use in navigating the host vehicle relative to the at least one target trajectory of the road segment.
Regarding claim 36, the claim is substantially similar to claim 10. See the rejection for that claim.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 8, 30, and 34 are rejected under 35 U.S.C. 103 as being unpatentable over Fridman (US2018/0025235) in view of Shalev-Shwartz et al. (US2019/0333381), both disclosures from Mobileye, who is also the assignee of the present disclosure.
Regarding claim 8, Fridman teaches the system of claim 1.
Fridman further teaches:
The system of claim 1, wherein
the indicators are inputted into one or more trained neural networks (see paragraph 00379 for teaching that “applications processor 180 and/or image processor 190 may execute the instructions stored in any of the modules 2502, 2504, 2506, and 2508. Although paragraph 0384 leaves module 2508 off of the list of the other three modules that may use “a trained system (such as a neural network…)” paragraph 0097 teaches that “processors 180 and/or image processor 190” together with the memory and instructions with which they run, including “various databases and image processing software, as well as a trained system, such as a neural network, or a deep neural network”. So according to paragraph 0097, items 180 and 190 are running a trained neural network, which paragraph 00379 says executes module 2508. See paragraph 0383 for the “navigational response module 2508” which “may store software executable by processing unit 110 to determine a desired navigational response based on data derived from execution of sparse map module 2502, image analysis module 2504 and/or road surface features module 2506.” See paragraph 0379 for “processing unit 110 may refer to applications processor 180 and image processor 190 individually or collectively.”).
Yet Fridman does not further teach:
the one or more trained neural networks are updated based on the aggregated indicators collected from a plurality of vehicles traversing a road segment.
However, Shalev-Shwartz teaches:
the one or more trained neural networks are updated based on the aggregated indicators collected from a plurality of vehicles traversing a road segment (see paragraph 0376 for continuous training of a neural network using real world data”. Therefore the system generates “updates” to the “one or more neural networks”.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system, as taught by Fridman, to add the additional features as indicated as being taught by Shalev-Shwartz. The motivation for doing so would be to continuously improve the neural network, as recognized by Shalev-Shwartz (see paragraph 0376).
This conclusion of obviousness corresponds to KSR rationale “A”: it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined prior art elements according to known methods to yield predictable results. See MPEP § 2141, subsection III.
Regarding claim 30, the claim is substantially similar to claim 8. See the rejection for that claim.
Regarding claim 34, the claim is substantially similar to claim 8. See the rejection for that claim.
Claims 9, 31, and 35 are rejected under 35 U.S.C. 103 as being unpatentable over Fridman (US2018/0025235) in view of Shalev-Shwartz et al. (US2019/0333381), in further view of Casterton et al. (U.S. 12,434,725).
Regarding claim 9, Fridman and Shalev-Shwartz teaches the system of claim 8.
Yet Fridman and Shalev-Shwartz do not further teach:
The system of claim 8, wherein
the one or more trained neural networks output an updated image representation of road topography of the road segment based on the aggregated indicators.
However, Casterton teaches:
the one or more trained neural networks output an updated image representation of road topography of the road segment based on the aggregated indicators (in the present disclosure, present claim 1 recites that an input into the neural network is an image representation. The present claim states that an output of the neural network can be “an updated image representation”. What is an “image representation” according to the present disclosure? Present claim 25 recites that “the image representation of road topography of the road segment includes a two-dimensional top view of the road segment and the road topography of the road segment.” Based on this definition, the present claim is teaching that the trained neural network (which receives updated data, according to claim 8) is capable of outputting an updated 2D top-down view of the road segment including the road “topography” which means road features, such as signs, lane markings, etc.
With that in mind, see Casterton col. 6, lines 42-60).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system, as taught by Fridman and Shalev-Shwartz, to add the additional features as indicated as being taught by Casterton. The motivation for doing so would be to have an updated image representation of the roadway, as recognized by Casterton (see col. 6, lines 42-60).
This conclusion of obviousness corresponds to KSR rationale “A”: it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined prior art elements according to known methods to yield predictable results. See MPEP § 2141, subsection III.
Regarding claim 31, the claim is substantially similar to claim 9. See the rejection for that claim.
Regarding claim 35, the claim is substantially similar to claim 9. See the rejection for that claim.
Claims 12, 13, and 27 are rejected under 35 U.S.C. 103 as being unpatentable over Fridman (US2018/0025235) in view of Ferencz et al. (U.S. 12,330,639).
Regarding claim 12, Fridman teaches the system of claim 10.
Yet Friedman does not further teach:
The system of claim 10, wherein
the output of the one or more trained neural networks includes polygonal representations of the indicators.
However, Ferencz teaches:
the output of the one or more trained neural networks includes polygonal representations of the indicators (in the present specification, see paragraph 0387 for a polygon being a series of points in two or three dimensions arranged into a shape, such as a rectangular shape or a box.
With that in mind, see Ferencz col. 76, lines 11-33. See col. 80, lines 18-19 for the trained model 2720 being a neural network.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system, as taught by Fridman, to add the additional features as indicated as being taught by Ferencz. The motivation for doing so would be to help the vehicle make decisions about its navigation and identify roadside objects, as recognized by Ferencz (see col. 1, lines 22-48).
This conclusion of obviousness corresponds to KSR rationale “A”: it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined prior art elements according to known methods to yield predictable results. See MPEP § 2141, subsection III.
Regarding claim 13, Fridman teaches the system of claim 10.
Yet Friedman does not further teach:
The system of claim 10, wherein
the output of the one or more trained neural networks includes point information and metadata representing the indicators.
However, Ferencz teaches:
the output of the one or more trained neural networks includes point information and metadata representing the indicators (see Ferencz col. 21, lines 3-12).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system, as taught by Fridman, to add the additional features as indicated as being taught by Ferencz. The motivation for doing so would be to help the vehicle make decisions about its navigation and identify roadside objects, as recognized by Ferencz (see col. 1, lines 22-48).
This conclusion of obviousness corresponds to KSR rationale “A”: it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined prior art elements according to known methods to yield predictable results. See MPEP § 2141, subsection III.
Regarding claim 27, Fridman teaches the system of claim 1.
Yet Friedman does not further teach:
The system of claim 26, wherein
the map is stored as a plurality of tiles, each of which are edited and updated independently.
However, Ferencz teaches:
the map is stored as a plurality of tiles, each of which are edited and updated independently (see col. 35, lines 8-26).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system, as taught by Fridman, to add the additional features as indicated as being taught by Ferencz. The motivation for doing so would be to help the vehicle make decisions about its navigation and identify roadside objects, as recognized by Ferencz (see col. 1, lines 22-48).
This conclusion of obviousness corresponds to KSR rationale “A”: it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined prior art elements according to known methods to yield predictable results. See MPEP § 2141, subsection III.
Claims 19-22 are rejected under 35 U.S.C. 103 as being unpatentable over Fridman (US2018/0025235) in view of Jiang et al. (US2020/0125102).
Regarding claim 19, Fridman teaches the system of claim 1.
Yet Fridman does not further teach:
The system of claim 1, wherein
the road segment includes a junction and wherein the at least one target trajectory includes a plurality of target trajectories each representative of a different navigable path through the junction.
However, Jiang teaches:
the road segment includes a junction and wherein the at least one target trajectory includes a plurality of target trajectories each representative of a different navigable path through the junction (see paragraph 0059 for generating an average trajectory among all the trajectories that vehicles have driven on a “road segment”. This is crowdsourced as shown in Fig. 5 and performed using the system of Fig. 1 which includes the machine learning system 122. See Fig. 10, especially step 1005 for generating a path for an autonomous vehicle to “navigate through a road segment” based on the reference lines created from the crowdsourced data. See paragraph 0043 and 0044 for this applying to intersections and a “crossing route” and intersections where there can be a “left turn only lane”. See paragraph 0050 for searching the lane configuration database 313 to obtain one or more lane reference lines (aggregated past trajectories) associated with on “or more lanes within the road”.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system, as taught by Fridman, to add the additional features as indicated as being taught by Jiang. The motivation for doing so would be to allow a vehicle to travel with minimum human interaction, as recognized by Jiang (see paragraph 0002).
This conclusion of obviousness corresponds to KSR rationale “A”: it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined prior art elements according to known methods to yield predictable results. See MPEP § 2141, subsection III.
Regarding claim 20, Fridman and Jiang teach the system of claim 19.
Yet Fridman does not further teach:
The system of claim 19, wherein
the plurality of target trajectories are representative of all navigable paths through the junction.
However, Jiang teaches:
the plurality of target trajectories are representative of all navigable paths through the junction (see paragraph 0059 for generating an average trajectory among all the trajectories that vehicles have driven on a “road segment”. This is crowdsourced as shown in Fig. 5 and performed using the system of Fig. 1 which includes the machine learning system 122. See Fig. 10, especially step 1005 for generating a path for an autonomous vehicle to “navigate through a road segment” based on the reference lines created from the crowdsourced data. See paragraph 0043 and 0044 for this applying to intersections and a “crossing route” and intersections where there can be a “left turn only lane”. See paragraph 0050 for searching the lane configuration database 313 to obtain one or more lane reference lines (aggregated past trajectories) associated with on “or more lanes within the road”. After sufficient crowdsourcing it is essentially impossible that all navigable paths would not be covered and unreasonable to argue that the disclosure of Jiang does not teach that all paths are covered by the vehicles participating in the crowdsourcing.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system, as taught by Fridman and Jiang, to add the additional features as indicated as being taught by Jiang. The motivation for doing so would be to allow a vehicle to travel with minimum human interaction, as recognized by Jiang (see paragraph 0002).
This conclusion of obviousness corresponds to KSR rationale “A”: it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined prior art elements according to known methods to yield predictable results. See MPEP § 2141, subsection III.
Regarding claim 21, Fridman teaches the system of claim 1.
Yet Fridman does not further teach:
The system of claim 1, wherein
the road segment includes a roundabout and wherein the at least one target trajectory includes a plurality of target trajectories each representative of a different navigable path through the roundabout
However, Jiang teaches:
the road segment includes a roundabout and wherein the at least one target trajectory includes a plurality of target trajectories each representative of a different navigable path through the roundabout (see paragraph 0059 for generating an average trajectory among all the trajectories that vehicles have driven on a “road segment”. This is crowdsourced as shown in Fig. 5 and performed using the system of Fig. 1 which includes the machine learning system 122. See Fig. 10, especially step 1005 for generating a path for an autonomous vehicle to “navigate through a road segment” based on the reference lines created from the crowdsourced data. See paragraph 0043 and 0044 for this applying to intersections and a “crossing route” and intersections where there can be a “left turn only lane”. See paragraph 0050 for searching the lane configuration database 313 to obtain one or more lane reference lines (aggregated past trajectories) associated with on “or more lanes within the road”. After sufficient crowdsourcing it is essentially impossible that all navigable paths would not be covered and unreasonable to argue that the disclosure of Jiang does not teach that all paths are covered by the vehicles participating in the crowdsourcing. A roundabout is an intersection, which is covered by at least paragraphs 0043-0044.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system, as taught by Fridman and Jiang, to add the additional features as indicated as being taught by Jiang. The motivation for doing so would be to allow a vehicle to travel with minimum human interaction, as recognized by Jiang (see paragraph 0002).
This conclusion of obviousness corresponds to KSR rationale “A”: it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined prior art elements according to known methods to yield predictable results. See MPEP § 2141, subsection III.
Regarding claim 22, Fridman and Jiang teach the system of claim 21.
Yet Fridman does not further teach:
The system of claim 21, wherein
the plurality of target trajectories are representative of all navigable paths through the roundabout.
However, Jiang teaches:
the plurality of target trajectories are representative of all navigable paths through the roundabout (see paragraph 0059 for generating an average trajectory among all the trajectories that vehicles have driven on a “road segment”. This is crowdsourced as shown in Fig. 5 and performed using the system of Fig. 1 which includes the machine learning system 122. See Fig. 10, especially step 1005 for generating a path for an autonomous vehicle to “navigate through a road segment” based on the reference lines created from the crowdsourced data. See paragraph 0043 and 0044 for this applying to intersections and a “crossing route” and intersections where there can be a “left turn only lane”. See paragraph 0050 for searching the lane configuration database 313 to obtain one or more lane reference lines (aggregated past trajectories) associated with on “or more lanes within the road”. After sufficient crowdsourcing it is essentially impossible that all navigable paths would not be covered and unreasonable to argue that the disclosure of Jiang does not teach that all paths are covered by the vehicles participating in the crowdsourcing. A roundabout is an intersection, which is covered by at least paragraphs 0043-0044.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the system, as taught by Fridman and Jiang, to add the additional features as indicated as being taught by Jiang. The motivation for doing so would be to allow a vehicle to travel with minimum human interaction, as recognized by Jiang (see paragraph 0002).
This conclusion of obviousness corresponds to KSR rationale “A”: it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined prior art elements according to known methods to yield predictable results. See MPEP § 2141, subsection III.
Additional Art
The prior art made of record here, though not relied upon, is considered pertinent to the present disclosure.
Vaught et al. (U.S. 11,899,457) teaches a system that may “update its neural network”.
Zhou et al. (US2020/0348676). Zhou teaches a vehicle at a junction as shown in Fig. 4. Good on candidate trajectories. Not great on plurality of vehicles.
PNG
media_image1.png
509
552
media_image1.png
Greyscale
PNG
media_image2.png
682
562
media_image2.png
Greyscale
PNG
media_image3.png
778
564
media_image3.png
Greyscale
PNG
media_image4.png
810
572
media_image4.png
Greyscale
Funke (US 12,374,125), a Zoox disclosure. Funke teaches a neural network that generates 2D top-down images and predicts trajectories of actors. As seen in Figs. 1 and 10, the system ultimately controls the host vehicle.
PNG
media_image5.png
824
546
media_image5.png
Greyscale
Sun et al. (US 12,485,898). See the figures below.
PNG
media_image6.png
578
818
media_image6.png
Greyscale
PNG
media_image7.png
542
816
media_image7.png
Greyscale
PNG
media_image8.png
536
776
media_image8.png
Greyscale
See Sun col. 19, line 47 – col. 20 for:
The lane metric module 314 may receive lane estimation data (e.g., from lane estimation module 312) and determine a metric (e.g., a score) for one or more of the candidate driving paths. The metric may be associated with an evaluation of each of the candidate driving paths based on geometric properties of the respective path (e.g., favoring paths with smaller curvature) and relation to identified objects (e.g., favoring paths that do not intersect or maintain a predetermined minimum distance from cones, barriers, road signs, temporary lane markers, etc.). In some implementations, the lane metric module 314 includes a machine learning model that receives lane estimation data as input and generates one or more outputs indicating one or more metrics (e.g., scores) of the one or more candidate driving paths. The machine learning model may also generate one or more outputs that include a level of confidence associated with the one or more candidate driving paths. In some implementations, the machine learning model may act as a data filter. For example, N candidate driving paths that have N highest levels of confidence may be identified as adequately representing the one or more driving lanes of the driving environment. In another example, all candidate driving paths having a level of confidence above a particular (e.g., predefined) threshold may be identified as adequately representing driving lane data.
See col. 22, lines 13-57 for:
One type of machine learning model that may be used to perform some or all of the above tasks is an artificial neural network, such as a deep neural network or a graph neural network (GNN). Artificial neural networks generally include a feature representation component with a classifier or regression layers that map features to a desired output space. A convolutional neural network (CNN), for example, hosts multiple layers of convolutional filters. Pooling is performed, and non-linearities may be addressed, at lower layers, on top of which a multi-layer perceptron is commonly appended, mapping top layer features extracted by the convolutional layers to decisions (e.g. classification outputs). Deep learning is a class of machine learning algorithms that use a cascade of multiple layers of nonlinear processing units for feature extraction and transformation. Each successive layer uses the output from the previous layer as input. Deep neural networks may learn in a supervised (e.g., classification) and/or unsupervised (e.g., pattern analysis) manner. Deep neural networks include a hierarchy of layers, where the different layers learn different levels of representations that correspond to different levels of abstraction. In deep learning, each level learns to transform its input data into a slightly more abstract and composite representation. In a driving path selector, for example, the raw input (e.g., first set of layers) may be sensor data associated with a state of a driving environment; a second set of layers may compose encoded feature data associated with one or more mapped driving lane data or detected objects disposed within the driving environment (e.g., road primitive such as cones, signs, lane markers, etc.); a third set of layers may include identifying target candidate locations (e.g., target destinations 410 of FIG. 4) associated with the mapped driving lanes. Notably, a deep learning process can learn which features to optimally place in which level on its own. The “deep” in “deep learning” refers to the number of layers through which the data is transformed. More precisely, deep learning systems have a substantial credit assignment path (CAP) depth. The CAP is the chain of transformations from input to output. CAPs describe potentially causal connections between input and output. For a feedforward neural network, the depth of the CAPs may be that of the network and may be the number of hidden layers plus one. For recurrent neural networks, in which a signal may propagate through a layer more than once, the CAP depth is potentially unlimited.
Widjaja et al. (US2024/0127603). See paragraph 0091 for obtaining 2D and 3D lidar data. “the LiDAR data is processed to extract features in a bird's-eye view (BEV). In examples, a BEV is a top down view of the environment.”.
See paragraph 0095 for “In examples, feature maps determined from LiDAR data are used to augment the base map 602 with road geometry and road features. For example, as a vehicle navigates along a trajectory in a region corresponding to at least one base map, LiDAR scans of the environment are captured. In the example of FIG. 6, features are extracted from the overlapping LiDAR scans. The features are input to a trained neural network that outputs rich feature maps, augmented with polylines. The rich feature maps are aggregated to generate globally consistent polylines 610 creating road geometry instances of a geometric layer 604. Based on the generated globally consistent polylines 610, human annotators can draw bounding boxes to indicate insertion of globally consistent lane boundary annotations 620. As shown in FIG. 6, the semantic layer 606 includes semantic information such as globally consistent lane boundary annotations 620 that demarcate semantic features of the environment. In examples, lane boundary annotations are markings of a map that identify locations corresponding to boundaries associated with lanes of travel, such as lane boundaries that segment driveable areas into lanes, curbside boundaries, and other road geometry including connectivity properties, physical properties, and road features such as crosswalks, traffic signs or other travel signals of various types.
PNG
media_image9.png
754
546
media_image9.png
Greyscale
Marchetti-Bowick et al. (US2021/0004012), an UATC (Uber) disclosure. See Fig. 2A and 2B for possible trajectories. See Fig. 6 and paragraph 0084 for “lane segments”. See Fig. 9 for inputting a “BEV” (bird’s eye top down view) scene image into a neural network. See paragraphs 0030 and 0060. See paragraph 0043 for:
According to some aspects of the present disclosure, a topological representation of an environment can be utilized to capture information about the spatial relationships between lanes of a road network, thereby encoding semantic information about how actors may behave relative to the road network.
PNG
media_image10.png
374
542
media_image10.png
Greyscale
PNG
media_image11.png
778
540
media_image11.png
Greyscale
PNG
media_image12.png
550
822
media_image12.png
Greyscale
PNG
media_image13.png
524
842
media_image13.png
Greyscale
Urtasun et al. (US2021/0276587) path refinement models
0112] The one or more path refinement models 316 (e.g., one or more recurrent neural networks) can be configured and/or trained to receive output including information and/or data associated with the one or more path proposals generated by the one or more path proposal models 314. Further, the one or more path refinement models 316 can be configured to perform one or more operations including generating one or more confidence scores associated with the one or more path proposals.
Kroepfl et al. (US2021/0063200). The translation from Japanese in PE2E reads “cloud sourcing” as “crowdsourcing”.
[0006] Embodiments of the present disclosure relate to approaches for map creation and localization for autonomous driving applications. In particular, embodiments of the present disclosure include an end-to-end system for data generation, map creation using the generated data, and localization to the created map that can be used with universal, consumer-grade sensors in commercially available vehicles. For example, during the data generation process, data collection vehicles employing consumer quality sensors and/or consumer vehicles may be used to generate sensor data. The resulting data may correspond to mapstreams—that may include streams of sensor data, perception outputs from deep neural networks (DNNs), and/or relative trajectory (e.g., rotation and translation) data—corresponding to any number of drives by any number of vehicles. As such, in contrast to a systematic data collection effort of conventional systems, the current systems may crowdsource data generation using many vehicles and many drives. To reduce the bandwidth and memory requirements of the system, the data from the mapstreams may be minimized (e.g., by filtering out dynamic objects, executing LiDAR plane slicing or LiDAR point reduction, converting perception or camera based outputs to 3D location information, executing campaigns for particular data types only, etc.) and/or compressed (e.g., using delta compression techniques. As a result of the mapstream data being generated using consumer grade sensors, the sensor data—once converted into map form for localization—may be used directly for localization, rather than relying solely on GNSS data. Further, because the relative trajectory information corresponding to each drive is tracked, this information may be used to generate individual road segments (e.g., 25 meter, 50 meter, etc. sized road segments) that may be localized to, thereby allowing for localization accuracy within the centimeter range.
See paragraph 0049 for a “crowdsourced approach to mapstream generation” which helps avoid occasional “occlusions”
Hardy et al. (US2018/0203453).
PNG
media_image14.png
948
780
media_image14.png
Greyscale
PNG
media_image15.png
634
866
media_image15.png
Greyscale
Jiang et al. (US2020/0125102). Added here for additional notes. See paragraph 0035 for machine learning engine 122.
PNG
media_image16.png
644
902
media_image16.png
Greyscale
PNG
media_image17.png
640
914
media_image17.png
Greyscale
PNG
media_image18.png
876
554
media_image18.png
Greyscale
PNG
media_image19.png
186
386
media_image19.png
Greyscale
PNG
media_image20.png
598
568
media_image20.png
Greyscale
PNG
media_image21.png
692
1154
media_image21.png
Greyscale
Shashua et al. (US2017/0248960), a Mobileye disclosure. Published on August 31, 2017.
[0095]
In some embodiments, a system for navigating a vehicle on a road at least partially covered with snow may include at least one processor programmed to: receive from an image capture device, a plurality of images captured of an environment forward of the vehicle, including areas where snow covers a road on which the vehicle travels; analyze at least one of the plurality of images to identify a first free space boundary on a driver side of the vehicle and extending forward of the vehicle, a second free space boundary on a passenger side of the vehicle and extending forward of the vehicle, and a forward free space boundary forward of the vehicle and extending between the first free space boundary and the second free space boundary; wherein the first free space boundary, the second free space boundary, and the forward free space boundary define a free space region forward of the vehicle; determine a first proposed navigational path for the vehicle through the free space region; provide the at least one of the plurality of images to a neural network and receive from the neural network a second proposed navigational path for the vehicle based on analysis of the at least one of the plurality of images by the neural network; determine whether the first proposed navigational path agrees with the second proposed navigational path; and cause the vehicle to travel on at least a portion of the first proposed navigational path if the first proposed navigational path is determined to agree with the second proposed navigational path.
[0540]
FIG. 24 illustrates a method of determining a condensed signature representation of a landmark. The condensed signature representation (or condense signature, or signature) may be determined for a landmark that is not directly relevant to driving, such as a general sign. For example, condensed signature representation may be determined for a rectangular business sign (advertisement), such as sign or landmark 2205. The condensed signature, rather than an actual image of the general sign may be stored within the model or sparse map, which may be used for later comparison with a condensed signature derived by other vehicles. In the embodiment shown in FIG. 24, an image of the landmark 2205 may be mapped to a sequence of numbers of a predetermined data size, such as 32 bytes (or any other size, such as 16 bytes, 64 bytes, etc.). The mapping may be performed through a mapping function indicated by arrow 2405. Any suitable mapping function may be used. In some embodiments, a neural network may be used to learn the mapping function based on a plurality of training images. FIG. 24 shows an example array 2410 including 32 numbers within a range of −128 to 127. The array 2410 of numbers may be an example condensed signature representation or identifier of landmark 2205.
Teaches road edges. Etc.
Ozog et al. (US2020/0292322) See paragraph 0025 for “machine learning” that receives sensor data of objects in a driving environment and outputs identification information.
PNG
media_image22.png
994
610
media_image22.png
Greyscale
PNG
media_image23.png
964
638
media_image23.png
Greyscale
[0008]
In other embodiments, the disclosed systems and methods may construct a road model for autonomous vehicle navigation. For example, the disclosed systems and methods may use crowd sourced data for autonomous vehicle navigation including recommended trajectories. As other examples, the disclosed systems and methods may identify landmarks in an environment of a vehicle and refine landmark positions.
[0095] In some embodiments, a system for navigating a vehicle on a road at least partially covered with snow may include at least one processor programmed to: receive from an image capture device, a plurality of images captured of an environment forward of the vehicle, including areas where snow covers a road on which the vehicle travels; analyze at least one of the plurality of images to identify a first free space boundary on a driver side of the vehicle and extending forward of the vehicle, a second free space boundary on a passenger side of the vehicle and extending forward of the vehicle, and a forward free space boundary forward of the vehicle and extending between the first free space boundary and the second free space boundary; wherein the first free space boundary, the second free space boundary, and the forward free space boundary define a free space region forward of the vehicle; determine a first proposed navigational path for the vehicle through the free space region; provide the at least one of the plurality of images to a neural network and receive from the neural network a second proposed navigational path for the vehicle based on analysis of the at least one of the plurality of images by the neural network; determine whether the first proposed navigational path agrees with the second proposed navigational path; and cause the vehicle to travel on at least a portion of the first proposed navigational path if the first proposed navigational path is determined to agree with the second proposed navigational path.
[0413]
In some embodiments, the disclosed systems and methods may construct a road model for autonomous vehicle navigation. For example, the road model may include crowd sourced data. The disclosed systems and methods may refine the crowd sourced data based on observed local conditions. Further, the disclosed systems and methods may determine a refined trajectory for an autonomous vehicle based on sensor information. Still further, the disclosed systems and methods may identify landmarks for use in the road model, as well refine the positions of the landmarks in the road model. These systems and methods are disclosed in further detail in the following sections.
[0414]
Crowd Sourcing Data for Autonomous Vehicle Navigation
[0415]
In some embodiments, the disclosed systems and methods may construct a road model for autonomous vehicle navigation. For example, disclosed systems and methods may use crowd sourced data for generation of an autonomous vehicle road model that one or more autonomous vehicles may use to navigate along a system of roads. By crowd sourcing, it means that data are received from various vehicles (e.g., autonomous vehicles) travelling on a road segment at different times and such data are used to generate and/or update the road model. The model may, in turn, be transmitted to the vehicles or other vehicles later travelling along the road segment for assisting autonomous vehicle navigation. The road model may include a plurality of target trajectories representing preferred trajectories that autonomous vehicles should follow as they traverse a road segment. The target trajectories may be the same as a reconstructed actual trajectory collected from a vehicle traversing a road segment, which may be transmitted from the vehicle to a server. In some embodiments, the target trajectories may be different from actual trajectories that one or more vehicles previously took when traversing a road segment. The target trajectories may be generated based on actual trajectories (e.g., through averaging or any other suitable operation).
[0416]
The vehicle trajectory data that a vehicle may upload to a server may correspond with the actual reconstructed trajectory for the vehicle, or it may correspond to a recommended trajectory, which may be based on or related to the actual reconstructed trajectory of the vehicle, but may differ from the actual reconstructed trajectory. For example, vehicles may modify their actual, reconstructed trajectories and submit (e.g., recommend) to the server the modified actual trajectories. The road model may use the recommended, modified trajectories as target trajectories for autonomous navigation of other vehicles.
PNG
media_image24.png
202
348
media_image24.png
Greyscale
PNG
media_image25.png
328
358
media_image25.png
Greyscale
Walls et al. (US20200257901). See paragraph 0024 for “machine learning” that receives sensor data of objects in a driving environment and outputs identification information. See paragraph 0025 for labeling the objects, including the object class.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DANIEL M. ROBERT whose telephone number is (571)270-5841. The examiner can normally be reached M-F 7:30-4:30 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Hunter Lonsberry can be reached at 571-272-7298. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DANIEL M. ROBERT/Primary Examiner, Art Unit 3665
Prosecution Insights