DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Status
Claims 1-11 are pending for examination in the application filed 09/07/2023. Claims 1-9 are amended and claim 11 is new.
Priority
Acknowledgement is made of the present application as a national stage entry of PCT/JP2021/009583, international filing date: 03/10/2021.
Information Disclosure Statement
The information disclosure statements (IDS) submitted on 09/07/2023, 05/16/2024, 07/19/2024, and 11/01/2024 have been considered by the examiner.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 2 and claims depending therefrom are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention. Claim 2 recites the limitation "wherein the at least one processor is further configured to execute the instructions to determine a divided image region in the image based on the demarcation line, the divided image corresponding to a divided region each formed by a portion of a lane divided into a predetermined length on a real space, estimate a divided road parameter, based on the divided image region, and generate a road parameter, based on the divided road parameter and the position information of the vehicle at a time of capturing the image”. It is unclear what “each” is referring to since claim 2 only refers to one image, one divided image, and one divided image region. Please clarify. Furthermore, it is also unclear if in the limitation to “generate a road parameter” in claim 2 is the same “a road parameter” estimated in claim 1. Please clarify.
Claim 6 and claims depending therefrom are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention. Claim 6 recites the limitation “wherein the at least one processor is further configured to execute the instructions to acquire a first frame, a second frame being captured later than the first frame, and position information of the vehicle at a time of capturing each of the first and second frames, detect a demarcation line of a road for from each of the first and second frames, and in a case that the first frame has a shielded region in which a demarcation line is not detected, estimate a first road parameter of a region included in the first frame, based on the demarcation line of the second frame capturing the shielded region, and based on the position information of the vehicle at a time of capturing each of the first and second frames”. The claim language of “shielded region”, which the first frame and second frame both capture, is unclear, as the remainder of the claim language appears to indicate that the same demarcation line is shielded in the first frame and is not shielded in the second frame. Please clarify.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-2, and 9-10 are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Ferencz (US20200247431A1).
Regarding claim 1, Ferencz teaches a system ([0004] Embodiments consistent with the present disclosure provide systems and methods for autonomous vehicle navigation) comprising: at least one memory storing instructions (memory 140, 150); and at least one processor (applications processor 180, image processor 190) configured to execute the instructions to:
acquire an image indicating a surrounding of a vehicle, and position information of the vehicle ([0088] The image capture devices included on vehicle 200 as part of the image acquisition unit 120 may be positioned at any suitable location. In some embodiments, as shown in FIGS. 2A-2E and 3A-3C, image capture device 122 may be located in the vicinity of the rearview mirror. This position may provide a line of sight similar to that of the driver of vehicle 200. [0090] In addition to image capture devices, vehicle 200 may include various other components of system 100. For example, processing unit 110 may be included on vehicle 200 either integrated with or separate from an engine control unit (ECU) of the vehicle. Vehicle 200 may also be equipped with a position sensor 130, such as a GPS receiver and may also include a map database 160 and memory units 140 and 150);
detect a demarcation line of a road from the image ([0139] As described in connection with FIGS. 5A-5D below, monocular image analysis module 402 may include instructions for detecting a set of features within the set of images, such as lane markings, vehicles, pedestrians, road signs, highway exit ramps, traffic lights, hazardous objects, and any other feature associated with an environment of a vehicle);
estimate, based on the position information of the vehicle and the demarcation line in the image, a road parameter including at least one of a number of lanes included in a predetermined region on a real space, a length of a lane, a width of a lane, and a curve curvature ([0152] FIG. 5C is a flowchart showing an exemplary process 500C for detecting road marks and/or lane geometry information in a set of images, consistent with disclosed embodiments. Processing unit 110 may execute monocular image analysis module 402 to implement process 500C. At step 550, processing unit 110 may detect a set of objects by scanning one or more images. To detect segments of lane markings, lane geometry information, and other pertinent road marks, processing unit 110 may filter the set of objects to exclude those determined to be irrelevant (e.g., minor potholes, small rocks, etc.). At step 552, processing unit 110 may group together the segments detected in step 550 belonging to the same road mark or lane mark. Based on the grouping, processing unit 110 may develop a model to represent the detected segments, such as a mathematical model. [0263] The geometry of road segment 1200 may include lane structure and/or landmarks. The lane structure may include the total number of lanes of road segment 1200, the type of lanes (e.g., one-way lane, two-way lane, driving lane, passing lane, etc.), markings on lanes, width of lanes, etc. [0332] FIG. 26A is a flowchart showing an exemplary process 2600A for mapping a lane mark for use in autonomous vehicle navigation, consistent with disclosed embodiments. At step 2610, process 2600A may include receiving two or more location identifiers associated with a detected lane mark. For example, step 2610 may be performed by server 1230 or one or more processors associated with the server. The location identifiers may include locations in real-world coordinates of points associated with the detected lane mark, as described above with respect to FIG. 24E. In some embodiments, the location identifiers may also contain other data, such as additional information about the road segment or the lane mark. Additional data may also be received during step 2610, such as accelerometer data, speed data, landmarks data, road geometry or profile data, vehicle positioning data, ego motion data, or various other forms of data described above. The location identifiers may be generated by a vehicle, such as vehicles 1205, 1210, 1215, 1220, and 1225, based on images captured by the vehicle. For example, the identifiers may be determined based on acquisition, from a camera associated with a host vehicle, of at least one image representative of an environment of the host vehicle, analysis of the at least one image to detect the lane mark in the environment of the host vehicle, and analysis of the at least one image to determine a position of the detected lane mark relative to a location associated with the host vehicle);
and generate a simulated road, based on the road parameter ([0241] Data (e.g., reconstructed trajectories) collected by multiple vehicles in multiple drives along a road segment at different times may be used to construct the road model (e.g., including the target trajectories, etc.) included in sparse data map 800. Data collected by multiple vehicles in multiple drives along a road segment at different times may also be averaged to increase an accuracy of the model. In some embodiments, data regarding the road geometry and/or landmarks may be received from multiple vehicles that travel through the common road segment at different times. Such data received from different vehicles may be combined to generate the road model and/or to update the road model).
Regarding claim 2, Ferencz teaches the system of claim 1. Ferencz further teaches wherein the at least one processor is further configured to execute the instructions to determine a divided image region in the image based on the demarcation line, the divided image corresponding to a divided region each formed by a portion of a lane divided into a predetermined length on a real space ([0204] As shown in FIG. 9A, a lane 900 may be represented using polynomials (e.g., a first order, second order, third order, or any suitable order polynomials). For illustration, lane 900 is shown as a two-dimensional lane and the polynomials are shown as two-dimensional polynomials. As depicted in FIG. 9A, lane 900 includes a left side 910 and a right side 920. In some embodiments, more than one polynomial may be used to represent a location of each side of the road or lane boundary. For example, each of left side 910 and right side 920 may be represented by a plurality of polynomials of any suitable length. In some cases, the polynomials may have a length of about 100 m. [0205] In the example shown in FIG. 9A, left side 910 of lane 900 is represented by two groups of third order polynomials. The first group includes polynomial segments 911, 912, and 913. The second group includes polynomial segments 914, 915, and 916. The two groups, while substantially parallel to each other, follow the locations of their respective sides of the road. Polynomial segments 911, 912, 913, 914, 915, and 916 have a length of about 100 meters and overlap adjacent segments in the series by about 50 meters…Additionally, while FIG. 9A is shown as representing polynomials extending in 2D space (e.g., on the surface of the paper), it is to be understood that these polynomials may represent curves extending in three dimensions (e.g., including a height component) to represent elevation changes in a road segment in addition to X-Y curvature. In the example shown in FIG. 9A, right side 920 of lane 900 is further represented by a first group having polynomial segments 921, 922, and 923 and a second group having polynomial segments 924, 925, and 926),
estimate a divided road parameter, based on the divided image region, and generate a road parameter, based on the divided road parameter and the position information of the vehicle at a time of capturing the image ([0159] FIG. 5E is a flowchart showing an exemplary process 500E for causing one or more navigational responses in vehicle 200 based on a vehicle path, consistent with the disclosed embodiments. At step 570, processing unit 110 may construct an initial vehicle path associated with vehicle 200. The vehicle path may be represented using a set of points expressed in coordinates (x, z), and the distance d.sub.i between two points in the set of points may fall in the range of 1 to 5 meters. In one embodiment, processing unit 110 may construct the initial vehicle path using two polynomials, such as left and right road polynomials. Processing unit 110 may calculate the geometric midpoint between the two polynomials and offset each point included in the resultant vehicle path by a predetermined offset (e.g., a smart lane offset), if any (an offset of zero may correspond to travel in the middle of a lane). The offset may be in a direction perpendicular to a segment between any two points in the vehicle path. In another embodiment, processing unit 110 may use one polynomial and an estimated lane width to offset each point of the vehicle path by half the estimated lane width plus a predetermined offset (e.g., a smart lane offset)).
Regarding claim 9, Ferencz teaches a method ([0004] Embodiments consistent with the present disclosure provide systems and methods for autonomous vehicle navigation) comprising:
an acquisition step of acquiring an image indicating a surrounding of a vehicle, and position information of the vehicle ([0088] The image capture devices included on vehicle 200 as part of the image acquisition unit 120 may be positioned at any suitable location. In some embodiments, as shown in FIGS. 2A-2E and 3A-3C, image capture device 122 may be located in the vicinity of the rearview mirror. This position may provide a line of sight similar to that of the driver of vehicle 200. [0090] In addition to image capture devices, vehicle 200 may include various other components of system 100. For example, processing unit 110 may be included on vehicle 200 either integrated with or separate from an engine control unit (ECU) of the vehicle. Vehicle 200 may also be equipped with a position sensor 130, such as a GPS receiver and may also include a map database 160 and memory units 140 and 150);
a detection step of detecting a demarcation line of a road from the image ([0139] As described in connection with FIGS. 5A-5D below, monocular image analysis module 402 may include instructions for detecting a set of features within the set of images, such as lane markings, vehicles, pedestrians, road signs, highway exit ramps, traffic lights, hazardous objects, and any other feature associated with an environment of a vehicle);
an estimation step of estimating, based on the position information of the vehicle and the demarcation line in the image, a road parameter including at least one of a number of lanes included in a predetermined region on a real space, a length of a lane, a width of a lane, and a curve curvature ([0152] FIG. 5C is a flowchart showing an exemplary process 500C for detecting road marks and/or lane geometry information in a set of images, consistent with disclosed embodiments. Processing unit 110 may execute monocular image analysis module 402 to implement process 500C. At step 550, processing unit 110 may detect a set of objects by scanning one or more images. To detect segments of lane markings, lane geometry information, and other pertinent road marks, processing unit 110 may filter the set of objects to exclude those determined to be irrelevant (e.g., minor potholes, small rocks, etc.). At step 552, processing unit 110 may group together the segments detected in step 550 belonging to the same road mark or lane mark. Based on the grouping, processing unit 110 may develop a model to represent the detected segments, such as a mathematical model. [0263] The geometry of road segment 1200 may include lane structure and/or landmarks. The lane structure may include the total number of lanes of road segment 1200, the type of lanes (e.g., one-way lane, two-way lane, driving lane, passing lane, etc.), markings on lanes, width of lanes, etc. [0332] FIG. 26A is a flowchart showing an exemplary process 2600A for mapping a lane mark for use in autonomous vehicle navigation, consistent with disclosed embodiments. At step 2610, process 2600A may include receiving two or more location identifiers associated with a detected lane mark. For example, step 2610 may be performed by server 1230 or one or more processors associated with the server. The location identifiers may include locations in real-world coordinates of points associated with the detected lane mark, as described above with respect to FIG. 24E. In some embodiments, the location identifiers may also contain other data, such as additional information about the road segment or the lane mark. Additional data may also be received during step 2610, such as accelerometer data, speed data, landmarks data, road geometry or profile data, vehicle positioning data, ego motion data, or various other forms of data described above. The location identifiers may be generated by a vehicle, such as vehicles 1205, 1210, 1215, 1220, and 1225, based on images captured by the vehicle. For example, the identifiers may be determined based on acquisition, from a camera associated with a host vehicle, of at least one image representative of an environment of the host vehicle, analysis of the at least one image to detect the lane mark in the environment of the host vehicle, and analysis of the at least one image to determine a position of the detected lane mark relative to a location associated with the host vehicle);
and a generation step of generating a simulated road, based on the road parameter ([0241] Data (e.g., reconstructed trajectories) collected by multiple vehicles in multiple drives along a road segment at different times may be used to construct the road model (e.g., including the target trajectories, etc.) included in sparse data map 800. Data collected by multiple vehicles in multiple drives along a road segment at different times may also be averaged to increase an accuracy of the model. In some embodiments, data regarding the road geometry and/or landmarks may be received from multiple vehicles that travel through the common road segment at different times. Such data received from different vehicles may be combined to generate the road model and/or to update the road model).
Regarding claim 10, Ferencz teaches a non-transitory computer-readable medium storing a program for causing a computer to execute ([0009] Consistent with other disclosed embodiments, non-transitory computer-readable storage media may store program instructions, which are executed by at least one processing device and perform any of the methods described herein):
acquisition processing of acquiring an image indicating a surrounding of a vehicle, and position information of the vehicle ([0088] The image capture devices included on vehicle 200 as part of the image acquisition unit 120 may be positioned at any suitable location. In some embodiments, as shown in FIGS. 2A-2E and 3A-3C, image capture device 122 may be located in the vicinity of the rearview mirror. This position may provide a line of sight similar to that of the driver of vehicle 200. [0090] In addition to image capture devices, vehicle 200 may include various other components of system 100. For example, processing unit 110 may be included on vehicle 200 either integrated with or separate from an engine control unit (ECU) of the vehicle. Vehicle 200 may also be equipped with a position sensor 130, such as a GPS receiver and may also include a map database 160 and memory units 140 and 150);
detection processing of detecting a demarcation line of a road from the image ([0139] As described in connection with FIGS. 5A-5D below, monocular image analysis module 402 may include instructions for detecting a set of features within the set of images, such as lane markings, vehicles, pedestrians, road signs, highway exit ramps, traffic lights, hazardous objects, and any other feature associated with an environment of a vehicle);
estimation processing of estimating, based on the position information of the vehicle and the demarcation line in the image, a road parameter including at least one of a number of lanes included in a predetermined region on a real space, a length of a lane, a width of a lane, and a curve curvature ([0152] FIG. 5C is a flowchart showing an exemplary process 500C for detecting road marks and/or lane geometry information in a set of images, consistent with disclosed embodiments. Processing unit 110 may execute monocular image analysis module 402 to implement process 500C. At step 550, processing unit 110 may detect a set of objects by scanning one or more images. To detect segments of lane markings, lane geometry information, and other pertinent road marks, processing unit 110 may filter the set of objects to exclude those determined to be irrelevant (e.g., minor potholes, small rocks, etc.). At step 552, processing unit 110 may group together the segments detected in step 550 belonging to the same road mark or lane mark. Based on the grouping, processing unit 110 may develop a model to represent the detected segments, such as a mathematical model. [0263] The geometry of road segment 1200 may include lane structure and/or landmarks. The lane structure may include the total number of lanes of road segment 1200, the type of lanes (e.g., one-way lane, two-way lane, driving lane, passing lane, etc.), markings on lanes, width of lanes, etc. [0332] FIG. 26A is a flowchart showing an exemplary process 2600A for mapping a lane mark for use in autonomous vehicle navigation, consistent with disclosed embodiments. At step 2610, process 2600A may include receiving two or more location identifiers associated with a detected lane mark. For example, step 2610 may be performed by server 1230 or one or more processors associated with the server. The location identifiers may include locations in real-world coordinates of points associated with the detected lane mark, as described above with respect to FIG. 24E. In some embodiments, the location identifiers may also contain other data, such as additional information about the road segment or the lane mark. Additional data may also be received during step 2610, such as accelerometer data, speed data, landmarks data, road geometry or profile data, vehicle positioning data, ego motion data, or various other forms of data described above. The location identifiers may be generated by a vehicle, such as vehicles 1205, 1210, 1215, 1220, and 1225, based on images captured by the vehicle. For example, the identifiers may be determined based on acquisition, from a camera associated with a host vehicle, of at least one image representative of an environment of the host vehicle, analysis of the at least one image to detect the lane mark in the environment of the host vehicle, and analysis of the at least one image to determine a position of the detected lane mark relative to a location associated with the host vehicle);
and generation processing of generating a simulated road, based on the road parameter ([0241] Data (e.g., reconstructed trajectories) collected by multiple vehicles in multiple drives along a road segment at different times may be used to construct the road model (e.g., including the target trajectories, etc.) included in sparse data map 800. Data collected by multiple vehicles in multiple drives along a road segment at different times may also be averaged to increase an accuracy of the model. In some embodiments, data regarding the road geometry and/or landmarks may be received from multiple vehicles that travel through the common road segment at different times. Such data received from different vehicles may be combined to generate the road model and/or to update the road model).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim 3 is rejected under 35 U.S.C. 103 as being unpatentable over Ferencz in view of Lewin (US20230245414A1).
Regarding claim 3, Ferencz teaches the system of claim 2. Ferencz does not explicitly teach wherein the at least one processor is further configured to execute the instructions to determine the predetermined length, based on a speed of the vehicle.
Lewin, in the same field of endeavor of vehicle image analysis, teaches wherein the at least one processor is further configured to execute the instructions to determine the predetermined length, based on a speed of the vehicle ([0053] In particular, for this application a region of interest is typically defined to correspond to a surface over which a road wheel of the vehicle 100 is about to travel, namely a small area of the road surface, slightly wider than a tyre width of the respective wheel and a few meters ahead of the current position of the wheel. It is noted that this process is iterated for at least each of the two front road wheels of the vehicle (or the rear road wheels if the vehicle 100 is reversing), so that a respective region of interest is determined for each road wheel. [0054] The location, width and length of such a region of interest may depend on a current vehicle speed, driving direction and terrain type. Navigation system data, accelerometers and brake or accelerator pedal sensors may be used for predicting the vehicle's speed, direction and underlying road surface in the near future, which may also be used to redefine the region of interest).
Therefore, it would have been obvious to a person of ordinary skill in the art before the time of filing to modify the system of Ferencz with the teachings of Lewin to determine a the predetermined length based on the speed of the vehicle because "the control system may be configured to determine the region of interest to correspond to a location at least one hundred meters ahead of the vehicle with respect to a driving direction. This may be useful to detect vehicles ahead when the vehicle is travelling at high speed" [0022].
Claim 4 is rejected under 35 U.S.C. 103 as being unpatentable over Ferencz in view of Zhang (US20190156128A1).
Regarding claim 4, Ferencz teaches the system of claim 2. Ferencz does not explicitly teach wherein the image is an image acquired by superimposing a guide line on a captured image, the guide line having length and position known on a real space, the captured image being captured by a camera of the vehicle, and the at least one processor is further configured to execute the instructions to determine the divided image region in the image, based on the guide line in the image.
Zhang, in the same field of endeavor of vehicle image analysis, teaches wherein the image is an image acquired by superimposing a guide line on a captured image, the guide line having length and position known on a real space, the captured image being captured by a camera of the vehicle, and the at least one processor is further configured to execute the instructions to determine the divided image region in the image, based on the guide line in the image ([0051] While illustrated as an outline overlaid on the roadway image 130, the targeted region 139 may be a designation of pixels, or other data points when the roadway image 130 is another type of sensor data, as being included in the targeted region 139. The server 125 performs an image analysis on the image to determine when the targeted region 139 includes a pixel in common with one or more stripe-shaped objects. The server 125 may scan the image data or other sensor data in the targeted region 139 to determine when the targeted region 139 overlaps with a stripe-shaped object. [0059] The localization geometry generator 121 is further configured to perform an image analysis on the image to determine when the at least one target region includes a pixel in common with the one or more stripe-shaped object templates, which identifies the stripe-shaped object in the targeted region. The localization geometry generator 121 may trace an outline of the stripe-shaped object based on the pixels that are identified as matching a stripe-shaped object template. [0141] At act S203, the processor 200 accesses a three-dimensional map in response to the position of the map device for one or more stripe-shaped objects. The one or more stripe-shaped objects in the three-dimensional map is derived from at least one targeted region shaped to intersect the one or more stripe-shaped objects. [0118] FIG. 19 illustrates an aging procedure for the stripe-shaped objects. The vertical dotted lines divide the roadway in the chunks. The chunks are the predetermined length used to analyze the road in section. Example chunks may be 12 meters, 20 meters, or another length. The length may be measured in the length of the road or a length of a tile or grid that may not be parallel to the road).
Therefore, it would have been obvious to a person of ordinary skill in the art before the time of filing to modify the system of Ferencz with the teachings of Zhang to superimpose a guide line on a captured image and determine the divided image region based on the guide line because "computer resources are conserved because only the region of interest determined according to the relationship between the stripe-shaped object and the roadway is used in calculating the geometry and the region of interest is a small fraction of the image" [0041].
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over Ferencz in view of Nagata (US20230356653A1).
Regarding claim 5, Ferencz teaches the system of claim 2. Ferencz does not explicitly teach wherein the at least one processor is further configured to execute the instructions to determine the divided image region in the image, based on a size and a position of an image region of another vehicle included in the image.
Nagata, in the same field of endeavor of vehicle image analysis, teaches wherein the at least one processor is further configured to execute the instructions to determine the divided image region in the image, based on a size and a position of an image region of another vehicle included in the image ([0178] Specifically, the adjacent lane is determined from information regarding the travel lane output from the lane detection unit 32 or the like. Moreover, information regarding positions, sizes, and the like of other vehicles traveling on the adjacent lane is extracted from the output of the object detection unit 30. Then, position and size of the free space are calculated using the extracted information. Here, the free space is, for example, a space where no other vehicles exist. [0198] In addition, the space recognition unit 33 detects an adjacent lane 40b adjacent to the region of the travel lane 40a and calculates position and size of a free space 52 (a hatched region in the figure) on the basis of the position information and the like of the other vehicle 50 on the adjacent lane 40b).
Therefore, it would have been obvious to a person of ordinary skill in the art before the time of filing to modify the system of Ferencz with the teachings of Nagata to determine the divided image region based on the size and position of another vehicle because "using information regarding the position and size of the free space, whether there is a space where that vehicle 1 can perform a lane change (i.e., a movement-allowing space on the target lane that the vehicle 1 can enter) is determined" [0179].
Claims 6-8 are rejected under 35 U.S.C. 103 as being unpatentable over Ferencz in view of Kodaira (US20100121561A1).
Regarding claim 6, Ferencz teaches the system of claim 1. Ferencz further teaches wherein the at least one processor is further configured to execute the instructions to acquire a first frame, a second frame being captured later than the first frame, and position information of the vehicle at a time of capturing each of the first and second frames, detect a demarcation line of a road from each of the first and second frames ([0154] At step 556, processing unit 110 may perform multi-frame analysis by, for example, tracking the detected segments across consecutive image frames and accumulating frame-by-frame data associated with detected segments. As processing unit 110 performs multi-frame analysis, the set of measurements constructed at step 554 may become more reliable and associated with an increasingly higher confidence level. Thus, by performing steps 550, 552, 554, and 556, processing unit 110 may identify road marks appearing within the set of captured images and derive lane geometry information).
Ferencz does not explicitly teach in a case that the first frame has a shielded region in which a demarcation line is not detected, estimate a first road parameter of a region included in the first frame, based on the demarcation line of the second frame capturing the shielded region, and based on the position information of the vehicle at a time of capturing each of the first and second frames.
Kodaira, in the same field of endeavor of vehicle image analysis, teaches in a case that the first frame has a shielded region in which a demarcation line is not detected, estimate a first road parameter of a region included in the first frame, based on the demarcation line of the second frame capturing the shielded region, and based on the position information of the vehicle at a time of capturing each of the first and second frames ([0011] In the present invention, for example, when a road marking as a recognition target is hidden by another vehicle or a shadow of the vehicle, an image captured before the marking is hidden, among images stored in the past, is combined with the newly clipped image, and the mark is detected and recognized by using the composite image. [0070] The road surface marking recognizing unit 30 detects a road marking and recognizes the road marking by using the composite image 225. [0060] The image clip-out unit 13 changes the position of the image area which is to be clipped out from the image, according to a detected state or a recognized state of the road marking included in the composite image. [0111] For example, while the vehicle is traveling at a constant speed, a vehicle behind is also following at substantially the same speed, and therefore, between an image 223a portion and an image 223b portion having a high brightness value due to the influence of the headlight of the vehicle behind, there is little positional change, but a positional change between the arrow markings 222a, 222b portions is great).
Therefore, it would have been obvious to a person of ordinary skill in the art before the time of filing to modify the system of Ferencz with the teachings of Kodaira to estimate a first road parameter based on the demarcation line of the second frame capturing the shielded region "using the composite image which enables accurate recognition of the road marking such as a traffic directional marking, leading to improved accuracy in the recognition of a traveling direction of the vehicle" [0011].
Regarding claim 7, Ferencz and Kodaira teach the system of claim 6. Ferencz does not explicitly teach wherein the at least one processor is further configured to execute the instructions to estimate a first road parameter of a region included in the first frame, based on the position information of the vehicle at a time of capturing and the demarcation line of the first frame, estimate a second road parameter of the shielded region, based on the position information of the vehicle at a time of capturing and the demarcation line of the second frame, correct the first road parameter, based on the second road parameter, and generate a simulated road, based on the first road parameter after correction.
Kodaira, in the same field of endeavor of vehicle image analysis, teaches wherein the at least one processor is further configured to execute the instructions to estimate a first road parameter of a region included in the first frame, based on the position information of the vehicle at a time of capturing and the demarcation line of the first frame, estimate a second road parameter of the shielded region, based on the position information of the vehicle at a time of capturing and the demarcation line of the second frame, correct the first road parameter, based on the second road parameter, and generate a simulated road, based on the first road parameter after correction ([0111] For example, while the vehicle is traveling at a constant speed, a vehicle behind is also following at substantially the same speed, and therefore, between an image 223a portion and an image 223b portion having a high brightness value due to the influence of the headlight of the vehicle behind, there is little positional change, but a positional change between the arrow markings 222a, 222b portions is great. [0060] The image clip-out unit 13 changes the position of the image area which is to be clipped out from the image, according to a detected state or a recognized state of the road marking included in the composite image. [0195] This application example is an example to cope with a case where a road marking is hidden by a shadow on a road surface. [0197] The image 502 shown in FIG. 20 includes not only lane-line markings 521 and a road marking 522 but also a shadow 523 of a vehicle on an adjacent lane. [0198] The shadow 523 hides part of the road marking 522. [0199] When the road marking recognition processing is applied to the image including such a shadow 523, it is difficult to obtain the correct recognition result. [0200] FIG. 21 and FIG. 22 show images 502a, 502b resulting from the perspective transformation processing, as the pre-processing of the image clip-out unit 13, which is applied to successive scenery images. [0201] As in the first embodiment, since the images 502a, 502b are images from the camera 6 attached to the vehicle, the road markings in the image move from the lower side to the upper side. [0205] In this case, the image is generated by combining images in the clipped portions (clipped images), or several pieces of the whole images of respective screens downloaded from the camera 6 are held in the storage portion 7b and an image in good condition among them is used, which makes it possible to generate a composite image without the shadow, such as a road marking 532 in an image 503 shown in FIG. 23. [0207] Therefore, the image combining unit 22 selects, from the respective images, partial areas without the shadow and combines these partial areas).
Therefore, it would have been obvious to a person of ordinary skill in the art before the time of filing to modify the system of Ferencz with the teachings of Kodaira to correct the first road parameter based on the second road parameter and generate a simulated road "using the composite image which enables accurate recognition of the road marking such as a traffic directional marking, leading to improved accuracy in the recognition of a traveling direction of the vehicle" [0011].
Regarding claim 8, Ferencz and Kodaira teach the system of claim 6. Ferencz does not explicitly teach wherein the at least one processor is further configured to execute the instructions to generate, as the simulated road, an image acquired by superimposing a simulated lane generated based on the first road parameter on the first frame.
Kodaira, in the same field of endeavor of vehicle image analysis, teaches wherein the at least one processor is further configured to execute the instructions to generate, as the simulated road, an image acquired by superimposing a simulated lane generated based on the first road parameter on the first frame ([0197] The image 502 shown in FIG. 20 includes not only lane-line markings 521 and a road marking 522 but also a shadow 523 of a vehicle on an adjacent lane. [0198] The shadow 523 hides part of the road marking 522. [0199] When the road marking recognition processing is applied to the image including such a shadow 523, it is difficult to obtain the correct recognition result. [0200] FIG. 21 and FIG. 22 show images 502a, 502b resulting from the perspective transformation processing, as the pre-processing of the image clip-out unit 13, which is applied to successive scenery images. [0201] As in the first embodiment, since the images 502a, 502b are images from the camera 6 attached to the vehicle, the road markings in the image move from the lower side to the upper side. [0202] A road marking 522a included in the image 502a in FIG. 21 moves upward as shown by a road marking 522b in the image 502b in FIG. 22. [0203] In both of the images 502a, 502b in FIG. 21 and FIG. 22, the road markings 522a, 522b are partly hidden by shadows 523a, 523b of the adjacent vehicle. [0204] Therefore, if a clipped image 524a in FIG. 21 and a clipped image 524b in FIG. 22 are used as they are for image composition, the shadow portion is left, which makes it difficult to generate an image suitable for the recognition of the road marking. [0205] In this case, the image is generated by combining images in the clipped portions (clipped images), or several pieces of the whole images of respective screens downloaded from the camera 6 are held in the storage portion 7b and an image in good condition among them is used, which makes it possible to generate a composite image without the shadow, such as a road marking 532 in an image 503 shown in FIG. 23. [0206] That is, in the road marking 522a in the image 502a in FIG. 21, part of the shadow 523a overlaps with a stick portion of the arrow, but in the road marking 522b in the image 502b in FIG. 22, the shadow 523b does not overlap with the stick portion of the arrow. [0207] Therefore, the image combining unit 22 selects, from the respective images, partial areas without the shadow and combines these partial areas. [0070] The road surface marking recognizing unit 30 detects a road marking and recognizes the road marking by using the composite image 225).
Therefore, it would have been obvious to a person of ordinary skill in the art before the time of filing to modify the system of Ferencz with the teachings of Kodaira to superimpose a simulated lane on the first frame because it "makes it possible to generate a composite image without the shadow" [0205].
Claim 11 is rejected under 35 U.S.C. 103 as being unpatentable over Ferencz in view of Kodaira and Ogawa (US20230115290A1).
Regarding claim 11, Ferencz and Kodaira teach the system of claim 8. Ferencz does not explicitly teach wherein the at least one processor is further configured to execute the instructions to decide whether a dangerous event related to another vehicle exists, based on the simulated lane and an image region of the another vehicle included in the simulated road.
Ogawa, in the same field of endeavor of vehicle image analysis, teaches wherein the at least one processor is further configured to execute the instructions to decide whether a dangerous event related to another vehicle exists, based on the simulated lane and an image region of the another vehicle included in the simulated road ([0009] The edge server 126 reconstructs an actual situation on a road on a virtual space by integrating the sensor data, and creates and maintains the traffic situation bird's-eye view map 52. The edge server 126 transmits information for assisting driving or the like to each communication terminal based on the traffic situation bird's-eye view map 52 maintained in this manner. [0076] The edge server 128 further includes an information transmission unit 228 that collates the moving object information of the traffic situation bird's-eye view map 225 and the vehicle information 221 stored in the vehicle information storage unit 222, and performs a process of notifying a vehicle located within a predetermined range from an object of information for traffic assistance such as information on a moving object having an attribute considered to be dangerous, such as a child or a pedestrian walking while looking at a smartphone, an accident vehicle on a road, a failed vehicle, a falling object, or the like in the integrated moving object information, and a transmission processing unit 230 that transmits a signal for information notification by the information transmission unit 228 to a target vehicle).
Therefore, it would have been obvious to a person of ordinary skill in the art before the time of filing to modify the system of Ferencz with the teachings of Ogawa to decide whether a dangerous event related to another vehicle exists based on the simulated lane and an image region of the another vehicle to "transmit a signal for information notification by the information transmission unit 228 to a target vehicle" [0076].
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Mori (US20210279484A1) teaches a lane estimation system.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Jacqueline R Zak whose telephone number is (571)272-4077. The examiner can normally be reached M-F 9-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Emily Terrell can be reached at (571) 270-3717. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JACQUELINE R ZAK/Examiner, Art Unit 2666
/SJ Park/Primary Examiner, Art Unit 2675
Prosecution Insights