DETAILED ACTION
Claims 1-10 are pending in this application and have been examined with the priority date of 06/27/2022 in accordance with applicant’s claim to foreign priority.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 11/07/2024 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitations are:
first landmark detection unit in claims 1 and 10.
imaging unit in claims 1 and 10.
imaging plane estimation unit in claims 1, 2, 3, 4, 7, and 10.
image transform estimation unit in claims 1, 4, 5, 7, 8 and 10.
feature collation unit in claims 1, 4, 5, and 9.
position estimation unit in claims 1, 8, and 9.
transform selection unit in claims 5 and 6.
error estimation unit in claims 7 and 8.
parallax calculation unit in claim 9.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-10 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea or mental process without significantly more.
Regarding claims 1 and 10, the claims recite; “An external world recognition device comprising:
a first landmark detection unit that detects a landmark on a basis of a first image acquired from at least a first imaging unit among a plurality of imaging units in which at least a part of an imaging visual field with respect to an external world overlaps; (Mental process in which a person could look at an image with multiple landmarks and determine that a landmark is in the field of view)
a second landmark detection unit that detects a landmark on a basis of a second image acquired from a second imaging unit among the plurality of imaging units; (Mental process in which a person could look at an image with multiple landmarks and determine that a landmark is in the field of view)
an imaging plane estimation unit that estimates an imaging plane in the first image on a basis of a detection result of the landmark by the first landmark detection unit; (Mathematical process where a human could manually generate a bounding box or other plane around a landmark on an image)
an image transform estimation unit that estimates an image transform parameter matching an imaging plane in the second image on a basis of information on the imaging plane estimated by the imaging plane estimation unit and imaging parameters of the plurality of imaging units; (Mathematical process where a human could calculate or obtain a parameter such as size, angle, rotation, or the like which would align two images of the same object)
a feature collation unit that collates a feature of the landmark detected from the first image subjected to image transform using an image transform parameter by the image transform estimation unit with a feature of the landmark detected from the second image; (Mental process of comparing features between images)
and a position estimation unit that estimates a three dimensional position of the landmark on a basis of a collation result of the feature collation unit. (Mathematical process in which a human could estimate a position of an object based on comparing two images of it)”
Under step 2A, prong 1 (see MPEP 2106), this judicial exception is not integrated into a practical application because the steps above could be practically performed as mental processes or mathematical analysis performed by a human. Under step 2A, prong 2, the claim recites the additional elements of a first and second landmark detection unit, imaging units, image transform estimation unit, image plane estimation unit and a feature collation unit. Under step 2B, the claims additional elements are not sufficient to amount to significantly more than the judicial exception because they are recited with a high level of generality and do not meaningfully translate the claim into practical application.
Dependent claims 2-9 follow the same logic and do not add limitations that translate the claims into practical application or amount to significantly more.
Regarding claim 2, claim 2 recites;
“wherein the imaging plane estimation unit estimates the imaging plane on a basis of a size of the landmark and a distance to the landmark. (Mathematical process of estimating a size of a landmark which could be performed manually)”
The above limitations are drawn to an abstract idea, mathematical process or mental process without significantly more. The claim recites the additional element of a plane estimation unit, which is not sufficient to amount to significantly more than the judicial exception because it is recited with a high level of generality and does not meaningfully translate the claim into practical application.
Regarding claim 3, claim 3 recites;
“wherein the imaging plane estimation unit estimates the imaging plane in an image region in which the landmark is imaged. (Step of mere data gathering where a human could put a bounding box around an object in an image manually)”
The above limitations are drawn to an abstract idea, mathematical process or mental process without significantly more. The claim recites the additional element of a plane estimation unit, which is not sufficient to amount to significantly more than the judicial exception because it is recited with a high level of generality and does not meaningfully translate the claim into practical application.
Regarding claim 4, claim 4 recites;
“wherein a plurality of imaging plane estimation units and a plurality of image transform estimation units are provided for the landmark detected from the first image and the landmark detected from the second image, respectively, (Step of mere data gathering)
and the feature collation unit acquires a collation result between a result of image transform of the first image by using the image transform parameter estimated by one of the image transform estimation units and the second image, (Mathematical estimation of a parameter which could be assessed by a human manually)
acquires a collation result between a result of image transform of the second image by using the image transform parameter estimated by other one of the image transform estimation units and the first image, (Mental process of comparison)
compares the respective collation results, (Mental process of comparison)
and in a case where the landmark detected from the first image and the landmark detected from the second image are collated at a same three-dimensional position, gives high reliability to the collation result. (Mental process of determination of a landmark position and reliability”
The above limitations are drawn to an abstract idea, mathematical process or mental process without significantly more. The claim recites the additional elements of plane estimation units, image transform estimation units, and collation unit which are not sufficient to amount to significantly more than the judicial exception because they are recited with a high level of generality and do not meaningfully translate the claim into practical application.
Regarding claim 5, claim 5 recites;
“comprising a transform selection unit that selects, from among a plurality of landmarks detected from the first image and the second image, the landmark at a short distance from an own vehicle, (mental process of selecting a closer object in an image)
and selects the image transform parameter estimated by the image transform estimation unit for the selected landmark, wherein the feature collation unit performs image transform of the landmark detected from the first image by using the image transform parameter selected by the transform selection unit. (Mental process where a person could calculate and discern a rotation or translation to adjust a landmark)”
The above limitations are drawn to an abstract idea, mathematical process or mental process without significantly more. The claim recites the additional elements of a transform selection unit and collation unit which are not sufficient to amount to significantly more than the judicial exception because they are recited with a high level of generality and do not meaningfully translate the claim into practical application.
Regarding claim 6, claim 6 recites;
“wherein the transform selection unit preferentially selects the landmark having an attribute that greatly affects traveling of the own vehicle. (Mental process of selecting a landmark in an image)”
The above limitations are drawn to an abstract idea, mathematical process or mental process without significantly more. The claim recites the additional element of a transform selection unit, which is not sufficient to amount to significantly more than the judicial exception because it is recited with a high level of generality and does not meaningfully translate the claim into practical application.
Regarding claim 7, claim 7 recites; “comprising an error estimation unit that estimates an error of the imaging plane estimated by the imaging plane estimation unit, wherein the image transform estimation unit determines a range of error of the imaging plane to be subjected to image transform on a basis of an error of the imaging plane. (Mathematical process where a human could estimate the error manually)”
The above limitations are drawn to an abstract idea, mathematical process or mental process without significantly more. The claim recites the additional elements of a transform selection unit and an error estimation unit which are not sufficient to amount to significantly more than the judicial exception because they are recited with a high level of generality and do not meaningfully translate the claim into practical application.
Regarding claim 8, claim 8 recites;
“wherein the image transform estimation unit generates a plurality of imaging planes on a basis of an error of the imaging plane estimated by the error estimation unit and estimates a plurality of image transform parameters for each of the plurality of imaging planes, (Math and mere data gathering/generation where a person could compute the error and then adjust/transform bounding boxes on an image)
and the feature collation unit performs collation processing between a feature detected from the first image subjected to image transform using the plurality of image transform parameters and a feature detected from the second image a plurality of times, (Mental process of comparing two images)
and outputs a result of the collation processing with high evaluation to the position estimation unit. (Mere data gathering/transmission)”
The above limitations are drawn to an abstract idea, mathematical process or mental process without significantly more. The claim recites the additional elements of a transform estimation unit, a feature collation unit, a position estimation unit, and an error estimation unit which are not sufficient to amount to significantly more than the judicial exception because they are recited with a high level of generality and do not meaningfully translate the claim into practical application.
Regarding claim 9, claim 9 recites;
“comprising a parallax calculation unit that calculates a parallax from the first image and the second image and collates a position of a landmark appearing in the first image and the second image, (Mathematical process where a human could compute the parallax information from two images of an object and determine its position manually)
wherein the feature collation unit compares a first collation result obtained by collation between a feature of the landmark detected from the first image subjected to image transform using the image transform parameter and a feature of the landmark detected from the second image with a second collation result obtained by collation on a basis of the parallax (Mental process where a person could calculate and discern a rotation or translation to adjust a landmark in two images),
and outputs the first collation result or the second collation result to the position estimation unit on a basis of validity of a comparison result. (Mere data gathering/transmission)”
The above limitations are drawn to an abstract idea, mathematical process or mental process without significantly more. The claim recites the additional elements of a transform estimation unit, a feature collation unit, a position estimation unit, and a parallax calculation unit which are not sufficient to amount to significantly more than the judicial exception because they are recited with a high level of generality and do not meaningfully translate the claim into practical application.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1-10 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Hayat (US 20210101616 A1).
Regarding claim 1 Hayat discloses; An external world recognition device comprising (Hayat, [0005] navigation system (external world recognition device) includes a processor, cameras and other devices to detect the environment of the vehicle and navigate accordingly):
a first landmark detection unit (Hayat, [0125] multiple processors are used to execute the image processing method disclosed, as per [0049] of applicant’s specification, a CPU or other hardware acts as the units disclosed, therefore the multiple processing units/processors of Hayat are analogous the claimed units) that detects a landmark on a basis of a first image acquired from at least a first imaging unit among a plurality of imaging units in which at least a part of an imaging visual field with respect to an external world overlaps (Hayat, [0176] the system may have multiple image capturing devices (imaging units) to generate a sparse roadmap, which includes multiple identified landmarks, [0170] processing units (landmark detection unit) may detect from a plurality of images (at least 2 images) landmarks/road markings/signs that are common to the images);
PNG
media_image1.png
338
368
media_image1.png
Greyscale
(Hayat, [0170])
PNG
media_image2.png
404
374
media_image2.png
Greyscale
(Hayat, [0176])
a second landmark detection unit (Hayat, [0125] multiple processors are used to execute the image processing method disclosed, as per [0049] of applicant’s specification, a CPU or other hardware acts as the units disclosed, therefore the multiple processing units/processors of Hayat are analogous the claimed units) that detects a landmark on a basis of a second image acquired from a second imaging unit among the plurality of imaging units (Hayat, [0170] a processing unit (landmark detection unit) may detect a landmark/lane markings/signs that are common between two or more images captured from each of a plurality of image capture devices (at least a first and a second capture device));
an imaging plane estimation unit (Hayat, [0125] multiple processors are used to execute the image processing method disclosed, as per [0049] of applicant’s specification, a CPU or other hardware acts as the units disclosed, therefore the multiple processing units/processors of Hayat are analogous the claimed units) that estimates an imaging plane in the first image on a basis of a detection result of the landmark by the first landmark detection unit (Hayat, [0176] the system may have multiple image capturing devices (imaging units) to generate a sparse roadmap, which includes multiple identified landmarks, [0170] processing units (landmark detection unit) may detect from a plurality of images (at least 2 images) landmarks/road markings/signs that are common to the images, [0201] polynomials for the landmarks in the sparse map (based on the images) are determined in 2D space (image plane estimation) paragraph [0045] of the applicant’s specification defines the imaging plane as the plane in the image where the landmark is detected based on the size of the landmark);
PNG
media_image3.png
404
370
media_image3.png
Greyscale
(Hayat, [0201])
an image transform estimation unit (Hayat, [0125] multiple processors are used to execute the image processing method disclosed, as per [0049] of applicant’s specification, a CPU or other hardware acts as the units disclosed, therefore the multiple processing units/processors of Hayat are analogous the claimed units)that estimates an image transform parameter matching an imaging plane in the second image on a basis of information on the imaging plane estimated by the imaging plane estimation unit and imaging parameters of the plurality of imaging units (Hayat, [0123] stereo analysis is performed between images based on determined variations of the image capture parameter, [0128] multiple processing devices may be used to determine parameters in the images, where a processing device may be used to detect landmarks such as road objects and signs and determined a distance or disparity between pixels for each image, [0270] a matching algorithm (image transform estimation unit) may be used to align landmarks appearing in the multiple maps or images, [0271] figure 16 shows the rectangular plane associated with a sign (landmark) being aligned/matched from multiple sparse map image inputs using the matching algorithm previously described in [0270],);
PNG
media_image4.png
108
294
media_image4.png
Greyscale
(Hayat, [0123])
PNG
media_image5.png
150
296
media_image5.png
Greyscale
(Hayat, [0128], emphasis added)
PNG
media_image6.png
584
674
media_image6.png
Greyscale
(Hayat, figure 16)
a feature collation unit (Hayat, [0125] multiple processors are used to execute the image processing method disclosed, as per [0049] of applicant’s specification, a CPU or other hardware acts as the units disclosed, therefore the multiple processing units/processors of Hayat are analogous the claimed units) that collates a feature of the landmark detected from the first image subjected to image transform using an image transform parameter by the image transform estimation unit with a feature of the landmark detected from the second image (Hayat, [0136] a processing unit (feature collation unit) may take the first and second images, and their parameters to detect a set of features common to both, [0145] the processing unit may compare the object/landmark’s features/parameters across multiple frames);
PNG
media_image7.png
446
302
media_image7.png
Greyscale
(Hayat, [0136])
PNG
media_image8.png
138
294
media_image8.png
Greyscale
(Hayat, [0145])
and a position estimation unit (Hayat, [0125] multiple processors are used to execute the image processing method disclosed, as per [0049] of applicant’s specification, a CPU or other hardware acts as the units disclosed, therefore the multiple processing units/processors of Hayat are analogous the claimed units) that estimates a three-dimensional position of the landmark on a basis of a collation result of the feature collation unit (Hayat, [0145] a processing unit (position estimation unit) compares the objects positions across frames and estimates and predicts a position of the object, [0141] the frames and comparisons are used to create a 3D mapping of the road surface/landmarks, therefore the positions are in 3D).
Regarding claim 2 Hayat discloses; The external world recognition device according to claim 1, wherein the imaging plane estimation unit estimates the imaging plane on a basis of a size of the landmark and a distance to the landmark (Hayat, [0265] landmark identification includes a measure of size of the landmarks as well as a distance to the landmark,[0266] bounding boxes/pixel groups are generated using this data, where per [0045] of applicant’s specification, the imaging plane is simply a plane containing the landmark, therefore a bounding box in the image containing the landmark is analogous to the plane).
PNG
media_image9.png
390
300
media_image9.png
Greyscale
(Hayat, [0265])
PNG
media_image10.png
242
298
media_image10.png
Greyscale
(Hayat, [0266])
Regarding claim 3 Hayat discloses; The external world recognition device according to claim 2, wherein the imaging plane estimation unit estimates the imaging plane in an image region in which the landmark is imaged (Hayat, [0263] the landmarks may be visible within the field of view for the cameras and have images captured of it to extract information about the landmark in the photo, therefore the imaging plane would be estimated in an image regions where the landmark is imaged).
PNG
media_image11.png
404
300
media_image11.png
Greyscale
(Hayat, [0263])
Regarding claim 4 Hayat discloses; The external world recognition device according to claim 3, wherein a plurality of imaging plane estimation units and a plurality of image transform estimation units are provided for the landmark detected from the first image and the landmark detected from the second image, respectively (Hayat, [0134] each processing unit comprises multiple processing devices, further each step may be performed by multiple processing devices, [0136] multiple imaging units provide multiple images for each landmark, where the landmark is analyzed by the processing units across frames from different fields of view),
PNG
media_image12.png
150
296
media_image12.png
Greyscale
PNG
media_image13.png
40
294
media_image13.png
Greyscale
(Hayat, [0134])
and the feature collation unit (Hayat, [0125] multiple processors are used to execute the image processing method disclosed, as per [0049] of applicant’s specification, a CPU or other hardware acts as the units disclosed, therefore the multiple processing units/processors of Hayat are analogous the claimed units) acquires a collation result between a result of image transform of the first image by using the image transform parameter estimated by one of the image transform estimation units and the second image (Hayat, [0136] a processing unit (feature collation unit) may take the first and second images, and there parameters to detect a set of features common to both, [0145] the processing unit may compare the object/landmark’s features/parameters across multiple frames),
acquires a collation result between a result of image transform of the second image by using the image transform parameter estimated by other one of the image transform estimation units and the first image (Hayat, [0238]-[0239] a geometric reconstruction may be performed on each image of a landmark frame by frame based on the camera’s angles/rotations/translation to generate a stitched reconstruction of the road surface, where each image may be collected from a different camera and differing field of view and then transformed and compared),
PNG
media_image14.png
488
298
media_image14.png
Greyscale
(Hayat, [0238]-[0239])
compares the respective collation results (Hayat, [0251] the system may compare landmarks and their trajectories collected from multiple clients and construct a model of the road based on this),
and in a case where the landmark detected from the first image and the landmark detected from the second image are collated at a same three-dimensional position, gives high reliability to the collation result (Hayat, [0252] the system may average the coordinates and properties (collation results) of the a landmark in multiple images taken by multiple cameras, and determine that the landmark is the same and is at the same position to improve the road reconstruction’s results).
PNG
media_image15.png
138
292
media_image15.png
Greyscale
(Hayat, [0251])
PNG
media_image16.png
308
296
media_image16.png
Greyscale
(Hayat, [0252])
Regarding claim 5 Hayat discloses; The external world recognition device according to claim 3, comprising a transform selection unit (Hayat, [0125] multiple processors are used to execute the image processing method disclosed, as per [0049] of applicant’s specification, a CPU or other hardware acts as the units disclosed, therefore the multiple processing units/processors of Hayat are analogous the claimed units) that selects, from among a plurality of landmarks detected from the first image and the second image, the landmark at a short distance from an own vehicle (Hayat, [0112] images of objects are acquired at desired ranges relative the vehicle, where images be captured of objects closer to or further from the vehicle, and this focal length may be selectable such that this distance is selectable, [0153]-[0154] as the vehicle approaches a junction (landmark) the processor may adjust the and update the confidence in the lane markings/landmark and adjust the geometry of the detected landmark based on it being closer to the vehicle, the updating and modifying this landmark information would be analogous to selecting a closer appearing or approaching landmark for analysis),
PNG
media_image17.png
46
298
media_image17.png
Greyscale
PNG
media_image18.png
206
300
media_image18.png
Greyscale
(Hayat, [0112])
PNG
media_image19.png
352
294
media_image19.png
Greyscale
(Hayat, [0153]-[0154]
and selects the image transform parameter estimated by the image transform estimation unit for the selected landmark (Hayat, [0153]-[0154] as the vehicle approaches a junction (landmark) the processor may adjust the and update the confidence in the lane markings/landmark and adjust the geometry of the detected landmark based on it being closer to the vehicle, the updating and modifying this landmark information would be analogous to selecting a closer appearing or approaching landmark for analysis, further, the geometry of the landmark/junction is updated for the selected closer landmark, [0155] offsets (transform parameters) may be computed for landmarks selected, such as lane lines making up a traffic junction),
wherein the feature collation unit performs image transform of the landmark detected from the first image by using the image transform parameter selected by the transform selection unit (Hayat, [0155] the polynomials used to model the vehicle path have offsets computed for each lane line/landmark, these offsets may be applied to the vehicle path data, [0156] this path including the landmarks is then reconstructed after the offsets are applied, which is analogous to applying the transform parameter to the landmark detected).
PNG
media_image20.png
352
294
media_image20.png
Greyscale
PNG
media_image21.png
106
298
media_image21.png
Greyscale
(Hayat, [0155])
Regarding claim 6 Hayat discloses; The external world recognition device according to claim 5, wherein the transform selection unit preferentially selects the landmark having an attribute that greatly affects traveling of the own vehicle (Hayat, [0224] landmarks such as lane lines are used to determine a preferred vehicle path, as well as if the vehicle should transition between road segments, which indicates landmarks are selected or used in the analysis based on their relevance to the vehicle’s preferred path).
PNG
media_image22.png
350
296
media_image22.png
Greyscale
(Hayat, [0224])
Regarding claim 7 Hayat discloses; The external world recognition device according to claim 3, comprising an error estimation unit (Hayat, [0125] multiple processors are used to execute the image processing method disclosed, as per [0049] of applicant’s specification, a CPU or other hardware acts as the units disclosed, therefore the multiple processing units/processors of Hayat are analogous the claimed units) that estimates an error of the imaging plane estimated by the imaging plane estimation unit (Hayat, [0266] the error in landmark bounding box/imaging plane detection is determined using distance estimation and size estimation errors),
wherein the image transform estimation unit determines a range of error of the imaging plane to be subjected to image transform on a basis of an error of the imaging plane (Hayat, [0322] an error threshold may be used to determine if lane markings (landmarks) detected require correction, if a point or portion of a detected landmark (imaging plane) falls outside of a threshold (error range) determined based on other landmark points, it is corrected based upon the other points (image transform based on an error threshold)).
PNG
media_image23.png
344
304
media_image23.png
Greyscale
(Hayat, [0322])
Regarding claim 8 Hayat discloses; The external world recognition device according to claim 7, wherein the image transform estimation unit generates a plurality of imaging planes on a basis of an error of the imaging plane estimated by the error estimation unit and estimates a plurality of image transform parameters for each of the plurality of imaging planes (Hayat, [0350] a “binary mask” may be projected and overlaid on the road image showing detected locations/bounding boxes of detected landmarks plurality of imaging planes), [0351] due to errors in the estimated positions the masks/bounding boxes/image planes may not be aligned with the corresponding landmarks in the image, therefore translation and rotation parameters (transform parameters) may be determined for each feature/masks/plane to correct the mask/image plane.),
PNG
media_image24.png
428
298
media_image24.png
Greyscale
(Hayat, [0350]-[0351])
and the feature collation unit performs collation processing between a feature detected from the first image subjected to image transform using the plurality of image transform parameters and a feature detected from the second image a plurality of times (Hayat, [0136] a processing unit (feature collation unit) may take the first and second images, and their parameters to detect a set of features common to both, [0145] the processing unit may compare the object/landmark’s features/parameters across multiple frames),
and outputs a result of the collation processing with high evaluation to the position estimation unit (Hayat, [0143] images are compared to match objects between frames, where objects not matched are excluded and matched objects are sent to be processed and have their positions tracked as described in [0143]-[0147], therefore those objects that are matched, would have a high collation result, and then would be sent to have a position estimated).
Regarding claim 9 Hayat discloses; The external world recognition device according to claim 3, comprising a parallax calculation unit that calculates a parallax from the first image and the second image (Hayat, [0104] parallax information is computed between image capture devices to obtain the lateral displacement between the devices, [0123] parallax information can be obtained between multiple images from multiple cameras)
PNG
media_image25.png
510
294
media_image25.png
Greyscale
(Hayat, [0123])
and collates a position of a landmark appearing in the first image and the second image (Hayat, [0127]-[0128] a disparity in the pixel locations of objects between multiple images may be determined),
PNG
media_image26.png
312
294
media_image26.png
Greyscale
(Hayat, [0127]-[0128])
wherein the feature collation unit compares a first collation result obtained by collation between a feature of the landmark detected from the first image subjected to image transform using the image transform parameter and a feature of the landmark detected from the second image with a second collation result obtained by collation on a basis of the parallax (Hayat, [0123] stereo analysis is performed between a first and second image, which is a comparison of the two images of the same object using parallax computations, where [0127] a first image/set of images is reconstructed using map data and other camera information (first image subjected to transform) and then disparities between the pixels in the image are computed (first collation result), [0128] a second image or set of images may be received and disparities of pixels between the images are determined (second collation result on the basis of parallax), [0131] the processed images from the first and second sets (first and second collation results) are compared for validation of the results),
and outputs the first collation result or the second collation result to the position estimation unit on a basis of validity of a comparison result (Hayat, [0131] the results may be used or supplemented by the system based upon validation, [0137] the data that is output from the stereo processing method of [0131] is provided to a processing unit (position estimation unit) and used for positional estimation).
Regarding claim 10 Hayat discloses; An external world recognition method comprising:
a process of detecting, by a first landmark detection unit (Hayat, [0125] multiple processors are used to execute the image processing method disclosed, as per [0049] of applicant’s specification, a CPU or other hardware acts as the units disclosed, therefore the multiple processing units/processors of Hayat are analogous the claimed units), a landmark on a basis of a first image acquired from at least a first imaging unit among a plurality of imaging units in which at least a part of an imaging visual field with respect to an external world overlaps (Hayat, [0176] the system may have multiple image capturing devices (imaging units) to generate a sparse roadmap, which includes multiple identified landmarks, [0170] processing units (landmark detection unit) may detect from a plurality of images (at least 2 images) landmarks/road markings/signs that are common to the images);
a process of detecting, by a second landmark detection unit (Hayat, [0125] multiple processors are used to execute the image processing method disclosed, as per [0049] of applicant’s specification, a CPU or other hardware acts as the units disclosed, therefore the multiple processing units/processors of Hayat are analogous the claimed units), a landmark on a basis of a second image acquired from a second imaging unit among the plurality of imaging units (Hayat, [0170] a processing unit (landmark detection unit) may detect a landmark/lane markings/signs that are common between two or more images captured from each of a plurality of image capture devices (at least a first and a second capture device));
a process of estimating, by an imaging plane estimation unit (Hayat, [0125] multiple processors are used to execute the image processing method disclosed, as per [0049] of applicant’s specification, a CPU or other hardware acts as the units disclosed, therefore the multiple processing units/processors of Hayat are analogous the claimed units), an imaging plane in the first image on a basis of a detection result of the landmark by the first landmark detection unit (Hayat, [0176] the system may have multiple image capturing devices (imaging units) to generate a sparse roadmap, which includes multiple identified landmarks, [0170] processing units (landmark detection unit) may detect from a plurality of images (at least 2 images) landmarks/road markings/signs that are common to the images, [0201] polynomials for the landmarks in the sparse map (based on the images) are determined in 2D space (image plane estimation) paragraph [0045] of the applicant’s specification defines the imaging plane as the plane in the image where the landmark is detected based on the size of the landmark);
a process of estimating, by an image transform estimation unit (Hayat, [0125] multiple processors are used to execute the image processing method disclosed, as per [0049] of applicant’s specification, a CPU or other hardware acts as the units disclosed, therefore the multiple processing units/processors of Hayat are analogous the claimed units), an image transform parameter in accordance with an imaging plane in the second image on a basis of information of the imaging plane estimated by the imaging plane estimation unit and imaging parameters of the plurality of imaging units (Hayat, [0123] stereo analysis is performed between images based on determined variations of the image capture parameter, [0128] multiple processing devices may be used to determine parameters in the images, where a processing device may be used to detect landmarks such as road objects and signs and determined a distance or disparity between pixels for each image, [0270] a matching algorithm (image transform estimation unit) may be used to align landmarks appearing in the multiple maps or images, [0271] figure 16 shows the rectangular plane associated with a sign (landmark) being aligned/matched from multiple sparse map image inputs using the matching algorithm previously described in [0270]);
a process of collating a feature of the landmark detected from the first image subjected to image transform using an image transform parameter by the image transform estimation unit with a feature of the landmark detected from the second image (Hayat, [0136] a processing unit (feature collation unit) may take the first and second images, and their parameters to detect a set of features common to both, [0145] the processing unit may compare the object/landmark’s features/parameters across multiple frames);
and a process of estimating a three-dimensional position of the landmark on a basis of a result of collation processing (Hayat, [0145] a processing unit (position estimation unit) compares the objects positions across frames and estimates and predicts a position of the object, [0141] the frames and comparisons are used to create a 3D mapping of the road surface/landmarks; therefore, the positions are in 3D).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. For a full listing of analogous prior art as cited by the examiner, please see the attached PTO-892 Notice of References cited form.
Shashua, (US 20170010618 A1), pertains to a system using stereo imaging and parallax computations to determine landmarks near a vehicle.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JORDAN M ELLIOTT whose telephone number is (703)756-5463. The examiner can normally be reached M-F 8AM-5PM ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Emily Terrell can be reached at (571) 270-3717. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/J.M.E./Examiner, Art Unit 2666 /Molly Wilburn/Primary Examiner, Art Unit 2666
Prosecution Insights