DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 06/02/2026 has been entered.
Response to Arguments
Applicant’s arguments, filed 06/02/2026, with respect to claims 1-2, 4-10 and 11-20 have been fully considered but are moot because they are not applicable to the current references and/or current grounds for rejection.
Claim Objections
Claims 1, 15-16 and 18-19 are objected to because of the following informalities:
In claim 1, Line 11-12, the term “estimating, by the controller, a pose of the vehicle depending on the change amount in pose of the camera, wherein the setting of the area around the vanishing point as the template includes:” should be corrected “estimating, by the controller, a pose of the vehicle depending on the change amount in pose of the camera[[,]]; wherein the setting of the area around the vanishing point as the template includes:” in order to replace a comma with a semi-colon to improve the quality and clarity of the claim, and ensure the metes and bounds of the claim language are defined. Appropriate corrections required.
In claim 1, Line 11-12, the term “and changing a position of the template, based on the reliability of the template being low, and wherein the estimating of the change amount in the pose of the camera” should be corrected “and changing a position of the template, based on the reliability of the template being low[[,]]; and wherein the estimating of the change amount in the pose of the camera” in order to replace a comma with a semi-colon to improve the quality and clarity of the claim, and ensure the metes and bounds of the claim language are defined. Appropriate corrections required.
In claim 15, Line 6-7, the term “set, as a template, an area around a vanishing point in a previous frame of an image input from the camera,” should be corrected “set, as a template, an area around a vanishing point in a previous frame of an image input from the camera[[,]];” in order to replace a comma with a semi-colon to improve the quality and clarity of the claim, and ensure the metes and bounds of the claim language are defined. Appropriate corrections required.
In claim 15, Line 8-9, the term “determine a matching area matching with the template by performing template matching in a current frame,” should be corrected “determine a matching area matching with the template by performing template matching in a current frame[[,]];” in order to replace a comma with a semi-colon to improve the quality and clarity of the claim, and ensure the metes and bounds of the claim language are defined. Appropriate corrections required.
In claim 15, Line 10-11, the term “determine an amount of position change of the vanishing point based on an amount of position change between the template and the matching area,” should be corrected “determine an amount of position change of the vanishing point based on an amount of position change between the template and the matching area[[,]];” in order to replace a comma with a semi-colon to improve the quality and clarity of the claim, and ensure the metes and bounds of the claim language are defined. Appropriate corrections required.
In claim 15, Line 12-13, the term “estimate a change amount in a pose of the camera based on the amount of position change of the vanishing point,” should be corrected “estimate a change amount in a pose of the camera based on the amount of position change of the vanishing point[[,]];” in order to replace a comma with a semi-colon to improve the quality and clarity of the claim, and ensure the metes and bounds of the claim language are defined. Appropriate corrections required.
In claim 15, Line 14-15, the term “and estimate a pose of the vehicle depending on the change amount in pose of the camera,” should be corrected “and estimate a pose of the vehicle depending on the change amount in pose of the camera[[,]];” in order to replace a comma with a semi-colon to improve the quality and clarity of the claim, and ensure the metes and bounds of the claim language are defined. Appropriate corrections required.
In claim 15, Line 19-20, the term “and change a position of the template, based on the reliability of the template being low,” should be corrected “and change a position of the template, based on the reliability of the template being low[[,]];” in order to replace a comma with a semi-colon to improve the quality and clarity of the claim, and ensure the metes and bounds of the claim language are defined. Appropriate corrections required.
In claim 16, Line 1-2, the term “wherein the controller is configured to change a position of the template based on the variance value of the template,” should be corrected “wherein the controller is configured to change a position of the template based on the variance value of the template[[,]];” in order to replace a comma with a semi-colon to improve the quality and clarity of the claim, and ensure the metes and bounds of the claim language are defined. Appropriate corrections required.
In claim 18, Line 2-3, the term “wherein the camera is a front camera configured to obtain image data for a field of view facing a front of the vehicle,” should be corrected “wherein the camera is a front camera configured to obtain image data for a field of view facing a front of the vehicle[[,]];” in order to replace a comma with a semi-colon to improve the quality and clarity of the claim, and ensure the metes and bounds of the claim language are defined. Appropriate corrections required.
In claim 18, Line 5-6, the term “determine a change amount in y-axis of the template based on the amount of position change between the template and the matching area,” should be corrected “determine a change amount in y-axis of the template based on the amount of position change between the template and the matching area[[,]];” in order to replace a comma with a semi-colon to improve the quality and clarity of the claim, and ensure the metes and bounds of the claim language are defined. Appropriate corrections required.
In claim 18, Line 7-9, the term “determine the change amount in y-axis of the vanishing point based on a change amount in y-axis where a roll slope at which the front camera is mounted is compensated from the change amount in y-axis of the template,” should be corrected “determine the change amount in y-axis of the vanishing point based on a change amount in y-axis where a roll slope at which the front camera is mounted is compensated from the change amount in y-axis of the template[[,]];” in order to replace a comma with a semi-colon to improve the quality and clarity of the claim, and ensure the metes and bounds of the claim language are defined. Appropriate corrections required.
In claim 18, Line 10-11, the term “estimate an amount of pitch change of the front camera based on the change amount in y-axis of the vanishing point,,” should be corrected “estimate an amount of pitch change of the front camera based on the change amount in y-axis of the vanishing point[[,]];” in order to replace a comma with a semi-colon to improve the quality and clarity of the claim, and ensure the metes and bounds of the claim language are defined. Appropriate corrections required.
In claim 19, Line 2-3, the term “wherein the camera is a multi-camera configured to obtain image data for a field of view facing a plurality of directions of the vehicle,” should be corrected “wherein the camera is a multi-camera configured to obtain image data for a field of view facing a plurality of directions of the vehicle[[,]];” in order to replace a comma with a semi-colon to improve the quality and clarity of the claim, and ensure the metes and bounds of the claim language are defined. Appropriate corrections required.
In claim 19, Line 4-5, the term “fuse an amount of position change of a vanishing point corresponding to each camera of the multi-camera,” should be corrected “fuse an amount of position change of a vanishing point corresponding to each camera of the multi-camera[[,]];” in order to replace a comma with a semi-colon to improve the quality and clarity of the claim, and ensure the metes and bounds of the claim language are defined. Appropriate corrections required.
In claim 19, Line 6-7, the term “estimate a change amount in pose of each of the cameras of the multi-camera based on the fused amount of position change of the vanishing point,” should be corrected “estimate a change amount in pose of each of the cameras of the multi-camera based on the fused amount of position change of the vanishing point[[,]];” in order to replace a comma with a semi-colon to improve the quality and clarity of the claim, and ensure the metes and bounds of the claim language are defined. Appropriate corrections required.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
Claims 1 and 15-20 recites limitations that use words like “means” (or “step”) or similar terms with functional language but do not invoke 35 U.S.C. 112(f):
Claims 1; recites the limitation, “determining, by a controller…” [Line 4 and 6].
Claims 1; recites the limitation, “estimating, by the controller…” [Line 8 and 10].
Claims 15; recites the limitation, “wherein the controller is configured to…” [Line 4 and 20].
Claims 15; recites the limitation, “wherein the controller is further configured to…” [Line 15].
Claims 16; recites the limitation, “wherein the controller is configured to…” [Line 2].
Claims 17; recites the limitation, “wherein the controller is configured to…” [Line 1].
Claims 18; recites the limitation, “wherein the controller is configured to…” [Line 4].
Claims 19; recites the limitation, “wherein the controller is configured to….” [Line 4].
Claims 20; recites the limitation, “wherein the controller is configured to…” [Line 1].
Because these claim limitation(s) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, they are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
After a careful analysis, as disclosed above, and a careful review of the specification the following limitations in claims 1 and 15-20:
(i) “controller” (Fig. 2, #200. Paragraph [0076-0079 and 0081-0082]-A controller #200 is described as performing the overall control of vehicle 1 and being configured to identify objects, estimate the change amount in pose of the front camera, estimate a pose of the vehicle and generate a front image. The controller 200 is further described as including an image signal processor #210, memory #220 and a communicator #230. In Fig. 2, the controller is illustrated as a blackbox containing the processor #210, memory #220 and a communicator #230. The communicator #230 is described as containing at least one component facilitating communication between an external device and constituent components, such as cameras, displays, brake devices, etc. Thus, the controller #200 has sufficient associated structure and/or material, and the controller is a computer and/or system with a processor, memory and communicator).
If applicant does not intend to have these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Please note: The Office has respectfully added 112(f) interpretation to enhance clarity and quality on the record.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2, 8-10, 12-16, and 18-20 are rejected under 35 U.S.C. 103 as being unpatentable over SUGIE et al. (US 20200082179 A1), hereinafter referenced as SUGIE in view of NAGAI et al. (US 20180322655 A1, hereinafter referenced as NAGAI and in further view of AKITA et al. (US 20100027844 A1), hereinafter referenced as AKITA.
Regarding claim 1, SUGIE explicitly teaches control method (Fig. 28. Paragraph [0139]-SUGIE discloses the image processing section 162 executes a process shown in FIG. 28. The process shown in FIG. 28 is obtained by adding S4A to the process shown in FIG. 2 between S4 and S5) of a vehicle (Fig. 24, #1 called a vehicle. Paragraph [0058]-SUGIE discloses the road map information update system 100 includes a camera 110, a parameter storage portion 120, a GPS receiver 130, a behavior detection sensor 140, a map storage portion 150, a computation portion 160, and an accumulated data storage portion 170. The road map information update system 100 is mounted on a vehicle 1. In paragraph [0066]-SUGIE discloses the computation portion 160 functions as a vehicle position detection section 161, an image processing section 162, a collation region identification section 163, and a collation processing section 164. When these functions are performed, a method corresponding to the program is performed. Please also see Fig. 28), the control method comprising:
setting, as a template (Fig. 30, #386 called a vanishing point region. Paragraph [0141]-SUGIE discloses FIG. 30 shows the vanishing point region image 386 obtained in the previous frame), an area around a vanishing point in a previous frame (Fig. 30. Paragraph [0141]-SUGIE discloses FIG. 30 shows the vanishing point region image 386 obtained in the previous frame) of an image input from a camera (Fig. 24, #110 called a camera. Paragraph [0058]. In paragraph [0140]-SUGIE discloses in S41 in FIG. 29, a vanishing point region image 386 is determined. The vanishing point region image 386 is an image of a region determined on the basis of the vanishing point in the corrected forward view image 181. The vanishing point region images 386 shown in FIGS. 30 and 31 have a horizontally-long rectangular shape which has a pair of short sides extending in the vertical direction and a pair of long sides extending in the horizontal direction with the vanishing point 183 (see FIGS. 11A and 11B) as a center. Please also see Fig. 28-29 and read paragraph [0141-0143]);
determining, by a controller (Fig. 24, #260 called a computation portion. Paragraph [0066]-SUGIE discloses the computation portion 160 is a computer provided with a CPU, a ROM, a RAM, and the like. The CPU executes a program stored in a non-transitory tangible storage medium such as a ROM, while using a temporary storage function of the RAM. In paragraph [0067]-SUGIE discloses a part or all of the functional blocks included in the computation portion 160 may be achieved by a combination of software execution by the CPU and hardware members. Please also read paragraph [0131]), a matching area matching with the template by performing template matching in a current frame (Fig. 29. Paragraph [0141]-SUGIE discloses in S42, a matching process is performed for matching the vanishing point region image 386 obtained in the previous frame, that is, the vanishing point region image 386 determined in the previous execution of the process in FIG. 28, and the vanishing point region image 386 obtained in S41 performed immediately before S42. FIG. 30 shows the vanishing point region image 386 obtained in the previous frame, and FIG. 31 shows the vanishing point region image 386 obtained in the current frame. Please also see Fig. 28 and read paragraph [0142-0143]);
determining, by the controller (Fig. 24, #260 called a computation portion. Paragraph [0066 and 0131]), an amount of position change of the vanishing point based on an amount of position change between the template and the matching area (Fig. 31. Paragraph [0142]-SUGIE discloses in S43, the amount of movement of the vanishing point region image 386 obtained in the current frame in S42 is calculated. The amount of movement is used as a value that indicates a pitch variation occurring in the vehicle 1 within one frame. S41 to S43 correspond to an amount of movement calculation section. In paragraph [0143]-SUGIE discloses in S44, the amount of movement calculated in S43 is sequentially integrated in the processes, shown in FIGS. 28 and 29, which are repeatedly executed each time the forward view image 180 is input. The integrated value is referred to as a positional deviation integration value. The positional deviation integration value includes the positional deviation of the vanishing point region image 386 due to the pitch variation and a change in the position of the vanishing point region image 386 due to static reasons other than the pitch variation. The static reasons other than the pitch variation include, for example, a change in the optical axis of the camera due to the vehicle 1 traveling on an uphill. S44 corresponds to an integration value calculation section. Please also see Fig. 28);
estimating, by the controller (Fig. 24, #260 called a computation portion. Paragraph [0066 and 0131]), a change amount in a pose of the camera (Fig. 24, #110 called a camera. Paragraph [0058]) based on the amount of position change of the vanishing point (Fig. 29. Paragraph [0146]-SUGIE discloses after the process in S46 is executed, the process flow proceeds to S5A in FIG. 28. In S5A, the orientation of the camera 110 is detected. Specifically, the orientation of the camera 110 is indicated by the yaw angle rx and the tilt angle ry. The yaw angle rx is obtained from Equation 4. The tilt angle ry is obtained from Equation 10. As shown in Equation 10, the yaw angle rx is calculated from the distance Δy between the image center 184 and the vanishing point 183 in the y direction and the amount of pitch variation δy. In paragraph [0147]-SUGIE discloses S6 to S9 are executed to output the bird's-eye view image. In paragraph [0148]-SUGIE discloses the bird's-eye view image subjected to pitch correction is the bird's-eye view image obtained by performing the process in FIG. 28. Please also see Fig. 5-6, 11A-B and 28); and
and wherein the estimating of the change amount in the pose of the camera (Fig. 24, #110 called a camera. Paragraph [0058]) includes estimating an amount of pitch change of the camera (Fig. 28. Paragraph [0146]-SUGIE discloses after the process in S46 is executed, the process flow proceeds to S5A in FIG. 28. In S5A, the orientation of the camera 110 is detected. Specifically, the orientation of the camera 110 is indicated by the yaw angle rx and the tilt angle ry. The yaw angle rx is obtained from Equation 4. The tilt angle ry is obtained from Equation 10. As shown in Equation 10, the yaw angle rx is calculated from the distance Δy between the image center 184 and the vanishing point 183 in the y direction and the amount of pitch variation δy) based on a change amount in y-axis of the vanishing point (Fig. 28. Paragraph [0142]-SUGIE discloses in S43, the amount of movement of the vanishing point region image 386 obtained in the current frame in S42 is calculated. The amount of movement is used as a value that indicates a pitch variation occurring in the vehicle 1 within one frame. In paragraph [0143]-SUGIE discloses in S44, the amount of movement calculated in S43 is sequentially integrated in the processes, shown in FIGS. 28 and 29, which are repeatedly executed each time the forward view image 180 is input. The integrated value is referred to as a positional deviation integration value. The positional deviation integration value includes the positional deviation of the vanishing point region image 386 due to the pitch variation and a change in the position of the vanishing point region image 386 due to static reasons other than the pitch variation. The static reasons other than the pitch variation include, for example, a change in the optical axis of the camera due to the vehicle 1 traveling on an uphill).
Although SUGIE explicitly teaches wherein the setting of the area around the vanishing point as the template includes: determining a reliability of the template based on a variance value of the template (Fig. 29. Paragraph [0143]-SUGIE discloses the integrated value is referred to as a positional deviation integration value. The positional deviation integration value includes the positional deviation of the vanishing point region image 386 due to the pitch variation and a change in the position of the vanishing point region image 386 due to static reasons other than the pitch variation. The static reasons other than the pitch variation include, for example, a change in the optical axis of the camera due to the vehicle 1 traveling on an uphill. S44 corresponds to an integration value calculation section. In paragraph [0145]-SUGIE discloses in S46, the difference between the positional deviation integration value calculated in S44 and the moving average value calculated in S45 is calculated. This difference is referred to as an amount of pitch variation δy. If determined in advance, the amount of pitch variation δy may be calculated from either the equation of (positional deviation integration value)−(moving average value) or the equation of (moving average value)−(positional deviation integration value). S45 and S46 correspond to an amount of pitch variation calculation section. FIG. 32 shows the positional deviation integration value, the moving average value, and the amount of pitch variation δy. Please also see Fig. 23 and read paragraph [0119-0124, 0129 and 0144]);
SUGIE fail to teach estimating, by the controller, a pose of the vehicle depending on the change amount in pose of the camera.
However, NAGAI explicitly teaches estimating, by the controller (Fig. #40 and #50 called a bird's-eye-view-image generating device and a posture-change determining device, respectively. Paragraph [0046]-NAGAI discloses the bird's-eye-view-image generating device 40 is an arithmetic processing unit that is constituted of, for example, a CPU (central processing unit) or the like. The bird's-eye-view-image generating device 40 loads a program that is stored in the storage device 30 into a memory, and executes a command included in the program. The bird's-eye-view-image generating device 40 includes the image acquiring unit 41, a vehicle-information acquiring unit 42, and a control unit 43 that includes a posture-change determining unit 44, a bird's-eye-view-image generating unit 45, and the display control unit 46. Please also read paragraph [0047]), a pose of the vehicle (Fig. 1, #V called a vehicle. Paragraph [0051]-NAGAI discloses the posture-change determining unit 44 then determines whether the posture of the vehicle V has changed based on the variations of the tilt angle θX1 to the tilt angle θX4 of the periphery imaging camera 21 to the periphery imaging camera 24, the variations of the height y1 to the height y4, the variations of the rotation angle θZ1 to the rotation angle θZ4 based on multiple periphery images that are acquired by the image acquiring unit 41. In paragraph [0052]-NAGAI discloses the bird's-eye-view-image generating unit 45 generates the bird's-eye view image 300 by subjecting the periphery images acquired by the image acquiring unit 41 to the eye point conversion so that an image looking the vehicle V down from above is obtained, and by superimposing the virtual own-vehicle image A looking the vehicle V from above) depending on the change amount in pose (Fig. 1. Paragraph [0044]-NAGAI discloses the storage device 30 stores a periphery image of the latest frame and a periphery image of a next previous frame acquired by the image acquiring unit 41. In paragraph [0045]-NAGAI discloses the storage device 30 associates a difference in a height direction (height direction variation) of a vanishing point P in two periphery imagers acquired by the forward-periphery imaging camera 21 and a difference in the tilt angle θX1 (variation of the tilt angle θX1) of the forward-periphery imaging camera 21 with each other, to store as tilt-angle variation data. The storage device 30 stores a difference in a height direction of a vanishing point P in two periphery imagers acquired by the rearward-periphery imaging camera 22 and a difference in the tilt angle θX2 (variation of the tilt angle θX2) of the rearward-periphery imaging camera 22 with each other, to store as tilt-angle variation data. The storage device 30 stores a difference in a height direction of a vanishing point P in two periphery imagers acquired by the leftward-periphery imaging camera 23 and a difference in the tilt angle θX3 (variation of the tilt angle θX3) of the leftward-periphery imaging camera 23 with each other, to store as tilt-angle variation data. The storage device 30 stores a difference in a height direction of a vanishing point P in two periphery imagers acquired by the rightward-periphery imaging camera 24 and a difference in the tilt angle θX4 (variation of the tilt angle θX4) of the rightward-periphery imaging camera 24 with each other, to store as tilt-angle variation data (wherein the posture/pose of each camera is estimated using six axes of coordinates (x3, y3, z3) and rotation components about the X axis, the Y axis, and the Z axis (θX3, θY3, θZ3), and the change in pose of each camera is based on a change in the y-axis of a vanishing point in each camera). Please also see Fig. 8-9 and read paragraph [0063]) of the camera (Fig. 1, #21, #22, #23, #24 and #41 called a forward-periphery imaging camera, a rearward-periphery imaging camera, a leftward-periphery imaging camera, and a rightward-periphery imaging camera, and image acquiring unit, respectively. Paragraph [0029 and 0046]. In paragraph [0051]-NAGAI discloses the posture-change determining unit 44 calculates variations of the tilt angle θX1 to the tilt angle θX4 of the periphery imaging camera 21 to the periphery imaging camera 24, variations of the height y1 to the height y4, variations of the rotation angle θZ1 to the rotation angle θZ4 based on multiple periphery images that are acquired by the image acquiring unit 41 (wherein the posture/pose of each camera is estimated using six axes of coordinates (x3, y3, z3) and rotation components about the X axis, the Y axis, and the Z axis (θX3, θY3, θZ3), and the change in pose of each camera is based on a change in the y-axis of a vanishing point in each camera). Please also read paragraph [0054-0056]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of SUGIE of having a control method of a vehicle, the control method comprising: setting, as a template, an area around a vanishing point in a previous frame of an image input from a camera; determining, by a controller, a matching area matching with the template by performing template matching in a current frame; determining, by the controller, an amount of position change of the vanishing point based on an amount of position change between the template and the matching area; estimating, by the controller, a change amount in a pose of the camera based on the amount of position change of the vanishing point, with the teachings of NAGAI of having estimating, by the controller, a pose of the vehicle depending on the change amount in pose of the camera.
Wherein SUGIE’s method having estimating, by the controller, a pose of the vehicle depending on the change amount in pose of the camera.
The motivation behind the modification would have been to obtain a method of that improves autonomous driving and camera calibration, since both CHOI and NAGAI concern vehicles and image analysis. Wherein SUGIE provides systems and methods that improve the bird's-eye view image and the accuracy of the subsequent processes performed based on this image, while NAGAI provides systems and methods that improves the ability to generate a birds-eye image from multiple cameras as well as determine the posture of a vehicle and cameras. Please see SUGIE et al. (US 20200082179 A1), Abstract and Paragraph [0129 and 0150] and NAGAI et al. (US 20180322655 A1), Abstract and Paragraph [0003-0010].
Although NAGAI explicitly teaches and wherein the estimating of the change amount in the pose of the camera includes estimating an amount of pitch change of the camera based on a change amount in y-axis of the vanishing point.
SUGIE in view of NAGAI fail to explicitly teach and changing a position of the template, based on the reliability of the template being low.
However, AKITA explicitly teaches and changing a position of the template (Fig. 6. Paragraph [0115]-AKITA discloses optical flows of the single moving object 40 are extended. These extended straight lines L intersect at one point, which is the focus of expansion: FOE, which is defined as the so-called point at infinity or the vanishing point. Each point feature of the moving object 40 which translates in the three-dimensional space has a common focus of expansion FOE in the image plane. In paragraph [0142]-AKITA discloses the optical flow calculation unit 4 sets a template around the feature point obtained as above and searches for an area having high correlativity with this template, from the subsequent image in the time series. (template matching)), based on the reliability of the template being low (Fig. 6. Paragraph [0142]-AKITA discloses as the result of the search, its movement vector is obtained as a motion vector (optical flow). As a method of such template matching, a normalized cross correlation technique (NCC) can be used (wherein template matching may use sequential similarity search or least squares). Further in paragraph [0143]-AKITA discloses a template image T (i, j) having (M.sub.T.times.N.sub.T) pixels is moved on the pixels present within a search area, thereby to search a point in the template image which point has the maximum correlativity coefficient (correlativity value) R.sub.NCC (a, b) in the following Formula (24). In paragraph [0144]-AKITA discloses (a, b) represents the position of the template image T within the image in the search area and I.sub.(a, b) (i, j) represents a partial image within the image in the search area. In paragraph [0147]-AKITA discloses it is possible to restrict the search area in advance in the image processing area setting step, according to the position of the moving object recognized in the previous flow (wherein various reliability values and thresholds may be used to adjust recognition results such as recognition reliability, feature amount reliability, optical flow reliability, grouping reliability, fused reliability, continuity reliability, moving object determination reliability and/or secondary intermediate reliability). Please also read paragraph [0150-0151, 0177-0189, 0204-0207, and 0209]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of SUGIE in view of NAGAI of having a control method of a vehicle, the control method comprising: setting, as a template, an area around a vanishing point in a previous frame of an image input from a camera; determining, by a controller, a matching area matching with the template by performing template matching in a current frame; determining, by the controller, an amount of position change of the vanishing point based on an amount of position change between the template and the matching area; estimating, by the controller, a change amount in a pose of the camera based on the amount of position change of the vanishing point, estimating, by the controller, a pose of the vehicle depending on the change amount in pose of the camera, with the teachings of AKITA of having and changing a position of the template, based on the reliability of the template being low.
Wherein SUGIE’s method having and changing a position of the template, based on the reliability of the template being low, and wherein the estimating of the change amount in the pose of the camera includes estimating an amount of pitch change of the camera based on a change amount in y-axis of the vanishing point.
The motivation behind the modification would have been to obtain a method of that improves efficiency and accuracy of image generation and calibration, since both SUGIE and AKITA concern vehicles, image analysis and vanishing point detection. Wherein SUGIE provides systems and methods that improve the bird's-eye view image and the accuracy of the subsequent processes performed based on this image, while AKITA provides systems and methods that improves the safety in vehicle driving. Please see SUGIE et al. (US 20200082179 A1), Abstract and Paragraph [0129 and 0150] and AKITA et al. (US 20100027844 A1), Abstract and Paragraph [0071].
Regarding claim 2, SUGIE in view of NAGAI and in further view of AKITA explicitly teach the control method of claim 1, SUGIE in view of NAGAI fails to explicitly teach wherein the setting of the area around the vanishing point as the template further includes changing a position of the template based on a variance value of the template, and wherein the variance value of the template is determined based on a degree of contrast of the image.
However, AKITA explicitly teaches wherein the setting of the area around the vanishing point as the template (Fig. 6. Paragraph [0115]-AKITA discloses optical flows of the single moving object 40 are extended. These extended straight lines L intersect at one point, which is the focus of expansion: FOE, which is defined as the so-called point at infinity or the vanishing point. Each point feature of the moving object 40 which translates in the three-dimensional space has a common focus of expansion FOE in the image plane. In paragraph [0142]-AKITA discloses the optical flow calculation unit 4 sets a template around the feature point obtained as above and searches for an area having high correlativity with this template, from the subsequent image in the time series. (template matching))) further includes changing a position of the template based on a variance value of the template, and wherein the variance value of the template is determined based on a degree of contrast of the image (Fig. 6. Paragraph [0143]-AKITA discloses a template image T (i, j) having (M.sub.T.times.N.sub.T) pixels is moved on the pixels present within a search area, thereby to search a point in the template image which point has the maximum correlativity coefficient (correlativity value) R.sub.NCC (a, b) in the following Formula (24) (wherein various reliability values and thresholds may be used to adjust recognition results such as recognition reliability, feature amount reliability, optical flow reliability, grouping reliability, fused reliability, continuity reliability, moving object determination reliability and/or secondary intermediate reliability). In paragraph [0144]-AKITA discloses (a, b) represents the position of the template image T within the image in the search area and I.sub.(a, b) (i, j) represents a partial image within the image in the search area. In paragraph [0147]-AKITA discloses it is possible to restrict the search area in advance in the image processing area setting step, according to the position of the moving object recognized in the previous flow. Please also read paragraph [0009 and 0030]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of SUGIE in view of NAGAI and in further view of AKITA of having a control method of a vehicle, the control method comprising: setting, as a template, an area around a vanishing point in a previous frame of an image input from a camera; determining, by a controller, a matching area matching with the template by performing template matching in a current frame, with the teachings of AKITA of having wherein the setting of the area around the vanishing point as the template further includes changing a position of the template based on a variance value of the template, and wherein the variance value of the template is determined based on a degree of contrast of the image.
Wherein SUGIE’s method having wherein the setting of the area around the vanishing point as the template further includes changing a position of the template based on a variance value of the template, and wherein the variance value of the template is determined based on a degree of contrast of the image.
The motivation behind the modification would have been to obtain a method of that improves efficiency and accuracy of image generation and calibration, since both SUGIE and AKITA concern vehicles, image analysis and vanishing point detection. Wherein SUGIE provides systems and methods that improve the bird's-eye view image and the accuracy of the subsequent processes performed based on this image, while AKITA provides systems and methods that improves the safety in vehicle driving. Please see SUGIE et al. (US 20200082179 A1), Abstract and Paragraph [0129 and 0150] and AKITA et al. (US 20100027844 A1), Abstract and Paragraph [0071].
Regarding claim 8, SUGIE in view of NAGAI and in further view of AKITA explicitly teach the control method of claim 1, SUGIE in view of NAGAI fail to explicitly teach wherein the determining of the matching area includes performing the template matching using a normalized cross correlation matching.
However, AKITA explicitly teaches wherein the determining of the matching area includes performing the template matching using a normalized cross correlation matching (Fig. 6. Paragraph [0142]-AKITA discloses the optical flow calculation unit 4 sets a template around the feature point obtained as above and searches for an area having high correlativity with this template, from the subsequent image in the time series. (template matching) As the result of the search, its movement vector is obtained as a motion vector (optical flow). As a method of such template matching, a normalized cross correlation technique (NCC) can be used).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of SUGIE in view of NAGAI and in further view of AKITA of having a control method of a vehicle, the control method comprising: setting, as a template, an area around a vanishing point in a previous frame of an image input from a camera; determining, by a controller, a matching area matching with the template by performing template matching in a current frame, with the teachings of AKITA of having wherein the determining of the matching area includes performing the template matching using a normalized cross correlation matching.
Wherein SUGIE’s method having wherein the determining of the matching area includes performing the template matching using a normalized cross correlation matching.
The motivation behind the modification would have been to obtain a method of that improves efficiency and accuracy of image generation and calibration, since both SUGIE and AKITA concern vehicles, image analysis and vanishing point detection. Wherein SUGIE provides systems and methods that improve the bird's-eye view image and the accuracy of the subsequent processes performed based on this image, while AKITA provides systems and methods that improves the safety in vehicle driving. Please see SUGIE et al. (US 20200082179 A1), Abstract and Paragraph [0129 and 0150] and AKITA et al. (US 20100027844 A1), Abstract and Paragraph [0071].
Regarding claim 9, SUGIE in view of NAGAI and in further view of AKITA explicitly teach the control method of claim 8, SUGIE in view of NAGAI fail to explicitly teach wherein the determining of the matching area includes performing the template matching using the normalized cross correlation matching in the current frame consecutive from the previous frame.
However, AKITA explicitly teaches wherein the determining of the matching area includes performing the template matching using the normalized cross correlation matching (Fig. 6. Paragraph [0142]-AKITA discloses the optical flow calculation unit 4 sets a template around the feature point obtained as above and searches for an area having high correlativity with this template, from the subsequent image in the time series. (template matching). As a method of such template matching, a normalized cross correlation technique (NCC) can be used (wherein template matching may use sequential similarity search or least squares). Please also read paragraph [0150-0151, 0177-0189, 0204-0207, and 0209]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of SUGIE in view of NAGAI and in further view of AKITA of having a control method of a vehicle, the control method comprising: setting, as a template, an area around a vanishing point in a previous frame of an image input from a camera; determining, by a controller, a matching area matching with the template by performing template matching in a current frame, with the teachings of AKITA of having wherein the determining of the matching area includes performing the template matching using the normalized cross correlation matching in the current frame consecutive from the previous frame.
Wherein SUGIE’s method having wherein the determining of the matching area includes performing the template matching using the normalized cross correlation matching in the current frame consecutive from the previous frame.
The motivation behind the modification would have been to obtain a method that improves efficiency and accuracy of image generation and calibration, since both SUGIE and AKITA concern vehicles, image analysis and vanishing point detection. Wherein SUGIE provides systems and methods that improve the bird's-eye view image and the accuracy of the subsequent processes performed based on this image, while AKITA provides systems and methods that improves the safety in vehicle driving. Please see SUGIE et al. (US 20200082179 A1), Abstract and Paragraph [0129 and 0150] and AKITA et al. (US 20100027844 A1), Abstract and Paragraph [0071].
Regarding claim 10, SUGIE in view of NAGAI and in further view of AKITA explicitly teach the control method of claim 1, SUGIE explicitly teaches wherein the camera (Fig. 24, #110 called a camera. Paragraph [0058]) is a front camera configured to obtain image data for a field of view facing a front of the vehicle (Fig. 1. Paragraph [0059]-SUGIE discloses the camera 110 is installed in a direction so that the area in front of the vehicle 1 can be captured, and captures an image of the area in front of the vehicle 1 in the surrounding areas of the vehicle 1. In the following description, an image captured by the camera 110 is referred to as a forward view image 180. FIG. 3 shows an example of the forward view image 180), and wherein the determining of the amount of position change of the vanishing point includes:
determining a change amount in y-axis of the template based on the amount of position change between the template and the matching area (Fig. 31. Paragraph [0142]-SUGIE discloses in S43, the amount of movement of the vanishing point region image 386 obtained in the current frame in S42 is calculated. The amount of movement is used as a value that indicates a pitch variation occurring in the vehicle 1 within one frame. S41 to S43 correspond to an amount of movement calculation section. In paragraph [0143]-SUGIE discloses in S44, the amount of movement calculated in S43 is sequentially integrated in the processes, shown in FIGS. 28 and 29, which are repeatedly executed each time the forward view image 180 is input. The integrated value is referred to as a positional deviation integration value. The positional deviation integration value includes the positional deviation of the vanishing point region image 386 due to the pitch variation and a change in the position of the vanishing point region image 386 due to static reasons other than the pitch variation. The static reasons other than the pitch variation include, for example, a change in the optical axis of the camera due to the vehicle 1 traveling on an uphill. S44 corresponds to an integration value calculation section. Please also see Fig. 28).
Although SUGIE explicitly teaches and determining the change amount in y-axis of the vanishing point based on a change amount in y-axis where a slope at which the front camera is mounted is compensated from the change amount in y-axis of the template (Fig. 28. Paragraph [0146]-SUGIE discloses after the process in S46 is executed, the process flow proceeds to S5A in FIG. 28. In S5A, the orientation of the camera 110 is detected. Specifically, the orientation of the camera 110 is indicated by the yaw angle rx and the tilt angle ry. The yaw angle rx is obtained from Equation 4. The tilt angle ry is obtained from Equation 10. As shown in Equation 10, the yaw angle rx is calculated from the distance Δy between the image center 184 and the vanishing point 183 in the y direction and the amount of pitch variation δy).
SUGIE fail to teach where a roll slope at which the front camera is mounted is compensated from the change amount in y-axis.
However, NAGAI explicitly teaches where a roll slope at which the front camera is mounted is compensated from the change amount in y-axis (Fig. 1. Paragraph [0044]-NAGAI discloses the storage device 30 stores a periphery image of the latest frame and a periphery image of a next previous frame acquired by the image acquiring unit 41. In paragraph [0045]-NAGAI discloses the storage device 30 associates a difference in a height direction (height direction variation) of a vanishing point P in two periphery imagers acquired by the forward-periphery imaging camera 21 and a difference in the tilt angle θX1 (variation of the tilt angle θX1) of the forward-periphery imaging camera 21 with each other, to store as tilt-angle variation data (wherein the height direction is in the direction of the y-axis)).
where a roll slope at which the front camera (Fig. 6, #21 called a forward-periphery imaging camera. Paragraph [0032]-NAGAI discloses the posture of the forward-periphery imaging camera 21 is identified by six axes of coordinates (x1, y1, z1) and rotation components about the X axis, the Y axis, and the Z axis (θX1, θY1, θZ1). In the present embodiment, a tilt angle θX1, which is the rotation component about the X axis, a height y1 in the direction of the Y axis indicating a height from the contact ground surface, and a rotation angle θZ1, which is a rotation component about the Z axis vary in the forward-periphery imaging camera 21. In paragraph [0044]-NAGAI discloses the storage device 30 stores a periphery image of the latest frame and a periphery image of a next previous frame acquired by the image acquiring unit 41. In paragraph [0045]-NAGAI discloses the storage device 30 associates a difference in a height direction (height direction variation) of a vanishing point P in two periphery imagers acquired by the forward-periphery imaging camera 21 and a difference in the tilt angle θX1 (variation of the tilt angle θX1) of the forward-periphery imaging camera 21 with each other, to store as tilt-angle variation data (wherein the height direction is in the direction of the y-axis)). Please also read paragraph [0054-0056]) is mounted is compensated from the change amount in y-axis (Fig. 6. Paragraph [0051]-NAGAI discloses the posture-change determining unit 44 determines a change of the posture of the vehicle V from a periphery image acquired by the image acquiring unit 41. The posture-change determining unit 44 calculates variations of the tilt angle θX1 to the tilt angle θX4 of the periphery imaging camera 21 to the periphery imaging camera 24, variations of the height y1 to the height y4, variations of the rotation angle θZ1 to the rotation angle θZ4 based on multiple periphery images that are acquired by the image acquiring unit 41. The posture-change determining unit 44 then determines whether the posture of the vehicle V has changed based on the variations).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of SUGIE in view of NAGAI and in further view of AKITA of having a control method of a vehicle, the control method comprising: setting, as a template, an area around a vanishing point in a previous frame of an image input from a camera; determining, by a controller, a matching area matching with the template by performing template matching in a current frame, with the teachings of NAGAI of having where a roll slope at which the front camera is mounted is compensated from the change amount in y-axis.
Wherein SUGIE’s method having and determining the change amount in y-axis of the vanishing point based on a change amount in y-axis where a roll slope at which the front camera is mounted is compensated from the change amount in y-axis of the template.
The motivation behind the modification would have been to obtain a method of that improves autonomous driving and camera calibration, since both SUGIE and NAGAI concern vehicles and image analysis. Wherein SUGIE provides systems and methods that improve the bird's-eye view image and the accuracy of the subsequent processes performed based on this image, while NAGAI provides systems and methods that improves the ability to generate a birds-eye image from multiple cameras as well as determine the posture of a vehicle and cameras. Please see SUGIE et al. (US 20200082179 A1), Abstract and Paragraph [0129 and 0150] and NAGAI et al. (US 20180322655 A1), Abstract and Paragraph [0003-0010].
Regarding claim 12, SUGIE in view of NAGAI and in further view of AKITA explicitly teaches the control method of claim 10, SUGIE in view of AKITA is silent on wherein the estimating of the pose of the vehicle includes estimating a pitch pose of the vehicle based on the amount of pitch change of the front camera.
However, NAGAI explicitly teaches wherein the estimating of the pose of the vehicle includes estimating a pitch pose of the vehicle (Fig. 1. Paragraph [0051]-NAGAI discloses the posture-change determining unit 44 determines a change of the posture of the vehicle V from a periphery image acquired by the image acquiring unit 41. More specifically, the posture-change determining unit 44 calculates variations of the tilt angle θX1 to the tilt angle θX4 of the periphery imaging camera 21 to the periphery imaging camera 24, variations of the height y1 to the height y4, variations of the rotation angle θZ1 to the rotation angle θZ4 based on multiple periphery images that are acquired by the image acquiring unit 41. The posture-change determining unit 44 then determines whether the posture of the vehicle V has changed based on the variations of the tilt angle θX1 to the tilt angle θX4 of the periphery imaging camera 21 to the periphery imaging camera 24, the variations of the height y1 to the height y4, the variations of the rotation angle θZ1 to the rotation angle θZ4 based on multiple periphery images that are acquired by the image acquiring unit 41) based on the amount of pitch change of the front camera (Fig. 1, #21 called a forward-periphery imaging camera. Paragraph [0032]-NAGAI discloses the posture of the forward-periphery imaging camera 21 is explained, using FIG. 6. FIG. 6 is a schematic diagram explaining a posture of the forward-periphery imaging camera of the bird's-eye-view-image generating system according to the first embodiment. In a coordinate system of the forward-periphery imaging camera 21, an optical axis direction of the forward-periphery imaging camera 21 is a Z axis, and directions perpendicular to the Z axis are an X axis and a Y axis, and a center of the forward-periphery imaging camera 21 is the origin point of the coordinate system. The posture of the forward-periphery imaging camera 21 is identified by six axes of coordinates (x1, y1, z1) and rotation components about the X axis, the Y axis, and the Z axis (θX1, θY1, θZ1). In the present embodiment, a tilt angle θX1, which is the rotation component about the X axis, a height y1 in the direction of the Y axis indicating a height from the contact ground surface, and a rotation angle θZ1, which is a rotation component about the Z axis vary in the forward-periphery imaging camera 21. Please also read paragraph [0054-0056]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of SUGIE in view of NAGAI and in further view of AKITA of having a control method of a vehicle, the control method comprising: setting, as a template, an area around a vanishing point in a previous frame of an image input from a camera; determining, by a controller, a matching area matching with the template by performing template matching in a current frame, with the teachings of NAGAI of having wherein the estimating of the pose of the vehicle includes estimating a pitch pose of the vehicle based on the amount of pitch change of the front camera.
Wherein SUGIE’s method having wherein the estimating of the pose of the vehicle includes estimating a pitch pose of the vehicle based on the amount of pitch change of the front camera.
The motivation behind the modification would have been to obtain a method of that improves autonomous driving and camera calibration, since both SUGIE and NAGAI concern vehicles and image analysis. Wherein SUGIE provides systems and methods that improve the bird's-eye view image and the accuracy of the subsequent processes performed based on this image, while NAGAI provides systems and methods that improves the ability to generate a birds-eye image from multiple cameras as well as determine the posture of a vehicle and cameras. Please see SUGIE et al. (US 20200082179 A1), Abstract and Paragraph [0129 and 0150] and NAGAI et al. (US 20180322655 A1), Abstract and Paragraph [0003-0010].
Regarding claim 13, SUGIE in view of NAGAI and in further view of AKITA explicitly teaches the control method of claim 1, SUGIE in view of AKITA fail to explicitly teach wherein the camera is a multi-camera configured to obtain image data for a field of view facing a plurality of directions of the vehicle, and wherein the estimating of the change amount in the pose of the camera includes: fusing an amount of position change of a vanishing point corresponding to each camera of the multi-camera, estimating a change amount in pose of each of the cameras of the multi-camera based on the fused amount of position change of the vanishing point, and estimating the pose of the vehicle based on the estimated change amount in the pose of each of the cameras.
However, NAGAI explicitly teaches wherein the camera (Fig. 1, #21, #22, #23, and #24 called a forward-periphery imaging camera, a rearward-periphery imaging camera, a leftward-periphery imaging camera, and a rightward-periphery imaging camera, respectively. Paragraph [0030]) is a multi-camera (Fig. 1, #41 called an image acquiring unit. Paragraph [0046]) configured to obtain image data for a field of view facing a plurality of directions of the vehicle, and wherein the estimating of the change amount in the pose of the camera (Fig. 1. Paragraph [0029]-NAGAI discloses the bird's-eye-view-image generating system 10 includes a forward-periphery imaging camera (imaging device) 21, a rearward-periphery imaging camera (imaging device) 22, a leftward-periphery imaging camera (imaging device) 23, a rightward-periphery imaging camera (imaging device) 24, a storage device 30, the bird's-eye-view-image generating device 40, and a posture-change determining device 50 (wherein the posture of each camera is estimated using six axes of coordinates (x3, y3, z3) and rotation components about the X axis, the Y axis, and the Z axis (θX3, θY3, θZ3)). In paragraph [0046]-NAGAI discloses the bird's-eye-view-image generating device 40 includes the image acquiring unit 41, a vehicle-information acquiring unit 42, and a control unit 43 that includes a posture-change determining unit 44, a bird's-eye-view-image generating unit 45, and the display control unit 46. In paragraph [0044]-NAGAI discloses the storage device 30 stores a periphery image of the latest frame and a periphery image of a next previous frame acquired by the image acquiring unit 41. Please also read paragraph [0032-0038]) includes:
fusing an amount of position change of a vanishing point corresponding to each camera of the multi-camera (Fig. 1. Paragraph [0026]-NAGAI discloses when determining that a posture of the vehicle V has changed, the bird's-eye-view-image generating system 10 generates the bird's-eye view image 300 corrected according to the change in the posture of the vehicle V. In paragraph [0054]-NAGAI discloses the bird's-eye view image 300 includes the virtual own-vehicle image A and at least one of a front image 301, a rear image 302, a left side image 303, and a right side image 304. The bird's-eye view image 300 is generated in a rectangular shape. The bird's-eye view image 300 includes at least one of a first region F1 showing the front image 301, a second region F2 showing the rear image 302, a third region F3 showing the left side image 303, and a fourth region F4 showing the right side image 304. the bird's-eye view image 300 includes the first region F1, the second region F2, the third region F3, and the fourth region F4. Please also read paragraph [0045 and 0063]), estimating a change amount in pose of each of the cameras of the multi-camera based on the fused amount of position change of the vanishing point (Fig. 1. Paragraph [0045]-NAGAI discloses the storage device 30 associates a difference in a height direction (height direction variation) of a vanishing point P in two periphery imagers acquired by the forward-periphery imaging camera 21 and a difference in the tilt angle θX1 (variation of the tilt angle θX1) of the forward-periphery imaging camera 21 with each other, to store as tilt-angle variation data. The storage device 30 stores a difference in a height direction of a vanishing point P in two periphery imagers acquired by the rearward-periphery imaging camera 22 and a difference in the tilt angle θX2 (variation of the tilt angle θX2) of the rearward-periphery imaging camera 22 with each other, to store as tilt-angle variation data. The storage device 30 stores a difference in a height direction of a vanishing point P in two periphery imagers acquired by the leftward-periphery imaging camera 23 and a difference in the tilt angle θX3 (variation of the tilt angle θX3) of the leftward-periphery imaging camera 23 with each other, to store as tilt-angle variation data. The storage device 30 stores a difference in a height direction of a vanishing point P in two periphery imagers acquired by the rightward-periphery imaging camera 24 and a difference in the tilt angle θX4 (variation of the tilt angle θX4) of the rightward-periphery imaging camera 24 with each other, to store as tilt-angle variation data. Please also read paragraph [0063]), and estimating the pose of the vehicle based on the estimated change amount in the pose of each of the cameras (Fig. 1. Paragraph [0055]-NAGAI discloses as correction processing, the bird's-eye-view-image generating unit 45 corrects, when a determination result by the posture-change determining unit 44 indicates that the posture of at least either one of the periphery imaging camera 21 to the periphery imaging camera 24 has changed, the periphery images acquired by the image acquiring unit 41 according to the change of the posture of the periphery imaging camera 21 to the periphery imaging camera 24, and performs the eye point conversion so that an image looking the vehicle V down from above is obtained, and superimposes the virtual own-vehicle image A looking the vehicle V down from above thereon, to generate the bird's-eye view image 300. The bird's-eye-view-image generating unit 45 outputs the generated bird's-eye view image 300 to the display control unit 46. Please also read paragraph [0051-0054]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of SUGIE in view of NAGAI and in further view of AKITA of having a control method of a vehicle, the control method comprising: setting, as a template, an area around a vanishing point in a previous frame of an image input from a camera; determining, by a controller, a matching area matching with the template by performing template matching in a current frame, with the teachings of NAGAI of having the control method of wherein the camera is a multi-camera configured to obtain image data for a field of view facing a plurality of directions of the vehicle, and wherein the estimating of the change amount in the pose of the camera includes: fusing an amount of position change corresponding to each camera of the multi-camera, estimating a change amount in pose of each of the cameras of the multi-camera based on the fused amount of position change, and estimating the pose of the vehicle based on the estimated change amount in the pose of each of the cameras.
Wherein SUGIE’s method having the control method of wherein the camera is a multi-camera configured to obtain image data for a field of view facing a plurality of directions of the vehicle, and wherein the estimating of the change amount in the pose of the camera includes: fusing an amount of position change of a vanishing point corresponding to each camera of the multi-camera, estimating a change amount in pose of each of the cameras of the multi-camera based on the fused amount of position change of the vanishing point, and estimating the pose of the vehicle based on the estimated change amount in the pose of each of the cameras.
The motivation behind the modification would have been to obtain a method of that improves autonomous driving and camera calibration, since both SUGIE and NAGAI concern vehicles and image analysis. Wherein SUGIE provides systems and methods that improve the bird's-eye view image and the accuracy of the subsequent processes performed based on this image, while NAGAI provides systems and methods that improves the ability to generate a birds-eye image from multiple cameras as well as determine the posture of a vehicle and cameras. Please see SUGIE et al. (US 20200082179 A1), Abstract and Paragraph [0129 and 0150] and NAGAI et al. (US 20180322655 A1), Abstract and Paragraph [0003-0010].
Regarding claim 14, SUGIE in view of NAGAI and in further view of AKITA explicitly teach the control method of claim 13, SUGIE in view of AKITA is silent on wherein the estimating of the pose of the vehicle (Fig. 1, #V called a vehicle. Paragraph [0051]-NAGAI discloses the posture-change determining unit 44 then determines whether the posture of the vehicle V has changed based on the variations of the tilt angle θX1 to the tilt angle θX4 of the periphery imaging camera 21 to the periphery imaging camera 24, the variations of the height y1 to the height y4, the variations of the rotation angle θZ1 to the rotation angle θZ4 based on multiple periphery images that are acquired by the image acquiring unit 41. The posture-change determining unit 44 calculates variations of the tilt angle θX1 to the tilt angle θX4 of the periphery imaging camera 21 to the periphery imaging camera 24, variations of the height y1 to the height y4, variations of the rotation angle θZ1 to the rotation angle θZ4 based on multiple periphery images that are acquired by the image acquiring unit 41) includes estimating the pose of the vehicle as at least one of rolling, pitching, yawing, height, or going straight (Fig. 6. Paragraph [0032]-NAGAI discloses FIG. 6 is a schematic diagram explaining a posture of the forward-periphery imaging camera of the bird's-eye-view-image generating system. In a coordinate system of the forward-periphery imaging camera 21, an optical axis direction of the forward-periphery imaging camera 21 is a Z axis, and directions perpendicular to the Z axis are an X axis and a Y axis, and a center of the forward-periphery imaging camera 21 is the origin point of the coordinate system. The posture of the forward-periphery imaging camera 21 is identified by six axes of coordinates (x1, y1, z1) and rotation components about the X axis, the Y axis, and the Z axis (θX1, θY1, θZ1). A tilt angle θX1, which is the rotation component about the X axis, a height y1 in the direction of the Y axis indicating a height from the contact ground surface, and a rotation angle θZ1, which is a rotation component about the Z axis vary in the forward-periphery imaging camera 21 Please also read paragraph [0035, 0037 and 0041]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of SUGIE in view of NAGAI and in further view of AKITA of having a control method of a vehicle, the control method comprising: setting, as a template, an area around a vanishing point in a previous frame of an image input from a camera; determining, by a controller, a matching area matching with the template by performing template matching in a current frame, with the teachings of NAGAI of having wherein the estimating of the pose of the vehicle includes estimating the pose of the vehicle as at least one of rolling, pitching, yawing, height, or going straight.
Wherein SUGIE’s method having wherein the estimating of the pose of the vehicle includes estimating the pose of the vehicle as at least one of rolling, pitching, yawing, height, or going straight.
The motivation behind the modification would have been to obtain a method of that improves autonomous driving and camera calibration, since both SUGIE and NAGAI concern vehicles and image analysis. Wherein SUGIE provides systems and methods that improve the bird's-eye view image and the accuracy of the subsequent processes performed based on this image, while NAGAI provides systems and methods that improves the ability to generate a birds-eye image from multiple cameras as well as determine the posture of a vehicle and cameras. Please see SUGIE et al. (US 20200082179 A1), Abstract and Paragraph [0129 and 0150] and NAGAI et al. (US 20180322655 A1), Abstract and Paragraph [0003-0010].
Regarding claim 15, SUGIE explicitly teaches a vehicle (Fig. 24, #1 called a vehicle. Paragraph [0058]-SUGIE discloses the road map information update system 100 includes a camera 110, a parameter storage portion 120, a GPS receiver 130, a behavior detection sensor 140, a map storage portion 150, a computation portion 160, and an accumulated data storage portion 170. The road map information update system 100 is mounted on a vehicle 1. In paragraph [0066]-SUGIE discloses the computation portion 160 functions as a vehicle position detection section 161, an image processing section 162, a collation region identification section 163, and a collation processing section 164. When these functions are performed, a method corresponding to the program is performed. Please also see Fig. 28-32), comprising:
a camera (Fig. 24, #110 called a camera. Paragraph [0058]) configured to photograph an area around of the vehicle (Fig. 3. Paragraph [0059]-SUGIE discloses the camera 110 is installed in a direction so that the area in front of the vehicle 1 can be captured, and captures an image of the area in front of the vehicle 1 in the surrounding areas of the vehicle 1. In the following description, an image captured by the camera 110 is referred to as a forward view image 180. FIG. 3 shows an example of the forward view image 180); and
a controller electrically connected to the camera, wherein the controller (Fig. 24, #260 called a computation portion. Paragraph [0066]-SUGIE discloses the computation portion 160 is a computer provided with a CPU, a ROM, a RAM, and the like. The CPU executes a program stored in a non-transitory tangible storage medium such as a ROM, while using a temporary storage function of the RAM. In paragraph [0067]-SUGIE discloses a part or all of the functional blocks included in the computation portion 160 may be achieved by a combination of software execution by the CPU and hardware members. Please also read paragraph [0131]) is configured to:
set, as a template (Fig. 30, #386 called a vanishing point region. Paragraph [0141]), an area around a vanishing point in a previous frame of an image input from the camera (Fig. 30. Paragraph (Fig. 28. Paragraph [0140]-SUGIE discloses in S41 in FIG. 29, a vanishing point region image 386 is determined. The vanishing point region image 386 is an image of a region determined on the basis of the vanishing point in the corrected forward view image 181. The vanishing point region images 386 shown in FIGS. 30 and 31 have a horizontally-long rectangular shape which has a pair of short sides extending in the vertical direction and a pair of long sides extending in the horizontal direction with the vanishing point 183 (see FIGS. 11A and 11B) as a center. Please also see Fig. 28-29 and read paragraph [0141-0143]). In paragraph [0141]-SUGIE discloses FIG. 30 shows the vanishing point region image 386 obtained in the previous frame),
determine a matching area matching with the template by performing template matching in a current frame ((Fig. 29. Paragraph [0141]-SUGIE discloses in S42, a matching process is performed for matching the vanishing point region image 386 obtained in the previous frame, that is, the vanishing point region image 386 determined in the previous execution of the process in FIG. 28, and the vanishing point region image 386 obtained in S41 performed immediately before S42. FIG. 30 shows the vanishing point region image 386 obtained in the previous frame, and FIG. 31 shows the vanishing point region image 386 obtained in the current frame. Please also see Fig. 28 and read paragraph [0142-0143]),
determine an amount of position change of the vanishing point based on an amount of position change between the template and the matching area (Fig. 31. Paragraph [0142]-SUGIE discloses in S43, the amount of movement of the vanishing point region image 386 obtained in the current frame in S42 is calculated. The amount of movement is used as a value that indicates a pitch variation occurring in the vehicle 1 within one frame. S41 to S43 correspond to an amount of movement calculation section. In paragraph [0143]-SUGIE discloses in S44, the amount of movement calculated in S43 is sequentially integrated in the processes, shown in FIGS. 28 and 29, which are repeatedly executed each time the forward view image 180 is input. The integrated value is referred to as a positional deviation integration value. The positional deviation integration value includes the positional deviation of the vanishing point region image 386 due to the pitch variation and a change in the position of the vanishing point region image 386 due to static reasons other than the pitch variation. The static reasons other than the pitch variation include, for example, a change in the optical axis of the camera due to the vehicle 1 traveling on an uphill. S44 corresponds to an integration value calculation section. Please also see Fig. 28),
estimate a change amount in a pose of the camera based on the amount of position change of the vanishing point (Fig. 29. Paragraph [0146]-SUGIE discloses after the process in S46 is executed, the process flow proceeds to S5A in FIG. 28. In S5A, the orientation of the camera 110 is detected. Specifically, the orientation of the camera 110 is indicated by the yaw angle rx and the tilt angle ry. The yaw angle rx is obtained from Equation 4. The tilt angle ry is obtained from Equation 10. As shown in Equation 10, the yaw angle rx is calculated from the distance Δy between the image center 184 and the vanishing point 183 in the y direction and the amount of pitch variation δy. In paragraph [0147]-SUGIE discloses S6 to S9 are executed to output the bird's-eye view image. In paragraph [0148]-SUGIE discloses the bird's-eye view image subjected to pitch correction is the bird's-eye view image obtained by performing the process in FIG. 28. Please also see Fig. 5-6, 11A-B and 28),
and wherein the controller (Fig. 24, #260 called a computation portion. Paragraph [0066 and 0131]) is further configured to:
determine a reliability of the template based on a variance value of the template (Fig. 29. Paragraph [0143]-SUGIE discloses the positional deviation integration value includes the positional deviation of the vanishing point region image 386 due to the pitch variation and a change in the position of the vanishing point region image 386 due to static reasons other than the pitch variation. The static reasons other than the pitch variation include, for example, a change in the optical axis of the camera due to the vehicle 1 traveling on an uphill. In paragraph [0145]-SUGIE discloses in S46, the difference between the positional deviation integration value calculated in S44 and the moving average value calculated in S45 is calculated. This difference is referred to as an amount of pitch variation δy. The amount of pitch variation δy may be calculated from either the equation of (positional deviation integration value)−(moving average value) or the equation of (moving average value)−(positional deviation integration value). FIG. 32 shows the positional deviation integration value, the moving average value, and the amount of pitch variation δy. Please also see Fig. 23 and read paragraph [0119-0124, 0129 and 0144]);
and wherein the controller (Fig. 24, #260 called a computation portion. Paragraph [0066 and 0131]) is configured to:
estimate an amount of pitch change of the camera (Fig. 28. Paragraph [0146]-SUGIE discloses after the process in S46 is executed, the process flow proceeds to S5A in FIG. 28. In S5A, the orientation of the camera 110 is detected. Specifically, the orientation of the camera 110 is indicated by the yaw angle rx and the tilt angle ry. The yaw angle rx is obtained from Equation 4. The tilt angle ry is obtained from Equation 10. As shown in Equation 10, the yaw angle rx is calculated from the distance Δy between the image center 184 and the vanishing point 183 in the y direction and the amount of pitch variation δy) based on a change amount in v-axis of the vanishing point (Fig. 28. Paragraph [0142]-SUGIE discloses in S43, the amount of movement of the vanishing point region image 386 obtained in the current frame in S42 is calculated. The amount of movement is used as a value that indicates a pitch variation occurring in the vehicle 1 within one frame. In paragraph [0143]-SUGIE discloses in S44, the amount of movement calculated in S43 is sequentially integrated in the processes, shown in FIGS. 28 and 29, which are repeatedly executed each time the forward view image 180 is input. The integrated value is referred to as a positional deviation integration value. The positional deviation integration value includes the positional deviation of the vanishing point region image 386 due to the pitch variation and a change in the position of the vanishing point region image 386 due to static reasons other than the pitch variation. The static reasons other than the pitch variation include, for example, a change in the optical axis of the camera due to the vehicle 1 traveling on an uphill).
SUGIE fail to teach and estimate a pose of the vehicle depending on the change amount in pose of the camera.
However, NAGAI explicitly teaches and estimate a pose of the vehicle (Fig. 1, #V called a vehicle. Paragraph [0051]-NAGAI discloses the posture-change determining unit 44 then determines whether the posture of the vehicle V has changed based on the variations of the tilt angle θX1 to the tilt angle θX4 of the periphery imaging camera 21 to the periphery imaging camera 24, the variations of the height y1 to the height y4, the variations of the rotation angle θZ1 to the rotation angle θZ4 based on multiple periphery images that are acquired by the image acquiring unit 41. In paragraph [0052]-NAGAI discloses the bird's-eye-view-image generating unit 45 generates the bird's-eye view image 300 by subjecting the periphery images acquired by the image acquiring unit 41 to the eye point conversion so that an image looking the vehicle V down from above is obtained, and by superimposing the virtual own-vehicle image A looking the vehicle V from above) depending on the change amount in pose (Fig. 1. Paragraph [0032]-NAGAI discloses the posture of the forward-periphery imaging camera 21 is identified by six axes of coordinates (x1, y1, z1) and rotation components about the X axis, the Y axis, and the Z axis (θX1, θY1, θZ1). In paragraph [0044]-NAGAI discloses the storage device 30 stores a periphery image of the latest frame and a periphery image of a next previous frame acquired by the image acquiring unit 41. In paragraph [0045]-NAGAI discloses the storage device 30 associates a difference in a height direction (height direction variation) of a vanishing point P in two periphery imagers acquired by the forward-periphery imaging camera 21 and a difference in the tilt angle θX1 (variation of the tilt angle θX1) of the forward-periphery imaging camera 21 with each other, to store as tilt-angle variation data (wherein this information/process and posture/pose is similarly determined for each rearward, rightward and leftward camera, and the change in pose of each camera is based on a change in the y-axis of a vanishing point in each camera). Please also see Fig. 8-9 and read paragraph [0063]) of the camera (Fig. 1, #21, #22, #23, and #24 called a forward-periphery imaging camera, a rearward-periphery imaging camera, a leftward-periphery imaging camera, and a rightward-periphery imaging camera, respectively. Paragraph [0029 and 0046]. In paragraph [0051]-NAGAI discloses the posture-change determining unit 44 calculates variations of the tilt angle θX1 to the tilt angle θX4 of the periphery imaging camera 21 to the periphery imaging camera 24, variations of the height y1 to the height y4, variations of the rotation angle θZ1 to the rotation angle θZ4 based on multiple periphery images that are acquired by the image acquiring unit 41 (wherein the posture/pose of each camera is estimated using six axes of coordinates (x3, y3, z3) and rotation components about the X axis, the Y axis, and the Z axis (θX3, θY3, θZ3), and the change in pose of each camera is based on a change in the y-axis of a vanishing point in each camera). Please also read paragraph [0054-0056]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of SUGIE of having a vehicle, comprising: set, as a template, an area around a vanishing point in a previous frame of an image input from the camera, determine a matching area matching with the template by performing template matching in a current frame, determine an amount of position change of the vanishing point based on an amount of position change between the template and the matching area, estimate a change amount in a pose of the camera based on the amount of position change of the vanishing point, and wherein the controller is further configured to: determine a reliability of the template based on a variance value of the template; and wherein the controller is configured to: estimate an amount of pitch change of the camera based on a change amount in v-axis of the vanishing point, with the teachings of NAGAI of having and estimate a pose of the vehicle depending on the change amount in pose of the camera.
Wherein SUGIE’s vehicle having and estimate a pose of the vehicle depending on the change amount in pose of the camera.
The motivation behind the modification would have been to obtain a vehicle that improves autonomous driving and camera calibration, since both SUGIE and NAGAI concern vehicles and image analysis. Wherein SUGIE provides systems and methods that improve the bird's-eye view image and the accuracy of the subsequent processes performed based on this image, while NAGAI provides systems and methods that improves the ability to generate a birds-eye image from multiple cameras as well as determine the posture of a vehicle and cameras. Please see SUGIE et al. (US 20200082179 A1), Abstract and Paragraph [0129 and 0150] and NAGAI et al. (US 20180322655 A1), Abstract and Paragraph [0003-0010].
SUGIE in view of NAGAI fails to explicitly teach change a position of the template, based on the reliability of the template being low.
However, AKITA explicitly teaches change a position of the template (Fig. 6. Paragraph [0115]-AKITA discloses optical flows of the single moving object 40 are extended. These extended straight lines L intersect at one point, which is the focus of expansion: FOE, which is defined as the so-called point at infinity or the vanishing point. Each point feature of the moving object 40 which translates in the three-dimensional space has a common focus of expansion FOE in the image plane. In paragraph [0142]-AKITA discloses the optical flow calculation unit 4 sets a template around the feature point obtained as above and searches for an area having high correlativity with this template, from the subsequent image in the time series. (template matching) As the result of the search, its movement vector is obtained as a motion vector (optical flow). As a method of such template matching, a normalized cross correlation technique (NCC) can be used (wherein template matching may use sequential similarity search or least squares)), based on the reliability of the template being low (Fig. 6. Paragraph [0143]-AKITA discloses a template image T (i, j) having (M.sub.T.times.N.sub.T) pixels is moved on the pixels present within a search area, thereby to search a point in the template image which point has the maximum correlativity coefficient (correlativity value) R.sub.NCC (a, b) in the following Formula (24). In paragraph [0144]-AKITA discloses (a, b) represents the position of the template image T within the image in the search area and I.sub.(a, b) (i, j) represents a partial image within the image in the search area. In paragraph [0147]-AKITA discloses it is possible to restrict the search area in advance in the image processing area setting step, according to the position of the moving object recognized in the previous flow (wherein various reliability values and thresholds may be used to adjust recognition results such as recognition reliability, feature amount reliability, optical flow reliability, grouping reliability, fused reliability, continuity reliability, moving object determination reliability and/or secondary intermediate reliability). Please also read paragraph [0150-0151, 0177-0189, 0204-0207, and 0209]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of SUGIE in view of NAGAI of having a vehicle, comprising: set, as a template, an area around a vanishing point in a previous frame of an image input from the camera, determine a matching area matching with the template by performing template matching in a current frame, determine an amount of position change of the vanishing point based on an amount of position change between the template and the matching area, with the teachings of AKITA of having and change a position of the template, based on the reliability of based on the reliability of the template being low.
Wherein SUGIE’s vehicle having and change a position of the template, based on the reliability of based on the reliability of the template being low.
The motivation behind the modification would have been to obtain a vehicle that improves efficiency and accuracy of image generation and calibration, since both SUGIE and AKITA concern vehicles, image analysis and vanishing point detection. Wherein SUGIE provides systems and methods that improve the bird's-eye view image and the accuracy of the subsequent processes performed based on this image, while AKITA provides systems and methods that improves the safety in vehicle driving. Please see SUGIE et al. (US 20200082179 A1), Abstract and Paragraph [0129 and 0150] and AKITA et al. (US 20100027844 A1), Abstract and Paragraph [0071].
Regarding claim 16, SUGIE in view of NAGAI and in further view of AKITA explicitly teach the vehicle of claim 15, SUGIE in view of NAGAI fail to explicitly teach wherein the controller is configured to change a position of the template based on the variance value of the template, and wherein the variance value of the template is determined based on a degree of contrast of the image.
However, AKITA explicitly teach wherein the controller is configured to change a position of the template based on the variance value of the template (Fig. 6. Paragraph [0115]-AKITA discloses optical flows of the single moving object 40 are extended. These extended straight lines L intersect at one point, which is the focus of expansion: FOE, which is defined as the so-called point at infinity or the vanishing point. Each point feature of the moving object 40 which translates in the three-dimensional space has a common focus of expansion FOE in the image plane. In paragraph [0142]-AKITA discloses the optical flow calculation unit 4 sets a template around the feature point obtained as above and searches for an area having high correlativity with this template, from the subsequent image in the time series. (template matching) As the result of the search, its movement vector is obtained as a motion vector (optical flow). As a method of such template matching, a normalized cross correlation technique (NCC) can be used (wherein template matching may use sequential similarity search or least squares)), and wherein the variance value of the template is determined based on a degree of contrast of the image (Fig. 6. Paragraph [0143]-AKITA discloses a template image T (i, j) having (M.sub.T.times.N.sub.T) pixels is moved on the pixels present within a search area, thereby to search a point in the template image which point has the maximum correlativity coefficient (correlativity value) R.sub.NCC (a, b) in the following Formula (24). In paragraph [0144]-AKITA discloses (a, b) represents the position of the template image T within the image in the search area and I.sub.(a, b) (i, j) represents a partial image within the image in the search area. In paragraph [0147]-AKITA discloses it is possible to restrict the search area in advance in the image processing area setting step, according to the position of the moving object recognized in the previous flow (wherein various reliability values and thresholds may be used to adjust recognition results such as recognition reliability, feature amount reliability, optical flow reliability, grouping reliability, fused reliability, continuity reliability, moving object determination reliability and/or secondary intermediate reliability). Please also read paragraph [0009 and 0030]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of SUGIE in view of NAGAI and in further view of AKITA of having a vehicle, comprising: set, as a template, an area around a vanishing point in a previous frame of an image input from the camera, determine a matching area matching with the template by performing template matching in a current frame, determine an amount of position change of the vanishing point based on an amount of position change between the template and the matching area, with the teachings of AKITA of having wherein the controller is configured to change a position of the template based on the variance value of the template, and wherein the variance value of the template is determined based on a degree of contrast of the image.
Wherein SUGIE’s vehicle having wherein the controller is configured to change a position of the template based on the variance value of the template, and wherein the variance value of the template is determined based on a degree of contrast of the image.
The motivation behind the modification would have been to obtain a vehicle that improves efficiency and accuracy of image generation and calibration, since both SUGIE and AKITA concern vehicles, image analysis and vanishing point detection. Wherein SUGIE provides systems and methods that improve the bird's-eye view image and the accuracy of the subsequent processes performed based on this image, while AKITA provides systems and methods that improves the safety in vehicle driving. Please see SUGIE et al. (US 20200082179 A1), Abstract and Paragraph [0129 and 0150] and AKITA et al. (US 20100027844 A1), Abstract and Paragraph [0071].
Regarding claim 18, SUGIE in view of NAGAI and in further view of AKITA explicitly teach the vehicle of claim 15, although SUGIE further teaches the vehicle of wherein the camera is a front camera (Fig. 24, #110 called a camera. Paragraph [0058]) configured to obtain image data for a field of view facing a front of the vehicle (Fig. 1. Paragraph [0059]-SUGIE discloses the camera 110 is installed in a direction so that the area in front of the vehicle 1 can be captured, and captures an image of the area in front of the vehicle 1 in the surrounding areas of the vehicle 1. In the following description, an image captured by the camera 110 is referred to as a forward view image 180. FIG. 3 shows an example of the forward view image 180), and wherein the controller Fig. 24, #260 called a computation portion. Paragraph [0066 and 0131]) is configured to:
determine a change amount in y-axis of the template based on the amount of position change between the template and the matching area (Fig. 31. Paragraph [0142]-SUGIE discloses in S43, the amount of movement of the vanishing point region image 386 obtained in the current frame in S42 is calculated. The amount of movement is used as a value that indicates a pitch variation occurring in the vehicle 1 within one frame. S41 to S43 correspond to an amount of movement calculation section. In paragraph [0143]-SUGIE discloses in S44, the amount of movement calculated in S43 is sequentially integrated in the processes, shown in FIGS. 28 and 29, which are repeatedly executed each time the forward view image 180 is input. The integrated value is referred to as a positional deviation integration value. The positional deviation integration value includes the positional deviation of the vanishing point region image 386 due to the pitch variation and a change in the position of the vanishing point region image 386 due to static reasons other than the pitch variation. The static reasons other than the pitch variation include, for example, a change in the optical axis of the camera due to the vehicle 1 traveling on an uphill. S44 corresponds to an integration value calculation section. Please also see Fig. 28), estimate an amount of pitch change of the front camera based on the change amount in y-axis of the vanishing point (Fig. 28. Paragraph [0146]-SUGIE discloses after the process in S46 is executed, the process flow proceeds to S5A in FIG. 28. In S5A, the orientation of the camera 110 is detected. Specifically, the orientation of the camera 110 is indicated by the yaw angle rx and the tilt angle ry. The yaw angle rx is obtained from Equation 4. The tilt angle ry is obtained from Equation 10. As shown in Equation 10, the yaw angle rx is calculated from the distance Δy between the image center 184 and the vanishing point 183 in the y direction and the amount of pitch variation δy), determine the change amount in y-axis of the vanishing point based on a change amount in y-axis where a slope at which the front camera is mounted is compensated from the change amount in y-axis of the template (Fig. 28. Paragraph [0147]-SUGIE discloses the processes after the execution of S5A are the same as those of the first embodiment. S6 to S9 are executed to output the bird's-eye view image to be used for collation process and the current position to the collation region identification section 163. In paragraph [0149]-SUGIE discloses FIG. 34 shows a bird's-eye view image subjected to pitch correction. The bird's-eye view image subjected to pitch correction is the bird's-eye view image obtained by performing the process in FIG. 28).
SUGIE in view of AKITA fail to teach where a roll slope at which the front camera is mounted is compensated from the change amount in y-axis, and estimate a pitch pose of the vehicle based on the amount of pitch change of the front camera.
However, NAGAI explicitly teaches where a roll slope at which the front camera (Fig. 1, #21 called a forward-periphery imaging camera. Paragraph [0032]-NAGAI discloses the posture of the forward-periphery imaging camera 21 is explained, using FIG. 6. FIG. 6 is a schematic diagram explaining a posture of the forward-periphery imaging camera of the bird's-eye-view-image generating system according to the first embodiment. In a coordinate system of the forward-periphery imaging camera 21, an optical axis direction of the forward-periphery imaging camera 21 is a Z axis, and directions perpendicular to the Z axis are an X axis and a Y axis, and a center of the forward-periphery imaging camera 21 is the origin point of the coordinate system. The posture of the forward-periphery imaging camera 21 is identified by six axes of coordinates (x1, y1, z1) and rotation components about the X axis, the Y axis, and the Z axis (θX1, θY1, θZ1). A tilt angle θX1, which is the rotation component about the X axis, a height y1 in the direction of the Y axis indicating a height from the contact ground surface, and a rotation angle θZ1, which is a rotation component about the Z axis vary in the forward-periphery imaging camera 21. Please also read paragraph [0054-0056]) is mounted is compensated from the change amount in y-axis (Fig. 1. Paragraph [0044]-NAGAI discloses the storage device 30 stores a periphery image of the latest frame and a periphery image of a next previous frame acquired by the image acquiring unit 41. In paragraph [0045]-NAGAI discloses the storage device 30 associates a difference in a height direction (height direction variation) of a vanishing point P in two periphery imagers acquired by the forward-periphery imaging camera 21 and a difference in the tilt angle θX1 (variation of the tilt angle θX1) of the forward-periphery imaging camera 21 with each other, to store as tilt-angle variation data (wherein the height direction is in the direction of the y-axis)), and estimate a pitch pose of the vehicle based on the amount of pitch change of the front camera (Fig. 1. Paragraph [0051]-NAGAI discloses the posture-change determining unit 44 determines a change of the posture of the vehicle V from a periphery image acquired by the image acquiring unit 41. The posture-change determining unit 44 calculates variations of the tilt angle θX1 to the tilt angle θX4 of the periphery imaging camera 21 to the periphery imaging camera 24, variations of the height y1 to the height y4, variations of the rotation angle θZ1 to the rotation angle θZ4 based on multiple periphery images that are acquired by the image acquiring unit 41. The posture-change determining unit 44 then determines whether the posture of the vehicle V has changed based on the variations of the tilt angle θX1 to the tilt angle θX4 of the periphery imaging camera 21 to the periphery imaging camera 24, the variations of the height y1 to the height y4, the variations of the rotation angle θZ1 to the rotation angle θZ4 based on multiple periphery images that are acquired by the image acquiring unit 41).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of SUGIE in view of NAGAI and in further view of AKITA of having a vehicle, comprising: set, as a template, an area around a vanishing point in a previous frame of an image input from the camera, determine a matching area matching with the template by performing template matching in a current frame, determine an amount of position change of the vanishing point based on an amount of position change between the template and the matching area, with the teachings of NAGAI of having determine the change amount in y-axis of the vanishing point based on a change amount in y-axis where a roll slope at which the front camera is mounted is compensated from the change amount in y-axis of the template, and estimate a pitch pose of the vehicle based on the amount of pitch change of the front camera.
Wherein SUGIE’s vehicle having determine the change amount in y-axis of the vanishing point based on a change amount in y-axis where a roll slope at which the front camera is mounted is compensated from the change amount in y-axis of the template, and estimate a pitch pose of the vehicle based on the amount of pitch change of the front camera.
The motivation behind the modification would have been to obtain a vehicle that improves autonomous driving and camera calibration, since both SUGIE and NAGAI concern vehicles and image analysis. Wherein SUGIE provides systems and methods that improve the bird's-eye view image and the accuracy of the subsequent processes performed based on this image, while NAGAI provides systems and methods that improves the ability to generate a birds-eye image from multiple cameras as well as determine the posture of a vehicle and cameras. Please see SUGIE et al. (US 20200082179 A1), Abstract and Paragraph [0129 and 0150] and NAGAI et al. (US 20180322655 A1), Abstract and Paragraph [0003-0010].
Regarding claim 19, SUGIE in view of NAGAI and in further view of AKITA explicitly teaches the vehicle of claim 15, SUGIE in view of AKITA fails to explicitly teach wherein the camera is a multi-camera configured to obtain image data for a field of view facing a plurality of directions of the vehicle, and wherein the controller is configured to: fuse an amount of position change of a vanishing point corresponding to each camera of the multi-camera, estimate a change amount in pose of each of the cameras of the multi-camera based on the fused amount of position change of the vanishing point, and estimate the pose of the vehicle based on the estimated change amount in the pose of each of the cameras.
However, NAGAI explicitly teaches wherein the camera (Fig. 1, #21, #22, #23, and #24 called a forward-periphery imaging camera, a rearward-periphery imaging camera, a leftward-periphery imaging camera, and a rightward-periphery imaging camera, respectively. Paragraph [0030].) is a multi-camera (Fig. 1, #41 called an image acquiring unit. Paragraph [0046]-NAGAI discloses the bird's-eye-view-image generating device 40 includes the image acquiring unit 41, a vehicle-information acquiring unit 42, and a control unit 43 that includes a posture-change determining unit 44, a bird's-eye-view-image generating unit 45, and the display control unit 46. In paragraph [0044]-NAGAI discloses the storage device 30 stores a periphery image of the latest frame and a periphery image of a next previous frame acquired by the image acquiring unit 41) configured to obtain image data for a field of view facing a plurality of directions of the vehicle (Fig. 1. Paragraph [0029]-NAGAI discloses the bird's-eye-view-image generating system 10 includes a forward-periphery imaging camera (imaging device) 21, a rearward-periphery imaging camera (imaging device) 22, a leftward-periphery imaging camera (imaging device) 23, a rightward-periphery imaging camera (imaging device) 24, a storage device 30, the bird's-eye-view-image generating device 40, and a posture-change determining device 50 (wherein the posture of each camera is estimated using six axes of coordinates (x3, y3, z3) and rotation components about the X axis, the Y axis, and the Z axis (θX3, θY3, θZ3)). In paragraph [0046]-NAGAI discloses the bird's-eye-view-image generating device 40 includes the image acquiring unit 41, a vehicle-information acquiring unit 42, and a control unit 43 that includes a posture-change determining unit 44, a bird's-eye-view-image generating unit 45, and the display control unit 46. In paragraph [0044]-NAGAI discloses the storage device 30 stores a periphery image of the latest frame and a periphery image of a next previous frame acquired by the image acquiring unit 41. Please also read paragraph [0032-0038]), and wherein the controller (Fig. #40 and #50 called a bird's-eye-view-image generating device and a posture-change determining device, respectively. Paragraph [0046]-NAGAI discloses the bird's-eye-view-image generating device 40 is an arithmetic processing unit that is constituted of, for example, a CPU (central processing unit) or the like. The bird's-eye-view-image generating device 40 loads a program that is stored in the storage device 30 into a memory, and executes a command included in the program. The bird's-eye-view-image generating device 40 includes the image acquiring unit 41, a vehicle-information acquiring unit 42, and a control unit 43 that includes a posture-change determining unit 44, a bird's-eye-view-image generating unit 45, and the display control unit 46. Please also read paragraph [0047]) is configured to:
fuse an amount of position change of a vanishing point (Fig. 1. Paragraph [0026]-NAGAI discloses when determining that a posture of the vehicle V has changed, the bird's-eye-view-image generating system 10 generates the bird's-eye view image 300 corrected according to the change in the posture of the vehicle V. In paragraph [0054]-NAGAI discloses the bird's-eye view image 300 includes the virtual own-vehicle image A and at least one of a front image 301, a rear image 302, a left side image 303, and a right side image 304. The bird's-eye view image 300 is generated in a rectangular shape. The bird's-eye view image 300 includes at least one of a first region F1 showing the front image 301, a second region F2 showing the rear image 302, a third region F3 showing the left side image 303, and a fourth region F4 showing the right side image 304. the bird's-eye view image 300 includes the first region F1, the second region F2, the third region F3, and the fourth region F4. Please also read paragraph [0045 and 0063]) corresponding to each camera of the multi-camera, estimate a change amount in pose of each of the cameras of the multi-camera based on the fused amount of position change of the vanishing point (Fig. 1. Paragraph [0045]-NAGAI discloses the storage device 30 associates a difference in a height direction (height direction variation) of a vanishing point P in two periphery imagers acquired by the forward-periphery imaging camera 21 and a difference in the tilt angle θX1 (variation of the tilt angle θX1) of the forward-periphery imaging camera 21 with each other, to store as tilt-angle variation data. The storage device 30 stores a difference in a height direction of a vanishing point P in two periphery imagers acquired by the rearward-periphery imaging camera 22 and a difference in the tilt angle θX2 (variation of the tilt angle θX2) of the rearward-periphery imaging camera 22 with each other, to store as tilt-angle variation data. The storage device 30 stores a difference in a height direction of a vanishing point P in two periphery imagers acquired by the leftward-periphery imaging camera 23 and a difference in the tilt angle θX3 (variation of the tilt angle θX3) of the leftward-periphery imaging camera 23 with each other, to store as tilt-angle variation data. The storage device 30 stores a difference in a height direction of a vanishing point P in two periphery imagers acquired by the rightward-periphery imaging camera 24 and a difference in the tilt angle θX4 (variation of the tilt angle θX4) of the rightward-periphery imaging camera 24 with each other, to store as tilt-angle variation data. Please also read paragraph [0063]), and estimate the pose of the vehicle based on the estimated change amount in the pose of each of the cameras (Fig. 1. Paragraph [0055]-NAGAI discloses as correction processing, the bird's-eye-view-image generating unit 45 corrects, when a determination result by the posture-change determining unit 44 indicates that the posture of at least either one of the periphery imaging camera 21 to the periphery imaging camera 24 has changed, the periphery images acquired by the image acquiring unit 41 according to the change of the posture of the periphery imaging camera 21 to the periphery imaging camera 24, and performs the eye point conversion so that an image looking the vehicle V down from above is obtained, and superimposes the virtual own-vehicle image A looking the vehicle V down from above thereon, to generate the bird's-eye view image 300. The bird's-eye-view-image generating unit 45 outputs the generated bird's-eye view image 300 to the display control unit 46. Please also read paragraph [0051-0054]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of SUGIE in view of NAGAI and in further view of AKITA of having a vehicle, comprising: set, as a template, an area around a vanishing point in a previous frame of an image input from the camera, determine a matching area matching with the template by performing template matching in a current frame, determine an amount of position change of the vanishing point based on an amount of position change between the template and the matching area, with the teachings of NAGAI of having wherein the camera is a multi-camera configured to obtain image data for a field of view facing a plurality of directions of the vehicle, and wherein the controller is configured to: fuse an amount of position change of a vanishing point corresponding to each camera of the multi-camera, estimate a change amount in pose of each of the cameras of the multi-camera based on the fused amount of position change of the vanishing point, and estimate the pose of the vehicle based on the estimated change amount in the pose of each of the cameras.
Wherein CHOI’s vehicle having wherein the camera is a multi-camera configured to obtain image data for a field of view facing a plurality of directions of the vehicle, and wherein the controller is configured to: fuse an amount of position change of a vanishing point corresponding to each camera of the multi-camera, estimate a change amount in pose of each of the cameras of the multi-camera based on the fused amount of position change of the vanishing point, and estimate the pose of the vehicle based on the estimated change amount in the pose of each of the cameras.
The motivation behind the modification would have been to obtain a vehicle that improves autonomous driving and camera calibration, since both SUGIE and NAGAI concern vehicles and image analysis. Wherein SUGIE provides systems and methods that improve the bird's-eye view image and the accuracy of the subsequent processes performed based on this image, while NAGAI provides systems and methods that improves the ability to generate a birds-eye image from multiple cameras as well as determine the posture of a vehicle and cameras. Please see SUGIE et al. (US 20200082179 A1), Abstract and Paragraph [0129 and 0150] and NAGAI et al. (US 20180322655 A1), Abstract and Paragraph [0003-0010].
Regarding claim 20, SUGIE in view of NAGAI and in further view of AKITA explicitly teaches vehicle of claim 19, SUGIE in view of AKITA fail to teach wherein the controller is configured for estimating the pose of the vehicle as at least one of rolling, pitching, yawing, height, or going straight.
However, NAGAI explicitly teaches wherein the controller (Fig. #40 and #50 called a bird's-eye-view-image generating device and a posture-change determining device, respectively. Paragraph [0046]-NAGAI discloses the bird's-eye-view-image generating device 40 is an arithmetic processing unit that is constituted of, for example, a CPU (central processing unit) or the like. The bird's-eye-view-image generating device 40 loads a program that is stored in the storage device 30 into a memory, and executes a command included in the program. The bird's-eye-view-image generating device 40 includes the image acquiring unit 41, a vehicle-information acquiring unit 42, and a control unit 43 that includes a posture-change determining unit 44, a bird's-eye-view-image generating unit 45, and the display control unit 46 (wherein the posture change device is an arithmetic processing unit that is a CPU used to implement part of the functions of the bird's-eye-view-image generating device). Please also read paragraph [0047]) is configured for estimating the pose of the vehicle (Fig. 1, #V called a vehicle. Paragraph [0051]-NAGAI discloses the posture-change determining unit 44 then determines whether the posture of the vehicle V has changed based on the variations of the tilt angle θX1 to the tilt angle θX4 of the periphery imaging camera 21 to the periphery imaging camera 24, the variations of the height y1 to the height y4, the variations of the rotation angle θZ1 to the rotation angle θZ4 based on multiple periphery images that are acquired by the image acquiring unit 41. The posture-change determining unit 44 calculates variations of the tilt angle θX1 to the tilt angle θX4 of the periphery imaging camera 21 to the periphery imaging camera 24, variations of the height y1 to the height y4, variations of the rotation angle θZ1 to the rotation angle θZ4 based on multiple periphery images that are acquired by the image acquiring unit 41) as at least one of rolling, pitching, yawing, height, or going straight (Fig. 6. Paragraph [0032]-NAGAI discloses FIG. 6 is a schematic diagram explaining a posture of the forward-periphery imaging camera of the bird's-eye-view-image generating system. In a coordinate system of the forward-periphery imaging camera 21, an optical axis direction of the forward-periphery imaging camera 21 is a Z axis, and directions perpendicular to the Z axis are an X axis and a Y axis, and a center of the forward-periphery imaging camera 21 is the origin point of the coordinate system. The posture of the forward-periphery imaging camera 21 is identified by six axes of coordinates (x1, y1, z1) and rotation components about the X axis, the Y axis, and the Z axis (θX1, θY1, θZ1). A tilt angle θX1, which is the rotation component about the X axis, a height y1 in the direction of the Y axis indicating a height from the contact ground surface, and a rotation angle θZ1, which is a rotation component about the Z axis vary in the forward-periphery imaging camera 21 Please also read paragraph [0035, 0037 and 0041]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of SUGIE in view of NAGAI and in further view of AKITA of having a vehicle, comprising: set, as a template, an area around a vanishing point in a previous frame of an image input from the camera, determine a matching area matching with the template by performing template matching in a current frame, determine an amount of position change of the vanishing point based on an amount of position change between the template and the matching area, with the teachings of NAGAI of having wherein the controller is configured for estimating the pose of the vehicle as at least one of rolling, pitching, yawing, height, or going straight.
Wherein SUGIE’s vehicle having wherein the controller is configured for estimating the pose of the vehicle as at least one of rolling, pitching, yawing, height, or going straight.
The motivation behind the modification would have been to obtain a vehicle that improves autonomous driving and camera calibration, since both SUGIE and NAGAI concern vehicles and image analysis. Wherein SUGIE provides systems and methods that improve the bird's-eye view image and the accuracy of the subsequent processes performed based on this image, while NAGAI provides systems and methods that improves the ability to generate a birds-eye image from multiple cameras as well as determine the posture of a vehicle and cameras. Please see SUGIE et al. (US 20200082179 A1), Abstract and Paragraph [0129 and 0150] and NAGAI et al. (US 20180322655 A1), Abstract and Paragraph [0003-0010].
Claims 4-5 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over SUGIE et al. (US 20200082179 A1), hereinafter referenced as SUGIE in view of NAGAI et al. (US 20180322655 A1, hereinafter referenced as NAGAI and in further view of AKITA et al. (US 20100027844 A1), hereinafter referenced as AKITA and in further view of GOMEZ et al. (US 20170163863 A1), hereinafter referenced as GOMEZ.
Regarding claim 4, SUGIE in view of NAGAI and in further view of AKITA explicitly teaches the control method of claim 2, SUGIE in view of NAGAI fail to explicitly teach wherein the changing of the position of the template includes moving the template according to a slope of a horizontal line based on a roll angle at which the camera is mounted.
However, GOMEZ explicitly teaches wherein the changing of the position of the template includes moving the template (Fig. 1. Paragraph [0029]-GOMEZ discloses FIG. 1 shows a schematic view of a rear vision system 1 installed in a vehicle. The system 1 comprises a camera 2 configured to capture an image 3, and a system controller 6 configured to display a first or a second area of the captured image 3 depending the vehicle driving and/or user information (wherein driving/user information includes, for example, increase/decrease in speed, a change in pitch angle, a change in roll angle by comparing representative parameters of actual and former captured images 3). In paragraph [0031]-GOMEZ discloses the system controller 6 is adapted for displaying a first area 5 of the captured image 3, and also for receiving and processing vehicle driving and/or user information for selecting a second area 8′-8″″′ of the captured image 3 depending on the received information. In paragraph [0063]-GOMEZ discloses the system controller 6 is adapted for performing road monitoring based on road monitoring information, and selecting as second area 8″″′ an area of the first area 5 that is related with the road vanishing point of the image. The system controller 6 may be adapted for selecting as second area 8″″′ a predetermined image that may be related with the road vanishing point. This predetermined image may be the initial image, i.e. the first area 5) according to a slope of a horizontal line (Fig. 1. Paragraph [0055]-GOMEZ discloses the system controller 6 is adapted for determining a relative rightward and leftward rotation of the system 1 from the change in the roll angle, selecting as second area 8″″ an area corresponding to a rightward rotation of the first area 5 such that the field of view is laterally rotated in a clockwise direction when the system 1 is rightwardly rotated about an horizontal axis, and selecting as second area 8″″ an area corresponding to an leftward rotation of the first area 5 such that the field of view is laterally rotated in a counterclockwise direction when the system 1 is leftwardly rotated about an horizontal axis) based on a roll angle (Fig. 1. Paragraph [0056]-GOMEZ discloses FIGS. 8 and 9 show the different areas that are displayed when the controller receives a change in the pitch angle value. In paragraph [0057]-GOMEZ discloses the system 1 comprises a roll angle sensor in communication with the controller 6 and operatively configured to obtain a roll angle value. The roll angle value may be obtained by image processing, comparing the actual captured image 3 with the former captured image 3. In paragraph [0058]-GOMEZ discloses the system controller 6 is configured to change from displaying the first area 5 to display a second area 8″″ corresponding to a rightward rotation (clockwise direction) of the first area 5, when the system 1 (and vehicle) is rightward and horizontally moved according to the roll sensor information. Please also read paragraph [0060]) at which the camera (Fig. 1, #2 called a camera. Paragraph [0029]-GOMEZ discloses the system 1 comprises a camera 2 configured to capture an image 3, and a system controller 6 configured to display a first or a second area of the captured image 3 depending the vehicle driving and/or user information. In paragraph [0054]-the rear vision system 1 further comprises at least one surroundings sensor, at least one steering sensor, and a processor. The surroundings sensor is configured to acquire data on at least a portion of the surroundings of the vehicle to the side and/or to the rear).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of SUGIE in view of NAGAI and in further view of AKITA of having a control method of a vehicle, the control method comprising: setting, as a template, an area around a vanishing point in a previous frame of an image input from a camera; determining, by a controller, a matching area matching with the template by performing template matching in a current frame, with the teachings of GOMEZ of having moving the template according to a slope of a horizontal line based on a roll angle at which the camera is mounted
Wherein SUGIE’s method having wherein the changing of the position of the template includes moving the template according to a slope of a horizontal line based on a roll angle at which the camera is mounted.
The motivation behind the modification would have been to obtain a method of that improves object detection and camera calibration, since both SUGIE and GOMEZ concern vehicles and image analysis. Wherein SUGIE provides systems and methods that improve the bird's-eye view image and the accuracy of the subsequent processes performed based on this image, while GOMEZ provides systems and methods that improves driver safety and comfort. Please see SUGIE et al. (US 20200082179 A1), Abstract and Paragraph [0129 and 0150] and GOMEZ et al. (US 20170163863 A1), Abstract and Paragraph [0001-0007].
Regarding claim 5, SUGIE in view of NAGAI and in further view of AKITA and in further view of GOMEZ explicitly teach the control method of claim 4, SUGIE in view of NAGAI and in further view of AKITA fail to explicitly teach wherein the changing of the position of the template includes moving the template according to the slope of the horizontal line in a direction opposite to a driving direction of the vehicle.
However, GOMEZ explicitly teaches wherein the changing of the position of the template (Fig. 1. Paragraph [0031]-GOMEZ discloses the system controller 6 is adapted for displaying a first area 5 of the captured image 3, and also for receiving and processing vehicle driving and/or user information for selecting a second area 8′-8″″′ of the captured image 3 depending on the received information (wherein driving/user information includes, for example, increase/decrease in speed, a change in pitch angle, a change in roll angle by comparing representative parameters of actual and former captured images 3). In paragraph [0063]-GOMEZ discloses the system controller 6 is adapted for performing road monitoring based on road monitoring information, and selecting as second area 8″″′ an area of the first area 5 that is related with the road vanishing point of the image. The system controller 6 may be adapted for selecting as second area 8″″′ a predetermined image that may be related with the road vanishing point. This predetermined image may be the initial image, i.e. the first area 5) includes moving the template according to the slope of the horizontal line (Fig. 1. Paragraph [0055]-GOMEZ discloses the system controller 6 is adapted for determining a relative rightward and leftward rotation of the system 1 from the change in the roll angle, selecting as second area 8″″ an area corresponding to a rightward rotation of the first area 5 such that the field of view is laterally rotated in a clockwise direction when the system 1 is rightwardly rotated about an horizontal axis, and selecting as second area 8″″ an area corresponding to an leftward rotation of the first area 5 such that the field of view is laterally rotated in a counterclockwise direction when the system 1 is leftwardly rotated about an horizontal axis. In paragraph [0056]-GOMEZ discloses FIGS. 8 and 9 show the different areas that are displayed when the controller receives a change in the pitch angle value. In paragraph [0057]-GOMEZ discloses the system 1 comprises a roll angle sensor in communication with the controller 6 and operatively configured to obtain a roll angle value. The roll angle value may be obtained by image processing, comparing the actual captured image 3 with the former captured image 3. In paragraph [0058]-GOMEZ discloses the system controller 6 is configured to change from displaying the first area 5 to display a second area 8″″ corresponding to a rightward rotation (clockwise direction) of the first area 5, when the system 1 (and vehicle) is rightward and horizontally moved according to the roll sensor information. Please also read paragraph [0060]) in a direction opposite to a driving direction of the vehicle (Fig. 1, #2 called a camera. Paragraph [0029]-GOMEZ discloses FIG. 1 shows a schematic view of a rear vision system 1 installed in a vehicle. The system 1 comprises a camera 2 configured to capture an image 3, and a system controller 6 configured to display a first or a second area of the captured image 3 depending the vehicle driving and/or user information. In paragraph. In paragraph [0054]-the rear vision system 1 further comprises at least one surroundings sensor, at least one steering sensor, and a processor. The surroundings sensor is configured to acquire data on at least a portion of the surroundings of the vehicle to the side and/or to the rear).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of SUGIE in view of NAGAI and in further view of AKITA and in further view of GOMEZ of having a control method of a vehicle, the control method comprising: setting, as a template, an area around a vanishing point in a previous frame of an image input from a camera; determining, by a controller, a matching area matching with the template by performing template matching in a current frame, with the teachings of GOMEZ of having wherein the changing of the position of the template includes moving the template according to the slope of the horizontal line in a direction opposite to a driving direction of the vehicle.
Wherein SUGIE’s method having wherein the changing of the position of the template includes moving the template according to the slope of the horizontal line in a direction opposite to a driving direction of the vehicle.
The motivation behind the modification would have been to obtain a method of that improves object detection and camera calibration, since both SUGIE and GOMEZ concern vehicles and image analysis. Wherein SUGIE provides systems and methods that improve the bird's-eye view image and the accuracy of the subsequent processes performed based on this image, while GOMEZ provides systems and methods that improves driver safety and comfort. Please see SUGIE et al. (US 20200082179 A1), Abstract and Paragraph [0129 and 0150] and GOMEZ et al. (US 20170163863 A1), Abstract and Paragraph [0001-0007].
Regarding claim 17, SUGIE in view of NAGAI and in further view of AKITA explicitly teach the vehicle of claim 16, SUGIE in view of NAGAI and in further view of AKITA fail to explicitly teach wherein the controller is configured for determining a movement direction of the template based on a driving direction of the vehicle and a roll angle at which the camera is mounted, and for moving the template in the determined movement direction.
However, GOMEZ explicitly teaches wherein the controller (Fig. 1, #6 called a system controller. Paragraph [0029]-GOMEZ discloses FIG. 1 shows a schematic view of a rear vision system 1 installed in a vehicle. The system 1 comprises a system controller 6. Please also read paragraph [0054 and 0062-0064]) is configured for determining a movement direction of the template (Fig. 2. Paragraph [0033]-GOMEZ discloses the system controller 6 is further adapted for determining a relative upward and downward movement of the system from a change in the pitch angle value. The system controller 6 is also adapted for selecting as second area 8′ an area corresponding to a downward vertical displacement of the first area 5 such that the field of view is vertically displaced in a downward direction, when the system 1 is downwardly moved, and selecting as second area 8′ an area corresponding to an upward vertical displacement of the first area 5 such that the field of view is vertically displaced in an upward direction, when the system 1 is upwardly moved. In paragraph [0063]-GOMEZ discloses the system controller 6 is adapted for performing road monitoring based on road monitoring information, and selecting as second area 8″″′ an area of the first area 5 that is related with the road vanishing point of the image. The system controller 6 may be adapted for selecting as second area 8″″′ a predetermined image that may be related with the road vanishing point. This predetermined image may be the initial image, i.e. the first area) based on a driving direction of the vehicle (Fig. 2. Paragraph [0031]-GOMEZ discloses the vehicle driving information at least corresponds to a change in driving direction from a forward driving direction to a reverse driving direction or vice versa, an increase or decrease in driving speed with respect to a predefined driving speed value, a lane change, a change in steering angle, a change in pitch angle, a change in roll angle, and road monitoring information. In paragraph [0032]-GOMEZ discloses system controller 6 is further adapted for selecting as second area 8′ an area corresponding to a downward vertical displacement of the first area 5 such that the field of view is vertically displaced in a downward direction, when the driving direction is changed from a forward driving direction to a reverse driving direction, and selecting as second area 8′ an area corresponding to an upward vertical displacement of the first area 5 such that the field of view is vertically displaced in an upward direction, when the driving direction is changed from a reverse driving direction to a forward driving direction. Please also read paragraph [0036]) and a roll angle at which the camera (Fig. 1, #2 called a camera. Paragraph [0029]-GOMEZ discloses the system 1 comprises a camera 2 configured to capture an image 3) is mounted (Fig. 2. Paragraph [0034] FIGS. 2 and 3 show the different areas that are displayed when the controller receives either a change in the driving direction or a change in the pitch angle value. In paragraph [0035]-GOMEZ discloses the system 1 comprises a roll angle sensor in communication with the controller 6 and operatively configured to obtain a pitch angle value. In paragraph [0057]-GOMEZ discloses the roll angle value may be obtained by image processing, comparing the actual captured image 3 with the former captured image 3), and for moving the template in the determined movement direction (Fig. 2. Paragraph [0046]-GOMEZ discloses the system controller 6 is adapted for selecting as second area 8′″ an area corresponding to a rightward lateral displacement of the first area 5 such that the field of view is laterally displaced in a leftward direction in the event of a change in a steering angle toward a leftward direction, and selecting as second area 8′″ an area corresponding to a leftward lateral displacement of the first area 5 such that the field of view is laterally displaced in a rightward direction in the event of a change in a steering angle toward a rightward direction).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of SUGIE in view of NAGAI and in further view of AKITA of having a vehicle, comprising: a camera configured to photograph an area around of the vehicle; and a controller electrically connected to the camera, wherein the controller is configured to: set, as a template, an area around a vanishing point in a previous frame of an image input from the camera, with the teachings of GOMEZ of having wherein the controller is configured for determining a movement direction of the template based on a driving direction of the vehicle and a roll angle at which the camera is mounted, and for moving the template in the determined movement direction.
Wherein SUGIE’s vehicle having wherein the controller is configured for determining a movement direction of the template based on a driving direction of the vehicle and a roll angle at which the camera is mounted, and for moving the template in the determined movement direction.
The motivation behind the modification would have been to obtain a vehicle that improves object detection and camera calibration, since both SUGIE and GOMEZ concern vehicles and image analysis. Wherein SUGIE provides systems and methods that improve the bird's-eye view image and the accuracy of the subsequent processes performed based on this image, while GOMEZ provides systems and methods that improves driver safety and comfort. Please see SUGIE et al. (US 20200082179 A1), Abstract and Paragraph [0129 and 0150] and GOMEZ et al. (US 20170163863 A1), Abstract and Paragraph [0001-0007].
Claims 6-7 are rejected under 35 U.S.C. 103 as being unpatentable over SUGIE et al. (US 20200082179 A1), hereinafter referenced as SUGIE in view of NAGAI et al. (US 20180322655 A1, hereinafter referenced as NAGAI and in further view of AKITA et al. (US 20100027844 A1), hereinafter referenced as AKITA and in further view of IMAI et al. (US 20200014898 A1), hereinafter referenced as IMAI.
Regarding claim 6, SUGIE in view of NAGAI and in further view of AKITA explicitly teach the control method of claim 2, SUGIE in view of NAGAI and in further view of AKITA fail to explicitly teach wherein the changing of the position of the template includes moving the template upward, based on a driving direction of the vehicle not being recognized or the vehicle going straight.
However, IMAI explicitly teaches wherein the changing of the position of the template includes moving the template upward, based on a driving direction of the vehicle not being recognized or the vehicle going straight (Fig. 2B. Paragraph [0032]-IMAI discloses the driving direction detecting section 15 detects a travel direction (forward or backward) of the vehicle including the image processing apparatus and notifies the attention area setting section 18 of travel direction information representing the travel direction. In paragraph [0033]-IMAI discloses the attention area setting section 18 sets the attention area on the basis of the moving object detection information, the vanishing point information, the travel direction information, the speed information, and the steering angle information and notifies the image correction processing section 19 and the imaging control section 20 of attention area information that represents a position and a size thereof. In paragraph [0063]-IMAI discloses the attention area is moved corresponding to a moving direction of the vanishing point. Specifically, in a case where a traveling road is curved left or right or is uphill or downhill, the attention area is moved up, down, left, or right. Please also read paragraph [0047, 0056, and 0062]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of SUGIE in view of NAGAI and in further view of AKITA of having a control method of a vehicle, the control method comprising: setting, as a template, an area around a vanishing point in a previous frame of an image input from a camera; determining, by a controller, a matching area matching with the template by performing template matching in a current frame, with the teachings of IMAI of having wherein the changing of the position of the template includes moving the template upward, based on a driving direction of the vehicle not being recognized or the vehicle going straight.
Wherein SUGIE’s method having wherein the changing of the position of the template includes moving the template upward, based on a driving direction of the vehicle not being recognized or the vehicle going straight.
The motivation behind the modification would have been to obtain a method of that improves camera calibration and object detection, since both SUGIE and IMAI concern vehicles and image analysis. Wherein SUGIE provides systems and methods that improve the bird's-eye view image and the accuracy of the subsequent processes performed based on this image, while IMAI provides systems and methods that improves the ability to accurately detect an object. Please see SUGIE et al. (US 20200082179 A1), Abstract and Paragraph [0129 and 0150] and IMAI et al. (US 20200014898 A1), Abstract and Paragraph [0014].
Regarding claim 7, SUGIE in view of NAGAI and in further view of AKITA explicitly teach the control method of claim 1, SUGIE in view of NAGAI and in further view of AKITA fail to explicitly teach wherein the setting of the area around the vanishing point as the template includes changing a size of the template based on a speed of the vehicle.
However, IMAI explicitly teaches wherein the setting of the area around the vanishing point as the template includes changing a size of the template based on a speed of the vehicle (Fig. 2B. Paragraph [0033]-IMAI discloses the attention area setting section 18 sets the attention area on the basis of the moving object detection information, the vanishing point information, the travel direction information, the speed information, and the steering angle information and notifies the image correction processing section 19 and the imaging control section 20 of attention area information that represents a position and a size thereof. In paragraph [0062]-IMAI discloses the image processing apparatus, the size of the current attention area is decreased or increased depending on the speed of the vehicle including the image processing apparatus. Specifically, it assumes that the size of the attention area at the current speed is a frame 31. If the speed is increased, the size of the attention area is decreased to a frame 32 since it takes a short time to approach the vanishing point. In contrast, if the speed is decreased, the size of the attention area is increased to a frame 33 since it takes a long time to approach the vanishing point. Please also read paragraph [0047 and 0063]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of SUGIE in view of NAGAI and in further view of AKITA of having a control method of a vehicle, the control method comprising: setting, as a template, an area around a vanishing point in a previous frame of an image input from a camera; determining, by a controller, a matching area matching with the template by performing template matching in a current frame, with the teachings of IMAI of having wherein the setting of the area around the vanishing point as the template includes changing a size of the template based on a speed of the vehicle.
Wherein SUGIE’s method having wherein the setting of the area around the vanishing point as the template includes changing a size of the template based on a speed of the vehicle.
The motivation behind the modification would have been to obtain a method of that improves camera calibration and object detection, since both SUGIE and IMAI concern vehicles and image analysis. Wherein SUGIE provides systems and methods that improve the bird's-eye view image and the accuracy of the subsequent processes performed based on this image, while IMAI provides systems and methods that improves the ability to accurately detect an object. Please see SUGIE et al. (US 20200082179 A1), Abstract and Paragraph [0129 and 0150] and IMAI et al. (US 20200014898 A1), Abstract and Paragraph [0014].
Conclusion
Listed below are the prior arts made of record and not relied upon but are considered pertinent to applicant`s disclosure.
SUMITOMO et al. (US 20120013713 A1)- An image integration unit includes: an imaging section which is installed in a moving body and which images a plurality of time-series images at different times; a three-dimensional image information calculating section which calculates three-dimensional image information in each of the time-series images based on the time-series images imaged by the imaging section; a stationary body area extracting section which extracts stationary body areas in each of the time-series images based on the three-dimensional image information; and an integrating section which calculates the corresponding stationary body areas between the time-series images from each of the stationary body areas extracted in each of the time-series images, and matches the corresponding stationary body areas to integrate the time-series images..................................Please see Fig. 3 and paragraph [0020-0024 and 0038-0039] (Please note this reference with regard to claims 1-2, 8-9 15, and 16). Abstract.
RIGGS et al. (US 20200192374 A1)- Systems and methods for implementing one or more autonomous features for autonomous and semi-autonomous control of one or more vehicles are provided. More specifically, image data may be obtained from an image acquisition device and processed utilizing one or more machine learning models to identify, track, and extract one or more features of the image utilized in decision making processes for providing steering angle and/or acceleration/deceleration input to one or more vehicle controllers. In some instances, techniques may be employed such that the autonomous and semi-autonomous control of a vehicle may change between vehicle follow and lane follow modes. In some instances, at least a portion of the machine learning model may be updated based on one or more conditions.................................Please see paragraph [0196-0198]. Abstract.
KOJIMA et al. (US 20140180497 A1)- A road surface shape estimating device includes a target detecting unit, a coordinate calculating unit and an estimating unit. The target detecting unit detects a known target from a captured image capturing an area ahead of a vehicle. The known target is a target of which a size and positional relationship to a road surface is known. The coordinate calculating unit determines three-dimensional coordinates for each known target from a size and position in the captured image of the known target detected by the target detecting unit. The estimating unit estimates a road surface shape using the three-dimensional coordinates calculated by the coordinate calculating unit...........................Please see Fig. 2-5 and para. [0031-0039]. Abstract.
DAI et al. (US 20200160561 A1)- Vehicle-mounted camera pose estimation methods, apparatuses, and systems, and electronic devices involve performing lane line detection of a road on which a vehicle drives on the basis of a video stream of the road acquired by a vehicle-mounted camera; obtaining horizon information of the road on which the vehicle drives according to a lane line detection result; and obtaining pose information of the vehicle-mounted camera according to the horizon information..........................Please see Fig. 3-7, 9 and 10. Abstract.
Mukherjee et al. (US 20250037311 A1)- Methods and systems are provided for calibrating an on-board camera of a vehicle by detecting one or both of a lead vehicle geometry and a road lane width. In one example, a method includes identifying a known vehicle geometry of a lead vehicle; estimating a distance to the lead vehicle based on matching the known vehicle geometry to an image of the known vehicle geometry from the camera; and updating a calibration of the camera based on the estimated distance..........................Please see Fig. 4-5 and para. [0042-0052]. Abstract.
GUPTA et al. (US 20210268962 A1)- A vehicular vision system includes a camera disposed at a vehicle and operable to capture multiple frames of image data during a driving maneuver of the vehicle. A control includes an image processor that processes frames of captured image data to determine feature points in an image frame when the vehicle is operated within a first range of steering angles, and to determine motion trajectories of those feature points in subsequent image frames for the respective range of steering angles. The control determines a horizon line based on the determined motion trajectories. Responsive to determination that the determined horizon line is non-parallel to the horizontal axis of the image plane, at least one of pitch, roll or yaw of the camera is adjusted. Image data captured by the camera is processed at the control for object detection................................Please see Fig. 1E and paragraph [0044-0054 and 0069-0072]. Abstract.
BISWAS et al. (US 20210019897 A1)- An approach is provided for estimating a real-world depth information from a monocular image. The approach, for example, involves determining a vanishing point of the monocular image captured by a camera. The approach also involves generating a vanishing point ray from an optical center of the camera through the vanishing point on an image plane of the monocular image to infinity. The approach further involves generating a center line ray from the optical center through a geometric center of the image plane to a feature line that is parallel to the vanishing point ray at a lateral distance. The approach further involves generating a feature ray from the optical center through a location of the feature on the image plane to the feature line. The approach further involves computing the real-world distances of the feature based on image coordinates of the rays, lines, angles derived therefrom, and a known pixel-wise distance of the monocular image...............................Please see Fig. 3-5 and 8-9 and paragraph [0055-0057]. Abstract.
Sasaki et al. (US 20200198646 A1)- A slip angle estimation device for a vehicle comprises an imaging device for capturing an image of at least one of the front and the rear of the vehicle and a control unit. The imaging device is a CCD camera including a lens and an imaging sensor. The control unit is configured to determine a plurality of tracking points for a plurality of captured objects, determine an optical flow for the plurality of tracking points based on two images captured at predetermined elapsed time intervals, determine a vanishing point based on intersections of the plurality of optical flows, and calculate a slip angle of the, vehicle based on a ratio of a horizontal distance between an image center and the vanishing point to a distance between a lens center of the CCD camera and an imaging sensor.........................Please see Fig. 4-5 and para. [0099-0107]. Abstract.
CHOE et al. (US 20200410704 A1)- In response to a first image captured by a camera of an ADV, a horizon line is determined based on the camera's hardware settings, representing a vanishing point based on an initial or default pitch angle of the camera. One or more lane lines are determined based on the first image via a perception process performed on the first image. In response to a first input signal received from an input device, a position of the horizon line is updated based on the first input signal and a position of at least one of the lane lines is updated based on the updated horizon line. The input signal may represent an incremental adjustment for adjusting the position of the horizon line. A first calibration factor or first correction value is determined for calibrating a pitch angle of the camera based on a difference between the initial horizon line and the updated horizon line.............................Please see Fig. 5-6. Abstract.
LIU et al. (US 20190103026 A1)- A collision warning system determines probabilities of potential collisions between a vehicle and other objects such as other vehicles. In an embodiment, sensors of a client device capture sensor data including motion data and image frames from a forward-facing view of the vehicle. An orientation of the client device relative to the vehicle may be determined using the motion data. The collision warning system determines cropped portions of the image frames and detects an object captured the image frames by processing the cropped portions. The collision warning system determines a probability of a potential collision between the vehicle and the object by tracking motion of the object. Responsive to determining that the probability is greater than a threshold value, the collision warning system may provide a notification of the potential collision to a driver of the vehicle...............................Please see Fig. 2-3 and para. [0030-0033 and 0044]. Abstract.
CHOI (US 20210256720 A1)- Vanishing point extraction includes obtaining a straight line including a vanishing point of a first image; obtaining a plurality of sample points in the first image based on processing the first image according to an object included in the first image and the straight line including the vanishing point of a first image, such that the plurality of sample points are determined as pixels in the first image having coordinates that overlap with coordinates of pixels of both the straight line and the object included in the first image; obtaining at least one matching point, in a second image, that corresponds to at least one sample point of the plurality of sample points in the first image the second image generated subsequently to the first image being generated; and obtaining a vanishing point of the second image based on the at least one matching point of the second image............................Please see Fig. 1-12. Abstract.
Any inquiry concerning this communication or earlier communications from the examiner
should be directed to Aaron Bonansinga whose telephone number is (703) 756-5380 The examiner can normally be reached on Monday-Friday, 9:00 a.m. - 6:00 p.m. ET.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s
supervisor, Chineyere Wills-Burns can be reached by phone at (571) 272-9752. The fax phone number for the organization where this application or proceeding is assigned is (571) 273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/AARON TIMOTHY BONANSINGA/Examiner, Art Unit 2673
/CHINEYERE WILLS-BURNS/Supervisory Patent Examiner, Art Unit 2673
Prosecution Insights