Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Information Disclosure Statement
The information disclosure statements (IDS) submitted on 11/08/2024 and 06/18/2025 are filed and are in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 112
Claims 1-17 rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Regrading claims 1 and 17, the claims recite: “acquiring a correspondence relationship between position coordinates of the image captured by the imaging device and position coordinates of the calibration pattern in the light illumination range; calculating an amount of parallax difference between the light distribution variable lamp and the imaging device”. It is unclear to the examiner how the “coordinates of the calibration pattern in the light illumination range” is being determined as changes in distance between the lamp and the surface will distort the light pattern and thus changes the coordinates of the point on the light pattern. Even if there exists a system to calibrate for such changes, it still requires knowing such change of distance and change light pattern has happened, which means a recording of the light pattern from the angle of the lamp exists.
In addition, calculation of parallax typically requires two images. However, there exists only image within the claims. When looking at the spec, in particular Fig. 3 and Para [0039]-[0042], it is unclear how the coordinates of “luminous point p” is determined. Since no second image is captured from the angle of the lamp and the distance between the whole system to the surface is not determined as well, it is unclear how the coordinates of “luminous point p” is determined, which is essential to the calculation of parallax as it is the point to be observed from the point of view of the lamp and the point of view of the camera.
For the above reason, claims 1 and 17 are rejected under 35 U.S.C. 112(b) for indefinite issues.
Claims 2-16 are rejected for dependent upon claim 1.
For the purpose of applying prior art, the claims are being interpreted as having two images being captured. No further limitation is being assumed as none of the detail within the calculation of parallax of Fig. 3 and the specification is currently included in the independent claims 1 and 17.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1 and 17 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by HӦDLMOSER et al. (US 2023/0109225 A1, hereinafter Hӧdlmoser).
Regarding claims 1, Hӧdlmoser discloses
A calibration method of calibrating between a light illumination range of a light distribution variable lamp that illuminates a region ahead of a vehicle with light and an imaging range of an imaging device that captures an image of the region ahead of the vehicle (Abstract: “A device and a method for calibrating a light projector (1) with adaptable luminous elements (3) arranged in an image plane (2), comprising a camera (4) and a control unit (5) connected to the light projector (1) and the camera (4), wherein the light projector (1) is configured to project in temporal succession at least two test images (6, 6') with at least two comparison structures (9, 9') from the image plane (2) to an arbitrarily extending projection surface (7), the camera (4) is a depth camera configured to record at least two camera images of a recording surface (8) in a manner substantially synchronized temporally with the light projector (1), and to detect the three-dimensional position data of the comparison structures (9, 9') in the camera images, and the control unit (5) is configured to determine the position and orientation of the light projector (1 ), and to calculate the translational displacement and rotational twist between camera (4) and projector (1).”), the calibration method comprising:
forming a calibration pattern with the light distribution variable lamp onto a screen located at a distance from the light distribution variable lamp and the imaging device; capturing an image of the calibration pattern on the screen with the imaging device (Para [0016]: “The comparison structures may preferably be visually concise structures (markers) which are easily recognisable by the camera, even in the case where the projection surface is highly inhomogeneous and not flat. In the case of a headlight as a projector, for example, these may be uniformly distributed circles and/or projections of circles that are formed by the light beams hitting the projection surface. In order to simplify the detection of correlating comparison structures in at least two camera images, the projector may, for example, be designed to generate test images with different comparison structures, or also comparison structures in different colours or geometries, at defined time intervals.”, Para [0017]: “The two test images transmitted consecutively must be at least partially identical, so that the comparison structures used on them are also identical. This facilitates the recognition of the comparison structures in both test images, even in case of highly distorted captured images. The test images are projected while the distance between the light projector and the projection surface changes, that is, preferably when the vehicle is moving. However, it may also be provided that the vehicle is stationary and the projection surface is moving, for example in a test environment.”, Para [0018]: “The camera is designed to capture at least two camera images of a capturing surface substantially synchronised in time with the light projector, the capturing surface substantially comprising the projection surface. The capturing surface is located at a distance in front of the light projector and substantially comprises the projection surface.”, Para [0019]: “In other words, the projection surface is substantially part of the capturing surface so that the camera can capture the entire projected image. The capturing surface does not have to be flat as well, but may be structured as desired.”);
acquiring a correspondence relationship between position coordinates of the image captured by the imaging device and position coordinates of the calibration pattern in the light illumination range; calculating an amount of parallax difference between the light distribution variable lamp and the imaging device (Para [0021]: “According to the invention, the control unit is designed to determine the position and orientation of the light projector from the at least four detected three-dimensional position data of the comparison structures. This can be done by the control unit reconstructing two light beams that run through one and the same comparison structure, but at different points in time. The intersection of the two light beams determines the position of the light projector.” Para [0023]: “Additionally, the control unit may also be designed to calculate the translational displacement between camera and projector relative to an initial position by comparing the determined coordinates of the light projector with a previously stored initial position. Further, the control unit may also be designed to calculate the rotational torsion relative to an initial position by comparing the determined coordinates of the light projector with a previously stored initial orientation.”); and
correcting the correspondence relationship based on the amount of parallax difference and generating mutual position information between the light illumination range and the imaging range (Para [0025]: “By comparing this calculated position, an internally stored initial positional relationship between the light projector and the camera can be corrected. If no initial positional relationship is available yet, the determined translational displacement between camera and projector can be set as initial positional relationship. Further, an initially stored torsion between light projector and camera may also be corrected by determining the actual torsion based on the captured camera images. If no initial torsion is available yet, the determined rotational torsion between camera and projector can be set as initial torsion. The control unit may store the determined translational displacement and rotational torsion in the storage unit for later use.”).
Regarding claims 17, Hӧdlmoser discloses
A calibration device that calibrates between a light illumination range of a light distribution variable lamp that illuminates a region ahead of a vehicle with light and an imaging range of an imaging device that captures an image of the region ahead of the vehicle (Abstract: “A device and a method for calibrating a light projector (1) with adaptable luminous elements (3) arranged in an image plane (2), comprising a camera (4) and a control unit (5) connected to the light projector (1) and the camera (4), wherein the light projector (1) is configured to project in temporal succession at least two test images (6, 6') with at least two comparison structures (9, 9') from the image plane (2) to an arbitrarily extending projection surface (7), the camera (4) is a depth camera configured to record at least two camera images of a recording surface (8) in a manner substantially synchronized temporally with the light projector (1), and to detect the three-dimensional position data of the comparison structures (9, 9') in the camera images, and the control unit (5) is configured to determine the position and orientation of the light projector (1 ), and to calculate the translational displacement and rotational twist between camera (4) and projector (1).”), the calibration device:
acquires an image captured by the imaging device of a calibration pattern that the light distribution variable lamp forms on a screen located at a distance from the light distribution variable lamp and the imaging device (Para [0016]: “The comparison structures may preferably be visually concise structures (markers) which are easily recognisable by the camera, even in the case where the projection surface is highly inhomogeneous and not flat. In the case of a headlight as a projector, for example, these may be uniformly distributed circles and/or projections of circles that are formed by the light beams hitting the projection surface. In order to simplify the detection of correlating comparison structures in at least two camera images, the projector may, for example, be designed to generate test images with different comparison structures, or also comparison structures in different colours or geometries, at defined time intervals.”, Para [0017]: “The two test images transmitted consecutively must be at least partially identical, so that the comparison structures used on them are also identical. This facilitates the recognition of the comparison structures in both test images, even in case of highly distorted captured images. The test images are projected while the distance between the light projector and the projection surface changes, that is, preferably when the vehicle is moving. However, it may also be provided that the vehicle is stationary and the projection surface is moving, for example in a test environment.”, Para [0018]: “The camera is designed to capture at least two camera images of a capturing surface substantially synchronised in time with the light projector, the capturing surface substantially comprising the projection surface. The capturing surface is located at a distance in front of the light projector and substantially comprises the projection surface.”, Para [0019]: “In other words, the projection surface is substantially part of the capturing surface so that the camera can capture the entire projected image. The capturing surface does not have to be flat as well, but may be structured as desired.”);
acquires a correspondence relationship between position coordinates of the image and position coordinates of the calibration pattern in the light illumination range; calculates an amount of parallax difference between the light distribution variable lamp and the imaging device (Para [0021]: “According to the invention, the control unit is designed to determine the position and orientation of the light projector from the at least four detected three-dimensional position data of the comparison structures. This can be done by the control unit reconstructing two light beams that run through one and the same comparison structure, but at different points in time. The intersection of the two light beams determines the position of the light projector.” Para [0023]: “Additionally, the control unit may also be designed to calculate the translational displacement between camera and projector relative to an initial position by comparing the determined coordinates of the light projector with a previously stored initial position. Further, the control unit may also be designed to calculate the rotational torsion relative to an initial position by comparing the determined coordinates of the light projector with a previously stored initial orientation.”); and
corrects the correspondence relationship based on the amount of parallax difference and generates mutual position information between the light illumination range and the imaging range (Para [0025]: “By comparing this calculated position, an internally stored initial positional relationship between the light projector and the camera can be corrected. If no initial positional relationship is available yet, the determined translational displacement between camera and projector can be set as initial positional relationship. Further, an initially stored torsion between light projector and camera may also be corrected by determining the actual torsion based on the captured camera images. If no initial torsion is available yet, the determined rotational torsion between camera and projector can be set as initial torsion. The control unit may store the determined translational displacement and rotational torsion in the storage unit for later use.”).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 2-8 are rejected under 35 U.S.C. 103 as being unpatentable over HӦDLMOSER et al. (US 2023/0109225 A1, hereinafter Hӧdlmoser) in view of Schneider et al. (US 2019/0122390 A1, hereinafter Schneider).
Regarding claims 2, dependent upon claim 1, Hӧdlmoser discloses everything regarding claim 1.
However, Hӧdlmoser does not explicitly disclose
at least a part of the calibration pattern is linear.
Schneider teaches
at least a part of the calibration pattern is linear (Fig. 1, Para [0015]: “In other words, beams are calculated for points that are situated at equivalent positions within the projected light pattern (for example upper left hand corner of a light area in row x and column y of the projected light pattern when using a chessboard pattern).”).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Hӧdlmoser with a checker board pattern for calibration of camera and other aspects of Schneider to as both Hӧdlmoser and Schneider uses light pattern for calibration of camera. In addition, since using a checker board light pattern for calibration is common technique in this field, it can be expected for a person with normal skill in the art to switch out the circle light pattern of Hӧdlmoser with the checkerboard light pattern of Schneider with expected result of improving accuracy of calibration.
Regarding claims 3, dependent upon claim 1, Hӧdlmoser discloses everything regarding claim 1.
However, Hӧdlmoser does not explicitly disclose
the calibration pattern includes a plurality of straight lines.
Schneider teaches
the calibration pattern includes a plurality of straight lines (Fig. 1, Para [0015]: “In other words, beams are calculated for points that are situated at equivalent positions within the projected light pattern (for example upper left hand corner of a light area in row x and column y of the projected light pattern when using a chessboard pattern).”).
Regarding claims 4, dependent upon claim 2, Hӧdlmoser in view of Schneider teaches everything regarding claim 2.
Schneider further teaches
the calibration pattern includes a plurality of straight lines (Fig. 1, Para [0015]: “In other words, beams are calculated for points that are situated at equivalent positions within the projected light pattern (for example upper left hand corner of a light area in row x and column y of the projected light pattern when using a chessboard pattern).”).
Regarding claims 5, dependent upon claim 1, Hӧdlmoser discloses everything regarding claim 1.
However, Hӧdlmoser does not explicitly disclose
the light illumination range is rectangular, and at least a part of the calibration pattern is on at least one corner and a center portion of the light illumination range.
Schneider teaches
the light illumination range is rectangular, and at least a part of the calibration pattern is on at least one corner and a center portion of the light illumination range (Fig. 1, Para [0022]: “In a further embodiment, the calibration pattern can exhibit a pattern having light and dark polygonal areas, preferably a chessboard pattern. The light and dark polygonal areas can include substantially preferably rectangular areas. Light polygonal areas can be produced by individual light-emitting segments, for example by LEDs of an LED matrix headlight. Dark areas situated between the light areas can correspond to switched-off/deactivated light-emitting segments of the headlight.”, Para [0027]: “Once the image of the scene has been captured, image processing is employed to extract the projected light pattern 5 from the image, and characteristic features are detected therein. In the present example, the characteristic features are comer points 13 of the light filled fields 11 of the projected light pattern 5. As illustrated in FIG. 1, it may be the case that not all comer points 13 in the light pattern 5 are detected. The positions of the detected corner points 13 in the image of the scene are stored, wherein all are assigned to the distance D1 from the wall 4. ”).
Regarding claims 6, 7, and 8, dependent upon claims 2, 3, and 4 respectively, Hӧdlmoser in view of Schneider teaches everything regarding claim 2, 3, and 4.
Schneider further teaches
the light illumination range is rectangular, and at least a part of the calibration pattern is on at least one corner and a center portion of the light illumination range (Fig. 1, Para [0022]: “In a further embodiment, the calibration pattern can exhibit a pattern having light and dark polygonal areas, preferably a chessboard pattern. The light and dark polygonal areas can include substantially preferably rectangular areas. Light polygonal areas can be produced by individual light-emitting segments, for example by LEDs of an LED matrix headlight. Dark areas situated between the light areas can correspond to switched-off/deactivated light-emitting segments of the headlight.”, Para [0027]: “Once the image of the scene has been captured, image processing is employed to extract the projected light pattern 5 from the image, and characteristic features are detected therein. In the present example, the characteristic features are comer points 13 of the light filled fields 11 of the projected light pattern 5. As illustrated in FIG. 1, it may be the case that not all comer points 13 in the light pattern 5 are detected. The positions of the detected corner points 13 in the image of the scene are stored, wherein all are assigned to the distance D1 from the wall 4. ”).
Claims 9-12 are rejected under 35 U.S.C. 103 as being unpatentable over HӦDLMOSER et al. (US 2023/0109225 A1, hereinafter Hӧdlmoser) in view of Schneider et al. (US 2019/0122390 A1, hereinafter Schneider), OTA et al. (US 2022/0222843 A1, hereinafter Ota) and Mullis (US 2021/0281816 A1, hereinafter Mullis).
Regarding claims 9, 10, 11, and 12, dependent upon claims 5, 6, 7, and 8 respectively, Hӧdlmoser in view of Schneider teaches everything regarding claim 5, 6, 7, and 8.
Hӧdlmoser in view of Schneider does not explicitly teach
the calibration pattern includes a first pattern located at least partly on a first corner, a second corner, and the center portion of the light illumination range, and a second pattern located at least partly on a third corner, a fourth corner, and the center portion of the light illumination range, and the calibration method further includes identifying position coordinates of the first pattern in the light illumination range relative to the position coordinates of the image, and generating first mutual position information by adding the amount of parallax difference to the position coordinates identified, and identifying position coordinates of the second pattern in the light illumination range relative to the position coordinates of the image, and generating second mutual position information by adding the amount of parallax difference to the position coordinates identified.
Ota teaches
the calibration pattern includes a first pattern located at least partly on a first corner, a second corner, and the center portion of the light illumination range, and a second pattern located at least partly on a third corner, a fourth corner, and the center portion of the light illumination range (Fig. 3; Under BRI, the current claim language does not limit whether it is corner or corner portion. The current claim language also does not limit how big a corner portion is.), and the calibration method further includes
identifying position coordinates of the first pattern in the light illumination range relative to the position coordinates of the image, and identifying position coordinates of the second pattern in the light illumination range relative to the position coordinates of the image (Para [0115]: “step S405, the parallax predictor 112 calculates 2D coordinates resulting from projection of a point onto an image coordinate system for the rectified first image, and calculates 2D coordinates resulting from projection of the same point onto an image coordinate system for the rectified second image, based on the distance information about the point obtained in step S405. The parallax predictor 112 then calculates the difference in coordinates between the two images. The difference is predicted parallax. The parallax predictor 112 determines predicted parallax for all the points for which distance information is obtained in step S404, and outputs the data as a reference parallax map.”).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Hӧdlmoser in view of Schneider with using two light patterns of Ota to effectively increase the accuracy when measuring distance between the camera and an object, which is ultimately the purpose the system Hӧdlmoser and Schneider is for as the system is mounted on a vehicle.
However, Hӧdlmoser in view of Schneider and Ota does not explicitly teach
generating first mutual position information by adding the amount of parallax difference to the position coordinates identified, and
generating second mutual position information by adding the amount of parallax difference to the position coordinates identified.
Mullis teaches
generating first mutual position information by adding the amount of parallax difference to the position coordinates identified, and generating second mutual position information by adding the amount of parallax difference to the position coordinates identified (Figure 5, Para [0010]: “The intersection points from the captured test pattern images captured by each of the associate imaging components associated with the reference component are translated in accordance with an expected parallax shift for each of the associate imaging components relative to the reference component. A set of geometric corrections for each of the associate imaging components associated with the reference component to compensate for low frequency aberrations in the captured image of the test pattern by comparing the translated intersections points in the images captured by each of the associate imaging components to corresponding intersection points in the corrected image for the reference component. In some of these embodiments, the expected parallax shift for each of the associate imaging components is based upon at least one of the physical offset of a particular imaging component to the reference imaging component, the behavior of sensor optics in the particular associate imaging component, and distance of the test pattern from the array camera.”).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Hӧdlmoser in view of Schneider and Ota with shifting the test pattern in the captured image with expected parallax shift of Mullis to effectively increase the accuracy when calibrating the camera.
Claims 13-16 are rejected under 35 U.S.C. 103 as being unpatentable over HӦDLMOSER et al. (US 2023/0109225 A1, hereinafter Hӧdlmoser) in view of Schneider et al. (US 2019/0122390 A1, hereinafter Schneider), OTA et al. (US 2022/0222843 A1, hereinafter Ota), Mullis (US 2021/0281816 A1, hereinafter Mullis) and Yamazaki et al. (Simultaneous self-calibration of a projector and a camera using structured light, hereinafter Yamazaki).
Regarding claims 13, 14, 15, and 16, dependent upon claims 9, 10, 11, and 12 respectively, Hӧdlmoser in view of Schneider, Ota, and Mullis teaches everything regarding claim 9, 10, 11, and 12.
Ota further teaches
the first corner and the second corner are offset from each other in a first direction, the first pattern includes a first straight line, a second straight line, and a third straight line each extending in a second direction orthogonal to the first direction, the first straight line extends from the first corner to a center line, in the second direction, of the light illumination range, the second straight line extends from the second corner to the center line, the third straight line extends from a midpoint of a side of the light illumination range extending in the first direction to the center line (Fig. 3; The languages of claim 13-16 can include any light pattern that includes some straight lines, including checkerboards and rectangle black white strips).
However, Hӧdlmoser in view of Schneider, Ota, and Mullis does not explicitly teach
the second pattern has a shape that is a mirror image of the first pattern inverted along the center line.
Yamazaki teaches
the second pattern has a shape that is a mirror image of the first pattern inverted along the center line (P. 3 Section 4: “Gray code allows to encode
N
codewords into
l
o
g
N
pattens in black and white. In order to detect the binary patterns robustly, we also project the complementary patterns with black and white inverted, and detect the change of sign in their differences. The pixels whose difference is less than a certain threshold are discarded and unused in the following calibration.”; Using the calibration patterns of this art, inversion of black and white will achieve the same result as mirroring the pattern).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Hӧdlmoser in view of Schneider, Ota, and Mullis with inversion of black and white in lighting pattern of Yamazaki to effectively increase the accuracy of camera self-calibration.
Relevant Prior Art Directed to State of Art
Ito et al. (US 2016/0264042 A1, hereinafter Ito) is prior art not applied in the rejection(s) above. Ito discloses a vehicle lamp system that includes a camera which generates image information in a front region, a controller which generates a light distribution command for instructing a light distribution pattern based on the image information, a headlight which irradiates the front region such that the instructed light distribution pattern is obtained, and a position calibrator which detects a positional deviation between the camera and the headlight.
Skotheim et al. (US 11,763,518 B2, hereinafter Skotheim) is prior art not applied in the rejection(s) above. Skotheim discloses A method for generating a three-dimensional image of an object, comprising receiving a set of input point cloud data from an imaging system, the set of input point cloud data comprising two or more input point clouds, determining, for each point in the respective input point clouds, a value of a quality parameter that reflects a degree of uncertainty in the three dimensional coordinates as specified for that point, generating an output point cloud for generating a three-dimensional representation of the object surface, wherein, for each point in the output point cloud, the coordinates of the respective point are computed based on the coordinate values of the points in the input point clouds, wherein the extent to which the coordinate values in the respective input point clouds are taken into consideration in the computation is determined based on the values of the quality parameter associated with those respective points.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JOSHUA CHEN whose telephone number is (703)756-5394. The examiner can normally be reached M-Th: 9:30 am - 4:30pm ET F: 9:30 am - 2:30pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, STEPHEN R KOZIOL can be reached at (408)918-7630. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/J. C./ Examiner, Art Unit 2665
/Stephen R Koziol/ Supervisory Patent Examiner, Art Unit 2665
Prosecution Insights