DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Status
Claims 1-16 were pending for examination in the Application No. 18/248,322 filed April 7th, 2023. In the remarks and amendments received on September 19th, 2025, claims 1-3, 5-7, 9, 13, and 15 are amended and claims 17-23 are added. Accordingly, claims 1-23 are currently pending for examination in the application.
Response to Amendments
Applicant’s amendments filed September 19th, 2025, to the Specification and Claims have overcome each and every objection previously set forth to the specification and claims in the Non-Final Office Action mailed July 15th, 2025. Accordingly, these objections are withdrawn in response to the remarks and amendments filed. Examiner warmly thanks Applicant for considering the objections and the suggested amendments to be made to the disclosure.
In response to the terminal disclaimer filed December 8th, 2025, the nonstatutory double patenting rejection previously set forth in the Non-Final Office Action mailed July 15th, 2025, is withdrawn.
Response to Arguments
Applicant’s arguments filed September 19th, 2025, regarding the objection(s) of the drawings and rejection(s) of the claims have been fully considered but are not persuasive.
Drawing Objection(s)
The examiner respectfully disagrees that the reference numbers labeling a step of a process or an element of a system in Figs. 3, 5, and 6 of Applicant’s drawings with corresponding descriptions in the specification are sufficient in overcoming the objections to the drawings as required by 37 CFR 1.84(n) (pgs. 9-10 of Applicant’s Remarks). The statement in Rule 84(n) states “drawing symbols may be used for conventional elements when appropriate” (emphasis added). The steps or elements of Applicant’s process or system are not conventional elements. Therefore, the reference numbers labelling each block in Applicant’s drawings are not sufficient as descriptive labels; and thus, remain objected to as required by 37 CFR 1.84(n).
35 USC § 112(b) Rejection(s)
The examiner appreciates Applicant’s remarks regarding the use of the phrase “likely to be recorded” in the newly amended claim limitation “an input image of a landscape representative of a landscape likely to be recorded by the physical digital recording device through said physical windshield” in claims 1 and 15 to differentiate between using a “physical” image recording device and a “modelled” or ”simulated”/”virtual” image recording device (pgs. 10-12 of Applicant’s Remarks). However, the newly made amendments to the claims are not sufficient to overcome the rejection under 35 USC § 112(b).
The ”input image of a landscape representative of a landscape” remains indefinite as it is unclear if the claimed limitation of the representative “landscape likely to be recorded by the physical digital recording device through said physical windshield” is a limitation in the claim (i.e., if the claim requires the representative landscape to be recorded by the digital recording device). Further, it is unclear what type of landscapes are considered “likely to be recorded by the physical digital recording device through said physical windshield”. Therefore, the phrase “likely to be recorded” remains to render claims 1 and 15 indefinite, wherein claims 2-14 and 16 inherit this insufficient antecedent basis in view of their dependency to these claims.
35 USC § 103 Rejection(s)
The examiner respectfully disagrees with Applicant’s assertion that Lasaruk does not use any measured optical quality function because it “emphasizes the point that is directed to calibration approach, rather than a simulation approach that models the windshield base on its measured quality function” (pg. 13 of Applicant’s Remarks). As detailed in the rejection below, the citations of Lasaruk (i.e., para(s). [0003], [0006], [0042], and [0053]—see citations in the rejection of the first limitation of claim 1 below) disclose the measured quality function as the “equations” of the “distortion model”.
Related to Applicant’s assertion above, the examiner respectfully disagrees with Applicant’s assertion that Rowell does not make up for the deficiencies of Lasaruk such as applying a virtual camera because “in Lasaruk’s calibration approach an actual image must be taken through the physical windshield – which only a physical one can do” (pg. 13 of Applicant’s Remarks). As detailed in the rejection of claim 1 below, para. [0053] of Lasaruk discloses optical systems can be modelled through simulations (see citation in the motivation paragraph to combine Lasaruk and Rowell). Since Rowell discloses in the same field of endeavor of calibrating imaging systems that virtual simulations can improve calibration of the imaging systems (para. [0031]—see citation in the teaching of Rowell for claim 1 limitation “a computer implemented method for simulating…” below—, it would have been obvious to one of ordinary skill in the art before the effective filing date of the presently filed invention to modify the system of Lasaruk to incorporate using a virtual camera to output simulated input images of a landscape in a system of a camera viewed through a windshield to improve computer vision tasks like object detection—which is also disclosed in para. [0022] of Lasaruk—in an optical system of Lasaruk comprising of mounting a camera behind a windshield. Furthermore, a person of ordinary skill in the art before the effective filing date of the claimed invention would have recognized that the simulated output images of Rowell being captured in “the same image plane as an actual camera device having the same location within a scene” as recited in para. [0031] of Rowell would be simulating input images of a landscape as viewed through a windshield in the system of Lasaruk since the camera device of Lasaruk captures images of a landscape through a windshield as disclosed in para. [0003] of Lasaruk (see citation of Lasaruk in the rejection of the first limitation of claim 1 below).
The examiner notes that the newly amended claim of “wherein said method provides as an output an image simulating said input image of a landscape as viewed through said physical windshield” is not sufficient in conveying Applicant’s method of “simulat[ing] the effects the measured quality function will have on the input image” as remarked by applicant (pg. 12 of Applicant’s Remarks). As recited in the newly amended claim limitation above, the limitation merely requires simulating said input image of the landscape as viewed through said physical windshield and does not detail what said simulation entails. For example, the claim does not require applying the measured optical quality function to said input image of a landscape to output said simulated image nor do the steps in the claims (i.e., steps a-c) mention the generation of said simulated output image (e.g., step c of claim 1 merely recites modelling a virtual camera in front of a modelled sheet of transparent mineral glass using a virtual camera). The examiner suggests amending into the claims an additional step or within the current steps a-c in the independent claims to actively recite generating the claimed simulated output image from said input image of a landscape as viewed through said physical windshield using/applying the claimed elements recited in the claims (e.g., using/applying the “modelled sheet of transparent mineral glass” in step c, the “measured optical quality function” in step a, etc.) to distinguish Applicant’s simulation of windshield distorted images from the currently cited prior art.
Priority (Previously Presented)
Acknowledgment is made of applicant’s status as a U.S. National Stage Filing under 35 U.S.C. § 371 of International Application No. , filed on September 28th, 2021, which claims priority to European (EP) Patent Application No. 20315428.1, filed on October 8th, 2020.
Acknowledgment is made of applicant’s claim for foreign priority under 35 U.S.C. 119 (a)-(d). The certified copy has been filed as European (EP) Patent Application No. 20315428.1, filed on October 8th, 2020.
Drawings (Previously Presented)
The drawings are objected to because of the following informalities:
Figs. 3, 5, and 6 are objected to as depicting a block diagram without “readily identifiable” descriptors of each block, as required by 37 CFR 1.84(n). Rule 84(n) requires “labeled representations” of graphical symbols, such as blocks; and any that are “not universally recognized may be used, subject to approval by the Office, if they are not likely to be confused with existing conventional symbols, and if they are readily identifiable.” In the case of Figs. 3, 5, and 6, the blocks are not readily identifiable per se and therefore require the insertion of text that identifies the function of that block. That is, each vacant block should be provided with a corresponding label identifying its function or purpose. The examiner respectfully suggests applicant to include descriptive labels to correct this informality.
Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. The figure or figure number of an amended drawing should not be labeled as “amended.” If a drawing figure is to be canceled, the appropriate figure must be removed from the replacement sheet, and where necessary, the remaining figures must be renumbered and appropriate changes made to the brief description of the several views of the drawings for consistency. Additional replacement sheets may be necessary to show the renumbering of the remaining figures. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Claim Objections
Claims 2, 5-6, 7, 14-16, 19 and 21 are objected to because of the following informalities:
In claims 2 and 5, the phrase “the windshield” should be “the physical windshield” to maintain consistency in terminology between claims;
In claims 6-7 and 16, the phrase “the digital image recording device” should be “the physical digital image recording device” to maintain consistency in terminology between claims;
In claim 14, the phrase “calibrating digital image recording device” should be “calibrating a digital image recording device”;
In step d of claims 15 and 21, the phrase “feeding the set of images” should be “feeding the set of simulated input images” to maintain consistency in terminology between claims;
In claim 19, the phrase “said input image” and “the input image” should be “said reference input image” and “the reference input image”, respectively, to maintain consistency in terminology between claims; and
In claim 21, the phrase “each input image of the set of input images” should be “each reference input image of the set of reference input images” and to maintain consistency in terminology between claims.
Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-18 and 21-23 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Regarding claims 1 and 15, the phrase “likely to be recorded by the physical digital image recording device” recited in each claim renders these claims indefinite because it is unclear whether the claim requires the representative landscape to be recorded by the physical digital image recording device. For examination purposes, this phrase in these claims will be read as “[[likely to be]] recorded by the physical digital image recording device”. Furthermore, claims 2-14, 16-18, and 21-23 inherit this insufficient antecedent basis in view of their dependency to claims 1 and/or 15.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1, 3, 7, and 9-23 are rejected under 35 U.S.C. 103 as being unpatentable over Lasaruk et al. (Lasaruk; EP 3293701 A1, provided in Applicant’s IDS filed April 7th, 2023) in view of Rowell et al. (Rowell; US 2020/0342652 A1).
Regarding claim 1, Lasaruk discloses a computer implemented method for simulating effects of optical distortions of a physical windshield on an image recording quality of a physical digital image recording device, wherein said method takes as input a measured optical quality function related to the optical distortions of the physical windshield and an input image of a landscape representative of a landscape likely to be recorded by the physical digital recording device through said physical windshield (para(s). [0003], [0006], [0042], and [0053], recite(s)
[0003] “For ADAS purposes in a vehicle, the camera is typically mounted behind the windshield in the mirror cover. Evidently, this solves the problem of blocking the camera lenses by dirt or rain. Unfortunately, the particular location of the camera introduces the optical properties of the windshield into the projection geometry of the complete optical system. Windshield influences can introduce a substantially different behaviour of the optical paths compared to that of the coaxially ordered lenses inside the camera optics. Consequently, the optical system of the camera together with the windshield can yield significant image deviations compared to the same camera without the windshield in front of it. The above class of optical distortions are referred to as windshield distortions of the camera image.”
[0006] “Ignoring the impact of the windshield distortion can severely obscure the distance estimation to objects, in particular at short distances, in front of the vehicle. Consequently, it is essential to model and estimate the windshield distortions together with other parameters of the camera system. Unfortunately, some properties of the camera distortions can not be estimated accurately by observing natural traffic scenes during driving. Therefore, it is obvious to introduce a distortion model, which approximately applies for a particular class of windshields of commercial vehicles. Such a model is referred to as a static windshield model. The invention introduces a novel camera model and a method for estimation of static windshield relevant parameters for this model.”
[0042] “The method may be based on a model which represents a distant independent approximation of the windshield distortion. The distortion may be represented by the equations described below, for example equation (30) or equation (43), or by their respective inversions.”
[0053] “…Lens systems can be modelled by chains of planes in commercial optical system simulations…”
, where the “distortion model” modeled by “equations” are measured optical quality functions, the “traffic scenes during driving” captured by the “camera” are input images of a landscape representative of a landscape likely to be recorded by the “camera”, and the “camera… mounted behind the windshield” is a physical digital recording device recording through a windshield), wherein said method comprises the following steps:
(a) modelling said physical windshield as at least one sheet of transparent mineral glass comprising two parallel faces, wherein a surface of at least one of said two main parallel faces is modelled as being textured with the measured optical quality function, and wherein the sheet of transparent mineral glass is modelled as being placed in front of the input image of the landscape and is inclined, with respect to said input image an installation angle of said physical windshield in a transporting vehicle (para(s). [0038] and [0084-0085], recite(s)
[0038] “One aspect of the invention is directed at a method for calibration of a camera-based system of a vehicle including a windshield pane, the method comprising the steps of placing an imaging target in form of board with a known pattern in the field of view of a camera of the camera based system, such that the camera can acquire a calibration image of the board through the windshield pane, acquiring a calibration image of the board with the camera, comparing the calibration image to the known pattern, calculating a windshield distortion which is introduced by the windshield pane, and storing the windshield distortion in the camera-based system.”
[0084] “For ADAS applications, the windshield is typically pitched with respect to the viewing direction of the camera. Other angles appear insignificant compared to the pitch. Optical paths enter the wind[s]hield at different directions u. Figure 2, however, shows only a cross section of the corresponding shift along u. When modelled accurately, the shift occurs in the orthogonal projection direction of u to the wind[s]hield surface. Latter behaviour can be approximated by assuming that the shift occurs only in the direction parallel to the optical axis of the camera…”
[0085] “In other words, the windshield plane models the physics of a flat glass model pitched with respect to the local camera coordinate system by τ.”
, where calculating the “windshield distortion” is modelling the physical windshield as at least one sheet of transparent mineral glass comprising two parallel faces (i.e., the “wind[s]hield surface[s]”), the “distortion” is the measured optical quality function texturing the “windshield”, and “pitch[ing]” the “windshield” is inclining the windshield at an installation angle);
(b) calculating, with a stochastic ray tracing method, a global illuminance arriving through the modelled inclined sheet of transparent mineral glass from the input image of the landscape as a light source (para(s). [0040] and [0051-0052], recite(s)
[0040] “In particular, the method relies on comparison only of the acquired image with the known pattern, and does not rely on a second image that was taken without the windshield pane in place. Also, the windshield distortion, which may be referred to as windshield distortion component h, is an bijective and thus invertible transformation, that allows for application of the inverse procedure to projection. With a camera calibrated this way, an approximation of the optical rays that are projected to a pixel, i.e. a location on the image, is made possible.”
[0051] “The optical path of light dispersion is modelled with ray polygons. Moreover, several basic assumptions for the light dispersion are imposed. Besides the z = 1-plane, an additional fixed plane in the local coordinate system of the camera is introduced. This plane is referred to as the windshield plane. It should be noted, however, that latter is only for convenience; more planes can be introduced depending on the problem. Moreover, the introduced plane is not required to necessarily represent the windshield in any sense.”
[0052] “It shall now be specified how light paths pass the planes and how they are modified during this. In the model that motivated the present invention, a polygon segment intersecting a plane is allowed to shift along the plane. The light path from a space point to the image is changed this way from a ray to a polygon. A light path starting in a space point passes the windshield plane 1 first. It is required, as a further assumption, that the final polygon segment passing through the z = 1-plane has to be incident to the camera center. With that, the first-order optics and the distortions are left unchanged, as with the generalized pinhole camera model.”
, where the “approximation of the optical rays that are projected to a pixel” is a stochastic ray tracing method calculating a global illuminance (e.g., “optical rays”)), and
(c) calculating a 3D projection of the global illuminance from a view frustum of a(para(s). [0022], [0033], [0040], [0068], [0076], and [0104], recite(s)
[0022] “… A camera calibration method is called a volume calibration, if the calibration target extends in all three directions in space…”
[0033] “…More precisely, it is still true that for a fixed image pixel the space points projected to that pixel lie on a ray in the world coordinate system to a first approximation. However, the entirety of such rays for all pixels do not intersect or pass through the camera center in general. That is, for a fixed image pixel the amount of distortion correction in the image depends on the distance to the object mapped to this pixel.”
[0040] “…Also, the windshield distortion, which may be referred to as windshield distortion component h, is an bijective and thus invertible transformation, that allows for application of the inverse procedure to projection. With a camera calibrated this way, an approximation of the optical rays that are projected to a pixel, i.e. a location on the image, is made possible.”
[0068] “Consequently, one aspect of the invention is directed at a method of calibrating a camera-based system of a vehicle - preferably an ADAS system - comprising the steps of acquiring an image with the camera, and then calculating, using the windshield distortion, the set of points in space that is projected to a location on the image. This can be seen as the inverse procedure to projection.”
[0076] “For the purpose of rectification, the complete 3D location of an object can be determined by triangulation using at least two images acquired at different locations. These images can be acquired by a single moving camera, or by a stereo camera system comprising two cameras that are preferably oriented parallelly to each other.”
[0104] “Pictured is an apparatus for calibration of a camera-based system - not shown in detail - of a vehicle 3 including a windshield pane (not pictured). The apparatus in this case comprises a vehicle with a windshield pane which is installed in the windshield bracket of the vehicle. A camera - not explicitly shown - that is preferably mounted on the backside of an inside rear mirror (not pictured) is facing the windshield pane from a first side, being inside the vehicle 3, and a target 2 in form of board with a known pattern 4 is mounted in a fixed distance z (not explicitly marked) on the other side (outside of the vehicle) of the windshield inside a field of view of the camera, such that the camera can acquire an image of the target 2 through the windshield.”
, where performing “volume” (i.e., 3D) calibration in an optical system comprises of calculating pixel projections using optical rays is calculating a 3D projection of the global illuminance (e.g., optical rays) from a view frustum (e.g., the “field of view”) of a camera with an optical camera model (e.g., a “camera model”) of said camera installed at an installation position (e.g., “backside of an inside rear view mirror… facing the windshield pane”) of the digital image recording device).
Where Lasaruk does not specifically disclose
a computer implemented method for simulating effects of optical distortions of a physical windshield on an image recording quality of a physical digital image recording device…, wherein said method provides as an output an image simulating said input image of a landscape as viewed through said physical windshield…:
…a virtual camera…;
Rowell teaches in the same field of endeavor of modelling optical distortions in vehicular optical systems
a computer implemented method for simulating effects of optical distortions of a physical windshield on an image recording quality of a physical digital image recording device…, wherein said method provides as an output an image simulating said input image of a landscape as viewed through said physical windshield… (para(s). [0031], recite(s)
[0031] “Additionally, the synthetic image generation systems and methods described herein improve camera functionality by providing virtual simulations of device performance. By including identical parameters of actual camera devices in camera settings files used to generate projections of synthetic scenes, the synthetic image generation system allows virtual cameras to capture the same image plane as an actual camera device having the same location within a scene. Synthetic image capture using realistic parameters for camera settings accurately simulates device performance during capture. Post capture, the synthetic image generation system synthesizes synthetic images generated using camera settings of a particular device to provide feedback on the accuracy of depth information and camera calibration adjustments correcting for calibration errors that would be made by a camera capturing actual photos having the characteristics of the synthetic images.”
, where “generate[d] projections of synthetic scenes” from “virtual cameras” are simulated output images; where a person of ordinary skill in the art before the effective filing date of the claimed invention would have recognized that the simulated output images are captured in “the same image plane as an actual camera device having the same location within a scene” would output simulated images of a landscape as viewed through a windshield in the system of Lasaruk since the camera device of Lasaruk captures images of a landscape through a windshield—see para. [0003] of Lasaruk in the first limitation of claim 1 above):
…a virtual camera… (para(s). [0060], [0136], and [0145], recite(s)
[0060] “The 3D models and other foreground objects selected by the 3D model selection routine may be specific to the intended CV application of the synthetic images produced by the synthetic image generation system 101. A 3D model arrangement routine executed by the image scene generator 124 to position the 3D models and other foreground objects in a foreground portion of the scene may also be specific to the application of the synthetic images. The 3D model arrangement routine may define the horizontal and vertical location and depth of each object (e.g., 3D models and other foreground objects) in a scene. In one embodiment, to produce image scenes for generating synthetic images to train machine learning systems for object detection in autonomous vehicles, the image scene generator 124 selects objects commonly found near and/or on roads (e.g., cars, trucks, buses, trees, buildings, humans, animals, traffic lights, traffic signs, etc.) and positions the objects to generate a scene resembling the point of view from a car driving on the road. One or more affine transformations may also be applied to one or more background and/or foreground objects in the scene to simulate motion.”
[0136] “By mirroring the camera capture settings, calibration parameters, and other characteristics of an actual camera device in a camera device file, the synthetic image generation system 101 may simulate the performance of actual camera devices during synthetic image generation…”
[0145] “Distortion coefficients 526 included in camera settings files 500 for virtual cameras may be k.sub.1, k.sub.2, . . . , k.sub.n parameters describing the levels of lens distortion, as a function of the radius from the center of the captured image frame to the edge of the frame. In some embodiments, n can be, for example, between 1 and 16, depending on how precise the calibration needs to be and the characteristics of the particular lens. The distortion coefficients 526 describe how much distortion an image pixel has as a location of the pixel moves from the center of the image to the edge of the image. In some embodiments, the k.sub.1, k.sub.2, . . . , k.sub.n parameters are defined radially and do not depend on the circular angle of the pixel location. The distortion coefficients 526 are variable depending on the type of lenses used in the camera module. For example, different polynomial lens distortion models having different numbers of distortion coefficients 526 with different values and orders of magnitude are used to describe distortion levels for fisheye and non-fisheye lenses.”
, where the “virtual cameras” are cameras that “mirror” the “characteristics of an actual camera”).
Since Lasaruk also discloses using images comprising of optical distortions for computer vision tasks like object detection (para(s). [0022], recite(s)
[0022] “It should be recalled that camera calibration refers to the problem of estimating the camera parameters by observing images of objects in the world. The state of the art comprises camera calibration methods which are based on detecting point features of an object with known geometry…”
, where “detecting point features of an object” includes object detection) and modelling optical systems through simulations (para(s). [0053], recite(s)
[0053] “…Lens systems can be modelled by chains of planes in commercial optical system simulations…”
), it would have been obvious to one of ordinary skill in the art before the effective filing date of the presently filed invention to modify the system of Lasaruk to incorporate outputting an image simulating said input image of a landscape as viewed through said physical windshield and calculating a 3D projection of the global illuminance from a view frustum of a virtual camera with an optical camera model of said virtual camera, wherein said virtual camera is placed in the same position as the digital recording device, to improve computer vision tasks like object detection using images comprising of optical distortions of a particular windshield by training machine learning models (e.g., neural networks) using synthetic/virtual images using virtual cameras simulating actual camera optical distortions, which reduces the time and costs of acquiring actual camera images comprising of optical distortions, as taught by Rowell (para(s). [0007] and [0091], recite(s)
[0007] “Systems and methods of generating synthetic data described herein improve machine learning in the field of CV by removing barriers to obtaining image data for training neural networks and other machine learning systems. In particular, resource and time intensive manual image capture tasks including scene construction, camera operation for image capture, and post processing steps generating additional image channels are accelerated by the systems and methods of the present invention. Executing one or more routines to virtually perform scene creation, scene capture, and generation of additional image data channels allows the synthetic image generation system to produce vast quantities of image data at a fraction of the time and cost of conventional methods.”
[0091] “In some embodiments, it may be desirable to modify synthetic images provided by the synthetic image generation module 102 with noise and other defects. To increase the amount of variation in a synthetic image dataset, the image augmentation engine 104 may add noise and distortion to synthetic images. By adding other effects (e.g., lens flare, lens distortion, motion blur, and Gaussian blur), the image augmentation engine 104 may also simulate realistic capture errors that commonly occur under certain conditions with specific camera hardware…”
).
Regarding claim 3, Lasaruk in view of Rowell discloses the computer implemented method according to claim 1, wherein Lasaruk further discloses the input image of the landscape is modelled as being at a finite distance of the modelled sheet of transparent mineral glass (para(s). [0104] and [0110], recite(s)
[0104] “Pictured is an apparatus for calibration of a camera-based system - not shown in detail - of a vehicle 3 including a windshield pane (not pictured). The apparatus in this case comprises a vehicle with a windshield pane which is installed in the windshield bracket of the vehicle. A camera - not explicitly shown - that is preferably mounted on the backside of an inside rear mirror (not pictured) is facing the windshield pane from a first side, being inside the vehicle 3, and a target 2 in form of board with a known pattern 4 is mounted in a fixed distance z (not explicitly marked) on the other side (outside of the vehicle) of the windshield inside a field of view of the camera, such that the camera can acquire an image of the target 2 through the windshield.”
[0110] “The camera can be mounted to the windshield in the vehicle bracket on the windshield. The optical axis of the camera can be oriented approximately perpendicular to the target. With the above setup, the windshield calibration parameters for one of the optic modules can be obtained. These parameters can preferably be - but are not limited to - the tilt τ, the thickness t and/or the refraction index v of the windshield pane.”
, where mounting the camera on a “bracket” and at a “fixed distance z” from a “target… on the other side (outside of the vehicle) of the windshield” is the input image of the landscape (which is captured by the mounted camera) being a finite (e.g., “fixed”) distance from the windshield).
Regarding claim 7, Lasaruk in view of Rowell discloses the computer implemented method according to claim 1, wherein Lasaruk further discloses the optical camera model of the virtual camera is a projection matrix, a point-spread function, an optical transfer function or a real modelling of each interface of different optics of a camera lens of the physical digital image recording device (para(s). [0010], [0014-0015], [0017], and [0053], recite(s)
[0010] “A camera model in this document describes a parametric map, which explains the property of a physical camera device to map points in space to pixel locations in the image. In the state of the art, a so-called generalized pinhole camera model may be used in ADAS software for front view cameras.”
[0014] “This function maps space points in the local coordinate system of the camera to plane points. The range of π can be seen as to lie on the so-called z = 1-plane (or the more optically convenient z = -1-plane alternatively by dividing by -z).”
[0015] “Figure 1 shows a sketch of the central projection property of the camera in the x-direction with the addition of a windshield plane 1, which will be discussed later on. For simplicity, all figures are presented in the local camera coordinate system…”
[0017] “Finally, the distortion d… is an invertible deformation of the z = 1-plane, which resembles the optical distortions of the camera…”
[0053] “…Lens systems can be modelled by chains of planes in commercial optical system simulations…”
, where the “camera model” is at least an optical camera model comprising of at least a real modelling of each interface of different optics (e.g., “optical distortions”) of camera lens of a physical digital image recording device).
Regarding claim 9, Lasaruk in view of Rowell discloses a computer implemented method according to claim 1, wherein Lasaruk further discloses the texture of the textured surface is modelled with a bump map or displacement map (para(s). [0021] and [0032], recite(s)
[0021] “Crucial to the observations so far is the note that the direct optical path (not pictured) without displacement by a windshield plane 1 from a space point s to its imager point is given by a ray from s passing through the origin of the camera's local coordinate system and intersecting the z = 1-plane. In the latter is a shift within the plane, given by the distortion function d. From the result of the shift, the first-order optics maps from the plane to the pixel coordinates in the image.”
[0032] “To address the windshield distortion estimation for stereo rectification algorithms, some methods have been proposed. One basic assumption introduced by these methods is that there exists an image transformation which compensates the windshield distortion. If this transformation is additionally assumed sufficiently smooth, the transformation can, to a first approximation, be represented by a polynomial over the camera's images. The parameters of the distortion maps can be statically estimated, say by a carpet measurement on the ground plane, or dynamically during driving.”
, where the “first-order optics maps” mapping the “shift” or “displacement” of pixel points by a “windshield pane” is at least a displacement map modeling the texture (e.g., “distortion”) of the textured surface (e.g., surface of the windshield)).
Regarding claim 10, Lasaruk in view of Rowell discloses a computer implemented method according to claim 1, wherein Lasaruk further discloses the stochastic ray tracing method is a stochastic path tracing method (para(s). [0050] and [0078-0080], recite(s)
[0050] “For a better understanding, a physically motivated model of the optical path from a space point to it's pixel location on the imager shall be derived…”
[0078] “A sketch of this model is presented in Figure 2 , which shows a physically motivated low curvature windshield model, where the optical ray is shifted along the glass surface.
The refraction on this model is governed by Snell’s law.
v
with
0
<
v
∈
R
is the refraction index of the medium
.
θ
,
γ
∈
R
are angles of incidence of the incoming and emerging ray, and the angle of the refracted ray within the medium respectively. The angles are all measured with respect to the surface normal, which is uniquely defined up to sign. Then the angles at the point of first refraction relate with
s
i
n
θ
=
v
s
i
n
(
γ
)
. (equation 32)”
[0079] “The second refraction introduces
s
i
n
γ
v
=
s
i
n
(
θ
)
. (equation 33)”
[0080] “In other words, the transmission of a light ray through the windshield does not change the ray’s direction. The ray is merely shifted along the surface of the windshield. That is, the assumption of a shift along the plane is entirely motivated by optics. Equivalently to the physical model, the shift is performed only on one of the surfaces, in particular on the incident surface.”
, where calculating the path of an “optical ray” or “light ray” through “the surface of the windshield” is a stochastic path tracing method).
Regarding claim 11, Lasaruk in view of Rowell discloses a data processing system comprising a processor and a memory coded with instruction for carrying out a method according to claim 1 (Lasaruk further discloses in para(s). [0104]:
[0104] “…The vehicle also includes a computing unit (not shown) which comprises program code which, when run by the computing unit, executes a method as described above.”
, where a person of ordinary skill in the art before the effective filing date of the claimed invention would have recognized that a system comprising a “computing unit” which “run[s]” a “program code” that “executes a method” must at least comprise of a processor (e.g., a CPU) in order to “run” a “program code” and at least a memory in order to store the coded instructions (e.g., the “program code”) executed by the processor).
Regarding claim 12, Lasaruk in view of Rowell discloses a non-transitory computer readable medium comprising instructions which, when the instructions are executed by a computer, cause the computer to carry out a method according to claim 1 (Lasaruk further discloses in para(s). [0104]:
[0104] “…The vehicle also includes a computing unit (not shown) which comprises program code which, when run by the computing unit, executes a method as described above.”
, where a person of ordinary skill in the art before the effective filing date of the claimed invention would have recognized that a “computing unit… which comprises program code” includes at least a computer in order to “run” the “program code” and at least a non-transitory computer readable medium (e.g., RAM) to store the instructions (e.g., “program code”) which are “run” by the computer).
Regarding claim 13, Lasaruk in view of Rowell discloses a method comprising performing a method according to claim 1 for evaluating an optical quality of a physical windshield for a use with a physical digital image recording device (Lasaruk further discloses in para(s). [0038]—see citation in claim 1 step (a) above—, where “calculating a windshield distortion” is evaluating an optical quality (e.g., distortion) of a windshield).
Regarding claim 14, Lasaruk in view of Rowell discloses a method comprising a method according to claim 1 for calibrating digital image recording device of an automated driving and advanced safety system (Lasaruk further discloses in para(s). [0001-0002], recite(s)
[0001] “The invention relates to the calibration of an ADAS camera…”
[0002] “In computer vision, a physical camera device is modelled essentially by its geometric projection properties. Given a point of an object in space, such a projection formally explains, where the point will appear in the image. For Advanced Driver Assistant Systems (ADAS), accurate estimates of the camera projection are essential for accurately estimating the distance to objects in front of the vehicle. Latter, in turn, is crucial for camera based traffic safety applications like, for example, automated braking or automated cruise control.”
, where “ADAS” is an automated driving and advanced safety system).
Regarding claim 15, Lasaruk in view of Rowell discloses a process for evaluating performance of an object detection and classification algorithm of an automated driving and advanced safety system (Rowell teaches in para(s). [0036] and [0060]:
[0036] “Image data provided by the synthetic image data generation system is compatible with a variety of machine learning systems including rules based classification algorithms, neural networks, and deep learning methods. More specifically, Naïve Bayes classification algorithms, decision tree classification algorithms, convolutional neural networks (CNNs), convolutional recurrent neural networks (CRNNs), hierarchical recurrent convolutional neural networks (HRNN), and HRNNs with attention vectors implemented in a machine learning framework (e.g., Keras, Scikitlearn, MXNet, or Tensorflow) are some example machine learning methods that may leverage training datasets provided by the synthetic image generation system. Using synthetic image data provided by the synthetic image data generation systems and methods described herein, machine learning systems may train one or more models to perform one or more CV tasks including generating disparity maps, depth maps, and other depth information; object tracking, classification, and counting; facial recognition; gesture tracking and control; 2D to 3D conversions; 3D scanning; image enhancement; image augmentation; and simultaneous localization and mapping (SLAM).”
[0060] “The 3D models and other foreground objects selected by the 3D model selection routine may be specific to the intended CV application of the synthetic images produced by the synthetic image generation system 101. A 3D model arrangement routine executed by the image scene generator 124 to position the 3D models and other foreground objects in a foreground portion of the scene may also be specific to the application of the synthetic images. The 3D model arrangement routine may define the horizontal and vertical location and depth of each object (e.g., 3D models and other foreground objects) in a scene. In one embodiment, to produce image scenes for generating synthetic images to train machine learning systems for object detection in autonomous vehicles, the image scene generator 124 selects objects commonly found near and/or on roads (e.g., cars, trucks, buses, trees, buildings, humans, animals, traffic lights, traffic signs, etc.) and positions the objects to generate a scene resembling the point of view from a car driving on the road. One or more affine transformations may also be applied to one or more background and/or foreground objects in the scene to simulate motion.”
, where the images obtained from virtual cameras (e.g., “synthetic images”) are used to “train machine learning systems for object detection in autonomous vehicles” are object detection and classification algorithms (e.g., “object tracking, classification, and counting”, etc.) for automated driving and safety systems (e.g., “autonomous vehicles”)), wherein said process comprises the following steps:
(a) providing a set of measured optical quality functions related to optical distortions of a set of physical windshields (Lasaruk discloses in para(s). [0006] and [0042]—see citations in the first limitation of claim 1—, where para(s). [0007] and [0102] further recite(s):
[0007] “…The invention introduces a novel general method to derive such distortion models systematically. This method can be applied to derive a concrete subclass of distortion models, which describes windshields with small curvature, i.e. windshields that are essentially flat. Latter class applies well to windshields of personal commercial vehicles.”
[0102] “One assumption may be that a camera is provided, which is intrinsically calibrated elsewhere. Moreover, multiple windshields are provided, to which the particular camera can be attached subsequently. The windshields can be placed in a stand or directly in the vehicle.”
, where the “distortion models” are a set of measured optical quality functions related to optical distortions of a set of physical windshields (e.g., “particular class[es] of windshields” or “multiple windshields”));
(b) providing a set of input images of landscapes representative of landscapes likely to be recorded by a physical digital image recording device through a physical windshield of said set of physical windshields (Lasaruk discloses in para(s). [0003] and [0006]—see citations in the first limitation of claim 1—, where the “traffic scenes during driving” captured by a “camera… mounted behind the windshield” are a set of input images of landscapes representative of landscapes likely to be recorded by the physical digital image recording device (i.e., the “camera”)),
(c) using a computer implemented method according to claim 1 for each input image of the set of input images of landscapes and each measured optical quality function of the set measured optical quality functions related to the optical distortion, in order to provide a set of images simulating the input images as viewed through the physical windshield of said set of physical windshields (Lasaruk in view of Rowell discloses the computer implemented method according to claim 1—see the rejection of claim 1 above including the rationale and reasoning to combine Lasaruk and Rowell—, where Lasaruk discloses each of the input images in the set of input images of landscapes in step (b) include a corresponding measured optical quality function (e.g., distortion)—see para(s). [0003], [0006], [0042], and [0053] of Lasaruk in the first limitation of claim 1 above—; wherein Rowell teaches simulating the input images as viewed through the physical windshield of said set of physical windshields as “generate[d] projections of synthetic scenes”—see the teachings of Rowell in claim 1 limitation “a computer implemented method for…” above);
(d) feeding the set of simulated input images obtained at step (c) to an object detection and classification algorithm of an automated driving and advanced safety system (Rowell further teaches in para(s). [0060], [0136], and [0145]—see citations in claim 1 above—that a set of simulated input images (e.g., “synthetic images”) can be fed to an object detection and classification algorithm of an automated driving and advanced safety system (e.g., “train[ing] machine learning systems for object detection in autonomous vehicles”); it would have been obvious to one of ordinary skill in the art before the effective filing date of the presently filed invention to modify the system of Lasaruk and Rowell to further incorporate feeding the set of images obtained at step (c) as disclosed by Lasaruk to further train the machine learning systems for object detection and classification in autonomous vehicles as the set of images obtained at step (c) also include distortions like the “synthetic images” of Rowell—see para. [0145] of Rowell);
(e) monitoring performance parameters in the object detection and classification of said algorithm while the algorithm is processing the set of images fed at step (d) (Rowell further teaches in para(s). [0033] and [0051]:
[0033] “Synthetic image datasets provided by the systems and methods described herein are highly varied and rapidly scalable depending on the application of the machine learning system using the synthetic dataset as training data. The size of synthetic image datasets produced by the systems and methods described herein is rapidly iterative. The size of synthetic image datasets provided by the systems and methods described herein can be modified depending on the prevalence of manual overfitting, available computational and power resources, value of a loss and/or error function or other metrics, or other feedback describing training performance and/or model accuracy…”
[0051] “…The view controlling the machine learning service 105 may allow users to select a machine learning system to progress training datasets generated by the training data assembly service. Feedback from one or more machine learning systems including training progress, processing performance, and accuracy of generated machine learning models may also be displayed on the machine learning service view included in the user interface 109…”
that the object detection and classification algorithms (e.g., “machine learning systems”) as disclosed in step (a) of the current claim above further comprises monitoring performance parameters (e.g., “feedback describing training performance and/or model accuracy” or “loss and/or error function or other metrics”) of the machine learning algorithms while the algorithm is processing the set of images fed at step (d) (e.g., “training datasets”)).
Regarding claim 16, Lasaruk in view of Rowell discloses the method according to claim 13, wherein Lasaruk further discloses the physical digital image recording device is a digital image recording device of an automated driving and advanced safety system (para(s). [0001-0002]—see citations in claim 14 above).
Regarding claim 17, Lasaruk in view of Rowell discloses the method according to claim 1, wherein Lasaruk further discloses the method according to claim 1 further comprising measuring the optical quality function related to the optical distortions of the physical windshield to obtain the measured optical quality function (para(s). [0038] and [0084-0085]—see citations in step a of claim 1 above—, where the calculating the “windshield distortion” is measuring the optical quality function (i.e., “distortion”) related to the optical distortions of the physical windshield to obtain the measured optical quality function (i.e., “windshield distortion” model)).
Regarding claim 18, the claim recites similar limitations to claim 17 and is rejected for similar rationale and reasoning (see the analysis for claim 17 above).
Regarding claim 19, the claim recites similar limitations to claim 1—where the “input image of a landscape” as recited in claim 1 is the “reference input image of a landscape” as recited in claim 19—and is rejected for similar rationale and reasoning (see the analysis for claim 1 above).
Regarding claim 20, the claim recites similar limitations to claim 17 and is rejected for similar rationale and reasoning (see the analysis for claim 17 above).
Regarding claim 21, the claim recites similar limitations to claim 15—where the set of input images of landscapes representative of landscapes likely to be recorded by a physical digital image recording device through a physical windshield of said set of physical windshields” as recited in claim 15 is the “set of reference input images of landscapes” as recited in claim 21—and is rejected for similar rationale and reasoning (see the analysis for claim 1 above).
Regarding claim 22, the claim recites similar limitations to claim 17 and is rejected for similar rationale and reasoning (see the analysis for claim 17 above).
Regarding claim 23, the claim recites similar limitations to claim 17 and is rejected for similar rationale and reasoning (see the analysis for claim 17 above).
Claim 2 is rejected under 35 U.S.C. 103 as being unpatentable over Lasaruk in view of Rowell as applied to claim 1 above, and further in view of Seger et al. (Seger; US 2012/0206601 A1).
Regarding claim 2, Lasaruk in view of Rowell discloses the computer implemented method according to claim 1, wherein Lasaruk further discloses the measured optical quality function related to the optical distortions of the physical windshield is a measured transmitted wavefront error of the physical windshield, measured surfaces profiles and/or a measured(para(s). [0047] and [0049], recite(s)
[0047] “Another aspect of the invention is directed at an apparatus providing a mechanical setup for calibration of the windshield parameters. The physical parameters are preferably the tilt and / or the refractive index and / or the thickness of the windshield, especially preferably all of these parameters combined.”
[0049] “The new model significantly extends older camera models from the state of the art and approaches to explain the effects of the windshield. In contrast to the state of the art, the approach according to the invention comprises an extension of the generalized pinhole camera model, which is physically motivated. In particular, as an instance of the new approach, physical windshield parameters are obtained, like tilt, thickness, and refraction index within the new model. On the other hand, the new special model, e.g. for low curvature windshields, significantly specializes existing works. The new model may introduce dependency of projection on the distance of the scene points.”
, where the “refraction index within the new model” is a measured complex refractive index).
Where Lasaruk in view of Rowell does not specifically disclose
…a measured distribution of complex refractive index;
Seger teaches in the same field of endeavor of modelling optical distortions due to windshields
…a measured distribution of complex refractive index (para(s). [0027], recite(s)
[0027] “The index of refraction of the rectifying pane which is held by the mounting device preferably differs from the index of refraction of the windshield; a spatial distribution of the index of refraction in the rectifying pane may also differ from a distribution of the index of refraction in the windshield. The optical effect of the rectifying pane may be provided complementarily to the optical effect of the windshield, in particular in consideration of the angle between the rectifying pane and the windshield, through these different indices of refraction or their distributions.”
).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the presently filed invention to modify the system of Lasaruk in view of Rowell to incorporate the measured complex refractive index as a measured distribution of complex refractive index to improve the measured quality function by taking into account the spatial distribution of the index of refraction of the windshield as taught by Seger above.
Claim 4 is rejected under 35 U.S.C. 103 as being unpatentable over Lasaruk in view of Rowell as applied to claim 1 above, and further in view of Lu et al. (Lu; “Efficient Simulation of Lens Distortion in OpenGL,” 2008).
Regarding claim 4, Lasaruk in view of Rowell discloses the computer implemented method according to claim 1, wherein Rowell further teaches the input image of the landscape is digitally preprocessed as an environment map projected into an inside side of an environment(para(s). [0053], [0144], and [0164], recite(s)
[0053] “The virtual camera application 122 may include an image scene generator 124 for generating synthetic image scenes, and image projector 126 for projecting one or more camera views, and a graphics rendering engine 128 rendering synthetic images capture the one or more projected camera views…”
[0144] “Distortion centers 524 included in camera settings files 500 for virtual cameras may be c.sub.x and c.sub.y parameters describing the distortion center of projection in the image frame captured by the lens. Since actual camera devices modeled by one or more camera settings files may have different lens shapes, the distortion center denoted by c.sub.x and c.sub.y may not be positioned at the geometric center of the image frame. Irregular shaped lenses including aspheric and fisheye lenses do not have prefect circular symmetry, therefore the distortion centers 524 included in camera settings files 500 for virtual cameras setup with irregular shaped lenses are not a the geometric center of the image frame.”
[0164] “The graphics rendering engine 128 then renders projection coordinates of the image frame at the camera view as pixel coordinates of a synthetic image. By constructing a camera view of a virtual using the same parameters actual camera devices use to construct camera views in the real world, the image projector simulates performance of an actual camera device. Therefore, synthetic images 808 rendered by the graphics rendering engine 128 have the appearance of a virtual scene captured by an actual camera device. Synthetic images rendered by the graphics rendering engine 128 may include single images, stereo images including right and left stereo image pairs, or image sets including synthetic images capturing multiple camera views of a scene using camera intrinsics, camera calibration metadata, and camera capture settings for an actual camera device…”
, where “constructing a camera view of a virtual image using the same parameters actual camera devices use to construct camera in the real world” is projecting an environment map (e.g., the “virtual scene”) into an inside side of an environment centered on the virtual camera (e.g., “geometric center of the image frame”).
Where Lasaruk in view of Rowell does not specifically disclose
the input image… projected into an inside side of an environment sphere…;
Lu teaches in the same field of endeavor of image projections and optical distortions
the input image… projected into an inside side of an environment sphere… (Fig. 2.2(a) and section 2.2 on pgs. 6-7, recite(s)
[pg. 6] “If we put a curved optical lens in front of the projection plane, the direction of the projection rays will be changed and we will see a distorted view. Different curved surfaces of the lenses create different deformation effects. Figure 2.2 shows three curved surfaces to alter projection rays.”
[pg. 7] “In Figure 2.2(a) a half-spherical lens is replaced between camera COP and the projection plane. The projection rays emitted from the COP to the objects in 3D space are altered along all directions when they penetrate the half-spherical lens. The final image on the projection plane, which is made of the intersection (pixels) between the rays and the projection plane, suffers nonlinear distortion in all directions. The deformation level depends on many factors, including the sphere’s radius, focus, and other optical features.”
PNG
media_image1.png
386
552
media_image1.png
Greyscale
, where the “projection rays” are projected spherically resulting in a spherical distortion for at least a “half-spherical lens” (see Fig. 2.2(a) above) is projecting an image into an inside side of an environment sphere).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the presently filed invention to modify the system of Lasaruk in view of Rowell to incorporate digitally preprocessing the input image of the landscape as an environment map projected into an inside side of an environment sphere centered on the virtual camera to improve optical distortion simulation by incorporating simulations corresponding to spherical optical distortions created by digital recording devices with spherical optical lens as taught by Lu above.
Claims 5 and 6 are rejected under 35 U.S.C. 103 as being unpatentable over Lasaruk in view of Rowell as applied to claim 1 above, and further in view of Sakamoto et al. (Sakamoto; US 2017/0015180 A1).
Regarding claim 5, Lasaruk in view of Rowell discloses the computer implemented method according to claim 1, wherein, in step (a), Sakamoto teaches in the same field of endeavor of a digital recording device recording images through a windshield the sheet of transparent mineral glass is modelled so that said sheet of transparent mineral glass represents only a demarcated zone of the physical windshield in front of which a digital image recording device is placed (para(s). [0130] and [0137], recite(s)
[0130] “When a camera is used as the information acquisition device as described above, the imaging range (passage range Z of visible light or infrared rays) of the camera is adjusted as described below in an opening 25 of the center mask layer 22. That is, as shown in FIG. 16, a distance s3 between the peripheral edge of the imaging range of a camera 80 and the peripheral edge of the opening is preferably 4 mm or more, and more preferably 6 mm or more.”
[0137] “As shown in FIG. 19, the camera unit 4 is configured as described below. FIG. 19 is a cross-sectional view of the windshield. As shown in this diagram, the camera unit 4 includes a housing 41 to be fixed to the glass sheet 1 and a camera 42 (imaging device) that is incorporated in this housing 41. The housing 41 is arranged opposite to the center mask layer 22, and an opening 43 is formed at a position that is opposite to the opening 231 of the center mask layer 22. The incorporated camera 42 takes images forward of the vehicle through the opening 43 of the housing 41 and the opening 231 of the center mask layer 22. At this time, a visual field Z (field angle) of the camera 42 is adjusted so as to be substantially coincident with the opening 231 of the center mask layer 22. An image processing device 8 is connected to the camera 42, and images taken by the camera 42 are transmitted to the image processing device 8.”
, where the “opening in the glass layer” is a demarcated zone of the windshield).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the presently filed invention to modify the system of Lasaruk in view of Rowell to incorporate modelling only the demarcated zone of the windshield in front of which a digital image recording device is placed to improve the modelling accuracy by only modelling the portion of the windshield that the digital image recording device receives light from as taught by Sakamoto (para(s). [0006], recite(s)
[0006] “Such a problem may arise not only in devices for measuring a distance between vehicles but also in information acquisition devices in general that acquire information from the outside of a vehicle by receiving light emitted by optical beacons, for example. Therefore, the present invention was made in order to solve the foregoing problems, and it is an object thereof to provide a windshield to which an information acquisition device that emits and/or receives light through the opening of the mask layer can be attached, and that enables accurate information processing.”
).
Regarding claim 6, Lasaruk in view of Rowell discloses the computer implemented method according to claim 1, wherein, in step (a), Sakamoto teaches in the same field of endeavor of a digital recording device recording images through a windshield the sheet of transparent mineral is modelled so that the global illuminance calculated at step (c) is only the global illuminance arriving through the part of the modelled sheet of transparent mineral glass which corresponds to a demarcated zone of the windshield through which the physical digital image recording device records an image (para(s). [0130] and [0137]—see claim 5 above—, where a person of ordinary skill in the art before the effective filing date of the claimed invention would have recognized that the camera’s “visual field” being “substantially coincident with the opening” (i.e., the demarcated zone) will only capture the global illuminance arriving through that part of windshield corresponding to the demarcated zone).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the presently filed invention to modify the system of Lasaruk in view of Rowell to incorporate modelling the sheet of transparent mineral using only the global illuminance arriving through the part of the modelled sheet of transparent mineral glass which corresponds to a demarcated zone of the windshield to improve the modelling by only calculating the global illuminance received by the digital image recording device.
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Lasaruk in view of Rowell as applied to claim 1 above, and further in view of Schofield (US 2010/0214791 A1).
Regarding claim 8, Lasaruk in view of Rowell discloses the computer implemented method according to claim 1, wherein, before step (a), Schofield teaches in the same field of endeavor of capturing images through a windshield the input image of a landscape is digitally preprocessed first with a color filter array, and/or second with a demosaicing algorithm, in particular with a nearest-neighbor interpolation kernel (para(s). [0027] and [0030], recite(s)
[0027] “The accessory module or housing assembly 18 includes an outer housing 18 a that is removably attached to an attachment plate 19 , which is fixedly attached to the upper central region of the inside surface of the vehicle's windshield 20 such that the optical system axis 17 is substantially horizontal and substantially parallel to the vehicle's principal central axis. The housing assembly and attachment plate. Preferably, the housing assembly 18 is positioned at the windshield such that the light rays that pass through the optical system 12 to the image sensor 13 also pass through a portion of the vehicle's windshield swept by the vehicle's windshield wiper system.”
[0030] “In order to extract color information from the image or image data, one of a number of filter types, each able to transmit a particular band of wavelength, covers the active surface of each of the photosensor elements of the array. The most commonly used color filter pattern, and therefore the most economically available as a standard feature of a commercially available image sensor, is the Bayer pattern in which either a blue, green or red pass filter is applied to the surface of each of the image sensor photosensor elements, such as shown in FIG. 4. Alternate rows of the photosensor elements may be covered by alternating red and green filters and alternating blue and green filters such that every 2 by 2 block of photosensors within the array contains one red filter, one blue filter and two diagonally opposite green filters. Since each photosensor is filtered to measure only the blue, green or red light energy content of the portion of the image formed by the optical system at its active surface, it is necessary to combine the measured values of filtered light from each of the photosensors of a 2 by 2 block such that an interpolated red, green and blue color value (RGB) may be assigned to the center point of the block. The color value for each of the 2 by 2 blocks of the array is calculated, thus creating a 639 by 479 array of color values with the same spacing as the photosensor array. This array of color picture elements is commonly termed a pixel array. A variety of algorithms may be used to perform this interpolation, commonly termed demosaicing, and the particular algorithm may be selected depending on the particular the application, the desired final image quality and available computing power. For the AHB control system described herein, each pixel RGB value is derived by combining the red value, the average of the two green values, and the blue value of its associated four photosensor elements.”
, where the “color filter” is at least a color filter array for digitally preprocessing an image).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the presently filed invention to modify the system of Lasaruk in view of Rowell to incorporate digitally preprocessed the input image of a landscape first with at least a color filter array to improve color information in the input image of the landscape as taught by Schofield above.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Pável et al (“Distortion Estimation Through Explicit Modeling of the Refractive Surface,” 2019) discloses in sections 1 on pg. 2, 6.1 on pg. 7, and 7 on pg. 10:
[1 Introduction] “…Due to the refractive material, tracing is more complex: as light enters or leaves a denser media, it changes direction, resulting in deviations from the pinhole model; called image distortions, as shown in Fig. 1(a). We construct the forward model
f
θ
p
:
Ω
→
R
3
, where – knowing the camera parameters, the refractive media and scene characteristics, jointly denoted
θ
– we map a pixel
p
from the image
Ω
⊂
R
2
to a point in the scene. We implement this function as a raycasting algorithm – see Sect. 4 –, allowing us to generate images given a set of parameters. After constructing the forward model, by using model inversion, we fit the parameters of the refractive media to a set of observations, given as displaced points. We build an RBF-network [11] based parametric model of the thickness of the refractive media and use ML estimation [2] to infer the optimal parameters that generated the distortions.”
[6.1 Synthetic Dataset] “In a first set of experiments we applied the forward image generation model to render synthetic images. We set the parameters of the camera, refractive surface, and the checkerboard pattern to similar values as in the real-world experiment, as shown in Fig. 4(a).”
[7 Conclusions] “In our work we presented a model for geometric distortions caused by refractive surface being placed between the camera and the scene. Based on an explicit model of the refractive surface, we presented a forward generative model of the distortions and the image generation; the generating process used the ray-casting mechanism. We assumed a conic refractive surface and used an additive model for the imperfections of the surface; the used model was a restricted Radial Basis Function network. Using model inversion and automated differentiation, we estimated the refractive surface with a set of checkerboard calibration target images. We validated our algorithm on synthetic and real-world data, and analyzed the observed image distortions.”
Matsushita et al. (JP 2002312406 A) discloses in description, para(s). [0023]. [0025], and [0040]:
[0023] “…Furthermore, the computer 12 generates a virtual plane 21 on the opposite side of the eye point EP with respect to the geometric model (i.e., on the outer surface side of the geometric model) (step S5).”
[0025] “Next, the computer 12 performs calculations to obtain the perspective distortion angle using the shape model 10, the eye point EP, and the virtual plane 21.
The calculation procedure involves first determining the intersection of a straight line representing the direction in which virtual rays traveling from each of three virtual points PO, POa, and POb constituting a target 22 toward the eye point EP after being refracted by the outer surface SU2 and the inner surface SU1 of the geometric model 10 with the virtual plane 21 (step S6).”
[0040] “Furthermore, since the automobile windshield can be simulated at the design stage (i.e., before a mold such as a heat bending mold is actually manufactured), if it is found that the windshield has perspective distortion as seen by the driver that actually needs to be corrected, the mold can be manufactured to match that distortion, thereby eliminating the waste of having to remanufacture the automobile windshield mold.”
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JULIA Z YAO whose telephone number is (571)272-2870. The examiner can normally be reached Monday - Friday (8:30AM - 5PM).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Emily Terrell can be reached at (571)270-3717. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/J.Z.Y./Examiner, Art Unit 2666
/EMILY C TERRELL/Supervisory Patent Examiner, Art Unit 2666
Prosecution Insights