Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Claims
2. This is a final action on the merits in response to the reply received 3/12/2026.
Response to Arguments
Applicant’s arguments have been considered but are moot in view of new grounds of rejections.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-3, 5-6, 10-11, 13, 15-20 are rejected under 35 U.S.C. 103 as being unpatentable over US 20150009296 A1-Crona, in view of US 20140232869 A1-May et al (Hereinafter referred to as “May”), in further view of US 20190385025 A1-McMichael et al (Hereinafter referred to as “McMichael”), in further view of US 20210076926 A1-Glik et al (Hereinafter referred to as “Glik”).
Regarding claim 1, Crona discloses a method ([0003], relates to a method) comprising:
accessing first imagery from a first lens and second imagery from a second lens, the first lens and the second lens being components of a stereo camera (abstract, provided with a first camera providing first images of said capturing area and a second camera providing second images. In addition, [0048], stereoscopic camera wherein comprises two cameras, a first camera and a second camera which in this example are placed next to each other such that their respective lenses are slightly spaced apart. Also see [0050]);
detecting a disparity between the first imagery and the second imagery ([0048], wherein the differences between first images from the first camera and second images from the second camera can be used to calculate distances. The examiner notes that a disparity is nothing more than a difference; [0051], wherein the span between the minimum and maximum values Bmin2, Bmax2 is always shorter for a contaminated evaluation area compared to a corresponding clean evaluation area. If there is a deviation between the historical image data for two corresponding evaluation areas, i.e. one in the first image and the other in the second image, it can be concluded that a lens of the stereoscopic camera is contaminated.);
identifying an obstruction on the first lens based on the detected disparity [0051], wherein the span between the minimum and maximum values Bmin2, Bmax2 is always shorter for a contaminated evaluation area compared to a corresponding clean evaluation area. If there is a deviation between the historical image data for two corresponding evaluation areas, i.e. one in the first image and the other in the second image, it can be concluded that a lens of the stereoscopic camera is contaminated.)
Crona fails to disclose in detail generating a visual output associated with the stereo camera based on the obstruction; and causing display of the generated visual output.
However, in the same field of endeavor, May discloses accessing first imagery from a first lens and second imagery from a second lens ([0005], wherein processing image data captured by two cameras;[0018], discloses one or more lens associated with one or more cameras), detecting a disparity between the first imagery and the second imagery ([0048], wherein the dirt detection algorithm comprises a pixel based image processing algorithm applied on input images and on some more images derived from input frames, such as difference images and contrast maximum images; [0056], wherein these images are the difference image, taken to a frame with n-frame distance to the current frame, the contrast maximum image and the color space diagonal distance image. The examiner notes that a disparity is nothing more than a difference between images); identifying an obstruction on the first lens based on the detected disparity[0036-0037]); generating a visual output associated with the cameras based on the obstruction ([0020], wherein if the dirt detection system identifies such locations at a more or less reliable level, the software or system is operable to generate an alert or warning, such as a visual or optical alert or warning and/or an audible or acoustical alert or warning to the user); and causing display of the generated visual output {[0064], wherein generate an alert to the driver of the vehicle and/or may generate an overlay at the displayed image. The image is being displayed along with the alert. Hence the alert is being caused to display).
Therefore, it would have been obvious to one of ordinary skilled in the art before the effective filing date of the claimed invention to modify the method disclosed by Crona to disclose generating a visual output associated with the stereo camera based on the obstruction; and causing display of the generated visual output as taught by May, to provide enhanced image processing of image data captured by cameras by recognizing obstructions on a lens of multiple cameras ([0025], May).
Crona and May fail to disclose identifying the obstruction comprises determining that a portion of the first imagery is not changing over time despite movement of the stereo camera and wherein the artificial neural network leverages a feature vector that includes output of the disparity detection engine.
However, in the same field of endeavor, McMichael discloses identifying the obstruction comprises determining that a portion of the first imagery is not changing over time despite movement of the stereo camera ([0024], wherein another feature does not move relative to the sequence of images, it may be an indication that the non-moving (stationary) feature is an obstruction) wherein the obstruction is identified using an artificial neural network ([0020], wherein neural network is used to detect obstruction), wherein the artificial neural network leverages a feature vector that includes output of the disparity detection engine (McMichael detects obstruction of a lens or mirror by taking a difference between object, the obstruction system may determine the presence of the obstruction as taught in [0096]. In addition, McMichael may determine presence of obstruction by comparing map data derived from the difference of images. In addition, [0103] discloses calculating difference and using those differences from images to determine obstruction. [0154] discloses comparing the differences between an object detected by multiple cameras. McMichael discloses identifying features from these images to determine obstruction in [0097]. The neural network in McMichael uses a memory that includes one or more maps as discloses in [0060]. These maps include texture information, spatial information. The examiner would like to note that disparity maps are nothing more than spatial information. The types of neural networks that are utilized are any type, not limited to Convolutional Neural Network (CNN), SVM (support vector machine), supervised learning, unsupervised learning, semi-supervised learning, ResNet70, ResNet101, VGG, DenseNet, PointNet, etc as taught in [0065-0066])
Therefore, it would have been obvious to one of ordinary skilled in the art before the effective filing date of the claimed invention to modify the method disclosed by Crona and May to disclose identifying the obstruction comprises determining that a portion of the first imagery is not changing over time despite movement of the stereo camera as taught by McMichael, to improve the effectiveness of the obstruction mitigation ([0035], McMichael) and wherein the obstruction is identified using an artificial neural network, wherein the artificial neural network leverages a feature vector that includes output of the disparity detection engine. Therefore, it would have been obvious to a person of ordinary skill to feed disparity data into a CNN because processes arbitrary feature vectors and the outcome would be predictable. In other words, Cnns taught in McMichael are well known for processing feature vectors of any input. The disparity information taught in Chrona and May combined with the difference and spatial information of McMichael is just another type of feature vector. This combination would enhance image analyses.
While the combined references teach a first imagery from a first lens and a second imagery from a second lens, Crona, May, and McMichael fail to expliclty disclose in detail wherein generating the visual output comprises removing the obstruction from the first imagery
by superimposing corresponding imagery from the second lens.
However, in the same field of endeavor, Glik discloses removing the obstruction from the first imagery by superimposing corresponding imagery ([0058], wherein during image processing the retinal images may be combined (e.g. image stacking is superimposing) to exclude deleterious image inclusions from the retinal images.
Therefore, it would have been obvious to one of ordinary skilled in the art before the effective filing date of the claimed invention to modify the method disclosed by Crona, May, and McMichael to disclose wherein generating the visual output comprises removing the obstruction from the first imagery by superimposing corresponding imagery from the second lens as taught by Glik , to ensure adequate image quality (Gilk, [0058]).
Regarding claim 2, May discloses the method of claim 1, wherein generating the visual output comprises: presenting the second imagery in the visual output ([0071], wherein displaying one or more images).
Regarding claim 3, May discloses the method of claim 1, wherein generating the visual output comprises: presenting the first imagery in the visual output to prompt a user to clean the first lens ([0019], wherein provide an alert or warning, warning (or confidence) level or parameter to the driver or to the image processing control, and that is operable to have the image processing algorithm lower or compensate the impairment of dirt on the vision system's camera lenses. The dirt detection system, responsive to detection of dirt at the camera or lens, may trigger a desired function, such as a camera cleaning cycle prompt) .
Regarding claim 5, May discloses the method of claim 1, wherein generating the visual output comprises: augmenting the first imagery to remove the obstruction ([0060-61], wherein suitable fractions may be cropped from previous images, shrunk, turned and distorted according to the projection plane (known motion compensation) to be blended over one or accordingly several regions identified to be statically covered by a blob of dirt by the dirt detection algorithm earlier. An example is shown in FIG. 6, where a fraction 25 of the source image taken earlier (tn-1) than the currently processed image (tn) becomes superimposed to the spots 26 covered by (identified) dirt blobs 20 in the compensated output image. The same dirt blobs 20 cover other areas in the source image (tn-1) than these 20 at image (tn)….. The blobs may blend in part by successive covering blobs)
Regarding claim 6, May discloses the method of claim 5, wherein augmenting the first imagery comprises: identifying a portion of the second imagery corresponding to the obstruction ([0060], wherein dirt spots are identified); modifying the first imagery based on the portion of the second imagery, wherein modifying the first imagery comprises: superimposing the portion of the second imagery into the first imagery ([0060], wherein suitable fractions may be cropped from previous images, shrunk, turned and distorted according to the projection plane (known motion compensation) to be blended over one or accordingly several regions identified to be statically covered by a blob of dirt by the dirt detection algorithm earlier. An example is shown in FIG. 6, where a fraction 25 of the source image taken earlier (tn-1) than the currently processed image (tn) becomes superimposed to the spots 26 covered by (identified) dirt blobs 20 in the compensated output image. The same dirt blobs 20 cover other areas in the source image (tn-1) than these 20 at image (tn)); and blending an edge of the superimposed portion of the second imagery to reduce a contrast along the edge ([0060-0061, The blobs may blend in part by successive covering blobs).
Regarding claim 10, May discloses the method of claim 1, wherein identifying the obstruction is based, at least in part, on a portion of the first imagery not changing upon movement of the first lens ([0040], wherein recognize dirt locations at the lens in real camera systems using millions of pixels).
Regarding claim 11, May discloses the method of claim 1, further comprising: identifying a classification of the obstruction based on at least one of a color, a transparency, or a translucency of the obstruction; and providing an output representing the identified classification ([0028]).
Regarding claim 13, May discloses the method of claim 1, wherein the obstruction obstructs a portion of a scene viewed by the first lens and reduces a quality of the first imagery ([0018], wherein dirt may impair the view. Impairing something will reduce the quality of that something).
Regarding claim 15, analyses are analogous to those presented for claim 1 and are applicable for claim 15.
Regarding claim 16, analyses are analogous to those presented for claim 2 and are applicable for claim 16.
Regarding claim 17, analyses are analogous to those presented for claim 3 and are applicable for claim 17.
Regarding claim 18, analyses are analogous to those presented for claim 1 and are applicable for claim 18, wherein processing circuitry (Crona, [0048]; memory ([Crona, [0028]).
Regarding claim 19, analyses are analogous to those presented for claim 1 and are applicable for claim 19.
Regarding claim 20, May discloses the system of claim 18, further comprising: a display unit to display the generated visual output ([0016]). Tutput
Regarding claim 21, McMichael discloses the method of claim 1, wherein the artificial neural network predicts a cause of the obstruction ([0064-0065, neural network predict obstruction as taught in [0090])).
Claim(s) 4, 8-9 are rejected under 35 U.S.C. 103 as being unpatentable over US 20150009296 A1-Crona, in view of US 20140232869 A1-May et al (Hereinafter referred to as “May”), in further view of US 20190385025 A1-McMichael et al (Hereinafter referred to as “McMichael”), in further view of US 20210076926 A1-Glik et al (Hereinafter referred to as “Glik”), in further view of US 20140293079 A1-Milanfar et al (Hereinafter referred to as “Mil”).
Regarding claim 4, May discloses the method of claim 3 (See claim 3),
Crona and May fail to explicitly disclose in detail presenting text prompting the user to clean the first lens overlaying the first imagery.
However, in the same field of endeavor, Mil discloses presenting text prompting the user to clean the first lens overlaying the first imagery ([0068], wherein text notification indicating lens needs to be cleaned)
Therefore, it would have been obvious to one of ordinary skilled in the art before the effective filing date of the claimed invention to modify the method disclosed by Crona and May to disclose presenting text prompting the user to clean the first lens overlaying the first imagery as taught by Mil, to provide enhanced image by removing obstructions on cameras ([0002], Mil).
Regarding claim 8, May discloses the method of claim 1 (See claim 1),
Crona and May fail to disclose causing cleaning of the first lens in response to identifying the obstruction; determining, after cleaning of the first lens, that the obstruction remains present; and providing an output for servicing the stereo camera to remove the obstruction.
However, in the same field of endeavor, Mil discloses causing cleaning of the first lens in response to identifying the obstruction; determining, after cleaning of the first lens, that the obstruction remains present; and providing an output for servicing the stereo camera to remove the obstruction (indicator 602 in FIG. 6A represents a camera with a handprint smudge, indicating that the camera needs cleaning. Indicator 602 may remain on the display until the computing device 600 determines that the view of the camera is substantially unobstructed. Once the computing device 600 determines that the view of the camera is substantially unobstructed, the computing device may provide instructions to remove the indicator).
Therefore, it would have been obvious to one of ordinary skilled in the art before the effective filing date of the claimed invention to modify the method disclosed by Crona and May to disclose causing cleaning of the first lens in response to identifying the obstruction; determining, after cleaning of the first lens, that the obstruction remains present; and providing an output for servicing the stereo camera to remove the obstruction as taught by Mil, to provide enhanced image by removing obstructions on cameras ([0002], Mil).
Regarding claim 9, Mil discloses the method of claim 8, wherein the output for servicing the stereo camera comprises text overlaying a displayed image, the text comprising a prompt for a user to service the stereo camera ([0068], wherein text notification indicating lens needs to be cleaned)
Claim(s) 7 is rejected under 35 U.S.C. 103 as being unpatentable over US 20150009296 A1-Crona, in view of US 20140232869 A1-May et al (Hereinafter referred to as “May”), in further view of US 20190385025 A1-McMichael et al (Hereinafter referred to as “McMichael”), in further view of US 20210076926 A1-Glik et al (Hereinafter referred to as “Glik”), in further view of US 20170193679 A1-Wu et al (Hereinafter referred to as “WU”).
Regarding claim 7, May discloses the method of claim 6 (See claim 6),
Crona and May fail to disclose wherein modifying the first imagery further comprises: adjusting at least one of a tone, a hue, or a saturation of the superimposed portion of the second imagery based on at least one of the tone, the hue, or the saturation of the first imagery.
However, in the same field of endeavor, Wu discloses wherein modifying the first imagery further comprises: adjusting at least one of a tone, a hue, or a saturation of the superimposed portion of the second imagery based on at least one of the tone, the hue, or the saturation of the first imagery [0076], wherein adjusting the tones in the boundary (portion) between real space image and additional image(first and second imagery) because when the additional image is displayed by the presentation processing unit in a manner superimposed on the real-space image, there is a large difference in the color tones between the real-space image and the additional image).
Therefore, it would have been obvious to one of ordinary skilled in the art before the effective filing date of the claimed invention to modify the method disclosed by Crona and May to disclose wherein modifying the first imagery further comprises: adjusting at least one of a tone, a hue, or a saturation of the superimposed portion of the second imagery based on at least one of the tone, the hue, or the saturation of the first imagery as taught by Wu, to provide enhanced image by using a system that adjusts color tones based on one image to conform with another image ([0076], Wu).
Claim(s) 12 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over US 20150009296 A1-Crona, in view of US 20140232869 A1-May et al (Hereinafter referred to as “May”), in further view of US 20190385025 A1-McMichael et al (Hereinafter referred to as “McMichael”), in further view of US 20190025773 A1-Yang et al (Hereinafter referred to as “Yang”).
Regarding claim 12, May discloses the method of claim 1 (See claim 1),
Crona and May fail to disclose wherein detecting the disparity and identifying the obstruction leverages an image processing engine, wherein the image processing engine includes at least one artificial neural network.
However, in the same field of endeavor, Yang discloses wherein detecting the disparity and identifying the obstruction leverages an image processing engine, wherein the image processing engine includes at least one artificial neural network ([0049])
Therefore, it would have been obvious to one of ordinary skilled in the art before the effective filing date of the claimed invention to modify the method disclosed by Crona and May to disclose wherein detecting the disparity and identifying the obstruction leverages an image processing engine, wherein the image processing engine includes at least one artificial neural network as taught by Yang, to provide a solution for weaknesses in systems that fails to have the ability to detect obstruction ([0048], Yang).
Regarding claim 14, analyses are analogous to those presented for claim 12 and are applicable for claim 14.
Claim(s) 22 rejected under 35 U.S.C. 103 as being unpatentable over US 20150009296 A1-Crona, in view of US 20140232869 A1-May et al (Hereinafter referred to as “May”), in further view of US 20190385025 A1-McMichael et al (Hereinafter referred to as “McMichael”), in further view of US 20210076926 A1-Glik et al (Hereinafter referred to as “Glik”), in further view of US 20220217287 A1-Adiri et al (Hereinafter referred to as “Adiri”)
Regarding claim 22. Crona discloses the method of claim 1 (see claim 1),
Crona, May, and McMichael fail to disclose wherein generating the visual output comprises: using an image completion technique to fill an obstructed part of the first imagery with content; and using a generative pretrained transformer engine to generate a prediction for the content based on the second imagery.
However, in the same field of endeavor, Adiri discloses using an image completion technique to fill an obstructed part of the first imagery with content ([0156], wherein an inpainting algorithm may refer to an algorithm which may fill in missing parts of an image to present a complete image); and using a generative pretrained transformer engine to generate a prediction for the content based on the second imagery ([0191], wherein a machine learning model (for example, a generative model, such as generative adversarial network, transformers based generative model, etc.) may be trained using training examples to generate images).
. Therefore, it would have been obvious to one of ordinary skilled in the art before the effective filing date of the claimed invention to modify the method disclosed by Crona, May, and McMichael to disclose wherein generating the visual output comprises: using an image completion technique to fill an obstructed part of the first imagery with content; and using a generative pretrained transformer engine to generate a prediction for the content based on the second imagery as taught by Adiri, to improve the quality of images ([0217], Adiri).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to LERON BECK whose telephone number is (571)270-1175. The examiner can normally be reached M-F 8 am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, David Czekaj can be reached at (571) 272-7327. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
LERON . BECK
Examiner
Art Unit 2487
/LERON BECK/Primary Examiner, Art Unit 2487
Prosecution Insights