DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 1-3, 5-9, 11-20 is/are rejected under 35 U.S.C. 102(a) as being unpatentable over Gassend et al. (US Patent Number 2020/0142041 -A1, hereinafter “Gassend”)
Regarding claim 1, Gassend teaches: A method comprising :at an electronic device including a non-transitory memory, one or more processors, a display and an image sensor: (Fig. 1)
obtaining, via the image sensor, a plurality of images of a physical environment of the electronic device ([0102], "Camera 334 may be any camera (e.g., a still camera, a video camera, etc.) configured to capture images of the environment in which the vehicle 300 is located. To that end, camera 334 may take any of the forms described above.")
while the electronic device is moving; ([0107], "Navigation and pathing system 348 may be any system configured to determine a driving path for vehicle 300. Navigation and pathing system 348 may additionally be configured to update a driving path of vehicle 300 dynamically while vehicle 300 is in operation. In some embodiments, navigation and pathing system 348 may be configured to incorporate data from sensor fusion algorithm 344, GPS 326, LIDAR unit 332, and/or one or more predetermined maps so as to determine a driving path for vehicle 300.")
detecting an occlusion of the image sensor based on a repeated occurrence of a static feature across the plurality of images; ([00136], "The identification of the portion of the FOV at block 408 could be based on the first scan, the second scan, or both the first scan and the second scan. Thus, in some examples, a system of method 400 may be configured to identify the portion of the FOV at block 408 based on one or more scans of the FOV obtained by the LIDAR device via a single optical window. In other examples, a system of method 400 may be configured to identify the portion of the FOV at block 408 based on one or more scans of the FOV obtained by the LIDAR device via both the first optical window and the second optical window.")
and modifying a weight associated with the static feature to decrease an impact of the occlusion on a performance of a function. ([0145], "In some examples, method 400 involves determining whether the occlusion (of block 408) rotates with the housing. As noted above, in some instances, a system of method 400 may respond to the detection of an occlusion depending on a type of the occlusion. In one scenario, if the occlusion rotates with the housing, the occlusion might correspond to an object (e.g., dirt, debris, etc.) that is disposed on an optical window. However, in another scenario, the rotating occlusion might alternatively correspond to an object (e.g., plastic sheet, etc.) that is attached to a rotating component of the LIDAR device (e.g., rotating platform 210, housing 250, etc.), where a portion of the attached object extends near one or both optical windows (e.g., portion of a plastic bag attached to the housing but extending near one or both of the optical windows."; [0146], "In some examples, a system of method 400 may be configured to attempt removing the occlusion (e.g., using cleaning apparatus 160, etc.) in response to a determination that the occlusion rotates with housing.")
Regarding claim 2, Gassend teaches: The method of claim 1, wherein the static feature includes a set of one or more points. ([0200], "In some examples, method 600 involves generating a combined point cloud representation of the first scan and the second scan. For example, computer system 310 may combine the two scans of the FOV (e.g., the first scan and the second scan simultaneously obtained via, respectively, the first optical window and the second optical window) to generate a single 3D point cloud representation having a greater number of data points (e.g., higher resolution, etc.) as compared to a point cloud that represents only one of the two scans.")
Regarding claim 3, Gassend teaches: The method of claim 2, further comprising performing feature generation and extraction on each of the plurality of images to generate respective point clouds. ([0200], "In some examples, method 600 involves generating a combined point cloud representation of the first scan and the second scan. For example, computer system 310 may combine the two scans of the FOV (e.g., the first scan and the second scan simultaneously obtained via, respectively, the first optical window and the second optical window) to generate a single 3D point cloud representation having a greater number of data points (e.g., higher resolution, etc.) as compared to a point cloud that represents only one of the two scans.")
Regarding claim 5, Gassend teaches: The method of claim 1, wherein detecting the occlusion comprises determining that a number of occurrences of the static feature exceeds a threshold number of occurrences. ([0143], "In some examples, method 400 involves determining an extent to which the occlusion occludes the identified portion of the FOV. In one example, identifying the portion of the FOV at block 408 may be based on a determination that the occlusion intercepts at least a threshold number of light pulses. In one example, the threshold number may be 10,000 pulses (e.g., in a system that requires detection of at least 10,000 pulses from the portion of the FOV, etc.). Other threshold numbers are possible as well.")
Regarding claim 6, Gassend teaches: The method of claim 1, wherein detecting the occlusion comprises detecting the occlusion when an ambient lighting level is less than a threshold lighting level. ([0143], "In some examples, method 400 involves determining an extent to which the occlusion occludes the identified portion of the FOV. In one example, identifying the portion of the FOV at block 408 may be based on a determination that the occlusion intercepts at least a threshold number of light pulses. In one example, the threshold number may be 10,000 pulses (e.g., in a system that requires detection of at least 10,000 pulses from the portion of the FOV, etc.). Other threshold numbers are possible as well."; Examiner's Note - Light produced by the device in prior art is a part of the ambient light.)
Regarding claim 7, Gassend teaches: The method of claim 1, wherein detecting the occlusion comprises detecting the occlusion when an infrared (IR) illuminator is located within a threshold distance of the image sensor. ([0042], "In a second example, where system 100 is configured as an active infrared (IR) camera, transmitter 120 may include one or more emitters 122 configured to emit IR radiation to illuminate a scene. To that end, transmitter 120 may include any type of emitter (e.g., light source, etc.) configured to provide the IR radiation."; [0145], "In some examples, method 400 involves determining whether the occlusion (of block 408) rotates with the housing. As noted above, in some instances, a system of method 400 may respond to the detection of an occlusion depending on a type of the occlusion. In one scenario, if the occlusion rotates with the housing, the occlusion might correspond to an object (e.g., dirt, debris, etc.) that is disposed on an optical window. However, in another scenario, the rotating occlusion might alternatively correspond to an object (e.g., plastic sheet, etc.) that is attached to a rotating component of the LIDAR device (e.g., rotating platform 210, housing 250, etc.), where a portion of the attached object extends near one or both optical windows (e.g., portion of a plastic bag attached to the housing but extending near one or both of the optical windows.")
Regarding claim 8, Gassend teaches: The method of claim 1, further comprising classifying the occlusion into one of a plurality of occlusion types. ([0145], "In some examples, method 400 involves determining whether the occlusion (of block 408) rotates with the housing. As noted above, in some instances, a system of method 400 may respond to the detection of an occlusion depending on a type of the occlusion. In one scenario, if the occlusion rotates with the housing, the occlusion might correspond to an object (e.g., dirt, debris, etc.) that is disposed on an optical window. However, in another scenario, the rotating occlusion might alternatively correspond to an object (e.g., plastic sheet, etc.) that is attached to a rotating component of the LIDAR device (e.g., rotating platform 210, housing 250, etc.), where a portion of the attached object extends near one or both optical windows (e.g., portion of a plastic bag attached to the housing but extending near one or both of the optical windows."
Regarding claim 9, Gassend teaches: The method of claim 8, wherein classifying the occlusion comprises classifying the occlusion based on an intensity of the static feature. ([0139], "Accordingly, in some examples, a system of method 400 may be configured to determine a likelihood that the occlusion is physically coupled to the LIDAR device (e.g., attached to the LIDAR device, or attached to another nearby structure, etc.), the extent of the occlusion, and/or whether the occlusion is likely to remain physically coupled to the LIDAR device if no responsive action is taken (e.g., without activating a cleaning apparatus, etc.). For instance, the system can make these determinations by assessing various factors such as: returning light pulse intensities/numbers, prior information about the prevalence of a certain type of occlusion in a region of the environment where the LIDAR device is currently located, a speed of a vehicle on which the LIDAR device is mounted, and/or corroborating data from other sensors, among other possible factors.")
Regarding claim 11, Gassend teaches: The method of claim 1, wherein modifying the weight associated with the static feature comprises lowering the weight of the static feature. ([0145], "In some examples, method 400 involves determining whether the occlusion (of block 408) rotates with the housing. As noted above, in some instances, a system of method 400 may respond to the detection of an occlusion depending on a type of the occlusion. In one scenario, if the occlusion rotates with the housing, the occlusion might correspond to an object (e.g., dirt, debris, etc.) that is disposed on an optical window. However, in another scenario, the rotating occlusion might alternatively correspond to an object (e.g., plastic sheet, etc.) that is attached to a rotating component of the LIDAR device (e.g., rotating platform 210, housing 250, etc.), where a portion of the attached object extends near one or both optical windows (e.g., portion of a plastic bag attached to the housing but extending near one or both of the optical windows."; [0146], "In some examples, a system of method 400 may be configured to attempt removing the occlusion (e.g., using cleaning apparatus 160, etc.) in response to a determination that the occlusion rotates with housing.")
Regarding claim 12, Gassend teaches: The method of claim 1, wherein modifying the weight associated with the static feature comprises discarding the static feature. ([0137], "By way of example, a system of method 400 may monitor first respective numbers of light pulses transmitted toward respective portions of the FOV by the LIDAR device, and second respective numbers of reflected light pulses detected by the LIDAR device from the respective portions. In some instances, the system could also monitor numbers of transmitted/reflected light pulses transmitted through the first optical window (e.g., the first scan) separately from those transmitted through the second optical window (e.g., the second scan). Depending on a variety of factors, the system can use the monitored numbers to decide if a particular portion of the FOV is occluded.")
Regarding claim 13, Gassend teaches: The method of claim 1, further comprising prompting to clean the image sensor when the occlusion satisfies a cleaning criterion. ([0146], "In some examples, a system of method 400 may be configured to attempt removing the occlusion (e.g., using cleaning apparatus 160, etc.) in response to a determination that the occlusion rotates with housing.")
Regarding claim 14, Gassend teaches: The method of claim 1, further comprising prompting to replace the electronic device when the occlusion satisfies a replacement criterion. ([0037], "For instance, the third actuator can be used to move or replace a filter or other type of optical element 140 along an optical path of an emitted light pulse, or can be used to tilt rotating platform (e.g., to adjust the extents of a field-of-view (FOV) scanned by system 100, etc.), among other possibilities.")
Regarding claim 15, Gassend teaches: The method of claim 1, wherein the function comprises a localization and mapping operation. ([0106], "To that end, computer vision system 346 may use an object recognition algorithm, a Structure from Motion (SFM) algorithm, video tracking, or other computer vision techniques. In some embodiments, computer vision system 346 may additionally be configured to map the environment, track objects, estimate the speed of objects, etc.")
Regarding claim 16, Gassend teaches: The method of claim 15, wherein the localization and mapping operation includes generating or updating a map of the physical environment. ([0106], "To that end, computer vision system 346 may use an object recognition algorithm, a Structure from Motion (SFM) algorithm, video tracking, or other computer vision techniques. In some embodiments, computer vision system 346 may additionally be configured to map the environment, track objects, estimate the speed of objects, etc.")
Regarding claim 17, Gassend teaches: The method of claim 15, wherein the localization and mapping operation includes tracking a location of an object in the physical environment. ([0106], "To that end, computer vision system 346 may use an object recognition algorithm, a Structure from Motion (SFM) algorithm, video tracking, or other computer vision techniques. In some embodiments, computer vision system 346 may additionally be configured to map the environment, track objects, estimate the speed of objects, etc.")
Regarding claim 18, Gassend teaches: The method of claim 15, wherein the localization and mapping operation includes tracking a location of an object in the physical environment. ([0106], "To that end, computer vision system 346 may use an object recognition algorithm, a Structure from Motion (SFM) algorithm, video tracking, or other computer vision techniques. In some embodiments, computer vision system 346 may additionally be configured to map the environment, track objects, estimate the speed of objects, etc.")
Regarding claim 19, claim 19 has been analyzed with regard to claim 1 and is rejected for the same reasons of obviousness as used above as well as in accordance with Gassend further teaching on: An electronic device comprising:a display;an image sensor; one or more processors; a non-transitory memory; and one or more programs stored in the n on-transitory memory, which, when executed by the one or more processors, cause the device to: (Fig. 1)
Regarding claim 20, claim 20 has been analyzed with regard to claim 1 and is rejected for the same reasons of obviousness as used above as well as in accordance with Gassend further teaching on: A non-transitory memory storing one or more programs, which, when executed by one or more processors of an electronic device with an image sensor, cause the device to: (Fig. 1)
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim 4 is/are rejected under 35 U.S.C. 103 as being unpatentable over Gassend et al. (US Patent Number 2020/0142041 -A1, hereinafter “Gassend”) in view of Howard et al. (US Patent Number 2015/0116734-A1, hereinafter “Howard”)
Regarding claim 4, Gassend does not teach: The method of claim 1, wherein detecting the occlusion comprises utilizing a two-dimensional (2D) look-up table (LUT) to track occurrences of features across the plurality of images.
However, Howard does teaches: The method of claim 1, wherein detecting the occlusion comprises utilizing a two-dimensional (2D) look-up table (LUT) to track occurrences of features across the plurality of images. (Howard, [0075], "In some embodiments, the cross-track and in-track displacement functions can be represented using 2D look-up tables (2D LUTS), which indicate the displacement (.DELTA.x, .DELTA.y) for a lattice of (x,y) image positions.")
At the time the invention was made, it would have been obvious to one of ordinary skill in the art to modify occlusion detection (as taught by Gassend) to include look-up table for feature tracking (as taught by Howard) because such a modification is the result of simple substitution of one known element for another producing a predictable result. More specifically, the storage of data based on features and 2D look-up tables perform the same general and predictable function, the predictable function being able to track features and related data.Since each individual element and its function are shown in the prior art, albeit shown in separate references, the difference between the claimed subject matter and the prior art rests not on any individual element or function but in the very combination itself - that is in the substitution of storage of data based on features by replacing it with look up table. Thus, the simple substitution of one known element for another producing a predictable result renders the claim obvious.
Claim 10 is/are rejected under 35 U.S.C. 103 as being unpatentable over Gassend et al. (US Patent Number 2020/0142041 -A1, hereinafter “Gassend”) in view of Tariq et al. (US Patent Number 2021/0201464-A1, hereinafter “Tariq”)
Regarding claim 10, Gassend does no teach: The method of claim 8, wherein classifying the occlusion comprises classifying the occlusion based on a shape of the static feature.
However, Tariq does teach: The method of claim 8, wherein classifying the occlusion comprises classifying the occlusion based on a shape of the static feature. (Tariq, [0086], " Intermediate images 702a, 702b, and 702c (collectively “images 702”) illustrate examples of the techniques described above for detecting degradations within image data in which regions of potential degradation 704 are indicated by the outlined shapes. The regions of potential degradation 704 may be associated with a probability that there is a degradation at the particular region. The probabilities may be based on the outputs of the one or more degradation detection techniques applied.")
At the time the invention was made, it would have been obvious to one of ordinary skill in the art to modify occlusion detection (as taught by Gassend) to include occlusion classification based on shape (as taught by Tariq) because such a modification is the result of combining prior art elements according to known methods to yield predictable results. More specifically, occlusion detection as modified by occlusion classification based on shape can yield a predictable result of which regions have occluded vision. Thus, a person of ordinary skill would have appreciated including in occlusion detection the ability to do occlusion classification based on shape since the claimed invention is merely a combination of old elements, and in the combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that the results of the combination were predictable.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Jinsu Hwang whose telephone number is (703)756-1370. The examiner can normally be reached Mon -Thu 10am-8am EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Matthew Bella can be reached at (571) 272-7778. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JINSU HWANG/Examiner, Art Unit 2667
/MATTHEW C BELLA/Supervisory Patent Examiner, Art Unit 2667
Prosecution Insights