DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 12/16/2025 has been entered.
Status of Claims
This Office action is in reply to filing by applicant on 12/16/2025.
Claims 1, 9, and 18 have been amended by Applicant.
Claim 11 was previously presented by Applicant.
Claims 2 and 10 were cancelled by Applicant.
Claims 3 – 8, 12 – 17, 19, and 20 remain as original.
Claims 1, 3 – 9, and 11 – 20 are currently pending and have been examined.
The prior 35 USC 101 claim rejections set forth in the Final rejection of 09/16/2025 as to claims 1, 3 – 9, and 11 – 20 are maintained in view of Applicant's arguments and amendments.
The prior 35 USC 103 claim rejections set forth in the Final rejection of 09/16/2025 as to claims 1, 3 – 9, and 11 – 20 are maintained in view of Applicant's arguments and amendments.
This action is made non-final.
Response to Arguments
With regard to the limitations of claims 1, 3 – 9, and 11 – 20, Applicant argues that the claims as amended are patent eligible under 35 USC 101 because they meet the analysis set forth by the Supreme Court. Remarks 12 – 20. Examiner respectfully disagrees. The subject claims noted were analyzed pursuant to MPEP 2106, et seq., and are still considered ineligible. Step 1 is met because the claims are directed towards one of the four statutory categories; Part 2A-Prong1 of the test is trying to evaluate if the claims recite a judicial exception (an abstract idea enumerated in the MPEP 2106.04(a)); Part 2A-Prong 2 is to evaluate whether the subject claims recite additional elements that integrate the exception into a practical application, and, lastly, Part 2B checks whether the claims amount to significantly more than the abstract idea. A detailed and formal analysis pursuant to 35 USC 101 as the same applies to the amended claim set will follow below.
As respects 35 USC 101, the claims as a whole amount to a drafting effort designed to monopolize the exception. The additional limitations when taken individually and in combination are not sufficient to amount to significantly more than the judicial exception. The MPEP sets forth several ways to positively demonstrate that claim limitations do not integrate a judicial exception into a practical application, such as:
Merely reciting the words “apply it” (or an equivalent) with the judicial exception, or merely including instructions to implement an abstract idea on a computer, or merely using a computer as a tool to perform an abstract idea, as discussed in MPEP § 2106.05(f);
Adding insignificant extra-solution activity to the judicial exception, as discussed in MPEP § 2106.05(g); and
Generally linking the use of a judicial exception to a particular technological environment or field of use, as discussed in MPEP § 2106.05(h).
Applicant’s actual arguments begin at page 14 of its Remarks. There, Applicant argues pursuant to 35 USC 101:
As recited in independent claim 1, the detection of the contaminated portion of a lens of the at least one camera is performed by a contamination detection model of the electronic device. Additionally, the determination of whether an operation of the electronic device is hindered by the contaminated portion, in response to the contamination detection model detecting the contaminated portion in the lens of the at least one camera, is performed by a processor. The detection of the contaminated portion of a lens of the at least one camera is clearly not performed in the human mind.
Accordingly, Applicants respectfully submit that the above-noted claimed features are not, and/or would/could not be, practically performed in the human mind and/or correspond to mental activities, since the claim specifically recites (1) the contaminated portion of the lens is detected by the contamination detection model, and (2) a processor determines, by reading the contamination detection model, whether an operation of the electronic device is hindered by the contaminated portion.
Examiner respectfully disagrees with the above Applicant assessment. The above italicized portion of Applicant’s response actually far overstates which portions of the claims (of 6/17/25) examiner established (in the last Final rejection, 9/16/25) as representing an abstract idea, one that could easily be performed in one’s mind (mental process). Namely. the actual claim passages as above that constituted an abstract idea were:
- detect a contaminated portion of a lens of the at least one camera, in response to the image being input;
and
- determine whether an operation of the electronic device is hindered by the contaminated portion, in response to the contamination detection model detecting the contaminated portion in the lens of the at least one camera,
Whether any task claimed as above can be performed by a processor / computer is not the present question per 35 USC 101. The question rather is whether those same segregated claim tasks could be mentally performed in one’s mind without using generic computer equipment, including a processer. In a phrase, they can. As previously noted in the last Final Rejection:
For example, the claim encompasses a person looking at the lens of a camera for a contaminated region, then determining whether any photo taken by that camera is hindered due to the contaminated region of the lens.
Applicant next argues per 35 USC 101 the “Hannun” case before the Board is on point. Remarks 16. Respectfully, it is not. The “Hannun” case involved very specific, computer related improvements to the technical field of speech recognition. To wit:
[the Board stated] We disagree with the Examiner that the claims recite either a method of organizing human activity or a mental process. While transcription generally can be performed by a human, the claims here are directed to a specific implementation including the steps of normalizing an input file, generating a jitter set of audio files, generating a set of spectrogram frames, obtaining predicted character probabilities from a trained neural network and decoding a transcription of the input audio using the predicted character probability outputs. These are not steps that can practically be performed mentally." PTAB noted that the claims were directed to a specific implementation comprising technical elements including AI and computer speech recognition and most importantly noted the importance of the specification describing how the claimed invention provides an improvement to the technical field of speech recognition …
The “Hannun” case made very specific improvements to computer technology. Here, no argument has ever been presented that any of the claim sets presented herein improve the generic computer related technology of detecting a flaw in a camera lens.
Applicant next argues per 35 USC 101 Prong II of the 2019 Revised 101 Guidance that the claims are patent eligible. Remarks 18. Examiner respectfully disagrees, and further notes that this eligibility argument morphs into the exact same mental process abstract idea argument that Applicant made as above and which was already addressed as above.
Applicant next argues per 35 USC 101 that certain portions of the Specification, Remarks 20 (paragraphs 051, 054, 055), apparently support of the notion that the claims now amount to significantly more than the abstract idea. Remark 21. Examiner respectfully disagrees for the reasons set forth above, and further notes that he is barred from importing those specifically cited (as above) portions of the Specification into the claims in this matter.
Applicant argues per 35 USC 103 that the citations last used by examiner (Takemura and Smolyanski) were inapropos. These arguments focus however on amended portions of the claims which have not yet been analyzed, nor otherwise even addressed, by examiner. To this end, the amended portions of the claims are specifically analyzed in the following 35 USC 103 analysis.
Applicant argues per 35 USC 103 that “one of ordinary skill in the art would certainly understand that the "contamination detection model" as claimed does not correspond to the "image self-diagnosis unit". Remarks 23. Examiner respectfully disagrees. The two worded differently monikers refer to the exact same thing, namely, using either of them to detect a contaminated portion of a camera lens.
Applicant lastly argues per 35 USC 103 the notion of the recently amended claims now go on to update info pertinent to the lens analysis. Remarks 25. As to such “updating” argument, that once again is part of the amended claims herein, and, as above noted, is analyzed in the below 35 USC 103 analysis.
Generally as to obviousness, examiner submits that it is determined on the basis of the evidence as a whole and the relative persuasiveness of the arguments. See In re Oetiker, 977 F.2d 1443, 1445, 24 USPQ2d 1443, 1444 (Fed. Cir. 1992); In re Hedges, 783 F.2d 1038, 1039, 228 USPQ 685,686 (Fed. Cir. 1992); In re Piasecki, 745 F.2d 1468, 1472, 223 USPQ 785,788 (Fed. Cir. 1984); and In re Rinehart, 531 F.2d 1048, 1052, 189 USPQ 143,147 (CCPA 1976). Using this standard, examiner submits that the burden of presenting a prima facie case of obviousness was successfully established in the prior Office Action of 09/16/2025, and also respecting the pending amended claim set of 12/06/2025, as seen below.
Examiner recognizes that references cannot be arbitrarily altered or modified, and that there must be some reason why a person having ordinary skill in the relevant art would be motivated to make the proposed modifications. Although the motivation or suggestion to make modifications must be articulated, it is respectfully submitted that there is no requirement that the motivation to make modifications must be expressly articulated within the references themselves. References are evaluated by what they suggest to one versed in the art, rather than by their specific disclosures, In re Bozek, 163 USPQ 545 (CCPA 1969).
Examiner also notes that the motivation to combine the applied references is, where appropriate in the below detailed analysis pursuant to 35 USC 103, additionally accompanied by select passages from the respective references which specifically support that particular motivation. It is also respectfully submitted that motivation based on the logic and scientific reasoning of one ordinarily skilled in the art at the time of the invention, which evidence can also support a finding of obviousness, is otherwise provided in the detailed 35 USC 103 analysis of the claim set below. In re Nilssen, 851 F.2d 1401, 1403, 7 USPQ2d 1500, 1502 (Fed. Cir. 1988) (references do not have to explicitly suggest combining teachings); Ex parte Clapp, 227 USPQ 972 (Bd. Pat. App. & Inter. 1985) (examiner must present convincing line of reasoning supporting rejection); and Ex parte Levengood, 28 USPQ2d 1300 (Bd. Pat. App. & Inter. 1993) (reliance on logic and sound scientific reasoning).
Examiner recognizes that obviousness can only be established by combining or modifying the teachings of the prior art to produce the claimed invention where there is some teaching, suggestion, or motivation to do so found either in the references themselves or in the knowledge generally available to a person of ordinary skill in the art. See In re Fine, 837 F.2d 1071, 5 USPQ2d 1596 (Fed. Cir. 1988) and In re Jones, 958 F.2d 347.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Independent claims 1, 9, and 18 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Analysis of Independent claims 1, 9, and 18 (using claim 1):
An electronic device, comprising:
at least one camera configured to capture an image;
a memory configured to store the image, and
a contamination detection model configured to
detect a contaminated portion of a lens of the at least one camera, in response to the image being input,
and output a result image;
and a processor configured to
determine whether an operation of the electronic device is hindered by the contaminated portion, [in response to the contamination detection model] detecting the contaminated portion in the lens of the at least one camera,
wherein the contamination detection model comprises a model trained to detect a location of the contaminated portion within the image using a grid,
wherein the contamination detection model is updated with the captured image and an upsampled result image, and
wherein a resolution of the grid is determined based on an operation in which the contamination detection model is implemented.
101 Analysis - Step 1: Statutory category – Yes
The claims again recite a device (machine, claims 1 and 18) and a method (process, claim 9). Thus, these claims all fall within one of the four statutory categories. MPEP 2106.03
101 Analysis - Step 2A Prong one evaluation: Judicial Exception – Yes – Mental processes
In Step 2A, Prong one of the 2019 Patent Eligibility Guidance (PEG), a claim is to be analyzed to determine whether it recites subject matter that falls within one of the following groups of abstract ideas: a) mathematical concepts, b) mental processes, and/or c) certain methods of organizing human activity.
The Office submits that the foregoing bolded limitation(s) constitutes judicial exceptions in terms of “mental processes” because under its broadest reasonable interpretation, the limitations can be “performed in the human mind, or by a human using a pen and paper”. See MPEP 2106.04(a)(2)(III).
The claim recites the limitations of detect a contaminated portion of a lens of the at least one camera, in response to the image being input; and determine whether an operation of the electronic device is hindered by the contaminated portion, in response to … detecting the contaminated portion in the lens of the at least one camera. These several limitations, as drafted, and under their broadest reasonable interpretation, cover performance of the limitations in the mind but for the recitations of being performed using: an electronic device, memory, a contamination detection model, and processor. The camera is also recited at a high level of generality and is merely a device upon which the mental process is applied to determine its functionality. That is, other than reciting an electronic device, camera, memory, a contamination detection model, and processor, nothing in the claim elements precludes the steps from practically being performed in the mind. For example, the claim encompasses a person looking at the lens of a camera for a contaminated region, then determining whether any photo taken by that camera is hindered due to the contaminated region of the lens The mere nominal recitations of an electronic device, camera, memory, contamination detection model, and processor do not take the claim limitations out of the mental process grouping. Thus, the claims recite a mental process.
101 Analysis - Step 2A Prong two evaluation: Practical Application – No
In Step 2A, Prong two of the 2019 PEG, a claim is to be evaluated whether, as a whole, it integrates the recited judicial exception into a practical application. As noted in MPEP 2106.04(d), it must be determined whether any additional elements in the claim beyond the abstract idea integrate the exception into a practical application in a manner that imposes a meaningful limit on the judicial exception, such that the claim is more than a drafting effort designed to monopolize the judicial exception. The courts have indicated that additional elements such as: merely using a computer to implement an abstract idea, adding insignificant extra solution activity, or generally linking use of a judicial exception to a particular technological environment or field of use do not integrate a judicial exception into a “practical application.”
The Office submits that the foregoing underlined limitations recite additional elements that do not integrate the recited judicial exception into a practical application.
The independent claims 1, 9, and 18 recite additional elements or steps of using
an electronic device, camera, memory, contamination detection model and a processor, all combined to detect a contaminated portion of a camera lens and determine whether the same hinders the camera’s operation. These additional elements are recited at a high level of generality (i.e. as a general means for detecting contamination on the lens and determining whether the same is hindering) and amount to mere data gathering, which is a form of insignificant extra-solution activity. Moreover, these limitations merely describe generally “applying” the otherwise mental judgements using a generic or general-purpose electronic devices, cameras, memories, contamination detection models and processors as noted above. The electronic device, camera, memory, contamination detection model and processor are all recited at a high level of generality and they merely automate the several detecting and determining steps.
As to the claimed contamination detection model being updated:
wherein the contamination detection model is updated with the captured image and an upsampled result image, and wherein a resolution of the grid is determined based on an operation in which the contamination detection model is implemented.
The same amounts to mere data gathering as noted and does not amount to a practical application.
Accordingly, even in combination, these additional elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea.
101 Analysis - Step 2B evaluation: Inventive concept - No
In Step 2B of the 2019 PEG, a claim is to be evaluated as to whether the claim, as a whole, amounts to significantly more than the recited exception, i.e., whether any additional element, or combination of additional elements, adds an inventive concept to the claim. See MPEP 2106.05.
As discussed with respect to Step 2A Prong Two, the additional elements in the claim amount to no more than mere instructions to apply the exception using a generic computer and computer related components. The same analysis applies here in 2B, i.e., mere instructions to apply an exception using generic electronic devices, cameras, memories, contamination detection models, and processors cannot integrate a judicial exception into a practical application at Step 2A or provide an inventive concept in Step 2B.
Under the 2019 PEG, a conclusion that an additional element is insignificant extra-solution activity in Step 2A should be re-evaluated in Step 2B. Here, the detecting and determining steps were considered to be insignificant extra-solution activity in Step 2A, and thus they are re-evaluated in Step 2B to determine if they are more than what is well-understood, routine, conventional activity in the field. There is nothing in the disclosure that recites that the electronic device, memory, contamination detection model, and processor, are anything other than a conventional, generic, devices and/or computer/ computer controlled components. The camera is recited at a high level of generality and is merely a device upon which the mental process is applied to determine its functionality. MPEP 2106.05(d)(II), and the cases cited therein, including Intellectual Ventures I, LLC v. Symantec Corp., 838 F.3d 1307, 1321 (Fed. Cir. 2016), TLI Communications LLC v. AV Auto. LLC, 823 F.3d 607, 610 (Fed. Cir. 2016), and OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363 (Fed. Cir. 2015), indicate that mere collection or receipt of data over a network is a well‐understood, routine, and conventional function when it is claimed in a merely generic manner (as it is here). Further, the Federal Circuit in Trading Techs. Int’l v. IBG LLC, 921 F.3d 1084, 1093 (Fed. Cir. 2019), and Intellectual Ventures I LLC v. Erie Indemnity Co., 850 F.3d 1315, 1331 (Fed. Cir. 2017), for example, indicated that the mere displaying of data is a well understood, routine, and conventional function. Accordingly, a conclusion that the above underlined several elements / steps of detecting and determining whether a contaminated camera lens may be hindered amount to well-understood, routine, conventional activity and are supported under Berkheimer.
Thus, independent claims 1, 9, and 18 are ineligible.
Dependent Claims
Dependent claims 3 – 8, 11 – 17, and 19 – 20, do not recite any further limitations that cause the claim(s) to be patent eligible. Rather, the limitations of these dependent claims are directed toward additional aspects of the judicial exception. As discussed with respect to Step 2A Prong Two, the additional elements in the claim amount to no more than mere instructions to apply the exception using a generic computer component(s). The same analysis applies here in 2B, i.e., mere instructions to apply an exception on a generic computer cannot integrate a judicial exception into a practical application at Step 2A or provide an inventive concept in Step 2B.
Dependent claims 3 – 8, 11 – 17, and 19 – 20 are not patent eligible under the same rationale as provided for in the above rejection of independent claims 1, 9, and 18.
Given the above analyses, all claims 1, 3 – 9, and 11 – 20 are ineligible under 35 USC §101.
Claim Rejections – 35 USC 103
In the event the determination of the status of the application as subject to AIA 35 USC 102 and 103 is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 USC 103 which forms the basis for all obviousness rejections set forth in this Office Action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 USC 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating
obviousness or nonobviousness.
Claims 1, 3 – 9, and 11 – 20 are rejected pursuant to 35 USC 103 as being unpatentable over Takemura (US20150334385A1) in view of Smolyanski (US20210150230A1).
Regarding independent claims 1, 9, and 18: (Note that claim 18 is mapped immediately below, as it reads on independent claims 1 and 9).
Takemura discloses:
at least one camera; and (“The present invention relates to a device for recognizing vehicle-mounted environment of a subject vehicle using a vehicle-mounted camera.”, [001]);
a processor (“…environment recognition device 10 according to an embodiment of the present invention. An imaging unit 100 acquires an image from a vehicle-mounted camera (not shown). The image is utilized in an image self-diagnosis unit 200 to detect lens water droplets, lens cloudiness, lens reflection, a low visibility region, contamination, road surface reflection, road surface water film, road surface sludge, light source environment, weather and the like.”, [026], and see [027, 031]), and see Fig. 1;
configured to load a contamination detection model Applying broadest reasonable interpretation to the claim term “contamination detection model”, consistent with the light of the Specification [007], said term includes an “image self-diagnosis unit”, both of which entities are capable of detecting various areas and/or defects within a camera image, … (“In order to achieve the object, a vehicle-mounted environment recognition device according to the present invention includes an imaging unit that acquires an image taken by an imaging device; an image self-diagnosis unit that diagnoses contamination of a lens of the imaging device with respect to the image acquired by the imaging unit; an application execution unit that executes an application selected from a predetermined application group; and a fail determination unit that determines whether, based on a diagnosis result from the image self-diagnosis unit, a lens contamination state is in an endurance range of the application, and that sets an operation of the application executed by the application execution unit to a suppression mode if within the endurance range, or that implements contamination removal control or a fail determination for the application if outside the endurance range.”, [007]);
configured to detect a contaminated portion of a lens of the at least one camera, (“In order to achieve the object, a vehicle-mounted environment recognition device according to the present invention includes an imaging unit that acquires an image taken by an imaging device; an image self-diagnosis unit that diagnoses contamination of a lens of the imaging device with respect to the image acquired by the imaging unit; an application execution unit that executes an application selected from a predetermined application group; and a fail determination unit that determines whether, based on a diagnosis result from the image self-diagnosis unit, a lens contamination state is in an endurance range of the application, and that sets an operation of the application executed by the application execution unit to a suppression mode if within the endurance range, or that implements contamination removal control or a fail determination for the application if outside the endurance range.”, [007]);
in response to an input of an image captured by the at least one camera to the contamination detection model, and (“The image is utilized in an image self-diagnosis unit 200 to detect lens water droplets, lens cloudiness, lens reflection, a low visibility region, contamination, road surface reflection, road surface water film, road surface sludge, light source environment, weather and the like. The result of detection is utilized for determining a status where erroneous detection, lack of detection and the like tends to be caused during multi-application execution, and to then determine a subsequent response method.”, [026]);
and configured to output a result image, and (“The image self-diagnosis unit 200 produces an output as illustrated in FIG. 6 where an image is divided into a grid creating a map.”, [40]);
determine whether a location of the contaminated portion in the lens of the camera hinders an operation of the electronic device, (“a vehicle-mounted environment recognition device according to the present invention includes an imaging unit that acquires an image taken by an imaging device; an image self-diagnosis unit that diagnoses contamination of a lens of the imaging device with respect to the image acquired by the imaging unit; an application execution unit that executes an application selected from a predetermined application group; and a fail determination unit that determines whether, based on a diagnosis result from the image self-diagnosis unit, a lens contamination state is in an endurance range of the application, and that sets an operation of the application executed by the application execution unit to a suppression mode if within the endurance range, or that implements contamination removal control or a fail determination for the application if outside the endurance range.”, [007]) and see Abstract, published 11/19/2015, note that if the contaminated portion of the view does not hinder operation of the vehicle, then that portion falls within the “endurance” range, meaning that it does not hinder operation of the vehicle, and is therefore suppressed;
based on the location of the contaminated portion being provided by the contamination detection model using a grid, (“The image self-diagnosis unit 200 produces an output as illustrated in FIG. 6 where an image is divided into a grid creating a map. On the map, there are shown positions at which locations with higher brightness than in surrounding regions have been discovered, and scores indicating the degree of attachment of water droplets on the lens, the scores indicating the probability of water droplets attachment on a position by position basis. The water droplets scores shown on the map are utilized in the application-by-application fail determination unit to determine the response technique in the event of water droplets attachment on an application-by-application basis.”, [040]);
wherein the contamination detection model comprises a model trained to detect a location of the contaminated portion within the image using a grid. (“The image self-diagnosis unit 200 produces an output as illustrated in FIG. 6 where an image is divided into a grid creating a map. On the map, there are shown positions at which locations with higher brightness than in surrounding regions have been discovered, and scores indicating the degree of attachment of water droplets on the lens, the scores indicating the probability of water droplets attachment on a position by position basis. The water droplets scores shown on the map are utilized in the application-by-application fail determination unit to determine the response technique in the event of water droplets attachment on an application-by-application basis.”, [040]);
Takemura does not expressly disclose, but Smolyanski teaches:
wherein the contamination detection model is updated with the captured image and an upsampled result image, and (“A world model manager 426 may be used to generate, update, and/or define a world model. The world model manager 426 may use information generated by and received from the perception component(s) of the drive stack 422 (e.g., the locations of detected obstacles). The perception component(s) may include an obstacle perceiver, a path perceiver, a wait perceiver, a map perceiver, and/or other perception component(s). For example, the world model may be defined, at least in part, based on affordances for obstacles, paths, and wait conditions that can be perceived in real-time or near real-time by the obstacle perceiver, the path perceiver, the wait perceiver, and/or the map perceiver. The world model manager 426 may continually update the world model based on newly generated and/or received inputs (e.g., data) from the obstacle perceiver, the path perceiver, the wait perceiver, the map perceiver, and/or other components of the autonomous vehicle control system.”, [084]);
wherein a resolution of the grid is determined based on an operation in which the contamination detection model is implemented. Examiner interprets this claim limitation broadly to include that grid resolution is simply determined by the model … (“In some embodiments, when reflections are binned together in a pixel of a range image, the reflection with the closest range may be represented in the range image and the other reflections may be dropped. Additionally or alternatively, the resolution of the range image may be selected in such a way as to reduce the loss of sensor data and/or limit the loss of accuracy. For example, the height (or vertical resolution) of the range image may be set to correspond with the number of horizontal scan lines of the sensor capturing the sensor data (e.g., one row of pixels in the range image per scan line of a corresponding LiDAR sensor).”, [036]);
wherein the contamination detection model comprises a convolution layer (“In some embodiments, the machine learning model(s) 408 may include a common trunk (or stream of layers) with several heads (or at least partially discrete streams of layers) for predicting different outputs based on the input data 406. For example, the machine learning model(s) 408 may include, without limitation, a feature extractor (e.g., a DNN, an encoder/decoder, etc.) including convolutional layers, pooling layers, and/or other layer types, where the output of the feature extractor is provided as input to a first head for predicting classification data and a second head for predicting location, geometry, and/or orientation of detected objects. The first head and the second head may receive parallel inputs, in some examples, and thus may produce different outputs from similar input data.”, [055]) and see Abstract, published 05/20/2021;
and is periodically updated for an environment in which the electronic device is used. (“The world model manager 426 may continually update the world model based on newly generated and/or received inputs (e.g., data) from the obstacle perceiver, the path perceiver, the wait perceiver, the map perceiver, and/or other components of the autonomous vehicle control system.”, [084]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of this application to have modified Takemura to incorporate the teachings of Smolyanski because Takemura would be more efficient and versatile if it could additionally utilize convolution layering as done in Smolyanski as a machine learning tool (“For example, the machine learning model(s) 408 may include, without limitation, a feature extractor (e.g., a DNN, an encoder/decoder, etc.) including convolutional layers, pooling layers, and/or other layer types, where the output of the feature extractor is provided as input to a first head for predicting classification data and a second head for predicting location, geometry, and/or orientation of detected objects.”, Smolyanski at [055]).
Regarding claims 3 and 12:
The combination of Takemura and Smolyanski disclose the limitations of claims 1 and 9, respectively:
Smolyanski further teaches:
determine whether to supplement the contaminated portion with an overlapping area of another image captured by another camera, and to determine whether an operation of the electronic device is hindered by the contaminated portion based on the overlapping area of the another image. (“In some situations, forming a range scan image such as a LiDAR range image may result in some sensor data being lost. For example, it may be possible for reflections from multiple objects in a scene to be binned together into one range scan pixel when accumulating detections over time while the ego-object is moving, when accumulating detections from different sensors mounted at different locations of the ego-object (i.e., capturing sensor data from different views of the scene), and/or when collapsing sensor data into a range image with a resolution that is insufficient to represent adjacent sensor data. In some embodiments, when reflections are binned together in a pixel of a range image, the reflection with the closest range may be represented in the range image and the other reflections may be dropped.”, [036]), and see [048].
It would have been obvious to one of ordinary skill in the art before the effective filing date of this application to have modified Takemura to incorporate the teachings of Smolyanski because Takemura would be more efficient and versatile if it could additionally utilize convolution layering as done in Smolyanski as a machine learning tool (“For example, the machine learning model(s) 408 may include, without limitation, a feature extractor (e.g., a DNN, an encoder/decoder, etc.) including convolutional layers, pooling layers, and/or other layer types, where the output of the feature extractor is provided as input to a first head for predicting classification data and a second head for predicting location, geometry, and/or orientation of detected objects.”, Smolyanski at [055]).
Regarding claims 4 and 13:
The combination of Takemura and Smolyanski disclose the limitations of claims 1 and 9, respectively:
Smolyanski further teaches:
wherein the processor is further configured to update the contamination detection model based on a reference image obtained from the electronic device in an environment of use of the electronic device. (“The world model manager 426 may continually update the world model based on newly generated and/or received inputs (e.g., data) from the obstacle perceiver, the path perceiver, the wait perceiver, the map perceiver, and/or other components of the autonomous vehicle control system.”, [084]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of this application to have modified Takemura to incorporate the teachings of Smolyanski because Takemura would be more efficient and versatile if it could additionally utilize convolution layering as done in Smolyanski as a machine learning tool (“For example, the machine learning model(s) 408 may include, without limitation, a feature extractor (e.g., a DNN, an encoder/decoder, etc.) including convolutional layers, pooling layers, and/or other layer types, where the output of the feature extractor is provided as input to a first head for predicting classification data and a second head for predicting location, geometry, and/or orientation of detected objects.”, Smolyanski at [055]).
Regarding claims 5 and 14:
The combination of Takemura and Smolyanski disclose the limitations of claims 4 and 13, respectively:
Smolyanski further teaches:
update the contamination detection model, using, as training data, the reference image and a label of the reference image determined based on an output of the contamination detection model to which the reference image is input. (“The world model manager 426 may continually update the world model based on newly generated and/or received inputs (e.g., data) from the obstacle perceiver, the path perceiver, the wait perceiver, the map perceiver, and/or other components of the autonomous vehicle control system.”, [084]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of this application to have modified Takemura to incorporate the teachings of Smolyanski because Takemura would be more efficient and versatile if it could additionally utilize convolution layering as done in Smolyanski as a machine learning tool (“For example, the machine learning model(s) 408 may include, without limitation, a feature extractor (e.g., a DNN, an encoder/decoder, etc.) including convolutional layers, pooling layers, and/or other layer types, where the output of the feature extractor is provided as input to a first head for predicting classification data and a second head for predicting location, geometry, and/or orientation of detected objects.”, Smolyanski at [055]).
Regarding claims 6 and 15:
The combination of Takemura and Smolyanski disclose the limitations of claims 5 and 14, respectively:
Smolyanski further teaches:
preprocess the training data and to update the contamination detection model based on the preprocessed training data. (“A world model manager 426 may be used to generate, update, and/or define a world model. The world model manager 426 may use information generated by and received from the perception component(s) of the drive stack 422 (e.g., the locations of detected obstacles). The perception component(s) may include an obstacle perceiver, a path perceiver, a wait perceiver, a map perceiver, and/or other perception component(s). For example, the world model may be defined, at least in part, based on affordances for obstacles, paths, and wait conditions that can be perceived in real-time or near real-time by the obstacle perceiver, the path perceiver, the wait perceiver, and/or the map perceiver. The world model manager 426 may continually update the world model based on newly generated and/or received inputs (e.g., data) from the obstacle perceiver, the path perceiver, the wait perceiver, the map perceiver, and/or other components of the autonomous vehicle control system.”, [084]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of this application to have modified Takemura to incorporate the teachings of Smolyanski because Takemura would be more efficient and versatile if it could additionally utilize convolution layering as done in Smolyanski as a machine learning tool (“For example, the machine learning model(s) 408 may include, without limitation, a feature extractor (e.g., a DNN, an encoder/decoder, etc.) including convolutional layers, pooling layers, and/or other layer types, where the output of the feature extractor is provided as input to a first head for predicting classification data and a second head for predicting location, geometry, and/or orientation of detected objects.”, Smolyanski at [055]).
Regarding claims 7 and 16:
The combination of Takemura and Smolyanski disclose the limitations of claims 1 and 9, respectively:
Smolyanski further teaches:
a communication module configured to communicate with a server, wherein the server is configured to receive reference images from each of a plurality of electronic devices, (“Generally, object detection may be performed using sensor data 402 from any number and any type of sensor, such as, without limitation, LiDAR sensors, RADAR sensors, cameras, and/or other sensor types such as those described below with respect to the autonomous vehicle 1600 of FIGS.”, [043]);
to update a super model using each of the reference images, and to update respective contamination detection models stored in each of the plurality of electronic devices using the updated super model, (“A world model manager 426 may be used to generate, update, and/or define a world model. The world model manager 426 may use information generated by and received from the perception component(s) of the drive stack 422 (e.g., the locations of detected obstacles). The perception component(s) may include an obstacle perceiver, a path perceiver, a wait perceiver, a map perceiver, and/or other perception component(s). For example, the world model may be defined, at least in part, based on affordances for obstacles, paths, and wait conditions that can be perceived in real-time or near real-time by the obstacle perceiver, the path perceiver, the wait perceiver, and/or the map perceiver. The world model manager 426 may continually update the world model based on newly generated and/or received inputs (e.g., data) from the obstacle perceiver, the path perceiver, the wait perceiver, the map perceiver, and/or other components of the autonomous vehicle control system.”, [084]);
wherein the super model comprises weights of all the contamination detection models comprised in each of the plurality of electronic devices, and (“In some embodiments, a total loss may be computed as a sum of classification loss (e.g., from the first and/or the second stage of the machine learning model(s) 408) and regression loss (e.g., from the second stage of the machine learning model(s) 408), in some embodiments, the contribution to the loss from the different tasks may be weighted with fixed weights and/or autoweights. Additionally or alternatively, classification loss may be weighted to counteract a class imbalance present in a training dataset. These and other variations may be implemented within the scope of the present disclosure.”, [0129]);
wherein each of the respective contamination detection models comprise a weight extracted from the super model to be used by the respective electronic device of the plurality of electronic devices. (“In some embodiments, a total loss may be computed as a sum of classification loss (e.g., from the first and/or the second stage of the machine learning model(s) 408) and regression loss (e.g., from the second stage of the machine learning model(s) 408), in some embodiments, the contribution to the loss from the different tasks may be weighted with fixed weights and/or autoweights. Additionally or alternatively, classification loss may be weighted to counteract a class imbalance present in a training dataset. These and other variations may be implemented within the scope of the present disclosure.”, [0129]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of this application to have modified Takemura to incorporate the teachings of Smolyanski because Takemura would be more efficient and versatile if it could additionally utilize convolution layering as done in Smolyanski as a machine learning tool (“For example, the machine learning model(s) 408 may include, without limitation, a feature extractor (e.g., a DNN, an encoder/decoder, etc.) including convolutional layers, pooling layers, and/or other layer types, where the output of the feature extractor is provided as input to a first head for predicting classification data and a second head for predicting location, geometry, and/or orientation of detected objects.”, Smolyanski at [055]).
Regarding claims 8 and 17:
The combination of Takemura and Smolyanski disclose the limitations of claims 7 and 16, respectively:
Smolyanski further teaches:
wherein the server is further configured to extract a weight of the contamination detection model of the electronic device from the super model before updating the contamination detection model of one of the plurality of electronic devices, and to learn the extracted weight using an image received from the electronic device. (“In some embodiments, a total loss may be computed as a sum of classification loss (e.g., from the first and/or the second stage of the machine learning model(s) 408) and regression loss (e.g., from the second stage of the machine learning model(s) 408), in some embodiments, the contribution to the loss from the different tasks may be weighted with fixed weights and/or autoweights. Additionally or alternatively, classification loss may be weighted to counteract a class imbalance present in a training dataset. These and other variations may be implemented within the scope of the present disclosure.”, [0129]), and see [084] as above.
It would have been obvious to one of ordinary skill in the art before the effective filing date of this application to have modified Takemura to incorporate the teachings of Smolyanski because Takemura would be more efficient and versatile if it could additionally utilize convolution layering as done in Smolyanski as a machine learning tool (“For example, the machine learning model(s) 408 may include, without limitation, a feature extractor (e.g., a DNN, an encoder/decoder, etc.) including convolutional layers, pooling layers, and/or other layer types, where the output of the feature extractor is provided as input to a first head for predicting classification data and a second head for predicting location, geometry, and/or orientation of detected objects.”, Smolyanski at [055]).
Regarding claim 11:
The combination of Takemura and Smolyanski disclose the limitations of claim 9:
Smolyanski further teaches:
wherein an accuracy of the contamination detection model increases, in response to an increase in a granularity of the grid. (“In some situations, forming a range scan image such as a LiDAR range image may result in some sensor data being lost. For example, it may be possible for reflections from multiple objects in a scene to be binned together into one range scan pixel when accumulating detections over time while the ego-object is moving, when accumulating detections from different sensors mounted at different locations of the ego-object (i.e., capturing sensor data from different views of the scene), and/or when collapsing sensor data into a range image with a resolution that is insufficient to represent adjacent sensor data. In some embodiments, when reflections are binned together in a pixel of a range image, the reflection with the closest range may be represented in the range image and the other reflections may be dropped. Additionally or alternatively, the resolution of the range image may be selected in such a way as to reduce the loss of sensor data and/or limit the loss of accuracy. For example, the height (or vertical resolution) of the range image may be set to correspond with the number of horizontal scan lines of the sensor capturing the sensor data (e.g., one row of pixels in the range image per scan line of a corresponding LiDAR sensor). The width (or horizontal resolution) of the range image may be set based on the horizontal resolution of the sensor capturing the sensor data. Generally, horizontal resolution may be a design choice: a lower resolution may have fewer collisions, but may be easier to process (and vice versa).”, [036]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of this application to have modified Takemura to incorporate the teachings of Smolyanski because Takemura would be more efficient and versatile if it could additionally utilize convolution layering as done in Smolyanski as a machine learning tool (“For example, the machine learning model(s) 408 may include, without limitation, a feature extractor (e.g., a DNN, an encoder/decoder, etc.) including convolutional layers, pooling layers, and/or other layer types, where the output of the feature extractor is provided as input to a first head for predicting classification data and a second head for predicting location, geometry, and/or orientation of detected objects.”, Smolyanski at [055]).
Regarding claim 19:
The combination of Takemura and Smolyanski disclose the limitations of claim 18:
Takemura further teaches:
wherein the electronic device is installed in a vehicle, and the processor is further configured to terminate an autonomous driving mode of the vehicle, in response to determining that the contamination portion hinders the operation of the electronic device. (“Thus, an object of the present invention to provide a vehicle-mounted environment recognition device that conducts an image-based self-diagnosis, at the optimum timing for each application, as to whether parameter adjustment, contamination removal using hardware, or fail determination is to be implemented.”, [006]).
Regarding claim 20:
The combination of Takemura and Smolyanski disclose the limitations of claim 19:
Takemura further teaches:
wherein the processor is further configured to activate an output device to notify the user that the autonomous driving mode has terminated and to commence manual driving. (“Alternatively, the system may be configured such that the user can be notified of the fail state on an application-by-application basis in greater detail. “, [088]).
CONCLUSION
The following prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Chen (US11341614B1) – An apparatus including an interface and a processor. The interface may be configured to receive video frames generated by a plurality of capture devices. The processor may be configured to perform operations to detect objects in the video frames received from a first of the capture devices, determine depth information corresponding to the objects detected, determine blending lines in response to the depth information, perform video stitching operations on the video frames from the capture devices based on the blending lines and generate panoramic video frames in response to the video stitching operations. The blending lines may correspond to gaps in a field of view of the panoramic video frames. The blending lines may be determined to prevent the objects from being in the gaps in the field of view. The panoramic video frames may be generated to fit a size of a display.
Sun (US20190149813A1) - A system for camera fault detection, notification, and recovery is provided. The system may include at least one camera and at least one physical processing unit configured with program instructions. The processing unit may be configured to analyze image data received from the at least one camera and, based on the image analysis, detect a camera fault or malfunction.
Caine (US20220180193A1) – Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training a neural network to perform 3D object detection. One of the methods includes training a student neural network to perform 3D object detection using pseudo-labels generated by a teacher neural network.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MATTHEW COBB whose telephone number is (571) 272-3850. The examiner can normally be reached 9 - 5, M - F.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to call examiner Cobb as above, or to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Peter Nolan, can be reached at (571) 270-7016. The fax phone number for the organization where this application or proceeding is assigned is (571) 273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at (866) 217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call (800) 786-9199 (IN USA OR CANADA) or (571) 272-1000.
/MATTHEW COBB/Examiner, Art Unit 3661
/PETER D NOLAN/Supervisory Patent Examiner, Art Unit 3661
Prosecution Insights