Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Interpretation
Regarding Claim 22, examiner considered if the claim language one or more computer-readable storage media directs to a non-statutory subject matter. Based on the language within specification Para [0114], examiner believes that the one or more computer-readable storage media of this application does not direct to a non-statutory subject matter. As such, claim 22 is not rejected under 35 U.S.C. 101 non-statutory subject matter.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim(s) 1-6, 9-16, 19-22 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The limitations, under their broadest reasonable interpretation, cover mental processes (concepts performed in a human mind, including as an observation, evaluation, judgment, opinion, organizing human activity and/or mathematical concepts and calculations). The independent claim(s) 1, 11, and 22 recite(s) a system for performing frame correction using ML model. This judicial exception is not integrated into a practical application because the steps do not add meaningful limitations to be considered specifically applied to a particular technological problem to be solved .The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception because the steps of the claimed invention can be done mentally and no additional features in the claims would preclude them from being performed as such except for the generic computer elements at high level of generality (i.e., processor, memory).
According to the USPTO guidelines, a claim is directed to non-statutory subject matter if:
STEP 1: the claim does not fall within one of the four statutory categories of invention (process, machine, manufacture or composition of matter), or
STEP 2: the claim recites a judicial exception, e.g. an abstract idea, without reciting additional elements that amount to significantly more than the judicial exception, as determined using the following analysis:
STEP 2A (PRONG 1): Does the claim recite an abstract idea, law of nature, or natural phenomenon?
STEP 2A (PRONG 2): Does the claim recite additional elements that integrate the judicial exception into a practical application?
STEP 2B: Does the claim recite additional elements that amount to significantly more than the judicial exception?
Using the two-step inquiry, it is clear that the independent claims 1, 11, and 22 are directed to an abstract idea as shown below:
STEP 1: Do the claims fall within one of the statutory categories? YES. Independent claims 1, 11, and 22 are directed to a system for performing frame correction using ML model using reference frame.
STEP 2A (PRONG 1): Is the claim directed to a law of nature, a natural phenomenon or an abstract idea? YES, the claims are directed toward a mental processes and/or mathematical concepts (i.e. abstract idea).
With regard to STEP 2A (PRONG 1), the guidelines provide three groupings of subject matter that are considered abstract ideas:
Mathematical concepts – mathematical relationships, mathematical formulas or equations, mathematical calculations;
Certain methods of organizing human activity – fundamental economic principles or practices (including hedging, insurance, mitigating risk); commercial or legal interactions (including agreements in the form of contracts; legal obligations; advertising, marketing or sales activities or behaviors; business relations); managing personal behavior or relationships or interactions between people (including social activities, teaching, and following rules or instructions); and
Mental processes – concepts that are practicably performed in the human mind (including an observation, evaluation, judgment, opinion).
Independent claims 1, 11, and 22 comprise mental processes and/or mathematical concepts that can be practicably performed in the human mind (or generic computers or components configured to perform the method) and, therefore, an abstract idea.
Regarding independent claim(s) 1, 11, and 22, the limitations recite:
performing, with the frame correction ML model executing on the processing circuitry, image frame correction to generate a corrected image frame based on weights or biases of the frame correction ML model applied to two or more of: samples of the image frame, samples of previously captured image frames from the first camera, or samples from image frames from other cameras of the plurality of cameras (The step of performing frame correction falls into the “mental processes” grouping of abstract ideas because performing frame correction can be performed in the human mind as an observation, evaluation, judgement or opinion. A person mentally fill up a missing part of the image, for instance the sign is blurred, using another image as reference.).
These limitations, as drafted, is a simple process that, under their broadest reasonable interpretation, covers performance of the limitations in the mind or by a human. The Examiner notes that under MPEP 2106.04(a)(2)(III), the courts consider a mental process (thinking) that “can be performed in the human mind, or by a human using a pen and paper" to be an abstract idea. CyberSource Corp. v. Retail Decisions, Inc., 654 F.3d 1366, 1372, 99 USPQ2d 1690, 1695 (Fed. Cir. 2011). As the Federal Circuit explained, "methods which can be performed mentally, or which are the equivalent of human mental work, are unpatentable abstract ideas the ‘basic tools of scientific and technological work’ that are open to all.’" 654 F.3d at 1371, 99 USPQ2d at 1694 (citing Gottschalk v. Benson, 409 U.S. 63, 175 USPQ 673 (1972)). See also Mayo Collaborative Servs. v. Prometheus Labs. Inc., 566 U.S. 66, 71, 101 USPQ2d 1961, 1965 ("‘[M]ental processes[] and abstract intellectual concepts are not patentable, as they are the basic tools of scientific and technological work’" (quoting Benson, 409 U.S. at 67, 175 USPQ at 675)); Parker v. Flook, 437 U.S. 584, 589, 198 USPQ 193, 197 (1978) (same).
As such, a person could mentally perform frame correction. The mere nominal recitation that the various steps are being executed by a processor does not take the limitations out of the mental process and/or mathematical concepts groupings. Thus, the claims recite a mental process.
STEP 2A (PRONG 2): Does the claim recite additional elements that integrate the judicial exception into a practical application? NO, the claims do not recite additional elements that integrate the judicial exception into a practical application.
With regard to STEP 2A (prong 2), whether the claim recites additional elements that integrate the judicial exception into a practical application, the guidelines provide the following exemplary considerations that are indicative that an additional element (or combination of elements) may have integrated the judicial exception into a practical application:
an additional element reflects an improvement in the functioning of a computer, or an improvement to other technology or technical field;
an additional element that applies or uses a judicial exception to affect a particular treatment or prophylaxis for a disease or medical condition;
an additional element implements a judicial exception with, or uses a judicial exception in conjunction with, a particular machine or manufacture that is integral to the claim;
an additional element effects a transformation or reduction of a particular article to a different state or thing; and
an additional element applies or uses the judicial exception in some other meaningful way beyond generally linking the use of the judicial exception to a particular technological environment, such that the claim as a whole is more than a drafting effort designed to monopolize the exception.
While the guidelines further state that the exemplary considerations are not an exhaustive list and that there may be other examples of integrating the exception into a practical application, the guidelines also list examples in which a judicial exception has not been integrated into a practical application:
an additional element merely recites the words “apply it” (or an equivalent) with the judicial exception, or merely includes instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea;
an additional element adds insignificant extra-solution activity to the judicial exception; and
an additional element does no more than generally link the use of a judicial exception to a particular technological environment or field of use.
Independent claims 1, 11, and 22 do not recite any of the exemplary considerations that are indicative of an abstract idea having been integrated into a practical application.
Independent claims 1, 11, and 22 discloses a processing circuitry or a processor, which are generic computer components that do not add a meaningful limitation to the abstract idea because they amount to simply implementing the abstract idea in a system.
Independent claims 1, 11, and 22 discloses using ML model, which is generic computer components that do not add a meaningful limitation to the abstract idea because they amount to simply implementing the abstract idea in a system. The MPEP Section 2106.04(a)(1) Examples of Claims That Do Not Recite Abstract Ideas only exempts training a neural network.
Independent claims 1, 11, and 22 discloses receiving, with a frame correction machine-learning (ML) model executing on processing circuitry, an image frame captured from a first camera of a plurality of cameras, which is insignificant pre-solution extra activity of gathering information that do not add a meaningful limitation to the abstract idea because they amount to simply implementing the abstract idea in a system.
Independent claims 1, 11, and 22 discloses performing, with the processing circuitry, post-processing based on the corrected image frame., which is insignificant post-solution extra activity of generating information that do not add a meaningful limitation to the abstract idea because they amount to simply implementing the abstract idea in a system.
Independent claim 11 discloses a memory, which is generic computer components that do not add a meaningful limitation to the abstract idea because they amount to simply implementing the abstract idea in a system.
Independent claim 22 discloses a computer-readable medium, which is generic computer components that do not add a meaningful limitation to the abstract idea because they amount to simply implementing the abstract idea in a system.
These limitations are recited at a high level of generality (i.e. as a general action or change being taken based on the results of the acquiring step) and amounts to mere post solution actions, which is a form of insignificant extra-solution activity. Further, the claims are claimed generically and are operating in their ordinary capacity such that they do not use the judicial exception in a manner that imposes a meaningful limit on the judicial exception. Accordingly, even in combination, these additional elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea.
STEP 2B: Does the claim recite additional elements that amount to significantly more than the judicial exception? No, the claims do not recite additional elements that amount to significantly more than the judicial exception.
With regard to STEP 2B, whether the claims recite additional elements that provide significantly more than the recited judicial exception, the guidelines specify that the pre-guideline procedure is still in effect. Specifically, that examiners should continue to consider whether an additional element or combination of elements:
adds a specific limitation or combination of limitations that are not well-understood, routine, conventional activity in the field, which is indicative that an inventive concept may be present; or
simply appends well-understood, routine, conventional activities previously known to the industry, specified at a high level of generality, to the judicial exception, which is indicative that an inventive concept may not be present.
Independent claim(s) 1, 11, and 22 do not recite any additional elements that are not well-understood, routine or conventional. The use of a generic computer elements are routine, well-understood and conventional process that is performed by computers.
Thus, since independent claims 1, 11, and 22 are: (a) directed toward an abstract idea, (b) do not recite additional elements that integrate the judicial exception into a practical application, and (c) do not recite additional elements that amount to significantly more than the judicial exception, it is clear that independent claims 1, 11, and 22 are not eligible subject matter under 35 U.S.C 101.
Regarding claim(s) 2 and 12: the additional limitations do not integrate the mental process into a practical application or add significantly more to the mental process. The limitations:
wherein the plurality of cameras are cameras of a vehicle merely adds definition to previous limitation that do not add a meaningful limitation to the abstract idea.
Regarding claim(s) 3 and 13: the additional limitations do not integrate the mental process into a practical application or add significantly more to the mental process. The limitations:
generating, with the processing circuitry, a confidence value indicative of confidence of accuracy of the corrected image frame,
wherein performing post-processing comprises performing post-processing based on the corrected image frame and the confidence value are insignificant post-solution activity of generating data that does not add a meaningful limitation to the abstract idea.
Regarding claim(s) 4 and 14: the additional limitations do not integrate the mental process into a practical application or add significantly more to the mental process. The limitations:
performing image frame correction comprises performing the image frame correction to generate the corrected image frame based on the weights or biases of the frame correction ML model applied to two or more of: the samples of the image frame, the samples of previously captured image frames from the first camera, the samples from image frames from other cameras, depth data, or samples of a satellite image frame merely adds definition to previous limitation that do not add a meaningful limitation to the abstract idea.
Regarding claim(s) 5 and 15: the additional limitations do not integrate the mental process into a practical application or add significantly more to the mental process. The limitations:
receiving, with a classifier ML model executing on the processing circuitry, the image frame prior to the frame correction ML model receiving the image frame, wherein the classifier ML model is configured to classify image frames into at least one of having no degradation or partial degradation is insignificant pre-solution activity of gathering data that does not add a meaningful limitation to the abstract idea;
classifying, with the classifier ML model executing on the processing circuitry, the image frame as partial degradation; and
outputting the image frame to the frame correction ML model based on the image frame being classified as partial degradation are insignificant pre-solution activity of generating data that does not add a meaningful limitation to the abstract idea;
using ML model, processor and processing circuitry are generic computer components.
Regarding claim(s) 6 and 16: the additional limitations do not integrate the mental process into a practical application or add significantly more to the mental process. The limitations:
generating a reliability weight indicative of a level of degradation of the image frame, is insignificant pre-solution activity of gathering data that does not add a meaningful limitation to the abstract idea;
wherein performing image frame correction comprises performing image frame correction to generate corrected image frame based on weights or biases of the frame correction ML model applied to two or more of: samples of the image frame, samples of previously captured image frames from the first camera, or samples from image frames from other cameras, and further based on the reliability weight merely adds definition to previous limitation that do not add a meaningful limitation to the abstract idea.
Regarding claim(s) 9 and 19: the additional limitations do not integrate the mental process into a practical application or add significantly more to the mental process. The limitation(s):
wherein performing image frame correction comprises performing inpainting on the image frame falls into the mathematical concepts grouping of abstract ideas.
Regarding claim(s) 10 and 20: the additional limitations do not integrate the mental process into a practical application or add significantly more to the mental process. The limitations:
wherein performing post-processing comprises one or more of: generating bird’s-eye-view image content based on the corrected image frame;
performing object detection based on the corrected image frame; or
performing path planning based on the corrected image frame are insignificant pre-solution activity of gathering data that does not add a meaningful limitation to the abstract idea;
Regarding claim(s) 21: the additional limitations do not integrate the mental process into a practical application or add significantly more to the mental process. The limitations:
further comprising a vehicle, wherein the vehicle includes the plurality of cameras, the memory, and the processing circuitry merely adds definition to previous limitation that do not add a meaningful limitation to the abstract idea.
Regarding claim(s) 7 and 17, the additional limitation(s):
during operation of a vehicle that includes the plurality of cameras, updating a first instance of the frame correction ML model to generate the second, updated instance of the frame correction ML model is NOT directed toward an abstract idea since it recites additional elements that integrate the judicial exception into a practical application and add significantly more that the judicial exception. Therefore, claim(s) 7 and 17 are not directed to an abstract idea and therefore is/are not rejected under 35 USC 101.
Regarding claim(s) 8 and 18, the claims are not rejected under 35 USC 101 due to their dependency to claims 7 and 17 respectively.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1-2, 4, 7, 9-12, 14, 17, 19-20, and 22 are rejected under 35 U.S.C. 103 as being unpatentable over Yadav et al. (US 2020/0204732 A1, hereinafter Yadav) in view of FROSIO et al. (US 2024/0251171 A1, hereinafter Frosio).
Regarding claims 1, 11, and 22, Yadav discloses
Claim 1: A method of processing image data, the method comprising:
Claim 11: A system for processing image data, the system comprising: memory configured to store a frame correction machine-learning (ML) model (Para [0042]: “In some embodiments, the machine learning model may be stored on a shared repository associated with the surround view generating device 107, which is globally accessible by all the vehicles”); and processing circuitry (Fig. 2A Processor 109) coupled to the memory and configured to:
Claim 22: One or more computer-readable storage media comprising instructions that when executed by one or more processors cause the one or more processors to:
receiving, with a frame correction machine-learning (ML) model executing on processing circuitry, an image frame captured from a first camera of a plurality of cameras (Para [0055]: “At block 301, the method 300 may include receiving, by a processor 109 of the surround view generating device 107, a plurality of image frames captured from each of a plurality of image capturing devices 103 associated with a vehicle 101, and sensor data 209 from a plurality of sensors associated with the vehicle 101. In some embodiments, the plurality of image capturing devices 103 may be mounted on the vehicle 101 in a way that Field of Views (FOVs) of each of the plurality of image capturing devices 103 is non overlapping”; Even though it is plurality of frames, it is suggested in Para[0053] that Yadav is “a method of handling occluded regions in an image frame to generate a surround view”; Para [0057]-[0058]: “At block 305, the method 300 may include predicting, by the processor 109, the one or more occluded blocks by mapping the plurality of image frames and the sensor data 209 with pre-stored image data using a machine learning model. As an example, the machine learning model may be a Location-specific Global Prediction Model (LGPM). At block 307, the method 300 may include generating, by the processor 109, one or more corrected image frames by stitching the predicted one or more occluded blocks to the plurality of image frames comprising the one or more occluded blocks”);
performing, with the frame correction ML model executing on the processing circuitry, image frame correction to generate a corrected image frame based on weights or biases of the frame correction ML model applied to two or more of: samples of the image frame, samples of previously captured image frames from the first camera, or samples from image frames from other cameras of the plurality of cameras (Para [0025]: “The present disclosure enables the prediction of the occlusions caused due to dust or water depositions on the plurality of image capturing devices associated with the vehicle by using the pre-stored images that were obtained from at least one of, image capturing devices of the vehicle during previous navigation along the same path or images obtained from image capturing devices of other vehicles passing through the same path or images obtained from image capturing devices associated with civic infrastructure”); and
performing, with the processing circuitry, post-processing based on the corrected image frame (Para [0059]: “At block 309, the method 300 may include generating, by the processor 109, a surround view of the vehicle 101 using at least one of the one or more corrected image frames and the plurality of image frames. In some embodiments, the surround view may be a top-view or a bird's eye view of the vehicle 101 along with a coverage of 360-degree surrounding area of the vehicle 101 which assists driver of the vehicle 101 in maneuvering the vehicle 101”, Para [0061]: “FIG. 4 illustrates a block diagram of an exemplary computer system 400 for implementing embodiments consistent with the present invention. In some embodiments, the computer system 400 can be surround view generating device 107 that is used for handling occluded regions in an image frame to generate a surround view.”).
However, Yadav does not explicitly disclose
receiving, with a frame correction machine-learning (ML) model executing on processing circuitry, an image frame captured from a first camera of a plurality of cameras;
performing, with the frame correction ML model executing on the processing circuitry, image frame correction to generate a corrected image frame based on weights or biases of the frame correction ML model applied to two or more of: samples of the image frame, samples of previously captured image frames from the first camera, or samples from image frames from other cameras of the plurality of cameras.
Frosio teaches
receiving, with a frame correction machine-learning (ML) model executing on processing circuitry, an image frame captured from a first camera of a plurality of cameras (Para [0052]: “ In various embodiments where live video processing subsystem 405 receives video captured by a low dynamic range (LDR) device, many frames of the live video received by video capture engine 410 have over or under exposed areas. Hallucination network 430 comprises a trans former based deep neural network model used to hallucinate HDR details in the over or underexposed areas. Hallucination network 430 receives the current frame from history buffer 410 and the reference frames selected by the reference frame selection network 420 and stored in reference frame buffer 425”);
performing, with the frame correction ML model executing on the processing circuitry, image frame correction to generate a corrected image frame based on weights or biases of the frame correction ML model applied to two or more of: samples of the image frame, samples of previously captured image frames from the first camera, or samples from image frames from other cameras of the plurality of cameras (Para [0050]: “In various embodiments, an image classification machine learning models may include one or more recurrent neural networks (RNNs), one or more convolutional neural networks (CNNs), one or more deep neural networks (DNNs), one or more deep convolutional networks (DCNs), one or more residual neural networks (ResNets), one or more graph neural networks, one or more autoencoders, transformer neural networks, one or more deep stereo geometry networks (DSGNs), one or more stereo R-CNNs, or other types of artificial neural networks or components of artificial neural networks. Reference frame selection network 420 is trained using reinforcement learning”; A CNN inherently performs sampling of the input images when processing; Claim 1: “identifying a set of reference frames included in the one or more frames based on at least the current frame, wherein each frame in the set of reference frames has a different exposure level relative to the current frame; determining, using one or more neural networks, a set of missing details for one or more regions of the current frame based on the set of reference frames; generating an updated version of the current frame based on the set of details”, Claim 7: “The method of claim 1, wherein determining the set of missing details comprises inputting the current frame and at least one frame in the set of reference frames into the one or more neural networks that generate the set of missing details”).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Yadav with hallucinating missing portions of an image frame base on reference frames using a CNN or other kinds of neural networks and other aspects of Frosio to effectively increase the quality of inpainting while reducing the computational cost when performing inpainting.
Regarding claims 2 and 12, dependent upon claims 1 and 11 respectively, Yadav in view of Frosio teaches everything regarding claims 1 and 11.
Yadav further discloses
wherein the plurality of cameras are cameras of a vehicle (Fig. 1B, Para [0028]: “The architecture 100 comprises a vehicle 101, image capturing devices 1031 to 103n (also referred as plurality of image capturing devices 103), stationary image capturing devices 1051 to 105n (also referred as one or more stationary image capturing devices 105) and a surround view generating device 107 (shown in FIG. 1B)… As an example, the plurality of image capturing devices 103 may include, but not limited to, a camera. The plurality of image capturing devices 103 may be configured on external surface of the vehicle 101 to capture plurality of image frames. Each side of the vehicle 101 may be configured with at least one of the plurality of image capturing devices 103 based on factors such as Field of View (FOV) of the image capturing device 103, dimensions of the vehicle 101 and the like”).
Regarding claims 4 and 14, dependent upon claims 1 and 11 respectively, Yadav in view of Frosio teaches everything regarding claims 1 and 11.
Yadav further discloses
performing, with the frame correction ML model executing on the processing circuitry, image frame correction to generate a corrected image frame based on weights or biases of the frame correction ML model applied to two or more of: the samples of the image frame, the samples of previously captured image frames from the first camera, the samples from image frames from other cameras, depth data, or samples of a satellite image frame (Para [0025]: “The present disclosure enables the prediction of the occlusions caused due to dust or water depositions on the plurality of image capturing devices associated with the vehicle by using the pre-stored images that were obtained from at least one of, image capturing devices of the vehicle during previous navigation along the same path or images obtained from image capturing devices of other vehicles passing through the same path or images obtained from image capturing devices associated with civic infrastructure”).
Frosio further teaches
performing, with the frame correction ML model executing on the processing circuitry, image frame correction to generate a corrected image frame based on weights or biases of the frame correction ML model applied to two or more of: the samples of the image frame, the samples of previously captured image frames from the first camera, the samples from image frames from other cameras, depth data, or samples of a satellite image frame (Para [0050]: “In various embodiments, an image classification machine learning models may include one or more recurrent neural networks (RNNs), one or more convolutional neural networks (CNNs), one or more deep neural networks (DNNs), one or more deep convolutional networks (DCNs), one or more residual neural networks (ResNets), one or more graph neural networks, one or more autoencoders, transformer neural networks, one or more deep stereo geometry networks (DSGNs), one or more stereo R-CNNs, or other types of artificial neural networks or components of artificial neural networks. Reference frame selection network 420 is trained using reinforcement learning”; A CNN inherently performs sampling of the input images when processing; Claim 1: “identifying a set of reference frames included in the one or more frames based on at least the current frame, wherein each frame in the set of reference frames has a different exposure level relative to the current frame; determining, using one or more neural networks, a set of missing details for one or more regions of the current frame based on the set of reference frames; generating an updated version of the current frame based on the set of details”, Claim 7: “The method of claim 1, wherein determining the set of missing details comprises inputting the current frame and at least one frame in the set of reference frames into the one or more neural networks that generate the set of missing details”).
Regarding claims 7 and 17, dependent upon claims 1 and 11 respectively, Yadav in view of Frosio teaches everything regarding claims 1 and 11.
Yadav further discloses
wherein the frame correction ML model comprises a second, updated instance of the frame correction ML model, the method further comprising: during operation of a vehicle that includes the plurality of cameras, updating a first instance of the frame correction ML model to generate the second, updated instance of the frame correction ML model (Para [0042]: “The machine learning model may be trained incrementally in real-time by correlating the image data 207 and the sensor data 209 received from the vehicle 101 and one or more other vehicles that had passed along the same path as the vehicle 101, and plurality of images received from one or more stationary image capturing devices 105 associated with civic infrastructure, of a same location from different vehicles at different times”).
Regarding claims 9 and 19, dependent upon claims 1 and 11 respectively, Yadav in view of Frosio teaches everything regarding claims 1 and 11.
Frosio further teaches
wherein performing image frame correction comprises performing inpainting on the image frame (Para [0073]: “One advantage of the systems disclosed herein is the causal reference frame selection. Using only currently available reference frames and no future frames allows the system to work on-the-fly and without full camera control. In addition, the reference frame selection DNN stores the most promising frames that could be used for hallucinating HDR details in the future. Finally, the selection of fewer, good reference frames, rather than a large set of reference frames results in better inpainting results and has smaller computational cost.”).
Regarding claims 10 and 20, dependent upon claims 1 and 11 respectively, Yadav in view of Frosio teaches everything regarding claims 1 and 11.
Yadav further discloses
wherein performing post-processing comprises one or more of: generating bird’s-eye-view image content based on the corrected image frame; performing object detection based on the corrected image frame; or performing path planning based on the corrected image frame (Para [0059]: “At block 309, the method 300 may include generating, by the processor 109, a surround view of the vehicle 101 using at least one of the one or more corrected image frames and the plurality of image frames. In some embodiments, the surround view may be a top-view or a bird's eye view of the vehicle 101 along with a coverage of 360-degree surrounding area of the vehicle 101 which assists driver of the vehicle 101 in maneuvering the vehicle 101”).
Claim(s) 3 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Yadav et al. (US 2020/0204732 A1, hereinafter Yadav) in view of FROSIO et al. (US 2024/0251171 A1, hereinafter Frosio and Li et al. (US 2021/0133947 A1, hereinafter Li).
Regarding claims 3 and 13, dependent upon claims 1 and 11 respectively, Yadav in view of Frosio teaches everything regarding claims 1 and 11.
However, Yadav in view of Frosio does not explicitly teach
generating, with the processing circuitry, a confidence value indicative of confidence of accuracy of the corrected image frame,
wherein performing post-processing comprises performing post-processing based on the corrected image frame and the confidence value.
Li teaches
generating, with the processing circuitry, a confidence value indicative of confidence of accuracy of the corrected image frame (Para [0025]: “At 316, the controller 112 determines whether the image quality metric for the current image frame satisfies the image quality threshold. When the image quality metric satisfies the image quality threshold, the method 300 proceeds to 324. Otherwise, the method 300 proceeds to 320”; Under BRI, examiner interprets the "confidence value indicative of confidence of accuracy of the corrected image frame" as quality score of the corrected image frame. In addition, the process of determining the quality score of the corrected image frame is no different from determining the quality score of any image frame as this process is after the generation of the corrected image. Since this art is in a similar endeavor as Yadav, it can be understood that the quality determination of Li can work on the corrected images produced by the Yadav),
wherein performing post-processing comprises performing post-processing based on the corrected image frame and the confidence value (Para [0025]: “At 316, the controller 112 determines whether the image quality metric for the current image frame satisfies the image quality threshold. When the image quality metric satisfies the image quality threshold, the method 300 proceeds to 324. Otherwise, the method 300 proceeds to 320”, Para [0026]: “At 324, the controller 112 utilizes the first DNN and at least the current image frame (optionally with additional data, such as LIDAR point cloud data) to perform object detection… At 340, the list of one or more detected objects is used as part of an ADAS function of the vehicle 100 (adaptive cruise control, collision avoidance, etc.)”).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Yadav in view of Frosio with determining a quality score for an image and performing actions based on the image of Li to effectively improve the quality of images used for object detection in autonomous vehicle driving.
Claim(s) 5-6 and 15-16 are rejected under 35 U.S.C. 103 as being unpatentable over Yadav et al. (US 2020/0204732 A1, hereinafter Yadav) in view of FROSIO et al. (US 2024/0251171 A1, hereinafter Frosio and JONNALAGEDDA et al. (US 2022/0350996 A1, hereinafter Jonnalagedda).
Regarding claims 5 and 15, dependent upon claims 1 and 11 respectively, Yadav in view of Frosio teaches everything regarding claims 1 and 11.
However, Yadav in view of Frosio does not explicitly teach
receiving, with a classifier ML model executing on the processing circuitry, the image frame prior to the frame correction ML model receiving the image frame, wherein the classifier ML model is configured to classify image frames into at least one of having no degradation or partial degradation;
classifying, with the classifier ML model executing on the processing circuitry, the image frame as partial degradation; and
outputting the image frame to the frame correction ML model based on the image frame being classified as partial degradation.
Jonnalagedda teaches
receiving, with a classifier ML model executing on the processing circuitry, the image frame prior to the frame correction ML model receiving the image frame, wherein the classifier ML model is configured to classify image frames into at least one of having no degradation or partial degradation (Para [0048]: “At step 408, back-end computing system 104 identifies one or more distortions in the image. For example, distortion detector 218 may detect the types of distortions present in the image. Distortion detector 218 may utilize a trained DCT CNN to detect one or more distortions present in the image. In some embodiments, distortion detector 218 may detect one or more distortions in each patch of the one or more patches corresponding to the image.”);
classifying, with the classifier ML model executing on the processing circuitry, the image frame as partial degradation (Para [0049]: “At step 410, back-end computing system 104 determines if the image meets a threshold level of quality. For example, distortion detector 218 may determine whether the image is suitable for correction, based on the one or more distortions identified. If, at step 410, distortion detector 218 determines that the image does not meet a threshold level of quality, i.e., the image is not suitable for correction, then method 400 proceeds to step 412”); and
outputting the image frame to the frame correction ML model based on the image frame being classified as partial degradation (Para [0050]: “If, however, at step 410, distortion detector 218 determines that the image meets a threshold level of quality, i.e., the image is suitable for correction, then method 400 proceeds to step 414”).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Yadav in view of Frosio with only performing image correction only if a level of distortion of the current image is reached of Jonnalagedda to effectively reduce the computing power needed for the system.
Regarding claims 6 and 16, dependent upon claims 5 and 15 respectively, Yadav in view of Frosio and Jonnalagedda teaches everything regarding claims 5 and 15.
Jonnalagedda further teaches
generating a reliability weight indicative of a level of degradation of the image frame (Para [0049]: “At step 410, back-end computing system 104 determines if the image meets a threshold level of quality. For example, distortion detector 218 may determine whether the image is suitable for correction, based on the one or more distortions identified. If, at step 410, distortion detector 218 determines that the image does not meet a threshold level of quality, i.e., the image is not suitable for correction, then method 400 proceeds to step 412”, having a threshold implies there is a values that measures the level of degradation of an image),
wherein performing image frame correction comprises performing image frame correction to generate corrected image frame based on weights or biases of the frame correction ML model applied to two or more of: samples of the image frame, samples of previously captured image frames from the first camera, or samples from image frames from other cameras, and further based on the reliability weight (Fig. 4, Para [0051]: “At step 414, back-end computing system 104 generates a clean version of the image. For example, distortion corrector 220 may generate a clean version of the image based on the uploaded image and/or the one or more patches of the uploaded image. Distortion corrector 220 translates the image from the distorted domain to the clean domain. In some embodiments, distortion corrector 220 corrects distortion in the image on a patch-by-patch basis”, Step 414 is not reached unless the threshold (reliability weight) is reached. As such, under BRI, the operation distortion correction of Jonnalagedda is based upon its distortion threshold from the previous step).
Claim(s) 8 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Yadav et al. (US 2020/0204732 A1, hereinafter Yadav) in view of FROSIO et al. (US 2024/0251171 A1, hereinafter Frosio and Pathak et al. (Context Encoders: Feature Learning by Inpainting, hereinafter Pathak).
Regarding claims 8 and 18, dependent upon claims 7 and 17 respectively, Yadav in view of Frosio teaches everything regarding claims 7 and 17.
Yadav further discloses
wherein the image frame comprises a second image frame, wherein the corrected image frame comprises a second corrected image frame, and wherein updating the first instance of the frame correction ML model comprises: corrupting a ground truth image frame captured with one of the plurality of cameras before the first camera captured the second image frame to generate a corrupted image frame (Para [0025]: “The present disclosure enables the prediction of the occlusions caused due to dust or water depositions on the plurality of image capturing devices associated with the vehicle by using the pre-stored images that were obtained from at least one of, image capturing devices of the vehicle during previous navigation along the same path or images obtained from image capturing devices of other vehicles passing through the same path or images obtained from image capturing devices associated with civic infrastructure”, Para [0042]: “The machine learning model may be trained incrementally in real-time by correlating the image data 207 and the sensor data 209 received from the vehicle 101 and one or more other vehicles that had passed along the same path as the vehicle 101, and plurality of images received from one or more stationary image capturing devices 105 associated with civic infrastructure, of a same location from different vehicles at different times”, Para [0043]: “Therefore, if the occluded block predicting module 227 predicts absence of the one or more occluded blocks in the plurality of image frames, the plurality of image frames may be transmitted to the shared repository for training the machine learning model incrementally”; Ground truth images can be taken from the same camera of the vehicle when navigating the same path at a different time point);
However Yadav in view of Frosio does not explicitly teach
corrupting a ground truth image frame captured with one of the plurality of cameras before the first camera captured the second image frame to generate a corrupted image frame;
applying the first instance of the frame correction ML model to the corrupted image frame to generate a first corrected image frame;
comparing the first corrected image frame and the ground truth image frame; and
updating weights or biases of the first instance of the frame correction ML model based on comparing the first corrected image frame and the ground truth image frame to generate the second, updated instance of the frame correction ML model.
Pathak teaches
corrupting a ground truth image frame captured with one of the plurality of cameras before the first camera captured the second image frame to generate a corrupted image frame (Figure 3-4, Section 3.3. Region masks: “The input to a context encoder is an image with one or more of its regions “dropped out”; i.e., set to zero, assuming zero-centered inputs”);
applying the first instance of the frame correction ML model to the corrupted image frame to generate a first corrected image frame (Figure 2, Section 3.1 Encoder-decoder pipeline: “The overall architecture is a simple encoder-decoder pipeline. The encoder takes an input image with missing regions and produces a latent feature representation of that image. The decoder takes this feature representation and produces the missing image content. We found it important to connect the encoder and the decoder through a channel wise fully-connected layer, which allows each unit in the decoder to reason about the entire image content. Figure 2 shows an overview of our architecture”, Section 3.1-Decoder: “We now discuss the second half of our pipeline, the decoder, which generates pixels of the image using the encoder features. The “encoder features” are connected to the “decoder features” using a channel-wise fully connected layer”);
comparing the first corrected image frame and the ground truth image frame (Figure 2, Section 3.2 Loss Function: “We train our context encoders by regressing to the ground truth content of the missing (dropped out) region. However, there are often multiple equally plausible ways to fill a missing image region which are consistent with the context. We model this behavior by having a decoupled joint loss function to handle both continuity within the con text and multiple modes in the output. The reconstruction (L2) loss is responsible for capturing the overall structure of the missing region and coherence with regards to its context, but tends to average together the multiple modes in predictions. The adversarial loss [16], on the other hand, tries to make prediction look real, and has the effect of picking a particular mode from the distribution”); and
updating weights or biases of the first instance of the frame correction ML model based on comparing the first corrected image frame and the ground truth image frame to generate the second, updated instance of the frame correction ML model (Section 5.2 Feature Learning: “For consistency with prior work, we use the AlexNet [26] architecture for our encoder. Unfortunately, we did not manage to make the adversarial loss converge with AlexNet, so we used just the reconstruction loss. The networks were trained with a constant learning rate of 103 for the center-region masks. However, for random region corruption, we found a learning rate of 104 to perform better. We apply dropout with a rate of 0.5 just for the channel-wise fully connected layer, since it has more parameters than other layers and might be prone to overfitting. The training process is fast and converges in about 100K iterations: 14 hours on a Titan X GPU”; Each iteration of training is to update the model with new weights and biases based on the loss).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Yadav in view of Frosio with generating corrupted ground truth images using masks and training image correction model using these corrupted ground truth images of Pathak to effectively increase the quality of inpainting using neural network.
Claim(s) 21 is rejected under 35 U.S.C. 103 as being unpatentable over Yadav et al. (US 2020/0204732 A1, hereinafter Yadav) in view of FROSIO et al. (US 2024/0251171 A1, hereinafter Frosio and Micks et al. (US 2018/0197048 A1, hereinafter Micks).
Regar