DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Acknowledgment is made of applicant’s claim for foreign priority under 35 U.S.C. 119 (a)-(d). The certified copy has been filed in parent Application No. 18862094, filed on 10/31/2024.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 10/31/2024 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Specification
The abstract of the disclosure is objected to for the inclusion of indented limitations with respective bullets (written as “-“). A corrected abstract of the disclosure is required and must be presented on a separate sheet, apart from any other text. See MPEP § 608.01(b).
Claim Objections
Claims 1-20 are objected to because of the following informalities: These claims are written in an improper, and sometimes narrative, format. For example, see the use of “on the one hand” and “on the other hand”. Additionally, note that the bullets must be removed and the claims must be rewritten accordingly to comply with USPTO standards. Appropriate correction is required.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that use the word “means” or “step” but are nonetheless not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph because the claim limitation(s) recite(s) sufficient structure, materials, or acts to entirely perform the recited function. Such claim limitation(s) is/are: “means” and “processing means” in claim 16.
Because this/these claim limitation(s) is/are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are not being interpreted to cover only the corresponding structure, material, or acts described in the specification as performing the claimed function, and equivalents thereof.
If applicant intends to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to remove the structure, materials, or acts that performs the claimed function; or (2) present a sufficient showing that the claim limitation(s) does/do not recite sufficient structure, materials, or acts to perform the claimed function.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-16 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 1 recites, “the at least one input image originating from at least one optical sensor provided with photosites of different colors and being obtained through at least one optical imaging system, each sensor being associated with an optical imaging system”. The bolded “each sensor” lacks antecedence and is being interpreted as referring to the ‘at least one optical sensor’.
Claim 1 recites, “a first term D, which depends on a comparison between, on the one hand, the at least one input image le and, on the other hand, a result Ick of the image IRK being rendered at the iteration k reprocessed by information relating to the at least one imaging system”. Firstly, the description of D as ‘depends on a comparison’ is indefinite because the nature of the dependence cannot be ascertained. This is being interpreted as any kind of dependence, including that D is equal to the comparison. Next, Applicant should remove the “on the one hand” and “on the other hand”, which lack antecedence and are improper.
Claim 1 recites, “a second term P, which depends on one or more anomalies or penalties or defects within the image IRK being rendered at the iteration k”. Similar to above, the exact scope of the dependence of P on one or more anomalies/penalties/defects cannot be ascertained. This is being interpreted broadly such that any dependence satisfies the claim language, but additional explanation is required to clarify the scope.
Claim 1 recites, “until minimizing, at least below a certain minimization threshold or after a certain number of iterations, a cumulative effect: of one or more differences of the first term D between the at least one input image Ie and the result Ick; and of one or more anomalies or penalties or defects on which the second term P depends within the image IRk being rendered at the iteration k;”. This limitation overall lacks clarity and requires revision to clarify the scope, particularly due to the bolded elements. Also evaluate punctuation for grammatical clarity. Overall, the limitation is being interpreted as performing the iterations of the previous limitations until a cumulative effect meets a minimization threshold or iteration threshold, wherein the cumulative effect relates to a D and P value at a particular iteration k.
Claim 1 recites, “different iterations k” and “the iteration k”. This creates an antecedence issue by referring to k as both a set of iterations (i.e. plural) and also one particular iteration (i.e. singular).
Claim 1 recites, “so that the rendered image IR corresponds to the image being rendered at the iteration for which this minimization is obtained.” The bolded elements lack proper antecedence.
Claims 2-15 are rejected as dependent on claim 1.
Claim 16 is rejected for the same reasons at claim 1.
Claim 6 recites, “characterized in that the result Ick may comprise and/or consist of a convolution product of the image IRk being rendered at the iteration k by the function describing the response of the at least one imaging system, and possibly processed by a geometric transformation GT.” The bolded limitation renders the claim indefinite due to the recitation of “possibly”.
Claim 11 recites the following antecedence issue: “the colors”. This is being interpreted as a new element.
Claim 12 recites the following antecedence issue: “the first iteration k=1”. This is being interpreted as a new element.
Claim 12 additionally recites the following antecedence issue: “these several input images Ie.”. In the claim set, Ie is used to refer to “the at least one input image Ie”. Ie should therefore not be used to refer to the “these several input images”.
Claims 13-15 each recite one the following relative terms: “small intensity differences”, “small differences”, and “low frequencies”. These terms are not defined by the claim, the specification does not provide a standard for ascertaining the requisite degree, and one of ordinary skill in the art would not be reasonably apprised of the scope of the invention. Accordingly, the above terms render each respective limitation indefinite.
Claim 15 recites “the method according to claim 1, characterized in that the second term P comprises at least one component P2 whose effect is minimized for low frequencies of direction changes between neighboring pixels of the image IRk drawing a contour.” This claim is grammatically unclear, particularly with respect to the bolded element, which renders the claim indefinite. It is unclear what “drawing a contour” is in relation to. This is being interpreted such that the direction changes are related to some type of contour/edge/line/curve, etc.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1-10, 13, and 16 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Levin (Image and Depth from a Conventional Camera with a Coded Aperture).
Regarding claim 1, Levin teaches “A method for correcting at least one input image Ie into an image being rendered IRk and then a rendered image IR, the at least one input image originating from at least one optical sensor provided with photosites of different colors and being obtained through at least one optical imaging system, each sensor being associated with an optical imaging system,” (Levin, Section 5.1, “The best performing filter under the criterion of Eqn. 8 was cut from gray card and inserted into an off-the-shelf Canon 50mm f/1.8 lens (shown in Figure 3(b)) mounted on a Canon 20D DSLR. To calibrate the lens the focus was locked at D = 2m and the camera was moved back until Dk = 3m in 10cm increments. At each interval, a planar pattern of random curves was captured. After aligning the focused calibration image with each of the blurry versions the blur kernel was deduced in a least-squares fashion, using a small amount of regularization to constrain the high-frequencies within the kernel. When Dk is close to D the blur is very small (< 4 pixels) making depth discrimination impossible due to lack of structure in the blur, although the image remains relatively sharp. For our setup, this “dead-zone” extends up to 35cm from the focal plane. Since the lens does not perfectly obey the thin lens model, the kernel varies slightly across the image, the distortion being more pronounced in the horizontal plane. Consequently, kernels were inferred at 7 different horizontal locations within the image. The computed kernels at a number of depths are shown in Figure 9. To enable a direct comparison between a conventional and coded apertures, we also calibrated an unmodified Canon 50mm f /1.8 lens in the same fashion.”)
“said method comprising: receiving the at least one input image Ie;” (Levin, Section 3 Paragraph 1, “Having identified the correct blur scale of an observed image y, the next objective is to remove the blur, reconstructing the original sharp image x. This task is known as deblurring or deconvolution.”)
“iteratively modifying the image IRk being rendered at different iterations k by iteratively processing a function E comprising two terms, that is: a first term D, which depends on a comparison between, on the one hand, the at least one input image Ie and, on the other hand, a result Ick of the image IRk being rendered at the iteration k reprocessed by information relating to the at least one imaging system; and a second term P, which depends on one or more anomalies or penalties or defects within the image IRk being rendered at the iteration k; until minimizing, at least below a certain minimization threshold or after a certain number of iterations, a cumulative effect: of one or more differences of the first term D between the at least one input image Ie and the result Ick; and of one or more anomalies or penalties or defects on which the second term P depends within the image IRk being rendered at the iteration k; so that the rendered image IR corresponds to the image being rendered at the iteration for which this minimization is obtained.” (Levin, Section 3, with emphasis on equation 12: “Having identified the correct blur scale of an observed image y, the next objective is to remove the blur, reconstructing the original sharp image x. This task is known as deblurring or deconvolution. Under our probabilistic model
PNG
media_image1.png
48
428
media_image1.png
Greyscale
The deblurring problem can thus be posed as finding the maximum likelihood explanation for y, x∗ = argmaxPk(x|y). For a Gaussian distribution, this reduces to a least squares optimization problem
PNG
media_image2.png
45
433
media_image2.png
Greyscale
By minimizing Eqn. 10 we search for the x minimizing the re construction error |Cfk x −y|2, with the prior preferring x to be as smooth as possible. We note that the optimal solution to Eqn. 10 can be found by solving a sparse set of linear equations: Ax = b for
PNG
media_image3.png
54
418
media_image3.png
Greyscale
Eqn. 11 can be solved in the frequency domain in a few seconds for megapixel sized image. While this approach does produce wrap-around artifacts along the image boundaries, these are usually unimportant in large images. Deblurring with a Gaussian prior on image derivatives is simple and efficient, but tends to over-smooth the result. To produce sharper decoded images, a stronger natural image prior is required, and a sparse derivatives prior was used. Thus, to solve for x we minimize
PNG
media_image4.png
48
439
media_image4.png
Greyscale
where ρ is a heavy-tailed function, in our implementation ρ(z) = |z|0.8. While a Gaussian prior prefers to distribute derivatives equally over the image, a sparse prior opts to concentrate derivatives at a small number of pixels, leaving the majority of image pixels constant. This produces sharper edges, reduces noise and helps to remove unwanted image artifacts such as ringing. The drawback of a sparse prior is that the optimization problem is no longer a simple least squares one, and cannot be minimized in closed form (in fact, the optimization is no longer convex). To optimize this, we use an iterative reweighted least squares process e.g. [Levin and Weiss To appear] which poses the optimization as a sequence of least squares problems while the weight of each derivative is up dated based on the previous iteration solution. The re-weighting means that Eqn. 11 cannot be solved in the frequency domain, so we are forced to work in the spatial domain using the Conjugate Gradient algorithm e.g. [Barrett et al. 1994]. The bottleneck in each iteration of this algorithm is the multiplication of each residual vector by the matrix A. Luckily the form of A (Eqn. 11) enables this to be performed efficiently as a concatenation of convolution operations. However, this procedure still takes around 1 hour on a 2.4Ghz CPU for a 2 megapixel image. Our sparse deblurring code is available on the project webpage: http://groups.csail. mit.edu/graphics/CodedAperture.” Note that E is mapped to Equation 12; D is mapped to “|Cfkx − y|”; the input image “Ie” is mapped to y; the “result Ick” is mapped to Cfkx; the “image Irk” is mapped to x, wherein x represents the image at the current iteration; note that the minimizing of the cumulative effect of D and P at least below a threshold is embodied in the iterative reweighted least squares process, resulting in a final x (rendered image IR).)
Regarding claim 2, Levin teaches “The method according to claim 1,”
“characterized in that it simultaneously corrects at least two of the following types of defect in the at least one input image: Optical, geometric and/or chromatic aberration, and/or Distortion, and/or Mosaicing, and/or Detection noise, and/or Blurring, and/or Residual non-compensation of a movement, and/or Artifacts induced by spatial discretization.” (Levin, Section 3, Paragraph 3, “Deblurring with a Gaussian prior on image derivatives is simple and efficient, but tends to over-smooth the result. To produce sharper decoded images, a stronger natural image prior is required, and a sparse derivatives prior was used. Thus, to solve for x we minimize
PNG
media_image5.png
65
616
media_image5.png
Greyscale
where ρ is a heavy-tailed function, in our implementation ρ(z) = |z|0.8. While a Gaussian prior prefers to distribute derivatives equally over the image, a sparse prior opts to concentrate derivatives at a small number of pixels, leaving the majority of image pixels constant. This produces sharper edges, reduces noise and helps to remove unwanted image artifacts such as ringing. The drawback of a sparse prior is that the optimization problem is no longer a simple least squares one, and cannot be minimized in closed form (in fact, the optimization is no longer convex). To optimize this, we use an iterative reweighted least squares process e.g. [Levin and Weiss To appear] which poses the optimization as a sequence of least squares problems while the weight of each derivative is up dated based on the previous iteration solution. The re-weighting means that Eqn. 11 cannot be solved in the frequency domain, so we are forced to work in the spatial domain using the Conjugate Gradient algorithm e.g. [Barrett et al. 1994]. The bottleneck in each iteration of this algorithm is the multiplication of each residual vec tor by the matrix A. Luckily the form of A (Eqn. 11) enables this to be performed efficiently as a concatenation of convolution op erations. However, this procedure still takes around 1 hour on a 2.4Ghz CPU for a 2 megapixel image. Our sparse deblurring code is available on the project webpage: http://groups.csail. mit.edu/graphics/CodedAperture.”)
Regarding claim 3, Levin teaches “The method according to claim 1,”
“characterized in that the minimization of the cumulative effect corresponds to a minimization of the function E.” (Levin, Section 3, Paragraph 3, “Deblurring with a Gaussian prior on image derivatives is simple and efficient, but tends to over-smooth the result. To produce sharper decoded images, a stronger natural image prior is required, and a sparse derivatives prior was used. Thus, to solve for x we minimize
PNG
media_image5.png
65
616
media_image5.png
Greyscale
where ρ is a heavy-tailed function, in our implementation ρ(z) = |z|0.8. While a Gaussian prior prefers to distribute derivatives equally over the image, a sparse prior opts to concentrate derivatives at a small number of pixels, leaving the majority of image pixels constant. This produces sharper edges, reduces noise and helps to remove unwanted image artifacts such as ringing. The drawback of a sparse prior is that the optimization problem is no longer a simple least squares one, and cannot be minimized in closed form (in fact, the optimization is no longer convex). To optimize this, we use an iterative reweighted least squares process e.g. [Levin and Weiss To appear] which poses the optimization as a sequence of least squares problems while the weight of each derivative is up dated based on the previous iteration solution. The re-weighting means that Eqn. 11 cannot be solved in the frequency domain, so we are forced to work in the spatial domain using the Conjugate Gradient algorithm e.g. [Barrett et al. 1994]. The bottleneck in each iteration of this algorithm is the multiplication of each residual vec tor by the matrix A. Luckily the form of A (Eqn. 11) enables this to be performed efficiently as a concatenation of convolution op erations. However, this procedure still takes around 1 hour on a 2.4Ghz CPU for a 2 megapixel image. Our sparse deblurring code is available on the project webpage: http://groups.csail. mit.edu/graphics/CodedAperture.”)
Regarding claim 4, Levin teaches “The method according to claim 1,”
“characterized in that the function E comprises the sum of the first term D and the second term P.” (Levin, equation 12 (function E) shows the sum of D and P).
Regarding claim 5, Levin teaches “The method according to claim 1,”
“characterized in that the first term D depends on the difference(s) between, on the one hand, the at least one input image Ie and, on the other hand, the result Ick of a modification of the image IRk being rendered at the iteration k at least by a function describing the response of the at least one imaging system.” (As described above, D is mapped to “|Cfkx −y|2”. According to sections 3 and 1.2 of Levin, y is the input image Ie, and Cfkx (mapped to Ick) is a modification of x (the image IRk being rendered at the iteration k) by a convolution matrix corresponding to a filter (a function describing the response of the at least one imaging system).)
Regarding claim 6, Levin teaches “The method according to claim 5,”
“characterized in that the result Ick may comprise and/or consist of a convolution product of the image IRk being rendered at the iteration k by the function describing the response of the at least one imaging system, and possibly processed by a geometric transformation GT.” (According to sections 3 and 1.2 of Levin, Cfkx (mapped to Ick) comprises is a convolutional product of x (image IRk rendered at the iteration k) by the convolution matrix corresponding to a filter (function describing the response of the at least one imaging system).
Regarding claim 7, Levin teaches “The method according to claim 5,”
“characterized in that the function describing the response of the at least one imaging system is an optical transfer function (OTF) of the at least one imaging system or a point spread function (PSF) of the at least one imaging system.” (According to section 1, subsection “Principle” of Levin, the function describing the response of the at least one imaging system (fk) is described as a “blur filter” that is “a scaled version of the aperture shape” (i.e. a PSF).) Levin also presents OTF as an alternative in section 1.2: “Throughout the paper we will use lower case symbols to denote spatial domain signals with upper case corresponding to their frequency domain representations. Also, for a filter f, we define Cf to be the corresponding convolution matrix (i.e. Cfx ≡ f ∗x). Similarly, CF will denote a convolution in the frequency domain (in this case, a diagonal matrix).”)
Regarding claim 8, Levin teaches “The method according to claim 5,”
“characterized in that the function describing the response of the at least one imaging system depends on: a distance (Z) between the at least one sensor and an object imaged by the at least one sensor, and/or a distance (zco) between a part of the at least one imaging system and the at least one sensor, and/or a distance (zos) between a part of the at least one imaging system and an object imaged by the at least one sensor, and/or a state of the at least one imaging system, such as a zoom or focus or digital aperture setting of the at least one imaging system, and/or the pixel of the image being rendered and/or the photosite of the at least one sensor, and/or one or more angles between the at least one sensor and the at least one imaging system.” (Levin, Section 5.1, “To calibrate the lens the focus was locked at D= 2m and the camera was moved back until Dk = 3m in 10cm increments. At each interval, a planar pattern of random curves was captured. After aligning the focused calibration image with each of the blurry versions the blur kernel was deduced in a least-squares fashion, using a small amount of regularization to constrain the high-frequencies within the kernel. When Dk is close to D the blur is very small (< 4 pixels) making depth discrimination impossible due to lack of structure in the blur, although the image remains relatively sharp. For our setup, this “dead-zone” extends up to 35cm from the focal plane. Since the lens does not perfectly obey the thin lens model, the kernel varies slightly across the image, the distortion being more pronounced in the horizontal plane. Consequently, kernels were inferred at 7 different horizontal locations within the image. The computed kernels at a number of depths are shown in Figure 9. To enable a direct comparison between a conventional and coded apertures, we also calibrated an unmodified Canon 50mm f/1.8 lens in the same fashion.”)
Regarding claim 9, Levin teaches “The method according to claim 1,”
“characterized in that it comprises passing light through the at least one optical system to the at least one sensor so as to generate the at least one input image.” (Levin, Section 5.1 describes the use of the “Canon 50mm f/1.8 lens” that is “mounted on a Canon 20D DSLR” for image capture in the paper, which necessarily embodies the limitations of the claim.)
Regarding claim 10, Levin teaches “The method according to claim 1,”
“characterized in that it comprises displaying the rendered image on a screen.” (Levin, Figure 10, shows the displayed rendered images that were necessarily displayed on a screen in the workflow of Levin.)
Regarding claim 13, Levin teaches “The method according to claim 1,”
“characterized in that the second term P comprises at least one component P1 whose effect is minimized for small intensity differences between neighboring pixels of the image IRk being rendered at the iteration k.” (Levin, in Equation 12, the second term P comprises P1,
PNG
media_image6.png
59
461
media_image6.png
Greyscale
wherein this term is decreased (minimized) for small intensity differences between neighboring pixels. In the equation, i and j correspond to coordinates in image x (image IRk being rendered at the iteration k), such that x(i,j) amounts to an intensity value at a coordinate location in image x. Note that the equation calculates differences between intensity values at neighboring coordinates (i+1 and j+1), and is therefore minimized when differences are small.)
Regarding claim 16, Claim 16 recites a device comprising “means” for receiving an image and “processing means” for modifying the image, with elements corresponding to the steps recited in Claim 1. Therefore, the recited elements of this claim are mapped to the analogous steps in the corresponding method claim. Levin additionally discloses a device with means of receiving and processing the image (Levin, Section 3, “However, this procedure still takes around 1 hour on a 2.4Ghz CPU for a 2 megapixel image. Our sparse deblurring code is available on the project webpage: http://groups.csail. mit.edu/graphics/CodedAperture.”)
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 11 is/are rejected under 35 U.S.C. 103 as being unpatentable over Levin in view of Alacoque (US20120063682A1).
Regarding claim 11, Levin teaches “The method according to claim 1,”
Levin does not expressly disclose “characterized in that the rendered image has a resolution greater than or equal to that of the combination of all the photosites of all the colors of the at least one sensor.”
Alacoque discloses “characterized in that the rendered image has a resolution greater than or equal to that of the combination of all the photosites of all the colors of the at least one sensor.” (Alacoque, Paragraphs 13 and 73, “The invention therefore relates to a method of demosaicing a digital raw image by a sensor optical device with monochrome photosites filtered by a matrix of color filters, the raw image being in the form of a matrix of pixels, each dedicated to a single color among several predetermined colors from the color filters, comprising a step for obtaining a color digital image of the same resolution as the raw image and in which each pixel has multiple color components and results from corresponding pixels in a luminance image and in chrominance images reconstructed from the raw image, further comprising the following steps: transformation of the raw image by applying local convolution kernels to its pixels, taking into account neighboring pixels of different colors, to obtain an image of low frequency coefficients using a low frequency local kernel and images of high frequency coefficients using high frequency local kernels, reconstruction of the luminance image using at least the image of low frequency coefficients, and reconstruction of the chrominance images using at least the images of high frequency coefficients.”; “Following the steps 112 and 116, during a step 118 performing a linear combination CL of images 10L, 10C1, and 10C2, an RGB color digital image 12 is obtained. In this color digital image 12 of the same resolution as the raw image 10, each red, green, or blue component of each pixel results from corresponding pixels in the luminance image 10L and in the chrominance images 10C1 and 10C2.”)
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to use the above strategy of Alacoque, wherein the rendered image has a resolution equal to the combined number of photosites, with respect to the image rendering of Levin.
The motivation for doing so would have been to ensure full sensor resolution is maintained (i.e. prevent resolution loss) for sharper final images. Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Levin with the above teaching of Alacoque to fully disclose “characterized in that the rendered image has a resolution greater than or equal to that of the combination of all the photosites of all the colors of the at least one sensor.”
Claim(s) 12, 14 and 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Levin in view of Farsiu (Multiframe demosaicing and super-resolution of color images)
Regarding claim 12, Levin teaches “The method according to claim 1,”
Levin does not expressly describe “characterized in that it comprises generating, from several input images, an initial version of the image being rendered IRk for the first iteration k=1 by a combination between these several input images Ie.”
Farsiu teaches generating an initial rendered image by combining several input images prior to subsequent iterative image processes (Farsiu, Sections III.F. and VI, “The optimality of this method is extensively discussed in [3], where it is shown that Zˆ–– is the weighted mean (mean or median operators, for the cases of L2 norm and L1 norm, respectively) of all measurements at a given pixel, after proper zero filling and motion compensation. We call this operation shift-and-add, which greatly speeds up the task of multiframe image fusion under the assumptions made. To compute the shift-and-add image, first the relative motion between all LR frames is computed. Then, a set of HR images is constructed by up-sampling each LR frame by zero filling. Then, these HR frames are registered with respect to the relative motion of the corresponding LR frames. A pixel-wise mean or median operation on the nonzero values of these HR frames will result in the shift-and-add image.”; “the blur kernel is space invariant, we can use the fast model of (16) to reconstruct the blurry image on the HR grid. The shift-and-add result of the demosaiced LR frames after bilinear interpolation,14 before deblurring and demosaicing is shown in Fig. 4(d). We used the result of the shift-and-add method as the initialization of the iterative multiframe demosaicing methods. We used the original set of frames (raw data) to reconstruct a HR image with reduced color artifacts. Fig. 5(a)–(c) shows the effect of the individual implementation of each regularization term (luminance, chrominance, and intercolor dependencies), described in Section IV.”)
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to generate the initial version of the image being rendered of Levin by using a combination of several input images, as taught by Farsiu.
The motivation for doing so would have been to compensate for noise and artifacts, as described in the Introduction of Farsiu: “Several distorting processes affect the quality of images acquired by commercial digital cameras. Some of the more important distorting effects include warping, blurring, color-filtering, and additive noise. A common image formation model for such imaging systems is illustrated in Fig. 1 1 . In this model, a real-world scene is seen to be warped at the camera lens because of the relative motion between the scene and camera. The imperfections of the optical lens results in the blurring of this warped image which is then subsampled and color-filtered at the CCD. The additive readout noise at the CCD will further degrade the quality of captured images. There is a growing interest in the multiframe image reconstruction algorithms that compensate for the shortcomings of the imaging system. Such methods can achieve high-quality images using less expensive imaging chips and optical components by capturing multiple images and fusing them.” Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Levin with the above teaching of Farsiu to fully disclose “characterized in that it comprises generating, from several input images, an initial version of the image being rendered IRk for the first iteration k=1 by a combination between these several input images Ie.”
Regarding claim 14, Levin teaches “The method according to claim 1,”
Levin does not expressly teach, “characterized in that the second term P comprises at least one component P3 whose effect is minimized for small differences in hue between neighboring pixels of the image being rendered IRk at the iteration k.”
Farsiu teaches at term that is minimized for small differences in hue between neighboring pixels of an image being rendered. (Farsiu, Section IV describes a cost function with “a penalty term to encourage smoothness in the chrominance component of the HR image (spatial chrominance penalty term);”. As a spatial penalty term of a cost function that encourages smooth chrominance, it decreases (is minimized) for small differences in hue (hue is part of chrominance) between neighboring pixels.)
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to incorporate, into the second term P of Levin that operates with respect to the image IRk, a component that is minimized for small differences in hue between neighboring pixels, as taught by Fasiu.
The motivation for doing so would have been to compensate for noise related to chrominance (hue/saturation), thereby reducing color artifacts. Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Levin with the above teaching of Farsiu to fully disclose “characterized in that the second term P comprises at least one component P3 whose effect is minimized for small differences in hue between neighboring pixels of the image being rendered IRk at the iteration k.”
Regarding claim 15, Levin teaches “The method according to claim 1,”
Levin does not expressly teach, “characterized in that the second term P comprises at least one component P2 whose effect is minimized for low frequencies of direction changes between neighboring pixels of the image IRk drawing a contour.
Farsiu teaches, “characterized in that the second term P comprises at least one component P2 whose effect is minimized for low frequencies of direction changes between neighboring pixels of the image IRk drawing a contour.” (Farsiu, Section IV describes a cost function with “a penalty term to encourage homogeneity of the location and orientation in different color bands (intercolor dependencies penalty term).” Section IV.D. elaborates, “This term penalizes the mismatch between locations or orientations of edges across the color bands. As described in Section III-D, the authors of [19] suggest a pixelwise intercolor dependencies cost function to be minimized. This term has the vector outer product norm of all pairs of neighboring pixels, which is solved by the finite element method.” Section III.C. further describes this penalty as between “adjacent color pixels”.)
It would have been obvious to a person having ordinary skill in the art before the time of the effective filing date of the claimed invention of the instant application to incorporate, into the second term P of Levin that operates with respect to the image IRk , a component that is minimized for low frequencies of direction changes between neighboring pixels, as taught by Fasiu.
The motivation for doing so would have been to reduce color artifacts, as described by Farsiu in the first sentence of section IIID. Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Levin with the above teaching of Farsiu to fully disclose “characterized in that the second term P comprises at least one component P2 whose effect is minimized for low frequencies of direction changes between neighboring pixels of the image IRk drawing a contour.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to AARON JOSEPH SORRIN whose telephone number is (703)756-1565. The examiner can normally be reached Monday - Friday 9am - 5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Sumati Lefkowitz can be reached at (571) 272-3638. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/AARON JOSEPH SORRIN/Examiner, Art Unit 2672
/SUMATI LEFKOWITZ/Supervisory Patent Examiner, Art Unit 2672
Prosecution Insights