DETAILED ACTION
Response to Amendment
Claims 1-14 were previously pending. Applicant’s amendment filed January 15, 2026, has been entered in full. Claims 1, 5, and 8 are amended. No claims are added or cancelled. Accordingly, claims 1-14 are now pending.
Examiner notes that the markup provided in the amendment does not accurately reflect all changes made to the claims, such as the deletion of “relevant” from claim 5 or the deletion of “familiar faces” from claim 6.
Response to Arguments
Applicant has amended claim 1 and asserts that the sub-sampling step “is fully supported by the written description, is enabled, and is set forth in definite terms” (Remarks filed January 15, 2026, hereinafter Remarks: Page 6). Examiner respectfully disagrees.
Applicant points to three different portions of the specification as providing support for the amended sub-sampling step (Remarks: Page 6).
First Applicant points to page 3, lines 10-25 as describing “definition of sub-sampling as omitting or downsizing pixels” (Remarks: Page 6). Examiner has reviewed this portion of the originally-filed specification, but finds that it merely restates the subject matter of original claim 1 and makes no mention of omitting or downsizing pixels.
Next, Applicant points to page 7, line 23 to page 8, line 6 as describing “execution of sub-sampling during capture to store a reduced-pixel image” (Remarks: Page 6). Examiner has reviewed this portion of the originally-filed specification, but finds that it merely defines the term “processor” and makes no mention of sub-sampling or storing a reduced-pixel image.
Next Applicant points to page 13, lines 14-21 as describing “generation and storage of a sub-sampled image comprising fewer pixels” (Remarks: Page 6). Examiner has reviewed this portion of the originally-filed specification, but finds that it merely provides an overview of the elements in Fig. 1 without any mention of generation or storage of a sub-sampled image comprising fewer pixels.
Examiner requests further explanation as to how the cited portions of the specification relate to the features Applicant has referenced in their Remarks.
Applicant further argues that the language of the amended claims “directly corresponds to the specification’s express definition of sub-sampling” (Remarks: Page 6). Examiner agrees that this phrase is reflected in the specification, but Applicant’s Remarks and the amendment still do not address the grounds of rejection in the previous Office Action, which were based on an inconsistency between the specification’s definition of sub-sampling and the examples provided, none of which meet that definition, and a lack of any description of an algorithm for performing the sub-sampling.
Applicant asserts that “sub-sampling by omitting or downsizing pixels during image capture was a well-known image processing technique at the time of filing, including structured approaches such as Bayer-pattern sampling and pixel binning, and thus would have been readily understood and implemented by a person of ordinary skill in the art without undue experimentation” (Remarks: Page 7).
First, Examiner notes that the rejection was for lack of adequate written description, rather than enablement. Second, as explained in the Non-Final Rejection and the rejections made below, Bayer sampling does not reduce a number of pixels in an image as required by the claimed invention. Third, pixel binning is not mentioned anywhere in the specification. Fourth, as was noted in the Non-Final Rejection (Page 9), “It is not enough that one skilled in the art could write a program to achieve the claimed function because the specification must explain how the inventor intends to achieve the claimed function to satisfy the written description requirement.” MPEP 2161.01, Subsection I. I.e., even if one of ordinary skill in the art would have known of ways to perform sub-sampling (e.g., via pixel binning), this would not be enough to satisfy the written description requirement.
Applicant asserts, without pointing to any specific evidence, that the description of executing the pre-trained neural network is adequate and argues that disclosure of an algorithm for executing the pre-trained neural network is not required because the claims do not invoke 35 U.S.C. 112(f) (Remarks: Pages 7-8). Examiner respectfully disagrees for the reasons presented in the previous rejection and below. As discussed extensively in the previous rejection, (a) disclosure of an algorithm is required for computer-implemented functional claim limitations, even if they do not invoke 35 U.S.C. 112(f) – see MPEP 2161.01, Subsection I – and (b) the specification does not disclose an algorithm.
Applicant makes similar assertions regarding the reconstructing, neural filling, and generating an output image features of claim 1 (Remarks: Pages 9-13), which are respectively non-persuasive for substantially the same reasons discussed above and the reasons presented in the rejections.
Applicant argues that, at point 10 of the Non-Final Rejection, Examiner has misapplied the written description standard to the present claims because the written description requirement does not mandate disclosure of source code or step-by-step algorithms (Remarks: Pages 13-14). Applicant also argues that “written description is not negated merely because a person of ordinary skill in the art could implement the disclosed functionality” (Remarks: Page 14). Examiner respectfully disagrees. Examiner has not rejected the claims for failure to disclose source code or because one of ordinary skill in the art could implement the disclosed functionality. Instead, Examiner has rejected the claims for failure to disclose an algorithm for performing computer-implemented functional claim limitations in sufficient detail such that one of ordinary skill in the art can reasonably conclude that the inventor possessed the claimed subject matter at the time of filing. The MPEP expressly requires that “[w]hen examining computer-implemented functional claims, examiners should determine whether the specification discloses the computer and the algorithm (e.g., the necessary steps and/or flowcharts) that perform the claimed function in sufficient detail such that one of ordinary skill in the art can reasonably conclude that the inventor possessed the claimed subject matter at the time of filing.” MPEP 2161.01, Subsection I. Examiner’s rejections clearly explain the basis for a determination that the algorithms are not described in sufficient detail, especially considering the MPEP’s instruction that “It is not enough that one skilled in the art could write a program to achieve the claimed function because the specification must explain how the inventor intends to achieve the claimed function to satisfy the written description requirement.” Id.
Applicant traverses Examiner’s determination that the scope of “sub-sampling” is unclear (Remarks: Page 15). Applicant argues that the examples of Bayer patterns and faded old photographs in par. [0030] are not examples of how sub-sampling is performed (Remarks: Page 15). Examiner respectfully disagrees. For example, par. [0030] states “In an implementation, a Bayer pattern that refers to a grid of color filters placed over the pixels of the image sensor can be used to perform sub-sampling. The Bayer pattern allows a single pixel to capture one of the color channels (i.e., red, green, and blue color channels) by omitting the information of another color.” (emphasis added). A plain reading of these statements clearly suggests that the Bayer pattern is performing the sub-sampling by omitting that is referenced earlier in the paragraph. As discussed regarding the written description rejections, there is no description of any other algorithm for performing the sub-sampling, nor any explanation of how Bayer sampling or old photograph fading may somehow be used in conjunction with sub-sampling, yet not provide the sub-sampling.
Applicant traverses Examiner’s determination that the scope of “one or more memory features” is indefinite (Remarks: Pages 15-17). Examiner respectfully disagrees. Familiar and/or human faces, as are specifically mentioned as examples of memory features in [0033] and original claim 6, plainly do not relate to text and neither the specification nor Applicant’s Remarks have resolved this contradiction with the express definition of “memory features” in the specification.
Applicant traverses the previous rejections under 35 U.S.C. 102, arguing that Ganguly does not disclose neural filling that has been added to claim 1 (Remarks: Pages 17-18). Examiner agrees. The previous rejections under 35 U.S.C. 102 are withdrawn. However, as explained in the previous rejection of claim 2 that is referenced in Applicant’s Remarks, Gupta does teach this feature. Accordingly, new rejections under 35 U.S.C. 103 over Ganguly in view of Gupta are presented below.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claim(s) 1-14 is/are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention.
The claims recite several limitations reciting functions that are apparently computer-implemented. MPEP 2161.01, Subsection I, provides instructions for determining whether there is adequate written description for a computer-implemented functional claim limitation, including the following:
“Similarly, original claims may lack written description when the claims define the invention in functional language specifying a desired result but the specification does not sufficiently describe how the function is performed or the result is achieved. For software, this can occur when the algorithm or steps/procedure for performing the computer function are not explained at all or are not explained in sufficient detail (simply restating the function recited in the claim is not necessarily sufficient). In other words, the algorithm or steps/procedure taken to perform the function must be described with sufficient detail so that one of ordinary skill in the art would understand how the inventor intended the function to be performed. See MPEP §§ 2163.02 and 2181, subsection IV. ”
“When examining computer-implemented functional claims, examiners should determine whether the specification discloses the computer and the algorithm (e.g., the necessary steps and/or flowcharts) that perform the claimed function in sufficient detail such that one of ordinary skill in the art can reasonably conclude that the inventor possessed the claimed subject matter at the time of filing. An algorithm is defined, for example, as "a finite sequence of steps for solving a logical or mathematical problem or performing a task." Microsoft Computer Dictionary (5th ed., 2002). Applicant may "express that algorithm in any understandable terms including as a mathematical formula, in prose, or as a flow chart, or in any other manner that provides sufficient structure." Finisar Corp. v. DirecTV Grp., Inc., 523 F.3d 1323, 1340, 86 USPQ2d 1609, 1623 (Fed. Cir. 2008) (internal citation omitted). It is not enough that one skilled in the art could write a program to achieve the claimed function because the specification must explain how the inventor intends to achieve the claimed function to satisfy the written description requirement. See, e.g., Vasudevan Software, Inc. v. MicroStrategy, Inc., 782 F.3d 671, 681-683, 114 USPQ2d 1349, 1356, 1357 (Fed. Cir. 2015) (reversing and remanding the district court’s grant of summary judgment of invalidity for lack of adequate written description where there were genuine issues of material fact regarding "whether the specification show[ed] possession by the inventor of how accessing disparate databases is achieved"). If the specification does not provide a disclosure of the computer and algorithm in sufficient detail to demonstrate to one of ordinary skill in the art that the inventor possessed the invention a rejection under 35 U.S.C. 112(a) or pre-AIA 35 U.S.C. 112, first paragraph, for lack of written description must be made.”
Claim 1 recites a step to “execute a sub-sampling routine during capture of the image by the image sensor to generate a sub-sampled input image by omitting or downsizing a subset of pixels of the first number of pixels in the image such that the sub-sampled input image comprises a second number of pixels less than the first number of pixels.” This describes a function of sub-sampling and appears to be computer-implemented at least because it relates to data processing and the claim has been amended to recite a processor being configured to perform it.
As explained in the rejections under 35 U.S.C. 112(b) below, the specification describes two examples of sub-sampling in the specification, but neither results in storing “a sub-sampled input image comprising a second number of pixels less than the first number of pixels” as required by the claimed invention. Providing only examples that do not accomplish the result required by the claimed invention does not describe the sub-sampling function “with sufficient detail so that one of ordinary skill in the art would understand how the inventor intended the function to be performed” (see MPEP quotation above), so claim 1 does not comply with the written description requirement of 35 U.S.C. 112(a).
Furthermore, as noted above, the claim has been amended to recite “a processor … configured to” execute the sub-sampling. Examiner has not identified adequate written description for this amendment and it appears to be new matter. The claim itself states that the sub-sampling occurs “during capture of the image by the image sensor”, rather than during post-processing of the captured image data by the processor. Furthermore, the examples of sub-sampling given in par. [0030] do not involve a processor. Bayer sampling involves the use of analog color filters and occurs in an image sensor – it is not performed by a processor. Indeed, the additional color information is “omitted” by the filters in the Bayer pattern before the image is even digitized. The fading of an old photograph also is not executed by a processor. The specification does mention a processor, but it is only described with respect to later processing of an already sub-sampled image. For example, [0032] states that “the processor is configured to execute the pre-trained neural network model on the sub-sampled input image”.
Claim 8 recites similar limitations and is also rejected under 35 U.S.C. 112(a) for substantially the same reasons as claim 1. Claims 2-7 and 9-14 are also rejected at least because they include the limitations of claim 1 or claim 8.
Claim 1 recites a step to “execute a pre-trained neural network model on the sub-sampled input image to detect one or more memory features in the sub-sampled input image.” This describes a function of detecting one or more memory features and is computer implemented at least because it is performed by a processor.
Examiner looks to the specification for description of an algorithm for how to perform this function, but finds none. The closest description is at par. [0057], which describes Fig. 2 and states in part:
“At step 206, a pre-trained neural network model is executed on the subsampled input image to detect one or more memory features in the sub-sampled input image and to reconstruct missing or
sub-sampled pixels corresponding to the detected one or
more memory features in the sub-sampled input image.”
This merely restates the claim and specifies a desired result (i.e., detection of one or more memory features) without any description of an algorithm for obtaining that desired result or any explanation of “how the inventor intended the function to be performed” (see MPEP quotation above). I.e., there is no explanation of how the pre-trained neural network is executed or any steps taken to achieve the detection. In view of this lack of detail, one of ordinary skill in the art could not reasonably conclude that the inventor possessed the claimed subject matter at the time of filing. Therefore, claim 1 does not comply with the written description requirement of 35 U.S.C. 112(a).
Claim 8 recites a similar limitation and is also rejected under 35 U.S.C. 112(a) for substantially the same reason as claim 1. Claims 2-7 and 9-14 are also rejected at least because they include the limitations of claim 1 or claim 8.
Claim 1 recites a step to “reconstruct missing or sub-sampled pixels corresponding to the detected one or more memory features in the sub-sampled input image using neural filling to generate replacement pixel values for the missing or sub-sampled pixels.” This describes a function of reconstructing one or more memory features and is computer implemented at least because it is performed by a processor.
Examiner looks to the specification for description of an algorithm for how to perform this function, but finds none. Relevant portions of the description are at par. [0057], which describes Fig. 2 and states in part:
“At step 206, a pre-trained neural network model is executed on the subsampled input image to detect one or more memory features in the sub-sampled input image and to reconstruct missing or
sub-sampled pixels corresponding to the detected one or
more memory features in the sub-sampled input image.”
and at par. [0045], which states in part:
“The reconstruction process is used for restoring image details and
enhancing the overall quality of the image, thereby ensuring that memory features, which may have been lost or blurred due to sub-sampling, are made more legible and recognizable. Furthermore, the term "neural filling" refers to a technique that utilizes a pre-trained neural network to generate or fill in missing or degraded pixels in order to generate a visually clear image with an enhanced image resolution. The technical effect of employing neural filling to increase
the resolution of detected memory features in sub-sampled images is to enhance the clarity, recognition, and memory so retention of these features, ultimately leading to an improved user experience in various applications, including image processing and computer vision tasks.”
This merely restates the claim and specifies a desired result (i.e., reconstruction of one or more memory features) without any description of an algorithm for obtaining that desired result or any explanation of “how the inventor intended the function to be performed” (see MPEP quotation above). I.e., there is no explanation of how the pre-trained neural network is executed, any steps taken to achieve the reconstruction, how the neural filling is performed, or any steps taken to generate replacement pixel values. For example, there is no explanation of how the pre-trained neural network performs the neural filling or is utilized to accomplish neural filling. In view of this lack of detail, one of ordinary skill in the art could not reasonably conclude that the inventor possessed the claimed subject matter at the time of filing. Therefore, claim 1 does not comply with the written description requirement of 35 U.S.C. 112(a).
Claim 8 recites a similar limitation and is also rejected under 35 U.S.C. 112(a) for substantially the same reason as claim 1. Claims 2-7 and 9-14 are also rejected at least because they include the limitations of claim 1 or claim 8. Claims 2 and 9 further recite employing neural filling and further lack adequate written description for substantially the same reasons discussed above.
Claim 1 recites a step to “generate an output image in which the one or more memory features are enhanced and present the output image in a legible form.” This describes a function of generating an image and is computer implemented at least because it is performed by a processor.
Examiner looks to the specification for description of an algorithm for how to perform this function, but finds none. The closest description is at par. [0057], which describes Fig. 2 and states in part:
“At step 208, an output image is generated with an enhanced one or more memory features present in a legible form.”
This merely restates the claim and specifies a desired result (i.e., generation of an output image) without any description of an algorithm for obtaining that desired result or any explanation of “how the inventor intended the function to be performed” (see MPEP quotation above). I.e., there is no explanation of how the output image is generated, how the memory features are enhanced, or any steps taken to achieve the generation of an output image with enhanced one or more memory features present in a legible form. In view of this lack of detail, one of ordinary skill in the art could not reasonably conclude that the inventor possessed the claimed subject matter at the time of filing. Therefore, claim 1 does not comply with the written description requirement of 35 U.S.C. 112(a).
Claim 8 recites a similar limitation and is also rejected under 35 U.S.C. 112(a) for substantially the same reason as claim 1. Claims 2-7 and 9-14 are also rejected at least because they include the limitations of claim 1 or claim 8.
Examiner notes that the prior art did include some techniques capable of achieving the desired results expressed in the claim limitations noted above. See, for example, the prior art rejections below. However, Examiner also notes that “It is not enough that one skilled in the art could write a program to achieve the claimed function because the specification must explain how the inventor intends to achieve the claimed function to satisfy the written description requirement.” MPEP 2161.01, Subsection I.
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim(s) 1-14 is/are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 1 recites “captur[ing] an image comprising a first number of pixels” and a processor being configured to “execute a sub-sampling routine during capture of the image by the image sensor to generate a sub-sampled input image by omitting or downsizing a subset of pixels of the first number of pixels in the image such that the sub-sampled input image comprises a second number of pixels less than the first number of pixels”.
“A claim, although clear on its face, may also be indefinite when a conflict or inconsistency between the claimed subject matter and the specification disclosure renders the scope of the claim uncertain as inconsistency with the specification disclosure or prior art teachings may make an otherwise definite claim take on an unreasonable degree of uncertainty.” MPEP 2173.03.
Sub-sampling is described at par. [0030] of the specification. (Note that this and all further references to the specification are to the as-published specification – i.e., US 2025/0166127 A1 – unless otherwise stated). Par. [0030] is reproduced below:
PNG
media_image1.png
200
400
media_image1.png
Greyscale
This description includes inconsistencies and contradictions that make the scope of the term “sub-sampling” unclear and therefore renders the scope of claim 1, which recites executing sub-sampling, indefinite.
On the one hand, both the claim and the specification state that sub-sampling produces an image “comprising a second number of pixels less than the first number of pixels.” This appears to clearly require a reduction in the total number of pixels.
On the other hand, the specification describes two examples of sub-sampling: (A) one using a Bayer pattern and (B) “a partially faded old photograph of a family gathering”, but neither of these examples includes any reduction in a number of pixels.
Regarding example (A), as is plainly described in par. [0030], use of a Bayer pattern reduces the amount of color information captured at each pixel, but does not decrease the total number of pixels in an image. I.e., the same number of pixels are present in a Bayer pattern image, they just have less color information (e.g., only one of red, green, or blue colors, rather than all three). The specification describes Bayer sampling as “omitting the information of another color” ([0030]). While the total amount of information in a Bayer pattern image is reduced relative to, for example, an image with full 3-channel color information at each pixel (e.g., a demosaiced image), the number of pixels is unchanged.
Regarding example (B), the phrase “partially faded old photograph” suggests an analog, printed image at least because digital images do not fade over time. Analog images do not have any pixels, so it is unclear how an analog image could have a lower number of pixels. Furthermore, the claim recites that the sub-sampling is executed “during capture”, but any sub-sampling (or, more generally, loss of information) caused by the fading of a photograph is caused by later degradation of that photograph that occurs after capture.
To summarize, the claim recites sub-sampling performed during image capture that reduces a number of pixels. Some of this language is repeated in the specification, but neither of the examples given in the specification involve any reduction in a number of pixels and one of the examples does not occur during image capture. This inconsistency and contradiction makes the scope of the claimed “sub-sampling” unclear. For example, does capturing an image with a Bayer pattern read on the claimed sub-sampling, even though it does not reduce a number of pixels? This ambiguity makes the scope of the claim indefinite.
Claim 8 recites a similar limitation and is also indefinite for substantially the same reason as claim 1. Claims 2-7 and 9-14 are also indefinite at least because they include the limitations of claim 1 or claim 8.
Claim 1 recites detecting “one or more memory features.”
“A claim, although clear on its face, may also be indefinite when a conflict or inconsistency between the claimed subject matter and the specification disclosure renders the scope of the claim uncertain as inconsistency with the specification disclosure or prior art teachings may make an otherwise definite claim take on an unreasonable degree of uncertainty.” MPEP 2173.03.
The phrase “one or more memory features” is defined at par. [0033] of the specification, which is reproduced below:
PNG
media_image2.png
200
400
media_image2.png
Greyscale
This description includes inconsistencies and contradictions that make the scope of the term “one or more memory features” unclear and therefore renders the scope of claim 1, which recites detecting one or more memory features, indefinite.
On the one hand, the specification includes many statements clearly stating that “one or more memory features” pertain to text. For example, the first sentence states that “the term ‘one or more memory features’ refers to elements or characteristics within the input image or visual content that are related to text” (emphasis added).
On the other hand, the second sentence of par. [0033] states that “The one or more memory features include components, human faces, patterns, or details in the input image that involve written or printed words, characters, symbols, or any other form of textual information.” This does mention “human faces,” but human faces do not have any apparent relevance to textual information. A vast majority of human faces do not include any text.
The specification includes contradiction and inconsistency because it clearly states that “one or more memory features” refers to elements of an image “that are related to text”, yet also recites that one or more memory features may include human faces, which do not have any apparent relation to textual information. This is further compounded by additional limitations the claims. For example, original claim 1 requires that enhanced “one or more memory features” are “present in a legible form” (emphasis added), where “legible” generally relates to text, while original claim 6 recites that one or more memory features can include not only letter features, but also “familiar faces or user interface elements or components”.
This inconsistency and contradiction makes the scope of the claimed “one or more memory features” unclear. For example, does detecting a face read on detecting “one or more memory features,” even if the face is unrelated to any text? This ambiguity makes the scope of the claim indefinite.
Additionally, or alternatively, the claim recites “one or more memory features” (emphasis added). The ordinary meaning of “memory features” is some kind of feature relating to memory. However, none of the “memory features” described in par. [0033] of the specification appear to relate to memory. For example, “texts, which are not easily legible or recognizable due to various factors, such as poor image quality, distortion, or other forms of degradation” have low quality, but do not appear to relate to memory in any way.
“Consistent with the well-established axiom in patent law that a patentee or applicant is free to be his or her own lexicographer, a patentee or applicant may use terms in a manner contrary to or inconsistent with one or more of their ordinary meanings if the written description clearly redefines the terms” (emphasis added). MPEP 2173.05(a), Subsection III.
As explained above, Applicant appears to be using the term “memory features” contrary to or inconsistent with its ordinary meaning because the examples given in the specification do not relate to memory. However, as also explained above, the written description does not clearly redefine the term. This use of the term contrary to its ordinary meaning further makes the scope of the claim unclear and renders the claim indefinite.
Claim 8 recites a similar limitation and is also indefinite for substantially the same reason as claim 1. Claims 2-7 and 9-14 are also indefinite at least because they include the limitations of claim 1 or claim 8.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-2, 4, 6-9, 11, and 13-14 is/are rejected under 35 U.S.C. 103 as being unpatentable over ‘Ganguly’ (US 2023/0049339 A1) in view of ‘Gupta’ (“Del-Net: A Single-Stage Network for Mobile Camera ISP,” 3 Aug. 2021).
Regarding claim 1, Ganguly teaches an apparatus (see, e.g., Figures 1-3B, 4B, 7A-C; For providing a clear and concise explanation, the mapping below mostly focuses on Fig. 4B), comprising:
an image sensor (e.g., Fig. 4B, sensor 402) to capture an image comprising a first number of pixels (e.g., Fig. 4B, full-resolution image with 12 Mpix);
a processor (e.g., Fig. 1, processors 112 and 118; Fig. 4B, high power ISP 406 and/or low power ISP 408; While these figures illustrate two separate processors and the claim recites “a” processor, Examiner notes that (A) par. [0046] of Ganguly states that “Although multiple processor blocks (e.g., processor 112, processor 118, processor 126, etc.) are depicted, a single processor may be utilized to carry out all processing tasks”, and (B) par. [0006] of the published specification [i.e., US 2025/0166127 A1] states that “where the indefinite article is used, the specification is to be understood as contemplating plurality as well as singularity, unless the context requires otherwise” and par. [0031] of the published specification states that the term “processor” “may refer to one or more individual processors”) communicably coupled to the image sensor (e.g., Fig. 1, arrows show communicative coupling), the processor being configured to:
execute a sub-sampling routine during capture of the image by the image sensor to generate a sub-sampled input image by omitting or downsizing a subset of pixels of the first number of pixels in the image such that the sub-sampled input image comprises a second number of pixels less than the first number of pixels (e.g., Fig. 4B, low-resolution image produced by sensor 402, which includes only 504x376 [i.e., 189,504] pixels, which is less than “12 Mpix” [i.e., 12 megapixels, which is approximately 12 million pixels]; The sub-sampling may be, e.g., 8x8 binning, which omits a subset of pixels to produce an image with a lower number of pixels – Fig. 4B);
execute a pre-trained neural network model on the sub-sampled input image to detect one or more memory features in the sub-sampled input image (e.g., [0087], Fig. 4B, low power computations 412 are performed to detect and object and/or region of interest (ROI) in the low-resolution/sub-sampled input; e.g., [0072], [0074], the low-power/resolution ROI detection may be performed using a pre-trained neural network; While the scope of “one or more memory features” is unclear – see ‘112(b) rejection – they appear to relate to textual and/or facial features and Ganguly describes detecting these as regions/objects of interest at, e.g., [0041], [0042], [0087], [0145]) and
reconstruct missing or sub-sampled pixels corresponding to the detected one or more memory features in the sub-sampled input image using neural filling (see Note Regarding Neural Filling below) to generate replacement pixel values for the missing or sub-sampled pixels (e.g., [0091], Fig. 4B, high-power ISP 406 produces a cropped, processed, full-resolution image of the region/object of interest [i.e., the memory feature] detected in the sub-sampled input image; At least two different aspects of this processing can be seen as “reconstructing missing or sub-sampled pixels …”; First, the processing performed by high power image signal processor 406 includes debayering; RAW images captured in a Bayer format include samples for only one color channel value per pixel – samples for other colors/channels are missing; Accordingly, pixels of RAW Bayer format images are at least “sub-sampled pixels”; The debayering process [also known in the prior art as demosaicing or demosaicking] reconstructs a full set of channel/color samples at each pixel, thereby generating replacement pixel values for the missing color information; Second, the output of high power processor 406 is a full-resolution version of the ROI/memory feature detected in the sub-sampled input image, so the high-power processor 406 can be seen as reconstructing and generating replacement pixel values for the missing or sub-sampled pixels that were removed from the full-resolution image to create the low-resolution image for the ROI detected as a memory feature, and to thus fall within the scope of the limitation); and
generate an output image in which the one or more memory features are enhanced and present the output image in a legible form (e.g., [0091], Fig. 4B, output image 418 is generated and output for use in further processing; The output image is “legible” at least because it can be further processed to recognize text, faces, etc. – see, e.g., [0044]-[0045], and the face in image 418 as illustrated in Fig. 4B).
Note Regarding Neural Filling. As discussed above, the reconstruction of missing or sub-sampled pixels corresponding to the detected one or more memory features in Ganguly comprises using a high-power image signal processor (ISP) to provide a processed image data that has been “debayered, color corrected, shading corrected, etc.” ([0091]). The processed image data is a high-resolution version of the detected one or more memory features in the sub-sampled input image (see rejection of claim 1).
Ganguly does not discuss details of the ISP’s operations. In particular, it does not teach the ISP employing neural filling.
However, Gupta does teach a neural ISP pipeline that performs various image processing steps including debayering (e.g., Section 2.1, demosaicing; Sec. 3, 1st par.), color correction (e.g., Sec. 2.1), and shading correction (e.g., Sec. 2.1). The neural ISP taught by Gupta performs neural filling to generate replacement pixel values for the missing or sub-sampled pixels at least because the neural ISP fills in channel data missing from the input RAW image (e.g., Sec. 3, 1st par., Input image I_0 with only 1 channel is filled in by the neural network to produce output image I_F that has been filled in by the neural network to have 3 channels).
Gupta teaches that “it has been shown that the performance of a smartphone camera ISP can be significantly improved by the use of deep learning techniques” and cites several examples from the prior art (Sec. 2.2). Gupta teaches that its specific neural ISP technique provides image quality comparable to the state of the art with advantageously reduced computational complexity (e.g., Sec. 7).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to modify the apparatus of Ganguly with the neural ISP including neural filling of Gupta in order to improve the apparatus with the reasonable expectation that this would result in an apparatus that could advantageously produce high-quality image data with relatively low computational complexity. This technique for improving the apparatus of Ganguly was within the ordinary ability of one of ordinary skill in the art based on the teachings of Gupta.
Therefore, it would have been obvious to one of ordinary skill in the art to combine the teachings of Ganguly and Guptato obtain the invention as specified in claim 1.
Regarding claim 2, Ganguly in view of Gupta teaches the apparatus of claim 1, and Gupta further teaches that the reconstruction of missing or sub-sampled pixels corresponding to the detected one or more memory features comprises employing neural filling to increase the resolution of the detected one or more memory features in the sub-sampled input image (e.g., Sec. 3, 1st par., Input image I_0 with only 1 channel is filled in by the neural network to produce output image I_F that has been filled in by the neural network to have 3 channels).
Regarding claim 4, Ganguly in view of Gupta teaches the apparatus of claim 1, and Ganguly further teaches that the image sensor is a video-see-through (VST) color camera sensor (The definition of this term at par. [0029] of the published specification is noted; e.g., [0078], [0081], [0083], Figs. 3A-B, wearable including VST display 304 and camera 316; Fig. 4B, sensor 402 produces images with color information).
Regarding claim 6, Ganguly in view of Gupta teaches the apparatus of claim 1, and Ganguly further teaches that the one or more memory features include one or more of: letter features (e.g., [0043], Fig. 1, ROI 130 includes text), familiar faces (e.g., [0075], Fig. 4B, image 418 [best seen when zoomed]; Note that any face will be familiar to at least someone, and therefore falls within the BRI of a “familiar” face), or user interface elements or components (e.g., [0043], Fig. 1, ROI 128 includes analog clock, which is an element or component of a user interface for telling time).
Regarding claim 7, Ganguly in view of Gupta teaches the apparatus of claim 6, and Ganguly further teaches that the user interface elements or components are related to one or more of: control device, mechanical devices, or display devices used in a training and simulation system (First, Examiner notes that memory features including user interface elements are not required by the claim – see “or” in claim 6; Second, [0043], Fig. 1, ROI 128 includes an analog clock, which is at least a time display device; Clocks may be used in a variety of training and simulation systems; For example, clocks may be used to time a scrimmage in which athletes are being trained in a simulated game; For at least these reasons, an analog clock – such as the one described by Ganguly – falls within the scope of “display devices used in a training and simulation system”).
Regarding claim 8, Examiner notes that the claim recites a method that is substantially the same as the method performed by the apparatus of claim 1. Ganguly in view of Gupta teaches the apparatus of claim 1 (see above). Accordingly, claim 8 is also rejected under 35 U.S.C. 103 as being unpatentable over Ganguly in view of Gupta for substantially the same reasons as claim 1.
Regarding claim 9, Examiner notes that the claim recites a method that is substantially the same as the method performed by the apparatus of claim 2. Ganguly in view of Gupta teaches the apparatus of claim 2 (see above). Accordingly, claim 9 is also rejected under 35 U.S.C. 103 as being unpatentable over Ganguly in view of Gupta for substantially the same reasons as claim 2.
Regarding claim 11, Examiner notes that the claim recites a method that is substantially the same as the method performed by the apparatus of claim 4. Ganguly in view of Gupta teaches the apparatus of claim 4 (see above). Accordingly, claim 11 is also rejected under 35 U.S.C. 103 as being unpatentable over Ganguly in view of Gupta for substantially the same reasons as claim 4.
Regarding claim 13, Examiner notes that the claim recites a method that is substantially the same as the method performed by the apparatus of claim 6. Ganguly in view of Gupta teaches the apparatus of claim 6 (see above). Accordingly, claim 13 is also rejected under 35 U.S.C. 103 as being unpatentable over Ganguly in view of Gupta for substantially the same reasons as claim 6.
Regarding claim 14, Examiner notes that the claim recites a method that is substantially the same as the method performed by the apparatus of claim 7. Ganguly in view of Gupta teaches the apparatus of claim 7 (see above). Accordingly, claim 14 is also rejected under 35 U.S.C. 103 as being unpatentable over Ganguly in view of Gupta for substantially the same reasons as claim 7.
Claim(s) 3 and 10 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ganguly in view of Gupta as applied above and further in view of ‘Chen’ (“TinyDet: Accurate Small Object Detection in Lightweight Generic Detectors,” 7 April 2023).
Regarding claim 3, Ganguly in view of Gupta teaches the apparatus of claim 1.
Ganguly teaches detecting objects or regions of interest in a low-resolution, sub-sampled image as “one or more memory features” (see, e.g., [0087] and rejection of claim 1). The object/region of interest detection may use machine learning, such as a neural network (e.g., [0072], [0074]).
Ganguly does not teach any specific object sizes or ranges of object sizes present in the low-resolution, sub-sampled image, including a specific “size range of 5 to 25 pixels” as recited in the claim.
However, Chen does teach a neural network for object/region of interest detection (e.g., Sec. 3 and Fig. 4) that does detect objects/regions “present on a size range of 5 to 25 pixels” of an input image. For example, small objects are detected, with small objects being defined as being in regions “≤ (32 x 32)” pixels (Sec. 5), which includes the range recited in the claim. In another example, the proposed region proposal network (RPN) includes anchors with heights and widths of 12.8 pixels (Sec. 4.1, last par.), and a size (as measured by height or width) of 12.8 falls within the claimed range of 5 to 25. Note that an RPN performs “objectness” tests within anchor boxes, so the size of an anchor box generally corresponds to the size/scale of objects that it will detect.
Ganguly requires a low-power neural network for object detection (e.g., [0075]; [0087], Fig. 4B, low power compute 412). Chen teaches such a low-power neural network for object detection (e.g., second page, 1st paragraph, TinyDet), and further teaches that it can advantageously detect objects of various sizes including small sizes (e.g., Sec. 4.2 and Table 2, various versions of TinyDet are able to detect small, medium and large objects) with improved performance relative to similar networks (e.g., Sec. 5; Sec. 4.2; Table 3).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to modify the apparatus of Ganguly in view of Gupta as applied above with the object/region of interest detector of Chen in order to improve the apparatus with the reasonable expectation that this would result in an apparatus that could advantageously perform lightweight detection for a variety of object/region sizes, including small ones. This technique for improving the apparatus of Ganguly in view of Gupta was within the ordinary ability of one of ordinary skill in the art based on the teachings of Chen and Ganguly.
Therefore, it would have been obvious to one of ordinary skill in the art to combine the teachings of Ganguly, Gupta and Chen to obtain the invention as specified in claim 3.
Regarding claim 10, Examiner notes that the claim recites a method that is substantially the same as the method performed by the apparatus of claim 3. Ganguly in view of Gupta and Chen teaches the apparatus of claim 3 (see above). Accordingly, claim 10 is also rejected under 35 U.S.C. 103 as being unpatentable over Ganguly in view of Gupta and Chen for substantially the same reasons as claim 3.
Claim(s) 5 and 12 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ganguly in view of Gupta as applied above, and further in view of ‘Ribani’ (“A Survey of Transfer Learning for Convolutional Neural Networks,” 2019).
Regarding claim 5, Ganguly in view of Gupta teaches the apparatus of claim 1.
Ganguly teaches using a neural network that has been trained to accept input sub-sampled images and output object detections (e.g., [0074]).
Ganguly does not explicitly teach details of how the training is performed. In particular, Ganguly does not teach training with transfer learning.
However, Ribani does teach how to perform neural network training using transfer learning. For example, at page 47, Ribani teaches:
PNG
media_image3.png
200
400
media_image3.png
Greyscale
As one of ordinary skill in the art would recognize, the object/region detection neural network in Ganguly could be trained using the transfer learning approach outlined by Ribani. This would include acquiring a training dataset of images captured using the sub-sampling, which is analogous to the “smaller dataset” labelled with “different types of guitar like stratocaster, telecaster or les paul” in Ribani’s example. The transfer learning would also include selecting a relevant pre-trained neural network model, such as one of the pre-trained models provided by the most-used machine learning frameworks in Ribani’s example, and fine-tuning it using the smaller training dataset as in Ribani’s example.
Ribani teaches that “it’s very common to have a problem to solve in the same domain that is not labeled” and that transfer learning can solve this problem (see quotation above). Applying transfer learning in this way would advantageously allow Ganguly to obtain a trained object/region of interest detector with a relatively smaller dataset of sub-sampled images, thereby advantageously reducing labeling cost.
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to modify the apparatus of Ganguly in viewof Gupta as applied above with the transfer learning of Ribani in order to improve the apparatus with the reasonable expectation that this would result in an apparatus that could train its neural network with a relatively smaller dataset, thereby advantageously reducing labeling cost. This technique for improving the apparatus of Ganguly in view of Gupta was within the ordinary ability of one of ordinary skill in the art based on the teachings of Ribani.
Therefore, it would have been obvious to one of ordinary skill in the art to combine the teachings of Ganguly, Gupta and Ribani to obtain the invention as specified in claim 5.
Regarding claim 12, Examiner notes that the claim recites a method that is substantially the same as the method performed by the apparatus of claim 5. Ganguly in view of Gupta and Ribani teaches the apparatus of claim 5 (see above). Accordingly, claim 12 is also rejected under 35 U.S.C. 103 as being unpatentable over Ganguly in view of Gupta and Ribani for substantially the same reasons as claim 5.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to GEOFFREY E SUMMERS whose telephone number is (571)272-9915. The examiner can normally be reached Monday-Friday, 7:00 AM to 3:30 PM ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chan Park can be reached at (571) 272-7409. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/GEOFFREY E SUMMERS/Examiner, Art Unit 2669