DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Receipt is acknowledged that application claims priority to foreign application with application number JAPAN 2023-182752 dated 10/24/2023. Copies of certified papers required by 37 CFR 1.55 have been received. Priority is acknowledged under 35 USC 119(e) and 37 CFR 1.78.
Specification
The title of the invention is not descriptive. A new title is required that is clearly indicative of the invention to which the claims are directed. The following title is suggested: IMAGE PROCESSING APPARATUS, IMAGE CAPTURING APPARATUS, CONTROL METHOD, AND RECORDING MEDIUM FOR EFFICIENT EDGE PROCESSING.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are:
"a computation unit configured to" in claim 1, in the specification “a computation unit” is defined as “ the CNN computation unit 101 is configured to include a CPU 102, multiply accumulation processing unit 103, and a shared memory 105.” ¶0025.
" an obtainment unit configured to" in claim 1, in the specification “an obtainment unit” is defined as “ obtains the plurality of tiles based on the image that has been obtained through image capture performed by the image capturing unit.” ¶0010.
" a control unit configured to " in claim 1, in the specification “a control unit” is defined as “ wherein the control unit controls the computation unit” ¶0009.
" an output unit configured to " in claim 6, in the specification “output unit” is not defined, it is described in claim 6 as “ at least one processor and/or circuit to function as an output unit”.
" an generation unit configured to " in claim 12 , in the specification “an generation unit” is defined as “under control of the CPU 102, the tile generation unit 107 determines whether the number of pixels in output data of a target tile is different from the number of pixels in input data of this target tile. In a case where the tile generation unit 107 has determined that the number of pixels in the output data of the target tile is different from the number of pixels in the input data thereof, the tile generation unit 107 causes processing to proceed to step S602; in a case where the tile generation unit 107 has determined that the numbers are the same, it ends the present generation processing.” In ¶0073.
" an image capturing unit" in claim 13 , in the specification “an image capturing unit” is defined as “based on the image that has been obtained through image capture performed by the image capturing unit.” In ¶0010.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
The claims in this application are given their broadest reasonable interpretation using the
plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification.
Under MPEP 2143.03, "All words in a claim must be considered in judging the patentability of that claim against the prior art." In re Wilson, 424 F.2d 1382, 1385, 165 USPQ 494, 496 (CCPA 1970). As a general matter, the grammar and ordinary meaning of terms as understood by one having ordinary skill in the art used in a claim will dictate whether, and to what extent, the language limits the claim scope. Language that suggests or makes a feature or step optional but does not require that feature or step does not limit the scope of a claim under the broadest reasonable claim interpretation. In addition, when a claim requires selection of an element from a list of alternatives, the prior art teaches the element if one of the alternatives is taught by the prior art. See, e.g., Fresenius USA, Inc. v. Baxter Int’l, Inc., 582 F.3d 1288, 1298, 92 USPQ2d 1163, 1171 (Fed. Cir. 2009).
Claims 1, 6 and 12 recite “and/or” then listing “at least one processor and/or circuit”. Since “and/or” is disjunctive, any one of the elements found in the prior art is sufficient to reject the claim. While citations have been provided for completeness and rapid prosecution, only one element is required. Because, on balance, it appears the disjunctive interpretation enjoys the most specification support and for that reason the disjunctive interpretation (one of A, B OR C) is being adopted for the purposes of this Office Action. Applicant’s comments and/or amendments relating to this issue are invited to clarify the claim language and the prosecution history.
Claim Rejections - 35 USC § 112
Claim limitation “output unit” in claim 6 invokes 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. However, the written description fails to disclose the corresponding structure, material, or acts for performing the entire claimed function and to clearly link the structure, material, or acts to the function. The specification is silent on what structure or hardware makes up the output unit or how it performs its functions, the only context given is in the claim itself which states that “an output unit configured to generate output data” which does not convey structure to perform the task. Therefore, the claim is indefinite and is rejected under 35 U.S.C. 112(b) or pre-AIA 35 U.S.C. 112, second paragraph.
Applicant may:
(a) Amend the claim so that the claim limitation will no longer be interpreted as a limitation under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph;
(b) Amend the written description of the specification such that it expressly recites what structure, material, or acts perform the entire claimed function, without introducing any new matter (35 U.S.C. 132(a)); or
(c) Amend the written description of the specification such that it clearly links the structure, material, or acts disclosed therein to the function recited in the claim, without introducing any new matter (35 U.S.C. 132(a)).
If applicant is of the opinion that the written description of the specification already implicitly or inherently discloses the corresponding structure, material, or acts and clearly links them to the function so that one of ordinary skill in the art would recognize what structure, material, or acts perform the claimed function, applicant should clarify the record by either:
(a) Amending the written description of the specification such that it expressly recites the corresponding structure, material, or acts for performing the claimed function and clearly links or associates the structure, material, or acts to the claimed function, without introducing any new matter (35 U.S.C. 132(a)); or
(b) Stating on the record what the corresponding structure, material, or acts, which are implicitly or inherently set forth in the written description of the specification, perform the claimed function. For more information, see 37 CFR 1.75(d) and MPEP §§ 608.01(o) and 2181.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
The USPTO “Interim Guidelines for Examination of Patent Applications for Patent Subject Matter Eligibility” (Official Gazette notice of 23 February 2010), Annex IV, reads as follows:
The USPTO recognizes that applicants may have claims directed to computer readable media that cover signals per se, which the USPTO must reject under 35 U.S.C. § 101 as covering both non-statutory subject matter and statutory subject matter. In an effort to assist the patent community in overcoming a rejection or potential rejection under 35 U.S.C. § 101 in this situation, the USPTO suggests the following approach.
A claim drawn to such a computer readable medium that covers both transitory and non-transitory embodiments may be amended to narrow the claim to cover only statutory embodiments to avoid a rejection under 35 U.S.C. § 101 by adding the limitation "non-transitory" to the claim. Cf. Animals - Patentability, 1077 Off. Gaz. Pat. Office 24 (April 21, 1987) (suggesting that applicants add the limitation "non-human" to a claim covering a multi-cellular organism to avoid a rejection under 35 U.S.C. § 101). Such an amendment would typically not raise the issue of new matter, even when the specification is silent because the broadest reasonable interpretation relies on the ordinary and customary meaning that includes signals per se. The limited situations in which such an amendment could raise issues of new matter occur, for example, when the specification does not support a non-transitory embodiment because a signal per se is the only viable embodiment such that the amended claim is impermissibly broadened beyond the supporting disclosure. See, e.g., Gentry Gallery, Inc. v. Berkline Corp., 134 F.3d 1473(Fed. Cir. 1998).
Claim 15 is rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter as follows. Claim 15 defines a “computer-readable recording medium” embodying functional descriptive material. However, the claim does not define a non-transitory computer-readable medium or memory and is thus non-statutory for that reason (i.e., “examination the pending claims must be interpreted as broadly as their terms reasonably allow). The broadest reasonable interpretation of a claim drawn to a computer readable medium (also called machine readable medium and other such variations) typically covers forms of non-transitory tangible media and transitory propagating signals per se in view of the ordinary and customary meaning of computer readable media, particularly when the specification is silent. See MPEP 2111.01.
When the broadest reasonable interpretation of a claim covers a signal per se, the claim must be rejected under 35 U.S.C. § 101 as covering non-statutory subject matter. See In see Official Gazette Notice 1351 OG212, February 23,2010). That is, the scope of the presently claimed “computer program product ” typically covers forms of non-transitory tangible media and transitory propagating signals per se. The examiner suggests amending the claim to embody the program on a “computer readable recording medium” and adding the limitation ”non-transitory ” to the claim or equivalent in order to make the claim statutory. Any amendment to the claim should be commensurate with its corresponding disclosure.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1-12, 14, and 15 are rejected under 35 U.S.C. 103 as unpatentable over Bronder (US Patent Publication US 2025/0371661 A1, hereafter referred to as Bronder) in view of Valpreda, Emanuele, et al. "Hw-flow-fusion: Inter-layer scheduling for convolutional neural network accelerators with dataflow architectures." Electronics 11.18 (2022): 2933, hereafter referred to as Valpreda.
Regarding Claim 1, Bronder teaches an image processing apparatus (Bronder ¶0035 and Fig 1 120 discloses a rendering engine that includes processes for altering the attributes of the images being processed), comprising:
at least one processor and/or circuit (Bronder ¶0085, ¶0089, discloses the disclosed methods may be practiced by a computer system comprising a computer including one or more processors); and
at least one memory storing computer program, which causes the at least one processor and/or circuit to function (Bronder ¶0083, ¶0085, discloses a computer system comprising a computer readable media such as computer memory. In particular, the computer memory may store computer-executable instructions that when executed by one or more processors cause various functions to be performed) as following units:
a computation unit configured to execute (Bronder ¶0037 discloses the super resolution model performing extensive computations) convolutional computation processing in a neural network with respect to input data (Bronder ¶0015, ¶0016, ¶0049 discloses using convolution operations of the resolution machine learning model to the plurality of input tiles),
an obtainment unit configured to obtain (Bronder ¶0006, ¶0013 discloses super resolution processing systems configured to subdivide input images into a plurality of smaller tiles) a plurality of tiles that respectively correspond to partial regions in an image (Bronder Fig 7, 730, ¶0015, ¶0018 discloses splitting the input images into a plurality of input tiles, where the input tiles have a tile size and halo region, the halo region being the edge region of the tiles which can change based on the halo region size), and
a control unit configured to perform control (Bronder ¶0012 discloses dynamically controlling the tiling of images for processing by super-resolution models) so as to cause the computation unit to execute the convolutional computation processing (Bronder ¶0037 discloses the super resolution model performing extensive computations) while using each of the plurality of tiles as the input data (Bronder ¶0015, ¶0016, ¶0049 discloses using convolution operations of the resolution model to the plurality of input tiles),
wherein the control unit controls (Bronder ¶0012 discloses dynamically controlling the tiling of images for processing by super-resolution models) the computation unit (Bronder ¶0037 discloses the super resolution model performing extensive computations) so that, with respect to at least a part of the plurality of tiles (Bronder ¶0015, ¶0016, ¶0049 discloses using convolution operations of the resolution model to the plurality of input tiles), overlapping pixels which are included in the at least the part of the plurality of tiles (Bronder Fig 3, Fig 7, 730, ¶0015, ¶0053, discloses overlap for multiple adjacent tiles in the same area).
Bronder does not explicitly disclose and which correspond to the same region in the image as another tile are excluded from a target of the convolutional computation processing.
Valpreda is in the same field of using CNN’s for image analysis in which the input image is tiled. Further, Valpreda teaches and which correspond to the same region in the image as another tile (Valpreda Section 4.1 and Fig 6 discloses the overlapping region of adjacent tiles) are excluded from a target of the convolutional computation processing (Valpreda, Section 4.2 and Fig 7 disclose only unique pixels are scheduled for computation to reduce the overlapping. To produce new blocks of unique pixels in intermediate layers, the tile set must be shaped accordingly to exclude the recomputation of pixels stored in reuse buffers).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Bronder by incorporating the exclusions of already processed overlapping image tiles using the CNN architecture as taught by Valpreda to make an invention that can automatically preserve edge integrity without the high computational cost of reprocessing multiple tiles; thus one of ordinary skilled in the art would be motivated to combine the references since there is a need for an improved data reuse model for zero recomputation when executing layers with data dependencies, leveraging multiple tile-sets to compute only non-redundant intermediate data. (Valpreda, Introduction).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Regarding Claim 2, Bronder in view of Valpreda teaches the image processing apparatus (Bronder ¶0035 and Fig 1 120 discloses a rendering engine that includes processes for altering the attributes of the images being processed)according to claim 1, wherein
the control unit (Bronder ¶0012 discloses dynamically controlling the tiling of images for processing by super-resolution models) excludes the overlapping pixels from the target of the convolutional computation processing (Valpreda, Section 4.2 and Fig 7 disclose only unique pixels are scheduled for computation to reduce the overlapping. To produce new blocks of unique pixels in intermediate layers, the tile set must be shaped accordingly to exclude the recomputation of pixels stored in reuse buffers) executed by the computation unit (Bronder ¶0037 discloses the super resolution model performing extensive computations) by using, as the input data, an image configured by excluding the overlapping pixels from the at least the part of the plurality of tiles (Vapreda Section 3.1 discloses using an internal control unit to store input and Section 4.2 and Fig 7 disclose only unique pixels are scheduled for computation to reduce the overlapping. To produce new blocks of unique pixels in intermediate layers, the tile set must be shaped accordingly to exclude the recomputation of pixels stored in reuse buffers). See Claim 1 for rationale, its parent claim.
Regarding Claim 3, Bronder in view of Valpreda teaches the image processing apparatus (Bronder ¶0035 and Fig 1 120 discloses a rendering engine that includes processes for altering the attributes of the images being processed) according to claim 1, wherein
the plurality of tiles are a predetermined number of partial regions set in the image (Bronder Fig 7, 730, ¶0015, ¶0018 discloses splitting the input images into a plurality of input tiles, where the input tiles have a tile size and halo region, the halo region being the edge region of the tiles which can change based on the halo region size, therefore the predetermined partial regions are two regions), and are each an image corresponding to one of the predetermined number of partial regions (Bronder Fig 7, 730, ¶0015, ¶0018 discloses splitting the input images into a plurality of input tiles with each tile being its own image as shown in Fig 4 , where the input tiles have a tile size and halo region, the halo region being the edge region of the tiles which can change based on the halo region size, therefore the predetermined partial regions are two regions) that have been set to overlap at least another partial region (Valpreda Section 4.1 and Fig 6 discloses the overlapping region of adjacent tiles), and
the overlapping pixels (Bronder Fig 3, Fig 7, 730, ¶0015, ¶0053, discloses overlap for multiple adjacent tiles in the same area) correspond to a region in which, among the predetermined number of partial regions (Bronder Fig 7, 730, ¶0015, ¶0018 discloses splitting the input images into a plurality of input tiles, where the input tiles have a tile size and halo region, the halo region being the edge region of the tiles which can change based on the halo region size, therefore the predetermined partial regions are two regions), partial regions that have been set to neighbor each other overlap (Bronder Fig 7 730, ¶0008, ¶0015, and discloses overlapping adjacent tiles with the tile size and halo region being predetermined). See Claim 1 for rationale, its parent claim.
Regarding Claim 4, Bronder in view of Valpreda teaches the image processing apparatus (Bronder ¶0035 and Fig 1 120 discloses a rendering engine that includes processes for altering the attributes of the images being processed) according to claim 1, wherein
the control unit controls the computation unit (Bronder ¶0012 discloses dynamically controlling the tiling of images for processing by super-resolution models) so as to cause the computation unit to execute the convolutional computation processing (Bronder ¶0015, ¶0016, ¶0049 discloses using convolution operations of the resolution model to the plurality of input tiles) while selecting each of the plurality of tiles as the input data in order, (Bronder ¶0053 discloses dynamically selecting the halo region tiles on the border for processing based on their position of order on the edge), and
exclude the overlapping pixels included in the at least the part of the plurality of tiles from the target of the convolutional computation processing (Valpreda, Section 4.2 and Fig 7 disclose only unique pixels are scheduled for computation to reduce the overlapping. To produce new blocks of unique pixels in intermediate layers, the tile set must be shaped accordingly to exclude the recomputation of pixels stored in reuse buffers) on a condition that the convolutional computation processing has been completed for the another tile (Valpreda Fig 7 and Section 4.1 and Section 4.2 discloses that to avoid recomputation of a tile that has already been processed that tile value is stored so that the scheduling skips that tile based on the fact that it has already been computed). See Claim 1 for rationale, its parent claim.
Regarding Claim 5, Bronder in view of Valpreda teaches the image processing apparatus (Bronder ¶0035 and Fig 1 120 discloses a rendering engine that includes processes for altering the attributes of the images being processed) according to claim 4, wherein
the control unit selects, as the input data to be used next (Bronder ¶0012 discloses dynamically controlling the tiling of images for processing by super-resolution models, a tile corresponding to a partial region that neighbors a partial region (Valpreda Section 4.1 and Fig 6 discloses the overlapping region of adjacent tiles) corresponding to a tile that has been selected as the input data most recently in a vertical direction in the image (Valpreda Fig 6 and Section 4.1 disclose that the tiles are processed in vertical order for example from time 3 to 4 and discloses the equation for when a processing direction is vertical). See Claim 1 for rationale, its parent claim.
Regarding Claim 6, Bronder in view of Valpreda teaches the image processing apparatus (Bronder ¶0035 and Fig 1 120 discloses a rendering engine that includes processes for altering the attributes of the images being processed) according to claim 1, wherein
the computer program further causes the at least one processor and/or circuit to function (Bronder ¶0083, ¶0085, discloses a computer system comprising a computer readable media such as computer memory. In particular, the computer memory may store computer-executable instructions that when executed by one or more processors cause various functions to be performed) as an output unit configured to generate output data of each of the plurality of tiles (Bronder Fig 7, 720-750 discloses output tiles based on the input tiles and using the super resolution model) a result of the convolutional computation processing (Bronder ¶0015, ¶0016, ¶0049 discloses using convolution operations of the resolution model to the plurality of input tiles) executed by the computation unit (Bronder ¶0037 discloses the super resolution model performing extensive computations), and
with respect to the at least the part of the plurality of tiles (Bronder Fig 3, Fig 7, 730, ¶0015, ¶0053, discloses overlap for multiple adjacent tiles in the same area), the output unit generates the output data (Bronder Fig 7, 720-750 discloses output tiles based on the input tiles and using the super resolution model) based on a result of the convolutional computation processing executed (Bronder ¶0015, ¶0016, ¶0049 discloses using convolution operations of the resolution model to the plurality of input tiles) for the at least the part of the plurality of tiles (Bronder Fig 3, Fig 7, 730, ¶0015, ¶0053, discloses overlap for multiple adjacent tiles in the same area), and on a result of the convolutional computation processing executed for the another tile (Valpreda Section 4.1 and Fig 6 discloses the overlapping region of adjacent tiles and Fig 7 discloses the processing order for the tiles which shows that the processing is dependent on the tile before the current processing tile) with respect to the overlapping pixels (Bronder Fig 3, Fig 7, 730, ¶0015, ¶0053, discloses overlap for multiple adjacent tiles in the same area). See Claim 1 for rationale, its parent claim.
Regarding Claim 7, Bronder in view of Valpreda teaches the image processing apparatus (Bronder ¶0035 and Fig 1 120 discloses a rendering engine that includes processes for altering the attributes of the images being processed) according to claim 6, wherein
the neural network is a convolutional neural network (Valpreda Introduction and Fig 11 discloses the use of convolutional neural networks and the CNN model) that includes a plurality of levels as convolutional layers (Valpreda Section 2.1 discloses the CNN having multiple convolutional layers), and repeatedly executes the convolutional computation processing in the plurality of levels with respect to the image (Valpreda Section 2.1 and Section 3 discloses that the convolutional layers are executed in loop techniques),
the control unit (Bronder ¶0012 discloses dynamically controlling the tiling of images for processing by super-resolution models) causes the computation unit to execute the convolutional computation processing (Bronder ¶0037 discloses the super resolution model performing extensive computations) with respect to each of the plurality of tiles (Bronder ¶0015, ¶0016, ¶0049 discloses using convolution operations of the resolution model to the plurality of input tiles), in each of the levels of the convolutional neural network (Valpreda Section 4.4 and Fig 11 discloses the convolutional processing for each layer of the CNN),
with respect to an input layer of the convolutional neural network (Valpreda Section 3.4 discloses the input layer), the obtainment unit obtains images (Bronder ¶0006, ¶0013 discloses super resolution processing systems configured to subdivide input images into a plurality of smaller tiles) included in the plurality of partial regions set in the image as the plurality of tiles (Bronder Fig 7, 730, ¶0015, ¶0018 discloses splitting the input images into a plurality of input tiles, where the input tiles have a tile size and halo region, the halo region being the edge region of the tiles which can change based on the halo region size), and with
respect to a succeeding layer of the convolutional neural network (Valpreda Fig 5 and Section 3.4 discloses that each output of the previous layer is then used as an input for the next layer), the obtainment unit (Bronder ¶0006, ¶0013 discloses super resolution processing systems configured to subdivide input images into a plurality of smaller tiles) obtains a plurality of pieces of the output data that have been generated by the output unit (Bronder Fig 7, 720-750 discloses output tiles based on the input tiles and using the super resolution model) with respect to a preceding level as the plurality of tiles for the succeeding layer (Valpreda Fig 5 and Section 3.4 discloses that each output of the previous layer is then used as an input for the next layer). See Claim 1 for rationale, its parent claim.
Regarding Claim 8, Bronder in view of Valpreda teaches the image processing apparatus (Bronder ¶0035 and Fig 1 120 discloses a rendering engine that includes processes for altering the attributes of the images being processed) according to claim 7, wherein
with respect to the at least the part of the plurality of tiles corresponding to the same partial region (Bronder Fig 7, 730, ¶0015, ¶0018 discloses splitting the input images into a plurality of input tiles, where the input tiles have a tile size and halo region, the halo region being the edge region of the tiles which can change based on the halo region size), the control unit (Bronder ¶0012 discloses dynamically controlling the tiling of images for processing by super-resolution models) differentiates pixels to be excluded as the overlapping pixels in accordance with a level of the convolutional neural network (Valpreda, Section 4.2 and Fig 7 disclose only unique pixels are scheduled for computation to reduce the overlapping. To produce new blocks of unique pixels in intermediate layers, the tile set must be shaped accordingly to exclude the recomputation of pixels stored in reuse buffers). See Claim 1 for rationale, its parent claim.
Regarding Claim 9, Bronder in view of Valpreda teaches the image processing apparatus (Bronder ¶0035 and Fig 1 120 discloses a rendering engine that includes processes for altering the attributes of the images being processed) according to claim 8, wherein
in each level of the convolutional neural network (Valpreda Section 4.4 and Fig 11 discloses the convolutional processing for each layer of the CNN), the computation unit executes the convolutional computation processing with respect to each of pixels included in the input data (Bronder ¶0047 discloses the convolutional operations performed by the super-resolution model on a pixel-by-pixel basis will utilize all of the original neighboring pixels within a predefined receptive field for each pixel being processed) with use of pixel values of pixels included in a predetermined region (Bronder ¶0067, ¶0040 discloses including pixel values to get an accurate representation of the halo) that has been determined based on the pixels included in the input data (Bronder ¶0047-¶0049, ¶0058 discloses the convolutional operations performed by the super-resolution model on a pixel-by-pixel basis will utilize all of the original neighboring pixels within a predefined receptive field for each pixel being processed),
a size of a reception field in the image that contributes to a result of the convolutional computation processing (Bronder ¶0047 discloses in a 3x3 convolutional operation, a set of nine input pixels (in a symmetric pattern of three rows and three columns) can be transformed into a single output pixel based on the pixel properties of the nine input pixels) for one pixel is determined in accordance with a level of the convolutional neural network in which the convolutional computation processing has been executed (Bronder ¶0047 discloses the convolutional operations performed by the super-resolution model on a pixel-by-pixel basis will utilize all of the original neighboring pixels within a predefined receptive field for each pixel being processed), and
the control unit (Bronder ¶0012 discloses dynamically controlling the tiling of images for processing by super-resolution models) determines pixels to be excluded as the overlapping pixels (Valpreda, Section 4.2 and Fig 7 disclose only unique pixels are scheduled for computation to reduce the overlapping. To produce new blocks of unique pixels in intermediate layers, the tile set must be shaped accordingly to exclude the recomputation of pixels stored in reuse buffers) based on the size of the reception field in the convolutional computation processing executed by the computation unit (Bronder ¶0015, ¶0016, ¶0049 discloses using convolution operations of the resolution model). See Claim 1 for rationale, its parent claim.
Regarding Claim 10, Bronder in view of Valpreda teaches the image processing apparatus (Bronder ¶0035 and Fig 1 120 discloses a rendering engine that includes processes for altering the attributes of the images being processed) according to claim 9, wherein
the size of the reception field becomes larger in a deeper level of the convolutional neural network (Bronder ¶0050 discloses pixel being processed will be based on the receptive field size that is established for processing the pixels ( e.g., a 5x5 receptive field with 2 pixels in each direction of the underlying pixel being processed). It will be appreciated that the size of the receptive field of view may vary to accommodate different needs, which the examiner is interpreting to being able to adapt to deeper levels of the convolution network), and
with respect to the at least the part of the plurality of tiles corresponding to the same partial region (Bronder Fig 3, Fig 7, 730, ¶0015, ¶0053, discloses overlap for multiple adjacent tiles in the same area), the control unit (Bronder ¶0012 discloses dynamically controlling the tiling of images for processing by super-resolution models) causes the number of pixels to be excluded as the overlapping pixels (Valpreda, Section 4.2 and Fig 7 disclose only unique pixels are scheduled for computation to reduce the overlapping. To produce new blocks of unique pixels in intermediate layers, the tile set must be shaped accordingly to exclude the recomputation of pixels stored in reuse buffers). to become smaller in a deeper level of the convolutional neural network (Valpreda Section 5.2 discloses tile sets used to block the processing of pixels around the center of the feature map spatial dimensions are inherently smaller than the others, meaning less pixels are removed). See Claim 1 for rationale, its parent claim.
Regarding Claim 11, Bronder in view of Valpreda teaches the image processing apparatus (Bronder ¶0035 and Fig 1 120 discloses a rendering engine that includes processes for altering the attributes of the images being processed) according to claim 9, wherein
in the convolutional computation processing (Bronder ¶0037 discloses the super resolution model performing extensive computations) for the another tile (Valpreda Section 4.1 and Fig 6 discloses the overlapping region of adjacent tiles), the control unit (Bronder ¶0012 discloses dynamically controlling the tiling of images for processing by super-resolution models) does not exclude a pixel corresponding to the reception field including a pixel that is not included (Bronder ¶0017 discloses the halo region are discarded after the image processing but are used during the image processing to provide a retained interior tile output based on a more thoroughly populated receptive field) in the another tile as the overlapping pixels (Valpreda Section 4.1 and Fig 6 discloses the overlapping region of adjacent tiles). See Claim 1 for rationale, its parent claim.
Regarding Claim 12, Bronder in view of Valpreda teaches the image processing apparatus (Bronder ¶0035 and Fig 1 120 discloses a rendering engine that includes processes for altering the attributes of the images being processed)according to claim 1, wherein
the computer program further causes the at least one processor and/or circuit to function (Bronder ¶0083, ¶0085, discloses a computer system comprising a computer readable media such as computer memory. In particular, the computer memory may store computer-executable instructions that when executed by one or more processors cause various functions to be performed) as a generation unit configured to generate the plurality of tiles based on the image (Bronder ¶0013, ¶0015, ¶0018 discloses a system for dynamically splitting input images into a plurality of input tiles for processing), and
the generation unit generates the plurality of tiles (Bronder ¶0013, ¶0015, ¶0018 discloses a system for dynamically splitting input images into a plurality of input tiles for processing) by setting partial regions so as not to segmentalize the image in a horizontal direction (Bronder Fig 7, 730, and ¶0013, ¶0015, ¶0029 discloses that the images a split based on resolution not based on horizontal direction). See Claim 1 for rationale, its parent claim.
Regarding Claim 14, Bronder teaches a control method (Bronder Fig 1 and ¶0022, ¶0032, ¶0035 and discloses an image processing flow) for an image processing apparatus (Bronder ¶0035 and Fig 1 120 discloses a rendering engine that includes processes for altering the attributes of the images being processed), the control method comprising:
executing convolutional computation processing (Bronder ¶0037 discloses the super resolution model performing extensive computations) in a neural network with respect to input data (Bronder ¶0015, ¶0016, ¶0049 discloses using convolution operations of the resolution model to the plurality of input tiles);
obtaining a plurality of tiles (Bronder ¶0006, ¶0013 discloses super resolution processing systems configured to subdivide input images into a plurality of smaller tiles) that respectively correspond to partial regions in an image (Bronder Fig 7, 730, ¶0015, ¶0018 discloses splitting the input images into a plurality of input tiles, where the input tiles have a tile size and halo region, the halo region being the edge region of the tiles which can change based on the halo region size); and
performing control so as to cause (Bronder ¶0012 discloses dynamically controlling the tiling of images for processing by super-resolution models) the convolutional computation processing (Bronder ¶0037 discloses the super resolution model performing extensive computations) to be executed while using each of the plurality of tiles as the input data (Bronder ¶0015, ¶0016, ¶0049 discloses using convolution operations of the resolution model to the plurality of input tiles),
wherein the control is performed (Bronder ¶0012 discloses dynamically controlling the tiling of images for processing by super-resolution models) so that, with respect to at least a part of the plurality of tiles (Bronder Fig 7, 730, ¶0015, ¶0018 discloses splitting the input images into a plurality of input tiles, where the input tiles have a tile size and halo region, the halo region being the edge region of the tiles which can change based on the halo region size), overlapping pixels which are included in the at least the part of the plurality of tiles (Bronder Fig 3, Fig 7, 730, ¶0015, ¶0053, discloses overlap for multiple adjacent tiles in the same area).
Bronder does not explicitly disclose and which correspond to the same region in the image as another tile are excluded from a target of the convolutional computation processing.
Valpreda is in the same field of using CNN’s for image analysis in which the input image is tiled. Further, Valpreda teaches and which correspond to the same region in the image as another tile (Valpreda Section 4.1 and Fig 6 discloses the overlapping region of adjacent tiles) are excluded from a target of the convolutional computation processing (Valpreda, Section 4.2 and Fig 7 disclose only unique pixels are scheduled for computation to reduce the overlapping. To produce new blocks of unique pixels in intermediate layers, the tile set must be shaped accordingly to exclude the recomputation of pixels stored in reuse buffers).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Bronder by incorporating the exclusions of already processed overlapping image tiles using the CNN architecture as taught by Valpreda to make an invention that can automatically preserve edge integrity without the high computational cost of reprocessing multiple tiles; thus one of ordinary skilled in the art would be motivated to combine the references since there is a need for an improved data reuse model for zero recomputation when executing layers with data dependencies, leveraging multiple tile-sets to compute only non-redundant intermediate data. (Valpreda, Introduction).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Regarding Claim 15, Bronder teaches a computer-readable recording medium storing a program for causing a computer to execute (Bronder ¶0083, ¶0085, discloses a computer system comprising a computer readable media such as computer memory. In particular, the computer memory may store computer-executable instructions that when executed by one or more processors cause various functions to be performed) the control method (Bronder Fig 1 and ¶0022, ¶0032, ¶0035 and discloses an image processing flow).
Bronder does not explicitly disclose according to claim 14.
Valpreda is in the same field of using CNN’s for image analysis in which the input image is tiled. Further, Valpreda teaches according to claim 14.
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Bronder by incorporating the exclusions of already processed overlapping image tiles using the CNN architecture as taught by Valpreda to make an invention that can automatically preserve edge integrity without the high computational cost of reprocessing multiple tiles; thus one of ordinary skilled in the art would be motivated to combine the references since there is a need for an improved data reuse model for zero recomputation when executing layers with data dependencies, leveraging multiple tile-sets to compute only non-redundant intermediate data. (Valpreda, Introduction).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Claim 13 are rejected under 35 U.S.C. 103 as unpatentable over Bronder in view of Valpreda in further view of Colleman, Steven, and Marian Verhelst. "High-utilization, high-flexibility depth-first CNN coprocessor for image pixel processing on FPGA." IEEE Transactions on Very Large Scale Integration (VLSI) Systems 29.3 (2021): 461-471, hereafter referred to as Colleman.
Regarding Claim 13, Bronder teaches the image processing apparatus (Bronder ¶0035 and Fig 1 120 discloses a rendering engine that includes processes for altering the attributes of the images being processed), wherein
the obtainment unit (Bronder ¶0006, ¶0013 discloses super resolution processing systems configured to subdivide input images into a plurality of smaller tiles) obtains the plurality of tiles based on the image that has been obtained (Bronder Fig 7, 730, ¶0015, ¶0018 discloses splitting the input images into a plurality of input tiles, where the input tiles have a tile size and halo region, the halo region being the edge region of the tiles which can change based on the halo region size)
Bronder does not explicitly disclose according to claim 1.
Valpreda is in the same field of using CNN’s for image analysis in which the input image is tiled. Further, Valpreda teaches according to claim 1.
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Bronder by incorporating the exclusions of already processed overlapping image tiles using the CNN architecture as taught by Valpreda to make an invention that can automatically preserve edge integrity without the high computational cost of reprocessing multiple tiles; thus one of ordinary skilled in the art would be motivated to combine the references since there is a need for an improved data reuse model for zero recomputation when executing layers with data dependencies, leveraging multiple tile-sets to compute only non-redundant intermediate data. (Valpreda, Introduction).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention.
Bronder and Valpreda in combination do not explicitly disclose an image capturing apparatus, an image capturing unit, and through image capture performed by the image capturing unit.
Colleman is in the same field of using CNN’s for image analysis in which the input image is tiled. Further, Colleman teaches an image capturing apparatus (Colleman, Introduction, discloses embedding the method onto a camera), an image capturing unit (Colleman, Introduction, discloses embedding the method onto a camera), and through image capture performed by the image capturing unit (Colleman, Introduction, discloses embedding the method onto a camera).
Therefore, it would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to modify the invention of Bronder in view of Valpreda by incorporating the implementation of the method with an image capturing device as taught by Colleman to make an invention that can automatically capture images and preserve edge integrity without the high computational cost of reprocessing multiple tiles; thus one of ordinary skilled in the art would be motivated to combine the references since there is a need to enable these algorithms at low cost into systems that already embed an FPGA, such as a LiDAR or other camera systems. (Colleman, Introduction).
Thus, the claimed subject matter would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention
Reference Cited
The prior art made of record and not relied upon is considered pertinent to applicant’s disclosure.
Gondimalla, Ashish, et al. "Occam: Optimal data reuse for convolutional neural networks." ACM Transactions on Architecture and Code Optimization 20.1 (2022): 1-25 discloses a method for optimal reuse of data in CNN’s for faster processing speeds.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to RACHEL ROBERTS whose telephone number is (571)272-6413. The examiner can normally be reached Monday- Friday 7:30am- 5:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Oneal Mistry can be reached on (313) 446-4912. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/RACHEL L ROBERTS/Examiner, Art Unit 2674
/ONEAL R MISTRY/Supervisory Patent Examiner, Art Unit 2674
Prosecution Insights