DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Specification
The lengthy specification has not been checked to the extent necessary to determine the presence of all possible minor errors. Applicant’s cooperation is requested in correcting any errors of which applicant may become aware in the specification.
Drawings
The applicant’s submitted drawings appear to be acceptable for examination purposes. Applicant’s cooperation is requested in correcting any errors of which applicant may become aware in the drawings.
Information Disclosure Statement
As required by M.P.E.P. 609(c), the applicant's submission of the Information Disclosure Statements, dated 20 May 2025, 10 July 2025, and 22 September 2025, are acknowledged by the examiner and the cited references have been considered in the examination of the claims now pending. As required by M.P.E.P 609 C(2), a copy of the PTOL-1449 forms, initialed and dated by the examiner, are attached to the instant office action.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1, 2, 5-9, 11, 13, 14, and 17-20 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Majurski et al. (Exact Tile-Based Segmentation Inference for Images Larger than GPU Memory, June 2021, pgs. 1-16 – cited in an IDS).
As per claim 1, Majurski teaches a method performed by one or more computers [the system/method is implemented by at least one GPU from data stored in GPU memory (abstract, etc.)], the method comprising:
receiving a new input to be processed by a fully convolutional neural network deployed on a hardware accelerator, the new input having a first size that is different from a fixed size that the fully convolutional neural network is configured to process when deployed on the hardware accelerator [a fully convolutional neural network (FCN) receives input images of arbitrarily large size and selects a tile size that will fit into GPU (hardware accelerator) memory and based on the receptive field size (abstract, etc.); where the image size is the first size that is different than the fixed tile size];
determining, one or more fixed-size inputs from the new input, each fixed-size input having the fixed size [a (fixed) tile size is selected and then stride across the input image by the tile size (without a halo) to produce input tiles (abstract, etc.)];
providing each of the one or more fixed-size inputs to the hardware accelerator for performing inference computations using the fully convolutional neural network [a fully convolutional neural network (FCN) receives input images of arbitrarily large size and selects a tile size that will fit into GPU (hardware accelerator) memory and based on the receptive field size, then stride across the input image by the tile size (without a halo) to produce input tiles (abstract, etc.)];
obtaining, from the hardware accelerator, a respective fixed-size output generated by the fully convolutional neural network for each of the one or more fixed-size inputs [the FCN on the GPU produces output tiles based on the selected input tile size and halo (abstract; pg. 5, paragraphs 1-3; etc.); which are fixed-size (see above)], wherein the respective fixed-size outputs comprise one or more inaccurate pixel-wise results [the zone of responsibility (ZoR) needs to be cropped out from the prediction output so only those pixel with full context are used to build the final result (pg. 5, paragraphs 1-3; etc.); where the remaining cropped/halo portion comprises the inaccurate pixel-wise results]; and
generating, from the respective fixed-size outputs, a final output that is equivalent to an output that would be generated by processing the new input using the fully convolutional neural network [after passing the input tile through the model, the ZoR within the prediction (without the halo) is copied to the output image being constructed in central processing unit (CPU) memory (pg. 5, paragraphs 1-3; etc.) which enables the output to be the same as if the FCN processed the entire input image (abstract, etc.)].
As per claim 2, Majurski teaches further comprising:
prior to deploying the fully convolutional neural network on the hardware accelerator, determining the fixed size based at least on characteristics of the fully convolutional neural network [the (fixed) tile and halo sizes are based upon the receptive field size (abstract, etc.), which is based on characteristics of the FCN (pg. 2, paragraph 4; pg. 3, paragraphs 5-7; etc.)].
As per claim 5, Majurski teaches wherein determining the fixed size further comprises:
determining that the first size of the new input is smaller than the fixed size; and
generating a fixed-size input by padding zeros around the new input up to the fixed size [the tiling scheme utilizes SAME type convolutions that include zero padding applied to each feature map within each convolutional layer so that the input remains the same fixed size (pg. 14, Appendix D; etc.)].
As per claim 6, Majurski teaches wherein each of the respective fixed-size outputs comprises a central valid region, and a peripheral dummy region at a width of a first number of pixels, wherein the central valid region includes at least a portion of the final output, wherein the peripheral dummy region comprises one or more inaccurate pixel-wise results [the zone of responsibility (ZoR) needs to be cropped out from the prediction output so only those pixel with full context are used to build the final result (pg. 5, paragraphs 1-3; etc.); where the ZoR is the central valid region and the cropped/halo portion is the peripheral dummy region comprising one or more inaccurate pixel-wise results].
As per claim 7, Majurski teaches wherein the first number of pixels is determined based on characteristics of the fully convolutional neural network [the (fixed) tile and halo sizes are based upon the receptive field size (abstract, etc.), which is based on characteristics of the FCN (pg. 2, paragraph 4; pg. 3, paragraphs 5-7; etc.)].
As per claim 8, Majurski teaches wherein generating the final output from the respective fixed-size outputs, comprises:
combining the central valid regions of the respective fixed-size outputs based on relations between coordinates of the respective fixed-size outputs and coordinates of respective corresponding fixed-size inputs used for generating the respective fixed-size outputs [the zone of responsibility (ZoR) needs to be cropped out from the prediction output so only those pixel with full context are used to build the final result by striding across the image at the size of the ZoR and combining the output ZoRs (pg. 2, paragraphs 5-7; pg. 5, paragraphs 1-3; pg. 7, section 3.2.1; etc.); where the output images are combined based upon the stride, which are respective coordinates of the inputs used to produce the outputs].
As per claim 9, Majurski teaches wherein combining the respective fixed-size outputs further comprises:
determining data representing a respective coordinate shift for each of the respective fixed-size outputs; and
combining the central valid regions of the respective fixed-size outputs based on the determined data [the zone of responsibility (ZoR) needs to be cropped out from the prediction output so only those pixel with full context are used to build the final result by striding across the image at the size of the ZoR and combining the output ZoRs (pg. 2, paragraphs 5-7; pg. 5, paragraphs 1-3; pg. 7, section 3.2.1; etc.); where the output image ZoRs are the central valid regions and are combined based upon the stride, which is the coordinate shift].
As per claim 11, Majurski teaches wherein determining data representing the respective coordinate shift comprises:
determining, based on characteristics of the fully convolutional neural network, overall alignment information; and
determining, based on the determined overall alignment information, the respective coordinate shift for each of the respective fixed-size outputs [the zone of responsibility (ZoR) needs to be cropped out from the prediction output so only those pixel with full context are used to build the final result by striding across the image at the size of the ZoR and combining the output ZoRs (pg. 2, paragraphs 5-7; pg. 5, paragraphs 1-3; pg. 7, section 3.2.1; etc.) where the size of the ZoR/halo/stride is based on the receptive field size (abstract, etc.), which is based on characteristics of the FCN (pg. 2, paragraph 4; pg. 3, paragraphs 5-7; etc.), where the determined halos and tile sizes are alignment information based on the FCN characteristics, and the shift determined from them is the coordinate shift].
As per claim 13, see the rejection of claim 1 above, wherein Majurski also teaches a system comprising one or more computers and one or more storage devices storing instructions that, when executed by the one or more computers, cause the one or more computers to perform [the method] [the system/method is implemented by at least one GPU from data stored in GPU memory (abstract, etc.) and store outputs in a CPU memory (pg. 5, paragraph 2; etc.); which includes performing instructions from memory, at least by the GPU].
As per claim 14, see the rejection of claim 2 above.
As per claim 17, see the rejection of claim 5 above.
As per claim 18, see the rejection of claim 6 above.
As per claim 19, see the rejection of claim 7 above.
As per claim 20, see the rejection of claim 1 above, wherein Majurski also teaches one or more non-transitory computer storage media storing instructions that, when executed by one or more computers, cause the one or more computers to perform [the method] [the system/method is implemented by at least one GPU from data stored in GPU memory (abstract, etc.) and store outputs in a CPU memory (pg. 5, paragraph 2; etc.); which includes performing instructions from memory, at least by the GPU].
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 3 and 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Majurski et al. (Exact Tile-Based Segmentation Inference for Images Larger than GPU Memory, June 2021, pgs. 1-16 – cited in an IDS) in view of well-known practices in the art.
As per claim 3, Majurski teaches the method of claim 2, as described above.
While Majurski teaches determining the fixed size based upon resource constraints and characteristics of the FCN (see above), it has not been relied upon for teaching wherein determining the fixed size further comprises: providing a plurality of candidate sizes to a user for selecting one candidate size of the plurality of candidate sizes as the fixed size.
However, the examiner takes official notice that providing a user a selection of available network and/or memory sizes and allowing the user to select desired configurations is old and well-known within the art, such as for implementation on user selected systems. Therefore, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to provide a plurality of candidate sizes to a user for selecting one candidate size of the plurality of candidate sizes as the fixed size, in the determination of the fixed size in the method/system taught by Majurski, to achieve the predictable result of allowing the user to select a desired system configuration and neural network based upon their own requirements and/or available hardware.
As per claim 15, see the rejection of claim 3 above.
Claim(s) 4 and 16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Majurski et al. (Exact Tile-Based Segmentation Inference for Images Larger than GPU Memory, June 2021, pgs. 1-16 – cited in an IDS) in view of Reda (US 2020/0160079 – cited in an IDS).
As per claim 4, Majurski teaches the method of claim 1, as described above.
While Majurski teaches determining the fixed size based upon resource constraints and characteristics of the FCN (see above), it has not been relied upon for teaching generating a plurality of candidate sizes for the fully convolutional neural network based on the characteristics of the fully convolutional neural network; for each of the candidate sizes: deploying a copy of the fully convolutional neural network on a respective hardware accelerator for processing inputs of the candidate size; measuring a total execution time of performing inference computations for the deployed copy of the fully convolutional neural network on the respective hardware accelerator; selecting, as the fixed size, a candidate size from the plurality of candidate sizes based at least upon the measured total execution times for the candidate sizes generating a plurality of candidate sizes for the fully convolutional neural network based on the characteristics of the fully convolutional neural network; for each of the candidate sizes: deploying a copy of the fully convolutional neural network on a respective hardware accelerator for processing inputs of the candidate size; measuring a total execution time of performing inference computations for the deployed copy of the fully convolutional neural network on the respective hardware accelerator; selecting, as the fixed size, a candidate size from the plurality of candidate sizes based at least upon the measured total execution times for the candidate sizes.
Reda teaches generating a plurality of candidate sizes for the fully convolutional neural network based on the characteristics of the fully convolutional neural network [we first create a large pool of candidate FCN models with varying computational costs (para. 0051, etc.) based upon differing input image scaling, numbers of layers, channels per layer, etc. (paras. 0054-61; etc.); where the fixed size is based upon the FCN models, as taught by Majurski above];
for each of the candidate sizes:
deploying a copy of the fully convolutional neural network on a respective hardware accelerator for processing inputs of the candidate size [the FCNs can be deployed on a hardware accelerator (para. 0008, etc.) or varying combinations of CPU/GPU/accelerator/FPGA (paras. 0033, 0074, etc.)];
measuring a total execution time of performing inference computations for the deployed copy of the fully convolutional neural network on the respective hardware accelerator [the execution times of performing inference are measured for different FCN models and different hardware, including a hardware accelerator (fig. 11; paras. 0070-71, 0090-91; etc.)];
selecting, as the fixed size, a candidate size from the plurality of candidate sizes based at least upon the measured total execution times for the candidate sizes [the most efficient set of models from the candidate FCNs are selected (paras. 0024, 0026, etc.) and sizes of tiling buffers are also selected based upon the candidate models selected (para. 0076, etc.); where the fixed size is based upon the FCN, so the fixed size candidate associated with the selected FCN is also selected (see Majurski, above)].
Majurski and Reda are analogous art, as they are within the same field of endeavor, namely improving FCN efficiency.
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to produce and compare a number of different FCN models and associated buffers, of different sizes and costs/runtimes, to select from among the candidate models/sizes, as taught by Reda, to select the FCN size and buffer size from which the tile size is produced in the system/method taught by Majurski.
Reda provides motivation as [producing and comparing different models and hardware setups/sizes provides improved efficiency and accuracy for the FCN (para. 0024, etc.)].
As per claim 16, see the rejection of claim 4 above.
Claim(s) 10 is/are rejected under 35 U.S.C. 103 as being unpatentable over Majurski et al. (Exact Tile-Based Segmentation Inference for Images Larger than GPU Memory, June 2021, pgs. 1-16 – cited in an IDS) in view of Minchenkov (US 2020/0349698).
As per claim 10, Majurski teaches the method of claim 9, as described above.
While Majurski also teaches removing the halo from the ZoR and merging the ZoRs of the outputs based upon the stride/coordinate shift (see above), it has not been relied upon for teaching wherein determining data representing the respective coordinate shift comprises: determining the respective coordinate shift using local search, wherein the local search comprises determining a relation between coordinates of a fixed-size output and coordinates of a corresponding fixed-size input used for generating the fixed-size output.
Minchenkov teaches wherein determining data representing the respective coordinate shift comprises:
determining the respective coordinate shift using local search, wherein the local search comprises determining a relation between coordinates of a fixed-size output and coordinates of a corresponding fixed-size input used for generating the fixed-size output [image registration is performed for blended images using local searches around points to match points of the two images (para. 0046, etc.), for aligning the coordinates of the outputs in Majurski, above].
Majurski and Minchenkov are analogous art, as they are within the same field of endeavor, namely using convolutional neural networks on images.
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to use local search to match points for aligning overlapping/blended images, as taught by Minchenkov, to align the outputs that include overlapping halo regions in the system/method taught by Majurski.
Minchenkov provides motivation as [performing registration on the images (and possibly 3D models) using local search allows the images to be integrated into a common reference frame (paras. 0045-46, etc.)].
Claim(s) 12 is/are rejected under 35 U.S.C. 103 as being unpatentable over Majurski et al. (Exact Tile-Based Segmentation Inference for Images Larger than GPU Memory, June 2021, pgs. 1-16 – cited in an IDS) in view of Wang (US 2021/0183022).
As per claim 12, Majurski teaches wherein the characteristics of the fully convolutional neural network comprises: a zero padding size, [and] stride size, for each network layer of the fully convolutional neural network [the size of the strode os based on the size of the ZoR and halo, which are based on the receptive field size (abstract, etc.), which is based on characteristics of the FCN (pg. 2, paragraph 4; pg. 3, paragraphs 5-7; etc.) and zero padding size required for each layer (pg. 14, Appendix D; etc.)].
While Majurski teaches utilizing some characteristics of the FCN (see above), it has not been relied upon for teaching wherein the characteristics of the fully convolutional neural network comprises: a respective filter size and scale factor.
Wang teaches wherein the characteristics of the fully convolutional neural network comprises: a respective filter size and scale factor [the convolutional neural network receptive field is based upon a scale (factor) (paras. 0052-56, 0067-68, etc.) and convolutional kernel size (filter size) (para. 0073, etc.)].
Majurski and Wang are analogous art, as they are within the same field of endeavor, namely utilizing a CNN based upon its receptive field size(s).
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to include the scale factor and kernel (filter) size in determining the appropriate receptive field, as taught by Wang, in determining the appropriate receptive field for determining the appropriate tile/ZoR/Halo sizes in the system/method taught by Majurski.
Wang provides motivation as [utilizing different receptive field sizes based upon different filter sizes and spatial resolutions different levels of information may be captured (para. 0073, etc.) to perform feature extraction on regions of the image(s) (abstract, etc.)]. Majurski also notes that the effective receptive field must be determined from multiple characteristics (see, e.g., Majurski: pgs. 2-3, section 2; pgs. 11-12, section 4.6; etc.).
Conclusion
The following is a summary of the treatment and status of all claims in the application as recommended by M.P.E.P. 707.07(i): claims 1-20 are rejected.
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Saito et al. (Multiple Object Extraction from Aerial Imagery with Convolutional Neural Networks, Dec 2015, pgs. 1-9) – cited by Majurski, above, and discloses predicting a center patch given a large local context, including adding a model averaging component by predicting eight slightly offset versions of the same patch and then combining the predictions.
Huang et al. (Tiling and stitching segmentation output for remote sensing: basic challenges and recommendations, Feb 2019, pgs. 1-9) – cited by Majurski, above, and discloses slicing an input image tile into patches of a selected size then stitching the outputs back together, using a sliding window approach, as well as the relation between patch sizes and execution/runtime.
Shi (US 10,489,913) – discloses an image segmentation method including trying multiple candidate regions of different sizes and unifying the candidates of different sizes to a fixed size.
Wang (US 2018/0150740 – cited in an IDS) – discloses a CNN system using multiple resolution limited CNNs processing overlapping subimages of a selected size that cover a larger input image, as well as merging the outputs back together with additional merging and CNN modules.
The examiner requests, in response to this Office action, that support be shown for language added to any original claims on amendment and any new claims. That is, indicate support for newly added claim language by specifically pointing to page(s) and line number(s) in the specification and/or drawing figure(s). This will assist the examiner in prosecuting the application.
When responding to this office action, Applicant is advised to clearly point out the patentable novelty which he or she thinks the claims present, in view of the state of the art disclosed by the references cited or the objections made. He or she must also show how the amendments avoid such references or objections. See 37 CFR 1.111(c).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to GEORGE GIROUX whose telephone number is (571)272-9769. The examiner can normally be reached M-F 10am-6pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Omar Fernandez Rivas can be reached at 571-272-2589. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/GEORGE GIROUX/Primary Examiner, Art Unit 2128
Prosecution Insights