DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Drawings
The drawings are objected to because:
Reference sign 215, identified in paragraph [0039] as the kernel in FIG. 2, does not appear in the drawings. FIG. 2 identifies the kernel using reference sign 105.
FIG. 8 is objected to because the smiley-face symbol adjacent to the legend “Proper Tiling” constitutes superfluous, nontechnical illustrative matter.
Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. The figure or figure number of an amended drawing should not be labeled as “amended.” If a drawing figure is to be canceled, the appropriate figure must be removed from the replacement sheet, and where necessary, the remaining figures must be renumbered and appropriate changes made to the brief description of the several views of the drawings for consistency. Additional replacement sheets may be necessary to show the renumbering of the remaining figures. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Specification
The disclosure is objected to because of the following informalities:
Paragraphs [0026] and [0027] reverse the operations shown in FIGS. 1 and 2. FIG. 1 depicts convolution without padding: a 5×5 image 110 and a 3×3 kernel 105 produce a reduced 3×3 feature map 115. FIG. 2 depicts zero-padding or boundary treatment: a zero border surrounds image 205 and output feature map 210 retains the original 5×5 dimensions. Paragraph [0026] must therefore describe FIG. 1 as convolution without padding, and paragraph [0027] must describe FIG. 2 as convolution with zero padding or boundary treatment.
Incorrect descriptions of FIGS. 5 and 6: paragraphs [0030], [0031], and [0092] reverse the operations shown in FIGS. 5 and 6. FIG. 5 expressly shows “Reduction in Output,” corresponding to convolution without boundary treatment. FIG. 6 shows a same-sized output having a central “Valid Output” surrounded by “Contaminated Output,” corresponding to convolution with boundary treatment. Applicant must correct: (i) Paragraph [0030] to describe FIG. 5 as convolution without boundary treatment resulting in reduced output size; (ii) Paragraph [0031] to describe FIG. 6 as convolution with boundary treatment resulting in a same-sized output containing contaminated boundary values; (iii) Paragraph [0092] so that the discussion of boundary treatment refers to FIG. 6 and the discussion of no boundary treatment refers to FIG. 5; (iv) the concluding “respectively” clause in paragraph [0092] so that reduction is associated with no padding and contamination is associated with padding.
Paragraph [0039] refers to kernel 215, whereas FIG. 2 identifies the kernel as 105, as separately addressed above.
Incorrect numerical example in paragraph [0054]: paragraph [0054] states that a 1000×1000-pixel image divided into 100×100-pixel patches comprises 10 patches, each containing 1000 pixels. A regular division produces 100 patches, each should containing 10,000 pixels. Applicant must correct or clarify the numerical example.
Paragraph [0056] identifies the fixation/ embedding system as 305. FIG. 3 identifies the fixation/ embedding system as 310; 305 identifies the image-generation system.
Paragraph [0060] identifies the tissue slicer as 310. FIG. 3 identifies the tissue slicer as 315.
Paragraph [0073] refers to “training dataset 345.” Reference numeral 345 identifies the training controller; the training dataset is 350.
Paragraph [0076] refers to “remote system 350”. Reference numeral 350 identifies the training dataset; the remote system is 335.
Paragraph [0090] refers to “contraction path 315”. FIG. 4 and paragraph [0089] identify the contracting path as 405.
Appropriate correction is required.
Claim Objections
Claims 1–2, 4, 6, 8–9, 11, 13, 15–16, 18, and 20 are objected to because of the following informalities:
Claims 1, 8, and 15: these claims recite “wherein the generating the tiling element comprises.” The superfluous first occurrence of “the” should be deleted so that the limitation reads “wherein generating the tiling element comprises”. These claims also recite “using at least convolutional layers, the kernel, and the down-sampling layers”. The omitted article before “convolutional layers” must be supplied or the phrase otherwise revised to express the intended grammatical relationship.
Claims 2, 9, and 16: these claims recite “the generating the tiling element further comprises”. The superfluous first occurrence of “the” should be deleted. These claims also recite “the number of down-sampling iterations, the size of the kernel, and number of convolutional layers”. For grammatical parallelism, “the” should be inserted before “number of convolutional layers”.
Claims 4, 11, and 18: these claims recite “for each portion of the image that the tiling element is aligned”. The phrase lacks the preposition necessary to express the stated relationship and should be corrected, for example, to “for each portion of the image with which the tiling element is aligned” or “for each portion of the image that the tiling element is aligned with”.
Claims 6, 13, and 20: these claims recite “wherein generating, for each tile, the convolved portion of the image comprise”. The singular gerund phrase requires “comprises”. Limitation (iv) further recites “summing the results of the element-wise multiplication are summed up to obtain a single value”. The duplicative predicate must be corrected, for example, to “summing the results of the element-wise multiplication to obtain a single value”. These claims further recite limitation (iii): “...corresponding values in a input region that the kernel covers...”. The indefinite article should properly read “an input region”.
Appropriate correction is required.
Claim Rejections - 35 USC § 112(b)
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1–20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claims 1, 8 and 15 recite the limitation "calculated from the number of down-sampling iterations and the size of the kernel" in claims. There is insufficient antecedent basis for this limitation in the claim. Claims 1, 8 and 15 first recite "determining a number of down-sampling layers to be implemented in a machine learning model," and then later recite that limitation. There is no antecedent basis for "down-sampling iterations" in the claim. It is unclear whether "down-sampling iterations" refers to the same feature as the previously recited "down-sampling layers," or to some additional, undefined concept. Because claims 2-7, 9-14 and 16-20 depend from claims 1, 8 and 15, they inherit this ambiguity, fail to cure the deficiency.
Claims 1, 8 and 15 further recite the limitation "generating, for each tile, a convolved portion of the image using at least convolutional layers, the kernel, and the down-sampling layers" in claims. There is insufficient antecedent basis for this limitation in the claim. The terms "the kernel" and "the down-sampling layers" have proper antecedent basis from earlier recitations of "a kernel" and "down-sampling layers." However, "convolutional layers" appears here for the first time in the claim without an introductory article (e.g., "one or more convolutional layers") and without any prior antecedent. It is unclear what structure, or step establishes these "convolutional layers," and whether they are part of the previously recited machine learning model or a separate, unclaimed element. Because claims 2-7, 9-14 and 16-20 depend from claims 1, 8 and 15, they inherit this ambiguity, fail to cure the deficiency.
Each of claims 3, 10, and 17 recites that “the non-overlapping core region is only used to generate each of the convolved portions of the image.” The placement of “only” permits materially different reasonable interpretations. The limitation may mean: (i) only pixels from the non-overlapping core region, and not pixels from the surrounding overlapping border region, participate in generating the convolved portion; (ii) the complete tile, including the overlapping border region that supplies convolutional context, is processed, but only output corresponding to the non-overlapping core region is retained to form the convolved portion; or (iii) the non-overlapping core region is used only for the stated purpose, without excluding use of the overlapping border region. The written description does not resolve the uncertainty. Paragraphs [0095] and [0103] describe substantially the same operation without clearly distinguishing between the tile regions processed as input and the output region retained to generate each convolved portion. Consequently, one of ordinary skill in the art cannot ascertain the metes and bounds of the claimed limitation. The rejection may be overcome by amending the claims to identify expressly what is processed and what is retained. If the second interpretation is intended, suitable language could recite, for example: “wherein the machine learning model processes each tile, including the overlapping border region, and only output corresponding to the non-overlapping core region is retained to generate each convolved portion of the image.”
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1–20 are rejected under 35 U.S.C. §102(a)(1) as being anticipated by NIST (Majurski, M., & Bajcsy, P. (2021). Exact Tile-Based Segmentation Inference for Images Larger than GPU Memory. Journal of Research of the National Institute of Standards and Technology, 126), as provided by Applicant's IDS filed on 12/27/2024.
Regarding claim 1, NIST teaches a computer-implemented method comprising:
accessing an image;
( Abstract, Sec. 1, Sec. 3-4 and Fig. 1: "Performing inference on arbitrarily large input images requires that we operate based on image tiles...". "We used a publicly accessible data set... roughly 20000 × 20000 pixels per stitched image frame...". NIST processes arbitrarily large images, including whole-slide microscopy images generated by a slide scanner and gigapixel microscopy images accessed for tile-based semantic-segmentation inference. )
generating a tiling element for the image, wherein the generating the tiling element comprises:
( [Abstract]; [Sec. 1]; [Sec. 3 and Fig. 2]; [Sec. 4.2]: a “tiling element” is defined as a spatial bounding template/ tile-configuration construct comprising a central non-overlapping core region surrounded by an overlapping border margin in Applicant’s specification [0042] & [0094]. NIST expressly defines and generates this exact composite tiling element (input tile template) configured as a defined spatial window comprising a central non-overlapping Zone of Responsibility (ZoR) (the claimed non-overlapping core region) surrounded by a Halo ghost region (the claimed overlapping border region), mathematically structured by the relationship
inputTileSize
=
ZoR
+
2
×
Halo
. Accordingly, NIST explicitly generates and establishes a
1024
×
1024
pixel tiling element template composed of an
832
×
832
pixel core ZoR and a surrounding
96
-pixel border Halo to decompose and extract matching tiles across the image. )
determining a number of down-sampling layers to be implemented in a machine learning model that is to be used for processing the image;
( Sec. 3.1.1, Sec. 3.1.4, Appendix B > Eq. 8: NIST determines the network level (lc) from the number of max-pool and up-convolution layers between the input and each convolutional layer. Each max-pool increases the level and halves the spatial resolution, and NIST expressly identifies the U-Net levels through level four. )
determining a size of a kernel to be applied during convolution operations in the machine learning model; and
( Sec. 3.1.1, Eq. 1, Appendix A > Eq. 6: NIST associates each convolutional layer with a square kernel (kc x kc), uses the kernel size (kc) in calculating the required Halo, and expressly uses a kernel size (k=3) for U-Net. )
generating a non-overlapping core region and an overlapping border region of the tiling element, wherein the overlapping border region surrounds the non-overlapping core region and a size of the non-overlapping core region is determined based on a dimension-reduction attribute N calculated from the number of down-sampling iterations and the size of the kernel;
( [Sec. 1], [Fig. 2 and Sec. 3], [Eq. 1 and Sec. 4.2]: NIST’s Zone of Responsibility (ZoR) corresponds to the non-overlapping core, and NIST’s surrounding Halo corresponds to the overlapping border. NIST defines (inputTileSize = ZoR + 2 x Halo), calculates Halo from the architecture level (lc), reflecting the down-sampling operations, and kernel size (kc), and therefore determines (ZoR = inputTileSize - Nclaim), where (Nclaim = 2 x Halo). NIST exemplifies this relationship as ( 832 = 1024 - (2 x 96) ). )
extracting tiles from the image using the tiling element, wherein each tile has the same non-overlapping core region and the same overlapping border region as the tiling element;
( [Sec. 1], [Sec. 3 and Fig. 2], [Sec. 4.2]: NIST’s predefined 1024 × 1024-pixel input-tile template comprising an 832 × 832-pixel non-overlapping ZoR core surrounded by a 96-pixel-wide Halo border. NIST positions this tiling element at successive locations across the source image using a fixed 832-pixel stride and extracts the image data encompassed by the tiling element at each full-tile location. Accordingly, each extracted full input tile has the same 832 × 832-pixel non-overlapping core region and the same 96-pixel-wide overlapping border region defined by the tiling element. )
inputting each tile into the machine learning model;
( [Abstract], [Sec. 3], [Sec. 4.2]: NIST passes each Halo-containing input tile through the fully convolutional U-Net in a separate GPU forward pass. )
generating, for each tile, a convolved portion of the image using at least convolutional layers, the kernel, and the down-sampling layers;
( [Sec. 1], [Secs. 3.1.1–3.1.4], [Sec. 3.2]: NIST’s U-Net includes convolutional layers having kernels and max-pool down-sampling layers. Each tile forward pass constructs a spatial subregion of the feature maps and produces the corresponding ZoR prediction. )
generating a convolved version of the image using the convolved portions of the image; and
( [Fig. 2], [Sec. 3], [Sec. 4.3]: NIST extracts the ZoR from each tile prediction and copies the separately generated ZoR portions into the output image in CPU memory, thereby constructing the network output as if the entire image had been inferred in one pass. )
outputting the convolved version of the image.
( [Fig. 1], [Sec. 3], [Sec. 4.3]: NIST produces and saves the assembled segmentation output, including output softmax values and the resulting segmentation mask. )
Regarding claim 2, NIST teaches the computer-implemented method of claim 1, wherein:
the machine learning model is configured to include the convolutional layers at each of one or more resolutions;
( [Sec. 3.1.1], [Sec. 3.1.4], [Appendix B > Eqs. 7–8]: NIST associates each convolutional layer with a network level (lc) and expressly identifies convolutional layers at successive levels. Equation 8 enumerates the convolutional-layer levels as ({0,0,1,1,2,2,3,3,4,4,3,3,2,2,1,1,0,0}), thereby disclosing convolutional layers operating at the different spatial resolutions represented by those levels. )
each of the down-sampling iterations leads to a different resolution of an image version being assessed;
( [Sec. 3.1.1], [Sec. 3.1.4], [Appendix D]: NIST states that each max-pool increases the level, that the first max-pool halves the spatial resolution, and that this process continues through level four. NIST further illustrates successive max-pooling resolutions of 500, 250, 125, 62, and 31 pixels. )
the generating the tiling element further comprises determining, for each level or each resolution, a number of convolutional layers being implemented by the machine learning model; and
( [Sec. 3.1.1], [Appendix A > Eq. 6], [Appendix B > Eqs. 7–9]: As part of generating the input-tile configuration corresponding to the claimed tiling element, NIST determines the number of convolutional layers implemented at each network level or resolution for use in calculating the required Halo. NIST defines nₗ as the number of convolutional layers at a level on each encoder or decoder side and determines that U-Net has nₗ = n = 2 convolutional layers at each such stage. )
the dimension-reduction attribute N is calculated from the number of down-sampling iterations, the size of the kernel, and number of convolutional layers.
( [Eq. 1], [Sec. 3.1.4], [Appendix A > Eq. 6]: NIST calculates "Halo"
=
∑
c
=
0
N
-
1
2
l
c
k
c
2
"; where (lc) represents the down-sampling level, (kc) is the convolution-kernel size, and the summation is performed over the convolutional layers. For a U-Net having a constant kernel and (n) convolutional layers per level, NIST simplifies the calculation to "Halo"
=
k
2
∑
m
=
0
M
-
1
2
m
n
+
2
M
n
=
k
2
n
3
⋅
2
M
-
2
, where (M) is the maximum down-sampling level. The claimed dimension-reduction attribute is (Nclaim = 2 x Halo). For (k=3), (n=2), and (M=4), NIST obtains a 92-pixel Halo and expressly identifies the corresponding total dimensional reduction as (2 x 92=184) pixels, reducing a 572-pixel input to a 388-pixel output. )
Regarding claim 3, NIST teaches the computer-implemented method of claim 2, wherein the convolutional layers without padding or border treatment trim down each tile such that the non-overlapping core region is only used to generate each of the convolved portions of the image.
( [Sec. 3.1], [Secs. 3.1.3–3.1.4], [Appendix D]: NIST states that its tiling methodology applies to fully convolutional networks and expressly discusses the original U-Net employing VALID convolutions. NIST explains that VALID convolutions use no zero padding and shrink the spatial size of each feature map by two pixels per convolutional layer. NIST further relates that shrinkage to its Halo/core geometry: a 92-pixel Halo on each side produces a total 184-pixel reduction, such that a 572-pixel input tile is reduced to a 388-pixel retained central output. NIST teaches that this full-context central ZoR is the non-overlapping output portion used to construct the final image. )
Regarding claim 4, NIST teaches the computer-implemented method of claim 1, wherein extracting the tiles from the image using the tiling element comprises:
sliding the tiling element through the image until the tiling element traverses the entire image; and
( [Abstract], [Sec. 1], [Sec. 3]: NIST states that the method strides across the source image by the tile size without the Halo, defines the stride as fixed at the ZoR size, and explains that the whole image is broken into non-overlapping regions by striding across the image with a fixed pixel stride. )
for each portion of the image that the tiling element is aligned as it slides through the image, a corresponding tile is extracted from the portion of the image, wherein each tile will have the same dimensions as the tiling element including the overlapping border region and the non-overlapping core region.
( [Sec. 1], [Sec. 3 and Fig. 2], [Sec. 4.2]: NIST defines each standard input tile as a ZoR together with a surrounding Halo according to (inputTileSize = ZoR + 2 x Halo). Figure 2 illustrates the repeated fixed geometry of a 1024-by-1024 input tile comprising an 832-by-832 ZoR and a 96-pixel surrounding Halo, and NIST uses the fixed ZoR dimension as the stride for extracting successive full tiles. )
Regarding claim 5, NIST teaches the computer-implemented method of claim 4, wherein as the tiling element slides through the image, the non-overlapping core regions of tiles being extracted from neighboring portions of the image are adjacent to one another but do not overlap with one another and the overlapping border regions of tiles being extracted from neighboring portions of the image do overlap with one another.
( [Abstract], [Sec. 1], [Sec. 3, Fig. 2]: NIST expressly teaches that input-tile Halos overlap while the output tiles join at their seams. NIST defines the image decomposition as non-overlapping ZoRs with the stride fixed at the ZoR size. Accordingly, neighboring ZoRs are adjacent and non-overlapping, while the Halos surrounding those neighboring ZoRs overlap. )
Regarding claim 6, NIST teaches the computer-implemented method of claim 1, wherein generating, for each tile, the convolved portion of the image comprise:
(i) sliding the kernel over the tile one step at a time;
( [Sec. 3.1.1], [Secs. 3.2.1–3.2.2], [Fig. 3]: NIST teaches conventional spatial convolution using a square kernel, discusses the kernel’s stride relative to its feature map, and illustrates a size-three kernel aligned at spatially offset positions. NIST’s disclosed stride-one U-Net convolution necessarily applies the kernel at successive spatial positions. )
(ii) for each step, a center of the kernel is placed at a specific position on the tile;
( [Sec. 3.1.1], [Sec. 3.2.1], [Fig. 3]: NIST associates each convolutional layer with an odd square kernel, including a 3-by-3 kernel, and illustrates that kernel aligned with corresponding input regions. An odd 3-by-3 kernel necessarily has a center element aligned with the corresponding spatial output position. )
(iii) during the sliding process, values of the kernel are element-wise multiplied with corresponding values in a input region that the kernel covers on the tile to obtain results;
( [Sec. 1], [Sec. 3.1.1], [Fig. 3]: NIST discloses conventional CNN convolutional layers applying kernels to corresponding image or feature-map regions. Under the ordinary mathematical operation necessarily performed by the disclosed convolutional layer, the aligned kernel coefficients are element-wise multiplied by the corresponding input-region values. )
(iv) summing the results of the element-wise multiplication are summed up to obtain a single value, wherein the single value represents output of the kernel at the specific position; and
( [Sec. 3.1.1], [Sec. 3.2.1], [Fig. 3]: NIST’s Figure 3 expressly illustrates a size-three unit-weight kernel producing individual output values by summing the covered input values, including (1+2+3=6) and (2+3+4=9). More generally, the disclosed convolution necessarily sums the aligned kernel-input products to form the kernel output at the corresponding position. )
(v) repeating (i)-(iv) for every possible position where the kernel can fit on the tile, wherein each convolutional layer outputs a new feature map, and wherein each element of the new feature map represents the output of applying the kernel to the specific position on the tile.
( [Sec. 1], [Sec. 3], [Secs. 3.1.1–3.1.4]: NIST describes a fully convolutional U-Net having successive convolutional layers that generate feature maps and explains that each tile-based forward pass computes a spatial subregion of those feature maps. Conventional stride-one convolution necessarily repeats the kernel operation at the available spatial positions, with the result at each position forming the corresponding element of the output feature map. )
Regarding claim 7, NIST teaches the computer-implemented method of claim 6, wherein no padding or border treatment is applied to each tile.
( [Sec. 3.1], [Secs. 3.1.3–3.1.4], [Appendix D]: NIST states that its tiling methodology applies to fully convolutional networks and expressly identifies original U-Net’s VALID convolutions as the alternative to SAME convolution. NIST explains that VALID convolution uses no zero padding, shrinks the feature maps by two pixels per convolutional layer, and reduces the 572-pixel input to the 388-pixel retained output. Therefore, NIST expressly discloses the no-convolution-padding VALID implementation. )
Regarding claims 8–20, the rationales provided in the rejection of claims 1–7 are incorporated herein. Further, the computer-implemented method of claims 1–7 correspond to the system of claims 8–14, as well as the computer-program product tangibly embodied in a non-transitory machine-readable storage medium of claims 15–20 ( [Sec. 3], [Sec. 4.3], [Sec. 6]: NIST discloses performing the tile-based inference process through a series of computer-executed forward passes using an Intel Xeon 4114 CPU and an NVIDIA Titan RTX GPU running Python 3.6 and TensorFlow 2.1.0. NIST further discloses the corresponding TensorFlow source-code implementation and the use of CPU and GPU memory. The disclosed program instructions are necessarily stored on a non-transitory machine-readable storage medium and, when executed by the disclosed CPU and GPU processors, cause those processors to perform the method steps mapped above. ), and performs the steps disclosed herein. Therefore, the claims are all rejected.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KEN KUDO whose telephone number is (571)272-4498. The examiner can normally be reached M-F 8am - 5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Vincent Rudolph can be reached at 571-272-8243. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
KEN KUDO
Examiner
Art Unit 2671
/KEN KUDO/Examiner, Art Unit 2671
/VINCENT RUDOLPH/Supervisory Patent Examiner, Art Unit 2671
