DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Information Disclosure Statement
For the sake of compact prosecution, the examiner requests that any substantive foreign prosecution documents falling under the duty to disclose and that are relevant to the instant application be timely filed with an IDS. See 37 C.F.R. 1.56. It is noted that to date, no IDS has been filed. At least two corresponding foreign applications have been found, CN120014254A filed 16 November 2023 and TW202522394A filed 21 November 2023. For the Taiwanese application, non-final Office action was mailed 15 November 2024, an amendment was filed 7 January 2025, and the granted application was published 11 June 2025.
Applicant is reminded, per 37 C.F.R. 1.56(b)(1), that foreign prosecution documents, e.g., Office actions, written opinions, search reports, cited prior art, and so forth, fall under the duty to disclose as they establish a prima facie case of unpatentability of a claim. Content requirements for an IDS are set forth in MPEP 609.04(a) and timing requirements for an IDS are set forth in MPEP 609.04(b).
Drawings
The drawings are objected to because they do not use the abbreviation “FIG.” See 37 C.F.R. 1.84(u)(1). Instead, the drawings use “Fig.”
Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. The figure or figure number of an amended drawing should not be labeled as “amended.” If a drawing figure is to be canceled, the appropriate figure must be removed from the replacement sheet, and where necessary, the remaining figures must be renumbered and appropriate changes made to the brief description of the several views of the drawings for consistency. Additional replacement sheets may be necessary to show the renumbering of the remaining figures. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Specification
The abstract of the disclosure is objected to because “A segmentation model training method is disclosed” can be implied by the following portions of the Abstract and the title.
A suggested amendment to the abstract that would obviate this objection is to remove the first sentence and change “The segmentation” in the second sentence to “A segmentation”.
A corrected abstract of the disclosure is required and must be presented on a separate sheet, apart from any other text. See MPEP § 608.01(b).
Applicant is reminded of the proper language and format for an abstract of the disclosure.
The abstract should be in narrative form and generally limited to a single paragraph on a separate sheet within the range of 50 to 150 words in length. The abstract should describe the disclosure sufficiently to assist readers in deciding whether there is a need for consulting the full patent text for details.
The language should be clear and concise and should not repeat information given in the title. It should avoid using phrases which can be implied, such as, “The disclosure concerns,” “The disclosure defined by this invention,” “The disclosure describes,” etc. In addition, the form and legal phraseology often used in patent claims, such as “means” and “said,” should be avoided.
Claim Interpretation
The term “small” in claims 1, 2, 5, and 9 and the term “large” in claims 1, 5, and 9 are relative terms. See MPEP 2173.05. The use of relative terminology in claim language, including terms of degree, does not automatically render the claim indefinite under 35 U.S.C. 112(b). MPEP 2173.05(b). The claim is not indefinite if the specification provides examples or teachings that can be used to measure a degree even without a precise numerical measurement. Id. Here, FIG. 12 provides a visual comparison between a large sample set (LS) and a small sample set (SS), showing that the large sample set is larger than the small sample set. Similarly, paragraph 73 of the specification provides, “In some embodiments, the large sample set LS is a sample set with a larger number of samples, while small sample set SS is a sample set with a smaller number of samples.” One of ordinary skill in the art would understand that what is being claimed, in light of the specification, and given the context of the rest of the claim language including “pre-training” and “fine-tuning” in relation to “sample groups” generated from the large and small sample sets, that the segmentation model training in the independent claims is leveraging the large sample set for pre-training and the small set for fine-tuning, and that the “large sample set” is larger than the “small sample set”. One of ordinary skill in the art would be familiar with the common practice of pretraining on larger, more generalized datasets and then finetuning with smaller, more task-specific datasets to tailor the training for a specific task. Thus, the relative terminology does not render the claims indefinite.
According to the Federal Circuit’s decision in SuperGuide v. DirecTV, claim language of the type “at least one of … and …” creates a presumption that Applicant intended the plain and ordinary meaning of the claim language to be a conjunctive list, unless the Specification supports an interpretation of the claim language that rebuts the presumption.1
Claims 3, 4, 7 and 8 recite limitations that raise the presumption of a conjunctive list per SuperGuide:
[Claim 3] The segmentation model training method of claim 2, further comprising:
adjusting at least one of a size and an angle of the target image ...
[Claim 4] The segmentation model training method of claim 1, further comprising:
adjusting at least one of a size, an angle, a color, and a position of a plurality of first images of the plurality of mix sample groups ... adjusting at least one of a size, an angle, a color, and a position of a plurality of second images of the plurality of first sample ... adjusting at least one of a size, an angle, a color, and a position of a plurality of first masks of the plurality of mix sample groups corresponding to the plurality of first images ... and adjusting at least one of a size, an angle, a color, and a position of a plurality of second masks ...
[Claim 7] The segmentation model training device of claim 6, wherein the processor is further configured to perform the following operations:
adjusting at least one of a size and an angle of the target image according to ...
[Claim 8] The segmentation model training device of claim 5, wherein the processor is further configured to perform the following operations:
adjusting at least one of a size, an angle, a color, and a position of ... adjusting at least one of a size, an angle, a color, and a position of a plurality of second images ... adjusting at least one of a size, an angle, a color, and a position of a plurality of first masks ... and adjusting at least one of a size, an angle, a color, and a position of a plurality of second masks ... (emphasis added).
No definition of “at least one of … and …” was found the specification, nor was any embodiment of a clear disclosure of a disjunctive version of any of the limitations reproduced above that would indicate Applicant intended to rebut the SuperGuide presumption. Accordingly, the presumption is not rebutted and the lists of claim elements recited in claims 3, 4, 7 and 8 are given the plain and ordinary meaning of being conjunctive lists, meaning that at a minimum, at least one of each element in each list is required. If Applicant had intended the interpretation to be disjunctive, the claims would have been written in the format of: “at least one of … or …” (emphasis added).
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-10 are rejected under 35 U.S.C. 101 because the claimed inventions are directed to a judicial exception without significantly more.
[Claim 1] A segmentation model training method, comprising:
(a) inputting a plurality of first sample groups of a large sample set to a data augmentation model to generate a plurality of augmentation sample groups;
(b) generating a plurality of mix sample groups based on a plurality of second sample groups of a small sample set;
(c) inputting the plurality of mix sample groups to the data augmentation model to generate a plurality of augmentation mix sample groups; and
(d) training a segmentation model according to the plurality of augmentation sample groups and the plurality of augmentation mix sample groups, comprising:
(e) performing pre-training to the segmentation model according to the plurality of augmentation sample groups; and
(f) performing fine-tuning training to the segmentation model corresponding to the plurality of augmentation mix sample groups.
[Claim 5] A segmentation model training device, comprising:
a memory, configured to store a segmentation model and a data augmentation model; and
a processor, coupled to the memory, configured to perform the following operations:
[limitations (a) - (f).]
[Claim 9] A non-transitory computer readable storage medium, configured to store a computer program, wherein when the computer program is executed, one or more processors are executed to perform a plurality of operations, wherein the plurality of operations comprise:
[limitations (a) - (f).]
Claim Interpretation
Under the broadest reasonable interpretation, the terms of the claims are presumed to have their plain meaning consistent with the specification as it would be interpreted by one of ordinary skill in the art. See MPEP 2111.
Based on the plain meaning of the words in the independent claims, the broadest reasonable interpretation of the independent claims is a method, a device having processor and a memory (i.e., a general purpose computer) storing models for segmentation and data augmentation for performing the method, and a non-transitory storage medium storing instructions that are executed by a processor to perform the method, which includes limitations (a) - (f).
Regarding limitations (a) and (b), the claims do not put any explicit limits on an absolute size or range of sizes of “a large sample set” and a “small sample set”. However, one of ordinary skill in the art would at least understand that the large sample set is larger relative to the small sample set, but would not put any further limit on the absolute sizes of the sample sets. In the context of “augmentation”, the phrase “mix sample” in limitations (b) - (d) could be interpreted as relating the claims to the technological environment of mixed-sample data augmentation algorithms and strategies such as Cutmix and Mixup. However, under the broadest reasonable interpretation, a “mix sample group” is merely a group of samples where at least one sample does not share a characteristic with the other samples in the group. Limitations (a) - (f) describe different steps or stages of a segmentation model training process/strategy that do not put any substantial limits on the involvement or lack thereof of a person in relation to the general purpose computer that is either implied as in the case of claim 1 (i.e., pre-training and fine-tuning are operations performed by a computer) or positively recited as in claims 5 and 9 (i.e., memory and processor in claim 5 and the storage medium and one or more processors of claim 9).
Concerning the “augmentation model” and “segmentation model”, the inclusion of “performing pre-training ... and fine-tuning to the segmentation model” according/corresponding to the respective sample groups implicitly discloses the “segmentation model” as including, at a minimum, an input, parameters that can be adjusted through optimization of an error metric, and a mapping from input(s) to output(s). This describes any model with those characteristics from deep-learning convolutional neural network-based models and/or combinations of such models with other models like transformer-based models with millions of adjustable/learnable parameters and many layers, down to simpler and less-sophisticated algorithms. For example, a “segmentation model” could segment an image by simply dividing the pixels into groups according to a threshold.
Likewise, “the augmentation model”, which is separate from the “segmentation model” and is not recited as being trainable, could simply be a set of stored affine transformations, which are well-known in the machine learning community, to perform basic linear translations and/or rotations on an input image.2
Step 1: do the claims fall within any statutory category?
Claim 1 recites a series of steps and therefore, is a process. Claim 5 recites a device comprising a memory and a processor and therefore, is a machine. Claim 9 recites a non-transitory computer readable storage medium. The disclosure gives magnetic media such as hard disks, floppy disks, and magnetic tapes, optical media such as an optical disc, and hardware devices such as read-only memory (ROM), flash memory, HDD, SSD, DRAM, and SRAM as non-limiting examples of a computer-readable recording medium. The broadest reasonable interpretation of claim 9 covers only statutory embodiments of a computer-readable storage medium and not a transitory signal. A non-transitory computer readable storage medium falls within the “manufacture” category of invention. See MPEP 2106.03. (Step 1: YES).
Step 2A, Prong One: do the claims recite a judicial exception?
Overall, the subject matter of the independent claims describes data augmentation and training a segmentation model via pre-training and fine-tuning. The “segmentation model” is described at a high level of generality and is so broadly-recited that it amounts to an attempt to claim an entire class of strategies and algorithms. Despite reciting words and phrases that typically accompany descriptions of machine learning, e.g., “augmentation”, “training”, “pre-training” and “fine-tuning”, the independent claims do not explicitly recite any images, pixels, or specific image-based features (e.g., intensity or color values) that are extracted and applied during training of the model. Despite dependent claims further specifying the inclusion of images, e.g., “a plurality of first images of the plurality of mix sample groups” in claim 4, narrowing the meaning of the models, input or augmented input samples to be explicitly image-based would not negate the abstract idea of being human-centric as opposed to technology-centric where the involvement of a human is incidental to a technological improvement to a technical problem being claimed.
The method represented by limitations (a) - (f) is embodied in each independent claim, where claim 1 is the method itself and each step of the method is reasonably interpreted as a manual (by-hand), unautomated act and/or decision made by a person who interacts with an implied/unrecited generic computer. Even though the claims require “the plurality of mix sample groups” be input or provided “to the data augmentation model” in order “to generate a plurality of augmentation mix sample groups”, for example, the act of “inputting” is still reasonably interpreted to be describing the actions of a person using a computer as the end result of their internal mental decision making, judgments, and opinions, e.g., interaction of a person with a graphical user interface (GUI) by opening, dragging and dropping, or otherwise providing a dataset to a software program via user input(s). The recitation of structures in the claims, e.g., “memory” and “processor” in claim 5, does not negate the mental nature of these limitations because the claims merely use the structural components as tools to carry out the aims and decisions of the model’s human designer(s). See MPEP 2106.04(a)(2), subsection III.C.
Under the broadest reasonable interpretation, the claims describe the observations, evaluations, judgments, and opinions of a person preparing to and subsequently train a segmentation model as they complete a mental checklist of various stages of training, which falls within the mental process grouping of abstract ideas. See MPEP 2106.04(a)(2), subsection III. (Step 2A, Prong One: YES).
Step 2A, Prong Two: Do the claims as a whole integrate the recited judicial exception into a practical application of the exception?
Claims 1, 5, and 9 recite six additional elements: (i) “a memory, configured to store a segmentation model and a data augmentation model; and a processor, coupled to the memory, configured to perform the following operations” in claim 5, (ii) “A non-transitory computer readable storage medium, configured to store a computer program, wherein when the computer program is executed, one or more processors are executed to perform a plurality of operations, wherein the plurality of operations comprise” in claim 9, and in claims 1, 5 and 9: (iii) “a data augmentation model to generate a plurality of augmentation sample groups”, (iv) “a plurality of augmentation mix sample groups”, (v) “pre-training to the segmentation model”, and (vi) “fine-tuning training to the segmentation model corresponding to the plurality of augmentation mix sample groups”.
The additional elements (i), (ii), (v), and (vi) describe generic computer components and/or operations thereof (e.g., using a pre-built PC, data augmentation using Microsoft Paint to crop, copy, and paste image portions to mix them together is enabled out of the box assuming the operating system is installed) recited at a high level of generality and do not amount to any of the relevant considerations for evaluating whether additional limitations integrate a judicial exception into a practical application provided in MPEP 2106.04(d), subsection I. The additional elements amount to merely including instructions to implement the abstract idea on a computer or merely using a computer as a tool to perform the abstract idea. See MPEP 2106.05(f). The additional elements of the claims invoke computers merely as a tool to perform an existing process: augmenting training data, pretraining a model, and then fine-tuning the model for a specific task or set of tasks. See MPEP 2106.05(f)(2).
Limitations (a) - (c), which include additional elements (iii) and (iv), amount to mere data gathering (of samples groups) in conjunction with the abstract idea recited at a high-level of generality. The additional elements of “pre-training to the segmentation model” and “fine-tuning training to the segmentation model corresponding to the plurality of augmentation mix sample groups” merely confine the use of the abstract idea to a particular technological environment (ML/AI model training) and thus fail to add an inventive concept to the claims. See MPEP 2106.05(h).
It should be noted that because the courts have made it clear that mere physicality or tangibility of an additional element or elements is not a relevant consideration in the eligibility analysis, the physical nature of any underlying hardware in the additional elements, e.g., “augmentation model”, “training”, “pre-training” and “fine-tuning” does not affect this analysis. See MPEP 2106.05(I).
Dependent claims 2-4, 6-8, and 10 generally describe further actions taken by a person using a recited or implied general-purpose computer to execute simple image-based techniques at the person’s direction to generate more training data, i.e., using masks to locate corresponding regions in multiple images to cut and paste (i.e., swap) those regions to generate new images as in claims 2, 6 and 10, performing standard data augmentation techniques like re-sizing and rotation as in claims 3 and 7, and re-coloring and position shifting as in claims 4 and 8. While the image masks and data augmentation operations in these claims are additional elements, the claims invoke computers merely as a tool to perform an existing process: augmenting training data, pretraining a model, and then fine-tuning the model for a specific task or set of tasks. See MPEP 2106.05(f)(2).
The additional elements do not improve the functioning of a computer. See MPEP 2106.04(d)(1). The specification’s background section sets forth a technical problem “[automatic] segmentation technology has made great progress through deep neural network (DNN). However, since the acquisition and annotation of medical images takes a lot of time, the current training challenge of the segmentation model is that the number of samples is insufficient and the training can be performed only with limited samples, which results in a decrease in performance” (par. 3). The figures, e.g., Fig. 3, appear to show skin lesion images. Assuming that a particular type of lesion has an inadequate number of available training samples, e.g., a foot ulcer3, then, per the disclosed problems in the prior art, training a segmentation model to segment foot ulcers would likely not yield a vary accurate segmentation if presented with an input image containing a foot ulcer with a significantly different appearance than any ulcer in the training data. Without admitting that any particular language, if brought from the specification into the independent claims would constitute an “improvement” per MPEP 2106.04(d)(1), and for the sake of compact prosecution, the following portions of the specification may describe technical features that could contribute towards an “improvement”:
paragraph 73, unlike the independent claims, provides, “In an embodiment, the large sample set LS is a sample set that is public and covers a wider domain, while the small sample set SS is a sample set that is not public and covers a narrower domain.”
paragraph 74, unlike claims 2-4, 6-8 and 10 which do not further define the “pre-training” in the independent claims, provides, “[the] processor 110 as illustrated in Fig. 1 performs fine-tuning training to the segmentation model SM which includes the pre-training parameters PW based on the augmentation mix sample set AMS.”
Whether evaluated individually or in combination, the additional elements do not integrate the recited judicial exception into a practical application and the independent claims, therefore, are directed to the judicial exception. (Step 2A, Prong Two: NO).
Step 2B: do the claims as a whole amount to significantly more than the judicial exception?
As explained with respect to Step 2A Prong Two, the additional elements of the pending claims amount to performing the abstract idea using a computer as a tool to perform an existing process (retrieve data and manipulate data at a user’s direction), which cannot provide an inventive concept. See MPEP 2106.05(f). Based on the high-level of specify of the technical aspects of the independent and dependent claims as compared to subject matter in the specification, including the drawings, that is in certain aspects more specific and rooted in the technical problems being solved in the additional elements, the additional elements do not constitute an improvement to the functioning of a computer or to another technology because they represent what is well-understood, routine, conventional activity4. See MPEP 2106.04(d)(1).
Also explained above, limitations (a) - (c) of claims 1, 5, and 9 amount to mere data gathering, which is a form of insignificant extra-solution activity, in conjunction with the abstract idea recited at a high-level of generality (e.g., a user opens image files in Microsoft Paint, cuts out random rectangular patches of equal size, and swaps them randomly). Under the 2019 PEG, a conclusion that an additional element is insignificant extra-solution activity in Step 2A should be re-evaluated in Step 2B. 2019 PEG Section III(B), 84 Fed. Reg. at 56. At Step 2B, the evaluation of the insignificant extra-solution activity consideration takes into account whether or not the extra-solution activity is well-known. See MPEP 2106.05(g). Here, the recitation of providing inputs to a “data augmentation model” to obtain “sample groups” as outputs is recited at a high level of generality, and is also well-known.5 Limitations (a) - (c) therefore remain insignificant extra-solution activity.
Dependent claims 2-4, 6-8, and 10 also describe image mixing or cut-paste operations at a high level of generality and/or describe routine data augmentation operations in the field of machine learning, e.g., re-sizing, rotating, re-coloring, shifting position, which merely generally links the abstract idea to a particular field of use: machine learning. See MPEP 2106.05(h). Additionally, the dependent claims do not include any subject matter that amounts to an improvement of the functioning of a computer, or an improvement to any other technology or technical field, as they merely describe and apply known concepts in the fields of image analysis and machine learning. See MPEP 2106.04(d)(1).
The additional elements in combination with the judicial exception do not provide an improvement to the functioning of a computer or any other technology or technical field. (Step 2A, Prong Two: NO). Even considering each claim as a whole, the claims do not amount to significantly more than the recited judicial exception and fail to encompass an inventive concept (Step 2B: NO). Claims 1-10, therefore, are not eligible.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed inventions absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-10 are rejected under 35 U.S.C. 103 as being unpatentable over Automatic Foot Ulcer Segmentation Using an Ensemble of Convolutional Neural Networks to Mahbod et al. (hereinafter “Mahbod”) in view of Object-Based Augmentation for Building Semantic Segmentation: Ventura and Santa Rosa Case Study to Illarionova et al. (hereinafter “Illarionova”), in view of Improving Concrete Crack Segmentation Networks through CutMix Data Synthesis and Temporal Data Fusion to Jamshidi et al. (hereinafter “Jamshidi”), and in further view of WSNet: Towards An Effective Method for Wound Image Segmentation to Oota et al. (hereinafter “Oota”).
Regarding claim 1, Mahbod teaches a segmentation model training method, comprising:
inputting a plurality of first sample groups (image-mask pairs are obtained for pre-training from the Medetec dataset, which includes 152 images and their segmentation masks for different types of open foot wounds. See Mahbod at section II.A. The 152 image+mask pairs are a dataset, which includes (is equivalent to) a series of groups, e.g., the first half in a first group and the second half in a second group.) of a sample set (Mahbod, section II.A, “Medetec dataset”) to generate a plurality of augmentation sample groups (The Medetec dataset is input to the computer system to generate sample group of image-mask pairs for training. See Mahbod at section II);
generating a plurality of sample groups based on a plurality of second sample groups of a sample set (The FUSeg dataset is used for the task-specific training following the pre-training, i.e., fine-tuning. 1010 images and their masks are sampled from FUSeg. See Mahbod at section II.A.);
inputting the plurality of sample groups to generate a plurality of augmentation sample groups (The sampled dataset for fine-tuning is augmented using random scaling, 90-degree rotations, horizontal flipping, vertical flipping, brightness shifting, and contrast shifting. See Mahbod at section II.C.); and
training a segmentation model according to the plurality of sample groups and the plurality of augmentation sample groups (See Mahbod at Fig. 1 and section II.C), comprising:
performing pre-training to the segmentation model according to the plurality of sample groups (Mahbod, Fig. 1, “Medetec Pre-training”); and
performing fine-tuning training to the segmentation model corresponding to the plurality of augmentation sample groups (Mahbod, Fig. 1, “Foot Ulcer Training”), but does not teach that which is explicitly taught by Illarionova.
Illarionova teaches inputting a plurality of first sample groups of a sample set (Illarionova, pg. 1665, section 5.1, “start training with a bigger augmented set”) to a data augmentation model (A software package with built-in transform operations is used to augment the training images. See Illarionova at section 3.1. Software that performs the same algorithm for image transformation when requested by a user follows a model of image transformation. Illarionova, pg. 1662, section 3.1, “OBA includes the following options … Shadows addition (length and intensity may vary) … Objects number per crop selection (default: up to 3 extra objects) … Selection of base color and geometrical transformations probability (default: 50%) … Background images selection (default: 60%) … Selection of original and generated samples mixing probability (default: 60%).”) to generate a plurality of augmentation sample groups (Augmentation is performed for pre-training and fine-tuning. See Illarionova at pg. 1665, section 5.1, “pretraining in the object-based augmentation mode (without original sample usage) for 10 epochs and further training in the base augmentation mode for 4 epochs for our task leads to the best result for the considered experiment.”; The object-based augmentation (OBA) mode includes augmentation options of base color, HueSaturationValue, and geometrical transformations, e.g., RandomRotate90 and flip, among others. See Illarionova at pg. 1662, section 3.1.), and
generating sample groups of a sample set (Illarionova, pg. 1665, section 5.1, “continue training with a smaller set”).
Mahbod discloses color and geometrical data augmentation transformations to improve fine-tuning an image segmentation model with a smaller sample set than a s ample set used for pre-training the model. Thus, Mahbod shows that it was known in the art before the effective filing date of the claimed invention to use color transformations and geometrical transformations to augment training data for fine-tuning a segmentation model during training, which is analogous to the claimed invention in that it is pertinent to the problem being solved by the claimed invention, training a segmentation model with a limited number of available training samples. Illarionova discloses sample-mixing, color and geometrical data augmentation transformations to improve fine-tuning an image segmentation model. Thus, Illarionova shows that it was known in the art before the effective filing date of the claimed invention to use color transformations and geometrical transformations to augment training data for pre-training a segmentation model prior to fine-tuning, which is analogous to the claimed invention in that it is pertinent to the problem being solved by the claimed invention, training a segmentation model with a limited number of available training samples.
A person of ordinary skill in the art would have been motivated to combine the data augmentation software model and the use of color and geometrical transformations as pre-training augmentation options as disclosed by Illarionova with the computer system and training sample datasets of Mahbod, to thereby perform color and geometrical transformations of the image-mask pairs use during pre-training and provide a designer of the model with the option to perform the same types of data augmentation for fine-tuning as for pre-training. Based on the foregoing, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have made such modification according to known methods to yield the predictable results to have the benefit of creating a larger training set without adding redundant features.
Mahbod in view of Illarionova does not teach that which is explicitly taught by Jamshidi.
Jamshidi teaches performing fine-tuning training to the segmentation model corresponding to the plurality of augmentation mix sample groups (Jamshidi, pg. 4, “the segmentation network trained on the public dataset is first fine-tuned using a dataset synthesized through the CutMix technique.”).
Mahbod in view of Illarionova is analogous to the claimed invention for the reasons provided above. Jamshidi discloses fine-tuning an image segmentation model with CutMix synthetic data. Thus, Jamshidi shows that it was known in the art before the effective filing date of the claimed invention to use a sampling-mixing algorithm, e.g., CutMix, to expand the number of training samples used in fine-tuning an image segmentation model, which is analogous to the claimed invention in that it is pertinent to the problem being solved by the claimed invention, training a segmentation model with a limited number of available training samples.
A person of ordinary skill in the art would have been motivated to combine CutMix during fine-tuning as disclosed by Jamshidi with the training procedure disclosed by Mahbod in view of Illarionova, to thereby apply CutMix to image-mask pairs to generate synthetic training mix sample groups of images and masks for fine-tuning the image segmentation model that are first expanded with CutMix and then expanded further with color and geometrical transformations. Based on the foregoing, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have made such modification according to known methods to yield the predictable results to have the benefit of increasing the variety of data available for fine-tuning.
Mahbod in view of Illarionova and in further view of Jamshidi, does not teach that which is explicitly taught by Oota.
Oota teaches a large sample set (pre-training on a dataset comprising 123,711 images labeled for wound classification and fine-tuning. See Oota at sections 3.2 and 3.3; Oota, pg. 3240, section 6, “We also contribute a better and larger wound image dataset, which can help the research community to advance wound image analysis further.”) and a small sample set (Table 2 indicates WoundSeg includes 2,686 image-mask pairs. Within the WoundSeg dataset, there are even smaller subsets of sample, e.g., 441 samples for diabetic-type wounds. See Oota at Fig. 1.).
Mahbod in view of Illarionova and in further view of Jamshidi is analogous to the claimed invention for the reasons provided above. Oota discloses generating pre-trained models on a large unlabeled sample set and fine-tuning an image segmentation model end-to-end with a small sample set of image-mask pairs. Thus, Oota shows that it was known in the art before the effective filing date of the claimed invention to strive for using a larger sample set for pre-training a segmentation model and a smaller sample set for fine-tuning the model, which is analogous to the claimed invention in that it is pertinent to the problem being solved by the claimed invention, training a segmentation model with a limited number of available training samples.
A person of ordinary skill in the art would have been motivated to replace or supplement the Medetec dataset comprising 152 samples disclosed by Mahbod in view of Illarionova and in further view of Jamshidi with the WoundSeg dataset comprising 2,686 samples disclosed by Oota, to thereby pre-train the segmentation model using a large sample set with at least 2,686 samples and fine-tune the segmentation model using the 1010 samples of the smaller FUSeg dataset. Based on the foregoing, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have made such modification according to known methods to yield the predictable results to have the benefit of improving the accuracy of the pre-training epochs and as a consequence, improve the accuracy of the end-to-end train segmentation model.
Regarding claim 2, Mahbod in view of Illarionova, in view of Jamshidi, and in further view of Oota teaches the segmentation model training method of claim 1, wherein generating the plurality of mix sample groups based on the plurality of second sample groups of the small sample set comprising:
capturing a target image (Jamshidi, pg. 12, “the area inside the rectangle is cropped from Image (A)”) of a first image (Jamshidi, pg. 12, “Image (A)”) based on the first image and a first mask (The area of “Label (A)” that spatially corresponds the (w × h) patch of “Image (A)” as shown in the “Label (CC)” mask image. See Jamshidi at Fig. 11.) of one of the plurality of second sample groups (Jamshidi, FIG. 11, “Label (A)”);
obtaining a target area (Jamshidi, pg. 12, “a rectangular patch with the center at coordinates (xc, yc), displayed by the “+” symbol in Figure 11, and width, w, and height, h, is randomly generated”) of a second image (Jamshidi, Fig. 11, “Image (B)”) based on the second image and a second mask (Jamshidi, Fig. 11, “Label (B)”) of another one of the plurality of second sample groups; and
pasting the target image to the target area of the second image (Jamshidi, pg. 12, “pasted in Image (B)”) to generate a third image (Jamshidi, pg. 12, “new combined Image (C)”) of one of the plurality of mix sample groups;
wherein a ratio value between an area of the target image (w × h) and an area of the target area (512 × 512) is located within a ratio value range (w and h depend on λ, which has a range of values. See Jamshidi, pg. 13, equations (11) and (12), thus establishing a range of values of the ratio between (w × h) and (512 × 512); pg. 12, “Here, both images have equal dimensions of 512 × 512 pixels. To combine these two images, a rectangular patch with the center at coordinates (xc, yc), displayed by the “+” symbol in Figure 11, and width, w, and height, h, is randomly generated”; pg. 13, “As shown in Equation (11), the width and height of the rectangle are calculated through variable λ, where λ is sampled from a uniform distribution (Equation (12)). In this manner, the resulting rectangle maintains the aspect ratio of the underlying image.”).
The rationale for obviousness is the same as provided for claim 1.
Regarding claim 3, Mahbod in view of Illarionova, in view of Jamshidi, and in further view of Oota teaches the segmentation model training method of claim 2, further comprising:
adjusting at least one of a size (Mahbod, section II.C, “scaling (scale limit 0.1 with 0.3 probability)”) and an angle of the target image (Mahbod, section II.C, “random 90-degree rotations (with 0.5 probability)”) according to an adjusting parameter (A scale factor and a rotation angle are both parameters for adjusting or transforming the image-mask pair. See Mahbod at section II.C) to generate an adjusted target image (the transformation is applied to the cropped object area to be pasted and its corresponding mask), wherein an area of the adjusted target image is the same as an area of the target area (image and its corresponding mask are transformed together);
pasting the adjusted target image (Jamshidi, pg. 12, “pasted in Image (B)”) to the target area of the second image to generate the third image (Jamshidi, pg. 12, “new combined Image (C)”) of the one of the plurality of mix sample groups; and
overlapping the first mask (The area of “Label (A)” that spatially corresponds the (w × h) patch of “Image (A)” as shown in the “Label (CC)” mask image. See Jamshidi at Fig. 11.) and the second mask (Jamshidi, Fig. 11, “Label (B)”) to generate a third mask (Jamshidi, Fig. 11, “Label (C)”) of the one of the plurality of mix sample groups, comprising:
adjusting the first mask according to the adjusting parameter to generate an adjusted mask (The mask after the augmentation has been applied. See Mahbod at section II.C); and
overlapping the adjusted mask and the second mask to generate the third mask of the one of the plurality of mix sample groups (The cropped segmentation mask region of the first image is pasted into the segmentation mask of the second image, thereby overlapping it. See Jamshidi at Fig. 11).
The rationale for obviousness is the same as provided for claim 1.
Regarding claim 4, Mahbod in view of Illarionova, in view of Jamshidi, and in further view of Oota teaches the segmentation model training method of claim 1, further comprising:
adjusting at least one of a size (Mahbod, section II.C, “scaling (scale limit 0.1 with 0.3 probability)”), an angle (Mahbod, section II.C, “random 90-degree rotations (with 0.5 probability)”), a color (Mahbod, Abstract, “applying a number of morphological-based and colour-based augmentation techniques.”; The training image datasets include color images. See Mahbod at Fig. 2. Mahbod, section II.C, “brightness and contrast shifts (limit of 0.15 with 0.4 probability)”), and a position (A flipping transformation changes the relative positions of pixels. See Mahbod at section II.C, “vertical and horizontal flipping (with 0.5 probability)”) of a plurality of first images of the plurality of mix sample groups according to the data augmentation model to generate a plurality of first augmentation images of the plurality of augmentation mix sample groups (The fine-tuning training images after CutMix and data augmentation are applied; Jamshidi, pg. 4, “the segmentation network trained on the public dataset is first fine-tuned using a dataset synthesized through the CutMix technique.”);
adjusting at least one of a size (Mahbod, section II.C, “scaling (scale limit 0.1 with 0.3 probability)”), an angle (Mahbod, section II.C, “random 90-degree rotations (with 0.5 probability)”), a color (Mahbod, Abstract, “colour-based augmentation techniques.”), and a position (Mahbod at section II.C, “vertical and horizontal flipping”) of a plurality of second images of the plurality of first sample groups (A second group of the 152 image-mask pairs. See Mahbod at section II) according to the data augmentation model to generate a plurality of second augmentation images of the plurality of augmentation sample groups (a second plurality of the augmented image-mask pairs (groups) for pre-training includes a plurality of second augmentation images. See Illarionova at pg. 1665, section 5.1, “pretraining in the object-based augmentation mode (without original sample usage) for 10 epochs and further training in the base augmentation mode for 4 epochs for our task leads to the best result for the considered experiment.”);
adjusting at least one of a size (Mahbod, section II.C, “scaling (scale limit 0.1 with 0.3 probability)”), an angle (Mahbod, section II.C, “random 90-degree rotations (with 0.5 probability)”), a color (Mahbod, Abstract, “colour-based augmentation techniques.”), and a position (Mahbod at section II.C, “vertical and horizontal flipping”) of a plurality of first masks of the plurality of mix sample groups corresponding to the plurality of first images of the plurality of mix sample groups to generate a plurality of first augmentation masks of the plurality of augmentation mix sample groups (augmenting the masks of the fine-tuning training samples generated with CutMix. See Jamshidi at pg. 4, “the segmentation network trained on the public dataset is first fine-tuned using a dataset synthesized through the CutMix technique.”); and
adjusting at least one of a size (Mahbod, section II.C, “scaling (scale limit 0.1 with 0.3 probability)”), an angle (Mahbod, section II.C, “random 90-degree rotations (with 0.5 probability)”), a color (Mahbod, Abstract, “colour-based augmentation techniques.”), and a position (Mahbod at section II.C, “vertical and horizontal flipping”) of a plurality of second masks of the plurality of first sample groups corresponding to the plurality of second images of the plurality of first sample groups to generate a plurality of second augmentation masks of the plurality of augmentation sample groups (the second plurality of the augmented image-mask pairs (groups) for pre-training includes a plurality of second masks paired with respective augmentation images. See Illarionova at pg. 1665, section 5.1.).
The rationale for obviousness is the same as provided for claim 1.
Claims 5-8 substantially correspond to claims 1-4 by reciting a segmentation model training device, comprising: a memory (A computer-aided neural network-based image segmentation model using an Nvidia GPU includes a non-transitory memory and processor. See Mahbod at section IV.), configured to store a segmentation model and a data augmentation model (GPU stores parameters needed to execute the model and program data stored in computer memory. See Mahbod at section IV); and a processor (A processor of the computer. See Mahbod at section IV.), coupled to the memory, configured to perform operations corresponding to the segmentation model training methods of claims 1-4 respectively. The rationale(s) for obviousness is/are the same as provided for claims 1-4.
Claims 9 and 10 substantially correspond to claims 1 and 2 by reciting non-transitory computer readable storage medium (A computer-aided neural network-based image segmentation model using an Nvidia GPU includes a non-transitory memory and processor. See Mahbod at section IV.), configured to store a computer program (GPU stores parameters needed to execute the model and program data is stored in computer memory. See Mahbod at section IV), wherein when the computer program is executed, one or more processors are executed to perform a plurality of operations corresponding to the segmentation model training methods of claims 1 and 2 respectively. The rationale(s) for obviousness is/are the same as provided for claims 1 and 2.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Cutmix: Regularization strategy to train strong classifiers with localizable features to Yun et al. is pertinent because it is a widely-adopted augmentation technique, gained improvements over other mixed-sample data augmentation techniques including CutOut and Mixup, and is conceptually similar to the augmentation method disclosed in the instant specification.
Improving Limited Supervised Foot Ulcer Segmentation Using Cross-Domain Augmentation Strategies to Kuo et al. is considered very pertinent to applicant’s disclosure because its authorship includes every co-inventor of the instant application except for Wei-Chao Chen and would anticipate much of the claimed subject matter, if not for being disqualified as prior art for publishing in April 2024 when the instant application was filed in January 2024. It is also considered very pertinent because to date, no IDS has been filed and no prior art has been indicated in the specification, it provides a detailed overview of the state of the art at the time the instant application was filed (section I), and it provides relevant citations that do qualify as prior art in terms of their publication dates, e.g., the 2019 CutMix paper of Yun et al. cited above.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to RYAN P POTTS whose telephone number is (571)272-6351. The examiner can normally be reached M-F, 9am-5pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Sumati Lefkowitz can be reached at 571-272-3638. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/RYAN P POTTS/Examiner, Art Unit 2672
/SUMATI LEFKOWITZ/Supervisory Patent Examiner, Art Unit 2672
1 See Superguide Corp. v. Direct TV Enterprises, Inc., 358 F.3d 870, 69 USPQ2d 1865 (Fed. Cir. 2004).
2 See https://web.archive.org/web/20180530050015/https://en.wikipedia.org/wiki/Affine_transformation
3 See Automatic Foot Ulcer Segmentation Using an Ensemble of Convolutional Neural Networks to Mahbod et al.
4 See e.g., https://web.archive.org/web/20180530050015/https://en.wikipedia.org/wiki/Affine_transformation as an example of affine transformations being well-known, understood, and routine in image processing and Cutmix: Regularization strategy to train strong classifiers with localizable features to Yun et al. as an example of mixed-sample data augmentation implemented with a computer as being well-understood, routine, and conventional.
5 See Object-based augmentation for building semantic segmentation: Ventura and santa rosa case study to Illarionova et al. (Section 3.1 describes a software package with built-in transform operations is used to augment the training images.).