DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Information Disclosure Statement
The IDS(s) has/have been considered and placed in the application file.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claim 15-16 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention.
Claim 15 recites “scoring a connection manner” and requires comparing “a score” against a first reference value and a second reference value. Claim 16 further conditions the iteration on the value of that score. The specification describes the score only in terms of the comparisons themselves: the processor “connects the two end points A and B at a default angle and scores a connection manner L” (¶ 42), and the score Y is then compared to a first reference value X and to 0.95 (¶ 43). The boundary enhancing operation is stated to be performed “through the discriminator network 320” (¶ 41), and Fig. 9 shows the connection manner L supplied to the discriminator network, which output an adopted or not adopted determination.
Identifying the discriminator as the score does not describe how the score is computed. The specification does not quantify what the score represents, no algorithm, metric, formula, nor criterion by which the score is computed is disclosed anywhere in the specification. The disclosure therefore does not convey possession of the claimed scoring step, and does not enable one of ordinary skill in the art to perform it without undue experimentation. “Computer-implemented functional claim language must still be evaluated for sufficient disclosure under the written description and enablement requirements of 35 U.S.C. 112(a).” MPEP 2161.01. the disclosure does not provide adequate written description of the claimed scoring step. Ariad Pharm., Inc. v. Eli Lilly & Co., 598 F.3d at 1336 (Fed. Cir. 2010) (en banc).
Claim(s) 16 depend either directly or indirectly from the rejection of Claim(s) 15, therefore they are also rejected.
CLAIM INTERPRETATION
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are: “the storage element stores” in claim(s) 18. The specification identifies corresponding structure at ¶ 22 (random access memory, read-only memory, flash memory, hard disk drive, and solid state drive).
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim 1 recites training the model “to obtain a weight value” and separately recites the first loss parameter is “a weighted sum” of the two recited averages. The weighted sum does not refer back to the obtained weight value, and no limitation of claim 1 requires the two to be the same. Claim 1 is therefore understood to cover a weighed sum using any weighting. Claim 5 is the first claim to tie the weight value to a source, namely a lookup table.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-4, 10, 17 and 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Anantatamukala et al., “Generative adversarial networks assisted machine learning based automated quantification of grain size from scanning electron microscope back scatter images” (hereinafter “ANANTATAMUKALA”) ,in view of Isola et al., “Image-to-Image Translation with Conditional Adversarial Networks” (hereinafter “ISOLA”) .
Claim 1.
Anantatamukala discloses a method for detecting an image boundary using a generative adversarial network model, wherein the generative adversarial network model comprises a generator network and a discriminator network (ANANTATAMUKALA: “The machine
learning model employed for this task was the Pix2Pix architecture which is a conditional
GAN” (Section 3.1, p. 3); “the task of segmenting the grains can be equivalently framed as
image translation (or mapping) of source microstructures into grain boundary maps”
(Section 2, p. 2). This teaches a conditional GAN having generator and discriminator
networks trained to produce grain boundary maps.), the method comprising:
training the generative adversarial network model using a plurality of image pictures to obtain a weight value, (ANANTATAMUKALA: "Successful training of the
GANs depends on the choice of network architecture ... loss functions (L1 loss, L2 loss,
binary cross-entropy etc. ... ), relative weights of the different losses (generator loss, and
discriminator loss), optimization algorithm used etc.” (Section 3.1, p. 5); 3,000 training
images were used (Section 3, p. 3). This teaches training on a plurality of images to obtain a
weight applied to a generator loss and a discriminator loss.);
updating the generator network with a first loss parameter, wherein the first loss parameter is (ANANTATAMUKALA: “generator's weights are adjusted to reduce this error (using the feedback from discriminator loss value) to generate better output in the subsequent iterations of training” (Section 3.1, p. 5). This teaches updating the generator from a combination of its own loss and the discriminator loss, the relative weighting of those two losses being a stated training parameter.);
Anantatamukala does not specifically teach the average value of each loss parameter, the weighted sum as an express objective, or updating the discriminator network with the discriminator loss alone. However, Isola, whose framework Anantatamukala adopts, teaches these limitations (ISOLA: the final objective at Section 2.1, Eq. (4) is G* = arg minGmaxD LcGAN(G, D) + λ LL1(G), where Eq. (3) defines LL1(G) as the expectation of IIY - G(x,z)ll1; “G tries to minimize this objective against an adversarial D that tries to maximize it” (Section 2.1); “We run this discriminator convolutionally across the image, averaging all responses to provide the ultimate output of D” (Section 2.2.2); and “we alternate between one gradient descent step on D, then one step on G” (Section 2.3). This teaches a generator objective that is the weighted sum of an averaged generator loss and an averaged discriminator loss, and a discriminator updated in a separate step on the discriminator loss alone.).
Therefore, it would have been obvious to one of ordinary skill in the art to combine Anantatamukala with Isola before the effective filing date of the claimed invention. Anantatamukala expressly adopts the Isola framework (ANANTATAMUKALA: “The architecture of the generator network employed in the current work is based on the original work of Pix2Pix frame work [32]” (Section 3.1, p. 5).) The motivation would have been to obtain the loss weighting and update schedule that Anantatamukala identifies as governing successful GAN training, thereby stabilizing training against the blurred, discontinuous boundary output that results from a reconstruction loss alone.
Claim 2.
Anantatamukala and Isola teaches wherein the plurality of image pictures comprise a sliced original image and a sliced true image, and the sliced original image and the sliced true image have a corresponding positional relationship (ANANTATAMUKALA: "a set of examples (source (BSE images) and target (grain boundary map/ground truths) image pairs) are fed to the model's generator and discriminator" (Section 3.1, p. 4); "each image was of 2040 x 2040 pixels and all such images were further split into smaller tiles of 512 x 512 pixels and later used in the model after resizing them to 256 x 256 pixels" (Section 3, p. 3). This teaches paired source and ground-truth tiles cut from corresponding positions of the same image.).
Claim 3.
Anantatamukala and Isola teaches wherein the sliced true image comprises a plurality of training sliced images, and steps for training the generative adversarial network model using the plurality of image pictures comprise: inputting the sliced original image into the generator network to generate a sliced fake image; and calculating a difference between the sliced fake image and the plurality of training sliced images, wherein the difference between the sliced fake image and the plurality of training sliced images is the generator network loss parameter (ISOLA: Eq. (3) defines LL1(G) as the expectation of IIY - G(x,z)ll1. This teaches the generator loss as the difference between the generated image and the ground-truth image.).
Claim 4.
Anantatamukala and Isola teaches wherein steps for training the generative adversarial network model using the plurality of image pictures further comprise: inputting the sliced fake image into the discriminator network to obtain a first value from 0 to 1, wherein a difference between the first value and 0 is the discriminator network loss parameter corresponding to the sliced fake image; inputting the plurality of training sliced images into the discriminator network to obtain a second value from 0 to 1, wherein a difference between the second value and 1 is the discriminator network loss parameter corresponding to the plurality of training sliced images; and calculating the weighted sum of the average value of the generator network loss parameter and the average value of the discriminator network loss parameter as the first loss parameter, wherein the discriminator network loss parameter comprises the discriminator network loss parameter of the sliced fake image and the discriminator network loss parameter of the plurality of training sliced images (ISOLA: Eq. (1) defines LcGAN(G, D) as Ex,y[log D(x, y)] + Ex,z[log(1 - D(x, G(x, z)))]; the discriminator “tries to classify if each N x N patch in an
image is real or fake” (Section 2.2.2). This teaches the discriminator scoring real and
generated images toward 1 and 0 respectively, the two terms of Eq. (1) being the corresponding losses.).
Claim 10.
Anantatamukala and Isola teaches wherein the sliced true image is the sliced original image added with a mark (ANANTATAMUKALA: "Acquired images have been manually annotated for the grain boundaries using GIMP (GNU Image Manipulation Program) [34] software to prepare the ground truth/target images for ML training and validation. Boundaries are marked in green color with a 3 pixel width lines" (Section 3, p. 3). This teaches a ground-truth image formed by marking boundaries onto the acquired image.).
Claims 17 and 18.
Anantatamukala and Isola teaches a computer readable recording media and an electronic apparatus comprising a processor and a storage element storing a computer program that executes the method of claim 1 (ISOLA: “Code is available at https://github.com/phillipi/pix2pix” (Section 1); training took “less than two hours of
training on a single Pascal Titan X GPU" (Section 3). This teaches a stored program executed by a processor.).
Claim 11 is rejected under 35 U.S.C. 103 as being unpatentable over Anantatamukala
in view of Isola, and further in view of Park et al., US 10,217,205 B2 (hereinafter “PARK”).
Claim 11.
The method for detecting the image boundary using the generative adversarial network model of claim 1, further comprising: detecting an image boundary of a picture to be detected using the updated generative adversarial network model (ANANTATAMUKALA: “The BSE image to
be segmented (for grain boundaries) is input to the input layer of the ML model, which
generates the required 'grain boundary map' as output, by employing the learned weights
of the trained ML model of generator network” (Section 2, p. 3). This teaches applying the
trained generator to an unseen image to produce its boundary map.), wherein the picture to be detected is a
Anantatamukala and Isola do not specifically teach that the picture to be detected is a
transmission electron microscope image. However, Park teaches grain boundary analysis performed on transmission electron microscope images (PARK: “Provided are a method and
system for analyzing grains using a high-resolution transmission electron microscopy (HRTEM) image. The method relates to analyzing nanometer grains, and includes receiving an HRTEM image” (Abstract). This teaches a transmission electron microscope
image as the input for grain boundary analysis.).
Therefore, it would have been obvious to one of ordinary skill in the art to apply the grain boundary segmentation of Anantatamukala and Isola to the transmission electron microscope images of Park before the effective filing date of the claimed invention. The motivation would have been to resolve grain boundaries at the nanometer scale that Park identifies as requiring transmission electron microscopy, while avoiding the manual and threshold-based analysis those images otherwise demand.
Allowable Subject Matter
Claims 5-9 and 12-16 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Ross Varndell whose telephone number is (571)270-1922. The examiner can normally be reached M-F, 9-5 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, O’Neal Mistry can be reached at (313)446-4912. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see https://ppair-my.uspto.gov/pair/PrivatePair. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Ross Varndell/Primary Examiner, Art Unit 2674
Prosecution Insights