DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are: “acquisition unit”, “computation unit”, “generator” in claims 1-10.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The following excerpt from the instant Application’s filed specification governs interpretation of “computation unit” and “generator”:
PNG
media_image1.png
1047
953
media_image1.png
Greyscale
Therefore, the computation unit and generator are interpreted to be integrated circuits executing programs steps for performing the claimed functions – or equivalents thereof.
The following excerpt from the instant Application’s filed specification, in addition to the except shown above, governs interpretation of “acquisition unit”:
PNG
media_image2.png
446
1266
media_image2.png
Greyscale
Therefore, the acquisition unit is interpreted to be circuitry for receiving image information from a remote device such as a server or equivalent.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 11 is rejected under 35 U.S.C. 102(a)(1) as being anticipated by M. Mokuwe, M. Burke and A. S. Bosman, "Black-Box Saliency Map Generation Using Bayesian Optimisation," 2020 International Joint Conference on Neural Networks (IJCNN), Glasgow, UK, 2020, pp. 1-8, doi: 10.1109/IJCNN48605.2020.9207343 (Mokuwe).
As per claim 11, Mokuwe teaches an information processing method causing a computer to execute the steps of:
acquiring first image information based on a first image (Mokuwe:
page 3, col 2, para 1: “Let X be an m × n input image given by a set {(u1, v1), ..., (um, vn)} of all possible pixel coordinate combinations, and let S = {s1, .., sl} be the set of variable window sizes of length l.”;
Fig. 1 (shown below): “input image”);
computing a difference between the first image and a second image in accordance with classification of an object based on the first image acquired in the acquiring step and the second image obtained by hiding the first image based on the first image information acquired in the acquiring step by a mask that is smaller than a size of the first image (Mokuwe:
Page 2, col 2, last para – page 3, col 1, para 2:
PNG
media_image3.png
695
911
media_image3.png
Greyscale
); and
generating, based on a difference in accordance with the classification computed in the computing step, a third image indicating a location where the difference occurs (Mokuwe:
Page 2, col 2, last para – page 3, col 1, para 2: “To evaluate how well f performs given X, we observe the relationship between the blanked image ˆX and the change in the model’s output, i.e. class probability, y = f(X) − f( ˆX ). The saliency map is formed by the y values, which are obtained by sequentially blanking the image per pixel location, and passing the blanked image ˆX to the model.”: See that class probability, y = f(X) − f( ˆX ) is used to generate the saliency map
Page 1: col 1, last para – col 2, para:
PNG
media_image4.png
1222
888
media_image4.png
Greyscale
PNG
media_image5.png
531
888
media_image5.png
Greyscale
Page 3, col 1, paras 5-6:
PNG
media_image6.png
1080
905
media_image6.png
Greyscale
PNG
media_image7.png
700
899
media_image7.png
Greyscale
Page 3: col 1, para 7 – col 2, para 1:
PNG
media_image8.png
1002
911
media_image8.png
Greyscale
Note that the occlusion window is the mask.
Fig. 3 and associated text: Page 5: col 1, para 2 – col 2, para 1 (shown below)).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-5, 9, 10, and 12 are rejected under 35 U.S.C. 103 as being unpatentable over Mokuwe in view of US 20210232865 A1 (Munoz Delgado).
As per claim 1, Mokuwe teaches an information processing apparatus comprising:
(Mokuwe:
page 3, col 2, para 1: “Let X be an m × n input image given by a set {(u1, v1), ..., (um, vn)} of all possible pixel coordinate combinations, and let S = {s1, .., sl} be the set of variable window sizes of length l.”;
Fig. 1 (shown below): “input image”);
(Mokuwe:
Page 2, col 2, last para – page 3, col 1, para 2:
PNG
media_image3.png
695
911
media_image3.png
Greyscale
); and
(Mokuwe:
Page 2, col 2, last para – page 3, col 1, para 2: “To evaluate how well f performs given X, we observe the relationship between the blanked image ˆX and the change in the model’s output, i.e. class probability, y = f(X) − f( ˆX ). The saliency map is formed by the y values, which are obtained by sequentially blanking the image per pixel location, and passing the blanked image ˆX to the model.”: See that class probability, y = f(X) − f( ˆX ) is used to generate the saliency map
Page 1: col 1, last para – col 2, para:
PNG
media_image4.png
1222
888
media_image4.png
Greyscale
PNG
media_image5.png
531
888
media_image5.png
Greyscale
Page 3, col 1, paras 5-6:
PNG
media_image6.png
1080
905
media_image6.png
Greyscale
PNG
media_image7.png
700
899
media_image7.png
Greyscale
Page 3: col 1, para 7 – col 2, para 1:
PNG
media_image8.png
1002
911
media_image8.png
Greyscale
Note that the occlusion window is the mask.
Fig. 3 and associated text: Page 5: col 1, para 2 – col 2, para 1 (shown below)).
Mokuwe is silent regarding an acquisition unit; a computation unit; and a generator.
Munoz Delgado teaches these limitations (Munoz Delgado:
Para 108: “the input image may be obtained from internal or external storage, via a server, downloaded from an external source, or obtained in any other way”;
Para 119: “The method(s) may be implemented on a computer as a computer implemented method, as dedicated hardware, or as a combination of both. As also illustrated in FIG. 6, instructions for the computer, e.g., executable code, may be stored on a computer readable medium 60”).
Thus, it would have been obvious for one of ordinary skill in the art, prior to filing, to implement the teachings of Munoz Delgado into Mokuwe since both Mokuwe and Munoz Delgado suggest a practical solution and field of endeavor of an explainable AI system that generates saliency maps indicating importance of different image locations for classification in general and Munoz Delgado additionally provides teachings that can be incorporated into Mokuwe in that the image data is retrieved from a server. One of ordinary skill would recognize the advantages of centralized access, data security, reliable backup, and access control. Also, Munoz Delgado additionally provides teachings that can be incorporated into Mokuwe in that the invention is implemented using a processor and software instructions as for the advantages of automation, increased accuracy, cost reduction, and enhanced efficiency. The teachings of Munoz Delgado can be incorporated into Mokuwe in that the image data is retrieved from a server and the invention is implemented using a processor and software instructions. Furthermore, one of ordinary skill in the art could have combined the elements as claimed by known methods and, in combination, each component functions the same as it does separately. One of ordinary skill in the art would have recognized that the results of the combination would be predictable.
As per claim 2, Mokuwe in view of Munoz Delgado teaches the information processing apparatus according to claim 1, wherein the computation unit computes a difference, in accordance with each of a plurality of second images, between the first image and each of the second images obtained by hiding the first image by a corresponding one of a plurality of different masks (Mokuwe: See arguments and citations offered in rejecting claim 1 above;
Page 3: col 1, para 7 – col 2, para 1 (shown above): “Let X be an m × n input image given by
a set {(u1, v1), ..., (um, vn)} of all possible pixel coordinate combinations, and let S = {s1, .., sl} be the set of variable window sizes of length l. We can fit g using a set of samples D1:i = {x1:i, y1:i}. The model”;
Page 3, col 2, para 2:
PNG
media_image9.png
330
888
media_image9.png
Greyscale
Plural images of different occlusion window sizes. Note that the process is iterative for each window variable size s_i. The GP model g is iteratively constructed for different occlusion windowed images
Page 5: col 1, para 2 – col 2, para 1:
PNG
media_image10.png
1014
892
media_image10.png
Greyscale
PNG
media_image11.png
800
1808
media_image11.png
Greyscale
).
As per claim 3, Mokuwe in view of Munoz Delgado teaches the information processing apparatus according to claim 1, wherein the computation unit is configured to:
compute a difference, in accordance with each of a plurality of second images, between the first image and each of the second images obtained by hiding different locations in the first image by a plurality of corresponding first masks each having a first size; and
compute a difference, in accordance with each of a plurality of second images, between the first image and each of the second images obtained by hiding different locations in the first image by a plurality of corresponding second masks each having a second size that is different from the first size, and
the generator generates a composite image of a third image related to the difference in accordance with the classification based on each of a plurality of the second images in accordance with a corresponding one of the first masks and a third image related to the difference in accordance with the classification based on each of a plurality of the second images in accordance with a corresponding one of the second masks (Mokuwe: See arguments and citations offered in rejecting claim 2 above:
Pae 1, col 2, para 2: “An important parameter for the exhaustive search is the size of the blanking
window. Multiple window sizes can be employed to improve the quality of the saliency map, however, every additional window size further reduces the computational efficiency of the approach”;
The GP model g is iteratively constructed for different occlusion windowed images.
Page 4, col 1, para 1:
PNG
media_image12.png
696
889
media_image12.png
Greyscale
Page 4, col 1, paras 2-4: Page 4, col 1: Algorithm 1:
PNG
media_image13.png
1412
889
media_image13.png
Greyscale
PNG
media_image14.png
571
887
media_image14.png
Greyscale
).
As per claim 4, Mokuwe in view of Munoz Delgado teaches the information processing apparatus according to claim 3, wherein the computation unit computes the difference based on a plurality of the first masks with a total number of which being odd and a plurality of the second masks with a total number of which being odd (Mokuwe: See arguments and citations offered in rejecting claim 3 above: Since the masks are iteratively updated and the saliency map is iteratively updated, with every other iteration, the number of masks will be odd.).
As per claim 5, Mokuwe in view of Munoz Delgado teaches the information processing apparatus according to claim 1, wherein the computation unit computes a difference between:
a first value obtained by entering the first image to a neural network having a learning model generated by learning the object in advance to be output for each classification of the object; and
a second value obtained by entering the second image to the neural network to be output for each classification of the object (Mokuwe: See arguments and citations offered in rejecting claim 1 above).
As per claim 9, Mokuwe in view of Munoz Delgado teaches the information processing apparatus according to claim 1, wherein the generator is configured to:
if a numerical value representing a difference computed by the computation unit is relatively great, presume that the object recorded in the first image, which is hidden by the mask, is relatively likely to correspond to classification of the difference computed by the computation unit; and
if a numerical value representing a difference computed by the computation unit is relatively small, presume that the object recorded in the first image, which is hidden by the mask, is relatively unlikely to correspond to the classification of the difference computed by the computation unit (Mokuwe: See arguments and citations offered in rejecting claim 1 above: page 1, col 1, para 2 (shown above): “The intuition behind the blanking operation is that for a given model f, an image X, and a partially blanked image ˆX, the model outputs f(X) and f( ˆX ) should vary significantly if an important feature of X was blanked in ˆX”;
Figures 1-3 (shown above);
: the saliency maps are heatmaps that show different colors representing increasing saliency values
PNG
media_image15.png
488
772
media_image15.png
Greyscale
PNG
media_image16.png
665
879
media_image16.png
Greyscale
).
As per claim 10, Mokuwe in view of Munoz Delgado teaches the information processing apparatus according to claim 9, wherein the generator is configured to:
if a numerical value representing a difference computed by the computation unit is relatively great, indicate a location where the difference occurs in a third image in a third mode; and
if a numerical value representing a difference computed by the computation unit is relatively small, indicate a location where the difference occurs in the third image in a fourth mode that is different from the third mode (Mokuwe: See arguments and citations offered in rejecting claim 9 above).
As per claim(s) 12, arguments made in rejecting claim(s) 1 are analogous. Mokuwe is silent regarding a non-transitory computer readable medium storing therein an information processing program causing a computer to. Munoz Delgado teaches these limitations (Munoz Delgado: See arguments and citations offered in rejecting claim 1 above).
Thus, it would have been obvious for one of ordinary skill in the art, prior to filing, to implement the teachings of Munoz Delgado into Mokuwe since both Mokuwe and Munoz Delgado suggest a practical solution and field of endeavor of an explainable AI system that generates saliency maps indicating importance of different image locations for classification in general and Munoz Delgado additionally provides teachings that can be incorporated into Mokuwe in that the invention involves “a non-transitory computer readable medium storing therein an information processing program causing a computer to” as to allow for more cost effective development, distribution, and maintenance. The teachings of Munoz Delgado can be incorporated into Mokuwe in that the invention involves “a non-transitory computer readable medium storing therein an information processing program causing a computer to”. Furthermore, one of ordinary skill in the art could have combined the elements as claimed by known methods and, in combination, each component functions the same as it does separately. One of ordinary skill in the art would have recognized that the results of the combination would be predictable.
Claim(s) 6 is rejected under 35 U.S.C. 103 as being unpatentable over Mokuwe in view of Munoz Delgado as applied to claim 5 above, and further in view of US 20210383262 A1 (Elen).
As per claim 6, Mokuwe in view of Munoz Delgado teaches the information processing apparatus according to claim 5, wherein the computation unit outputs the first and second values in accordance with a type of
Mokuwe in view of Munoz Delgado does not teach fundus disease and fundus diseases.
Elen a type of (Elen:
Para 14: “classifying the image (e.g. classification of an eye disease”;
Para 15: “For a certain application (e.g. detection of eye diseases by analyzing images of the eye), it can be determined which neural network architecture combined with which heat mapping technique (cf. explainability method) provides the best results (highest agreement score/metric)”;
Para 50: “detecting various eye diseases (e.g. age related macular degeneration and diabetic macular edema)”: both Age-Related Macular Degeneration (AMD) and Diabetic Macular Edema (DME) are fundus (retinal) diseases.
Para 51: “evaluating a performance of explainability methods used with artificial neural network models which are configured to classify images of retina of eyes of subjects, the method comprising operating one or more hardware processors to: receive a saliency map of a retina image generated by applying explainability method on a trained artificial neural network of a machine learning model”;
Para 61: “retina images (also called fundus images) … a retina/fundus image”;
Para 81, 86: fundus).
Thus, it would have been obvious for one of ordinary skill in the art, prior to filing, to implement the teachings of Elen into Mokuwe in view of Munoz Delgado since both Mokuwe in view of Munoz Delgado and Elen suggest a practical solution and field of endeavor of generating saliency maps for explaining the efficacy of a neural networks classifications and what image locations are major contributors in general and Elen additionally provides teachings that can be incorporated into Mokuwe in view of Munoz Delgado in that the classification is a type of fundus disease of a plurality of fundus diseases since “Detection of a disease at its early phase is often critical to the recovery of patients or to prevent the disease from advancing to more severe stages. For example, photographs of the body, e.g. the retina, may be analyzed to determine whether the subject has a medical condition, e.g. vision loss over time by analysis of the retina image. For example, continued proliferation of preventable eye diseases such as diabetic retinopathy, macular degeneration, and glaucoma, which can cause permanent vision loss over time if left undiagnosed, can be prevented” (Elen: para 2). The teachings of Elen can be incorporated into Mokuwe in view of Munoz Delgado in that the classification is a type of fundus disease of a plurality of fundus diseases. Furthermore, one of ordinary skill in the art could have combined the elements as claimed by known methods and, in combination, each component functions the same as it does separately. One of ordinary skill in the art would have recognized that the results of the combination would be predictable.
Allowable Subject Matter
Claims 7 and 8 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter: Limitations pertaining to “if a numerical value representing the difference computed by the computation unit indicates one of positive and negative values…
if a numerical value representing the difference computed by the computation unit indicates another of the positive and negative values”, in conjunction with other limitations present in independent claim 1 and claim 7, distinguish over the prior art.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Atiba Fitzpatrick whose telephone number is (571) 270-5255. The examiner can normally be reached on M-F 10:00am-6pm.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Bee can be reached on (571) 270-5183. The fax phone number for Atiba Fitzpatrick is (571) 270-6255.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
Atiba Fitzpatrick
/ATIBA O FITZPATRICK/
Primary Examiner, Art Unit 2677