DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claims 1-12 are pending in the application.
Priority
The present application is a 371 of PCT/JP2022/009573 filed on 03/04/2022.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 08/28/2024 and 05/28/2026 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Objections
The following claims are objected to.
Claim 4 (page 7) the clause “use the target image group as the candidate image group” has been recited twice.
Claim 10 2nd-3rd line “using” has been recited twice.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-12 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 1 9th line “the image” has no antecedent basis and claim 1 12th line “the image” has no antecedent basis. Note claim 1 recites “an input image” and “a similar image … within a candidate image group”. Further the specification states that adversarial attack can be applied to an input image (query attack), or to an image within a candidate image group (candidate attack) (PGPub para. [0036]). Therefore it is not clear to which image the adversarial attack is applied. For prior art consideration, Examiner considers the following limitations:
--- count the rank of the similar image with respect to the input image in the candidate image group in a case where adversarial perturbation is applied to the input image or an image within the candidate image group;.
--- calculate the rank of the similar image with respect to the input image in the candidate image group in a case where adversarial perturbation is not applied to the input image or an image within the candidate image group.
Similar treatment is applied to claims 2 and 7 in which “the image” is recited.
Similar issue exists in independent claims 11 and 12 and similar consideration is applied. Specifically, claim 11 (page 8) last line “the image” has no antecedent basis and claim 11 (page 9) 2nd-3rd line “the image” has no antecedent basis. Claim 12 7th line “the image” has no antecedent basis and claim 12 9th-10th line “the image” has no antecedent basis.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-12 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Analysis for claim 1 is provided in the following. Claim 1 is reproduced in the following (annotation added):
1. A robustness verification device comprising:
at least one memory configured to store instructions; and
at least one processor configured to execute the instructions to:
(a) use the similarity between feature amounts obtained by extracting a feature amount to identify a similar image having a predetermined rank of similarity with respect to an input image within a candidate image group;
(b) count the rank of the similar image with respect to the input image in the candidate image group in a case where adversarial perturbation is applied to the image;
(c) calculate the rank of the similar image with respect to the input image in the candidate image group in a case where adversarial perturbation is not applied to the image; and
(d) verify whether or not the counted rank of the similar image is within a predetermined range that includes the calculated rank of the similar image.
Step 1: Evaluating whether the claim belongs to one of the statutory categories.
Claim 1 recites a device. Thus, the claim is directed to a machine, which is one of the statutory categories of invention (Step 1: YES).
Step 2A Prong One: Evaluating whether the claim recites a judicial exception (an abstract idea enumerated in the 2019 PEG, a law of nature, or a natural phenomenon). If no exception is recited, the claim is eligible. This concludes the eligibility analysis. If the claim recites an exception, go to Step 2A Prong Two.
Claim 1 steps (a), (b) and (d) can be practically performed in the human mind. These concepts fall into the “mental processes” group of abstract ideas, which is observation, evaluation and/or judgment. The limitations, interpreted under their broadest reasonable interpretation and in consistent with the specification, cover performance of the limitations in the mind or by generic computer components. Step (c) recites mathematical calculations. See MPEP 2106.04 and the 2019 PEG. (Step 2A Prong One YES)
Step 2A Prong Two: Evaluating whether the claim recites additional elements that integrate the exception into a practical application of the exception. This evaluation is performed by (a) identifying whether there are any additional elements recited in the claim beyond the judicial exception, and (b) evaluating those additional elements individually and in combination to determine whether the claim as a whole integrates the exception into a practical application. If the answer to (a) is YES and (b) is NO, go to Step 2B; if the answer to (a) and (b) is YES, go to PATHWAY B, i.e., the claim is not directed to a judicial exception and the claim is eligible.
In claim 1, “at least one memory” and “at least one processor” can be regarded as additional elements. These components are merely recited as tools to perform the mental processes and mathematical calculations, specified at a high level of generality, to the judicial exception, e.g., a claim to an abstract idea requiring no more than a generic computer to perform generic computer functions. These additional elements do not integrate the abstract idea into a practical application. (Step 2A Prong Two NO).
Step 2B: This part of the eligibility analysis evaluates whether the claim as a whole amounts to significantly more than the recited exception, i.e., whether any additional element, or combination of additional elements, adds an inventive concept to the claim.
In claim 1, “at least one memory” and “at least one processor” can be regarded as additional elements. Simply appending well-understood, routine, conventional activities previously known to the industry, specified at a high level of generality, to the judicial exception, e.g., a claim to an abstract idea requiring no more than a generic computer to perform generic computer functions that are well-understood, routine and conventional activities previously known to the industry, as discussed in Alice Corp., 573 U.S. at 225, 110 USPQ2d at 1984 (see MPEP § 2106.05(d) (Step 2B: NO). Claim 1 is not eligible.
Claim 2 recites (annotation added), “calculate the upper limit and lower limit of the similarity between the feature amounts obtained by extracting the feature amount between the input image and the images of the candidate image group in a case where adversarial perturbation is applied to the image, and count the rank of the similar image with respect to the input image in the candidate image group in a case where adversarial perturbation is applied to the image, using the calculated upper limit and the calculated lower limit”. The “calculate” act recites mathematical calculations. The “count” act recites mental processes. Claim 2 is not eligible.
Claim 3 recites, “wherein the adversarial perturbation is applied to the input image”. These additional elements just provide extra information on which image the adversarial perturbation is applied to. These additional elements do not integrate the abstract idea recited in claims and 2 into a practical application. Claim 3 is not eligible.
Claim 4 recites (annotation added), [T]he robustness verification device according to claim 3, wherein the at least one processor is configured to execute the instructions to: include the similar image and an image in which the similarity between feature amounts obtained by extracting the feature amount with the similar image in an image of the candidate image group is equal to or greater than a predetermined value in a target image group, use the target image group as the candidate image group”. The “include” and “use” acts recite mental processes. Claim 4 is not eligible
Claim 5 recites, [T]he robustness verification device according to claim 2, wherein the adversarial perturbation is applied to one or more images in the candidate image group. These additional elements just provide extra information on which image the adversarial perturbation is applied to. These additional elements do not integrate the abstract idea recited in claims 1 and 2 into a practical application. Claim 5 is not eligible.
Claim 6 recites (annotation added), [T]he robustness verification device according to claim 5, wherein the at least one processor is configured to execute the instructions to: calculate the rank of the similar image from the candidate image group to which the adversarial perturbation is applied; and make the rank of the similar image the predetermined rank. The “calculate” act recites mathematical calculations. The “make” act recites mental processes. Claim 6 is not eligible.
Claim 7 recites, [T]he robustness verification device according to claim 2, wherein the image is one in which the magnitude of the adversarial perturbation is equal to or less than a predetermined value in the infinity norm. These additional elements just provide extra information on the adversarial perturbation. These additional elements do not integrate the abstract idea recited in claims 1 and 2 into a practical application. Claim 7 is not eligible.
Claim 8 recites (annotation added), [T]he robustness verification device according to claim 7, wherein the at least one processor is configured to execute the instructions to calculate the upper limit and the lower limit on the basis of an upper limit and lower limit of each element of the feature amounts obtained by extracting the feature amount an image to which adversarial perturbation is applied. Claim 8 recites additional mathematical calculations. Claim 8 is not eligible.
Claim 9 recites (annotation added), [T]he robustness verification device according to claim 1, wherein the extracting the feature amount includes extracting a feature amount using a deep learning model. Under broadest reasonable interpretation, extracting feature amount can be practically perform in human mind. The element “using deep learning model” is recited in high level of generality such that it amounts to using a computer with a generic deep learning model to apply the abstract idea. Claim 9 is not eligible.
Claim 10 recites (annotation added), [T]he robustness verification device according to claim 1, wherein the extracting the feature amount includes extracting a feature amount using Deep Metric Learning. Under broadest reasonable interpretation, extracting feature amount can be practically perform in human mind. The element “using Deep Metric Learning” is recited in high level of generality such that it amounts to using a computer with a generic deep learning model to apply the abstract idea. Claim 10 is not eligible.
Claim 11, an independent method claim, recites similar steps as recited in claim 1 without additional elements. Therefore claim 11 merely recites mental processes and mathematical calculations. Claim 11 is not eligible.
Claim 12, an independent medium claim, recites similar steps as recited in claim 1. Claim 12 further recites “a computer”. Similar analysis applied to claim 1 regarding additional elements is applicable here. Claim 12 is not eligible.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1 and 9-12 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Zhou et al. (Zhou M, Wang L, Niu Z, Zhang Q, Zheng N, Hua G. Adversarial Attack and Defense in Deep Ranking. arXiv preprint arXiv:2106.03614. 2021 Jun 7. Hereafter Zhou).
As per claim 1, Zhou teaches a robustness verification device (see below) comprising:
at least one memory configured to store instructions (The limitation of a memory storing instructions is inherently taught when a computer-implemented method is disclosed. See Zhou page 7-10 section 6 “EXPERIMENTS”); and
at least one processor (page 7 left col. last para. “We conduct experiments with Nvidia RTX3090 GPUs and Intel Xeon 6226R CPU”) configured to execute the instructions to:
use the similarity between feature amounts obtained by extracting a feature amount to identify a similar image having a predetermined rank of similarity with respect to an input image within a candidate image group (FIG. 1; page 2 left col. 2nd para. “a typical deep ranking model maps samples; i.e., queries and candidates) to a common embedding space, where the distances among them determine the final ranking order”; Here “a common embedding space” refers to a common feature space, and “the distance” refers to the similarity between a query image and a candidate image within a candidate image group; page 3 left col. 1st para. following section 3 ADVERSARIAL RANKING ATTACK “Given the query q and candidate set X = {c1,c2,...,cn}, deep ranking aims to learn a mapping f, which is usually implemented by a DNN to map all candidates and query into a common embedding space, such that the relative distances among the embedding vectors could satisfy the expected ranking order”; page 2 left col. 2nd para. “Therefore, the essential of adversarial ranking attack is to find a proper perturbation, which could push the sample to a desired position that leads to the expected ranking order. Specifically, we first represent the expected ranking order as a set of inequalities. Subsequently, a triplet-like objective function is designed according to those inequalities, and combined with Projected Gradient Descent (PGD) to efficiently obtain the desired adversarial perturbation”; Here “a desired position that leads to the expected ranking order” means a predetermined rank of similarity);
count the rank of the similar image with respect to the input image in the candidate image group in a case where adversarial perturbation is applied to the image (FIG. 1 1st col. CA+, CA-, QA+ and QA- representing various types of adversarial perturbations are applied to the image; right side “Ranking List” representing the rank of the similar image with respect to the input/query image is counted);
calculate the rank of the similar image with respect to the input image in the candidate image group in a case where adversarial perturbation is not applied to the image (FIG. 1 1st col. of 1st row “Original” means adversarial perturbation is not applied to the image, right side “Ranking List” representing the rank of the similar image with respect to the input/query image is calculated); and
verify whether or not the counted rank of the similar image is within a predetermined range that includes the calculated rank of the similar image (page 7 left col. section 6 EXPERIMENTS 1st para. “To validate the proposed attacks and defenses, and evaluate the ranking models with ERS, we use five ranking datasets …”; See the following para. from page 7 right col.; See also Table 1 and page 8-9 section 6.2 Candidate Attack & Query Attack).
PNG
media_image1.png
308
876
media_image1.png
Greyscale
As per claim 9, dependent upon claim 1, Zhou teaches wherein the extracting the feature amount includes extracting a feature amount using a deep learning model (Abstract; page 1 “Index Terms—Deep Ranking, Deep Metric Learning …”).
As per claim 10, dependent upon claim 1, Zhou teaches wherein the extracting the feature amount includes extracting a feature amount using using Deep Metric Learning (Abstract; page 1 “Index Terms—Deep Ranking, Deep Metric Learning … “; page 2 right col. first para. following section 2 RELATED WORKS “Deep Ranking is generally formularized as the deep metric learning (DML) problem”).
Claim 11, an independent method claim, recites steps corresponding to apparatus claim 1. Therefore the recited steps of claim 11 are mapped to Zhou in the same manner as corresponding steps in claim 1.
Claim 12, an independent medium claim, recites steps corresponding to apparatus claim 1. Therefore the recited steps of claim 12 are mapped to Zhou in the same manner as corresponding steps in claim 1. Claim 12 further recites a medium and a computer. These elements are taught by Zhou (page 7 left col. last para. “We conduct experiments with Nvidia RTX3090 GPUs and Intel Xeon 6226R CPU”). The limitation of a medium for storing program is inherently taught when a computer-implemented method is disclosed (Zhou page 7-10 section 6 “EXPERIMENTS”).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 2-8 are rejected under 35 U.S.C. 103 as being unpatentable over Zhou, as applied above to claim 1, in view of Gowal et al. (Gowal S, Dvijotham K, Stanforth R, Bunel R, Qin C, Uesato J, Arandjelovic R, Mann T, Kohli P. On the effectiveness of interval bound propagation for training verifiably robust models. arXiv preprint arXiv:1810.12715v4 [cs.LG] 29 Aug 2019, hereafter Gowal).
As per claim 2, Zhou teaches calculate the upper limit adversarial perturbation is applied to the image (See below para. from page 4 right col. 2nd para. following section 4.1 Defense with Embedding-Shifted Triplet; Note the distance represents the similarity between two images.); and
PNG
media_image2.png
433
861
media_image2.png
Greyscale
count the rank of the similar image with respect to the input image in the candidate image group in a case where adversarial perturbation is applied to the image, using the calculated upper limit
PNG
media_image3.png
195
864
media_image3.png
Greyscale
Zhou, however, does not teach calculating a lower limit.
Gowal in the same field of endeavor discloses a method of using interval bound propagation (IBP) to train large provably robust neural networks (Abstract). Specifically, Gowal calculates an upper bound, in which the upper bound represents the maximum difference between any pair of logits when the input can be perturbed within an
PNG
media_image4.png
37
42
media_image4.png
Greyscale
norm-bounded ball (page 2 left col. 2nd para.). Gowal further calculate a lower bound (see below para. from page 4 right col.
PNG
media_image5.png
162
649
media_image5.png
Greyscale
It would have been obvious for a person with ordinary skill in the art before the effective filing date of the claimed invention to modify Zhou to incorporate the teaching of Gowal to calculate a lower limit of the similarity between the feature amounts obtained by extracting the feature amount between the input image and the images of the candidate image group in a case where adversarial perturbation is applied to the image. The motivation for calculating a lower bound is that “the upper and lower bounds of the output logits zK can be used to construct an upper bound on the solution of (4)”, which is a constraint for a verification problem (see below para. from Gowal page 4).
PNG
media_image6.png
195
674
media_image6.png
Greyscale
As per claim 3, dependent upon claim 2, Zhou in view of Gowal teaches wherein the adversarial perturbation is applied to the input image (Zhou FIG. 1 1st col. and Table 1 QA+, QA-, where QA representing query/input image).
As per claim 4, dependent upon claim 3, Zhou in view of Gowal teaches wherein the at least one processor is configured to execute the instructions to: include the similar image and an image in which the similarity between feature amounts obtained by extracting the feature amount with the similar image in an image of the candidate image group is equal to or greater than a predetermined value in a target image group, use the target image group as the candidate image group, and use the target image group as the candidate image group (Zhou FIG. 1 “Ranking List”; See below para. from page 3 left col.; Note distance represents similarity.).
PNG
media_image7.png
399
627
media_image7.png
Greyscale
.
As per claim 5, dependent upon claim 2, Zhou in view of Gowal teaches wherein the adversarial perturbation is applied to one or more images in the candidate image group (Zhou FIG. 1 1st col. and Table 1 CA+, CA-, where CA representing candidate image).
As per claim 6, dependent upon claim 5, Zhou in view of Gowal teaches wherein the at least one processor is configured to execute the instructions to: calculate the rank of the similar image from the candidate image group to which the adversarial perturbation is applied; and make the rank of the similar image the predetermined rank (Zhou FIG. 1; page 2 left col. 2nd para. “Therefore, the essential of adversarial ranking attack is to find a proper perturbation, which could push the sample to a desired position that leads to the expected ranking order. Specifically, we first represent the expected ranking order as a set of inequalities. Subsequently, a triplet-like objective function is designed according to those inequalities, and combined with Projected Gradient Descent (PGD) to efficiently obtain the desired adversarial perturbation”).
As per claim 7, dependent upon claim 2, Zhou in view of Gowal teaches wherein the image is one in which the magnitude of the adversarial perturbation is equal to or less than a predetermined value in the infinity norm (See following para. from Zhou page 3 right col.).
PNG
media_image8.png
279
896
media_image8.png
Greyscale
As per claim 8, dependent upon claim 7, Zhou in view of Gowal teaches wherein the at least one processor is configured to execute the instructions to calculate the upper limit and the lower limit on the basis of an upper limit and lower limit of each element of the feature amounts obtained by extracting the feature amount in an image to which adversarial perturbation is applied (Gowal page 3 left col., page 4 left col, and page 4 right col. respectively as shown below).
PNG
media_image9.png
292
644
media_image9.png
Greyscale
PNG
media_image10.png
190
667
media_image10.png
Greyscale
PNG
media_image11.png
310
698
media_image11.png
Greyscale
Conclusion
Further reference GOWAL et al. (GOWAL et al., "Scalable Verified Training for Provably Robust Image Classification", 2019 IEEE/CVF International Conference on Computer Vision (ICCV), IEEE, 2019, pp. 4841-4850) discloses using interval bound propagation (IBP) to train large provably robust neural networks (Abstract). GOWAL et al. further discloses calculating an upper bound and a lower bound for logits similarity (page 4845 eq. (8) and eq. (10) and corresponding descriptions).
Contact
Any inquiry concerning this communication or earlier communications from the examiner should be directed to XUEMEI G CHEN whose telephone number is (571)270-3480. The examiner can normally be reached Monday-Friday 9am-6pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, John M Villecco can be reached at (571) 272-7319. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/XUEMEI G CHEN/Primary Examiner, Art Unit 2661
Prosecution Insights