DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Examiner’s Note
This second Non-Final action is issued upon further consideration, the prior office action is determined to be incomplete in that it did not fully address the whole requirements of the claimed limitation; therefore, this subsequent office action is issued. Applicant’s argument (regarding rejection under 35 U.S.C. § 103(a): (pgs. [10 – 11])), have been found persuasive with respect to the previously applied rejection.
Applicant's arguments filed 03/13/2026 (regarding rejection under 35 U.S.C. § 101: (pgs. [7 – 9])) have been fully considered but they are not persuasive.
Argument – 1: (page: 8) applicant contends: “At least for this reason the wording "identify one or more first features in the input image" and "identify one or more second features in the input image" does not recite an abstract idea in the claimed method. The Examiner further asserts that the feature "calculating a similarity measure between the first activation map and the second activation map" recites an abstract idea. Applicant disagrees and submits that calculating a similarity measure between two activation maps cannot be performed in the human mind and is not an abstract idea. According to 2A - Prong One, claim 1 does not recite any of the judicial exceptions such as mathematical relationships, formulas, or calculations, but rather a method of carrying out obtaining and labelling pertaining to first and second convolutional neural networks (CNNs),”
Regarding the above argument the Examiner respectfully notes that the rejected claims recite abstract idea such as: “identify one or more second features in the input image”. At a high level, identifying features from an image is abstract idea because it involves analyzing visual data to extract and recognize patterns (e.g., shapes, edges or color), which can be performed as a mental process. Under the broadest reasonable interpretation, this amounts to data analysis and information extraction, falling within the abstract idea category of mental process, (see MPEP 2106.04). In addition, "calculating a similarity measure between the first activation map and the second activation map" also recites an abstract idea. The claim recites the limitation at a high level of generalization, without specifying technical details, so it can be characterized as an abstract idea because it involves applying mathematical operations to compare numerical data. Under the broadest reasonable interpretation, this amounts to evaluating information using a mathematical relationship, which falls within the categories of mathematical relationship, under MPEP 2106.04.
Argument – 2: ((pgs. 8 – 9) Argument regarding Step 2A, Prong 2 and Step 2B): Applicants contends “Even if, for the sake of argument, claim 1 is considered to recite an abstract idea, Applicant submits that claim 1 is not directed to an abstract idea. Claim 1 is explicitly rooted in the technology of CNNs and image processing, requiring concrete, technical steps to be performed by a machine. Applicant submits at least that claim 1 recites additional elements that integrate the alleged abstract idea into a practical application…
… Even if, for the sake of argument, claim 1 is considered to be directed to an abstract idea, Applicant submits that claim 1 amounts to significantly more. As noted above, the claimed invention solves the problem of how to label filters in a CNN and thus includes the inventive concept of improving the labelling of filters in a CNN and provides an improvement in the field of CNNs and/or in the functioning of the computer implementing the method…”
Regarding the above argument, the Examiner respectfully disagrees with the Applicant’s assertion that claim 1 recites additional elements that integrate the abstract idea into a practical application or it amounts to significantly more. The recited steps of obtaining activation maps from first and second CNNs based on an input image merely constitutes data gathering and processing using generic neural network operation, which under the broadest reasonable interpretation, amounts to collecting and analyzing information to perform comparison or filter labeling. Such operation do not impose a meaningful limit on the abstract idea but instead represents insignificant extra-solution activity or mere instruction to apply the abstract idea using a computer, (see MPEP 2106.05(g)).
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim(s) 1 – 15 rejected under 35 U.S.C. 101 because the claimed invention is directed to a judicial exception (i.e. an abstract idea) without significantly more.
In step 1, of the 101-analysis set forth in the MPEP 2106, the examiner has determined that the following limitations recite a process that, under the broadest reasonable interpretation, falls within one or more statutory categories (processes).
In step 2A prong 1, of the 101-analysis set forth in MPEP 2106, the examiner has determined that the following limitations recite a process that, under broadest reasonable interpretation, covers a mental process but for the recitation of generic computer components:
Regarding claim 1:
identify one or more first features in the input image;
(i.e.: the broadest reasonable interpretation, the claim recites abstract idea: mental process: It involves making an observation to recognize or identify features and patterns in an image, which includes identifying edges, shapes or objects by sight. See (MPEP 2106.04)).
identify one or more second features in the input image;
(i.e.: the broadest reasonable interpretation, the claim recites abstract idea: mental process: It involves making an observation to recognize or identify features and patterns in an image, which includes identifying edges, shapes or objects by sight. See (MPEP 2106.04)).
calculating a similarity measure between the first activation map and the second activation map;
(i.e.: the broadest reasonable interpretation, the claim recites abstract idea: mathematical concept: It involves mathematical computing a similarity values based on two sets of numerical values (activation maps). See (MPEP 2106.04)).
labelling, when the similarity measure is equal to or above a threshold similarity, the filter of the second convolutional neural network with a label of the labelled filter of the first convolutional neural network.
(i.e.: the broadest reasonable interpretation, the claim recites abstract idea: mental process: It involves making a judgment based on the similarity measure and assigning label accordingly. See (MPEP 2106.04)).
If the claim limitations, under their broadest reasonable interpretation, covers performance of the limitations as a mental process, but for the recitation of generic computer components, then it falls within the mental process. Accordingly, the claim recites an abstract idea.
Step 2A Prong 2 of the 101-analysis, set forth in MPEP 2106, the examiner has determined that
the following additional elements do not integrate this judicial exception into a practical application:
As evaluated below:
• The preamble is deemed insufficient to transform the judicial exception to a patentable
invention to a patentable invention because the preamble generally links the use of a
judicial exception to a particular technological environment or field of use, see MPEP
2106.05(h).
obtaining, based on an input image, a first activation map of a labelled filter
(i.e.: deemed insufficient to transform the judicial exception to a patentable invention because the claim recites limitation directed to mere data gathering as deemed insufficient to transform the judicial exception because claimed elements are considered insignificant extra-solution activity, See MPEP (2106.05(g))).
of a first convolutional neural network, the first convolutional neural network being configured to
(i.e.: deemed insufficient to transform the judicial exception to a patentable invention because the claim recites limitation which does not amount to more than a recitation of the words "apply it" (or an equivalent), such as mere instructions to implement an abstract idea on a computer. See MPEP 2106.05(f)).
obtaining, based on the input image, a second activation map of a filter
(i.e.: deemed insufficient to transform the judicial exception to a patentable invention because the claim recites limitation directed to mere data gathering as deemed insufficient to transform the judicial exception because claimed elements are considered insignificant extra-solution activity, See MPEP (2106.05(g))).
of a second convolutional neural network, the second convolutional neural network being configured to
(i.e.: deemed insufficient to transform the judicial exception to a patentable invention because the claim recites limitation which does not amount to more than a recitation of the words "apply it" (or an equivalent), such as mere instructions to implement an abstract idea on a computer. See MPEP 2106.05(f)).
the first convolutional neural network being different from the second convolutional neural network;
(i.e.: deemed insufficient to transform the judicial exception to a patentable invention because the claim recites limitation simply links the judicial exception to a field of use and/or technology environment, see MPEP 2106.05(h)).
In Step 2B of the 101-analysis set forth in the 2019 PEG, the examiner has determined that the
claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception:
Regarding limitation (I and III), additional elements considered extra/post solution activity, as analyzed above, are activity that are well-understood routine and conventional, specifically: the courts have recognized the computer functions as well‐understood, routine, and conventional functions.
Receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 (utilizing an intermediary computer to forward information); TL| Communications LLC v. AV Auto. LLC, 823 F.3d 607, 610, 118 USPQ2d 1744, 1745 (Fed. Cir. 2016) (using a telephone for image transmission); OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015) (sending messages over a network); buySAFE, Inc. v. Google, Inc., 765 F.3d 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014) (computer receives and sends information over a network). See MPEP 2106.05(d)(II).
Regarding limitation (II and IV), recite mere application of the abstract idea or mere instructions to implement an abstract idea on a computer are deemed insufficient to transform the judicial exception to a patentable invention because the limitations generally apply the use of a generic computer and/or process with the judicial exception, see MPEP 2106.05(f).
Regarding limitation (V), additional elements are deemed insufficient to transform the judicial exception to a patentable invention to a patentable invention because they generally link the judicial exception to the technology environment, see MPEP 2106.05(h).
As analyzed above, the additional elements, analyzed above, do not integrate the noted judicial exception into a practical application because they do not impose any meaningful limits on practicing the abstract idea. Therefore, the claim is directed to an abstract idea.
Regarding claim 2, dependent upon claim 1, and fail to resolve the deficiencies identified above by
integrating the judicial exception into a practical application, or introducing significantly more than the judicial exception. The claim recites:
wherein the calculating of the similarity measure comprises converting the first activation map and the second activation map into first and second binary matrices, respectively, and calculating the similarity measure between the first and second binary matrices.
(i.e.: the broadest reasonable interpretation, the claim recites abstract idea: mathematical concept: It involves converting numerical activation maps into binary metrices and then computing a similarity metrics between these matrices. See (MPEP 2106.04)).
Regarding claim 3, dependent upon claim 2, and fail to resolve the deficiencies identified above by
integrating the judicial exception into a practical application, or introducing significantly more than the judicial exception. The claim recites:
further comprising, before the calculating of the similarity measure between the first and second binary matrices, scaling the first and second binary matrices to dimensions of the input image, optionally using nearest neighbors interpolation.
(i.e.: the broadest reasonable interpretation, the claim recites abstract idea: mathematical concept: It involves scaling binary matrices to new dimensions, including using interpolation techniques such as nearest neighbors. See (MPEP 2106.04)).
Regarding claim 4, dependent upon claim 2, and fail to resolve the deficiencies identified above by
integrating the judicial exception into a practical application, or introducing significantly more than the judicial exception. The claim recites:
wherein the converting of the first activation map and the second activation map into first and second binary matrices comprises: setting each activation value which has an absolute value above a threshold value in the first activation map and the second activation map to a first value, and setting each activation value which has an absolute value equal to or below the threshold value to a second value.
(i.e.: the broadest reasonable interpretation, the claim recites abstract idea: mental process: It involves evaluating activation values and deciding whether to assign them to a first value or a second value based on a comparison to a threshold. See (MPEP 2106.04)).
Regarding claim 5, dependent upon claim 2, and fail to resolve the deficiencies identified above by
integrating the judicial exception into a practical application, or introducing significantly more than the judicial exception. The claim recites:
wherein the calculating of the similarity measure comprises calculating an intersection-over-union (IoU) metric between the first and second binary matrices.
The recitation in the additional limitation simply links the judicial exception to a field of use and/or technology environment, see MPEP 2106.05(h).
Limitations directed to field of use cannot integrate a judicial exception into a practical application at Step 2A or provide an inventive concept in Step 2B.
Regarding claim 6, dependent upon claim 1, and fail to resolve the deficiencies identified above by
integrating the judicial exception into a practical application, or introducing significantly more than the judicial exception. The claim recites:
wherein the calculating of the similarity measure comprises calculating a cosine distance metric between the first activation map and the second activation map.
(i.e.: the broadest reasonable interpretation, the claim recites abstract idea: mathematical concept: It involves computing a cosine distance metric between two activation maps. See (MPEP 2106.04)).
Regarding claim 7, dependent upon claim 1, and fail to resolve the deficiencies identified above by
integrating the judicial exception into a practical application, or introducing significantly more than the judicial exception. The claim recites:
further comprising using the second convolutional neural network in control of an autonomous or semi-autonomous vehicle.
The recitation in the additional limitation simply links the judicial exception to a field of use and/or technology environment, see MPEP 2106.05(h).
Limitations directed to field of use cannot integrate a judicial exception into a practical application at Step 2A or provide an inventive concept in Step 2B.
Regarding claim 8, dependent upon claim 1, and fail to resolve the deficiencies identified above by
integrating the judicial exception into a practical application, or introducing significantly more than the judicial exception. The claim recites:
re-training the first convolutional neural network to provide the second convolutional neural network.
Deemed insufficient to transform the judicial exception to a patentable invention because the limitation is directed to mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea and are considered to adding the words “apply it” (or an equivalent) with the judicial exception, See MPEP 2106.05(f).
Limitations directed to using the computer as a tool for implementing an abstract idea cannot integrate a judicial exception into a practical application at Step 2A or provide an inventive concept in Step 2B.
Regarding claim 9, dependent upon claim 1, and fail to resolve the deficiencies identified above by
integrating the judicial exception into a practical application, or introducing significantly more than the judicial exception. The claim recites:
obtaining, based on the input image, a plurality of first activation maps including the first activation map of a plurality of labelled filters
The recitation in the additional limitation directed to mere data gathering as deemed insufficient to transform the judicial exception because claimed elements are considered insignificant extra-solution activity and well-understood routine and conventional (2106.05(d)).
Receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 (utilizing an intermediary computer to forward information); TL| Communications LLC v. AV Auto. LLC, 823 F.3d 607, 610, 118 USPQ2d 1744, 1745 (Fed. Cir. 2016) (using a telephone for image transmission); OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015) (sending messages over a network); buySAFE, Inc. v. Google, Inc., 765 F.3d 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014) (computer receives and sends information over a network). See MPEP 2106.05(d)(II).
The additional limitations as analyze failed to integrate a judicial exception into a practical application at Step 2A and provide an inventive concept in Step 2B, per the analysis above.
of the first convolutional neural network;
Deemed insufficient to transform the judicial exception to a patentable invention because the limitation is directed to mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea and are considered to adding the words “apply it” (or an equivalent) with the judicial exception, See MPEP 2106.05(f).
Limitations directed to using the computer as a tool for implementing an abstract idea cannot integrate a judicial exception into a practical application at Step 2A or provide an inventive concept in Step 2B.
calculating a similarity measure for each of a plurality of pairs each comprising the second activation map and one of the plurality of first activation maps; and
(i.e.: the broadest reasonable interpretation, the claim recites abstract idea: mathematical concept: It involves mathematical computing a similarity values based on two sets of numerical values (activation maps). See (MPEP 2106.04)).
labelling the filter of the second convolutional neural network with a label of the labelled filter corresponding to a first activation map belonging to the pair with a highest similarity measure.
(i.e.: the broadest reasonable interpretation, the claim recites abstract idea: mental process: It involves comparing similarity measure and selecting the highest value, then assigning a label based on that comparison. See (MPEP 2106.04)).
Regarding claim 10, dependent upon claim 1, and fail to resolve the deficiencies identified above by
integrating the judicial exception into a practical application, or introducing significantly more than the judicial exception. The claim recites:
obtaining, based on the input image, a plurality of first activation maps including the first activation map of a plurality of labelled filters
The recitation in the additional limitation directed to mere data gathering as deemed insufficient to transform the judicial exception because claimed elements are considered insignificant extra-solution activity and well-understood routine and conventional (2106.05(d)).
Receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 (utilizing an intermediary computer to forward information); TL| Communications LLC v. AV Auto. LLC, 823 F.3d 607, 610, 118 USPQ2d 1744, 1745 (Fed. Cir. 2016) (using a telephone for image transmission); OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015) (sending messages over a network); buySAFE, Inc. v. Google, Inc., 765 F.3d 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014) (computer receives and sends information over a network). See MPEP 2106.05(d)(II).
The additional limitations as analyze failed to integrate a judicial exception into a practical application at Step 2A and provide an inventive concept in Step 2B, per the analysis above.
of the first convolutional neural network;
Deemed insufficient to transform the judicial exception to a patentable invention because the limitation is directed to mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea and are considered to adding the words “apply it” (or an equivalent) with the judicial exception, See MPEP 2106.05(f).
Limitations directed to using the computer as a tool for implementing an abstract idea cannot integrate a judicial exception into a practical application at Step 2A or provide an inventive concept in Step 2B.
selecting at least one first activation map each having an activation score above a threshold activation score or having a highest activation score;
(i.e.: the broadest reasonable interpretation, the claim recites abstract idea: mental process: It involves selecting activation maps based on having an activation score above a certain threshold value. See (MPEP 2106.04)).
calculating a similarity measure for each of a plurality of pairs each comprising the second activation map and one of the at least one selected first activation map; and
(i.e.: the broadest reasonable interpretation, the claim recites abstract idea: mathematical concept: It involves mathematical computing a similarity values based on two sets of numerical values (activation maps). See (MPEP 2106.04)).
labelling the filter of the second convolutional neural network with a label of the labelled filter corresponding to a first activation map belonging to the pair with a highest similarity measure.
(i.e.: the broadest reasonable interpretation, the claim recites abstract idea: mental process: It involves comparing similarity measure and selecting the highest value, then assigning a label based on that comparison. See (MPEP 2106.04)).
Regarding claim 11, dependent upon claim 1, and fail to resolve the deficiencies identified above by
integrating the judicial exception into a practical application, or introducing significantly more than the judicial exception. The claim recites:
wherein for each of a plurality of input images in which the input image is included: obtaining, based on the input image, a plurality of first activation maps of a plurality of labelled filters
The recitation in the additional limitation directed to mere data gathering as deemed insufficient to transform the judicial exception because claimed elements are considered insignificant extra-solution activity and well-understood routine and conventional (2106.05(d)).
Receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 (utilizing an intermediary computer to forward information); TL| Communications LLC v. AV Auto. LLC, 823 F.3d 607, 610, 118 USPQ2d 1744, 1745 (Fed. Cir. 2016) (using a telephone for image transmission); OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015) (sending messages over a network); buySAFE, Inc. v. Google, Inc., 765 F.3d 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014) (computer receives and sends information over a network). See MPEP 2106.05(d)(II).
The additional limitations as analyze failed to integrate a judicial exception into a practical application at Step 2A and provide an inventive concept in Step 2B, per the analysis above.
of the first convolutional neural network
Deemed insufficient to transform the judicial exception to a patentable invention because the limitation is directed to mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea and are considered to adding the words “apply it” (or an equivalent) with the judicial exception, See MPEP 2106.05(f).
Limitations directed to using the computer as a tool for implementing an abstract idea cannot integrate a judicial exception into a practical application at Step 2A or provide an inventive concept in Step 2B.
obtaining, based on the input image, the second activation map of the filter
The recitation in the additional limitation directed to mere data gathering as deemed insufficient to transform the judicial exception because claimed elements are considered insignificant extra-solution activity and well-understood routine and conventional (2106.05(d)).
Receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 (utilizing an intermediary computer to forward information); TL| Communications LLC v. AV Auto. LLC, 823 F.3d 607, 610, 118 USPQ2d 1744, 1745 (Fed. Cir. 2016) (using a telephone for image transmission); OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015) (sending messages over a network); buySAFE, Inc. v. Google, Inc., 765 F.3d 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014) (computer receives and sends information over a network). See MPEP 2106.05(d)(II).
The additional limitations as analyze failed to integrate a judicial exception into a practical application at Step 2A and provide an inventive concept in Step 2B, per the analysis above.
of the second convolutional neural network
Deemed insufficient to transform the judicial exception to a patentable invention because the limitation is directed to mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea and are considered to adding the words “apply it” (or an equivalent) with the judicial exception, See MPEP 2106.05(f).
Limitations directed to using the computer as a tool for implementing an abstract idea cannot integrate a judicial exception into a practical application at Step 2A or provide an inventive concept in Step 2B.
for each of a plurality of pairs each comprising the second activation map and one of the plurality of first activation maps, calculating a similarity measure between the first activation map and the second activation map
(i.e.: the broadest reasonable interpretation, the claim recites abstract idea: mathematical concept: It involves computing similarity values between pairs of activation maps, which is a mathematical comparison operation performed across pairs of data (e.g.: correlation, distance, dot-product based similarity). See (MPEP 2106.04)).
wherein the computer-implemented method further comprises: labelling the filter of the second convolutional neural network with a label of the labelled filter corresponding to the first activation map belonging to the pair having a highest similarity measure among the pairs;
(i.e.: the broadest reasonable interpretation, the claim recites abstract idea: mental process: It involves comparing similarity measure and selecting the highest value, then assigning a label based on that comparison. See (MPEP 2106.04)).
selecting a label of the labelled filter corresponding to the first activation map belonging to each of at least one pair having the highest similarity measure among the pairs for each of the plurality of images, and when one label has been selected, labelling the filter of the second convolutional neural network with the selected label, and when a plurality of labels have been selected, labelling the filter of the second convolutional neural network with the label appearing most frequently among the selected plurality of labels or label the filter of the second convolutional neural network with a label selected at random from a plurality of labels appearing the most frequently among the selected plurality of labels;
(i.e.: the broadest reasonable interpretation, the claim recites abstract idea: mental process: It involves comparing similarity values to determine which label to select and then choosing either the most frequent occurring label or randomly selecting among tied labels. See (MPEP 2106.04)).
selecting a label of the labelled filter corresponding to the first activation map belonging to the or each pair having a said similarity measure above or equal to a threshold similarity, and when one label has been selected, labelling the filter of the second convolutional neural network with the selected label, and when a plurality of labels have been selected, labelling the filter of the second convolutional neural network with the label appearing most frequently among the selected plurality of labels or label the filter of the second convolutional neural network with a label selected at random from a plurality of labels appearing the most frequently among the selected plurality of labels.
(i.e.: the broadest reasonable interpretation, the claim recites abstract idea: mental process: It involves evaluating whether similarity values meets a threshold, selecting one or more labels based on the evaluation, and then choosing either the most frequently occurring label or randomly selecting among tied labels. See (MPEP 2106.04)).
Regarding claim 12, dependent upon claim 1, and fail to resolve the deficiencies identified above by
integrating the judicial exception into a practical application, or introducing significantly more than the judicial exception. The claim recites:
obtaining, based on a plurality of input images, a plurality of corresponding activation maps of a filter
The recitation in the additional limitation directed to mere data gathering as deemed insufficient to transform the judicial exception because claimed elements are considered insignificant extra-solution activity and well-understood routine and conventional (2106.05(d)).
Receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 (utilizing an intermediary computer to forward information); TL| Communications LLC v. AV Auto. LLC, 823 F.3d 607, 610, 118 USPQ2d 1744, 1745 (Fed. Cir. 2016) (using a telephone for image transmission); OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015) (sending messages over a network); buySAFE, Inc. v. Google, Inc., 765 F.3d 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014) (computer receives and sends information over a network). See MPEP 2106.05(d)(II).
The additional limitations as analyze failed to integrate a judicial exception into a practical application at Step 2A and provide an inventive concept in Step 2B, per the analysis above
of a second convolutional neural network, each activation map comprising activation values;
Deemed insufficient to transform the judicial exception to a patentable invention because the limitation is directed to mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea and are considered to adding the words “apply it” (or an equivalent) with the judicial exception, See MPEP 2106.05(f).
Limitations directed to using the computer as a tool for implementing an abstract idea cannot integrate a judicial exception into a practical application at Step 2A or provide an inventive concept in Step 2B.
for each input image, calculating an activation score as an aggregation of the activation values of the corresponding activation map and
(i.e.: the broadest reasonable interpretation, the claim recites abstract idea: mathematical concept: It involves calculating an activation score for each input image by combining activation values. See (MPEP 2106.04)).
selecting at least one input image having an activation score above a threshold activation score or having a highest activation score among the plurality of input images;
(i.e.: the broadest reasonable interpretation, the claim recites abstract idea: mental process: It involves selecting one or more images based on an activation score above a threshold score. See (MPEP 2106.04)).
using the at least one selected input image, implementing the method as claimed claim 1.
Deemed insufficient to transform the judicial exception to a patentable invention because the limitation is directed to mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea and are considered to adding the words “apply it” (or an equivalent) with the judicial exception, See MPEP 2106.05(f).
Limitations directed to using the computer as a tool for implementing an abstract idea cannot integrate a judicial exception into a practical application at Step 2A or provide an inventive concept in Step 2B.
Regarding claim 13, dependent upon claim 1, and fail to resolve the deficiencies identified above by
integrating the judicial exception into a practical application, or introducing significantly more than the judicial exception. The claim recites:
implementing the computer- implemented method as claimed in claim 1 for a plurality of filters of the second convolutional neural network.
Deemed insufficient to transform the judicial exception to a patentable invention because the limitation is directed to mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea and are considered to adding the words “apply it” (or an equivalent) with the judicial exception, See MPEP 2106.05(f).
Limitations directed to using the computer as a tool for implementing an abstract idea cannot integrate a judicial exception into a practical application at Step 2A or provide an inventive concept in Step 2B.
Regarding claim 14,
The rest of the limitations recite analogous subject matter as claim 1, so are rejected under similar rationale.
A non-transitory computer readable medium storing a program which, when run on a computer, causes the computer to carry out a method comprising:
Deemed insufficient to transform the judicial exception to a patentable invention because the limitation is directed to mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea and are considered to adding the words “apply it” (or an equivalent) with the judicial exception, See MPEP 2106.05(f).
Limitations directed to using the computer as a tool for implementing an abstract idea cannot integrate a judicial exception into a practical application at Step 2A or provide an inventive concept in Step 2B.
Regarding claim 15,
The rest of the limitations recite analogous subject matter as claim 1, so are rejected under similar rationale.
An information processing apparatus comprising: a memory, and a processor connected to the memory, wherein the processor is configured to:
Deemed insufficient to transform the judicial exception to a patentable invention because the limitation is directed to mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea and are considered to adding the words “apply it” (or an equivalent) with the judicial exception, See MPEP 2106.05(f).
Limitations directed to using the computer as a tool for implementing an abstract idea cannot integrate a judicial exception into a practical application at Step 2A or provide an inventive concept in Step 2B.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 9 – 15 are rejected under 35 U.S.C. 103 as being unpatentable over Zhang, et al., "Interpretable convolutional neural networks." in view of Frolova et al., Pub. No.: US20210081754A1, Nguyen et al., "Explaining convolutional neural networks by tagging filters" and Tang et al., Pub. No.: US20180129919A1.
Regarding claim 1, Zhang teaches: A computer-implemented method comprising:
obtaining, based on an input image, [ ], the first convolutional neural network being configured to identify one or more first features in the input image;
(Zhang, page 8829, “During the forward propagation, given each input image I [obtaining, based on an input image], the CNN computes a feature map x of the filter f after the ReLu operation [the first convolutional neural network being configured to identify one or more first features in the input image], where x is an n × n matrix, xij ≥ 0. Our method estimates the potential position of the object part on the feature map x as the neural unit with the strongest activation ˆµ=argmaxµ=[i,j]xij, 1≤i,j ≤n. Then, based on the estimated part position ˆµ, the CNN assigns a specific mask with x to filter out noisy activations.”)
a first activation map of a labelled filter of a first convolutional neural network
(Zhang, page 8829, “Our interpretable filters selectively model the most distinct parts of each category to minimize the conditional entropy of the final classification given feature maps of a conv-layer. 2) Each filter represents a single part of an object [a first activation map of a labelled filter of a first convolutional neural network], which maximizes the mutual information between the input image and middle-layer feature maps (i.e. “forgetting” as much irrelevant information as possible).”)
obtaining, based on the input image, a second activation map of a filter of a second convolutional neural network, the second convolutional neural network being configured to identify one or more second features in the input image,
(Zhang, page 8829, “Section 3.1 for the determination of the category c for fil ter f. Let X = {x|x = f(I),I ∈ I} denote feature maps of f after an ReLU operation for different training images. Given an input image I [obtaining, based on the input image,], if I ∈ Ic, we expect the feature map x = f(I) to exclusively activated at the target part’s location [a second activation map of a filter of a second convolutional neural network, the second convolutional neural network being configured to identify one or more second features in the input image]; otherwise, the feature map keeps inactivated. In other words, if I ∈ Ic, the feature map x is expected to fit to the assigned template Tˆµ; if I ∈ Ic, we design a negative template T− and hope that the feature map x matches to T−”)
Zhang does not teach:
calculating a similarity measure between the first activation map and the second activation map; and
… when the similarity measure is equal to or above a threshold similarity …
labelling, [ ], the filter of the second convolutional neural network with a label of the labelled filter of the first convolutional neural network.
the first convolutional neural network being different from the second convolutional neural network;
Frolova teaches:
calculating a similarity measure between the first activation map and the second activation map; and
(Frolova, “[0095] For example, as described herein, in certain implementations, various criteria can be defined to reflect whether a computed similarity/correlation [calculating a similarity measure between the first activation map and the second activation map] reflects a result that is satisfactory (e.g., within an image recognition process). For example, a Pearson correlation coefficient (PCC) value of 0.6 can be defined as a threshold that reflects a satisfactory result (e.g., with respect to identifying content within input 130). In scenarios in which the comparison between corresponding activation maps results in a PCC value below the defined threshold, such an activation map can be identified as a candidate for modification in the CNN. Such a candidate for modification can reflect, for example, an occlusion that may affect various aspects of the processing/identification of input 130.”)
… when the similarity measure is equal to or above a threshold similarity…
(Frolova, “[0108] By way of further illustration, as shown in FIG. 2 and described herein, the respective activation maps of set 150A (corresponding to input 130) and set 150B (corresponding to reference image(s) 170) can be compared and a statistical correlation (as expressed in a similarity value) can be computed for each respective comparison. In the scenario depicted in FIG. 2, the similarity value for activation maps 152A, 152B and 152D (as compared with activation maps 152W, 152X, and 152Z, respectively, of set 150B) meets or exceeds one or more defined criteria (e.g., a PCC value threshold of 0.6) [… when the similarity measure is equal to or above a threshold similarity…]. Accordingly, such activation maps can be determined to sufficiently correlate with the referenced reference image(s) (e.g., in order to enable the identification of content within input 130 via CNN 140).”)
Frolova and Zhang are related to the same field of endeavor (i.e.: Convolutional Neural Network optimization). It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to combine the teaching of Frolova with teachings of Zhang to use activation map correlations across different images to perform error correction within CNN (Frolova, Abstract).
Zhang in view of Frolova do not teach:
labelling, [ ], the filter of the second convolutional neural network with a label of the labelled filter of the first convolutional neural network.
the first convolutional neural network being different from the second convolutional neural network;
Nguyen teaches:
labelling, [ ], the filter of the second convolutional neural network with a label of the labelled filter of the first convolutional neural network.
(Nguyen, page: 2, “Filter Tagging. We tag the filters [labelling, [ ], the filter of the second convolutional neural network] according to their corresponding values received in 𝑧𝑐 (𝑚,𝑖) with the label of the input image class [with a label of the labelled filter of the first convolutional neural network]. We are interested in the feature maps with high activations of a certain class because they indicate important features associated with that class [4]. We define two methods to select those feature maps per class and per layer (because of the mentioned complexity in different layers): (i) 𝑘-best-method (choose the 𝑘 feature maps with highest activation values) and (ii) 𝑞-quantile-method (choose the 𝑞-quantile of feature maps with highest activation values).”)
Nguyen, Zhang and Frolova are related to the same field of endeavor (i.e.: Convolutional Neural Network optimization). It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to combine the teaching of Nguyen with teachings of Zhang and Frolova to introduce the idea of tagging filters based on class-specific activations, to allow the understanding of which features contributes to a classification and why error occurred (Nguyen, Abstract).
Zhang in view of Frolova and Nguyen do not teach:
the first convolutional neural network being different from the second convolutional neural network;
Tang teaches:
the first convolutional neural network being different from the second convolutional neural network;
(Tang, “[0012] In an aspect, disclosed is a method for generating a semantic image labeling model, comprising: forming a first CNN and a second CNN, respectively [the first convolutional neural network being different from the second convolutional neural network]; randomly initializing the first CNN; inputting a raw image and predetermined label ground truth annotations to the first CNN to iteratively update weights of the first CNN so that category label probabilities output from the first CNN approaches the predetermined label ground truth annotations; randomly initializing the second CNN; ...”)
Tang, Zhang, Frolova and Nguyen are related to the same field of endeavor (i.e.: Convolutional Neural Network optimization). It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to combine the teaching of Tang with teachings of Zhang, Frolova and Nguyen to add a unified concatenated network that performs pixel wise semantic segmentation (Tang, Abstract).
Regarding claim 9, Zhang in view of Frolova, Nguyen and Tang teach the method of claim 1.
Frolova further teaches: obtaining, based on the input image, a plurality of first activation maps including the first activation map of a plurality of labelled filters of the first convolutional neural network;
(Frolova, “[0076] At operation 420, a first activation map/set [a plurality of first activation maps including the first activation map of a plurality of labelled filters of the first convolutional neural network] of activation maps is generated, e.g., with respect to the input/image(s) received at 410 [obtaining, based on the input image]. In certain implementations, such an activation map/set of activation maps can be generated within one or more layers of the convolutional neural network (e.g., convolutional layers, RELU layers, pooling layers, fully connected layers, normalization layers, etc.). In certain implementations, the described operations can generate a set or vector of activation maps for an image (reflecting activation maps that correspond to various portions, regions, or aspects of the image) [the first convolutional neural network being configured to identify one or more first features in the input image].”)
calculating a similarity measure for each of a plurality of pairs each comprising the second activation map and one of the plurality of first activation maps; and
(Frolova, “[0095] For example, as described herein, in certain implementations, various criteria can be defined to reflect whether a computed similarity/correlation [calculating a similarity measure for each of a plurality of pairs each comprising the second activation map and one of the plurality of first activation maps] reflects a result that is satisfactory (e.g., within an image recognition process). For example, a Pearson correlation coefficient (PCC) value of 0.6 can be defined as a threshold that reflects a satisfactory result (e.g., with respect to identifying content within input 130). In scenarios in which the comparison between corresponding activation maps results in a PCC value below the defined threshold, such an activation map can be identified as a candidate for modification in the CNN. Such a candidate for modification can reflect, for example, an occlusion that may affect various aspects of the processing/identification of input 130.”)
It would have been obvious to one of ordinary skill in the art before the effective filling date of the present application to combine the teachings of Frolova with the teachings of Zhang, Nguyen and Tang for the same reasons disclosed for claim 1.
Nguyen further teaches: labelling the filter of the second convolutional neural network with a label of the labelled filter corresponding to a first activation map belonging to the pair with a highest similarity measure.
(Nguyen, page: 2, “Filter Tagging. We tag the filters [labelling the filter of the second convolutional neural network with a label of the labelled filter corresponding to a first activation map belonging to the pair] according to their corresponding values received in 𝑧𝑐 (𝑚,𝑖) with the label of the input image class. We are interested in the feature maps with high activations of a certain class [with a highest similarity measure] because they indicate important features associated with that class [4]. We define two methods to select those feature maps per class and per layer (because of the mentioned complexity in different layers): (i) 𝑘-best-method (choose the 𝑘 feature maps with highest activation values) and (ii) 𝑞-quantile-method (choose the 𝑞-quantile of feature maps with highest activation values).”)
It would have been obvious to one of ordinary skill in the art before the effective filling date of the present application to combine the teachings of Nguyen with teachings of Zhang, Frolova and Tang for the same reasons disclosed for claim 1.
Regarding claim 10, Zhang in view of Frolova, Nguyen and Tang teach the method of claim 1.
Frolova further teaches: obtaining, based on the input image, a plurality of first activation maps including the first activation map of a plurality of labelled filters of the first convolutional neural network;
(Frolova, “[0076] At operation 420, a first activation map/set [a plurality of first activation maps including the first activation map of a plurality of labelled filters of the first convolutional neural network] of activation maps is generated, e.g., with respect to the input/image(s) received at 410 [obtaining, based on the input image]. In certain implementations, such an activation map/set of activation maps can be generated within one or more layers of the convolutional neural network (e.g., convolutional layers, RELU layers, pooling layers, fully connected layers, normalization layers, etc.). In certain implementations, the described operations can generate a set or vector of activation maps for an image (reflecting activation maps that correspond to various portions, regions, or aspects of the image) [the first convolutional neural network being configured to identify one or more first features in the input image].”)
selecting at least one first activation map each having an activation score above a threshold activation score or having a highest activation score;
(Frolova, “[0108] By way of further illustration, as shown in FIG. 2 and described herein, the respective activation maps of set 150A (corresponding to input 130) and set 150B (corresponding to reference image(s) 170) can be compared and a statistical correlation (as expressed in a similarity value) can be computed for each respective comparison. In the scenario depicted in FIG. 2, the similarity value for activation maps [selecting at least one first activation map] 152A, 152B and 152D (as compared with activation maps 152W, 152X, and 152Z, respectively, of set 150B) meets or exceeds one or more defined criteria (e.g., a PCC value threshold of 0.6) [each having an activation score above a threshold activation score or having a highest activation score]. Accordingly, such activation maps can be determined to sufficiently correlate with the referenced reference image(s) (e.g., in order to enable the identification of content within input 130 via CNN 140).”)
calculating a similarity measure for each of a plurality of pairs each comprising the second activation map and one of the at least one selected first activation map; and
(Frolova, “[0095] For example, as described herein, in certain implementations, various criteria can be defined to reflect whether a computed similarity/correlation [calculating a similarity measure for each of a plurality of pairs each comprising the second activation map and one of the at least one selected first activation map] reflects a result that is satisfactory (e.g., within an image recognition process). For example, a Pearson correlation coefficient (PCC) value of 0.6 can be defined as a threshold that reflects a satisfactory result (e.g., with respect to identifying content within input 130). In scenarios in which the comparison between corresponding activation maps results in a PCC value below the defined threshold, such an activation map can be identified as a candidate for modification in the CNN. Such a candidate for modification can reflect, for example, an occlusion that may affect various aspects of the processing/identification of input 130.”)
It would have been obvious to one of ordinary skill in the art before the effective filling date of the present application to combine the teachings of Frolova with the teachings of Zhang, Nguyen and Tang for the same reasons disclosed for claim 1.
Nguyen further teaches: labelling the filter of the second convolutional neural network with a label of the labelled filter corresponding to a first activation map belonging to the pair with a highest similarity measure.
(Nguyen, page: 2, “Filter Tagging. We tag the filters [labelling the filter of the second convolutional neural network with a label of the labelled filter corresponding to a first activation map belonging to the pair] according to their corresponding values received in 𝑧𝑐 (𝑚,𝑖) with the label of the input image class. We are interested in the feature maps with high activations of a certain class [with a highest similarity measure] because they indicate important features associated with that class [4]. We define two methods to select those feature maps per class and per layer (because of the mentioned complexity in different layers): (i) 𝑘-best-method (choose the 𝑘 feature maps with highest activation values) and (ii) 𝑞-quantile-method (choose the 𝑞-quantile of feature maps with highest activation values).”)
It would have been obvious to one of ordinary skill in the art before the effective filling date of the present application to combine the teachings of Nguyen with teachings of Zhang, Frolova and Tang for the same reasons disclosed for claim 1.
Regarding claim 11, Zhang in view of Frolova, Nguyen and Tang teach the method of claim 1.
Frolova further teaches: wherein for each of a plurality of input images in which the input image is included: obtaining, based on the input image, a plurality of first activation maps of a plurality of labelled filters of the first convolutional neural network,
(Frolova, “[0076] At operation 420, a first activation map/set [a plurality of first activation maps of a plurality of labelled filters of the first convolutional neural network] of activation maps is generated, e.g., with respect to the input/image(s) received at 410 [obtaining, based on the input image]. In certain implementations, such an activation map/set of activation maps can be generated within one or more layers of the convolutional neural network (e.g., convolutional layers, RELU layers, pooling layers, fully connected layers, normalization layers, etc.). In certain implementations, the described operations can generate a set or vector of activation maps for an image (reflecting activation maps that correspond to various portions, regions, or aspects of the image) [the first convolutional neural network being configured to identify one or more first features in the input image].”)
obtaining, based on the input image, the second activation map of the filter of the second convolutional neural network,
(Frolova, “[0093] At operation 450, one or more candidate(s) for modification is/are identified. In certain implementations, such candidate(s) can be identified based on a computed correlation (e.g., a statistical correlation). In certain implementations, such candidate(s) for modification can be identified based on a correlation computed between data reflected in the first set of activation maps (e.g., the activation maps generated at 420) and data reflected in a second set of activation maps associated with a second image (e.g., from the set of activation maps identified at 440). In certain implementations, such a correlation between each pair of activation maps [the second activation map of the filter of the second convolutional neural network,] can reflect a correlation between the set of activation maps generated with respect to the first image [obtaining, based on the input image] and a set of activation maps associate with the reference image(s).”)
for each of a plurality of pairs each comprising the second activation map and one of the plurality of first activation maps, calculating a similarity measure between the first activation map and the second activation map
(Frolova, “[0095] For example, as described herein, in certain implementations, various criteria can be defined to reflect whether a computed similarity/correlation [for each of a plurality of pairs each comprising the second activation map and one of the plurality of first activation maps, calculating a similarity measure between the first activation map and the second activation map] reflects a result that is satisfactory (e.g., within an image recognition process). For example, a Pearson correlation coefficient (PCC) value of 0.6 can be defined as a threshold that reflects a satisfactory result (e.g., with respect to identifying content within input 130). In scenarios in which the comparison between corresponding activation maps results in a PCC value below the defined threshold, such an activation map can be identified as a candidate for modification in the CNN. Such a candidate for modification can reflect, for example, an occlusion that may affect various aspects of the processing/identification of input 130.”)
It would have been obvious to one of ordinary skill in the art before the effective filling date of the present application to combine the teachings of Frolova with the teachings of Zhang, Nguyen and Tang for the same reasons disclosed for claim 1.
Nguyen further teaches: wherein the computer-implemented method further comprises: labelling the filter of the second convolutional neural network with a label of the labelled filter corresponding to the first activation map belonging to the pair having a highest similarity measure among the pairs; OR selecting a label of the labelled filter corresponding to the first activation map belonging to each of at least one pair having the highest similarity measure among the pairs for each of the plurality of images, and when one label has been selected, labelling the filter of the second convolutional neural network with the selected label, and when a plurality of labels have been selected, labelling the filter of the second convolutional neural network with the label appearing most frequently among the selected plurality of labels or label the filter of the second convolutional neural network with a label selected at random from a plurality of labels appearing the most frequently among the selected plurality of labels; OR
selecting a label of the labelled filter corresponding to the first activation map belonging to the or each pair having a said similarity measure above or equal to a threshold
similarity, and when one label has been selected, labelling the filter of the second convolutional neural network with the selected label, and when a plurality of labels have been selected, labelling the filter of the second convolutional neural network with the label appearing most frequently among the selected plurality of labels or label the filter of the second convolutional neural network with a label selected at random from a plurality of labels appearing the most frequently among the selected plurality of labels.
(Nguyen, page: 2, “Filter Tagging. We tag the filters [labelling the filter of the second convolutional neural network with a label of the labelled filter corresponding to the first activation map belonging to the pair] according to their corresponding values received in 𝑧𝑐 (𝑚,𝑖) with the label of the input image class. We are interested in the feature maps with high activations of a certain class [having a highest similarity measure among the pairs] because they indicate important features associated with that class [4]. We define two methods to select those feature maps per class and per layer (because of the mentioned complexity in different layers): (i) 𝑘-best-method (choose the 𝑘 feature maps with highest activation values) and (ii) 𝑞-quantile-method (choose the 𝑞-quantile of feature maps with highest activation values).”)
It would have been obvious to one of ordinary skill in the art before the effective filling date of the present application to combine the teachings of Nguyen with teachings of Zhang, Frolova and Tang for the same reasons disclosed for claim 1.
Regarding claim 12, Zhang in view of Frolova, Nguyen and Tang teach the method of claim 1.
Frolova further teaches: obtaining, based on a plurality of input images, a plurality of corresponding activation maps of a filter of a second convolutional neural network, each activation map comprising activation values;
(Frolova, “[0076] At operation 420, a first activation map/set [a plurality of corresponding activation maps of a filter of a second convolutional neural network, each activation map comprising activation values] of activation maps is generated, e.g., with respect to the input/image(s) received at 410 [obtaining, based on a plurality of input images]. In certain implementations, such an activation map/set of activation maps can be generated within one or more layers of the convolutional neural network (e.g., convolutional layers, RELU layers, pooling layers, fully connected layers, normalization layers, etc.). In certain implementations, the described operations can generate a set or vector of activation maps for an image (reflecting activation maps that correspond to various portions, regions, or aspects of the image) [the first convolutional neural network being configured to identify one or more first features in the input image].”)
selecting at least one input image having an activation score above a threshold activation score or having a highest activation score among the plurality of input images; and
(Frolova, “[0108] By way of further illustration, as shown in FIG. 2 and described herein, the respective activation maps of set 150A (corresponding to input 130) and set 150B (corresponding to reference image(s) 170) can be compared and a statistical correlation (as expressed in a similarity value) can be computed for each respective comparison. In the scenario depicted in FIG. 2, the similarity value for activation maps 152A, 152B and 152D (as compared with activation maps 152W, 152X, and 152Z, respectively, of set 150B) meets or exceeds one or more defined criteria (e.g., a PCC value threshold of 0.6) [each having an activation score above a threshold activation score or having a highest activation score]. Accordingly, such activation maps can be determined to sufficiently correlate with the referenced reference image(s) [selecting at least one input image] (e.g., in order to enable the identification of content within input 130 via CNN 140).”)
using the at least one selected input image, implementing the method as claimed claim 1.
(Frolova, “[0029] When analyzing/processing images within a convolutional neural network, challenges arise in scenarios in which such images contain occlusions or other defects that obscure portions of the content within the image [using the at least one selected input image, implementing the method as claimed claim 1]. For example, in scenarios in which image(s) being analyzed via a convolutional neural network correspond to human heads/faces (e.g., to identify the angle/direction the head of such a user is oriented), certain images may include occlusions that obscure portions of such a head/face.”)
It would have been obvious to one of ordinary skill in the art before the effective filling date of the present application to combine the teachings of Frolova with the teachings of Zhang, Nguyen and Tang for the same reasons disclosed for claim 1.
Nguyen further teaches: for each input image, calculating an activation score as an aggregation of the activation values of the corresponding activation map and
(Nguyen, page: 2, “For example, the first convolutional layers detect simple patterns such as lines and edges whereas the layers in the end detect compositional structures which match better to human-understandable objects [9]. Let 𝑎(𝑚,𝑖, 𝑑, 𝑗) be such a scaled activation in the 𝑗th element in the feature map calculated from image 𝑑 [for each input image] and filter 𝑖 ∈ 𝐼𝑚 in convolutional layer 𝑚. In order to get a total activation value per feature map [calculating an activation score as an aggregation of the activation values of the corresponding activation map], we define 𝑎¯(𝑚,𝑖, 𝑑) = 1 𝑛 Í𝑛 𝑗 𝑎(𝑚,𝑖, 𝑑, 𝑗), 0 ≤ 𝑎¯(𝑚,𝑖, 𝑑) ≤ 1, as the arithmetic mean of the scaled activations in a feature map where 𝑛 is the number of activations in the feature map. We do this for all filters𝑖 ∈ 𝐼𝑚 and repeat these steps for all layers 𝑚 ∈ 𝑀.”)
It would have been obvious to one of ordinary skill in the art before the effective filling date of the present application to combine the teachings of Nguyen with teachings of Zhang, Frolova and Tang for the same reasons disclosed for claim 1.
Regarding claim 13, Zhang in view of Frolova, Nguyen and Tang teach the method of claim 1.
Frolova further teaches: implementing the computer-implemented method as claimed in claim 1 for a plurality of filters of the second convolutional neural network.
(Frolova, “[0078] It should be understood that, in certain implementations, the number of activation maps in the referenced set can be defined by the structure of CNN 140 and/or layer(s) 142. For example, in a scenario in which a selected convolutional layer 142A of CNN 140 [of the second convolutional neural network] includes 64 filters [implementing the computer-implemented method as claimed in claim 1 for a plurality of filters], the referenced set will have 64 corresponding activation maps.”)
It would have been obvious to one of ordinary skill in the art before the effective filling date of the present application to combine the teachings of Frolova with the teachings of Zhang, Nguyen and Tang for the same reasons disclosed for claim 1.
Regarding claim 14, Frolova teaches: A non-transitory computer readable medium storing a program which, when run on a computer, causes the computer to carry out a method comprising:
(Frolova, “[0132] Aspects and implementations of the disclosure also relate to an apparatus for performing the operations herein. A computer program to activate or configure a computing device accordingly may be stored in a computer readable storage medium [A non-transitory computer readable medium storing a program which, when run on a computer], such as, but not limited to, any type of disk including floppy disks, optical disks, CD-ROMs, and magnetic-optical disks, read-only memories (ROMs), random access memories (RAMs), EPROMs, EEPROMs, magnetic or optical cards, or any type of media suitable for storing electronic instructions.”)
The rest of the limitation is analogous to claim 1, so are rejected under similar rationale.
Regarding claim 15, Frolova teaches: An information processing apparatus comprising: a memory, and a processor connected to the memory, wherein the processor is configured to:
(Frolova, “[0148] The memory/storage 530 can include a memory 532 [a memory], such as a main memory, or other memory storage, and a storage unit 536, both accessible to the processors 510 such as via the bus 502 [and a processor connected to the memory]. The storage unit 536 and memory 532 store the instructions 516 embodying any one or more of the methodologies or functions described herein.”)
The rest of the limitation is analogous to claim 1, so are rejected under similar rationale.
Claim 2 is rejected under 35 U.S.C. 103 as being unpatentable over Zhang in view of Frolova, Nguyen, Tang and in further view of Muddamsetty et al., "Sidu: Similarity difference and uniqueness method for explainable ai."
Zhang in view of Frolova, Nguyen and Tang teach the method of claim 1.
Zhang in view of Frolova, Nguyen and Tang do not teach:
wherein the calculating of the similarity measure comprises converting the first activation map and the second activation map into first and second binary matrices, respectively, and calculating the similarity measure between the first and second binary matrices.
Muddamsetty teaches:
wherein the calculating of the similarity measure comprises: converting the first activation map and the second activation map into first and second binary matrices, respectively, and
(Muddamsetty, page: 2 section: 2.1, “We first generate masks from the last convolution layers of the deep CNN model. Let us consider any deep CNN model F with last convolution layers of size n × n × N where 0n 0 is size of that layer and ‘N’ is the number of features in the activation maps f of class c, i.e., f c = [f c i , ....f c N ]. Each feature activation map f c i is then converted into a binary mask Mc i corresponding to the feature activation map f c i in the convolution layer [converting the first activation map and the second activation map into first and second binary matrices, respectively]. Next, a bi-linear interpolation is applied to up-sample the binary mask for a given input image I of size W × H. After interpolation, the binary mask Mc i will be no longer binary and the values range between [0, 1].”)
calculating the similarity measure between the first and second binary matrices.
(Muddamsetty, page: section: 2.3, “In summary, to explain the decision of the predicted class c visually, we first extract the last convolution layer from the deep CNN model F which has N number of features activation maps of size n × n. We then generate N binary masks and point wise multiplication is performed between each generated binary mask Mi and the input image I. The similarity difference SD between probability scores of the predicted class and each point-wise multiplied image mask Ai and uniqueness measure U between the image masks are computed [calculating the similarity measure between the first and second binary matrices] (i.e.: compute a similarity among masked binary matrices, each derived from a respective activation map). Weights Wi of each image mask Ai computed by the dot product of SD and U. Finally, the visual explanation Sc is a weighted sum of feature activation image masks Ai given in Eq. 5”)
Muddamsetty, Zhang, Frolova, Nguyen and Tang are related to the same field of endeavor (i.e.: Convolutional Neural Network optimization). It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to combine the teaching of Muddamsetty with teachings of Zhang, Frolova, Nguyen and Tang to enhance transparency, support validation by experts, and helps build trust in the model’s output. (Muddamsetty, Abstract).
Claim 3 is rejected under 35 U.S.C. 103 as being unpatentable over Zhang in view of Frolova, Nguyen, Tang and Muddamsetty.
Zhang in view of Frolova, Nguyen, Tang and Muddamsetty teach the method of claim 2.
Nguyen further teaches: further comprising, before the calculating of the similarity measure between the first and second binary matrices, scaling the first and second binary matrices to dimensions of the input image, optionally using nearest neighbors interpolation
(Nguyen, page: 2, “First, we collect the activations in the feature map to get the importance of the filters regarding an input image, i.e. the output in the feature map for a given filter (see terminology in Figure 2). Second, we scale these activations per layer between [0, 1]. In scaling the activations, [further comprising, before the calculating of the similarity measure between the first and second binary matrices, scaling the first and second binary matrices to dimensions of the input image] we ensure that no image is overrepresented with overall high activation values. We scale the activations per layer because each layer has its specific pattern compositionality of filters”)
It would have been obvious to one of ordinary skill in the art before the effective filling date of the present application to combine the teachings of Nguyen with teachings of Zhang, Frolova, Tang and Muddamsetty for the same reasons disclosed for claim 2.
Claim 4 is rejected under 35 U.S.C. 103 as being unpatentable over Zhang in view of Frolova, Nguyen, Tang, Muddamsetty and in further view of Junget al., Pub. No.: US20190347550A1.
Zhang in view of Frolova, Nguyen, Tang and Muddamsetty teach the method of claim 2.
Zhang in view of Frolova, Nguyen, Tang and Muddamsetty do not teach:
setting each activation value which has an absolute value above a threshold value in the first activation map and the second activation map to a first value, and setting each activation value which has an absolute value equal to or below the threshold value to a second value.
Junget teaches:
setting each activation value which has an absolute value above a threshold value in the first activation map and the second activation map to a first value, and setting each activation value which has an absolute value equal to or below the threshold value to a second value.
(Junget, “[0013] The activation quantization parameter may include a first median value and a first difference value with respect to the output activation map, wherein the first difference value indicates a half of a difference between a first threshold and a second threshold, and the first median value indicates a middle value of the first threshold and the second threshold, wherein the first threshold indicates an upper limit of an activation map section with respect to the output activation map [setting each activation value which has an absolute value above a threshold value in the first activation map and the second activation map to a first value], and the second threshold indicates a lower limit of the activation map section [setting each activation value which has an absolute value equal to or below the threshold value to a second value], and the weight quantization parameter may include a second median value and a second difference value of an absolute value of the weight of the current layer, wherein the second difference value indicates a half of a difference between a third threshold and a fourth threshold, and the second median value indicates a middle value of the third threshold and the fourth threshold, wherein the third threshold indicates an upper limit of a weight section with respect to the absolute value of the weight, and the fourth threshold indicates a lower limit of the weight section.”)
Junget, Zhang, Frolova, Nguyen, Tang and Muddamsetty are related to the same field of endeavor (i.e.: Convolutional Neural Network optimization). It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to combine the teaching of Junget with teachings of Zhang, Frolova, Nguyen, Tang and Muddamsetty to improve regulating activation magnitudes through quantization to make feature comparisons more consistent and robust. (Junget, Abstract).
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over Zhang in view of Frolova, Nguyen, Tang, Muddamsetty and in further view of Kortylewski et al., "Compositional convolutional neural networks: A robust and interpretable model for object recognition under occlusion."
Zhang in view of Frolova, Nguyen, Tang and Muddamsetty teach the method of claim 2.
Zhang in view of Frolova, Nguyen, Tang and Muddamsetty do not teach:
wherein the calculating of the similarity measure comprises: calculating an intersection-over-union (IoU) metric between the first and second binary matrices.
Kortylewski teaches:
wherein the calculating of the similarity measure comprises: calculating an intersection-over-union (IoU) metric between the first and second binary matrices.
(Kortylewski, page: 19, “6.4.2 Interpretation of vMF Kernels We further investigate the interpretability of our Compositional Nets using network dissection as proposed by [4]. In short, network dissection looks at the top activation of the hidden units and correlates them with a large range of human labeled visual concepts in the Broden dataset. Most of the concepts are annotated as segmentation mask with input resolution and the activation maps are up-scaled to the same size to calculate the intersection over union (IoU) scores [calculating an intersection-over-union (IoU) metric between the first and second binary matrices]. By setting a threshold for the best matched score, Network Dissection studies the latent representations of various layers in a network.”)
Kortylewski, Zhang, Frolova, Nguyen, Tang and Muddamsetty are related to the same field of endeavor (i.e.: Convolutional Neural Network optimization). It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to combine the teaching of Kortylewski with teachings of Zhang, Frolova, Nguyen, Tang and Muddamsetty to improve performance under challenging conditions but also makes the model more interpretable by showing how object parts contribute to classification. (Kortylewski, Abstract).
Claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over Zhang in view of Frolova, Nguyen, Tang and in further view of JINDAL et al., Pub. No.: US20220366188A1.
Zhang in view of Frolova, Nguyen and Tang teach the method of claim 1.
Zhang in view of Frolova, Nguyen and Tang do not teach:
wherein the calculating of the similarity measure comprises: calculating a cosine distance metric between the first activation map and the second activation map.
JINDAL teaches:
wherein the calculating of the similarity measure comprises: calculating a cosine distance metric between the first activation map and the second activation map.
(JINDAL, “[0038] In various aspects, for each token activation map generated by the encoder based on the given sentence, the parametric component can identify k nearest neighbors of that token activation map in the library of token activation maps. That is, for a particular token activation map generated by the encoder, the parametric component can compute a Euclidean distance between that particular token activation map and every other token activation map that is stored in the library [calculating a cosine distance metric between the first activation map and the second activation map]. The parametric component can thus rank the token activation maps that are stored in the library in order of ascending and/or descending Euclidean distance, and the k token activation maps in the library that correspond to the k smallest Euclidean distances can be selected as the k nearest neighbors. In various cases, the parametric component can repeat this Euclidean distance computation for each token activation map generated by the encoder based on the given sentence.”)
JINDAL, Zhang, Frolova, Nguyen and Tang are related to the same field of endeavor (i.e.: Convolutional Neural Network optimization). It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to combine the teaching of JINDAL with teachings of Zhang, Frolova, Nguyen and Tang to dynamically adapt parts of the CNN using learned contextual parameters to improve performance after initial training. (JINDAL, Abstract).
Claim 7 is rejected under 35 U.S.C. 103 as being unpatentable over Zhang in view of Frolova, Nguyen, Tang and in further view of YOU et al., Pub. No.: US20180275657A1.
Zhang in view of Frolova, Nguyen and Tang teach the method of claim 1.
Zhang in view of Frolova, Nguyen and Tang do not teach:
further comprising using the second convolutional neural network in control of an autonomous or semi- autonomous vehicle.
YOU teaches:
further comprising using the second convolutional neural network in control of an autonomous or semi- autonomous vehicle.
(YOU, “[0069] Hereinafter, an expert rule setting method, according to exemplary embodiments of the present disclosure, will be described with reference to FIG. 4. An autonomous vehicle control system, according to exemplary embodiments of the present disclosure, may set a deep learning architecture in S110, and may set a deep learning-based output control parameter in S120. Here, the setting of the deep learning architecture refers to setting the number of hidden layers, the number of convolutional neural networks (CNNs) [using the second convolutional neural network], and the like, and the output control parameter refers to an output control value for autonomous vehicle control [in control of an autonomous or semi- autonomous vehicle]. Then, the autonomous vehicle control system may set an expert rule selected by a user or provided by an expert in S130.”)
YOU, Zhang, Frolova, Nguyen and Tang are related to the same field of endeavor (i.e.: Convolutional Neural Network optimization). It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to combine the teaching of YOU with teachings of Zhang, Frolova, Nguyen and Tang to detect and correct CNN misclassification in autonomous driving control , improving reliability in safety critical setting. (YOU, Abstract).
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Zhang in view of Frolova, Nguyen, Tang and in further view of Li et al., Pub. No.: US20190095764A1.
Zhang in view of Frolova, Nguyen and Tang teach the method of claim 1.
Zhang in view of Frolova, Nguyen and Tang do not teach:
further comprising: re-training the first convolutional neural network to provide the second convolutional neural network.
Li teaches:
further comprising: re-training the first convolutional neural network to provide the second convolutional neural network.
(Li, “[0013] In one embodiment, transfer learning is employed to build new classifiers on top of pre-trained machine learning models, such as pre-trained convolutional neural networks (CNNs), by re-training classification layers of the pre-trained machine learning models [re-training the first convolutional neural network] using new training data while keeping feature detection layers of the pre-trained models fixed [to provide the second convolutional neural network]. ”)
Li, Zhang, Frolova, Nguyen and Tang are related to the same field of endeavor (i.e.: Convolutional Neural Network optimization). It would have been obvious to a person having ordinary skill in the art before the effective filing date of the invention to combine the teaching of Li with teachings of Zhang, Frolova, Nguyen and Tang by incorporating re-training of specific layers to improve stability and accuracy. (Li, Abstract).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Liang, et al. "Training interpretable convolutional neural networks by differentiating class-specific filters.", 2020.
Liang proposes a novel strategy to train interpretable CNNs by encouraging class-specific filters, among which each filter responds to only one (or few) class.
Zhou, et al. "Object detectors emerge in deep scene cnns.", 2014.
Zhou show that object detectors emerge from training CNNs to perform scene classification. As scenes are composed of objects, the CNN for scene classification automatically discovers meaningful objects detectors, representative of the learned scene categories.
Any inquiry concerning this communication or earlier communications from the examiner
should be directed to MATIYAS T MARU whose telephone number is (571)270-0902. The examiner
can normally be reached Monday 8:00am - Friday 4:00pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a
USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to
use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor,
Michelle Bechtold can be reached on (571) 431 – 0762. The fax phone number for the organization were this application or proceeding is assigned is 571 – 273 – 8300.
Information regarding the status of published or unpublished applications may be obtained from
Patent Center. Unpublished application information in Patent Center is available to registered users.
To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit
https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and
https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional
questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like
assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA)
or 571 -272 – 1000.
/M.T.M./ Examiner, Art Unit 2148
/Ryan Barrett/Primary Examiner, Art Unit 2148
Prosecution Insights