DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Claims
Claims 1-19 have been cancelled by Applicant. New claims 20-38 have been added. However, the numbering of claims is not in accordance with 37 CFR 1.126 which requires the original numbering of the claims to be preserved throughout the prosecution. When claims are canceled, the remaining claims must not be renumbered. When new claims are presented, they must be numbered consecutively beginning with the number next following the highest numbered claims previously presented (whether entered or not). It is noted that claim 20 had been previously cancelled.
Hence, misnumbered claims 20-38 been renumbered 21-39.
Response to Arguments
Claim Objections
Objection to claims 1, 3, 4, 5, 7, 10, 12, and 17 have been rendered moot. The deficiencies stated in the Non-Final Office Action dated 5/28/2025 have been corrected in the new set of claims.
Claim Rejections under 35 U.S.C. 112
The rejection of claims 1-19 under 35 U.S.C. 112(b) have been rendered moot in view of the new set of claims correcting the deficiencies previously stated in the Non-Final Office Action.
Claim Rejections under 35 U.S.C. 101
Claim Rejections under 35 U.S.C. 101 have been made in the instant Office Action with regards to new claims 21-33, 35-39.
Applicant broadly argues (in page 12 of applicant’s remarks filed on 8/28/2025) that “the claims reflect an improvement to a computer or to a technological field” as explained in paragraphs 0007-0008, 0034, and 0058-0060 of Applicant’s specification.
Examiner notes that these sections of the specification appear to be concerned with the manner of training and/or learning of the generation and discrimination models. However, independent claims 20, 37, and 38 do not contain any details of how the training or the learning of these models is occurring, as required by the MPEP. For example, new claim 20, merely recites “wherein the at least one processor is configured to train the Generative Adversarial Network, using the real data, the fake data, and the generated mix data”.
Independent claim 21 presently stands rejected under 35 U.S.C. 101 and 35 U.S.C. 103. With respect to the rejection of claim 21 under 35 U.S.C. 101, the limitation of “wherein the at least one processor is configured to train the Generative Adversarial Network, using the real data, the fake data, and the generated mix data” was analyzed under Step 2A, Prong 2 and Step 2B. In both analysis, the claim limitation was considered, individually and in combination with the claim as a whole, as an additional element recited at a high-level of generality such that it amounts to no more than mere instructions to apply the exception using generic computer components. (see MPEP 2106.05(f)).
As stated in MPEP 2106.05(a)(II) “to show that the involvement of a computer assists in improving the technology, the claims must recite the details regarding how a computer aids the method, the extent to which the computer aids the method, or the significance of a computer to the performance of the method. Merely adding generic computer components to perform the method is not sufficient. Thus, the claim must include more than mere instructions to perform the method on a generic component or machinery to qualify as an improvement to an existing technology. As noted above, these details are not recited in new claim 20.
In view of the above, claim 21 has been rejected under 35 U.S.C. 101. For at least the same reasons, analogous claims 37 and 38 and dependent claims 22-37 stand rejected under 35 U.S.C. 101. (See claim rejections under 35 U.S.C. 101 for the details about each claim rejection).
Claim Rejections under 35 U.S.C. 103
New claims 21-39 stand rejected under 35 U.S.C. 103.
Regarding the rejection of claim 21, under 35 U.S.C. 103, Applicant argues (in page 14 of applicant’s remarks filed 8/28/2025) that the combination of Wang in view of Chen and Navarrete does not teach the new limitation of “…wherein the at least one processor configured to change a mix ratio that is used to generate a data element constituting the mix data based on a position of the data element in the mix data…”.
Applicant’s arguments with respect to new claim 21 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 21-33, 35-39 are rejected under 35 U.S.C. 101 because the claimed invention is directed to a judicial exception (i.e., abstract idea) without significantly more.
Regarding claim 21,
Step 1: Claim 21 is directed towards an apparatus.
Step 2A, Prong 1: Claim 21 recites the following limitation:
generate mix data by mixing the real data and the fake data, wherein the at least one processor is configured to change the mix ratio that is used to generate a data element of the mix data based on a position of the data element in the mix data. (i.e., a person can mentally, or with the aid of pen and paper, generate a dataset that combines real data and fake data, and change the mix ratio based on a position of the data elements in the mixed data according to the mix data equation set forth in paragraph [0067] of Applicant’s specification.)
generate, using a Generative Adversarial Network, fake data that imitates the real data (i.e., a person can generate fake data that imitates real data via the mind)
Hence, the limitations recite an abstract idea.
Step 2A, Prong 2: The additional elements of at least one memory configured to store instructions; at least one processor configured to execute the instructions; “using a Generative Adversarial Network”; and “wherein the at least one processor is configured to train the Generative Adversarial Network using the real data, the fake data, and the generated mix data” are recited at a high-level of generality such that it amounts to no more than mere instructions to apply the exception using generic computer components (see MPEP 2106.05(f)).
Furthermore, the step of “obtain real data”, is considered mere data transmission which is a form of insignificant extra-solution activity. (see MPEP 2106.05(g)) Hence, the claim does not recite additional elements that integrate the judicial exception into a practical application. Since the claim as a whole, looking at the additional elements individually and in combination, does not contain any other additional elements that are indicative of integration into a practical application, the claim is directed to an abstract idea.
Step 2B: Claim 21 does not include additional elements that are sufficient to amount to significantly more than the judicial exception nor does it recite an inventive concept. As discussed above with respect to the integration of the abstract idea into a practical application, the additional elements of at least one memory configured to store instructions; at least one processor configured to execute the instructions; “using a Generative Adversarial Network”; and “wherein the at least one processor is configured to train the Generative Adversarial Network using the real data, the fake data, and the generated mix data” are recited at a high-level of generality such that it amounts to no more than mere instructions to apply the exception using generic computer components (see MPEP 2106.05(f)). Furthermore, the step of “obtain real data”, was considered under Step 2A, Prong 1 as mere data transmission which is a form of insignificant extra-solution activity, and thus it is re-evaluated under Step 2B to determine if it is more than what the courts have recognized to be well-understood, routine conventional activity in the field. The court decisions cited in MPEP 2106.05(d)(II) indicate that merely “receiving or transmitting data over a network” (e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 (utilizing an intermediary computer to forward information)) is a well-understood, routine, conventional activity when it is claimed in a merely generic manner (as it is in the present claim). Thereby, a conclusion that the claimed “obtain real data” step, is a well-understood, routine, conventional activity supported under Berkheimer. Hence, the claim lacks limitations which amount to significantly more than the judicial exception or an inventive concept, and is rejected. Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
Regarding claim 22,
Step 2A, Prong 1: Claim 22 recites an abstract idea as inherited from claim 20. Claim 22 further recites the following limitations:
wherein the at least one processor is configured to execute the instructions to change the mix ratio that is used to generate each of a plurality of data elements of the mix data in a continuous manner by using a function in which the position of the data element in the mix data is an argument (i.e., a person can mentally or with the aid of pen and paper change the mix ratio used to generate a plurality of data elements of the mix data by using a mathematical equation set forth in paragraph [0067] of Applicant’s specification.)
Hence, claim 22 further recites an abstract idea.
Step 2A, Prong 2: The additional element “wherein the at least one processor is configured to execute the instructions to” to execute the step recited above is recited at a high-level of generality such that it amounts to no more than mere instructions to apply the exception using generic computer components (see MPEP 2106.05(f)). Hence, the claim does not recite additional elements that integrate the judicial exception into a practical application. Since the claim as a whole, looking at the additional elements individually and in combination, does not contain any other additional elements that are indicative of integration into a practical application, the claim is directed to an abstract idea.
Step 2B: Claim 22 does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional element of “wherein the at least one processor is configured to execute the instructions to” stated above amounts to no more than mere instructions to apply the exception using generic computer components (see MPEP 2106.05(f)). Hence, the claim lacks limitations which amount to significantly more than the judicial exception or an inventive concept, and is rejected. Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
Regarding claim 23,
Step 2A, Prong 1: Claim 23 recites an abstract idea as inherited from claim 21. Claim 23 further recites the following limitations:
set the mix ratio that is used to generate a first data element of the mix data to be a first ratio; (i.e., a person can mentally and/or with the aid of pen and paper set a first ratio that combines different types of data such as real data and fake data)
set the mix ratio that is used to generate a second data element of the mix data that is different from the first data element to be a second ratio that is different from the first ratio; and (i.e., a person can mentally and/or with the aid of pen and paper set a second ratio that is different from the first that combines different types of data such as real data and fake data)
change the mix ratio that is used to generate each of a plurality of third data elements of the mix data, which is between the first and the second data elements, from the first ratio to the second ratio in a continuous manner based on the position of the third data element in the mix data. (i.e., a person can mentally or with the aid of pen and paper change the mix ratio in a continuous manner based on a position of a data element in the mix data)
Hence, the claim further recites an abstract idea.
Step 2A, Prong 2: The additional element of “wherein the at least one processor is configured to execute instructions to” is recited at a high-level of generality such that it amounts to no more than mere instructions to apply the judicial exception using generic computer components (see MPEP 2106.05(f)). Hence, the claim does not recite additional elements that integrate the judicial exception into a practical application. Since the claim as a whole, looking at the additional elements individually and in combination, does not contain any other additional elements that are indicative of integration into a practical application, the claim is directed to an abstract idea.
Step 2B: Claim 23 does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional element of “wherein the at least one processor is configured to execute instructions to” stated above amounts to no more than mere instructions to apply the exception using generic computer components (see MPEP 2106.05(f)). Hence, the claim lacks limitations which amount to significantly more than the judicial exception or an inventive concept, and is rejected. Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
Regarding claim 24:
Step 2A, Prong 1: Claim 24 recites an abstract idea as inherited from claim 21. Claim 24 further recites the following limitations:
wherein the at least one processor is configured to execute the instructions to change the mix ratio that is used to generate each of a plurality of data elements that are included in one data part of the mix data in a continuous manner by using a function in which the position of the data element in the mix data is an argument (i.e., a person can mentally or with the aid of pen and paper change a mix ratio of data in a continuous manner by applying a mathematical equation as set forth in paragraph [0067] of Applicant’s specification).
Hence, the claim further recites an abstract idea.
Step 2A, Prong 2: The additional element of “wherein the at least one processor is configured to execute the instructions to” is recited at a high-level of generality such that it amounts to no more than mere instructions to apply the judicial exception using generic computer components (see MPEP 2106.05(f)). Hence, the claim does not recite additional elements that integrate the judicial exception into a practical application. Since the claim as a whole, looking at the additional elements individually and in combination, does not contain any other additional elements that are indicative of integration into a practical application, the claim is directed to an abstract idea.
Step 2B: Claim 24 does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional element of “wherein the at least one processor is configured to execute the instructions to” stated above amounts to no more than mere instructions to apply the exception using generic computer components (see MPEP 2106.05(f)). Hence, the claim lacks limitations which amount to significantly more than the judicial exception or an inventive concept, and is rejected. Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
Regarding claim 25,
Step 2A, Prong 1: Claim 25 recites an abstract idea as inherited from claim 21. Claim 25 further recites the following limitations:
wherein the at least one processor is configured to execute the instructions to:
fix the mix ratio that is used to generate a plurality of data elements included in a first data part of the mix data to be a third ratio; (i.e., a person can mentally or with the aid of pen and paper keep fixed a ratio included in a data part of the mix data and designate it as a third ratio)
fix the mix ratio that is used to generate a plurality of data elements included in a second data part of the mix data that is different from the first data part to be a fourth ratio that is different from the third ratio; (i.e., a person can mentally or with the aid of pen and paper keep fixed a ratio included in a data part of the mix data and designate it as a different ratio than the third ratio)
change the mix ratio that is used to generate each of a plurality of data elements included in a third part of the mix data, which is between the first and second data parts, from the third ratio to the fourth ratio in a continuous manner based on the position of the data element in the mix data. (i.e., a person can mentally or with the aid of pen and paper change the mix ratio between the third and fourth ratio in a continuous manner based on a condition such as the position of the data element in the mix data)
Hence, the claim further recites an abstract idea.
Step 2A, Prong 2: The additional element of “wherein the at least one processor is configured to execute the instructions to” recited above is recited at a high-level of generality such that it amounts to no more than mere instructions to apply the judicial exception using generic computer components (see MPEP 2106.05(f)). Hence, the claim does not recite additional elements that integrate the judicial exception into a practical application. Since the claim as a whole, looking at the additional elements individually and in combination, does not contain any other additional elements that are indicative of integration into a practical application, the claim is directed to an abstract idea.
Step 2B: Claim 25 does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional element of “wherein the at least one processor is configured to execute the instructions to” stated above amounts to no more than mere instructions to apply the exception using generic computer components (see MPEP 2106.05(f)). Hence, the claim lacks limitations which amount to significantly more than the judicial exception or an inventive concept, and is rejected. Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
Regarding claim 26,
Step 2A, Prong 1: Claim 26 recites an abstract idea as inherited from claim 21. Claim 26 further recites the following limitations:
wherein the at least one processor is configured to execute the instructions to change, among multiple values, the mix ratio that is used to generate each of a plurality of data elements of the mix data by using a function in which the position of the data element in the mix data is an argument. (i.e., a person can mentally or with the aid of pen and paper change the mix ratio among multiple values by using a mathematical equation as set forth in paragraph [0067] of Applicant’s specification.)
Step 2A, Prong 2: The additional element of “wherein the at least one processor is configured to execute the instructions to” is recited at a high-level of generality such that it amounts to no more than mere instructions to apply the judicial exception using generic computer components (see MPEP 2106.05(f)). Hence, the claim does not recite additional elements that integrate the judicial exception into a practical application. Since the claim as a whole, looking at the additional elements individually and in combination, does not contain any other additional elements that are indicative of integration into a practical application, the claim is directed to an abstract idea.
Step 2B: Claim 26 does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional element of “wherein the at least one processor is configured to execute the instructions to” stated above amounts to no more than mere instructions to apply the exception using generic computer components (see MPEP 2106.05(f)). Hence, the claim lacks limitations which amount to significantly more than the judicial exception or an inventive concept, and is rejected. Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
Regarding claim 27,
Step 2A, Prong 1: Claim 27 recites an abstract idea as inherited form claim 21. Claim 27 further recites the following limitations:
set the mix ratio that is used to generate a first data element of the mix data to be a first ratio; (i.e., a person can mentally or with the aid of pen and paper keep set a mix ratio and designate it as a first ratio, said first ratio used to generate a first data element of the mix data)
set the mix ratio that is used to generate a second data element of the mix data that is different from the first data element to be a second ratio that is different from the first ratio; (i.e., a person can mentally or with the aid of pen and paper keep set a mix ratio and designate it a second ratio different from the first ratio – the second ratio used to generate a second data element in the mix data)
change, among multiple values from the first ratio to the second ratio, the mix ratio that is used to generate each of a plurality of third data elements of the mix data, which is between the first and the second data elements, based on the position of the third data element in the mix data. (i.e., a person can mentally or with the aid of pen and paper change the mix ratio among multiple values between the first and second ratio based on a condition such as the position of the third data element in the mix data – see mathematical equation set forth in paragraph [0067] of Applicant’s specification.)
Hence, the claim further recites an abstract idea.
Step 2A, Prong 2: The additional element of “wherein the at least one processor is configured to execute the instructions to” is recited at a high-level of generality such that it amounts to no more than mere instructions to apply the judicial exception using generic computer components (see MPEP 2106.05(f)). Hence, the claim does not recite additional elements that integrate the judicial exception into a practical application. Since the claim as a whole, looking at the additional elements individually and in combination, does not contain any other additional elements that are indicative of integration into a practical application, the claim is directed to an abstract idea.
Step 2B: Claim 27 does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional element of “wherein the at least one processor is configured to execute the instructions to” stated above amounts to no more than mere instructions to apply the exception using generic computer components (see MPEP 2106.05(f)). Hence, the claim lacks limitations which amount to significantly more than the judicial exception or an inventive concept, and is rejected. Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
Regarding claim 28,
Step 2A, Prong 1: Claim 28 recites an abstract idea as inherited from claim 21. Claim 28 further recites the following limitations:
change, among multiple values, the mix ratio that is used to generate each of a plurality of data elements that are included in one data part of the mix data by using a function in which the position of the data element in the mix data is an argument. (i.e., a person can mentally or with the aid of pen and paper change the mix ratio among multiple values by applying a mathematical equation as set forth in paragraph [0067] of Applicant’s specification.)
Step 2A, Prong 2: The additional element of “wherein the at least one processor is configured to execute the instructions to” is recited at a high-level of generality such that it amounts to no more than mere instructions to apply the judicial exception using generic computer components (see MPEP 2106.05(f)). Hence, the claim does not recite additional elements that integrate the judicial exception into a practical application. Since the claim as a whole, looking at the additional elements individually and in combination, does not contain any other additional elements that are indicative of integration into a practical application, the claim is directed to an abstract idea.
Step 2B: Claim 28 does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional element of “wherein the at least one processor is configured to execute the instructions to” stated above amounts to no more than mere instructions to apply the exception using generic computer components (see MPEP 2106.05(f)). Hence, the claim lacks limitations which amount to significantly more than the judicial exception or an inventive concept, and is rejected. Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
Regarding claim 29,
Step 2A, Prong 1: Claim 29 recites an abstract idea as inherited from claim 21. Claim 29 further recites the following limitations:
fix the mix ratio that is used to generate a plurality of data elements included in a first data part of the mix data to be a third ratio; (i.e., a person can mentally or with the aid of pen and paper keep fixed a ratio included in a data part of the mix data and designate it as a third ratio)
fix the mix ratio that is used to generate a plurality of data elements included in a second data part of the mix data that is different from the first data part to be a fourth ratio that is different from the third ratio; (i.e., a person can mentally or with the aid of pen and paper keep fixed a ratio included in a data part of the mix data and designate it as a fourth ratio)
change, among multiple values from the third ratio to the fourth ratio, the mix ratio that is used to generate each of a plurality of data elements included in a third part of the mix data, which is between the first and second data parts, based on the position of the data element in the mix data. (i.e., a person can mentally or with the aid of pen and paper change the mix ratio among multiple values between the third and fourth ratio based on a condition such as the position of the data element in the mix data – see equation set forth in paragraph [0067] of Applicant’s specification)
Step 2A, Prong 2: The additional element of “wherein the at least one processor is configured to execute the instructions to” is recited at a high-level of generality such that it amounts to no more than mere instructions to apply the judicial exception using generic computer components (see MPEP 2106.05(f)). Hence, the claim does not recite additional elements that integrate the judicial exception into a practical application. Since the claim as a whole, looking at the additional elements individually and in combination, does not contain any other additional elements that are indicative of integration into a practical application, the claim is directed to an abstract idea.
Step 2B: Claim 29 does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional element of “wherein the at least one processor is configured to execute the instructions to” stated above amounts to no more than mere instructions to apply the exception using generic computer components (see MPEP 2106.05(f)). Hence, the claim does not recite additional elements that amount to significantly more than the judicial exception nor does it recite an inventive concept. Therefore, the claim is rejected. Hence, the claim lacks limitations which amount to significantly more than the judicial exception or an inventive concept, and is rejected. Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
Regarding claim 30,
Step 2A, Prong 1: Claim 30 recites an abstract idea as inherited from claim 21. Claim 30 further recites the following limitations:
wherein the at least one processor is configured to execute the instructions to change the mix ratio so that the mix data includes a fourth data part in which the real data is dominant, a fifth data part in which the fake data is dominant and a sixth data part in which the real data and the fake data are balanced. (i.e., a person can mentally or with the aid of pen and paper change a mix ratio so that the real data is dominant and a part of the data in which the fake data is dominant and a different data part in which the real and the fake data are balanced – merely a way of changing the ratio among different partitions or subsets data which can be done mentally or the aid of pen and paper)
Step 2A, Prong 2: The additional element of “wherein the at least one processor is configured to execute the instructions to” is recited at a high-level of generality such that it amounts to no more than mere instructions to apply the judicial exception using generic computer components (see MPEP 2106.05(f)). Hence, the claim does not recite additional elements that integrate the judicial exception into a practical application. Since the claim as a whole, looking at the additional elements individually and in combination, does not contain any other additional elements that are indicative of integration into a practical application, the claim is directed to an abstract idea.
Step 2B: Claim 30 does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional element of “wherein the at least one processor is configured to execute the instructions to” stated above amounts to no more than mere instructions to apply the exception using generic computer components (see MPEP 2106.05(f)). Hence, the claim does not recite additional elements that amount to significantly more than the judicial exception nor does it recite an inventive concept, and the claim is rejected. Hence, the claim lacks limitations which amount to significantly more than the judicial exception or an inventive concept, and is rejected. Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
Regarding claim 31,
Step 2A, Prong 1: Claim 31 recites an abstract idea as inherited from claims 21 and 30. Claim 31 further recites the following limitations:
wherein the at least one processor is configured to execute the instructions to change the mix ratio so that the sixth data part is located between the fourth data part and the fifth data part. (i.e., a person can mentally or with the aid of pen and paper change a ratio so that the value is between certain subsets of data.)
Hence the claim further recites an abstract idea.
Step 2A, Prong 2: The additional element of “wherein the at least one processor is configured to execute the instructions to” is recited at a high-level of generality such that it amounts to no more than mere instructions to apply the exception using generic computer components. (See MPEP 2106.05(f)). Hence, the claim does not recite additional elements that integrate the judicial exception into a practical application. Since the claim as a whole, looking at the additional elements individually and in combination, does not contain any other additional elements that are indicative of integration into a practical application, the claim is directed to an abstract idea.
Step 2B: Claim 31 does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional element of “wherein the at least one processor is configured to execute the instructions to” stated above amounts to no more than mere instructions to apply the exception using generic computer components (see MPEP 2106.05(f)). Hence, the claim does not recite additional elements that amount to significantly more than the judicial exception nor does it recite an inventive concept, and the claim is rejected. Hence, the claim lacks limitations which amount to significantly more than the judicial exception or an inventive concept, and is rejected. Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
Regarding claim 32,
Step 2A, Prong 1: Claim 32 recites an abstract idea as inherited from claim 21. Claim 32 further recites the following limitations:
change the mix ratio based on a time at which the mix data is generated so that the mix ratio that is used to generate the mix data in a first period is different from the mix ratio that is used to generate the mix data in a second period that is different from the first period. (i.e., a person can mentally or with the aid of pen and paper graph on a time-domain axis a change of a ratio based on a condition such as the time that the mix data is generated so that the ratio that is used to generate the mix data in a first period is different from the ratio that is used to generate the mix data in a second period)
Hence, the claim further recites an abstract idea.
Step 2A, Prong 2: The additional element of “wherein the at least one processor is configured to execute the instructions to” is recited at a high-level of generality such that it amounts to no more than mere instructions to apply the exception using generic computer components. (See MPEP 2106.05(f)) Hence, the claim does not recite additional elements that integrate the judicial exception into a practical application. Since the claim as a whole, looking at the additional elements individually and in combination, does not contain any other additional elements that are indicative of integration into a practical application, the claim is directed to an abstract idea.
Step 2B: Claim 32 does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional element of “wherein the at least one processor is configured to execute the instructions to” stated above amounts to no more than mere instructions to apply the exception using generic computer components (see MPEP 2106.05(f)). Hence, the claim does not recite additional elements that amount to significantly more than the judicial exception nor does it recite an inventive concept, and the claim is rejected. Hence, the claim lacks limitations which amount to significantly more than the judicial exception or an inventive concept, and is rejected. Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
Regarding claim 33,
Step 2A, Prong 1: Claim 33 recites an abstract idea as inherited from claims 21 and 32. Claim 33 further recites the following limitations:
set the mix ratio in the first period so that a ratio of a fifth data part in which the fake data is dominant to the mix data is equal to or larger than a ratio of a fourth data part in which the real data is dominant to the mix data; and set the mix ratio in the second period so that a ratio of the fourth data part to the mix data in the second period is larger than a ratio of the fourth data part to the mix data in the first period. (i.e., a person can mentally or with the aid of pen and paper graph on a time-domain axis a ratio in a first period based on a condition such as that a ratio in a certain subset of data in which the fake data is dominant to the mix data is equal or larger than a ratio of another subset of data in which the real data is dominant to the mix data in a second period)
Hence, the claim recites an abstract idea.
Step 2A, Prong 2: The additional element of “wherein the at least one processor is configured to execute the instructions to” is a recited at a high-level of generality such that it amounts to no more than mere instructions to apply the exception using generic computer components. (See MPEP 2106.05(f)) Hence, the claim does not recite additional elements that integrate the judicial exception into a practical application. Since the claim as a whole, looking at the additional elements individually and in combination, does not contain any other additional elements that are indicative of integration into a practical application, the claim is directed to an abstract idea.
Step 2B: Claim 33 does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional element of “wherein the at least one processor is configured to execute the instructions to” stated above amounts to no more than mere instructions to apply the exception using generic computer components (see MPEP 2106.05(f)). Hence, the claim does not recite additional elements that amount to significantly more than the judicial exception nor does it recite an inventive concept, and the claim is rejected. Hence, the claim lacks limitations which amount to significantly more than the judicial exception or an inventive concept, and is rejected. Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
Regarding claim 35,
Step 2A, Prong 1: Claim 35 recites an abstract idea as inherited from claim 21.
Step 2A, Prong 2: Claim 35 recites the additional elements of “wherein: each of the real data, the fake data and the mix data is data relating to an image, the data element of the mix data includes a pixel of the image, the position of the data element in the mix data is a position of the pixel in the image.” This claim merely defines what the data comprises and what the position comprises – which is merely generally linking the use of the judicial exception to a particular technological environment or field of use – see MPEP 2106.05(h). Hence, the claim does not recite additional elements that integrate the judicial exception into a practical application. Since the claim as a whole, looking at the additional elements individually and in combination, does not contain any other additional elements that are indicative of integration into a practical application, the claim is directed to an abstract idea.
Step 2B: Claim 35 does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above, this claim elements of “wherein: each of the real data, the fake data and the mix data is data relating to an image, the data element of the mix data includes a pixel of the image, the position of the data element in the mix data is a position of the pixel in the image.” merely generally links the use of the judicial exception to a particular technological environment or field of use – see MPEP 2106.05(h). Hence, the claim does not recite additional elements that amount to significantly more than the judicial exception nor does it recite an inventive concept, and the claim is rejected. Hence, the claim lacks limitations which amount to significantly more than the judicial exception or an inventive concept, and is rejected. Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
Regarding claim 36,
Step 2A, Prong 1: Claim 36 recites an abstract idea as inherited from claim 21. Claim 36 further recites the following limitations:
change the mix ratio that is used to generate each of a plurality of data elements of the mix data in a discontinuous manner or a stepwise manner by using a function in which the position of the data element in the mix data is an argument. (i.e., a person can mentally or with the aid of pen and paper change the mix ratio in a discontinuous manner by using a mathematical function as claimed.)
Hence, the claim further recites an abstract idea.
Step 2A, Prong 2: The additional element of “wherein the at least one processor is configured to execute the instructions to” is recited at a high-level of generality such that it amounts to no more than mere instructions to apply the judicial exception using generic computer components (see MPEP 2106.05(f)). Hence, the claim does not recite additional elements that integrate the judicial exception into a practical application. Since the claim as a whole, looking at the additional elements individually and in combination, does not contain any other additional elements that are indicative of integration into a practical application, the claim is directed to an abstract idea.
Step 2B: Claim 36 does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional element of “wherein the at least one processor is configured to execute the instructions to” stated above amounts to no more than mere instructions to apply the exception using generic computer components (see MPEP 2106.05(f)). Hence, the claim does not recite additional elements that amount to significantly more than the judicial exception nor does it recite an inventive concept, and the claim is rejected. Hence, the claim lacks limitations which amount to significantly more than the judicial exception or an inventive concept, and is rejected. Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
Regarding claim 37,
Step 2A, Prong 1: Claim 37 recites an abstract idea as inherited from claim 21. Claim 37 further recites the following limitations:
change the mix ratio so that the mix ratio changes, on a line that connects a first data element to a second element of the mix data, (i) from a fifth ratio that allows a ratio of the real data to the fake data is 1:0 to a sixth ratio that allows the ratio of the real data to the fake data is 1:1 or (ii) from the sixth ratio to the fifth ratio, or (iii) from a seventh ratio that allows a ratio of the real data to the fake data is 0:1 to the sixth ratio or (iv) from the sixth ratio to the seventh ratio. (i.e., a person can mentally or with the aid of pen and paper change the mix ratio such that the ratio of real data to fake data is 1:0 or the ratio of real data to fake data is 1:1 or the ratio of real data to fake data is 0:1 – for example, this could be represented graphically as in on a line that connects a first data element to a second element of the mix data)
Hence, the claim further recites an abstract idea.
Step 2A, Prong 2: The additional element of “wherein the at least one processor is configured to execute the instructions to” is recited at a high-level of generality such that it amounts to no more than mere instructions to apply the judicial exception using generic computer components (see MPEP 2106.05(f)). Hence, the claim does not recite additional elements that integrate the judicial exception into a practical application. Since the claim as a whole, looking at the additional elements individually and in combination, does not contain any other additional elements that are indicative of integration into a practical application, the claim is directed to an abstract idea.
Step 2B: Claim 37 does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to the integration of the abstract idea into a practical application, the additional element of “wherein the at least one processor is configured to execute the instructions to” stated above amounts to no more than mere instructions to apply the exception using generic computer components (see MPEP 2106.05(f)). Hence, the claim does not recite additional elements that amount to significantly more than the judicial exception nor does it recite an inventive concept, and the claim is rejected. Hence, the claim lacks limitations which amount to significantly more than the judicial exception or an inventive concept, and is rejected. Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
Regarding claim 38,
Step 1: Claim 38 is directed towards an apparatus.
Step 2A, Prong 1: Claim 38 recites the following limitations:
generate, using a Generative Adversarial Network, fake data that imitates the real data; (i.e., a person can generate fake that imitates real data data via the mind)
generate mix data by mixing the real data and the fake data; (i.e., a person can, mentally or with the aid of pen and paper, generate a mix of data by mixing real data and fake data)
discriminate discrimination target data including the real data, the fake data and the mix data by using a discrimination model executed by the Generative Adversarial Network, (i.e., a person can, mentally or with the aid of pen and paper, discriminate discrimination targe data that includes real data, fake data and mixed data)
wherein the at least one processor is configured to change a mix ratio based on a time at which the mix data is generated so that the mix ratio that is used to generate the mix data in a first period that includes a period before a predetermined time elapses from a timepoint at which a training the discrimination model starts is different from the mix ratio that is used to generate the mix data in a second period that is different from the first period and that includes a period after the predetermined time elapses from the timepoint at which the training of the discrimination model starts, (i.e., a person can mentally or with the aid of pen and paper change a mix ratio according to temporal conditions as it is stated in the present limitation)
Hence the claim recites an abstract idea.
Step 2A, Prong 2: Claim 38 recites the additional elements of “at least one memory configured to store instructions”, “at least one processor configured to execute the instructions to”, “by using a discrimination model executed by the Generative Adversarial Network,”, “wherein the at least one processor is configured to train the discrimination model based on a discriminated result of the discrimination target data,”, “wherein the at least one processor is configured to train the Generative Adversarial Network using the real data, the fake data and the generated mix data” are recited at a high-level of generality such that it amounts to no more than mere instructions to apply the exception using generic computer components (see MPEP 2106.05(f)).
Furthermore, the step of “obtain real data”, is considered mere data transmission which is a form of insignificant extra-solution activity. (see MPEP 2106.05(g)) Hence, the claim does not recite additional elements that integrate the judicial exception into a practical application. Since the claim as a whole, looking at the additional elements individually and in combination, does not contain any other additional elements that are indicative of integration into a practical application, the claim is directed to an abstract idea.
Step 2B: Claim 38 does not include additional elements that are sufficient to amount to significantly more than the judicial exception nor does it recite an inventive concept. As discussed above with respect to the integration of the abstract idea into a practical application, the additional elements of “at least one memory configured to store instructions”, “at least one processor configured to execute the instructions to”, “by using a discrimination model executed by the Generative Adversarial Network,”, “wherein the at least one processor is configured to train the discrimination model based on a discriminated result of the discrimination target data,”, “wherein the at least one processor is configured to train the Generative Adversarial Network using the real data, the fake data and the generated mix data” are recited at a high-level of generality such that it amounts to no more than mere instructions to apply the exception using generic computer components (see MPEP 2106.05(f)). Furthermore, the step of “obtain real data”, was considered under Step 2A, Prong 1 as mere data transmission which is a form of insignificant extra-solution activity, and thus it is re-evaluated under Step 2B to determine if it is more than what the courts have recognized to be well-understood, routine conventional activity in the field. The court decisions cited in MPEP 2106.05(d)(II) indicate that merely “receiving or transmitting data over a network” (e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 (utilizing an intermediary computer to forward information)) is a well-understood, routine, conventional activity when it is claimed in a merely generic manner (as it is in the present claim). Thereby, a conclusion that the claimed “obtain real data” step, is a well-understood, routine, conventional activity supported under Berkheimer. Hence, the claim lacks limitations which amount to significantly more than the judicial exception or an inventive concept, and is rejected. Considering the additional elements individually and in combination, and the claim as a whole, the additional elements do not provide significantly more than the abstract idea. Therefore, the claim is not patent eligible.
Regarding claim 39,
Step 1: Claim 39 is directed towards a method.
This claim recites the same or analogous limitations as claim 21. Therefore the same rationale is applied to reject claim 39 as the rationale set forth in claim 21.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or non-obviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 21, 32, 34, 35, 38, and 39 are rejected under 35 U.S.C. 103 as being unpatentable over Wang et al. (US 20220415019 A1, filed Jan. 23, 2020, Published Dec. 29, 2022) in view of Chen et al.(US 20200320769 A1, filed May 25, 2017, Published Oct. 8, 2020) and Burgin et al. (US 20190236614 A1, filed Jun. 28, 2019 and Published Aug. 1, 2019)
Regarding claim 21, Wang teaches a data generation apparatus comprising:
at least one memory configured to store instructions (Wang, Paragraph [0159] teaches memory containing instructions executable by processor); and
at least one processor configured to execute the instructions (Wang, Paragraph [0159] teaches processor that executes instructions stored in memory) to:
obtain real data (Wang, Paragraph [0077] teaches input to the discriminator may comprise real data; Fig. 2 teaches receive an image to be classified (the received image reading on obtained real data));
generate, using a Generative Adversarial Network, fake data that imitates the real data (Wang, Paragraph [0077] further teaches input to the discriminator may comprise fake data generated from the generator 21 – both the discriminator and the generator are components of a Generative Adversarial Network – see Fig. 3, elements 22 and 21 – and Paragraph [0075]); and
However, Wang does not distinctly disclose:
generate mix data by mixing the real data and the fake data,
wherein the at least one processor is configured to change the mix ratio that is used to generate a data element constituting the mix data based on a position of the data element in the mix data,
wherein the at least one processor is configured to train the Generative Adversarial Network using the real data, the fake data and the generated mix data.
Nevertheless, Chen teaches generate mix data by mixing the real data and the fake data,… (Chen, Paragraph [0272] teaches The “Synthetic/Real Photo Classifier” submodule (i.e. “Discriminator”) adopts a deep network architecture and loss functions for binary attribute classification, as described in Section 2.2.1. It takes the input of an image, and the output of the network is a binary label defining whether the input image is synthetic or not, and its associated label probability ranges between 0 and 1. The training data of the submodule is a balanced mixture of real model photos obtained from retailer websites and the internet and the synthetic renders [i.e., fake data] generated from the rendering pipeline; Chen, Paragraph [0276] teaches in the model training, the optimization of “Generator” and “Discriminator” are carried out in an alternating manner. In each epoch of training, the new batch of revised synthetic renders obtained from the “Render Modifier” may be mixed with real model photos for training the “Discriminator” in the next epoch of training.).
wherein the at least one processor is configured to train the Generative Adversarial Network using the real data, the fake data and the generated mix data (Chen, Paragraph [0276] teaches in the model training, the optimization of “Generator” and “Discriminator” are carried out in an alternating manner. In each epoch of training, the new batch of revised synthetic renders obtained from the “Render Modifier” may be mixed with real model photos for training the “Discriminator” in the next epoch of training.; Chen, Paragraph [0271] teaches to implement such a “Rendering Quality Improvement Module”, we can adopt a deep neural network model using an architecture of generative adversarial networks (GAN); See Fig. 9)
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified the method and apparatus for image classification, as taught by Wang, with the synthetic/real photo classifier submodule (i.e., discriminator) for binary classification defining whether an input image is synthetic or not, as taught by Chen, in order to improve the photo-realism of synthetic renders to fool the “Discriminator”. (Chen, Paragraph [0271])
However, the combination of Wang in view of Chen does not distinctly disclose: wherein the at least one processor is configured to change the mix ratio that is used to generate a data element constituting the mix data based on a position of the data element in the mix data.
Nevertheless, Burgin teaches wherein the at least one processor is configured to change the mix ratio that is used to generate a data element constituting the mix data based on a position of the data element in the mix data (Burgin, [0052] teaches At 603, the generator 502 may attempt to arrange the random distribution to match a pixel distribution of the training image 501. Since the attempted rearrangement is unlikely to perfectly reproduce the pixels of the training image, the generator 502 may introduce a change to a training image 501. In some examples, the change may be made randomly. For example, a portion of the pixels in the training image 501 may be altered in a random way, such as by changing its position randomly and/or changing a property such as color randomly. In some instances, the pixels to be changed may be selected randomly. Alternatively or additionally, in some embodiments, the generator 502 may make changes based on input from a user and/or from the discriminator 506. For example, the user may specify certain portions of the image to be altered and/or a specific manner to alter the portions (such as changing a property such as a font of text that is imprinted on a pill). In some instances, input from the discriminator 506 may be used to alter the image. For instance, the discriminator 506 may learn (as will be described in more detail) that certain regions of an item are important for identifying counterfeits. These regions may correspond to actual parts of a genuine item that may be difficult for a counterfeiter to reproduce, may be more noticeable to SMEs 513, and/or other reasons. Whatever the reason, certain regions may be more important for identifying counterfeits than other regions. As such, with this input from the discriminator 506, the generator 502 may make changes to the training image 501 accordingly. For example, the generator 502 may focus on key regions when making a change to a training image 501, may make more subtle changes, and/or otherwise alter the way it changes the training image. In doing so, over multiple iterations, the generator 502 may generate increasingly difficult to detect counterfeit images.; [Note: here, making changes to a training image over multiple iterations has been understood to read on “change the mix ratio that is used to generate a data element constituting the mix data”. These changes may be focused on key regions of the training image and the generator may attempt to arrange the random distribution to match a pixel distribution of the training image – pixel distribution reading on “based on a position of the data element in the mix data”].)
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified the method and apparatus for image classification, as taught by Wang in view of Chen, to further include the counterfeit detection implementing a Generative Adversarial Network, as taught by Burgin. The system may implement additional technical improvements to its counterfeit detection capabilities. For instance, the system may implement a CAM that identifies regions of an image that led to the discriminator's classification of whether the image is one of a faked or genuine item. Put another way, the CAM may identify the reasons why the discriminator classified an image as a counterfeit or genuine image. To do so, the CAM may analyze the convolutional layer outputs of the discriminator to identify the regions of the image and corresponding weights. (Burgin, [0023])
Regarding claim 32, the combination of Wang in view of Chen and Burgin teaches all of the limitations of claim 20, and the combination further teaches wherein the at least one processor is configured to execute the instructions to change the mix ratio based on a time at which the mix data is generated so that the mix ratio that is used to generate the mix data in a first period is different from the mix ratio that is used to generate the mix data in a second period that is different from the first period (Chen, Paragraph [0276] teaches in the model training, the optimization of “Generator” and “Discriminator” are carried out in an alternating manner. In each epoch of training [i.e., teaching period and/or based on a time, as claimed] the new batch of revised synthetic renders obtained from the “Render Modifier” may be mixed with real model photos for training the “Discriminator” in the next epoch of training.).
Motivation to combine same as stated in claim 20.
Regarding claim 34, the combination of Wang in view of Chen and Burgin teaches all of the limitations of claim 32, and the combination further teaches wherein the at least one processor is configured to execute instructions to: discriminate discrimination target data including the real data, the fake data and the mix data, generate the fake data by using a generation model that is learnable based on a discriminated result of the discrimination target data and that is for generating the fake data; and discriminate the discrimination target data by using a discrimination model that is learnable based on the discriminated result of the discrimination target data and that is for discriminating the discrimination target data, the first period including a period before a predetermined time elapses from a start of a learning of the generation model and the discrimination model, the second period including a period after the predetermined time elapses from the start of the learning of the generation model and the discrimination model (Chen, Paragraph [0272] teaches The “Synthetic/Real Photo Classifier” submodule (i.e. “Discriminator”) adopts a deep network architecture and loss functions for binary attribute classification, as described in Section 2.2.1. It takes the input of an image, and the output of the network is a binary label defining whether the input image is synthetic or not, and its associated label probability ranges between 0 and 1. Various network architectures (e.g. VGG11/16/19, GoogLeNet) can be adopted for convolutional and pooling layers. The training data of the submodule is a balanced mixture of real model photos obtained from retailer websites and the internet and the synthetic renders [i.e., fake data] generated from the rendering pipeline; Chen, Paragraph [0276] teaches in the model training, the optimization of “Generator” and “Discriminator” are carried out in an alternating manner. In each epoch of training [i.e., comprising at least a first period], the new batch of revised synthetic renders obtained from the “Render Modifier” may be mixed with real model photos for training the “Discriminator” in the next epoch of training. [i.e., reading on second period]; Note: the target data, as claimed, is being understood as the input image to be “discriminated” by the discriminator in Chen).
Motivation to combine same as stated in claim 21.
Regarding claim 35, the combination of Wang in view of Chen and Burgin teaches all of the limitations of claim 21, wherein each of the real data, the fake data and the mix data is data relating to an image (Chen, Paragraph [0272] teaches The “Synthetic/Real Photo Classifier” submodule (i.e. “Discriminator”) adopts a deep network architecture and loss functions for binary attribute classification, as described in Section 2.2.1. It takes the input of an image, and the output of the network is a binary label defining whether the input image is synthetic or not, and its associated label probability ranges between 0 and 1. Various network architectures (e.g. VGG11/16/19, GoogLeNet) can be adopted for convolutional and pooling layers. The training data of the submodule is a balanced mixture of real model photos obtained from retailer websites and the internet and the synthetic renders [i.e., fake data] generated from the rendering pipeline; Chen, Paragraph [0276] teaches in the model training, the optimization of “Generator” and “Discriminator” are carried out in an alternating manner. In each epoch of training [i.e., comprising at least a first period], the new batch of revised synthetic renders obtained from the “Render Modifier” may be mixed with real model photos for training the “Discriminator” in the next epoch of training.; [Note: Wang also teaches real data and fake data as referring to a real image and a generated fake image]), wherein the data element of the mix data includes a pixel of the image, and wherein the position of the data element in the mix data is a position of the pixel in the image (Burgin, [0052] teaches At 603, the generator 502 may attempt to arrange the random distribution to match a pixel distribution of the training image 501. Since the attempted rearrangement is unlikely to perfectly reproduce the pixels of the training image, the generator 502 may introduce a change to a training image 501. In some examples, the change may be made randomly. For example, a portion of the pixels in the training image 501 may be altered in a random way, such as by changing its position randomly and/or changing a property such as color randomly. In some instances, the pixels to be changed may be selected randomly. Alternatively or additionally, in some embodiments, the generator 502 may make changes based on input from a user and/or from the discriminator 506. For example, the user may specify certain portions of the image to be altered and/or a specific manner to alter the portions (such as changing a property such as a font of text that is imprinted on a pill). In some instances, input from the discriminator 506 may be used to alter the image. For instance, the discriminator 506 may learn (as will be described in more detail) that certain regions of an item are important for identifying counterfeits. These regions may correspond to actual parts of a genuine item that may be difficult for a counterfeiter to reproduce, may be more noticeable to SMEs 513, and/or other reasons. Whatever the reason, certain regions may be more important for identifying counterfeits than other regions. As such, with this input from the discriminator 506, the generator 502 may make changes to the training image 501 accordingly.)
Motivation to combine same as stated for claim 21.
Regarding claim 38, Wang teaches a learning apparatus comprising:
at least one memory configured to store instructions (Wang, Paragraph [0159] teaches memory containing instructions executable by processor); and
at least one processor configured to execute instructions (Wang, Paragraph [0159] teaches processor that executes instructions stored in memory) to:
obtain real data (Wang, Paragraph [0077] teaches input to the discriminator may comprise real data; Fig. 2 teaches receive an image to be classified (the received image reading on obtained real data));
generate, using a Generative Adversarial Network, fake data that imitates the real data (Wang, Paragraph [0077] further teaches input to the discriminator may comprise fake data generated from the generator 21 – both the discriminator and the generator are components of a Generative Adversarial Network – see Fig. 3 and Paragraph);
However, Wang does not distinctly disclose:
generate mix data by mixing the real data and the fake data;
discriminate discrimination target data including the real data, the fake data and the mix data by using a discrimination model executed by the Generative Adversarial Network, wherein the at least one processor is configured to train the discrimination model based on a discriminated result of the discrimination target area,
wherein the at least one processor is configured to change a mix ratio based on a time at which the mix data is generated so that the mix ratio that is used to generate the mix data in a first period that includes a period before a predetermined time elapses from a start of a learning of the generation model and the discrimination model is different from the mix ratio that is used to generate the mix data in a second period that is different from the first period that includes a period after the predetermined time elapses from the timepoint at which the training of the discriminator model starts, and
wherein the at least one processor is configured to train the Generative Adversarial Network using the real data, the fake data and the generated mix data.
Nevertheless, Chen teaches:
generate mix data by mixing the real data and the fake data (Chen, Paragraph [0272] teaches The “Synthetic/Real Photo Classifier” submodule (i.e. “Discriminator”) adopts a deep network architecture and loss functions for binary attribute classification, as described in Section 2.2.1. It takes the input of an image, and the output of the network is a binary label defining whether the input image is synthetic or not, and its associated label probability ranges between 0 and 1. The training data of the submodule is a balanced mixture of real model photos obtained from retailer websites and the internet and the synthetic renders [i.e., fake data] generated from the rendering pipeline; Chen, Paragraph [0276] teaches in the model training, the optimization of “Generator” and “Discriminator” are carried out in an alternating manner. In each epoch of training, the new batch of revised synthetic renders obtained from the “Render Modifier” may be mixed with real model photos for training the “Discriminator” in the next epoch of training.);
wherein the at least one processor is configured to change a mix ratio based on a time at which the mix data is generated so that the mix ratio that is used to generate the mix data in a first period that includes a period before a predetermined time elapses from a start of a learning of the generation model and the discrimination model is different from the mix ratio that is used to generate the mix data in a second period that is different from the first period that includes a period after the predetermined time elapses from the timepoint at which the training of the discriminator model starts (Paragraph [0272] teaches The “Synthetic/Real Photo Classifier” submodule (i.e. “Discriminator”) adopts a deep network architecture and loss functions for binary attribute classification, as described in Section 2.2.1. It takes the input of an image, and the output of the network is a binary label defining whether the input image is synthetic or not, and its associated label probability ranges between 0 and 1. The training data of the submodule is a balanced mixture of real model photos obtained from retailer websites and the internet and the synthetic renders [i.e., fake data] generated from the rendering pipeline; Chen, Paragraph [0276] teaches in the model training, the optimization of “Generator” and “Discriminator” are carried out in an alternating manner. In each epoch of training [i.e., including at least a first period], the new batch of revised synthetic renders obtained from the “Render Modifier” may be mixed with real model photos for training the “Discriminator” in the next epoch of training [i.e., understood to read on second period.).
wherein the at least one processor is configured to train the Generative Adversarial Network using the real data, the fake data and the generated mix data (Chen, Paragraph [0276] teaches in the model training, the optimization of “Generator” and “Discriminator” are carried out in an alternating manner. In each epoch of training, the new batch of revised synthetic renders obtained from the “Render Modifier” may be mixed with real model photos for training the “Discriminator” in the next epoch of training.; Chen, Paragraph [0271] teaches to implement such a “Rendering Quality Improvement Module”, we can adopt a deep neural network model using an architecture of generative adversarial networks (GAN); See Fig. 9).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified the method and apparatus for image classification, as taught by Wang, with the synthetic/real photo classifier submodule (i.e., discriminator) for binary classification defining whether an input image is synthetic or not, as taught by Chen. One or more advantages can be achieved, for example, the discriminator of the first GAN can output a result either indicating real or fake within one network/model instead of implementing two or more tasks/models separately. The detection time is reduced and there is no need to collect anomalous samples, which are much harder to get than the normal samples. Hence, the efficiency may be also improved. (Wang, Paragraph [0069])
However, the combination does not distinctly disclose discriminate discrimination target data including the real data, the fake data and the mix data by using a discrimination model executed by the Generative Adversarial Network, wherein the at least one processor is configured to train the discrimination model based on a discriminated result of the discrimination target area,
Nevertheless, Burgin teaches discriminate discrimination target data including the real data, the fake data and the mix data by using a discrimination model executed by the Generative Adversarial Network, wherein the at least one processor is configured to train the discrimination model based on a discriminated result of the discrimination target area (Burgin, [0052] teaches At 603, the generator 502 may attempt to arrange the random distribution to match a pixel distribution of the training image 501. Since the attempted rearrangement is unlikely to perfectly reproduce the pixels of the training image, the generator 502 may introduce a change to a training image 501. In some examples, the change may be made randomly. For example, a portion of the pixels in the training image 501 may be altered in a random way, such as by changing its position randomly and/or changing a property such as color randomly. In some instances, the pixels to be changed may be selected randomly. Alternatively or additionally, in some embodiments, the generator 502 may make changes based on input from a user and/or from the discriminator 506. For example, the user may specify certain portions of the image to be altered and/or a specific manner to alter the portions (such as changing a property such as a font of text that is imprinted on a pill). In some instances, input from the discriminator 506 may be used to alter the image. For instance, the discriminator 506 may learn (as will be described in more detail) that certain regions of an item are important for identifying counterfeits. These regions may correspond to actual parts of a genuine item that may be difficult for a counterfeiter to reproduce, may be more noticeable to SMEs 513, and/or other reasons. Whatever the reason, certain regions may be more important for identifying counterfeits than other regions. As such, with this input from the discriminator 506, the generator 502 may make changes to the training image 501 accordingly. For example, the generator 502 may focus on key regions when making a change to a training image 501, may make more subtle changes, and/or otherwise alter the way it changes the training image. In doing so, over multiple iterations, the generator 502 may generate increasingly difficult to detect counterfeit images.;)
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified the method and apparatus for image classification, as taught by Wang in view of Chen, to further include the counterfeit detection implementing a Generative Adversarial Network, as taught by Burgin. The system may implement additional technical improvements to its counterfeit detection capabilities. For instance, the system may implement a CAM that identifies regions of an image that led to the discriminator's classification of whether the image is one of a faked or genuine item. Put another way, the CAM may identify the reasons why the discriminator classified an image as a counterfeit or genuine image. To do so, the CAM may analyze the convolutional layer outputs of the discriminator to identify the regions of the image and corresponding weights. (Burgin, [0023])
Regarding claim 39,
This method claim recites the same or analogous limitations as claim 21. Hence, it is rejected under the same rationale and motivation as set forth in claim 21.
Claims 22, 24, and 36 are rejected under 35 U.S.C. 103 as being unpatentable over Wang et al. in view of Chen et al. and Burgin, as applied to claim 21, and further in view of Takeda (EP3518178A1, Published July, 31, 2019)
Regarding claim 22, the combination of Wang in view of Chen and Burgin teaches all of the limitations of claim 1, and the combination further teaches wherein the at least one processor is configured to execute instructions to
change the mix ratio that is used to generate each of a plurality of data elements of the mix data in a continuous manner … (Chen, Paragraph [0272] teaches training data of the submodule is a balanced mixture of real model photos obtained from retailer websites and the internet and the synthetic renders [i.e., fake data] generated from the rendering pipeline; Chen, Paragraph [0276] teaches in the model training, the optimization of “Generator” and “Discriminator” are carried out in an alternating manner. In each epoch of training [i.e., as in “in a continuous manner” as training is iterative], the new batch of revised synthetic renders obtained from the “Render Modifier” may be mixed with real model photos for training the “Discriminator” in the next epoch of training.)….
Motivation to combine same as stated in claim 20.
However, the combination does not distinctly disclose … by using a function in which the position of the data element in the mix data is an argument.
Nevertheless, Takeda teaches … by using a function in which the position of the data element in the mix data is an argument (Takeda, Paragraph [0054], teaches Function A is a function that takes pixel values difference S(d) and the average value AvrS of the pixel values as argument [i.e. understood as position of the pixel in an object of interest in the image(s)]…; Takeda, Paragraph [0056] teaches, in summary, when it is defined that the pixel value differences increase from a negative toward a positive, the smoothed image Smooth(d) with a smaller pixel value difference S(d) accommodates to smoothing along the direction of the structure when the average value AvrS of the pixel value differences is positive, and the smoothed image Smooth(d) with a larger pixel value difference S(d) accommodates to smoothing along the direction of the structure when the average value AvrS of the pixel value differences is negative.; Takeda, Paragraph [0031] further teaches in order to obtain one synthetic image Comp from the eight smoothed images Smooth(d), weighted synthesis processing is performed to heavily weight and synthesize a smoothed image Smooth(d) on which smoothing is performed in the direction along the structure of the subject among the D smoothed images Smooth(d) and lightly weight and synthesize a smoothed image Smooth(d) on which smoothing is performed in the direction across the structure of the subject among the D smoothed images Smooth(d). Thus, the contribution of a smoothed image smoothed between pixels belonging to the same structure of the subject can be increased to perform synthesis, and the contribution of a smoothed image smoothed across portions having the different structures of the subject can be reduced to perform synthesis, and thus blurring of the boundary caused by smoothing across the boundary of the structure of the subject can be significantly reduced or prevented while the noise components are reduced by smoothing.)
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified the method and apparatus for image classification, as taught by Wang in view of Chen and Burgin, to further include the function that uses pixel positions as an argument to generate synthetic images from real images, as taught by Takeda, in order to significantly reduce or prevent variations due to noise components included in pixel values of pixels of an image, and performs smoothing processing to obtain an image excellent in visibility in which changes in the pixel values of the image are smooth. (Takeda, Paragraph [0025])
Regarding claim 24, the combination of Wang in view of Chen and Burgin teaches all of the limitations of claim 21, and the combination further teaches wherein the at least one processor is configured to execute instructions to change the mix ratio that is used to generate each of a plurality of data elements that are included in one data part of the mix data in a continuous manner… (Chen, Paragraph [0272] teaches training data of the submodule is a balanced mixture of real model photos obtained from retailer websites and the internet and the synthetic renders [i.e., fake data] generated from the rendering pipeline; Chen, Paragraph [0276] teaches in the model training, the optimization of “Generator” and “Discriminator” are carried out in an alternating manner. In each epoch of training [i.e., as in “in a continuous manner” as training is iterative], the new batch of revised synthetic renders obtained from the “Render Modifier” may be mixed with real model photos for training the “Discriminator” in the next epoch of training.; )
However, the combination does not distinctly disclose … by using a function in which the position of the data element in the mix data is an argument.
Nevertheless, Takeda teaches by using a function in which the position of the data element in the mix data is an argument (Takeda, Paragraph [0054], teaches Function A is a function that takes pixel values difference S(d) and the average value AvrS of the pixel values as argument [i.e. understood as position of the pixel in an object of interest in the image(s)]…; Takeda, Paragraph [0056] teaches, in summary, when it is defined that the pixel value differences increase from a negative toward a positive, the smoothed image Smooth(d) with a smaller pixel value difference S(d) accommodates to smoothing along the direction of the structure when the average value AvrS of the pixel value differences is positive, and the smoothed image Smooth(d) with a larger pixel value difference S(d) accommodates to smoothing along the direction of the structure when the average value AvrS of the pixel value differences is negative.; Takeda, Paragraph [0031] further teaches in order to obtain one synthetic image Comp from the eight smoothed images Smooth(d), weighted synthesis processing is performed to heavily weight and synthesize a smoothed image Smooth(d) on which smoothing is performed in the direction along the structure of the subject among the D smoothed images Smooth(d) and lightly weight and synthesize a smoothed image Smooth(d) on which smoothing is performed in the direction across the structure of the subject among the D smoothed images Smooth(d). Thus, the contribution of a smoothed image smoothed between pixels belonging to the same structure of the subject can be increased to perform synthesis, and the contribution of a smoothed image smoothed across portions having the different structures of the subject can be reduced to perform synthesis, and thus blurring of the boundary caused by smoothing across the boundary of the structure of the subject can be significantly reduced or prevented while the noise components are reduced by smoothing.).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified the method and apparatus for image classification, as taught by Wang in view of Chen and Burgin, to further include the function that uses pixel positions as an argument to generate synthetic images from real images, as taught by Takeda, in order to significantly reduce or prevent variations due to noise components included in pixel values of pixels of an image, and performs smoothing processing to obtain an image excellent in visibility in which changes in the pixel values of the image are smooth. (Takeda, Paragraph [0025])
Regarding claim 36, the combination of Wang in view of Chen and Burgin teaches all of the limitations of claim 21, and the combination further teaches wherein the at least one processor is configured to execute the instructions to change the mix ratio that is used to generate each of a plurality of data elements of the mix data in a discontinuous manner or a stepwise manner (Chen, Paragraph [0272] teaches The “Synthetic/Real Photo Classifier” submodule (i.e. “Discriminator”) adopts a deep network architecture and loss functions for binary attribute classification, as described in Section 2.2.1. It takes the input of an image, and the output of the network is a binary label defining whether the input image is synthetic or not, and its associated label probability ranges between 0 and 1. Various network architectures (e.g. VGG11/16/19, GoogLeNet) can be adopted for convolutional and pooling layers. The training data of the submodule is a balanced mixture of real model photos obtained from retailer websites and the internet and the synthetic renders [i.e., fake data] generated from the rendering pipeline; Chen, Paragraph [0276] teaches in the model training, the optimization of “Generator” and “Discriminator” are carried out in an alternating manner (i.e., understood to read on “in a discontinuous manner”, as claimed). In each epoch of training [i.e., comprising at least a first epoch or period], the new batch of revised synthetic renders obtained from the “Render Modifier” may be mixed with real model photos for training the “Discriminator” in the next epoch of training.)…
However, the combination does not distinctly disclose by using a function in which the position of the data element in the mix data is an argument.
Nevertheless, Takeda teaches by using a function in which the position of the data element in the mix data is an argument (Takeda, Paragraph [0054], teaches Function A is a function that takes pixel values difference S(d) and the average value AvrS of the pixel values as argument [i.e. understood as position of the pixel in an object of interest in the image(s)]…; Takeda, Paragraph [0056] teaches, in summary, when it is defined that the pixel value differences increase from a negative toward a positive, the smoothed image Smooth(d) with a smaller pixel value difference S(d) accommodates to smoothing along the direction of the structure when the average value AvrS of the pixel value differences is positive, and the smoothed image Smooth(d) with a larger pixel value difference S(d) accommodates to smoothing along the direction of the structure when the average value AvrS of the pixel value differences is negative.; Takeda, Paragraph [0031] further teaches in order to obtain one synthetic image Comp from the eight smoothed images Smooth(d), weighted synthesis processing is performed to heavily weight and synthesize a smoothed image Smooth(d) on which smoothing is performed in the direction along the structure of the subject among the D smoothed images Smooth(d) and lightly weight and synthesize a smoothed image Smooth(d) on which smoothing is performed in the direction across the structure of the subject among the D smoothed images Smooth(d). Thus, the contribution of a smoothed image smoothed between pixels belonging to the same structure of the subject can be increased to perform synthesis, and the contribution of a smoothed image smoothed across portions having the different structures of the subject can be reduced to perform synthesis, and thus blurring of the boundary caused by smoothing across the boundary of the structure of the subject can be significantly reduced or prevented while the noise components are reduced by smoothing.)
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified the method and apparatus for image classification, as taught by Wang in view of Chen and Burgin, to further include the function that uses pixel positions as an argument to generate synthetic images from real images, as taught by Takeda, in order to significantly reduce or prevent variations due to noise components included in pixel values of pixels of an image, and performs smoothing processing to obtain an image excellent in visibility in which changes in the pixel values of the image are smooth. (Takeda, Paragraph [0025])
Claims 23, 25, 27, 30, and 37 are rejected under 35 U.S.C. 103 as being unpatentable over Wang et al. in view of Chen et al. and Burgin et al. , as applied to claim 21, and further in view of Nowruzi et al., “How much real data do we actually need: Analyzing object detection performance using synthetic and real data”, (Published July 16, 2019)
Regarding claim 23, the combination of Wang in view of Chen and Burgin teaches all of the limitations of claim 21, however, the combination does not distinctly disclose wherein the at least one processor is configured to execute the instructions to: set the mix ratio that is used to generate a first data element of the mix data to be a first ratio; set the mix ratio that is used to generate a second data element of the mix data that is different from the first data element to be a second ratio that is different from the first ratio; and change the mix ratio that is used to generate each of a plurality of third data elements of the mix data, which is between the first and the second data elements, from the first ratio to the second ratio in a continuous manner based on the position of the third data element in the mix data.
Nevertheless, Nowruzi teaches wherein the at least one processor is configured to execute the instructions to: sets the mix ratio that is used to generate a first data element of the mix data to be a first ratio; sets the mix ratio that is used to generate a second data element of the mix data that is different from the first data element to be a second ratio that is different from the first ratio; and changes the mix ratio that is used to generate each of a plurality of third data elements of the mix data, which is between the first and the second data elements, from the first ratio to the second ratio in a continuous manner based on the position of the third data element in the mix data (Nowruzi, Table 4. teaches Results for Mixed Training. 10%, 5%, and 2.5% of real data is used in a mixed training procedure with the synthetic data. Test results are reported on the test set of the corresponding test splits of the real datasets.; Nowruzi, Section 4.3 further teaches in an attempt to achieve the full real dataset performance with only using a fraction of it, we launch a study that uses a mixed set of synthetic and real datasets with various ratios. These per-class results are shown in table 2, while the averaged results are shown in figure 4. The mixed dataset is used in training; Nowruzi, Section 4.4 further teaches in mixed training, our model learns the general concepts from simulated datasets, and uses the real samples to adapt its domain. However, there is no scheduling in the mixed training sessions. To perform a more structured experiment, we take a transfer learning approach. Model is first trained on a synthetic dataset, and then fine-tuned on each of the real datasets. We use the same ratios defined in the previous section.; Nowruzi, Section 4.5 teaches in mixed training, our model learns the general concepts from simulated datasets, and uses the real samples to adapt its domain. However, there is no scheduling in the mixed training sessions. To perform a more structured experiment, we take a transfer learning approach. Model is first trained on a synthetic dataset, and then fine-tuned on each of the real datasets. We use the same ratios defined in the previous section.; Nowruzi, Figure 6 shows the experimental results of this task. We can see that combining all the synthetic datasets for training provides better results than individually using them, except in case of BDD. This could be attributed to the completeness of the combined training data.; See also Table 5 teaching Results for Fine-tuning with Real Data. Model is trained on the synthetic dataset and is then fine-tuned on a 10%, 5%, and 2.5% portion of the real dataset.; Nowruzi, Section 5 teaches we have cross-compared the performance of multiple datasets in car and person detection. We have extensively analyzed the effects of training using datasets with a large amount of synthetic data and a small number of real data in two folds; mixed training and fine-tuning. [Note: the object detection, as disclosed, understood to read on “based on the position of the third data element in the mix data” given that to detect the object you have to know it’s position in the image])
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified the method and apparatus for image classification, as taught by Wang in view of Chen and Burgin, to further include the variations in the ratio of the mix of real and synthetic data, as taught by Nowruzi. The impressive results of synthetic training are valuable, as real data is very expensive to annotate. Using simulated data as a cheaper source of training samples can provide significant savings of both cost and time. (Nowruzi, Section 5)
Regarding claim 25, the combination of Wang in view of Chen and Burgin teaches all of the limitations of claim 21, however the combination does not distinctly disclose wherein the at least one processor is configured to execute instructions to: fix the mix ratio that is used to generate a plurality of data elements included in a first data part of the mix data to be a third ratio; fix the mix ratio that is used to generate a plurality of data elements included in a second data part of the mix data that is different from the first data part to be a fourth ratio that is different from the third ratio; and change the mix ratio that is used to generate each of a plurality of data elements included in a third part of the mix data, which is between the first and second data parts, from the third ratio to the fourth ratio in a continuous manner based on the position of the data element in the mix data.
Nevertheless, Nowruzi teaches fixes the mix ratio that is used to generate a plurality of data elements included in a first data part of the mix data to be a third ratio; fixes the mix ratio that is used to generate a plurality of data elements included in a second data part of the mix data that is different from the first data part to be a fourth ratio that is different from the third ratio; and changes the mix ratio that is used to generate each of a plurality of data elements included in a third part of the mix data, which is between the first and second data parts, from the third ratio to the fourth ratio in a continuous manner based on the position of the data element in the mix data (Nowruzi, Table 4. teaches Results for Mixed Training. 10%, 5%, and 2.5% of real data is used in a mixed training procedure with the synthetic data. Test results are reported on the test set of the corresponding test splits of the real datasets.; Nowruzi, Section 4.3 further teaches in an attempt to achieve the full real dataset performance with only using a fraction of it, we launch a study that uses a mixed set of synthetic and real datasets with various ratios. These per-class results are shown in table 2, while the averaged results are shown in figure 4. The mixed dataset is used in training; Nowruzi, Section 4.4 further teaches in mixed training, our model learns the general concepts from simulated datasets, and uses the real samples to adapt its domain. However, there is no scheduling in the mixed training sessions. To perform a more structured experiment, we take a transfer learning approach. Model is first trained on a synthetic dataset, and then fine-tuned on each of the real datasets. We use the same ratios defined in the previous section.; Nowruzi, Section 4.5 teaches in mixed training, our model learns the general concepts from simulated datasets, and uses the real samples to adapt its domain. However, there is no scheduling in the mixed training sessions. To perform a more structured experiment, we take a transfer learning approach. Model is first trained on a synthetic dataset, and then fine-tuned on each of the real datasets. We use the same ratios defined in the previous section.; Nowruzi, Figure 6 shows the experimental results of this task. We can see that combining all the synthetic datasets for training provides better results than individually using them, except in case of BDD. This could be attributed to the completeness of the combined training data.; See also Table 5 teaching Results for Fine-tuning with Real Data. Model is trained on the synthetic dataset and is then fine-tuned on a 10%, 5%, and 2.5% portion of the real dataset.; Nowruzi, Section 5 teaches we have cross-compared the performance of multiple datasets in car and person detection [Note: the object detection, as disclosed, understood to read on “based on the position of the third data element in the mix data”]. We have extensively analyzed the effects of training using datasets with a large amount of synthetic data and a small number of real data in two folds; mixed training and fine-tuning.; Table 2 teaches four experiments for various datasets with different synthetic and real ratios reading on “third ratio” and “a fourth ratio that is different from the third ratio”, as claimed).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified the method and apparatus for image classification, as taught by Wang in view of Chen and Navarrete, to further include the variations in the ratio of the mix of real and synthetic data, as taught by Nowruzi. The impressive results of synthetic training are valuable, as real data is very expensive to annotate. Using simulated data as a cheaper source of training samples can provide significant savings of both cost and time. (Nowruzi, Section 5)
Regarding claim 27, the combination of the combination of Wang in view of Chen and Burgin teaches all of the limitations of claim 21, however, the combination does not distinctly disclose wherein the at least one processor is configured to execute instructions to: set the mix ratio that is used to generate a first data element of the mix data to be a first ratio; set the mix ratio that is used to generate a second data element of the mix data that is different from the first data element to be a second ratio that is different from the first ratio; and change, among multiple values from the first ratio to the second ratio, the mix ratio that is used to generate each of a plurality of third data elements of the mix data, which is between the first and the second data elements, based on the position of the third data element in the mix data
Nevertheless, Nowruzi teaches wherein the at least one processor is configured to execute instructions to: set the mix ratio that is used to generate a first data element of the mix data to be a first ratio; set the mix ratio that is used to generate a second data element of the mix data that is different from the first data element to be a second ratio that is different from the first ratio; and change, among multiple values from the first ratio to the second ratio, the mix ratio that is used to generate each of a plurality of third data elements of the mix data, which is between the first and the second data elements, based on the position of the third data element in the mix data (Nowruzi, Table 4. teaches Results for Mixed Training. 10%, 5%, and 2.5% of real data is used in a mixed training procedure with the synthetic data. Test results are reported on the test set of the corresponding test splits of the real datasets.; Nowruzi, Section 4.3 further teaches in an attempt to achieve the full real dataset performance with only using a fraction of it, we launch a study that uses a mixed set of synthetic and real datasets with various ratios. These per-class results are shown in table 2, while the averaged results are shown in figure 4. The mixed dataset is used in training; Nowruzi, Section 4.4 further teaches in mixed training, our model learns the general concepts from simulated datasets, and uses the real samples to adapt its domain. However, there is no scheduling in the mixed training sessions. To perform a more structured experiment, we take a transfer learning approach. Model is first trained on a synthetic dataset, and then fine-tuned on each of the real datasets. We use the same ratios defined in the previous section.; Nowruzi, Section 4.5 teaches in mixed training, our model learns the general concepts from simulated datasets, and uses the real samples to adapt its domain. However, there is no scheduling in the mixed training sessions. To perform a more structured experiment, we take a transfer learning approach. Model is first trained on a synthetic dataset, and then fine-tuned on each of the real datasets. We use the same ratios defined in the previous section.; Nowruzi, Figure 6 shows the experimental results of this task. We can see that combining all the synthetic datasets for training provides better results than individually using them, except in case of BDD. This could be attributed to the completeness of the combined training data.; See also Table 5 teaching Results for Fine-tuning with Real Data. Model is trained on the synthetic dataset and is then fine-tuned on a 10%, 5%, and 2.5% portion of the real dataset.; Nowruzi, Section 5 teaches we have cross-compared the performance of multiple datasets in car and person detection [Note: the object detection, as disclosed, understood to read on “based on the position of the third data element in the mix data”]. We have extensively analyzed the effects of training using datasets with a large amount of synthetic data and a small number of real data in two folds; mixed training and fine-tuning.; Table 2 teaches four experiments for various datasets with different synthetic and real ratios reading on “mix ratio that is used to generate a second data element of the mix data that is different from the first data element to be a second ratio that is different from the first ratio; and changes, among multiple values from the first ratio to the second ratio”, as claimed).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified the method and apparatus for image classification, as taught by Wang in view of Chen and Burgin, to further include the variations in the ratio of the mix of real and synthetic data, as taught by Nowruzi. The impressive results of synthetic training are valuable, as real data is very expensive to annotate. Using simulated data as a cheaper source of training samples can provide significant savings of both cost and time. (Nowruzi, Section 5)
Regarding claim 30, the combination of Wang in view of Chen and Burgin teaches all of the limitations of claim 21, and the combination further teaches wherein the at least one processor is configured to execute the instruction to changes the mix ratio so that …. a sixth data part in which the real data and the fake data are balanced (Chen, Paragraph [0272] teaches The “Synthetic/Real Photo Classifier” submodule (i.e. “Discriminator”) adopts a deep network architecture and loss functions for binary attribute classification, as described in Section 2.2.1. It takes the input of an image, and the output of the network is a binary label defining whether the input image is synthetic or not, and its associated label probability ranges between 0 and 1. Various network architectures (e.g. VGG11/16/19, GoogLeNet) can be adopted for convolutional and pooling layers. The training data of the submodule is a balanced mixture of real model photos [i.e., real data] obtained from retailer websites and the internet and the synthetic renders [i.e., fake data] generated from the rendering pipeline; ).
However, the combination does not distinctly disclose:
wherein the at least one processor is configured to execute the instruction to changes the mix ratio so that the mix data includes a fourth data part in which the real data is dominant, a fifth data part in which the fake data is dominant …
Nevertheless, Nowruzi teaches wherein the at least one processor is configured to execute the instruction to changes the mix ratio so that the mix data includes a fourth data part in which the real data is dominant (Nowruzi, Table 2 teaches training dataset where the real data ratio is 100% and the synthetic data ratio is 0% - reading on real data is dominant as claimed), a fifth data part in which the fake data is dominant (Nowruzi, Table 2 teaches training dataset where synthetic data ratio is 97.5% and the real data ratio is 2.5% - reading on fake data is dominant as claimed)…
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified the method and apparatus for image classification, as taught by Wang in view of Chen and Burgin, to further include the variations in the ratio of the mix of real and synthetic data, as taught by Nowruzi. The impressive results of synthetic training are valuable, as real data is very expensive to annotate. Using simulated data as a cheaper source of training samples can provide significant savings of both cost and time. (Nowruzi, Section 5)
Regarding claim 37, the combination of Wang in view of Chen and Burgin teaches all of the limitations of claim 20, and the combination further teaches a sixth ratio that allows the ratio of the real data to the fake data is 1:1 ((Chen, Paragraph [0272] teaches The “Synthetic/Real Photo Classifier” submodule (i.e. “Discriminator”) adopts a deep network architecture and loss functions for binary attribute classification, as described in Section 2.2.1. It takes the input of an image, and the output of the network is a binary label defining whether the input image is synthetic or not, and its associated label probability ranges between 0 and 1. Various network architectures (e.g. VGG11/16/19, GoogLeNet) can be adopted for convolutional and pooling layers. The training data of the submodule is a balanced mixture of real model photos [i.e., real data] obtained from retailer websites and the internet and the synthetic renders [i.e., fake data] generated from the rendering pipeline; [Note: “balanced” reading on 1:1 ratio])…
However the combination does not distinctly disclose wherein the at least one processor is configured to execute the instructions to change the mix ratio so that the mix ratio changes, on a line that connects a first data element to a second element of the mix data, (i) from a fifth ratio that allows a ratio of the real data to the fake data is 1:0 to a sixth ratio that allows the ratio of the real data to the fake data is 1:1 or (ii) from the sixth ratio to the fifth ratio, or (iii) from a seventh ratio that allows a ratio of the real data to the fake data is 0:1 to the sixth ratio or (iv) from the sixth ratio to the seventh ratio.
Nevertheless, Nowruzi teaches wherein the at least one processor is configured to execute the instructions to changes the mix ratio so that the mix ratio changes, on a line that connects a first data element to a second element of the mix data, (i) from a fifth ratio that allows a ratio of the real data to the fake data is 1:0 (Nowruzi, Table 2 teaches training data where the real data ratio is 100% and the fake data is 0% - understood to read on 1:0 ratio as claimed. Said table and corresponding graph also showing changes in the mix ratio as claimed.) to a sixth ratio that allows the ratio of the real data to the fake data is 1:1 or (ii) from the sixth ratio to the fifth ratio, or (iii) from a seventh ratio that allows a ratio of the real data to the fake data is 0:1 to the sixth ratio or (iv) from the sixth ratio to the seventh ratio.
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified the method and apparatus for image classification, as taught by Wang in view of Chen and Burgin, to further include the variations in the ratio of the mix of real and synthetic data, as taught by Nowruzi. The impressive results of synthetic training are valuable, as real data is very expensive to annotate. Using simulated data as a cheaper source of training samples can provide significant savings of both cost and time. (Nowruzi, Section 5)
Claims 26 and 28 are rejected under 35 U.S.C. 103 as being unpatentable over Wang in view of Chen and Burgin, as applied to claim 21, and further in view of Nowruzi and Takeda.
Regarding claim 26, the combination of the combination of Wang in view of Chen and Burgin teaches all of the limitations of claim 20, and the combination further teaches wherein the at least one processor is configured to execute the instructions to change, …., the mix ratio that is used to generate each of a plurality of data elements of the mix data constituting the mix data (Chen, Paragraph [0272] teaches training data of the submodule is a balanced mixture of real model photos obtained from retailer websites and the internet and the synthetic renders [i.e., fake data] generated from the rendering pipeline; Chen, Paragraph [0276] teaches in the model training, the optimization of “Generator” and “Discriminator” are carried out in an alternating manner. In each epoch of training [i.e., as in “in a continuous manner”], the new batch of revised synthetic renders obtained from the “Render Modifier” may be mixed with real model photos for training the “Discriminator” in the next epoch of training).
However, the combination does not distinctly disclose:
change, among multiple values, the mix ratio…
…by using a function in which the position of the data element in the mix data is an argument.
Nevertheless, Nowruzi teaches change, among multiple values, the mix ratio… (Nowruzi, Table 4. teaches Results for Mixed Training. 10%, 5%, and 2.5% of real data is used in a mixed training procedure with the synthetic data [i.e., reading on among multiple values]. Test results are reported on the test set of the corresponding test splits of the real datasets.; Nowruzi, Section 4.3 further teaches in an attempt to achieve the full real dataset performance with only using a fraction of it, we launch a study that uses a mixed set of synthetic and real datasets with various ratios. These per-class results are shown in table 2, while the averaged results are shown in figure 4. The mixed dataset is used in training; Nowruzi, Section 4.4 further teaches in mixed training, our model learns the general concepts from simulated datasets, and uses the real samples to adapt its domain. However, there is no scheduling in the mixed training sessions. To perform a more structured experiment, we take a transfer learning approach. Model is first trained on a synthetic dataset, and then fine-tuned on each of the real datasets. We use the same ratios defined in the previous section. [i.e., teaching among multiple values as disclosed in Nowruzi]; Nowruzi, Section 4.5 teaches in mixed training, our model learns the general concepts from simulated datasets, and uses the real samples to adapt its domain. However, there is no scheduling in the mixed training sessions. To perform a more structured experiment, we take a transfer learning approach. Model is first trained on a synthetic dataset, and then fine-tuned on each of the real datasets. We use the same ratios defined in the previous section. [i.e., teaching among multiple values as disclosed in Nowruzi]; Nowruzi, Figure 6 shows the experimental results of this task. We can see that combining all the synthetic datasets for training provides better results than individually using them, except in case of BDD. This could be attributed to the completeness of the combined training data.; See also Table 5 teaching Results for Fine-tuning with Real Data. Model is trained on the synthetic dataset and is then fine-tuned on a 10%, 5%, and 2.5% portion of the real dataset. [i.e., also reading “among multiple values” as claimed]; Nowruzi, Section 5 teaches we have cross-compared the performance of multiple datasets in car and person detection. We have extensively analyzed the effects of training using datasets with a large amount of synthetic data and a small number of real data in two folds; mixed training and fine-tuning. [Note: the object detection, as disclosed, understood to read on “based on the position of the data element in the mix data” given that to detect the object you have to know it’s position in the image])
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified the method and apparatus for image classification, as taught by Wang in view of Chen and Burgin, to further include the variations in the ratio of the mix of real and synthetic data, as taught by Nowruzi. The impressive results of synthetic training are valuable, as real data is very expensive to annotate. Using simulated data as a cheaper source of training samples can provide significant savings of both cost and time. (Nowruzi, Section 5)
However, the combination in view of Nowruzi does not distinctly or clearly disclose by using a function in which the position of the data element in the mix data is an argument.
Nevertheless, Takeda teaches by using a function in which the position of the data element in the mix data is an argument (Takeda, Paragraph [0054], teaches Function A is a function that takes pixel values difference S(d) and the average value AvrS of the pixel values as argument [i.e. understood as position of the pixel in an object of interest in the image(s), among multiple values]…; Takeda, Paragraph [0056] teaches, in summary, when it is defined that the pixel value differences increase from a negative toward a positive, the smoothed image Smooth(d) with a smaller pixel value difference S(d) accommodates to smoothing along the direction of the structure when the average value AvrS of the pixel value differences is positive, and the smoothed image Smooth(d) with a larger pixel value difference S(d) accommodates to smoothing along the direction of the structure when the average value AvrS of the pixel value differences is negative.; Takeda, Paragraph [0031] further teaches in order to obtain one synthetic image Comp from the eight smoothed images Smooth(d), weighted synthesis processing is performed to heavily weight and synthesize a smoothed image Smooth(d) on which smoothing is performed in the direction along the structure of the subject among the D smoothed images Smooth(d) and lightly weight and synthesize a smoothed image Smooth(d) on which smoothing is performed in the direction across the structure of the subject among the D smoothed images Smooth(d). Thus, the contribution of a smoothed image smoothed between pixels belonging to the same structure of the subject can be increased to perform synthesis, and the contribution of a smoothed image smoothed across portions having the different structures of the subject can be reduced to perform synthesis, and thus blurring of the boundary caused by smoothing across the boundary of the structure of the subject can be significantly reduced or prevented while the noise components are reduced by smoothing.)
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified the method and apparatus for image classification, as taught by Wang in view of Chen, Burgin and Nowruzi, to further include the function that uses pixel positions as an argument to generate synthetic images from real images, as taught by Takeda, in order to significantly reduce or prevent variations due to noise components included in pixel values of pixels of an image, and performs smoothing processing to obtain an image excellent in visibility in which changes in the pixel values of the image are smooth. (Takeda, Paragraph [0025])
Regarding claim 28, the combination of Wang in view of Chen and Burgin teaches all of the limitations of claim 21, and the combination further teaches wherein the at least one processor is configured to execute the instructions to change, …, the mix ratio that is used to generate each of a plurality of data elements that are included in one data part of the mix data… (Chen, Paragraph [0272] teaches training data of the submodule is a balanced mixture of real model photos [i.e., real data] obtained from retailer websites and the internet and the synthetic renders [i.e., fake data] generated from the rendering pipeline; Chen, Paragraph [0276] teaches in the model training, the optimization of “Generator” and “Discriminator” are carried out in an alternating manner. In each epoch of training [i.e., as in “in a continuous manner”], the new batch of revised synthetic renders obtained from the “Render Modifier” may be mixed with real model photos for training the “Discriminator” in the next epoch of training.;) ….
However the combination does not distinctly disclose:
change, among multiple values, the mix ratio…
…by using a function in which the position of the data element in the mix data is an argument.
Nevertheless, Nowruzi teaches change, among multiple values, the mix ratio…(Nowruzi, Table 4. teaches Results for Mixed Training. 10%, 5%, and 2.5% of real data is used in a mixed training procedure with the synthetic data [i.e., reading on among multiple values]. Test results are reported on the test set of the corresponding test splits of the real datasets.; Nowruzi, Section 4.3 further teaches in an attempt to achieve the full real dataset performance with only using a fraction of it, we launch a study that uses a mixed set of synthetic and real datasets with various ratios. These per-class results are shown in table 2, while the averaged results are shown in figure 4. The mixed dataset is used in training; Nowruzi, Section 4.4 further teaches in mixed training, our model learns the general concepts from simulated datasets, and uses the real samples to adapt its domain. However, there is no scheduling in the mixed training sessions. To perform a more structured experiment, we take a transfer learning approach. Model is first trained on a synthetic dataset, and then fine-tuned on each of the real datasets. We use the same ratios defined in the previous section. [i.e., teaching among multiple values as disclosed in Nowruzi]; Nowruzi, Section 4.5 teaches in mixed training, our model learns the general concepts from simulated datasets, and uses the real samples to adapt its domain. However, there is no scheduling in the mixed training sessions. To perform a more structured experiment, we take a transfer learning approach. Model is first trained on a synthetic dataset, and then fine-tuned on each of the real datasets. We use the same ratios defined in the previous section. [i.e., teaching among multiple values as disclosed in Nowruzi]; Nowruzi, Figure 6 shows the experimental results of this task. We can see that combining all the synthetic datasets for training provides better results than individually using them, except in case of BDD. This could be attributed to the completeness of the combined training data.; See also Table 5 teaching Results for Fine-tuning with Real Data. Model is trained on the synthetic dataset and is then fine-tuned on a 10%, 5%, and 2.5% portion of the real dataset. [i.e., also reading “among multiple values” as claimed]; Nowruzi, Section 5 teaches we have cross-compared the performance of multiple datasets in car and person detection. We have extensively analyzed the effects of training using datasets with a large amount of synthetic data and a small number of real data in two folds; mixed training and fine-tuning. [Note: the object detection, as disclosed, understood to read on “based on the position of the data element in the mix data” given that to detect the object you have to know it’s position in the image])
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified the method and apparatus for image classification, as taught by Wang in view of Chen and Navarrete, to further include the variations in the ratio of the mix of real and synthetic data, as taught by Nowruzi. The impressive results of synthetic training are valuable, as real data is very expensive to annotate. Using simulated data as a cheaper source of training samples can provide significant savings of both cost and time. (Nowruzi, Section 5)
However the combination in view of Nowruzi does not distinctly or clearly disclose …by using a function in which the position of the data element in the mix data is an argument.
Nevertheless, Takeda teaches …by using a function in which the position of the data element in the mix data is an argument (Takeda, Paragraph [0054], teaches Function A is a function that takes pixel values difference S(d) and the average value AvrS of the pixel values as argument [i.e. understood as position of the pixel in an object of interest in the image(s)]…; Takeda, Paragraph [0056] teaches, in summary, when it is defined that the pixel value differences increase from a negative toward a positive, the smoothed image Smooth(d) with a smaller pixel value difference S(d) accommodates to smoothing along the direction of the structure when the average value AvrS of the pixel value differences is positive, and the smoothed image Smooth(d) with a larger pixel value difference S(d) accommodates to smoothing along the direction of the structure when the average value AvrS of the pixel value differences is negative.; Takeda, Paragraph [0031] further teaches in order to obtain one synthetic image Comp from the eight smoothed images Smooth(d), weighted synthesis processing is performed to heavily weight and synthesize a smoothed image Smooth(d) on which smoothing is performed in the direction along the structure of the subject among the D smoothed images Smooth(d) and lightly weight and synthesize a smoothed image Smooth(d) on which smoothing is performed in the direction across the structure of the subject among the D smoothed images Smooth(d). Thus, the contribution of a smoothed image smoothed between pixels belonging to the same structure of the subject can be increased to perform synthesis, and the contribution of a smoothed image smoothed across portions having the different structures of the subject can be reduced to perform synthesis, and thus blurring of the boundary caused by smoothing across the boundary of the structure of the subject can be significantly reduced or prevented while the noise components are reduced by smoothing.).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to have modified the method and apparatus for image classification, as taught by Wang in view of Chen Burgin and, Nowruzi, to further include the function that uses pixel positions as an argument to generate synthetic images from real images, as taught by Takeda, in order to significantly reduce or prevent variations due to noise components included in pixel values of pixels of an image, and performs smoothing processing to obtain an image excellent in visibility in which changes in the pixel values of the image are smooth. (Takeda, Paragraph [0025])
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to BEATRIZ RAMIREZ BRAVO whose telephone number is 571-272-2156. The examiner can normally be reached Mon. - Fri. 7:30a.m.-5:00p.m..
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, USMAAN SAEED can be reached at 571-272-4046. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/B.R.B./Examiner, Art Unit 2146
/USMAAN SAEED/Supervisory Patent Examiner, Art Unit 2146
Prosecution Insights