Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 4/22/2024 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Status of Claims
The present application is being examined under the claims filed on 4/22/2024.
Claims 1-17 are rejected.
Claims 1-17 are pending.
Specification
The specification filed on 4/22/2024 is acceptable for examination purposes.
Drawings
The drawings filed on 4/22/2024 are acceptable for examination purposes.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-9 and 11-17 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Regarding Claim 1,
Step 1: Claim 1 is a method claim. Therefore, Claims 1-9 are directed to a process.
Step 2A Prong 1:
determining whether a preset termination condition is satisfied (mental process – determining whether a preset termination condition is satisfied may be performed manually by a user with the aid of pen and paper by observing/analyzing a preset termination condition and using a judgement to determine whether the preset termination condition is satisfied. See MPEP 2106.04(a)(2)(III)(C).)
Step 2A Prong 2: The judicial exceptions are not integrated into a practical application.
Additional Elements:
performed by a processor executing one or more instructions stored in memory (merely using a computer as a tool to perform an abstract idea. See MPEP 2106.05(f).)
transmitting a plurality of source data to a worker terminal (Adding insignificant extra-solution activity to the judicial exception. See MPEP 2106.05(g).)
receiving, from the worker terminal, first-round labeled data for which data labeling is performed on objects in first-round source data among the plurality of source data (Adding insignificant extra-solution activity to the judicial exception. See MPEP 2106.05(g).)
generating a first-round artificial intelligence model that automatically performs the data labeling on the objects in the source data by separating some of the first-round labeled data into first-round training data, and training an artificial neural network through the first-round training data (merely reciting the words "apply it" (or an equivalent) with the judicial exception. See MPEP 2106.05(f).)
receiving, from the worker terminal, nth-round labeled data for which the data labeling is performed on objects in nth-round source data among the plurality of source data (Adding insignificant extra-solution activity to the judicial exception. See MPEP 2106.05(g).)
generating an nth-round artificial intelligence model that automatically performs the data labeling on the objects in the source data by separating some of the nth-round labeled data into the nth-round training data, and training an n-1th-round artificial neural model through the nth-round training data (merely reciting the words "apply it" (or an equivalent) with the judicial exception. See MPEP 2106.05(f).)
repeatedly performing the steps (d) to (e) until the preset termination condition is satisfied, wherein the n is a natural number applied in ascending order from 2 based on the number of times the steps (d) and (e) are performed (merely reciting the words "apply it" (or an equivalent) with the judicial exception. See MPEP 2106.05(f).)
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The additional elements recite generic computer elements and programs at a high-level of generality to perform the judicial exception as well as recitation of generic computer functionality such as a processor, a first-round artificial intelligence model and a nth-round artificial intelligence model.
Additional Elements:
performed by a processor executing one or more instructions stored in memory (merely using a computer as a tool to perform an abstract idea. See MPEP 2106.05(f).)
transmitting a plurality of source data to a worker terminal (MPEP 2106.05(d)(II) indicates that merely “Receiving or transmitting data over a network” is a well-understood, routine, conventional function when it is claimed in a merely generic manner (as it is in the present claim). Thereby, a conclusion that the claimed limitation is well-understood, routine, conventional activity is supported under Berkheimer)
receiving, from the worker terminal, first-round labeled data for which data labeling is performed on objects in first-round source data among the plurality of source data (MPEP 2106.05(d)(II) indicates that merely “Receiving or transmitting data over a network” is a well-understood, routine, conventional function when it is claimed in a merely generic manner (as it is in the present claim). Thereby, a conclusion that the claimed limitation is well-understood, routine, conventional activity is supported under Berkheimer)
generating a first-round artificial intelligence model that automatically performs the data labeling on the objects in the source data by separating some of the first-round labeled data into first-round training data, and training an artificial neural network through the first-round training data (merely reciting the words "apply it" (or an equivalent) with the judicial exception. See MPEP 2106.05(f).)
receiving, from the worker terminal, nth-round labeled data for which the data labeling is performed on objects in nth-round source data among the plurality of source data (MPEP 2106.05(d)(II) indicates that merely “Receiving or transmitting data over a network” is a well-understood, routine, conventional function when it is claimed in a merely generic manner (as it is in the present claim). Thereby, a conclusion that the claimed limitation is well-understood, routine, conventional activity is supported under Berkheimer)
generating an nth-round artificial intelligence model that automatically performs the data labeling on the objects in the source data by separating some of the nth-round labeled data into the nth-round training data, and training an n-1th-round artificial neural model through the nth-round training data (merely reciting the words "apply it" (or an equivalent) with the judicial exception. See MPEP 2106.05(f).)
repeatedly performing the steps (d) to (e) until the preset termination condition is satisfied, wherein the n is a natural number applied in ascending order from 2 based on the number of times the steps (d) and (e) are performed (merely reciting the words "apply it" (or an equivalent) with the judicial exception. See MPEP 2106.05(f).)
For the reasons above, Claim 1 is rejected as being directed to an abstract idea without significantly more. This rejection applies equally to dependent claims 2-9. The additional limitations of the dependent claims are addressed below.
Regarding Claim 2,
Step 2A Prong 1:
[wherein the processor is configured to] determine a similarity among the source data to be transmitted to the worker terminal according to a preset similarity determination criterion (mental process – determining a similarity among the source data to be transmitted to the worker terminal according to a preset similarity determination criterion may be performed manually by a user with the aid of pen and paper by observing/analyzing the source data and a preset similarity determination criterion and using a judgement to determine a similarity among the source data to be transmitted to the worker terminal. See MPEP 2106.04(a)(2)(III)(C).)
divide the plurality of source data into rounds according to the similarity among the source data (mental process – dividing the plurality of source data into rounds according to the similarity among the source data may be performed manually by a user with the aid of pen and paper by observing/analyzing the plurality of source data and the similarity among the source data and using a judgement to divide the plurality of source data into rounds. See MPEP 2106.04(a)(2)(III)(C).)
[wherein the processor is configured to] include source data with low similarity to each other in the first-round source data transmitted to the worker terminal (mental process – including source data with low similarity to each other in the first-round source data transmitted to the worker terminal may be performed manually by a user with the aid of pen and paper by observing/analyzing the source data and the similarity among the source data and using a judgement to include the source data with low similarity to each other in the first-round source data transmitted to the worker terminal. See MPEP 2106.04(a)(2)(III)(C).)
include source data with higher similarity than the similarity among source data included in the first-round source data in the nth-round source data transmitted to the worker terminal (mental process – including source data with higher similarity than the similarity among source data included in the first-round source data in the nth-round source data transmitted to the worker terminal may be performed manually by a user with the aid of pen and paper by observing/analyzing the source data and the similarity among the source data and using a judgement to include the source data with higher similarity in the nth-round source data transmitted to the worker terminal. See MPEP 2106.04(a)(2)(III)(C).)
Step 2A Prong 2: The judicial exceptions are not integrated into a practical application.
Additional Elements:
wherein the processor is configured to (merely using a computer as a tool to perform an abstract idea. See MPEP 2106.05(f).)
transmit the source data to the worker terminal (Adding insignificant extra-solution activity to the judicial exception. See MPEP 2106.05(g).)
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Additional Elements:
wherein the processor is configured to (merely using a computer as a tool to perform an abstract idea. See MPEP 2106.05(f).)
transmit the source data to the worker terminal (MPEP 2106.05(d)(II) indicates that merely “Receiving or transmitting data over a network” is a well-understood, routine, conventional function when it is claimed in a merely generic manner (as it is in the present claim). Thereby, a conclusion that the claimed limitation is well-understood, routine, conventional activity is supported under Berkheimer)
Regarding Claim 3,
Step 2A Prong 1:
See the rejection of Claim 2 above, which Claim 3 depends on.
Step 2A Prong 2: The judicial exceptions are not integrated into a practical application.
Additional Elements:
wherein the number of source data included in the nth-round source data that the processor transmits to the worker terminal is smaller than the number of source data included in the first-round source data that the processor transmits to the worker terminal (merely reciting the words "apply it" (or an equivalent) with the judicial exception. See MPEP 2106.05(f).)
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Additional Elements:
wherein the number of source data included in the nth-round source data that the processor transmits to the worker terminal is smaller than the number of source data included in the first-round source data that the processor transmits to the worker terminal (merely reciting the words "apply it" (or an equivalent) with the judicial exception. See MPEP 2106.05(f).)
Regarding Claim 4,
Step 2A Prong 1:
calculates performance of the first-round artificial intelligence model utilizing the first-round verification data (mental process – calculating performance of the first-round artificial intelligence model utilizing the first-round verification data may be performed manually by a user with the aid of pen and paper by observing/analyzing the first-round verification data and using the first-round verification data to calculate performance of the first-round artificial intelligence model. See MPEP 2106.04(a)(2)(III)(C).)
determines whether performance of the first-round artificial intelligence model satisfies a preset minimum required performance (mental process – determining whether performance of the first-round artificial intelligence model satisfies a preset minimum required performance may be performed manually by a user with the aid of pen and paper by observing/analyzing the performance of the first-round artificial intelligence model and a preset minimum required and using a judgement to determine whether performance of the first-round artificial intelligence model satisfies the preset minimum performance. See MPEP 2106.04(a)(2)(III)(C).)
Step 2A Prong 2: The judicial exceptions are not integrated into a practical application.
Additional Elements:
wherein in the step (c), the processor separates the remainder of the first-round labeled data into first-round verification data (merely using a computer as a tool to perform an abstract idea. See MPEP 2106.05(f).)
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Additional Elements:
wherein in the step (c), the processor separates the remainder of the first-round labeled data into first-round verification data (merely using a computer as a tool to perform an abstract idea. See MPEP 2106.05(f).)
Regarding Claim 5,
Step 2A Prong 1:
See the rejection of Claim 4 above, which Claim 5 depends on.
Step 2A Prong 2: The judicial exceptions are not integrated into a practical application.
Additional Elements:
wherein when the performance of the first-round artificial intelligence model satisfies the preset minimum required performance (merely reciting the words "apply it" (or an equivalent) with the judicial exception. See MPEP 2106.05(f).)
the processor designates the first-round artificial intelligence model as a preprocessing engine and transmits the preprocessing engine to the worker terminal in order to allow the worker to utilize automated data labeling performed by the preprocessing engine when the worker performs data labeling with the worker terminal (merely using a computer as a tool to perform an abstract idea. See MPEP 2106.05(f).)
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Additional Elements:
wherein when the performance of the first-round artificial intelligence model satisfies the preset minimum required performance (merely reciting the words "apply it" (or an equivalent) with the judicial exception. See MPEP 2106.05(f).)
the processor designates the first-round artificial intelligence model as a preprocessing engine and transmits the preprocessing engine to the worker terminal in order to allow the worker to utilize automated data labeling performed by the preprocessing engine when the worker performs data labeling with the worker terminal (merely using a computer as a tool to perform an abstract idea. See MPEP 2106.05(f).)
Regarding Claim 6,
Step 2A Prong 1:
See the rejection of Claim 4 above, which Claim 6 depends on.
Step 2A Prong 2: The judicial exceptions are not integrated into a practical application.
Additional Elements:
wherein when the performance of the first-round artificial intelligence model does not satisfy the preset minimum required performance (merely reciting the words "apply it" (or an equivalent) with the judicial exception. See MPEP 2106.05(f).)
the processor does not designate the first-round artificial intelligence model as a preprocessing engine (merely using a computer as a tool to perform an abstract idea. See MPEP 2106.05(f).)
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Additional Elements:
wherein when the performance of the first-round artificial intelligence model does not satisfy the preset minimum required performance (merely reciting the words "apply it" (or an equivalent) with the judicial exception. See MPEP 2106.05(f).)
the processor does not designate the first-round artificial intelligence model as a preprocessing engine (merely using a computer as a tool to perform an abstract idea. See MPEP 2106.05(f).)
Regarding Claim 7,
Step 2A Prong 1:
calculates the performance of the first-round artificial intelligence model to the nth-round artificial intelligence model, respectively, utilizing all of the first-round verification data to the nth-round verification data (mental process – calculating the performance of the first-round artificial intelligence model to the nth-round artificial intelligence model, respectively, utilizing all of the first-round verification data to the nth-round verification data may be performed manually by a user with the aid of pen and paper by observing/analyzing the first-round verification data, nth-round verification data and using all of the first-round verification data to the nth-round verification data to calculate the performance of the first-round artificial intelligence model to the nth-round artificial intelligence model. See MPEP 2106.04(a)(2)(III)(C).)
selects an artificial intelligence model with highest performance among the first-round artificial intelligence model to the nth-round artificial intelligence model (mental process – selecting an artificial intelligence model with highest performance among the first-round artificial intelligence model to the nth-round artificial intelligence model may be performed manually by a user with the aid of pen and paper by observing/analyzing the performance of the first-round artificial intelligence model to the nth-round artificial intelligence model and using a judgement to select an artificial intelligence model with highest performance. See MPEP 2106.04(a)(2)(III)(C).)
Step 2A Prong 2: The judicial exceptions are not integrated into a practical application.
Additional Elements:
wherein in the step (e), the processor separates the remainder of the nth-round labeled data into nth-round verification data (merely using a computer as a tool to perform an abstract idea. See MPEP 2106.05(f).)
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Additional Elements:
wherein in the step (e), the processor separates the remainder of the nth-round labeled data into nth-round verification data (merely using a computer as a tool to perform an abstract idea. See MPEP 2106.05(f).)
Regarding Claim 8,
Step 2A Prong 1:
See the rejection of Claim 7 above, which Claim 8 depends on.
Step 2A Prong 2: The judicial exceptions are not integrated into a practical application.
Additional Elements:
wherein the processor determines whether performance of the selected artificial intelligence model satisfies the preset minimum required performance (merely using a computer as a tool to perform an abstract idea. See MPEP 2106.05(f).)
when the performance of the selected artificial intelligence model satisfies the preset minimum required performance (merely reciting the words "apply it" (or an equivalent) with the judicial exception. See MPEP 2106.05(f).)
the processor designates the selected artificial intelligence model as a preprocessing engine and transmits the preprocessing engine to the worker terminal in order to allow the worker to utilize automated data labeling performed by the preprocessing engine when the worker performs data labeling with the worker terminal (merely using a computer as a tool to perform an abstract idea. See MPEP 2106.05(f).)
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Additional Elements:
wherein the processor determines whether performance of the selected artificial intelligence model satisfies the preset minimum required performance (merely using a computer as a tool to perform an abstract idea. See MPEP 2106.05(f).)
when the performance of the selected artificial intelligence model satisfies the preset minimum required performance (merely reciting the words "apply it" (or an equivalent) with the judicial exception. See MPEP 2106.05(f).)
the processor designates the selected artificial intelligence model as a preprocessing engine and transmits the preprocessing engine to the worker terminal in order to allow the worker to utilize automated data labeling performed by the preprocessing engine when the worker performs data labeling with the worker terminal (merely using a computer as a tool to perform an abstract idea. See MPEP 2106.05(f).)
Regarding Claim 9,
Step 2A Prong 1:
See the rejection of Claim 7 above, which Claim 9 depends on.
Step 2A Prong 2: The judicial exceptions are not integrated into a practical application.
Additional Elements:
wherein the processor determines whether performance of the selected artificial intelligence model satisfies the preset minimum required performance, and does not designate the selected artificial intelligence model as a preprocessing engine when the performance of the selected artificial intelligence model does not satisfy the preset minimum required performance (merely using a computer as a tool to perform an abstract idea. See MPEP 2106.05(f).)
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Additional Elements:
wherein the processor determines whether performance of the selected artificial intelligence model satisfies the preset minimum required performance, and does not designate the selected artificial intelligence model as a preprocessing engine when the performance of the selected artificial intelligence model does not satisfy the preset minimum required performance (merely using a computer as a tool to perform an abstract idea. See MPEP 2106.05(f).)
Regarding Claim 11,
Step 1: Claim 11 is a method claim. Therefore, Claims 11-17 are directed to a process.
Step 2A Prong 1:
calculates performance of the prior artificial intelligence model and the first-round artificial intelligence model, respectively, utilizing the first-round verification data (mental process – calculating performance of the prior artificial intelligence model and first-round artificial intelligence model, respectively, utilizing the first-round verification data may be performed manually by a user with the aid of pen and paper by observing/analyzing the prior artificial intelligence model and the first-round artificial intelligence model and the first-round verification data and using the first-round verification data to calculate performance of the prior artificial intelligence model and the first-round artificial intelligence model. See MPEP 2106.04(a)(2)(III)(C).)
selects an artificial intelligence model with higher performance among the prior artificial intelligence model and the first-round artificial intelligence model (mental process – selecting an artificial intelligence model with higher performance among the prior artificial intelligence model and the first-round artificial intelligence model may be performed manually by a user with the aid of pen and paper by observing/analyzing the performance of the prior artificial intelligence model and the first-round artificial intelligence model and using a judgement to select an artificial intelligence model with higher performance. See MPEP 2106.04(a)(2)(III)(C).)
Step 2A Prong 2: The judicial exceptions are not integrated into a practical application.
Additional Elements:
performed by a processor executing one or more instructions stored in memory (merely using a computer as a tool to perform an abstract idea. See MPEP 2106.05(f).)
transmitting a plurality of source data to a worker terminal (Adding insignificant extra-solution activity to the judicial exception. See MPEP 2106.05(g).)
receiving, from the worker terminal, first-round labeled data for which data labeling is performed on objects in first-round source data among the plurality of source data (Adding insignificant extra-solution activity to the judicial exception. See MPEP 2106.05(g).)
generating a first-round artificial intelligence model that automatically performs the data labeling on the objects in the source data by separating some of the first-round labeled data into first-round training data, and training a prior artificial intelligence model that is pre-trained to automatically perform the data labeling on the objects in the source data through the first-round training data (merely reciting the words "apply it" (or an equivalent) with the judicial exception. See MPEP 2106.05(f).)
wherein in the step (c’), the processor separates the remainder of the first-round labeled data into first-round verification data (merely using a computer as a tool to perform an abstract idea. See MPEP 2106.05(f).)
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. The additional elements recite generic computer elements and programs at a high-level of generality to perform the judicial exception as well as recitation of generic computer functionality such as a processor, a first-round artificial intelligence model and a mth-round artificial intelligence model.
Additional Elements:
performed by a processor executing one or more instructions stored in memory (merely using a computer as a tool to perform an abstract idea. See MPEP 2106.05(f).)
transmitting a plurality of source data to a worker terminal (MPEP 2106.05(d)(II) indicates that merely “Receiving or transmitting data over a network” is a well-understood, routine, conventional function when it is claimed in a merely generic manner (as it is in the present claim). Thereby, a conclusion that the claimed limitation is well-understood, routine, conventional activity is supported under Berkheimer)
receiving, from the worker terminal, first-round labeled data for which data labeling is performed on objects in first-round source data among the plurality of source data (MPEP 2106.05(d)(II) indicates that merely “Receiving or transmitting data over a network” is a well-understood, routine, conventional function when it is claimed in a merely generic manner (as it is in the present claim). Thereby, a conclusion that the claimed limitation is well-understood, routine, conventional activity is supported under Berkheimer)
generating a first-round artificial intelligence model that automatically performs the data labeling on the objects in the source data by separating some of the first-round labeled data into first-round training data, and training a prior artificial intelligence model that is pre-trained to automatically perform the data labeling on the objects in the source data through the first-round training data (merely reciting the words "apply it" (or an equivalent) with the judicial exception. See MPEP 2106.05(f).)
wherein in the step (c’), the processor separates the remainder of the first-round labeled data into first-round verification data (merely using a computer as a tool to perform an abstract idea. See MPEP 2106.05(f).)
For the reasons above, Claim 11 is rejected as being directed to an abstract idea without significantly more. This rejection applies equally to dependent claims 12-17. The additional limitations of the dependent claims are addressed below.
Regarding Claim 12,
Step 2A Prong 1:
See the rejection of Claim 11 above, which Claim 12 depends on.
Step 2A Prong 2: The judicial exceptions are not integrated into a practical application.
Additional Elements:
wherein the processor determines whether the performance of the selected artificial intelligence model satisfies a preset minimum required performance (merely using a computer as a tool to perform an abstract idea. See MPEP 2106.05(f).)
when the performance of the selected artificial intelligence model satisfies the preset minimum required performance (merely reciting the words "apply it" (or an equivalent) with the judicial exception. See MPEP 2106.05(f).)
the processor designates the selected artificial intelligence model as a preprocessing engine and transmits the preprocessing engine to the worker terminal in order to allow the worker to utilize automated data labeling performed by the preprocessing engine when the worker performs data labeling with the worker terminal (merely using a computer as a tool to perform an abstract idea. See MPEP 2106.05(f).)
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Additional Elements:
wherein the processor determines whether the performance of the selected artificial intelligence model satisfies a preset minimum required performance (merely using a computer as a tool to perform an abstract idea. See MPEP 2106.05(f).)
when the performance of the selected artificial intelligence model satisfies the preset minimum required performance (merely reciting the words "apply it" (or an equivalent) with the judicial exception. See MPEP 2106.05(f).)
the processor designates the selected artificial intelligence model as a preprocessing engine and transmits the preprocessing engine to the worker terminal in order to allow the worker to utilize automated data labeling performed by the preprocessing engine when the worker performs data labeling with the worker terminal (merely using a computer as a tool to perform an abstract idea. See MPEP 2106.05(f).)
Regarding Claim 13,
Step 2A Prong 1:
See the rejection of Claim 11 above, which Claim 13 depends on.
Step 2A Prong 2: The judicial exceptions are not integrated into a practical application.
Additional Elements:
wherein the processor determines whether the performance of the selected artificial intelligence model satisfies a preset minimum required performance, and does not designate the selected artificial intelligence model as a preprocessing engine when the performance of the selected artificial intelligence model does not satisfy the preset minimum required performance (merely using a computer as a tool to perform an abstract idea. See MPEP 2106.05(f).)
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Additional Elements:
wherein the processor determines whether the performance of the selected artificial intelligence model satisfies a preset minimum required performance, and does not designate the selected artificial intelligence model as a preprocessing engine when the performance of the selected artificial intelligence model does not satisfy the preset minimum required performance (merely using a computer as a tool to perform an abstract idea. See MPEP 2106.05(f).)
Regarding Claim 14,
Step 2A Prong 1:
determining whether a preset termination condition is satisfied (mental process – determining whether a preset termination condition is satisfied may be performed manually by a user with the aid of pen and paper by observing/analyzing a preset termination condition and using a judgement to determine whether the preset termination condition is satisfied. See MPEP 2106.04(a)(2)(III)(C).)
Step 2A Prong 2: The judicial exceptions are not integrated into a practical application.
Additional Elements:
after the step (c’), by the processor (merely using a computer as a tool to perform an abstract idea. See MPEP 2106.05(f).)
receiving, from the worker terminal, mth-round labeled data for which the data labeling is performed on objects in mth-round source data among the plurality of source data (Adding insignificant extra-solution activity to the judicial exception. See MPEP 2106.05(g).)
generating an mth-round artificial intelligence model that automatically performs the data labeling on the objects in the source data by separating some of the mth-round labeled data into the mth-round training data and training an m-1th-round artificial intelligence model through the mth-round training data (merely reciting the words "apply it" (or an equivalent) with the judicial exception. See MPEP 2106.05(f).)
repeatedly performing the steps (d’) to (e’) until the preset termination condition is satisfied, wherein the m is a natural number applied in ascending order from 2 based on the number of times the steps (d’) and (e’) are performed (merely reciting the words "apply it" (or an equivalent) with the judicial exception. See MPEP 2106.05(f).)
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Additional Elements:
after the step (c’), by the processor (merely using a computer as a tool to perform an abstract idea. See MPEP 2106.05(f).)
receiving, from the worker terminal, mth-round labeled data for which the data labeling is performed on objects in mth-round source data among the plurality of source data (MPEP 2106.05(d)(II) indicates that merely “Receiving or transmitting data over a network” is a well-understood, routine, conventional function when it is claimed in a merely generic manner (as it is in the present claim). Thereby, a conclusion that the claimed limitation is well-understood, routine, conventional activity is supported under Berkheimer)
generating an mth-round artificial intelligence model that automatically performs the data labeling on the objects in the source data by separating some of the mth-round labeled data into the mth-round training data and training an m-1th-round artificial intelligence model through the mth-round training data (merely reciting the words "apply it" (or an equivalent) with the judicial exception. See MPEP 2106.05(f).)
repeatedly performing the steps (d’) to (e’) until the preset termination condition is satisfied, wherein the m is a natural number applied in ascending order from 2 based on the number of times the steps (d’) and (e’) are performed (merely reciting the words "apply it" (or an equivalent) with the judicial exception. See MPEP 2106.05(f).)
Regarding Claim 15,
Step 2A Prong 1:
calculates the performance of the prior artificial intelligence model, and the first-round artificial intelligence model to the mth-round artificial intelligence model, respectively, utilizing all of the first-round verification data to the mth-round verification data (mental process – calculating the performance of the prior artificial intelligence model, and the first-round artificial intelligence model to the mth-round artificial intelligence model, respectively, utilizing all of the first-round verification data to the mth-round verification data may be performed manually by a user with the aid of pen and paper by observing/analyzing the first-round verification data, mth-round verification data and using all of the first-round verification data to the mth-round verification data to calculate the performance of the prior artificial intelligence model, and the first-round artificial intelligence model to the mth-round artificial intelligence model. See MPEP 2106.04(a)(2)(III)(C).)
selects an artificial intelligence model with highest performance among the prior artificial intelligence model, and the first-round artificial intelligence model to the mth-round artificial intelligence model (mental process – selecting an artificial intelligence model with highest performance among the prior artificial intelligence model, and the first-round artificial intelligence model to the mth-round artificial intelligence model may be performed manually by a user with the aid of pen and paper by observing/analyzing the performance of the prior artificial intelligence model, and the first-round artificial intelligence model to the mth-round artificial intelligence model and using a judgement to select an artificial intelligence model with highest performance. See MPEP 2106.04(a)(2)(III)(C).)
Step 2A Prong 2: The judicial exceptions are not integrated into a practical application.
Additional Elements:
wherein in the step (e’), the processor separates the remainder of the mth-round labeled data into mth-round verification data (merely using a computer as a tool to perform an abstract idea. See MPEP 2106.05(f).)
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Additional Elements:
wherein in the step (e’), the processor separates the remainder of the mth-round labeled data into mth-round verification data (merely using a computer as a tool to perform an abstract idea. See MPEP 2106.05(f).)
Regarding Claim 16,
Step 2A Prong 1:
See the rejection of Claim 15 above, which Claim 16 depends on.
Step 2A Prong 2: The judicial exceptions are not integrated into a practical application.
Additional Elements:
wherein the processor determines whether the performance of the selected artificial intelligence model satisfies a preset minimum required performance (merely using a computer as a tool to perform an abstract idea. See MPEP 2106.05(f).)
when the performance of the selected artificial intelligence model satisfies the preset minimum required performance (merely reciting the words "apply it" (or an equivalent) with the judicial exception. See MPEP 2106.05(f).)
the processor designates the selected artificial intelligence model as a preprocessing engine and transmits the preprocessing engine to the worker terminal in order to allow the worker to utilize automated data labeling performed by the preprocessing engine when the worker performs data labeling with the worker terminal (merely using a computer as a tool to perform an abstract idea. See MPEP 2106.05(f).)
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Additional Elements:
wherein the processor determines whether the performance of the selected artificial intelligence model satisfies a preset minimum required performance (merely using a computer as a tool to perform an abstract idea. See MPEP 2106.05(f).)
when the performance of the selected artificial intelligence model satisfies the preset minimum required performance (merely reciting the words "apply it" (or an equivalent) with the judicial exception. See MPEP 2106.05(f).)
the processor designates the selected artificial intelligence model as a preprocessing engine and transmits the preprocessing engine to the worker terminal in order to allow the worker to utilize automated data labeling performed by the preprocessing engine when the worker performs data labeling with the worker terminal (merely using a computer as a tool to perform an abstract idea. See MPEP 2106.05(f).)
Regarding Claim 17,
Step 2A Prong 1:
See the rejection of Claim 15 above, which Claim 17 depends on.
Step 2A Prong 2: The judicial exceptions are not integrated into a practical application.
Additional Elements:
wherein the processor determines whether performance of the selected artificial intelligence model satisfies a preset minimum required performance, and does not designate the selected artificial intelligence model as a preprocessing engine when the performance of the selected artificial intelligence model does not satisfy the preset minimum required performance (merely using a computer as a tool to perform an abstract idea. See MPEP 2106.05(f).)
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception.
Additional Elements:
wherein the processor determines whether performance of the selected artificial intelligence model satisfies a preset minimum required performance, and does not designate the selected artificial intelligence model as a preprocessing engine when the performance of the selected artificial intelligence model does not satisfy the preset minimum required performance (merely using a computer as a tool to perform an abstract idea. See MPEP 2106.05(f).)
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2, 10 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over Johnson et al. (US 11263556 B2) (hereinafter Johnson) and in view of Deng et al. (US 20190318261 A1) (hereinafter Deng).
Regarding Claim 1,
Johnson teaches:
“An automated training-based data labeling method performed by a processor executing one or more instructions stored in memory” (Johnson, Col. 2, Lines 23-27 and Lines 57-60, “The apparatus comprises a processor and a memory that stores processor-executable instructions, wherein the processor interfaces with the memory to execute the processor-executable instructions […] the present disclosure provides a method in an apparatus for implementing a batch-mode active learning for technology-assisted review (TAR) of documents.”; Examiner’s note: an automated training-based data labeling method (i.e., a method for implementing a batch-mode active learning for technology-assisted review (TAR) of documents) performed by a processor executing one or more instructions stored in memory (i.e., a processor and a memory that stores processor-executable instructions) is taught.)
“receiving, from the worker terminal, first-round labeled data for which data labeling is performed on objects in first-round source data among the plurality of source data” (Johnson, Col. 20, Lines 39-47, “At step 702, the apparatus 600 is operable to obtain an unlabeled set of documents D. At step 704, the apparatus 600 is operable to obtain (e.g., from the expert 614) a batch size k. At step 706, the apparatus 600 is operable to construct a first batch of k documents D. At step 708, the apparatus 600 is operable to obtain (e.g., from the expert 614) labels for the first batch of k documents D, wherein the labeled first batch of k documents D are referred to as training data documents.”; Examiner’s note: receiving, from the worker terminal, first-round labeled data for which data labeling is performed on objects in first-round source data among the plurality of source data (i.e., obtaining an unlabeled set of documents D, constructing a first batch of k documents D and obtaining labels for the first batch of k documents D) is taught.)
“generating a first-round artificial intelligence model that automatically performs the data labeling on the objects in the source data by separating some of the first-round labeled data into first-round training data, and training an artificial neural network through the first-round training data” (Johnson, Col. 20, Lines 39-51, “At step 702, the apparatus 600 is operable to obtain an unlabeled set of documents D. At step 704, the apparatus 600 is operable to obtain (e.g., from the expert 614) a batch size k. At step 706, the apparatus 600 is operable to construct a first batch of k documents D. At step 708, the apparatus 600 is operable to obtain (e.g., from the expert 614) labels for the first batch of k documents D, wherein the labeled first batch of k documents D are referred to as training data documents. At step 710, the apparatus 600 is operable to construct a hyperplane hc(x) using the labeled first batch of k documents D. At step 712, the apparatus 600 is operable to perform an iteration of active learning using a support vector machine (SVM).”; Johnson, Col. 24, Lines 16-17, “automatically select a label for each unlabeled document in the collection”; Johnson, Col. 27-28, Lines 66-67 and 10-12, “the classification model Mc can be any type of classification model such as, for example, a support vector machine (SVM) model, a logistic regression model, a nearest neighbors model, decision forest model, neural network model, Bayesian model […]”; Examiner’s note: generating a first-round artificial intelligence model that automatically performs the data labeling on the objects in the source data by separating some of the first-round labeled data into first-round training data, and training an artificial neural network through the first-round training data (i.e., obtaining an unlabeled set of documents D, constructing a first batch of k documents D, obtaining labels for the first batch of k documents D, wherein the labeled first batch of k documents D are referred to as training data documents, constructing a hyperplane and performing an iteration of active learning using a support vector machine (SVM). Automatically, selecting a label for each unlabeled document is doable and the classification model can be neural network model as well) is taught.)
“receiving, from the worker terminal, nth-round labeled data for which the data labeling is performed on objects in nth-round source data among the plurality of source data” (Johnson, Col. 2, Lines 33-46, “(f) perform an iteration of active learning using a support vector machine (SVM), wherein the perform operation comprises: (i) select a new batch of unlabeled instances (documents) Bc using a current version of the hyperplane hc(x), an unlabeled set of available documents D, and the batch size k; (ii) obtain labels for the new batch of unlabeled instances Bc; (iii) add the labeled new batch of instances Bc to a current version of the training data documents referred to as extended training data documents Dc; (g) construct an updated hyperplane h(x) using the extended training data documents Dc; (h) determine whether a stopping criteria has been met; (i) based on the determination that the stopping criteria has not been met, repeat the perform operation […]”; Examiner’s note: receiving, from the worker terminal, nth-round labeled data for which the data labeling is performed on objects in nth-round source data among the plurality of source data (i.e., selecting a new batch of unlabeled instances(documents), obtaining labels for the new batch, adding the labeled new batch to a current version of the training data documents, constructing an updated hyperplane and repeating the operation if the stopping criteria has not been met) is taught.)
“generating an nth-round artificial intelligence model that automatically performs the data labeling on the objects in the source data by separating some of the nth-round labeled data into the nth-round training data, and training an n-1th-round artificial neural model through the nth-round training data” (Johnson, Col. 2, Lines 33-46, “(f) perform an iteration of active learning using a support vector machine (SVM), wherein the perform operation comprises: (i) select a new batch of unlabeled instances (documents) Bc using a current version of the hyperplane hc(x), an unlabeled set of available documents D, and the batch size k; (ii) obtain labels for the new batch of unlabeled instances Bc; (iii) add the labeled new batch of instances Bc to a current version of the training data documents referred to as extended training data documents Dc; (g) construct an updated hyperplane h(x) using the extended training data documents Dc; (h) determine whether a stopping criteria has been met; (i) based on the determination that the stopping criteria has not been met, repeat the perform operation […]”; Johnson, Col. 24, Lines 16-17, “automatically select a label for each unlabeled document in the collection”; Johnson, Col. 27-28, Lines 66-67 and 10-12, “the classification model Mc can be any type of classification model such as, for example, a support vector machine (SVM) model, a logistic regression model, a nearest neighbors model, decision forest model, neural network model, Bayesian model […]”; Examiner’s note: generating an nth-round artificial intelligence model that automatically performs the data labeling on the objects in the source data by separating some of the nth-round labeled data into the nth-round training data, and training an n-1th-round artificial neural model through the nth-round training data (i.e., performing an iteration of active learning using a support vector machine (SVM) including obtaining labels for the new batch, adding the labeled new batch to a current version of the training data documents, constructing an updated hyperplane and repeating the operation if the stopping criteria has not been met. Automatically, selecting a label for each unlabeled document is doable and the classification model can be neural network model as well) is taught.)
“determining whether a preset termination condition is satisfied, and repeatedly performing the steps (d) to (e) until the preset termination condition is satisfied, wherein the n is a natural number applied in ascending order from 2 based on the number of times the steps (d) and (e) are performed” (Johnson, Col. 2, Lines 33-46, “(f) perform an iteration of active learning using a support vector machine (SVM), wherein the perform operation comprises: (i) select a new batch of unlabeled instances (documents) Bc using a current version of the hyperplane hc(x), an unlabeled set of available documents D, and the batch size k; (ii) obtain labels for the new batch of unlabeled instances Bc; (iii) add the labeled new batch of instances Bc to a current version of the training data documents referred to as extended training data documents Dc; (g) construct an updated hyperplane h(x) using the extended training data documents Dc; (h) determine whether a stopping criteria has been met; (i) based on the determination that the stopping criteria has not been met, repeat the perform operation […]”; Examiner’s note: determining whether a preset termination condition is satisfied, and repeatedly performing the steps (d) to (e) until the preset termination condition is satisfied, wherein the n is a natural number applied in ascending order from 2 based on the number of times the steps (d) and (e) are performed (i.e., repeating the operation if the stopping criteria has not been met. The operation includes selecting a new batch of unlabeled documents, obtaining labels for the new batch, adding the labeled new batch to a current version of the training data documents, training and repeating if the stopping criteria not met) is taught.)
Johnson does not explicitly teach:
“transmitting a plurality of source data to a worker terminal”
Deng teaches:
“transmitting a plurality of source data to a worker terminal” (Deng, Paragraph [0082], “the first electronic device 602 transmits the unlabeled input over the network 606 to the second electronic device 604, which provides the unlabeled input 608 to the feature encoder 404.”; Examiner’s note: transmitting a plurality of source data (i.e., transmitting the unlabeled input) to a worker terminal (i.e., the second electronic device 604, which provides the unlabeled input to the feature encoder 404) is taught.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the apparatus and method of implementing batch-mode active learning for technology-assisted review of documents in Johnson, and the system and method for active machine learning as taught in Deng. Johnson teaches receiving the labeled data, generating the artificial intelligence model that automatically performs the data labeling, training an artificial neural network, determining whether a preset termination condition is satisfied and repeating the performing steps until the preset termination condition is satisfied. Deng teaches transmitting a plurality of source data to a worker terminal. One of ordinary skill would have motivation to combine Johnson and Deng to “improve the active learning efficiency, particularly on large datasets” (Deng, Paragraph [0037]).
Regarding Claim 2,
The combination of Johnson and Deng teaches:
“The automated training-based data labeling method of claim 1,” (preamble)
“wherein the processor is configured to determine a similarity among the source data to be transmitted to the worker terminal according to a preset similarity determination criterion, divide the plurality of source data into rounds according to the similarity among the source data, and transmit the source data to the worker terminal” (Deng, Paragraph [0006], “The at least one processor is configured to select one or more entries from a data set including unlabeled data based on a similarity between the one or more entries and labeled data.”; Deng, Paragraph [0082], “the first electronic device 602 transmits the unlabeled input over the network 606 to the second electronic device 604, which provides the unlabeled input 608 to the feature encoder 404.”; Deng, Paragraph [0089], “If the processor of the second electronic device determines that the similarity metric is above a threshold at decision block 710, the process 700 moves to block 712. At block 712, the processor of the second electronic device sends the unlabeled data to a labeling model that is executed by the processor of the second electronic device or another electronic device and that is configured to label data that is similar to already-labeled data.”; Examiner’s note: wherein the processor is configured to determine a similarity among the source data (i.e., the at least one processor configured to select one or more entries from a data set including unlabeled data based on a similarity between the one or more entries) to be transmitted to the worker terminal (i.e., transmitted to the second electronic device) according to a preset similarity determination criterion (i.e., the similarity metric above a threshold), divide the plurality of source data into rounds according to the similarity among the source data, and transmit the source data to the worker terminal (i.e., the process 700 moving to block 712 where the processor is configured to label data that is similar to already-labeled data) is taught.)
“wherein the processor is configured to include source data with low similarity to each other in the first-round source data transmitted to the worker terminal, and include source data with higher similarity than the similarity among source data included in the first-round source data in the nth-round source data transmitted to the worker terminal” (Deng, Paragraph [0060], “a labeling priority is offered to samples with low similarity scores by a score analysis and labeling module 218 […] If, for instance, one of the unlabeled images includes a plate of food, the image of the plate of food will be given a low score. The image with the low score is then prioritized for labeling, such as manual labeling or image captioning by a user, because an image of a plate of food will offer greater new knowledge to the system than will additional images of humans engaged in sporting activities.”; Johnson, Col. 2, Lines 33-46, “(f) perform an iteration of active learning using a support vector machine (SVM), wherein the perform operation comprises: (i) select a new batch of unlabeled instances (documents) Bc using a current version of the hyperplane hc(x), an unlabeled set of available documents D, and the batch size k; (ii) obtain labels for the new batch of unlabeled instances Bc; (iii) add the labeled new batch of instances Bc to a current version of the training data documents referred to as extended training data documents Dc; (g) construct an updated hyperplane h(x) using the extended training data documents Dc; (h) determine whether a stopping criteria has been met; (i) based on the determination that the stopping criteria has not been met, repeat the perform operation […]”; Examiner’s note: wherein the processor is configured to include source data with low similarity to each other in the first-round source data transmitted to the worker terminal (i.e., labeling priority with low similarity scores), and include source data with higher similarity than the similarity among source data included in the first-round source data in the nth-round source data transmitted to the worker terminal (i.e., source data with higher similarity scores being processed in the later rounds. Repeating operation steps are taught by Johnson) is taught.)
The reasons of obviousness have been noted in the rejection of Claim 1 above and applicable herein.
Regarding Claim 10,
Johnson teaches:
“An automated training-based data labeling method performed by a processor executing one or more instructions stored in memory” (Johnson, Col. 2, Lines 23-27 and Lines 57-60, “The apparatus comprises a processor and a memory that stores processor-executable instructions, wherein the processor interfaces with the memory to execute the processor-executable instructions […] the present disclosure provides a method in an apparatus for implementing a batch-mode active learning for technology-assisted review (TAR) of documents.”; Examiner’s note: an automated training-based data labeling method (i.e., a method for implementing a batch-mode active learning for technology-assisted review (TAR) of documents) performed by a processor executing one or more instructions stored in memory (i.e., a processor and a memory that stores processor-executable instructions) is taught.)
“receiving, from the worker terminal, first-round labeled data for which data labeling is performed on objects in first-round source data among the plurality of source data” (Johnson, Col. 20, Lines 39-47, “At step 702, the apparatus 600 is operable to obtain an unlabeled set of documents D. At step 704, the apparatus 600 is operable to obtain (e.g., from the expert 614) a batch size k. At step 706, the apparatus 600 is operable to construct a first batch of k documents D. At step 708, the apparatus 600 is operable to obtain (e.g., from the expert 614) labels for the first batch of k documents D, wherein the labeled first batch of k documents D are referred to as training data documents.”; Examiner’s note: receiving, from the worker terminal, first-round labeled data for which data labeling is performed on objects in first-round source data among the plurality of source data (i.e., obtaining an unlabeled set of documents D, constructing a first batch of k documents D and obtaining labels for the first batch of k documents D) is taught.)
“generating a first-round artificial intelligence model that automatically performs the data labeling on the objects in the source data by separating some of the first-round labeled data into first-round training data, and training a prior artificial intelligence model that is pre-trained to automatically perform the data labeling on the objects in the source data through the first-round training data” (Johnson, Col. 20, Lines 39-51, “At step 702, the apparatus 600 is operable to obtain an unlabeled set of documents D. At step 704, the apparatus 600 is operable to obtain (e.g., from the expert 614) a batch size k. At step 706, the apparatus 600 is operable to construct a first batch of k documents D. At step 708, the apparatus 600 is operable to obtain (e.g., from the expert 614) labels for the first batch of k documents D, wherein the labeled first batch of k documents D are referred to as training data documents. At step 710, the apparatus 600 is operable to construct a hyperplane hc(x) using the labeled first batch of k documents D. At step 712, the apparatus 600 is operable to perform an iteration of active learning using a support vector machine (SVM).”; Johnson, Col. 24, Lines 16-17 and 29-33, “[…] automatically select a label for each unlabeled document in the collection […] A classifier composes a method for taking a subset of document profiles along with their labels, called the training set, and produces a mathematical model that can be used to predict the labels for other document profiles.”; Examiner’s note: generating a first-round artificial intelligence model that automatically performs the data labeling on the objects in the source data by separating some of the first-round labeled data into first-round training data, and training a prior artificial intelligence model that is pre-trained to automatically perform the data labeling on the objects in the source data through the first-round training data (i.e., obtaining an unlabeled set of documents D, constructing a first batch of k documents D, obtaining labels for the first batch of k documents D, wherein the labeled first batch of k documents D are referred to as training data documents, constructing a hyperplane and performing an iteration of active learning using a support vector machine (SVM). Automatically, selecting a label for each unlabeled document is doable and a classifier taking subset of document profiles with their labels and producing a mathematical model to predict the labels for other document profiles teach training a prior artificial intelligence model that is pre-trained to automatically perform the data labeling on the objects) is taught.)
Johnson does not explicitly teach:
“transmitting a plurality of source data to a worker terminal”
Deng teaches:
“transmitting a plurality of source data to a worker terminal” (Deng, Paragraph [0082], “the first electronic device 602 transmits the unlabeled input over the network 606 to the second electronic device 604, which provides the unlabeled input 608 to the feature encoder 404.”; Examiner’s note: transmitting a plurality of source data (i.e., transmitting the unlabeled input) to a worker terminal (i.e., the second electronic device 604, which provides the unlabeled input to the feature encoder 404) is taught.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the apparatus and method of implementing batch-mode active learning for technology-assisted review of documents in Johnson, and the system and method for active machine learning as taught in Deng. Johnson teaches receiving the labeled data, generating the artificial intelligence model that automatically performs the data labeling, training an artificial neural network, determining whether a preset termination condition is satisfied and repeating the performing steps until the preset termination condition is satisfied. Deng teaches transmitting a plurality of source data to a worker terminal. One of ordinary skill would have motivation to combine Johnson and Deng to “improve the active learning efficiency, particularly on large datasets” (Deng, Paragraph [0037]).
Regarding Claim 14,
The combination of Johnson and Deng teaches:
“The automated training-based data labeling method of claim 11, further comprising: after the step (c’), by the processor,” (preamble)
“receiving, from the worker terminal, mth-round labeled data for which the data labeling is performed on objects in mth-round source data among the plurality of source data” (Johnson, Col. 2, Lines 33-46, “(f) perform an iteration of active learning using a support vector machine (SVM), wherein the perform operation comprises: (i) select a new batch of unlabeled instances (documents) Bc using a current version of the hyperplane hc(x), an unlabeled set of available documents D, and the batch size k; (ii) obtain labels for the new batch of unlabeled instances Bc; (iii) add the labeled new batch of instances Bc to a current version of the training data documents referred to as extended training data documents Dc; (g) construct an updated hyperplane h(x) using the extended training data documents Dc; (h) determine whether a stopping criteria has been met; (i) based on the determination that the stopping criteria has not been met, repeat the perform operation […]”; Examiner’s note: receiving, from the worker terminal, mth-round labeled data for which the data labeling is performed on objects in mth-round source data among the plurality of source data (i.e., selecting a new batch of unlabeled instances(documents), obtaining labels for the new batch, adding the labeled new batch to a current version of the training data documents, constructing an updated hyperplane and repeating the operation if the stopping criteria has not been met) is taught.)
“generating an mth-round artificial intelligence model that automatically performs the data labeling on the objects in the source data by separating some of the mth-round labeled data into the mth-round training data, and training an m-1th-round artificial neural model through the mth-round training data” (Johnson, Col. 2, Lines 33-46, “(f) perform an iteration of active learning using a support vector machine (SVM), wherein the perform operation comprises: (i) select a new batch of unlabeled instances (documents) Bc using a current version of the hyperplane hc(x), an unlabeled set of available documents D, and the batch size k; (ii) obtain labels for the new batch of unlabeled instances Bc; (iii) add the labeled new batch of instances Bc to a current version of the training data documents referred to as extended training data documents Dc; (g) construct an updated hyperplane h(x) using the extended training data documents Dc; (h) determine whether a stopping criteria has been met; (i) based on the determination that the stopping criteria has not been met, repeat the perform operation […]”; Johnson, Col. 24, Lines 16-17, “automatically select a label for each unlabeled document in the collection”; Johnson, Col. 27-28, Lines 66-67 and 10-12, “the classification model Mc can be any type of classification model such as, for example, a support vector machine (SVM) model, a logistic regression model, a nearest neighbors model, decision forest model, neural network model, Bayesian model […]”; Examiner’s note: generating an mth-round artificial intelligence model that automatically performs the data labeling on the objects in the source data by separating some of the mth-round labeled data into the mth-round training data, and training an m-1th-round artificial intelligence model through the mth-round training data (i.e., performing an iteration of active learning using a support vector machine (SVM) including obtaining labels for the new batch, adding the labeled new batch to a current version of the training data documents, constructing an updated hyperplane and repeating the operation if the stopping criteria has not been met. Automatically, selecting a label for each unlabeled document is doable) is taught.)
“determining whether a preset termination condition is satisfied, and repeatedly performing the steps (d’) to (e’) until the preset termination condition is satisfied, wherein the m is a natural number applied in ascending order from 2 based on the number of times the steps (d’) and (e’) are performed” (Johnson, Col. 2, Lines 33-46, “(f) perform an iteration of active learning using a support vector machine (SVM), wherein the perform operation comprises: (i) select a new batch of unlabeled instances (documents) Bc using a current version of the hyperplane hc(x), an unlabeled set of available documents D, and the batch size k; (ii) obtain labels for the new batch of unlabeled instances Bc; (iii) add the labeled new batch of instances Bc to a current version of the training data documents referred to as extended training data documents Dc; (g) construct an updated hyperplane h(x) using the extended training data documents Dc; (h) determine whether a stopping criteria has been met; (i) based on the determination that the stopping criteria has not been met, repeat the perform operation […]”; Examiner’s note: determining whether a preset termination condition is satisfied, and repeatedly performing the steps (d’) to (e’) until the preset termination condition is satisfied, wherein the n is a natural number applied in ascending order from 2 based on the number of times the steps (d’) and (e’) are performed (i.e., repeating the operation if the stopping criteria has not been met. The operation includes selecting a new batch of unlabeled documents, obtaining labels for the new batch, adding the labeled new batch to a current version of the training data documents, training and repeating if the stopping criteria not met) is taught.)
The reasons of obviousness have been noted in the rejection of Claim 11 above and applicable herein.
Claim 3 is rejected under 35 U.S.C. 103 as being unpatentable over Johnson in view of Deng as applied in claim 1, in view of Johnson et al. (US 20220067456 A1) (hereinafter Johnson2).
Regarding Claim 3,
The combination of Johnson and Deng teaches:
“The automated training-based data labeling method of claim 2,” (preamble)
the processor transmits to the worker terminal (Deng, Paragraph [0082], “the first electronic device 602 transmits the unlabeled input over the network 606 to the second electronic device 604, which provides the unlabeled input 608 to the feature encoder 404.”; Examiner’s note: the processor transmits to the worker terminal (i.e., the first electronic device 602 transmitting the unlabeled input) to the worker terminal (i.e., the second electronic device 604, which provides the unlabeled input to the feature encoder 404) is taught.)
The combination of Johnson and Deng does not explicitly teach:
wherein the number of source data included in the nth-round source data is smaller than the number of source data included in the first-round source data
Johnson2 teaches:
wherein the number of source data included in the nth-round source data is smaller than the number of source data included in the first-round source data (Johnson2, Paragraph [0021], “[…] the computing device increases the number of diverse documents in the subsequent batch by a smaller amount.”; Examiner’s note: wherein the number of source data included in the nth-round source data that the processor transmits to the worker terminal is smaller than the number of source data included in the first-round source data that the processor transmits to the worker terminal (i.e., the subsequent batch by a smaller amount teaches the number of source data in the nth-round source data is smaller than the number of source data in the first-round source data) is taught.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the apparatus and method of implementing batch-mode active learning for technology-assisted review of documents in Johnson, and the system and method for active machine learning in Deng, and the diversity sampling for technology-assisted document review as taught in Johnson2. Johnson teaches receiving the labeled data, generating the artificial intelligence model that automatically performs the data labeling, training an artificial neural network, determining whether a preset termination condition is satisfied and repeating the performing steps until the preset termination condition is satisfied. Deng teaches transmitting a plurality of source data to a worker terminal. Johnson2 teaches that the number of source data included in the nth-round source data is smaller than the number of source data included in the first-round source data. One of ordinary skill would have motivation to combine Johnson, Deng and Johnson2 to “improve[] methods 250 for selecting subsequent batches of documents and updating the classification model” (Johnson2, Paragraph [0039]).
Claims 4, 7, 11 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Johnson in view of Deng as applied in claim 1, in view of Ma et al. (US 20190258904 A1) (hereinafter Ma).
Regarding Claim 4,
The combination of Johnson and Deng teaches:
“The automated training-based data labeling method of claim 1,” (preamble)
The combination of Johnson and Deng does not explicitly teach:
“wherein in the step (c), the processor separates the remainder of the first-round labeled data into first-round verification data”
“calculates performance of the first-round artificial intelligence model utilizing the first-round verification data”
“determines whether performance of the first-round artificial intelligence model satisfies a preset minimum required performance”
Ma teaches:
“wherein in the step (c), the processor separates the remainder of the first-round labeled data into first-round verification data” (Ma, Paragraphs [0005], [0006] and [0045], “A remainder of the input dataset is allocated to a training/validation dataset. The training/validation dataset is partitioned into an oversampled training/validation dataset using an oversampling process based on a predefined value of the event assessment variable. (a) A validation sample is selected from the oversampled training/validation dataset using a second stratified sampling process based on the value of the event assessment variable. (b) A training sample is selected from the oversampled training/validation dataset using the second stratified sampling process based on the value of the event assessment variable. The validation sample and the training sample are mutually exclusive […] Processor 110 executes instructions as understood by those skilled in the art […] Processor 110 operably couples with input interface 102, with output interface 104, with communication interface 106, and with computer-readable medium 108 to receive, to send, and to process information.”; Examiner’s note: wherein in the step (c), the processor (i.e., the processor 110) separates the remainder of the first-round labeled data into first-round verification data (i.e., a remainder of the input dataset allocated to a training/validation dataset and a validation sample selected from the training/validation dataset) is taught.)
“calculates performance of the first-round artificial intelligence model utilizing the first-round verification data” (Ma, Paragraphs [0006] and [0149], “(d) The trained predictive type model is validated using the selected validation sample to compute a validation criterion value for the trained predictive type model that quantifies a validation error […] a maximum validation criterion value may be used to determine the best predictive model of the plurality of predictive type models”; Examiner’s note: calculates performance of the first-round artificial intelligence model (i.e., computing a validation criterion value for the trained predictive type model. The trained predictive type model teaches the first-round artificial intelligence model) utilizing the first-round verification data (i.e., a validation criterion value. A maximum criterion value being used to determine the best predictive model) is taught.)
“determines whether performance of the first-round artificial intelligence model satisfies a preset minimum required performance” (Ma, Paragraph [0006], “A best predictive model is determined based on the stored, computed assessment criterion value of each validated predictive type model.”; Examiner’s note: determines whether performance of the first-round artificial intelligence model (i.e., a best predictive model being determined teaches the performance of the first-round artificial intelligence model) satisfies a preset minimum required performance (i.e., stored, computed assessment criterion value) is taught.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the apparatus and method of implementing batch-mode active learning for technology-assisted review of documents in Johnson, and the system and method for active machine learning in Deng, and the analytic system for machine learning prediction model selection as taught in Ma. Johnson teaches receiving the labeled data, generating the artificial intelligence model that automatically performs the data labeling, training an artificial neural network, determining whether a preset termination condition is satisfied and repeating the performing steps until the preset termination condition is satisfied. Deng teaches transmitting a plurality of source data to a worker terminal. Ma teaches separating the remainder of data into the verification data, calculating performance of the artificial intelligence model using the verification data and determining whether the performance of the artificial intelligence model satisfies a preset minimum required. One of ordinary skill would have motivation to combine Johnson, Deng and Ma to “determine the best predictive model of the selected predictive type model” (Ma, Paragraph [0127]).
Regarding Claim 7,
The combination of Johnson, Deng and Ma teaches:
“The automated training-based data labeling method of claim 4,” (preamble)
“wherein in the step (e), the processor separates the remainder of the nth-round labeled data into nth-round verification data” (Ma, Paragraphs [0005], [0006] and [0045], “A remainder of the input dataset is allocated to a training/validation dataset. The training/validation dataset is partitioned into an oversampled training/validation dataset using an oversampling process based on a predefined value of the event assessment variable. (a) A validation sample is selected from the oversampled training/validation dataset using a second stratified sampling process based on the value of the event assessment variable […] (c) A predictive type model is trained using the selected training sample […] (d) The trained predictive type model is validated using the selected validation sample to compute a validation criterion value […] (e) The validated predictive type model is scored using the selected assessment dataset to compute a probability value for each observation vector and to compute an assessment criterion value […] (f) The computed assessment criterion value, a hyperparameter, model characteristics that define a trained model of the predictive type model, and the computed probability value for each observation vector of the selected assessment dataset are stored to the computer-readable medium. (c) to (f) are repeated for at least one additional predictive type model […] Processor 110 executes instructions as understood by those skilled in the art […] Processor 110 operably couples with input interface 102, with output interface 104, with communication interface 106, and with computer-readable medium 108 to receive, to send, and to process information.”; Examiner’s note: wherein in the step (e), the processor (i.e., the processor 110) separates the remainder of the nth-round labeled data into nth-round verification data (i.e., a remainder of the input dataset allocated to a training/validation dataset, a validation sample selected from the training/validation dataset, computing a validation criterion value and assessment criterion value. Training and computing a validation criterion and assessment criterion value are repeated for at least one additional predictive type model) is taught.)
“calculates the performance of the first-round artificial intelligence model to the nth-round artificial intelligence model, respectively, utilizing all of the first-round verification data to the nth-round verification data” (Ma, Paragraphs [0006] and [0149], “(d) The trained predictive type model is validated using the selected validation sample to compute a validation criterion value for the trained predictive type model that quantifies a validation error. (e) The validated predictive type model is scored using the selected assessment dataset to compute a probability value for each observation vector and to compute an assessment criterion value […] (f) The computed assessment criterion value, a hyperparameter, model characteristics that define a trained model of the predictive type model, and the computed probability value for each observation vector of the selected assessment dataset are stored to the computer-readable medium. (c) to (f) are repeated for at least one additional predictive type model […] a maximum validation criterion value may be used to determine the best predictive model of the plurality of predictive type models”; Examiner’s note: calculates performance of the first-round artificial intelligence model to the nth-round artificial intelligence model, respectively, utilizing all of the first-round verification data to the nth-round verification data (i.e., computing a validation criterion value for the trained predictive type model and a maximum criterion value being used to determine the best predictive model. Training and computing a validation criterion and assessment criterion value are repeated for at least one additional predictive type model) is taught.)
“selects an artificial intelligence model with highest performance among the first-round artificial intelligence model to the nth-round artificial intelligence model” (Ma, Paragraph [0006], “(c) to (f) are repeated for at least one additional predictive type model. A best predictive model is determined based on the stored, computed assessment criterion value of each validated predictive type model.”; Examiner’s note: selects an artificial intelligence model with highest performance (i.e., a best predictive model being determined teaches selecting an artificial intelligence model with highest performance) among the first-round artificial intelligence model (i.e., the first predictive type model) to the nth-round artificial intelligence model (i.e., additional predictive type model) is taught.)
The reasons of obviousness have been noted in the rejection of Claim 4 above and applicable herein.
Regarding Claim 11,
The combination of Johnson and Deng teaches:
“The automated training-based data labeling method of claim 10,” (preamble)
The combination of Johnson and Deng does not explicitly teach:
“wherein in the step (c’), the processor separates the remainder of the first-round labeled data into first-round verification data”
“calculates performance of the prior artificial intelligence model and the first-round artificial intelligence model, respectively, utilizing the first-round verification data”
“selects an artificial intelligence model with higher performance among the prior artificial intelligence model and the first-round artificial intelligence model”
Ma teaches:
“wherein in the step (c’), the processor separates the remainder of the first-round labeled data into first-round verification data” (Ma, Paragraphs [0005], [0006] and [0045], “A remainder of the input dataset is allocated to a training/validation dataset. The training/validation dataset is partitioned into an oversampled training/validation dataset using an oversampling process based on a predefined value of the event assessment variable. (a) A validation sample is selected from the oversampled training/validation dataset using a second stratified sampling process based on the value of the event assessment variable. (b) A training sample is selected from the oversampled training/validation dataset using the second stratified sampling process based on the value of the event assessment variable. The validation sample and the training sample are mutually exclusive […] Processor 110 executes instructions as understood by those skilled in the art […] Processor 110 operably couples with input interface 102, with output interface 104, with communication interface 106, and with computer-readable medium 108 to receive, to send, and to process information.”; Examiner’s note: wherein in the step (c’), the processor (i.e., the processor 110) separates the remainder of the first-round labeled data into first-round verification data (i.e., a remainder of the input dataset allocated to a training/validation dataset and a validation sample selected from the training/validation dataset) is taught.)
“calculates performance of the prior artificial intelligence model and the first-round artificial intelligence model, respectively, utilizing the first-round verification data” (Ma, Paragraphs [0006] and [0149], “(d) The trained predictive type model is validated using the selected validation sample to compute a validation criterion value for the trained predictive type model that quantifies a validation error […] a maximum validation criterion value may be used to determine the best predictive model of the plurality of predictive type models […] (c) to (f) are repeated for at least one additional predictive type model”; Examiner’s note: calculates performance of the prior artificial intelligence model and the first-round artificial intelligence model (i.e., computing a validation criterion value for the trained predictive type model and repeating the process for at least one additional predictive type model. The trained predictive type model and at least one additional predictive type model teach the prior artificial intelligence model and the first-round artificial intelligence model), respectively, utilizing the first-round verification data (i.e., a validation criterion value. A maximum criterion value being used to determine the best predictive model) is taught.)
“selects an artificial intelligence model with higher performance among the prior artificial intelligence model and the first-round artificial intelligence model” (Ma, Paragraph [0006], “(c) to (f) are repeated for at least one additional predictive type model. A best predictive model is determined based on the stored, computed assessment criterion value of each validated predictive type model.”; Examiner’s note: selects an artificial intelligence model with higher performance (i.e., a best predictive model being determined teaches selecting an artificial intelligence model with higher performance) among the prior artificial intelligence model and the first-round artificial intelligence model (i.e., the prior predictive type model in additional predictive type models and the first predictive type model) is taught.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the apparatus and method of implementing batch-mode active learning for technology-assisted review of documents in Johnson, and the system and method for active machine learning in Deng, and the analytic system for machine learning prediction model selection as taught in Ma. Johnson teaches receiving the labeled data, generating the artificial intelligence model that automatically performs the data labeling, training an artificial neural network, determining whether a preset termination condition is satisfied and repeating the performing steps until the preset termination condition is satisfied. Deng teaches transmitting a plurality of source data to a worker terminal. Ma teaches separating the remainder of data into the verification data, calculating performance of the artificial intelligence model using the verification data and selecting an artificial intelligence model with higher performance. One of ordinary skill would have motivation to combine Johnson, Deng and Ma to “determine the best predictive model of the selected predictive type model” (Ma, Paragraph [0127]).
Regarding Claim 15,
The combination of Johnson, Deng and Ma teaches:
“The automated training-based data labeling method of claim 14,” (preamble)
“wherein in the step (e’), the processor separates the remainder of the mth-round labeled data into mth-round verification data” (Ma, Paragraphs [0005], [0006] and [0045], “A remainder of the input dataset is allocated to a training/validation dataset. The training/validation dataset is partitioned into an oversampled training/validation dataset using an oversampling process based on a predefined value of the event assessment variable. (a) A validation sample is selected from the oversampled training/validation dataset using a second stratified sampling process based on the value of the event assessment variable. (b) A training sample is selected from the oversampled training/validation dataset using the second stratified sampling process based on the value of the event assessment variable. The validation sample and the training sample are mutually exclusive […] Processor 110 executes instructions as understood by those skilled in the art […] Processor 110 operably couples with input interface 102, with output interface 104, with communication interface 106, and with computer-readable medium 108 to receive, to send, and to process information.”; Examiner’s note: wherein in the step (e’), the processor (i.e., the processor 110) separates the remainder of the mth-round labeled data into mth-round verification data (i.e., a remainder of the input dataset allocated to a training/validation dataset and a validation sample selected from the training/validation dataset) is taught.)
“calculates the performance of the prior artificial intelligence model, and the first-round artificial intelligence model to the mth-round artificial intelligence model, respectively, utilizing all of the first-round verification data to the mth-round verification data” (Ma, Paragraphs [0006] and [0149], “(d) The trained predictive type model is validated using the selected validation sample to compute a validation criterion value for the trained predictive type model that quantifies a validation error […] a maximum validation criterion value may be used to determine the best predictive model of the plurality of predictive type models […] (c) to (f) are repeated for at least one additional predictive type model”; Examiner’s note: calculates the performance of the prior artificial intelligence model, and the first-round artificial intelligence model to the mth-round artificial intelligence model (i.e., computing a validation criterion value for the trained predictive type model and repeating the process for at least one additional predictive type model. The trained predictive type model and at least one additional predictive type model teach the prior artificial intelligence model and the first-round artificial intelligence model to the mth-round artificial intelligence model), respectively, utilizing all of the first-round verification data to the mth-round verification data (i.e., a validation criterion value computed at each iteration. A maximum criterion value being used to determine the best predictive model) is taught.)
“selects an artificial intelligence model with highest performance among the prior artificial intelligence model, and the first-round artificial intelligence model to the mth-artificial intelligence model” (Ma, Paragraph [0006], “(c) to (f) are repeated for at least one additional predictive type model. A best predictive model is determined based on the stored, computed assessment criterion value of each validated predictive type model.”; Examiner’s note: selects an artificial intelligence model with highest performance (i.e., a best predictive model being determined teaches selecting an artificial intelligence model with highest performance) among the prior artificial intelligence model and the first-round artificial intelligence model to the mth-artificial intelligence model (i.e., the prior predictive type model in additional predictive type models and the first predictive type model to the mth additional predictive type model) is taught.)
The reasons of obviousness have been noted in the rejection of Claim 14 above and applicable herein.
Claims 5-6, 8-9, 12-13 and 16-17 are rejected under 35 U.S.C. 103 as being unpatentable over Johnson, in view of Deng, and further in view of Ma as applied in claim 4, in view of Verbeke (US 20230260257 A1).
Regarding Claim 5,
The combination of Johnson, Deng and Ma teaches:
“The automated training-based data labeling method of claim 4,” (preamble)
“wherein when the performance of the first-round artificial intelligence model satisfies the preset minimum required performance” (Ma, Paragraph [0006], “A best predictive model is determined based on the stored, computed assessment criterion value of each validated predictive type model.”; Examiner’s note: wherein when the performance of the first-round artificial intelligence model satisfies the preset minimum required performance (i.e., a best predictive model being determined teaches the performance of the first-round artificial intelligence model) satisfies the preset minimum required performance (i.e., stored, computed assessment criterion value) is taught.)
The combination of Johnson, Deng and Ma does not explicitly teach:
“the processor designates the first-round artificial intelligence model as a preprocessing engine and transmits the preprocessing engine to the worker terminal in order to allow the worker to utilize automated data labeling performed by the preprocessing engine when the worker performs data labeling with the worker terminal”
Verbeke teaches:
“the processor designates the first-round artificial intelligence model as a preprocessing engine and transmits the preprocessing engine to the worker terminal in order to allow the worker to utilize automated data labeling performed by the preprocessing engine when the worker performs data labeling with the worker terminal” (Verbeke, Paragraphs [0029], [0031], [0044], [0046], [0051] and [0072], “ii) transmitting the annotated dataset to a remote entity comprising a second machine learning algorithm configured to use the annotated dataset as an input dataset in order to generate an output dataset comprising predictions of the one or more features […] iv) re-annotating the erroneous dataset so to form a re-annotated dataset […] the annotating entity will “pre-annotate” a dataset using a machine-learning algorithm configured to annotate images […] Thereby it is possible to obtain an automated and efficient evaluation of the annotations, which in turn enables for a better refinement of training datasets for machine-learning algorithms, in a cost effective manner […] the first machine-learning algorithm is configured to annotate the plurality of images (i.e. to label the images so that a “machine” can use it) by generating annotations for one or more features comprised in the plurality of images […] there is provided a non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of a processing system”; Examiner’s note: the processor (i.e., one or more processors of a processing system) designates the first-round artificial intelligence model as a preprocessing engine (i.e., pre-annotating a dataset using a machine-learning algorithm) and transmits the preprocessing engine to the worker terminal (i.e., transmitting the annotated dataset created by the first machine-learning algorithm to a remote entity comprising a second machine learning algorithm) in order to allow the worker to utilize automated data labeling (i.e., automated and efficient evaluation of the annotations) performed by the preprocessing engine (the first machine-learning algorithm) when the worker performs data labeling with the worker terminal (re-annotating using a second machine learning algorithm) is taught.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the present application to combine the apparatus and method of implementing batch-mode active learning for technology-assisted review of documents in Johnson, and the system and method for active machine learning in Deng, and the analytic system for machine learning prediction model selection in Ma, and the iterative refinement of annotated datasets as taught in Verbeke. Johnson teaches receiving the labeled data, generating the artificial intelligence model that automatically performs the data labeling, training an artificial neural network, determining whether a preset termination condition is satisfied and repeating the performing steps until the preset termination condition is satisfied. Deng teaches transmitting a plurality of source data to a worker terminal. Ma teaches separating the remainder of data into the verification data, calculating performance of the artificial intelligence model using the verification data and determining whether the performance of the artificial intelligence model satisfies a preset minimum required. Verbeke teaches designating the first-round artificial intelligence model as a preprocessing engine and transmitting the preprocessing engine to the worker terminal in order to allow the worker to utilize automated data labeling performed by the preprocessing engine when the worker performs data labeling with the worker terminal. One of ordinary skill would have motivation to combine Johnson, Deng, Ma and Verbeke to “improve the datasets used for training the machine-learning algorithms in a more efficient manner as compared to conventional solutions” (Verbeke, Paragraph [0045]).
Regarding Claim 6,
The combination of Johnson, Deng, Ma and Verbeke teaches:
“The automated training-based data labeling method of claim 4,” (preamble)
“wherein when the performance of the first-round artificial intelligence model does not satisfy the preset minimum required performance, the processor does not designate the first-round artificial intelligence model as a preprocessing engine” (Johnson, Col. 9, Lines 24 and 30-31, “the active learning process stops: […] when the entropy of each selected sample or error on prediction is less than a threshold”; Johnson, Col.20, Lines 30-32, “the apparatus's processor 606 can execute process-executable instructions stored in the memory 608 to enable the operations”; Examiner’s note: wherein when the performance of the first-round artificial intelligence model does not satisfy the preset minimum required performance (i.e., the entropy of each selected sample or error on prediction is less than a threshold), the processor (i.e., the apparatus’s processor 606) does not designate the first-round artificial intelligence model as a preprocessing engine (i.e., the active learning process being stopped teaches the processor does not designate the first-round artificial intelligence model as a preprocessing engine) is taught.)
The reasons of obviousness have been noted in the rejection of Claim 4 above and applicable herein.
Regarding Claim 8,
The combination of Johnson, Deng, Ma and Verbeke teaches:
“The automated training-based data labeling method of claim 7,” (preamble)
“wherein the processor determines whether performance of the selected artificial intelligence model satisfies the preset minimum required performance” (Ma, Paragraphs [0006] and [0045], “A best predictive model is determined based on the stored, computed assessment criterion value of each validated predictive type model […] Processor 110 executes instructions as understood by those skilled in the art […] Processor 110 operably couples with input interface 102, with output interface 104, with communication interface 106, and with computer-readable medium 108 to receive, to send, and to process information.”; Examiner’s note: wherein the processor (i.e., the processor 110) determines whether performance of the selected artificial intelligence model (i.e., a best predictive model being determined teaches the performance of the selected artificial intelligence model) satisfies the preset minimum required performance (i.e., stored, computed assessment criterion value) is taught.)
“when the performance of the selected artificial intelligence model satisfies the preset minimum required performance” (Ma, Paragraph [0006], “A best predictive model is determined based on the stored, computed assessment criterion value of each validated predictive type model.”; Examiner’s note: when the performance of the selected artificial intelligence model satisfies the preset minimum required performance (i.e., a best predictive model being determined teaches the performance of the selected artificial intelligence model) satisfies the preset minimum required performance (i.e., stored, computed assessment criterion value) is taught.)
“the processor designates the selected artificial intelligence model as a preprocessing engine and transmits the preprocessing engine to the worker terminal in order to allow the worker to utilize automated data labeling performed by the preprocessing engine when the worker performs data labeling with the worker terminal” (Verbeke, Paragraphs [0029], [0031], [0044], [0046], [0051] and [0072], “ii) transmitting the annotated dataset to a remote entity comprising a second machine learning algorithm configured to use the annotated dataset as an input dataset in order to generate an output dataset comprising predictions of the one or more features […] iv) re-annotating the erroneous dataset so to form a re-annotated dataset […] the annotating entity will “pre-annotate” a dataset using a machine-learning algorithm configured to annotate images […] Thereby it is possible to obtain an automated and efficient evaluation of the annotations, which in turn enables for a better refinement of training datasets for machine-learning algorithms, in a cost effective manner […] the first machine-learning algorithm is configured to annotate the plurality of images (i.e. to label the images so that a “machine” can use it) by generating annotations for one or more features comprised in the plurality of images […] there is provided a non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of a processing system”; Examiner’s note: the processor (i.e., one or more processors of a processing system) designates the selected artificial intelligence model as a preprocessing engine (i.e., pre-annotating a dataset using a machine-learning algorithm) and transmits the preprocessing engine to the worker terminal (i.e., transmitting the annotated dataset created by the first machine-learning algorithm to a remote entity comprising a second machine learning algorithm) in order to allow the worker to utilize automated data labeling (i.e., automated and efficient evaluation of the annotations) performed by the preprocessing engine (the first machine-learning algorithm) when the worker performs data labeling with the worker terminal (re-annotating using a second machine learning algorithm) is taught.)
The reasons of obviousness have been noted in the rejection of Claim 7 above and applicable herein.
Regarding Claim 9,
The combination of Johnson, Deng, Ma and Verbeke teaches:
“The automated training-based data labeling method of claim 7,” (preamble)
“wherein the processor determines whether performance of the selected artificial intelligence model satisfies the preset minimum required performance” (Ma, Paragraphs [0006] and [0045], “A best predictive model is determined based on the stored, computed assessment criterion value of each validated predictive type model […] Processor 110 executes instructions as understood by those skilled in the art […] Processor 110 operably couples with input interface 102, with output interface 104, with communication interface 106, and with computer-readable medium 108 to receive, to send, and to process information.”; Examiner’s note: wherein the processor (i.e., the processor 110) determines whether performance of the selected artificial intelligence model (i.e., a best predictive model being determined teaches the performance of the selected artificial intelligence model) satisfies the preset minimum required performance (i.e., stored, computed assessment criterion value) is taught.)
“the processor does not designate the selected artificial intelligence model as a preprocessing engine when the performance of the selected artificial intelligence model does not satisfy the preset minimum required performance” (Johnson, Col. 9, Lines 24 and 30-31, “the active learning process stops: […] when the entropy of each selected sample or error on prediction is less than a threshold”; Johnson, Col.20, Lines 30-32, “the apparatus's processor 606 can execute process-executable instructions stored in the memory 608 to enable the operations”; Examiner’s note: the processor (i.e., the apparatus’s processor 606) does not designate the selected artificial intelligence model as a preprocessing engine (i.e., the active learning process being stopped teaches the processor does not designate the selected artificial intelligence model as a preprocessing engine) when the performance of the selected artificial intelligence model does not satisfy the preset minimum required performance (i.e., the entropy of each selected sample or error on prediction is less than a threshold), is taught.)
The reasons of obviousness have been noted in the rejection of Claim 7 above and applicable herein.
Regarding Claim 12,
The combination of Johnson, Deng, Ma and Verbeke teaches:
“The automated training-based data labeling method of claim 11,” (preamble)
“wherein the processor determines whether the performance of the selected artificial intelligence model satisfies a preset minimum required performance” (Ma, Paragraphs [0006] and [0045], “A best predictive model is determined based on the stored, computed assessment criterion value of each validated predictive type model […] Processor 110 executes instructions as understood by those skilled in the art […] Processor 110 operably couples with input interface 102, with output interface 104, with communication interface 106, and with computer-readable medium 108 to receive, to send, and to process information.”; Examiner’s note: wherein the processor (i.e., the processor 110) determines whether the performance of the selected artificial intelligence model (i.e., a best predictive model being determined teaches the performance of the selected artificial intelligence model) satisfies a preset minimum required performance (i.e., stored, computed assessment criterion value) is taught.)
“when the performance of the selected artificial intelligence model satisfies the preset minimum required performance” (Ma, Paragraph [0006], “A best predictive model is determined based on the stored, computed assessment criterion value of each validated predictive type model.”; Examiner’s note: when the performance of the selected artificial intelligence model satisfies the preset minimum required performance (i.e., a best predictive model being determined teaches the performance of the selected artificial intelligence model) satisfies the preset minimum required performance (i.e., stored, computed assessment criterion value) is taught.)
“the processor designates the selected artificial intelligence model as a preprocessing engine and transmits the preprocessing engine to the worker terminal in order to allow the worker to utilize automated data labeling performed by the preprocessing engine when the worker performs data labeling with the worker terminal” (Verbeke, Paragraphs [0029], [0031], [0044], [0046], [0051] and [0072], “ii) transmitting the annotated dataset to a remote entity comprising a second machine learning algorithm configured to use the annotated dataset as an input dataset in order to generate an output dataset comprising predictions of the one or more features […] iv) re-annotating the erroneous dataset so to form a re-annotated dataset […] the annotating entity will “pre-annotate” a dataset using a machine-learning algorithm configured to annotate images […] Thereby it is possible to obtain an automated and efficient evaluation of the annotations, which in turn enables for a better refinement of training datasets for machine-learning algorithms, in a cost effective manner […] the first machine-learning algorithm is configured to annotate the plurality of images (i.e. to label the images so that a “machine” can use it) by generating annotations for one or more features comprised in the plurality of images […] there is provided a non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of a processing system”; Examiner’s note: the processor (i.e., one or more processors of a processing system) designates the selected artificial intelligence model as a preprocessing engine (i.e., pre-annotating a dataset using a machine-learning algorithm) and transmits the preprocessing engine to the worker terminal (i.e., transmitting the annotated dataset created by the first machine-learning algorithm to a remote entity comprising a second machine learning algorithm) in order to allow the worker to utilize automated data labeling (i.e., automated and efficient evaluation of the annotations) performed by the preprocessing engine (the first machine-learning algorithm) when the worker performs data labeling with the worker terminal (re-annotating using a second machine learning algorithm) is taught.)
The reasons of obviousness have been noted in the rejection of Claim 11 above and applicable herein.
Regarding Claim 13,
The combination of Johnson, Deng, Ma and Verbeke teaches:
“The automated training-based data labeling method of claim 11,” (preamble)
“wherein the processor determines whether the performance of the selected artificial intelligence model satisfies a preset minimum required performance” (Ma, Paragraphs [0006] and [0045], “A best predictive model is determined based on the stored, computed assessment criterion value of each validated predictive type model […] Processor 110 executes instructions as understood by those skilled in the art […] Processor 110 operably couples with input interface 102, with output interface 104, with communication interface 106, and with computer-readable medium 108 to receive, to send, and to process information.”; Examiner’s note: wherein the processor (i.e., the processor 110) determines whether the performance of the selected artificial intelligence model (i.e., a best predictive model being determined teaches the performance of the selected artificial intelligence model) satisfies a preset minimum required performance (i.e., stored, computed assessment criterion value) is taught.)
“the processor does not designate the selected artificial intelligence model as a preprocessing engine when the performance of the selected artificial intelligence model does not satisfy the preset minimum required performance” (Johnson, Col. 9, Lines 24 and 30-31, “the active learning process stops: […] when the entropy of each selected sample or error on prediction is less than a threshold”; Johnson, Col.20, Lines 30-32, “the apparatus's processor 606 can execute process-executable instructions stored in the memory 608 to enable the operations”; Examiner’s note: the processor (i.e., the apparatus’s processor 606) does not designate the selected artificial intelligence model as a preprocessing engine (i.e., the active learning process being stopped teaches the processor does not designate the selected artificial intelligence model as a preprocessing engine) when the performance of the selected artificial intelligence model does not satisfy the preset minimum required performance (i.e., the entropy of each selected sample or error on prediction is less than a threshold), is taught.)
The reasons of obviousness have been noted in the rejection of Claim 11 above and applicable herein.
Regarding Claim 16,
The combination of Johnson, Deng, Ma and Verbeke teaches:
“The automated training-based data labeling method of claim 15,” (preamble)
“wherein the processor determines whether the performance of the selected artificial intelligence model satisfies a preset minimum required performance” (Ma, Paragraphs [0006] and [0045], “A best predictive model is determined based on the stored, computed assessment criterion value of each validated predictive type model […] Processor 110 executes instructions as understood by those skilled in the art […] Processor 110 operably couples with input interface 102, with output interface 104, with communication interface 106, and with computer-readable medium 108 to receive, to send, and to process information.”; Examiner’s note: wherein the processor (i.e., the processor 110) determines whether the performance of the selected artificial intelligence model (i.e., a best predictive model being determined teaches the performance of the selected artificial intelligence model) satisfies a preset minimum required performance (i.e., stored, computed assessment criterion value) is taught.)
“when the performance of the selected artificial intelligence model satisfies the preset minimum required performance” (Ma, Paragraph [0006], “A best predictive model is determined based on the stored, computed assessment criterion value of each validated predictive type model.”; Examiner’s note: when the performance of the selected artificial intelligence model satisfies the preset minimum required performance (i.e., a best predictive model being determined teaches the performance of the selected artificial intelligence model) satisfies the preset minimum required performance (i.e., stored, computed assessment criterion value) is taught.)
“the processor designates the selected artificial intelligence model as a preprocessing engine and transmits the preprocessing engine to the worker terminal in order to allow the worker to utilize automated data labeling performed by the preprocessing engine when the worker performs data labeling with the worker terminal” (Verbeke, Paragraphs [0029], [0031], [0044], [0046], [0051] and [0072], “ii) transmitting the annotated dataset to a remote entity comprising a second machine learning algorithm configured to use the annotated dataset as an input dataset in order to generate an output dataset comprising predictions of the one or more features […] iv) re-annotating the erroneous dataset so to form a re-annotated dataset […] the annotating entity will “pre-annotate” a dataset using a machine-learning algorithm configured to annotate images […] Thereby it is possible to obtain an automated and efficient evaluation of the annotations, which in turn enables for a better refinement of training datasets for machine-learning algorithms, in a cost effective manner […] the first machine-learning algorithm is configured to annotate the plurality of images (i.e. to label the images so that a “machine” can use it) by generating annotations for one or more features comprised in the plurality of images […] there is provided a non-transitory computer-readable storage medium storing one or more programs configured to be executed by one or more processors of a processing system”; Examiner’s note: the processor (i.e., one or more processors of a processing system) designates the selected artificial intelligence model as a preprocessing engine (i.e., pre-annotating a dataset using a machine-learning algorithm) and transmits the preprocessing engine to the worker terminal (i.e., transmitting the annotated dataset created by the first machine-learning algorithm to a remote entity comprising a second machine learning algorithm) in order to allow the worker to utilize automated data labeling (i.e., automated and efficient evaluation of the annotations) performed by the preprocessing engine (the first machine-learning algorithm) when the worker performs data labeling with the worker terminal (re-annotating using a second machine learning algorithm) is taught.)
The reasons of obviousness have been noted in the rejection of Claim 15 above and applicable herein.
Regarding Claim 17,
The combination of Johnson, Deng, Ma and Verbeke teaches:
“The automated training-based data labeling method of claim 15,” (preamble)
“wherein the processor determines whether performance of the selected artificial intelligence model satisfies a preset minimum required performance” (Ma, Paragraphs [0006] and [0045], “A best predictive model is determined based on the stored, computed assessment criterion value of each validated predictive type model […] Processor 110 executes instructions as understood by those skilled in the art […] Processor 110 operably couples with input interface 102, with output interface 104, with communication interface 106, and with computer-readable medium 108 to receive, to send, and to process information.”; Examiner’s note: wherein the processor (i.e., the processor 110) determines whether performance of the selected artificial intelligence model (i.e., a best predictive model being determined teaches the performance of the selected artificial intelligence model) satisfies a preset minimum required performance (i.e., stored, computed assessment criterion value) is taught.)
“the processor does not designate the selected artificial intelligence model as a preprocessing engine when the performance of the selected artificial intelligence model does not satisfy the preset minimum required performance” (Johnson, Col. 9, Lines 24 and 30-31, “the active learning process stops: […] when the entropy of each selected sample or error on prediction is less than a threshold”; Johnson, Col.20, Lines 30-32, “the apparatus's processor 606 can execute process-executable instructions stored in the memory 608 to enable the operations”; Examiner’s note: the processor (i.e., the apparatus’s processor 606) does not designate the selected artificial intelligence model as a preprocessing engine (i.e., the active learning process being stopped teaches the processor does not designate the selected artificial intelligence model as a preprocessing engine) when the performance of the selected artificial intelligence model does not satisfy the preset minimum required performance (i.e., the entropy of each selected sample or error on prediction is less than a threshold), is taught.)
The reasons of obviousness have been noted in the rejection of Claim 15 above and applicable herein.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to YONG D RHO whose telephone number is (571)270-0194. The examiner can normally be reached 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Viker Lamardo can be reached at 5712705871. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/YONG DOO RHO/Examiner, Art Unit 2147
/ERIC NILSSON/Primary Examiner, Art Unit 2151
Prosecution Insights