Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statements (IDS) submitted on 05/31/2023 and 08/04/2023 are incompliance with provisions of 37 CFR 1.97. Accordingly, the information disclosure statements are being considered by the examiner.
Priority
The present application claims priority to International Application No. PCT/CN2022/135618, filed on 11/30/2022.
A certified copy of International Application No. PCT/CN2022/135618 in English has been received (on 07/18/2023), as required by 37 CFR 1.55.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1 – 20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The following sections following the 2019 PEG guidelines for analyzing subject matter eligibility.
The analysis below of the claims’ subject matter eligibility follows the 2019 Revised
Patent Subject Matter Eligibility Guidance, 84 Fed. Reg. 50-57 (January 7, 2019) (“2019 PEG”)
and the 2024 Guidance Update on Patent Subject Matter Eligibility, Including on Artificial Intelligence, 89 Fed. Reg. 58128-58138 (July 17, 2024) (“2024 AI SME Update”).
When considering subject matter eligibility under 35 U.S.C. 101, it must be determined
whether the claim is directed to one of the four statutory categories of invention, 1.e., process,
machine, manufacture, or composition of matter (Step 1). If the claim does fall within one of the
statutory categories, the second step in the analysis is to determine whether the claim is directed
to a judicial exception (Step 2A). The Step 2A analysis is broken into two prongs. In the first
prong (Step 2A, Prong 1), it is determined whether or not the claims recite a judicial exception
(e.g., mathematical concepts, mental processes, certain methods of organizing human activity). If
it is determined in Step 2A, Prong 1 that the claims recite a judicial exception, the analysis
proceeds to the second prong (Step 2A, Prong 2), where it is determined whether or not the
claims integrate the judicial exception into a practical application. If it is determined at step 2A,
Prong 2 that the claims do not integrate the judicial exception into a practical application, the
analysis proceeds to determining whether the claim is a patent-eligible application of the
exception (Step 2B). If an abstract idea is present in the claim, any element or combination of
elements in the claim must be sufficient to ensure that the claim integrates the judicial exception
into a practical application, or else amounts to significantly more than the abstract idea itself.
Claim 1
Step 1: The claim recites a method, which is one of the four statutory categories of eligible matter.
Step 2A Prong 1:
selecting a first subset of data from the dataset by applying one or more clustering algorithms to the labelled data in the dataset; (Mathematical Concepts: are defined as mathematical relationships, mathematical formulas or equations, or mathematical calculations.)
applying annotation to the labelled data in the first subset to refine the labels on the data in the first subset, wherein the annotation includes, at least in part, implementation of historical information for the data; (Mental Processes: Can be performed in the human mind, or by a human using a pen and paper, making observations, evaluations and judgments as claimed)
selecting a second subset of labelled data from the dataset, wherein the labels on the data in the second subset of data correspond to the labels on the data in the first subset after annotation; (Mental Processes: Can be performed in the human mind, or by a human using a pen and paper, making observations, evaluations and judgments as claimed)
selecting a portion of the data from the second subset to add to the first subset, wherein the portion of the data is selected based on the data with a given label in the portion having a measure of similarity, with respect to the data with the same given label in the first subset, that satisfies a predetermined threshold; (Mental Processes: Can be performed in the human mind, or by a human using a pen and paper, making observations, evaluations and judgments as claimed)
adding the portion of the data selected to the first subset to generate a training dataset; (Mental Processes: Can be performed in the human mind, or by a human using a pen and paper, making observations, evaluations and judgments as claimed)
Step 2A Prong 2: The judicial exceptions are not integrated into a practical application. In
particular, the claim recites these additional elements:
accessing, by a computer system, a dataset comprising a plurality of data, wherein the data is labelled corresponding to a plurality of categories; (Mere data gathering, Insignificant extra solution activity in MPEP § 2106.05(g))
providing the training dataset to a machine learning algorithm for training of the machine learning algorithm. (Mere data gathering, Insignificant extra solution activity in MPEP § 2106.05(g))
Step 2B: The claim does not include additional elements that are sufficient to amount to
significantly more than the judicial exception.
accessing, by a computer system, a dataset comprising a plurality of data, wherein the data is labelled corresponding to a plurality of categories; (receiving or transmitting data, using components and functions claimed at a high level of generality have been determined by the courts as being well-understood, routine, and conventional activities in the field of computer functions (See MPEP § 2106.05(d)(II)(i))
providing the training dataset to a machine learning algorithm for training of the machine learning algorithm. (receiving or transmitting data, using components and functions claimed at a high level of generality have been determined by the courts as being well-understood, routine, and conventional activities in the field of computer functions (See MPEP § 2106.05(d)(II)(i))
As an ordered whole, the claim is directed to a method of clustering datasets into subsets, this is nothing more than using machine learning models to group the provided data. Nothing in the claim provides significantly more than this. As such, the claim is not patent eligible.
Claim 2 incorporates the rejection of claim 1.
Step 1: The claim recites a method, which is one of the four statutory categories of eligible matter.
Step 2A Prong 1; The judicial exception of claim 1 are incorporated.
wherein the data with the given label in the portion having the measure of similarity satisfying the predetermined threshold indicates that the data in the portion with the given label has a nearest neighbor ranking for similarity, to the data with the same given label in the first subset, that satisfies a predetermined ranking threshold. (Mental Processes: Can be performed in the human mind, or by a human using a pen and paper, making observations, evaluations and judgments as claimed)
Step 2A Prong 2: The judicial exceptions are not integrated into a practical application. In particular, the claim recites these additional elements:
The claim does not recite any additional limitations. Therefore, there are no additional elements to integrate the abstract ideas into a practical application. (Merely asserting that a judicial exceptions is to be carried out on a generic computer (i.e. “a computer system” of base claim 1) cannot meaningfully integrate the judicial exceptions into a practical application. See MPEP § 2106.05(f))
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exceptions.
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. Mere instructions to apply an exception (i.e., “a computer system” of base claim 1) cannot provide an inventive concept. The claim is not patent eligible.
Claim 3 incorporates the rejection of claim 1.
Step 1: The claim recites a method, which is one of the four statutory categories of eligible matter.
Step 2A Prong 1; The judicial exception of claim 1 are incorporated.
determining, for the given label, similarity values between the data with the given label in the second subset and the data with the given label in the first subset; (Mental Processes: Can be performed in the human mind, or by a human using a pen and paper, making observations, evaluations and judgments as claimed)
determining data from the second subset to include in the portion of the data based on the data with the given label in the second subset having a similarity value that satisfies the predetermined threshold. (Mental Processes: Can be performed in the human mind, or by a human using a pen and paper, making observations, evaluations and judgments as claimed)
Step 2A Prong 2: The judicial exceptions are not integrated into a practical application. In
particular, the claim recites these additional elements:
The claim does not recite any additional limitations. Therefore, there are no additional elements to integrate the abstract ideas into a practical application. (Merely asserting that a judicial exceptions is to be carried out on a generic computer (i.e. “a computer system” of base claim 1) cannot meaningfully integrate the judicial exceptions into a practical application. See MPEP § 2106.05(f))
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exceptions.
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. Mere instructions to apply an exception (i.e., “evaluating subsets meeting thresholds” of base claim 1) cannot provide an inventive concept. The claim is not patent eligible.
Claim 4 incorporates the rejection of claim 1.
Step 1: The claim recites a method, which is one of the four statutory categories of eligible matter.
Step 2A Prong 1; The judicial exception of claim 1 are incorporated.
determining, for the given label, a ranking of similarity between the data with the given label in the second subset and the data with the given label in the first subset; (Mental Processes: Can be performed in the human mind, or by a human using a pen and paper, making observations, evaluations and judgments as claimed)
determining data from the second subset to include in the portion of the data based on the ranking of similarity for the data with the given label in the second subset satisfying the predetermined threshold, wherein the predetermined threshold is a threshold for the ranking of similarity. (Mental Processes: Can be performed in the human mind, or by a human using a pen and paper, making observations, evaluations and judgments as claimed)
Step 2A Prong 2: The judicial exceptions are not integrated into a practical application. In
particular, the claim recites these additional elements:
The claim does not recite any additional limitations. Therefore, there are no additional elements to integrate the abstract ideas into a practical application. (Merely asserting that a judicial exceptions is to be carried out on a generic computer (i.e. “a computer system” of base claim 1) cannot meaningfully integrate the judicial exceptions into a practical application. See MPEP § 2106.05(f))
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exceptions.
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. Mere instructions to apply an exception (i.e., “a computer system” of base claim 1) cannot provide an inventive concept. The claim is not patent eligible.
Claim 5 incorporates the rejection of claim 1.
Step 1: The claim recites a method, which is one of the four statutory categories of eligible matter.
Step 2A Prong 1; The judicial exception of claim 1 are incorporated.
determining, for every individual label present in both the first subset and the second subset, a ranking of similarity between the data with an individual label in the second subset and the data with the individual label in the first subset; (Mental Processes: Can be performed in the human mind, or by a human using a pen and paper, making observations, evaluations and judgments as claimed)
determining, for every individual label present in both the first subset and the second subset, data from the second subset to include in the portion of the data, wherein the data to be included is determined based on the ranking of similarity for the data with the individual label in the second subset satisfying the predetermined threshold, the predetermined threshold being a threshold for the ranking of similarity. (Mental Processes: Can be performed in the human mind, or by a human using a pen and paper, making observations, evaluations and judgments as claimed)
Step 2A Prong 2: The judicial exceptions are not integrated into a practical application. In
particular, the claim recites these additional elements:
The claim does not recite any additional limitations. Therefore, there are no additional elements to integrate the abstract ideas into a practical application. (Merely asserting that a judicial exceptions is to be carried out on a generic computer (i.e. “a computer system” of base claim 1) cannot meaningfully integrate the judicial exceptions into a practical application. See MPEP § 2106.05(f))
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exceptions.
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. Mere instructions to apply an exception (i.e., “a computer system” of base claim 1) cannot provide an inventive concept. The claim is not patent eligible.
Claim 6 incorporates the rejection of claim 1.
Step 1: The claim recites a method, which is one of the four statutory categories of eligible matter.
Step 2A Prong 1; The judicial exception of claim 1 are incorporated.
wherein the second subset of labelled data is selected from the dataset by determining additional data from the dataset that has labels that are identical to the labels on the data in the first subset after annotation. (Mental Processes: Can be performed in the human mind, or by a human using a pen and paper, making observations, evaluations and judgments as claimed)
Step 2A Prong 2: The judicial exceptions are not integrated into a practical application. In
particular, the claim recites these additional elements:
The claim does not recite any additional limitations. Therefore, there are no additional elements to integrate the abstract ideas into a practical application. (Merely asserting that a judicial exceptions is to be carried out on a generic computer (i.e. “a computer system” of base claim 1) cannot meaningfully integrate the judicial exceptions into a practical application. See MPEP § 2106.05(f))
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exceptions.
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. Mere instructions to apply an exception (i.e., “a computer system” of base claim 1) cannot provide an inventive concept. The claim is not patent eligible.
Claim 7 incorporates the rejection of claim 1.
Step 1: The claim recites a method, which is one of the four statutory categories of eligible matter.
Step 2A Prong 1; The judicial exception of claim 1 are incorporated.
wherein applying the annotation to the labelled data includes determining an agreement in the labels on the data in the first subset after implementation of the historical information for the data. (Mental Processes: Can be performed in the human mind, or by a human using a pen and paper, making observations, evaluations and judgments as claimed)
Step 2A Prong 2: The judicial exceptions are not integrated into a practical application. In
particular, the claim recites these additional elements:
The claim does not recite any additional limitations. Therefore, there are no additional elements to integrate the abstract ideas into a practical application. (Merely asserting that a judicial exceptions is to be carried out on a generic computer (i.e. “a computer system” of base claim 1) cannot meaningfully integrate the judicial exceptions into a practical application. See MPEP § 2106.05(f))
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exceptions.
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. Mere instructions to apply an exception (i.e., “a computer system” of base claim 1) cannot provide an inventive concept. The claim is not patent eligible.
Claim 8 incorporates the rejection of claim 1.
Step 1: The claim recites a method, which is one of the four statutory categories of eligible matter.
Step 2A Prong 1; The judicial exception of claim 1 are incorporated.
determining a set of nearest neighbors to a given item of data in the first subset based on data in the given item of data and data in the nearest neighbors, wherein the set of nearest neighbors includes a set of nearest items of data based on similarities between the data in the nearest neighbors and the data in the given item of data; (Mental Processes: Can be performed in the human mind, or by a human using a pen and paper, making observations, evaluations and judgments as claimed)
determining a number of nearest neighbors in the set of nearest neighbors that have labels identical to a label on the data in the given item of data; (Mental Processes: Can be performed in the human mind, or by a human using a pen and paper, making observations, evaluations and judgments as claimed)
retaining the given item of data in the first subset when the number of nearest neighbors that have identical labels satisfies a predetermined threshold for a minimum number of nearest neighbors having the same label; (Mental Processes: Can be performed in the human mind, or by a human using a pen and paper, making observations, evaluations and judgments as claimed)
removing the given item of data from the first subset when the number of nearest neighbors that have identical labels fails to satisfy the predetermined threshold for the minimum number of nearest neighbors having the same label. (Mental Processes: Can be performed in the human mind, or by a human using a pen and paper, making observations, evaluations and judgments as claimed)
Step 2A Prong 2: The judicial exceptions are not integrated into a practical application. In
particular, the claim recites these additional elements:
The claim does not recite any additional limitations. Therefore, there are no additional elements to integrate the abstract ideas into a practical application. (Merely asserting that a judicial exceptions is to be carried out on a generic computer (i.e. “a computer system” of base claim 1) cannot meaningfully integrate the judicial exceptions into a practical application. See MPEP § 2106.05(f))
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exceptions.
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. Mere instructions to apply an exception (i.e., “identifying similar data points” of base claim 1) cannot provide an inventive concept. The claim is not patent eligible.
Claim 9 incorporates the rejection of claim 1.
Step 1: The claim recites a method, which is one of the four statutory categories of eligible matter.
Step 2A Prong 1; The judicial exception of claim 1 are incorporated.
wherein the plurality of data in the dataset includes text data, and wherein at least one of the categories applied to the data is an intent category. (Mental Processes: Can be performed in the human mind, or by a human using a pen and paper, making observations, evaluations and judgments as claimed)
Step 2A Prong 2: The judicial exceptions are not integrated into a practical application. In
particular, the claim recites these additional elements:
The claim does not recite any additional limitations. Therefore, there are no additional elements to integrate the abstract ideas into a practical application. (Merely asserting that a judicial exceptions is to be carried out on a generic computer (i.e. “identifying data categories in a dataset” of base claim 1) cannot meaningfully integrate the judicial exceptions into a practical application. See MPEP § 2106.05(f))
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exceptions.
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. Mere instructions to apply an exception (i.e., “identifying data categories in a dataset” of base claim 1) cannot provide an inventive concept. The claim is not patent eligible.
Claim 10 incorporates the rejection of claim 1.
Step 1: The claim recites a method, which is one of the four statutory categories of eligible matter.
Step 2A Prong 1; The judicial exception of claim 1 are incorporated.
wherein at least one item of data in the first subset is labelled with a mislabeled category, and wherein applying the annotation to the first subset corrects the mislabeled category. (Mental Processes: Can be performed in the human mind, or by a human using a pen and paper, making observations, evaluations and judgments as claimed)
Step 2A Prong 2: The judicial exceptions are not integrated into a practical application. In
particular, the claim recites these additional elements:
The claim does not recite any additional limitations. Therefore, there are no additional elements to integrate the abstract ideas into a practical application. (Merely asserting that a judicial exceptions is to be carried out on a generic computer (i.e. “a computer system” of base claim 1) cannot meaningfully integrate the judicial exceptions into a practical application. See MPEP § 2106.05(f))
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exceptions.
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. Mere instructions to apply an exception (i.e., “a computer system” of base claim 1) cannot provide an inventive concept. The claim is not patent eligible.
Claim 11
Step 1: The claim recites a method, which is one of the four statutory categories of eligible matter.
Step 2A Prong 1:
selecting a subset of labelled data from the dataset, wherein the labels on the data in the subset of data correspond to the annotated labels on the data in the training dataset; (Mental Processes: Can be performed in the human mind, or by a human using a pen and paper, making observations, evaluations and judgments as claimed)
selecting a portion of the data from the subset to add to the training dataset, wherein the portion of the data is selected based on the data with a given label in the portion having a measure of similarity, with respect to the data with the same given label in the training dataset, that satisfies a predetermined threshold; (Mental Processes: Can be performed in the human mind, or by a human using a pen and paper, making observations, evaluations and judgments as claimed)
updating the training dataset by adding the portion of the data selected to the training dataset; (Mental Processes: Can be performed in the human mind, or by a human using a pen and paper, making observations, evaluations and judgments as claimed)
Step 2A Prong 2: The judicial exceptions are not integrated into a practical application. In
particular, the claim recites these additional elements:
receiving, by a computer system, an indication to update a training dataset for a machine learning algorithm; (Mere data gathering, Insignificant extra solution activity in MPEP § 2106.05(g))
accessing, by the computer system in response to the indication, data in the training dataset for the machine learning algorithm, wherein the training dataset includes annotated labels on the data; (Mere data gathering, Insignificant extra solution activity in MPEP § 2106.05(g))
accessing, by the computer system, a dataset comprising a plurality of data, wherein the data is labelled corresponding to a plurality of categories; (Mere data gathering, Insignificant extra solution activity in MPEP § 2106.05(g))
providing the updated training dataset to the machine learning algorithm for training of the machine learning algorithm. (Mere data gathering, Insignificant extra solution activity in MPEP § 2106.05(g))
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exceptions.
receiving, by a computer system, an indication to update a training dataset for a machine learning algorithm; (receiving or transmitting data, using components and functions claimed at a high level of generality have been determined by the courts as being well-understood, routine, and conventional activities in the field of computer functions (See MPEP § 2106.05(d)(II)(i))
accessing, by the computer system in response to the indication, data in the training dataset for the machine learning algorithm, wherein the training dataset includes annotated labels on the data; (receiving or transmitting data, using components and functions claimed at a high level of generality have been determined by the courts as being well-understood, routine, and conventional activities in the field of computer functions (See MPEP § 2106.05(d)(II)(i))
accessing, by the computer system, a dataset comprising a plurality of data, wherein the data is labelled corresponding to a plurality of categories; (receiving or transmitting data, using components and functions claimed at a high level of generality have been determined by the courts as being well-understood, routine, and conventional activities in the field of computer functions (See MPEP § 2106.05(d)(II)(i))
providing the updated training dataset to the machine learning algorithm for training of the machine learning algorithm. (receiving or transmitting data, using components and functions claimed at a high level of generality have been determined by the courts as being well-understood, routine, and conventional activities in the field of computer functions (See MPEP § 2106.05(d)(II)(i))
As an ordered whole, the claim is directed to a method of clustering datasets into subsets, this is nothing more than using machine learning models to group the provided data. Nothing in the claim provides significantly more than this. As such, the claim is not patent eligible.
Claim 12 incorporates the rejection of claim 11.
Step 1: The claim recites a method, which is one of the four statutory categories of eligible matter.
Step 2A Prong 1; The judicial exception of claim 11 are incorporated. Please see the analysis of claim 11 above. Regarding the method recited in claim 11, these steps cover mental processes based on grouping and updating datasets.
Therefore, claim 12 is directed to an abstract idea – Mental Processes (i.e., can be performed in the human mind, or by a human using a pen and paper, making observations, evaluations and judgements as claimed)
Step 2A Prong 2: The judicial exceptions are not integrated into a practical application. In
particular, the claim recites these additional elements:
wherein the indication to update the training dataset is received in response to a drift in performance of the machine learning algorithm being detected. (Mere instructions to apply an exception as it recites only the idea of a solution or outcome as discussed in MPEP § 2106.05(f))
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exceptions.
wherein the indication to update the training dataset is received in response to a drift in performance of the machine learning algorithm being detected. (Mere instructions to apply an exception as it recites only the idea of a solution or outcome as discussed in MPEP § 2106.05(f))
The courts have found that adding the words "apply it" (or an equivalent) with the
judicial exception, or mere instructions to implement an abstract idea on a computer does not
qualify as “significantly more”. (See MPEP § 2106.05(I)(A))
Claim 13 incorporates the rejection of claim 11.
Step 1: The claim recites a method, which is one of the four statutory categories of eligible matter.
Step 2A Prong 1; The judicial exception of claim 11 are incorporated. Please see the analysis of claim 11 above. Regarding the method recited in claim 11, these steps cover mental processes based on grouping and updating datasets.
Therefore, claim 13 is directed to an abstract idea – Mental Processes (i.e., can be performed in the human mind, or by a human using a pen and paper, making observations, evaluations and judgements as claimed)
Step 2A Prong 2: The judicial exceptions are not integrated into a practical application. In
particular, the claim recites these additional elements:
wherein the indication to update the training dataset is received in response to a new category being added to the dataset comprising the plurality of data. (Mere instructions to apply an exception as it recites only the idea of a solution or outcome as discussed in MPEP § 2106.05(f))
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exceptions.
wherein the indication to update the training dataset is received in response to a new category being added to the dataset comprising the plurality of data. (Mere instructions to apply an exception as it recites only the idea of a solution or outcome as discussed in MPEP § 2106.05(f))
The courts have found that adding the words "apply it" (or an equivalent) with the
judicial exception, or mere instructions to implement an abstract idea on a computer does not
qualify as “significantly more”. (See MPEP § 2106.05(I)(A))
Claim 14 incorporates the rejection of claim 11.
Step 1: The claim recites a method, which is one of the four statutory categories of eligible matter.
Step 2A Prong 1; The judicial exception of claim 11 are incorporated. Please see the analysis of claim 11 above. Regarding the method recited in claim 11, these steps cover mental processes based on grouping and updating datasets.
Therefore, claim 14 is directed to an abstract idea – Mental Processes (i.e., can be performed in the human mind, or by a human using a pen and paper, making observations, evaluations and judgements as claimed)
Step 2A Prong 2: The judicial exceptions are not integrated into a practical application. In
particular, the claim recites these additional elements:
wherein the indication to update the training dataset is received in response to additional data being added to the dataset comprising the plurality of data. (Mere instructions to apply an exception as it recites only the idea of a solution or outcome as discussed in MPEP § 2106.05(f))
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exceptions.
wherein the indication to update the training dataset is received in response to additional data being added to the dataset comprising the plurality of data. (Mere instructions to apply an exception as it recites only the idea of a solution or outcome as discussed in MPEP § 2106.05(f))
The courts have found that adding the words "apply it" (or an equivalent) with the
judicial exception, or mere instructions to implement an abstract idea on a computer does not
qualify as “significantly more”. (See MPEP § 2106.05(I)(A))
Claim 15 incorporates the rejection of claim 11.
Step 1: The claim recites a method, which is one of the four statutory categories of eligible matter.
Step 2A Prong 1; The judicial exception of claim 11 are incorporated.
determining, for every individual label present in both the training dataset and the subset, similarity values between items of data with an individual label in the subset and an item of data with the individual label in the training dataset; (Mental Processes: Can be performed in the human mind, or by a human using a pen and paper, making observations, evaluations and judgments as claimed)
ranking, for every individual label, the items of data with the individual label in the subset based on the determined similarity values; (Mental Processes: Can be performed in the human mind, or by a human using a pen and paper, making observations, evaluations and judgments as claimed)
selecting, for every individual label, a set of items of data with the individual label to add to the training dataset based on the ranking of the set of items of data with the individual label in the subset satisfying a predetermined threshold for ranking of similarity to the item of data with the individual label in the training dataset. (Mental Processes: Can be performed in the human mind, or by a human using a pen and paper, making observations, evaluations and judgments as claimed)
Step 2A Prong 2: The judicial exceptions are not integrated into a practical application. In
particular, the claim recites these additional elements:
The claim does not recite any additional limitations. Therefore, there are no additional elements to integrate the abstract ideas into a practical application. (Merely asserting that a judicial exceptions is to be carried out on a generic computer (i.e. “a computer system” of base claim 11) cannot meaningfully integrate the judicial exceptions into a practical application. See MPEP § 2106.05(f))
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exceptions.
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. Mere instructions to apply an exception (i.e., “a computer system” of base claim 11) cannot provide an inventive concept. The claim is not patent eligible.
Claim 16 incorporates the rejection of claim 15.
Step 1: The claim recites a method, which is one of the four statutory categories of eligible matter.
Step 2A Prong 1; The judicial exception of claim 15 are incorporated.
wherein at least some of the annotated labels in the training dataset have been applied, at least in part, by human-based annotation with implementation of historical information. (Mental Processes: Can be performed in the human mind, or by a human using a pen and paper, making observations, evaluations and judgments as claimed)
Step 2A Prong 2: The judicial exceptions are not integrated into a practical application. In
particular, the claim recites these additional elements:
The claim does not recite any additional limitations. Therefore, there are no additional elements to integrate the abstract ideas into a practical application. (Merely asserting that a judicial exceptions is to be carried out on a generic computer (i.e. “a computer system” of base claim 15) cannot meaningfully integrate the judicial exceptions into a practical application. See MPEP § 2106.05(f))
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exceptions.
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. Mere instructions to apply an exception (i.e., “a computer system” of base claim 15) cannot provide an inventive concept. The claim is not patent eligible.
Claim 17
Step 1: The claim recites a non-transitory computer-readable medium, which is one of the four statutory categories of eligible matter.
Step 2A Prong 1:
selecting a first subset of data from the dataset by applying one or more clustering algorithms to the labelled data in the dataset; (Mathematical Concepts: are defined as mathematical relationships, mathematical formulas or equations, or mathematical calculations.)
applying annotation to the labelled data in the first subset to refine the labels on the data in the first subset, wherein the annotation includes, at least in part, implementation of historical information for the data; (Mental Processes: Can be performed in the human mind, or by a human using a pen and paper, making observations, evaluations and judgments as claimed)
selecting a second subset of labelled data from the dataset, wherein the labels on the data in the second subset of data correspond to the labels on the data in the first subset after annotation; (Mental Processes: Can be performed in the human mind, or by a human using a pen and paper, making observations, evaluations and judgments as claimed)
selecting at least one item of data with a given label from the second subset to add to the first subset, wherein the at least one item of data with the given label is selected based on the at least one item of data with the given label having a ranking of similarity, with respect to the data with the same given label in the first subset, that satisfies a predetermined threshold; (Mental Processes: Can be performed in the human mind, or by a human using a pen and paper, making observations, evaluations and judgments as claimed)
adding the at least one item of data with the given label to the first subset to generate a training dataset for a machine learning algorithm. (Mental Processes: Can be performed in the human mind, or by a human using a pen and paper, making observations, evaluations and judgments as claimed)
Step 2A Prong 2: The judicial exceptions are not integrated into a practical application. In
particular, the claim recites these additional elements:
A non-transitory computer-readable medium having instructions stored thereon that are executable by a computing device to perform operations (Mere instructions to apply an exception as it recites only the idea of a solution or outcome as discussed in MPEP § 2106.05(f))
accessing a dataset comprising a plurality of data, wherein the data is labelled corresponding to a plurality of categories; (Mere data gathering, Insignificant extra solution activity in MPEP § 2106.05(g))
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exceptions.
A non-transitory computer-readable medium having instructions stored thereon that are executable by a computing device to perform operations (Mere instructions to apply an exception as it recites only the idea of a solution or outcome as discussed in MPEP § 2106.05(f))
accessing a dataset comprising a plurality of data, wherein the data is labelled corresponding to a plurality of categories; (receiving or transmitting data, using components and functions claimed at a high level of generality have been determined by the courts as being well-understood, routine, and conventional activities in the field of computer functions (See MPEP § 2106.05(d)(II)(i))
The courts have found that adding the words "apply it" (or an equivalent) with the
judicial exception, or mere instructions to implement an abstract idea on a computer does not
qualify as “significantly more”. (See MPEP § 2106.05(I)(A))
As an ordered whole, the claim is directed to a method of clustering datasets into subsets, this is nothing more than using machine learning models to group the provided data. Nothing in the claim provides significantly more than this. As such, the claim is not patent eligible.
Claim 18 incorporates the rejection of claim 17.
Step 1: The claim recites a non-transitory computer-readable medium, which is one of the four statutory categories of eligible matter.
Step 2A Prong 1; The judicial exception of claim 17 are incorporated.
selecting at least one additional item of data with the given label from the second subset to add to the first subset, wherein the at least one additional item of data with the given label is selected based on the at least one additional item of data with the given label having a ranking of similarity, with respect to the data with the same given label in the first subset, that satisfies the predetermined threshold; (Mental Processes: Can be performed in the human mind, or by a human using a pen and paper, making observations, evaluations and judgments as claimed)
adding the at least one additional item of data with the given label to the generated training dataset for the machine learning algorithm. (Mental Processes: Can be performed in the human mind, or by a human using a pen and paper, making observations, evaluations and judgments as claimed)
Step 2A Prong 2: The judicial exceptions are not integrated into a practical application. In
particular, the claim recites these additional elements:
The claim does not recite any additional limitations. Therefore, there are no additional elements to integrate the abstract ideas into a practical application. (Merely asserting that a judicial exceptions is to be carried out on a generic computer (i.e. “a computer system” of base claim 17) cannot meaningfully integrate the judicial exceptions into a practical application. See MPEP § 2106.05(f))
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exceptions.
The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. Mere instructions to apply an exception (i.e., “a computer system” of base claim 17) cannot provide an inventive concept. The claim is not patent eligible.
Claim 19 incorporates the rejection of claim 18.
Step 1: The claim recites a non-transitory computer-readable medium, which is one of the four statutory categories of eligible matter.
Step 2A Prong 1; The judicial exception of claim 18 are incorporated. Please see the analysis of claim 18 above. Regarding the non-transitory computer-readable medium recited in claim 18, these steps cover mental processes based on grouping and updating datasets.
Therefore, claim 19 is directed to an abstract idea – Mental Processes (i.e., can be performed in the human mind, or by a human using a pen and paper, making observations, evaluations and judgements as claimed)
Step 2A Prong 2: The judicial exceptions are not integrated into a practical application. In
particular, the claim recites these additional elements:
wherein the at least one additional item of data is selected in response to training of the machine learning algorithm failing to be verified. (Mere instructions to apply an exception as it recites only the idea of a solution or outcome as discussed in MPEP § 2106.05(f))
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exceptions.
wherein the at least one additional item of data is selected in response to training of the machine learning algorithm failing to be verified. (Mere instructions to apply an exception as it recites only the idea of a solution or outcome as discussed in MPEP § 2106.05(f))
The courts have found that adding the words "apply it" (or an equivalent) with the
judicial exception, or mere instructions to implement an abstract idea on a computer does not
qualify as “significantly more”. (See MPEP § 2106.05(I)(A))
Claim 20 incorporates the rejection of claim 17.
Step 1: The claim recites a non-transitory computer-readable medium, which is one of the four statutory categories of eligible matter.
Step 2A Prong 1; The judicial exception of claim 17 are incorporated. Please see the analysis of claim 17 above. Regarding the non-transitory computer-readable medium recited in claim 17, these steps cover mental processes based on grouping and updating datasets.
Therefore, claim 20 is directed to an abstract idea – Mental Processes (i.e., can be performed in the human mind, or by a human using a pen and paper, making observations, evaluations and judgements as claimed)
Step 2A Prong 2: The judicial exceptions are not integrated into a practical application. In
particular, the claim recites these additional elements:
further comprising implementing the training dataset in training of the machine learning algorithm to determine one or more trained classifiers for the machine learning algorithm. (Mere instructions to apply an exception as it recites only the idea of a solution or outcome as discussed in MPEP § 2106.05(f))
Step 2B: The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exceptions.
further comprising implementing the training dataset in training of the machine learning algorithm to determine one or more trained classifiers for the machine learning algorithm. (Mere instructions to apply an exception as it recites only the idea of a solution or outcome as discussed in MPEP § 2106.05(f))
The courts have found that adding the words "apply it" (or an equivalent) with the
judicial exception, or mere instructions to implement an abstract idea on a computer does not
qualify as “significantly more”. (See MPEP § 2106.05(I)(A))
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1, 3, 6, 7, 9 and 11 is rejected under 35 U.S.C. 103 as being unpatentable over Su (US 20170091471 A1) in view of JUNG (US 20180018562 A1)
Regarding claim 1, Su teaches accessing, by a computer system, a dataset comprising a plurality of data, wherein the data is labelled corresponding to a plurality of categories; (See e.g. [0035], “At 402, access indications for multiple users of multiple resources are received.” [i.e., accesses indications corresponding to a plurality of data] “Each of the access indications indicate a resource, in this example a file or file location in a file repository and a user of the multiple users. An example of this is shown in FIGS. 1-3. As noted, the access indications can indicate a file name, file location, resource name or metadata, and a user.” [i.e., access indications comprises file name, file location, resource name or metadata, and a user corresponding to a plurality of categories])
selecting a first subset of data from the dataset by applying one or more clustering algorithms to the labelled data in the dataset; (See e.g. [0044], “At 404, the access indications and the multiple users are correlated. This correlation creates clusters, which cluster together subsets” [i.e., correlating multiple users to create subsets corresponding to a clustering algorithm] “of the multiple users with subsets of the resources.” [i.e., multiple users with resources corresponding to labelled data in the dataset] See e.g. [0045], “Thus, each cluster correlates one of the subsets of the multiple users with files indicated in one of the subsets” [i.e., one of the subsets of the multiple users corresponding to the first subset] “of access indications.”)
applying annotation to the labelled data in the first subset to refine the labels on the data in the first subset, wherein the annotation includes, at least in part, implementation of historical information for the data; (See e.g. [0045], “Thus, each cluster correlates one of the subsets of the multiple users with files indicated in one of the subsets” [i.e., one of the subsets of the multiple users corresponding to the first subset] “of access indications. To perform the correlation or as part of building each cluster, each file proxy or file can by annotated with names or identifiers for each user accessing those files” [i.e., annotating with names or identifiers corresponding to applying annotation]. See e.g. [0047], “At 406, other access indications of another file repository are received. These access indications indicate files accessed by other users, though these access indications can be analyzed to determine at least some shared users of the other file repository” [i.e., files accessed by other users corresponding to historical information for the data] “as that of the first-mentioned file repository.”)
selecting a second subset of labelled data from the dataset, wherein the labels on the data in the second subset of data correspond to the labels on the data in the first subset after annotation; (See e.g. [0050], “Consider, for example, FIG. 5, which illustrates first clusters 502 of a first repository, such as through performing operations 402 and 404 of method 400, and second clusters 504” [i.e., the second clusters corresponding to the second subset] “of a second repository, such as through performing operations 406” [i.e., operations 406 is the first subset being annotated corresponding to the first subset after annotation] “and 408 of method 400.”)
selecting a portion of the data from the second subset to add to the first subset, wherein the portion of the data is selected based on the data with a given label in the portion having a measure of similarity, with respect to the data with the same given label in the first subset, that satisfies a predetermined threshold; (See e.g. [0049], “With the clusters determined for the two repositories, at 410, the clusters” [i.e., the clusters corresponding to the first subset] “and the other clusters” [i.e., the other clusters corresponding to the second subset] “are cascaded together based on having some shared users between the subsets of the other users and the subsets of the multiple users” [i.e., having some shared users between the subsets corresponding to a predetermined threshold]. “These cascaded clusters are total clusters of both repositories. This cascading can include adding or concatenating together file proxies from one repository into a cluster” [i.e., adding files proxies from one repository into a cluster corresponding to selecting a portion of data from the second subset to add to the first subset] “for another repository based on shared users” [i.e. shared users corresponding to a measure of similarity])
Su does not teach adding the portion of the data selected to the first subset to generate a training dataset; providing the training dataset to a machine learning algorithm for training of the machine learning algorithm.
JUNG teaches adding the portion of the data selected [to the first subset] to generate a training dataset; (See e.g. [0125], “Further, according to embodiments of the inventive concept, when learning data is added” [i.e., learning data added to the learning model corresponding to adding the portion of the data] “to the learning model for extracting the task, the cache model is crated” [sic - created][i.e., the cache model being created corresponding to generate a training dataset] “and merged with the existing model, thereby implementing a real-time learning process. Accordingly, the resources and the costs necessary for constructing the learning model may be reduced.”)
providing the training dataset to a machine learning algorithm for training of the machine learning algorithm. (See e.g. [0125], “Further, according to embodiments of the inventive concept, when learning data is added to the learning model for extracting the task, the cache model is crated and merged with the existing model, thereby implementing a real-time learning process” [i.e. implementing the learning process corresponding to providing the training dataset]. “Accordingly, the resources and the costs necessary for constructing the learning model may be reduced.”)
Accordingly, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention, having the teachings of Su and JUNG before them, to include JUNG’s updating of training data in Su’s clustering of data. One would have been motivated to make such a combination in order to reduce the resources needed for constructing a learning model, as suggested by JUNG (0125)
Regarding claim 3, Su and JUNG teach the method of claim 1. Su further teaches determining, for the given label, similarity values between the data with the given label in the second subset and the data with the given label in the first subset; (See e.g. [0050], “FIG. 5 illustrates total clusters 506 resulting from performing operation 410. Here the clusters of each of the repositories are correlated based on having same or similar shared users” [i.e., user corresponding to the given label] “of each cluster.”)
determining data from the second subset to include in the portion of the data based on the data with the given label in the second subset having a similarity value that satisfies the predetermined threshold. (See e.g. [0050], “FIG. 5 illustrates total clusters 506 resulting from performing operation 410. Here the clusters of each of the repositories are correlated based on having same or similar” [i.e., similar users corresponding to the predetermined threshold] “shared users” [i.e., user corresponding to the given label] “of each cluster.”)
Regarding claim 6, Su and JUNG teach the method of claim 1. Su further teaches wherein the second subset of labelled data is selected from the dataset by determining additional data from the dataset that has labels that are identical to the labels on the data in the first subset after annotation. (See e.g. [0050], “Consider, for example, FIG. 5, which illustrates first clusters 502 of a first repository, such as through performing operations 402 and 404 of method 400, and second clusters 504” [i.e., the second clusters corresponding to the second subset] “of a second repository, such as through performing operations 406” [i.e., operations 406 is the first subset being annotated corresponding to the first subset after annotation] “and 408 of method 400. FIG. 5 illustrates total clusters 506 resulting from performing operation 410. Here the clusters of each of the repositories are correlated based on having same” [i.e., clusters having the same users corresponding to labels that are identical] “or similar shared users of each cluster.”)
Regarding claim 7, Su and JUNG teach the method of claim 1. Su further teaches wherein applying the annotation to the labelled data includes determining an agreement in the labels on the data in the first subset after implementation of the historical information for the data. (See e.g. [0047], “At 406, other access indications of another file repository are received. These access indications indicate files accessed by other users, though these access indications can be analyzed to determine at least some shared users of the other file repository” [i.e., files accessed by other users corresponding to historical information for the data] “as that of the first-mentioned file repository.” See e.g. [0048], “At 408” [i.e., operation 408 correlate the other access indications to cluster subsets of other users with subsets of other resources corresponding to after implementation], “the other access indications and the other users are correlated to cluster together the subsets of the other users with subsets of the other files” [i.e., users that correlate to cluster together corresponding agreement in the labels]. “As in operation 404, these files or file locations can be arranged into, or analyzed through file proxies, which is illustrated above.”)
Regarding claim 9, Su and JUNG teach the method of claim 1. Su further teaches wherein the plurality of data in the dataset includes text data, and wherein at least one of the categories applied to the data is an intent category. (See e.g. [0035], “At 402, access indications for multiple users of multiple resources are received. Each of the access indications indicate a resource, in this example a file or file location in a file repository and a user of the multiple users. An example of this is shown in FIGS. 1-3. As noted, the access indications can indicate a file name, file location, resource name or metadata, and a user.” [i.e., the access indications user corresponding to intent category])
Regarding claim 11, Su teaches accessing, by the computer system in response to the indication, data in the training dataset [for the machine learning algorithm], wherein the [training] dataset includes annotated labels on the data; (See e.g. [0035], “At 402, access indications for multiple users of multiple resources are received. Each of the access indications indicate a resource, in this example a file or file location in a file repository and a user of the multiple users.” [i.e., user of the multiple users corresponding to annotated labels on the data])
accessing, by the computer system, a dataset comprising a plurality of data, wherein the data is labelled corresponding to a plurality of categories; (See e.g. [0035], “At 402, access indications for multiple users of multiple resources are received” [i.e., accesses indications corresponding to a plurality of data]. “Each of the access indications indicate a resource, in this example a file or file location in a file repository and a user of the multiple users. An example of this is shown in FIGS. 1-3. As noted, the access indications can indicate a file name, file location, resource name or metadata, and a user.” [i.e., access indications comprises file name, file location, resource name or metadata, and a user corresponding to a plurality of categories])
selecting a subset of labelled data from the dataset, wherein the labels on the data in the subset of data correspond to the annotated labels on the data [in the training dataset]; (See e.g. [0044], “At 404, the access indications and the multiple users are correlated. This correlation creates clusters, which cluster together subsets of the multiple users with subsets of the resources.” See e.g. [0045], “To perform the correlation or as part of building each cluster, each file proxy or file can by annotated with names or identifiers for each user accessing those files” [i.e., file can by annotated with names or identifiers for each user corresponding to annotated labels]. “With these annotations, the cluster module 110 may then arrange the file proxies and the users to visually cluster them for an administrator's benefit, to aid in his or her analysis, though this human-readable visual presentation is not required for many of the features described herein.”)
selecting a portion of the data from the subset to add to the [training] dataset, wherein the portion of the data is selected based on the data with a given label in the portion having a measure of similarity (See e.g. [0049], “With the clusters determined for the two repositories, at 410, the clusters and the other clusters are cascaded together based on having some shared users between the subsets” [i.e. shared users corresponding to a measure of similarity] “of the other users and the subsets of the multiple users. These cascaded clusters are total clusters of both repositories. This cascading can include adding” [i.e., adding files proxies from one repository into a cluster corresponding data from the subset to add to the dataset] “or concatenating together file proxies from one repository into a cluster for another repository based on shared users.”)
with respect to the data with the same given label in the [training] dataset that satisfies a predetermined threshold; (See e.g. [0049], “These cascaded clusters are total clusters of both repositories. This cascading can include adding or concatenating together file proxies from one repository into a cluster for another repository based on shared users.” [i.e., shared users corresponding to a predetermined threshold])
updating the training dataset by adding the portion of the data selected to the [training] dataset; (See e.g. [0052], “This operation is illustrated at 412 in FIG. 4, at which the techniques annotate the total cluster” [i.e., annotating the total cluster corresponding to updating the dataset] “based on annotations of one of the constituent clusters.”)
Su does not teach receiving, by a computer system, an indication to update a training dataset for a machine learning algorithm; providing the updated training dataset to the machine learning algorithm for training of the machine learning algorithm.
JUNG teaches receiving, by a computer system, an indication to update a training dataset for a machine learning algorithm; (See e.g. [0100], “The task extracting unit 303 may learn words selected through the CNN learning model as input data of the RNN learning model.” [i.e., RNN learning model corresponding to an indication to update a training dataset])
providing the updated training dataset to the machine learning algorithm for training of the machine learning algorithm. (See e.g. [0125], “Further, according to embodiments of the inventive concept, when learning data is added to the learning model for extracting the task, the cache model is crated and merged with the existing model” [i.e., merging the cache model with the existing model corresponding to providing the updated training dataset], “thereby implementing a real-time learning process. Accordingly, the resources and the costs necessary for constructing the learning model may be reduced.”)
Accordingly, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention, having the teachings of Su and JUNG before them, to include JUNG’s updating of training data in Su’s clustering of data. One would have been motivated to make such a combination in order to reduce the resources needed for constructing a learning model, as suggested by JUNG (0125)
Claim 2 is rejected under 35 U.S.C. 103 as being unpatentable over Su (US 20170091471 A1) in view of JUNG (US 20180018562 A1) further in view of BOUDREAU (US 20200265270 A1)
Regarding claim 2, Su and JUNG teach the method of claim 1.
Su and JUNG do not teach wherein the data with the given label in the portion having the measure of similarity satisfying the predetermined threshold indicates that the data in the portion with the given label has a nearest neighbor ranking for similarity, to the data with the same given label in the first subset, that satisfies a predetermined ranking threshold.
BOUDREAU teaches wherein the data with the given label in the portion having the measure of similarity satisfying the predetermined threshold indicates that the data in the portion with the given label has a nearest neighbor ranking for similarity, to the data with the same given label in the first subset, that satisfies a predetermined ranking threshold. (See e.g. [0030], “Each data set may have several clusters of data elements; such that a first subset” [i.e., the first subset] “of data elements in the data set belongs to a first cluster” See e.g. [0057], “At 514, the nearest neighbors for the data elements are compared to identify mutual neighbor relationships between data elements. As noted, two data elements are mutual neighbors at a value k if they appear in each other's list of k nearest neighbors. Scores are assigned based on the proportion of mutual neighbor relationships among each data element's k nearest neighbors, for values of k from k_min to k_max. [i.e., a list of k nearest neighbors based on score corresponding to a nearest neighbor ranking]” See e.g. [0059], “At 516, classification tool 32 selects from the unlabeled data set representative data elements which represent corresponding clusters of data elements, as described above. A data element is selected as representative if it has more than a threshold proportion of mutual neighbor relationships among its k_max mutual neighbors (e.g., more than half) [i.e., threshold proportion of mutual neighbors corresponding to satisfies a predetermined ranking threshold] , and if it has the highest cluster score among those k_max mutual neighbors.” See e.g. [0094], “Data element's “1”, “2”, “5”, “8” have been labeled “Red” while data elements “3”, “4”, “6”, “7” have been labeled “Green”.” [i.e., label red corresponding to a given label] “In some instances, unreached data elements may be considered as outliers. In the depicted example, data element “9” meets the above-referenced definition of outlier, namely, a k_max mutuality score below 0.5 and having the lowest cluster score among its k_max closest mutual neighbors.”)
Accordingly, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention, having the teachings of Su, JUNG and BOUDREAU before them, to include BOUDREAU’s ranking of neighbors Su and JUNG’s plurality of subsets. One would have been motivated to make such a combination in order to increase the accuracy of labeling and propagating labels, as suggested by BOUDREAU (0023)
Claims 4, 15, 17, 18 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Su (US 20170091471 A1) in view of JUNG (US 20180018562 A1) further in view of Hua (US 20160217349 A1)
Regarding claim 4, Su and JUNG teach the method of claim 1. Su further teaches determining, for the given label, a [ranking of] similarity between the data with the given label in the second subset and the data with the given label in the first subset; (See e.g. [0050], “Consider, for example, FIG. 5, which illustrates first clusters 502 of a first repository, such as through performing operations 402 and 404 of method 400, and second clusters 504 of a second repository, such as through performing operations 406 and 408 of method 400. FIG. 5 illustrates total clusters 506 resulting from performing operation 410. Here the clusters of each of the repositories are correlated based on having same or similar shared users” [i.e., user corresponding to the given label] “of each cluster.”)
Su and JUNG do not teach determining data [from the second subset] to include in the portion of the data based on the ranking of similarity for the data with the given label [in the second subset] satisfying the predetermined threshold, wherein the predetermined threshold is a threshold for the ranking of similarity.
Hua teaches determining data [from the second subset] to include in the portion of the data based on the ranking of similarity for the data with the given label [in the second subset] satisfying the predetermined threshold, wherein the predetermined threshold is a threshold for the ranking of similarity. (See e.g. [0091], “the classifying module 120 may select a label” [i.e., classifying module 120 selecting a label corresponding to the given label] “corresponding to the highest ranking similarity value as the label associated with the new multimedia data item 502. In some examples, the classifying module 120 may select a predetermined number [i.e. predetermined number of labels based on similarity values corresponding to a predetermined threshold is a threshold for the ranking of similarity] (e.g., 5, 3, 2) of the labels corresponding to similarity values above a predetermined threshold and may return the predetermined number of the labels with confidence scores as a recognition result 508.”)
Accordingly, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention, having the teachings of Su, JUNG and Hua before them, to include Hua’s ranking of similarity Su and JUNG’s plurality of subsets. One would have been motivated to make such a combination in order to increase the training efficiency of the model, as suggested by Hua (0019)
Regarding claim 15, Su and JUNG teach the method of claim 11.
Su and JUNG do not teach determining, for every individual label present in both the training dataset and the subset, similarity values between items of data with an individual label in the subset and an item of data with the individual label in the training dataset; ranking, for every individual label, the items of data with the individual label in the subset based on the determined similarity values; selecting, for every individual label, a set of items of data with the individual label to add to the training dataset based on the ranking of the set of items of data with the individual label in the subset satisfying a predetermined threshold for ranking of similarity to the item of data with the individual label in the training dataset.
Hua teaches determining, for every individual label present in both the training dataset and the subset, similarity values between items of data with an individual label in the subset and an item of data with the individual label in the training dataset; (See e.g. [0115], “the negative multimedia data items are not associated with any label of the plurality of labels; extracting a first set of features from the individual positive multimedia data items; training a classifier” [i.e., the first set of features corresponding to the training dataset] “based at least in part on the first set of features, the classifier including a plurality of model vectors each corresponding to one of the individual labels; based at least in part on applying the classifier to one or more of the individual positive multimedia data items, collecting statistics corresponding to each of the individual labels; extracting a second set of features” [i.e., the extracting a second set of features corresponding to the subset] “from a new multimedia data item; applying the classifier to the second set of features to determine similarity values” [i.e., determining a similarity values of features corresponding to similarity values between items] “corresponding to each of the individual labels;”)
ranking, for every individual label, the items of data with the individual label in the subset based on the determined similarity values; (See e.g. [0130], “determining that the new multimedia data item” [i.e., multimedia data is a collection of individual labels corresponding to the subset] “is associated with a particular individual label of the individual labels, the particular individual label being associated with a highest ranking similarity value.”)
selecting, for every individual label, a set of items of data with the individual label to add to the training dataset based on the ranking of the set of items of data with the individual label in the subset satisfying a predetermined threshold for ranking of similarity to the item of data with the individual label in the training dataset. (See e.g. [0049], “The updating module 208” [i.e., updating the module with new multimedia data items corresponding to adding to the training dataset] “may update the classifiers (e.g., multi-class classifiers) based at least in part on applying the classifiers to new multimedia data items” See e.g. [0137], “classifying the new multimedia data item comprises determining that the new multimedia data item is associated with at least one of the individual labels; and the acts further comprise ranking the similarity values”)
Accordingly, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention, having the teachings of Su, JUNG and Hua before them, to include Hua’s ranking of similarity Su and JUNG’s plurality of subsets. One would have been motivated to make such a combination in order to increase the training efficiency of the model, as suggested by Hua (0019)
Regarding claim 17, Su teaches A non-transitory computer-readable medium having instructions stored thereon that are executable by a computing device to perform operations (See e.g. [0004], “An example method for classifying data elements is executed by a processor coupled to a computer memory.”)
accessing, by a computer system, a dataset comprising a plurality of data, wherein the data is labelled corresponding to a plurality of categories; (See e.g. [0035], “At 402, access indications for multiple users of multiple resources are received.” [i.e., accesses indications corresponding to a plurality of data] “Each of the access indications indicate a resource, in this example a file or file location in a file repository and a user of the multiple users. An example of this is shown in FIGS. 1-3. As noted, the access indications can indicate a file name, file location, resource name or metadata, and a user.” [i.e., access indications comprises file name, file location, resource name or metadata, and a user corresponding to a plurality of categories])
selecting a first subset of data from the dataset by applying one or more clustering algorithms to the labelled data in the dataset; (See e.g. [0044], “At 404, the access indications and the multiple users are correlated. This correlation creates clusters, which cluster together subsets” [i.e., correlating multiple users to create subsets corresponding to a clustering algorithm] “of the multiple users with subsets of the resources.” [i.e., multiple users with resources corresponding to labelled data in the dataset] See e.g. [0045], “Thus, each cluster correlates one of the subsets of the multiple users with files indicated in one of the subsets” [i.e., one of the subsets of the multiple users corresponding to the first subset] “of access indications.”)
applying annotation to the labelled data in the first subset to refine the labels on the data in the first subset, wherein the annotation includes, at least in part, implementation of historical information for the data; (See e.g. [0045], “Thus, each cluster correlates one of the subsets of the multiple users with files indicated in one of the subsets” [i.e., one of the subsets of the multiple users corresponding to the first subset] “of access indications. To perform the correlation or as part of building each cluster, each file proxy or file can by annotated with names or identifiers for each user accessing those files” [i.e., annotating with names or identifiers corresponding to applying annotation]. See e.g. [0047], “At 406, other access indications of another file repository are received. These access indications indicate files accessed by other users, though these access indications can be analyzed to determine at least some shared users of the other file repository” [i.e., files accessed by other users corresponding to historical information for the data] “as that of the first-mentioned file repository.”)
selecting a second subset of labelled data from the dataset, wherein the labels on the data in the second subset of data correspond to the labels on the data in the first subset after annotation; (See e.g. [0050], “Consider, for example, FIG. 5, which illustrates first clusters 502 of a first repository, such as through performing operations 402 and 404 of method 400, and second clusters 504” [i.e., the second clusters corresponding to the second subset] “of a second repository, such as through performing operations 406” [i.e., operations 406 is the first subset being annotated corresponding to the first subset after annotation] “and 408 of method 400.”)
Su does not teach adding the at least one item of data with the given label to the first subset to generate a training dataset for a machine learning algorithm.
JUNG teaches adding the at least one item of data with the given label [to the first subset] to generate a training dataset for a machine learning algorithm. (See e.g. [0125], “Further, according to embodiments of the inventive concept, when learning data is added” [i.e., learning data added to the learning model corresponding to adding the portion of the data] “to the learning model for extracting the task, the cache model is” crated [i.e., the cache model being created corresponding to generate a training dataset] “and merged with the existing model, thereby implementing a real-time learning process. Accordingly, the resources and the costs necessary for constructing the learning model may be reduced.”)
Accordingly, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention, having the teachings of Su and JUNG before them, to include JUNG’s updating of training data in Su’s clustering of data. One would have been motivated to make such a combination in order to reduce the resources needed for constructing a learning model, as suggested by JUNG (0125)
Su and JUNG do not teach selecting at least one item of data with a given label from the second subset to add to the first subset, wherein the at least one item of data with the given label is selected based on the at least one item of data with the given label having a ranking of similarity, with respect to the data with the same given label in the first subset, that satisfies a predetermined threshold;
Hua teaches selecting at least one item of data with a given label [from the second subset to add to the first subset], wherein the at least one item of data with the given label is selected based on the at least one item of data with the given label having a ranking of similarity, with respect to the data with the same given label [in the first subset], that satisfies a predetermined threshold; and (See e.g. [0091], “the classifying module 120 may select a label” [i.e., classifying module 120 selecting a label corresponding to the given label] “corresponding to the highest ranking similarity value as the label associated with the new multimedia data item 502. In some examples, the classifying module 120 may select a predetermined number (e.g., 5, 3, 2) of the labels corresponding to similarity values above a predetermined threshold and may return the predetermined number of the labels with confidence scores as a recognition result 508.”)
Accordingly, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention, having the teachings of Su, JUNG and Hua before them, to include Hua’s ranking of similarity Su and JUNG’s plurality of subsets. One would have been motivated to make such a combination in order to increase the training efficiency of the model, as suggested by Hua (0019)
Regarding claim 18, Su, JUNG and Hua teach the computer-readable medium of claim 17. Su teaches selecting at least one additional item of data with the given label from the second subset to add to the first subset… (See e.g. [0049], “With the clusters determined for the two repositories, at 410, the clusters” [i.e., the clusters corresponding to the first subset] “and the other clusters” [i.e., the other clusters corresponding to the second subset] “are cascaded together based on having some shared users between the subsets of the other users and the subsets of the multiple users” [i.e., having some shared users between the subsets corresponding to a predetermined threshold]. “These cascaded clusters are total clusters of both repositories. This cascading can include adding or concatenating together file proxies from one repository into a cluster” [i.e., adding files proxies from one repository into a cluster corresponding to selecting a portion of data from the second subset to add to the first subset] “for another repository based on shared users” [i.e., shared users corresponding to a measure of similarity])
Su and JUNG do not teach [selecting at least one additional item of data with the given label from the second subset to add to the first subset], wherein the at least one additional item of data with the given label is selected based on the at least one additional item of data with the given label having a ranking of similarity, with respect to the data with the same given label in the first subset, that satisfies the predetermined threshold; and
Hua teaches selecting at least one additional item of data with the given label [from the second subset to add to the first subset], wherein the at least one additional item of data with the given label is selected based on the at least one additional item of data with the given label having a ranking of similarity, with respect to the data with the same given label [in the first subset], that satisfies the predetermined threshold; and (See e.g. [0091], “the classifying module 120 may select a label” [i.e., classifying module 120 selecting a label corresponding to the given label] “corresponding to the highest ranking similarity value as the label associated with the new multimedia data item 502. In some examples, the classifying module 120 may select a predetermined number (e.g., 5, 3, 2) of the labels corresponding to similarity values above a predetermined threshold and may return the predetermined number of the labels with confidence scores as a recognition result 508.” See e.g. [0087], “In some examples, as described above, the classifying module 120 may send the similarity values to the updating module 208 for comparing. The classifying module 120 may compare each of the similarity values to the statistics 506 to determine whether the new multimedia data item 502 is associated with any of the labels associated with the topic.”)
Accordingly, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention, having the teachings of Su, JUNG and Hua before them, to include Hua’s ranking of similarity Su and JUNG’s plurality of subsets. One would have been motivated to make such a combination in order to increase the training efficiency of the model, as suggested by Hua (0019)
Su and Hua do not teach adding the at least one additional item of data with the given label to the generated training dataset for the machine learning algorithm. JUNG teaches adding the at least one additional item of data with the given label to the generated training dataset for the machine learning algorithm. (See e.g. [0125], “Further, according to embodiments of the inventive concept, when learning data is added” [i.e., learning data added to the learning model corresponding to adding the portion of the data] “to the learning model for extracting the task, the cache model is crated” [sic - created][i.e., the cache model being created corresponding to generate a training dataset] “and merged with the existing model, thereby implementing a real-time learning process. Accordingly, the resources and the costs necessary for constructing the learning model may be reduced.”)
Accordingly, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention, having the teachings of Su, Hua and JUNG before them, to include JUNG’s updating of training data in Su and Hua’s clustering of data. One would have been motivated to make such a combination in order to reduce the resources needed for constructing a learning model, as suggested by JUNG (0125)
Regarding claim 20, Su and JUNG teach the computer-readable medium of claim 17.
Su and JUNG do not teach further comprising implementing the training dataset in training of the machine learning algorithm to determine one or more trained classifiers for the machine learning algorithm.
Hua teaches further comprising implementing the training dataset in training of the machine learning algorithm to determine one or more trained classifiers for the machine learning algorithm. (See e.g. [0040], “FIG. 2 is a diagram showing additional components of an example system 200 for training classifiers” [i.e., system 200 comprises classifier(s) 212 corresponding to determine one or more trained classifiers] “from positive multimedia data items” [i.e., multimedia data items corresponding to training dataset] “and negative multimedia data items and applying the trained classifiers to classify new multimedia data items.” See e.g. [0051], “The classifying module 120 may store one or more classifier(s) 212 (e.g., multi-class classifiers) and may be associated with a ranking module 214”)
Accordingly, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention, having the teachings of Su, JUNG and Hua before them, to include Hua’s ranking of similarity Su and JUNG’s plurality of subsets. One would have been motivated to make such a combination in order to increase the training efficiency of the model, as suggested by Hua (0019)
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over Su (US 20170091471 A1) in view of JUNG (US 20180018562 A1) further in view of Key (US 20230005042 A1)
Regarding claim 5, Su and JUNG teach the method of claim 1.
Su and JUNG do not teach determining, for every individual label present in both the first subset and the second subset, a ranking of similarity between the data a ranking of similarity between the data with an individual label in the second subset and the data with the individual label in the first subset; and determining, for every individual label present in both the first subset and the second subset, data from the second subset to include in the portion of the data, wherein the data to be included is determined based on the ranking of similarity for the data with the individual label in the second subset satisfying the predetermined threshold, the predetermined threshold being a threshold for the ranking of similarity.
Key teaches determining, for every individual label present in both the first subset and the second subset, a ranking of similarity between the data a ranking of similarity between the data with an individual label in the second subset and the data with the individual label in the first subset; (See e.g. [0083], “For example, the one or more computers (e.g., 104) may determine that the measures of similarity are each above one or more predetermined similarity thresholds (e.g., that distances between the purchase embeddings” [i.e., items comprise purchase embeddings corresponding the individual label] “are less than one or more predetermined distance thresholds). The one or more computers may also rank the measures of similarity between the first and the second pluralities of items” [i.e., rank the measure of similarity between the first and second pluralities of items corresponding a ranking of similarity between first and second subsets for every individual label present based on purchase embeddings], “and the predetermined thresholds include a predetermined minimum ranking.”)
and determining, for every individual label present in both the first subset and the second subset, (See e.g. [0083], “For example, the one or more computers (e.g., 104) may determine that the measures of similarity are each above one or more predetermined similarity thresholds (e.g., that distances between the purchase embeddings” [i.e., items comprise purchase embeddings corresponding the individual label in both subsets] “are less than one or more predetermined distance thresholds). The one or more computers may also rank the measures of similarity between the first and the second pluralities of items”)
data from the second subset to include in the portion of the data, wherein the data to be included is determined based on the ranking of similarity for the data with the individual label in the second subset satisfying the predetermined threshold, (See e.g. [0083], “At step 760, the one or more computing devices may identify, based on the descriptions and the calculating of step 750, a subset of the second plurality of items that are available for purchase within the second geographic region and that have measures of similarity to the first plurality of items that exceed predetermined thresholds [i.e., the plurality of items exceeding a predetermined threshold corresponding to the individual label in the second subset satisfying the predetermined threshold]” See e.g. [0085], “At step 770, the one or more computing devices (e.g., 104, 109A-C) may generate, based on the identifying of the correlation between the purchases of the first plurality of items and the identifying of the subset of the second plurality of items [i.e., identifying the items from the second subset of items corresponding to data from the second subset to include in the portion of the data], a proposed pattern for purchasing the subset within the second geographic region.”)
the predetermined threshold being a threshold for the ranking of similarity. (See e.g. [0083], “The one or more computers may also rank the measures of similarity between the first and the second pluralities of items, and the predetermined thresholds include a predetermined minimum ranking.”)
Accordingly, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention, having the teachings of Su, JUNG and Key before them, to include Key’s ranking of subsets Su and JUNG’s plurality of subsets. One would have been motivated to make such a combination in order to generate stronger rankings by increasing the number of parameters used in the comparison of subsets, as suggested by Key (0002)
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Su (US 20170091471 A1) in view of JUNG (US 20180018562 A1) further in view of BOUDREAU (US 20200265270 A1) further in view of Kopparapu (US 20220207875 A1)
Regarding claim 8, Su and JUNG teach the method of claim 1.
Su and JUNG do not teach determining a set of nearest neighbors to a given item of data in the first subset based on data in the given item of data and data in the nearest neighbors, wherein the set of nearest neighbors includes a set of nearest items of data based on similarities between the data in the nearest neighbors and the data in the given item of data; determining a number of nearest neighbors in the set of nearest neighbors that have labels identical to a label on the data in the given item of data; retaining the given item of data in the first subset when the number of nearest neighbors that have identical labels satisfies a predetermined threshold for a minimum number of nearest neighbors having the same label;
BOUDREAU teaches determining a set of nearest neighbors to a given item of data in the first subset based on data in the given item of data and data in the nearest neighbors, wherein the set of nearest neighbors includes a set of nearest items of data based on similarities between the data in the nearest neighbors and the data in the given item of data; (See e.g. [0030], “such that a first subset of data elements in the data set” [i.e., the data elements corresponding to a given item of data] “belongs to a first cluster, a second subset of data elements in the data set belongs to a second cluster, and so forth.” See e.g. [0043], “In an example, cluster scores may be based on mutuality scores, namely, scores indicating for each data element the proportion of nearest neighbors for which the data element is also a nearest neighbor. As used herein, data elements” [i.e., data elements among one another’s closest neighbor corresponding to a set of nearest items] “which are among one another's closest neighbors are referred to as mutual neighbors.” [i.e., mutual neighbors corresponding to similarities between the data in the nearest neighbors] “Therefore, the mutuality score for a given value of k represents the proportion of mutual neighbors among that data element's closest k neighbors.”)
determining a number of nearest neighbors in the set of nearest neighbors that have labels identical to a label on the data in the given item of data; (See e.g. [0042], “A threshold number of neighbors may be determined. The threshold number may be referred to as k_max, which may be received by RI module 53 as a parameter, where k_max is an integer greater than or equal to 1. To determine the closest neighbors to a data element in the feature space” [i.e., determine the closest neighbors corresponding to determining a number of nearest neighbors], “RI module 53 may first compute the distance between each two data elements in the feature space, for example, by computing the Euclidean distance between the data elements in the feature space, or by using other suitable formulas.” See e.g. [0020], “Groups of data elements having the same label” [i.e., same label corresponding to labels identical] “may be referred to as classes.”)
retaining the given item of data in the first subset when the number of nearest neighbors that have identical labels satisfies a predetermined threshold for a minimum number of nearest neighbors having the same label; See e.g. [0078], “Classification tool 32 may store in memory 24, at 530, a single” [i.e., single label in associated with each data element corresponding to identical labels] “or multiple label in association with each data element to generate a labeled data set.”)
Accordingly, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention, having the teachings of Su, JUNG and BOUDREAU before them, to include BOUDREAU’s ranking of neighbors Su and JUNG’s plurality of subsets. One would have been motivated to make such a combination in order to increase the accuracy of labeling and propagating labels, as suggested by BOUDREAU (0023)
Su, JUNG and BOUDREAU do not teach removing the given item of data from the first subset when the number of nearest neighbors that have identical labels fails to satisfy the predetermined threshold for the minimum number of nearest neighbors having the same label.
Kopparapu teaches removing the given item of data from the first subset [when the number of nearest neighbors that have identical labels] fails to satisfy the predetermined threshold [for the minimum number of nearest neighbors having the same label.] (See e.g. [0102], “In one or more embodiments, the frame filtering corresponds with determining a first subset of the video frames 530. The frame filtering and keyframe extraction module 532 removes, from the video frames 530, those video frames which are outside of an image quality threshold.” [i.e., outside of an image quality threshold corresponding to fails to satisfy];)
Accordingly, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention, having the teachings of Su, JUNG, BOUDREAU and Kopparapu before them, to include Kopparapu’s pruning of data in subsets Su, JUNG and BOUDREAU’s plurality of subsets. One would have been motivated to make such a combination in order to reduce computer resources to train a model, as suggested by Kopparapu (0131)
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Su (US 20170091471 A1) in view of JUNG (US 20180018562 A1) further in view of TRUONG (US 20230244987 A1)
Regarding claim 10, Su and JUNG teach the method of claim 1.
Su and JUNG do not teach wherein at least one item of data in the first subset is labelled with a mislabeled category, and wherein applying the annotation to the first subset corrects the mislabeled category.
TRUONG teaches wherein at least one item of data in the first subset is labelled with a mislabeled category, and wherein applying the annotation to the first subset corrects the mislabeled category. (See e.g. [0004], “The set of instructions, when executed by the one or more processors of the data labeling system, may cause the data labeling system to receive inputs to apply user-specified labels to data elements included in a first subset of the unlabeled data samples …The set of instructions, when executed by the one or more processors of the data labeling system, may cause the data labeling system to present a user interface to request feedback related to the automatic labels based on confidence levels associated with the automatic labels.” See e.g. [0031], “as shown by reference number 140, the data labeling system may modify one or more automatic labels and/or one or more manual labels based on the user feedback. For example, any labels that were identified as incorrect” [i.e., identified as incorrect corresponding to labelled with a mislabeled category] “may be updated with the correct user-defined label” [i.e., updated with correct user-defined label corresponding to correcting the mislabeled category].)
Accordingly, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention, having the teachings of Su, JUNG and TRUONG before them, to include TRUONG’s correcting and modifying training sets Su and JUNG’s plurality of subsets. One would have been motivated to make such a combination in order to increase the accuracy and consistency of the model, as suggested by TRUONG (0054)
Claims 12 – 14 are rejected under 35 U.S.C. 103 as being unpatentable over Su (US 20170091471 A1) in view of JUNG (US 20180018562 A1) further in view of Raveh (US 20210295213 A1)
Regarding claim 12, Su and JUNG teaches the method of claim 11.
Su and JUNG do not teach wherein the indication to update the training dataset is received in response to a drift in performance of the machine learning algorithm being detected.
Raveh teaches wherein the indication to update the training dataset is received in response to a drift in performance of the machine learning algorithm being detected. (See e.g. [0105], “in case a data drift is determined, the production dataset along with the subsets of defects may be provided to Training Dataset Selection Module 460. Training Dataset Selection Module 460 may be configured to determine which data should be selected for re-training the main classification model.”)
Accordingly, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention, having the teachings of Su, JUNG and Raveh before them, to include Raveh’s indicator for retraining the model Su and JUNG’s plurality of subsets. One would have been motivated to make such a combination in order to improve the accuracy, robustness and stability of the model, as suggested by Raveh (0038)
Regarding claim 13, Su and JUNG teaches the method of claim 11.
Su and JUNG do not teach wherein the indication to update the training dataset is received in response to a new category being added to the dataset comprising the plurality of data.
Raveh teaches wherein the indication to update the training dataset is received in response to a new category being added to the dataset comprising the plurality of data. (See e.g. [0124], “sets of images that introduce new categories of classification (e.g. new classes), may be selected to be in the training dataset as whole, may be provided with a highest weight, or the like; in order to introduce the new categories in the re-training.”)
Accordingly, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention, having the teachings of Su, JUNG and Raveh before them, to include Raveh’s indicator for retraining the model Su and JUNG’s plurality of subsets. One would have been motivated to make such a combination in order to improve the accuracy, robustness and stability of the model, as suggested by Raveh (0038)
Regarding claim 14, Su and JUNG teaches the method of claim 11.
Su and JUNG do not teach wherein the indication to update the training dataset is received in response to additional data being added to the dataset comprising the plurality of data.
Raveh teaches wherein the indication to update the training dataset is received in response to additional data being added to the dataset comprising the plurality of data. (See e.g. [0025], “As an example, in response to obtaining a new dataset of data, a retraining of the classification model may be performed with respect to the new dataset” [i.e., obtaining a new dataset of data corresponding to additional data being added], “while completely forgetting the last training dataset, the oldest samples thereof, training using only a portion of the last training dataset, or the like. Such solution, despite being able to potentially reduce the amount of the training data for re-training” [i.e., re-training the training data corresponding to update the training dataset], “may result in losing data important to the training, such as old data that is more relevant to current samples than newer data, or the like.”)
Accordingly, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention, having the teachings of Su, JUNG and Raveh before them, to include Raveh’s indicator for retraining the model Su and JUNG’s plurality of subsets. One would have been motivated to make such a combination in order to improve the accuracy, robustness and stability of the model, as suggested by Raveh (0038)
Claims 16 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Su (US 20170091471 A1) in view of JUNG (US 20180018562 A1) further in view of Hua (US 20160217349 A1) further in view of TRUONG (US 20230244987 A1)
Regarding claim 16, Su, JUNG and Hua teach the method of claim 15.
Su, JUNG and Hua do not teach wherein at least some of the annotated labels in the training dataset have been applied, at least in part, by human-based annotation with implementation of historical information.
TRUONG teaches wherein at least some of the annotated labels in the training dataset have been applied, at least in part, by human-based annotation with implementation of historical information. (See e.g. [0035], “The set of observations may be obtained and/or input from training data (e.g., historical data), such as data gathered during one or more processes described herein. For example, the set of observations may include data gathered from a data labeling system that is used to generate a labeled dataset based on a combination of manual labeling” [i.e., manual labeling corresponding to human-based annotation], “automatic labeling, and user feedback, as described elsewhere herein.”)
Accordingly, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention, having the teachings of Su, JUNG, Hua and TRUONG before them, to include TRUONG’s correcting and modifying training sets Su, JUNG and Hua’s plurality of subsets. One would have been motivated to make such a combination in order to increase the accuracy and consistency of the model, as suggested by TRUONG (0054)
Regarding claim 19, Su, JUNG and Hua teach the computer-readable medium of claim 18.
Su, JUNG and Hua do not teach wherein the at least one additional item of data is selected in response to training of the machine learning algorithm failing to be verified
TRUONG teaches wherein the at least one additional item of data is selected in response to training of the machine learning algorithm failing to be verified. (See e.g. [0014], “In some implementations, the data labeling system may then use one or more data profilers to detect a structure or other attributes associated with the manually labeled data samples, where the one or more data profilers may include or may use one or more machine learning models that are trained to identify other data samples with a similar structure as the manually labeled data samples…However, in cases where the confidence values are low (e.g., fail to satisfy a threshold” [i.e., confidence values that are too low to satisfy a threshold corresponding to failing to be verified]), “the data labeling system may present the automatic labels to one or more users and prompt the one or more users to review the automatic labels.”)
Accordingly, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention, having the teachings of Su, JUNG, Hua and TRUONG before them, to include TRUONG’s correcting and modifying training sets Su, JUNG and Hua’s plurality of subsets. One would have been motivated to make such a combination in order to increase the accuracy and consistency of the model, as suggested by TRUONG (0054)
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KYLE ALLMAN THOMPSON whose telephone number is (571)272-3671. The examiner can normally be reached Monday - Thursday, 6 a.m. - 3 p.m. ET..
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kamran Afshar can be reached at (571) 272-7796. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/K.A.T./Examiner, Art Unit 2125
/KAMRAN AFSHAR/Supervisory Patent Examiner, Art Unit 2125
Prosecution Insights