Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-16 rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Regarding claim 1
Claim 1 – Step 1 – Is the claim to a process, machine, manufacture or composition of matter?
Yes, the claim is to a process.
Claim 1 – Step 2A – Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes, the claim recites an abstract idea.
a first acquisition step including acquiring result information about a result of a classification executed by a living being on a target; - this limitation is collecting information about a human’s classification decision. A person could observe the target, ask the human inspector for their classification, and write down the result on paper. These are activities that can be performed mentally or with pen and paper and thus, fall under “mental processes” (including observation/evaluation/judgment) that can be performed in the human mind or by a human with pen and paper, per MPEP § 2106.04(a)(2)(III).
a second acquisition step including acquiring execution information about execution of the classification; - this limitation recites collecting meta-information (“execution information”) about how the human performed the classification (e.g., decision time, condition, etc.). A person could manually note timing, environment, or subjective condition while the human classifies the target, i.e., achievable mentally or with pen and paper. As such, this limitation is a part of the judicial exception, falling under MPEP § 2106.04(a)(2)(III).
and a generation step including generating data for machine learning based on the result information and the execution information, - this limitation uses the previously collected information to generate “data for machine learning”. A person could take the recorded result and execution information, and manually construct a table or set of records representing training examples (with fields for input data, labels, and meta-data). This is merely organizing and annotating information according to rules and, as such, constitutes a mental process under MPEP § 2106.04(a)(2)(III).
the data for machine learning including learning data and evaluation information about evaluation of the learning data. – this limitation recites conceptual data structuring/annotation that can be done by a human following rules: assign each record both a label and an evaluation value/score and write them into a table. Accordingly, this limitation falls under MPEP § 2106.04(a)(2)(III) (mental processes; human organization and evaluation of data) and remains a part of the abstract idea.
Claim 1 – Step 2A – Prong 2 – Does the claim recite additional elements that integrate the judicial exception into a practical application?
No. There are no additional elements that integrate the judicial exception into a practical application.
Claim 1 – Step 2B – Does the claim recite additional elements that amount to significantly more than the judicial exception?
No. There are no additional elements that amount to significantly more than the judicial exception.
Regarding claim 2
Claim 2 – Step 1 – Is the claim to a process, machine, manufacture or composition of matter?
Yes, the claim is to a process.
Claim 2 – Step 2A – Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes, the claim recites an abstract idea.
wherein the evaluation information includes an evaluation value indicating a degree of accuracy of the learning data. – this limitation recites assessing how accurate a training example (learning data) is, and assigning a value (e.g., a score) that quantifies that accuracy. A person could perform this entirely mentally or with pen and paper by reviewing each data item, judging its correctness/accuracy, and writing down a numerical or qualitative “degree of accuracy” as an evaluation value. Thus, this limitation is merely further evaluation and annotation of information, constituting a mental process under MPEP § 2106.04(a)(2)(III).
Claim 2 – Step 2A – Prong 2 – Does the claim recite additional elements that integrate the judicial exception into a practical application?
No. There are no additional elements that integrate the judicial exception into a practical application.
Claim 2 – Step 2B – Does the claim recite additional elements that amount to significantly more than the judicial exception?
No. There are no additional elements that amount to significantly more than the judicial exception.
Regarding claim 3
Claim 3 – Step 1 – Is the claim to a process, machine, manufacture or composition of matter?
Yes, the claim is to a process.
Claim 3 – Step 2A – Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes, the claim recites an abstract idea.
wherein the learning data is data representing correspondence between target information about the target and the result information. – this limitation describes organizing and representing information, which a human can perform entirely mentally or with pen and paper. A person could write down, for each target, the target information in one column and the classification result in another, thereby creating “data representing correspondence” between them. (See MPEP § 2106.04(a)(2)(III) (“concepts performed in the human mind (including an observation, evaluation, judgment, opinion)”).
Claim 3 – Step 2A – Prong 2 – Does the claim recite additional elements that integrate the judicial exception into a practical application?
No. There are no additional elements that integrate the judicial exception into a practical application.
Claim 3 – Step 2B – Does the claim recite additional elements that amount to significantly more than the judicial exception?
No. There are no additional elements that amount to significantly more than the judicial exception.
Regarding claim 4
Claim 4 – Step 1 – Is the claim to a process, machine, manufacture or composition of matter?
Yes, the claim is to a process.
Claim 4 – Step 2A – Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes, the claim recites an abstract idea.
wherein the learning data includes supervisor data. – this recites an informational categorization step: deciding that some of the learning data is to be treated as “supervisor data” (e.g., training data) and labeling/organizing it accordingly. A person can do this mentally or with pen and paper by selecting certain records in a table and marking them as “supervisor” or “training” data. As such, this limitation falls within “mental processes” under MPEP § 2106.04(a)(2)(III).
Claim 4 – Step 2A – Prong 2 – Does the claim recite additional elements that integrate the judicial exception into a practical application?
No. There are no additional elements that integrate the judicial exception into a practical application.
Claim 4 – Step 2B – Does the claim recite additional elements that amount to significantly more than the judicial exception?
No. There are no additional elements that amount to significantly more than the judicial exception.
Regarding claim 5
Claim 5 – Step 1 – Is the claim to a process, machine, manufacture or composition of matter?
Yes, the claim is to a process.
Claim 5 – Step 2A – Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes, the claim recites an abstract idea.
wherein the execution information includes time information about a time it has taken for the living being to have the classification done. – this recites specifying that one type of “execution information” is elapsed time for the human to complete the classification. A person could perform this mentally or with pen and paper (e.g., using a stopwatch and writing down the elapsed time) thus, falling within “concepts performed in the human mind (including an observation, evaluation, judgment, opinion)” as described in MPEP § 2106.04(a)(2)(III).
Claim 5 – Step 2A – Prong 2 – Does the claim recite additional elements that integrate the judicial exception into a practical application?
No. There are no additional elements that integrate the judicial exception into a practical application.
Claim 5 – Step 2B – Does the claim recite additional elements that amount to significantly more than the judicial exception?
No. There are no additional elements that amount to significantly more than the judicial exception.
Regarding claim 6
Claim 6 – Step 1 – Is the claim to a process, machine, manufacture or composition of matter?
Yes, the claim is to a process.
Claim 6 – Step 2A – Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes, the claim recites an abstract idea.
wherein the execution information includes condition information about a condition of the living being. – this limitation consists of obtaining, identifying, or recording subjective information about a subjective state of the living being. Such observation, assessment, and recordation of subjective information is an activity capable of being performed in the human mind or with pen and paper, and therefore falls within the mental process category of abstract ideas under MPEP § 2106.04(a)(2)(III).
Claim 6 – Step 2A – Prong 2 – Does the claim recite additional elements that integrate the judicial exception into a practical application?
No. There are no additional elements that integrate the judicial exception into a practical application.
Claim 6 – Step 2B – Does the claim recite additional elements that amount to significantly more than the judicial exception?
No. There are no additional elements that amount to significantly more than the judicial exception.
Regarding claim 7
Claim 7 – Step 1 – Is the claim to a process, machine, manufacture or composition of matter?
Yes, the claim is to a process.
Claim 7 – Step 2A – Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes, the claim recites an abstract idea.
wherein the condition of the living being includes at least one of mental and physical conditions of the living being. – this limitation recites identifying, characterizing, and using mental and physical conditions of the living being as part of the “condition” already used in the execution information. This merely constitutes further observation/evaluation/recordation of information about a person’s state, and thus, a mental process under MPEP § 2106.04(a)(2)(III).
Claim 7 – Step 2A – Prong 2 – Does the claim recite additional elements that integrate the judicial exception into a practical application?
No. There are no additional elements that integrate the judicial exception into a practical application.
Claim 7 – Step 2B – Does the claim recite additional elements that amount to significantly more than the judicial exception?
No. There are no additional elements that amount to significantly more than the judicial exception.
Regarding claim 8
Claim 8 – Step 1 – Is the claim to a process, machine, manufacture or composition of matter?
Yes, the claim is to a process.
Claim 8 – Step 2A – Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes, the claim recites an abstract idea.
wherein the result information includes results of classifications executed by a plurality of the living beings, - this limitation is directed to collecting, comparing, and organizing information about classifications made by multiple living beings and relative information among those classifications. Such observation, comparison, and recordation of plural human classification results and their relative relationships can be performed in the human mind or with pen and paper, and therefore fall within the mental processes category of abstract ideas. (See MPEP § 2106.04(a)(2)(III)).
and the execution information includes relative information about the respective classifications executed by the plurality of the living beings. - this limitation consists of obtaining, identifying, or recording subjective information about a subjective state of the plurality of living beings. Such observation, assessment, and recordation of subjective information is an activity capable of being performed in the human mind or with pen and paper, and therefore falls within the mental process category of abstract ideas under MPEP § 2106.04(a)(2)(III).
Claim 8 – Step 2A – Prong 2 – Does the claim recite additional elements that integrate the judicial exception into a practical application?
No. There are no additional elements that integrate the judicial exception into a practical application.
Claim 8 – Step 2B – Does the claim recite additional elements that amount to significantly more than the judicial exception?
No. There are no additional elements that amount to significantly more than the judicial exception.
Regarding claim 9
Claim 9 – Step 1 – Is the claim to a process, machine, manufacture or composition of matter?
Yes, the claim is to a process.
Claim 9 – Step 2A – Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes, the claim recites an abstract idea.
wherein the execution information includes statistical information about statistics of the results of classifications executed by a plurality of the living beings on the target. – this limitation concerns forming and using statistical information about the results of classifications made by multiple living beings. Computing, summarizing, and recording statistics (e.g., frequencies, averages, distributions) over such classification results can be done in the human mind or with pen and paper. It is an observation/evaluation/organization of information and therefore falls within the mental processes category of abstract ideas under MPEP § 2106.04(a)(2)(III).
Claim 9 – Step 2A – Prong 2 – Does the claim recite additional elements that integrate the judicial exception into a practical application?
No. There are no additional elements that integrate the judicial exception into a practical application.
Claim 9 – Step 2B – Does the claim recite additional elements that amount to significantly more than the judicial exception?
No. There are no additional elements that amount to significantly more than the judicial exception.
Regarding claim 10
Claim 10 – Step 1 – Is the claim to a process, machine, manufacture or composition of matter?
Yes, the claim is to a process.
Claim 10 – Step 2A – Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes, the claim recites an abstract idea.
wherein the execution information includes subjective information about a subjective opinion of the living being on the classification. – this limitation recites obtaining and using subjective information about a subjective opinion of the living being on the classification. Determining, expressing, and recording a subjective opinion (e.g., what the person thinks about their classification) are activities that can be performed in the human mind or with pen and paper. They are thus “concepts performed in the human mind (including an observation, evaluation, judgment, opinion)” and fall squarely within the mental process category of abstract ideas under MPEP § 2106.04(a)(2)(III).
Claim 10 – Step 2A – Prong 2 – Does the claim recite additional elements that integrate the judicial exception into a practical application?
No. There are no additional elements that integrate the judicial exception into a practical application.
Claim 10 – Step 2B – Does the claim recite additional elements that amount to significantly more than the judicial exception?
No. There are no additional elements that amount to significantly more than the judicial exception.
Regarding claim 11
Claim 11 – Step 1 – Is the claim to a process, machine, manufacture or composition of matter?
Yes, the claim is to a process.
Claim 11 – Step 2A – Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes, the claim recites an abstract idea.
wherein the target includes an image. – the limitation merely specifies that the “target” on which the classification is executed includes an image. It is still directed to using information (here, image information) as the subject of the human classification, and to handling that information in the same way as in claim 1. Choosing an image as the target and treating it as the subject of classification is an information choice; observing and classifying an image, and recording the result, can be performed in the human mind or with pen and paper. Accordingly, the limitation falls within the mental process category under MPEP § 2106.04(a)(2)(III).
Claim 11 – Step 2A – Prong 2 – Does the claim recite additional elements that integrate the judicial exception into a practical application?
No. There are no additional elements that integrate the judicial exception into a practical application.
Claim 11 – Step 2B – Does the claim recite additional elements that amount to significantly more than the judicial exception?
No. There are no additional elements that amount to significantly more than the judicial exception.
Regarding claim 12
Claim 12 – Step 1 – Is the claim to a process, machine, manufacture or composition of matter?
Yes, the claim is to a process.
Claim 12 – Step 2A – Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes, the claim recites an abstract idea.
further comprising an adjustment step including removing learning data, of which the evaluation information fails to meet a standard, from the data for machine learning. – this limitation recites identifying learning data whose associated evaluation information “fails to meet a standard”, and removing that learning data from the dataset. Determining whether evaluation information meets or fails to meet a standard, and then deciding to remove corresponding records, is an evaluation and data selection operation that can be done entirely in the human mind or with pen and paper (e.g., scanning a table for rows whose score is below a threshold and crossing them out). This limitation constitutes as a mental process under MPEP § 2106.04(a)(2)(III).
Claim 12 – Step 2A – Prong 2 – Does the claim recite additional elements that integrate the judicial exception into a practical application?
No. There are no additional elements that integrate the judicial exception into a practical application.
Claim 12 – Step 2B – Does the claim recite additional elements that amount to significantly more than the judicial exception?
No. There are no additional elements that amount to significantly more than the judicial exception.
Regarding claim 13
Claim 13 – Step 1 – Is the claim to a process, machine, manufacture or composition of matter?
Yes, the claim is to a process.
Claim 13 – Step 2A – Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes, the claim recites an abstract idea.
A decision method comprising executing a classification of the target using a learned model, the learned model having been generated by machine learning using the learning data of the data for machine learning that has been generated by the data generation method of claim 1. – this limitation recites a method of using a learned model to classify a target, where the learned model itself is defined by how it was trained (using learning data from the data generation method of claim 1). The operations of classifying a target using a model, and defining that model in terms of training data originating from a prior abstract data-generation method, are mathematical/data-processing operations that can be conceptualized as mental steps or purely algorithmic steps (e.g., applying a decision function to inputs and yielding a classification). Such classification and use of a learned model is a mental process / abstract data-processing concept under MPEP § 2106.04(a)(2)(III) and MPEP § 2106.04(a).
Claim 13 – Step 2A – Prong 2 – Does the claim recite additional elements that integrate the judicial exception into a practical application?
No. There are no additional elements that integrate the judicial exception into a practical application.
Claim 13 – Step 2B – Does the claim recite additional elements that amount to significantly more than the judicial exception?
No. There are no additional elements that amount to significantly more than the judicial exception.
Regarding claim 14
Claim 14 – Step 1 – Is the claim to a process, machine, manufacture or composition of matter?
Yes, the claim is to a manufacture.
Claim 14 – Step 2A – Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes, the claim recites an abstract idea.
A non-transitory computer-readable tangible recording medium storing a program designed to cause one or more processors to perform the data generation method of claim 1. – this merely requires a program, that, when executed, causes processors to perform the same data generation method of claim 1. Claim 14 as a whole is directed to encoding instructions for performing the same mental process / abstract data processing scheme already identified for claim 1, now recited in the form of a program stored on a computer-readable medium. Accordingly, this limitation constitutes a mental process, falling under MPEP § 2106.04(a)(2)(III).
Claim 14 – Step 2A – Prong 2 – Does the claim recite additional elements that integrate the judicial exception into a practical application?
No. There are no additional elements that integrate the judicial exception into a practical application.
The additional elements:
“A non-transitory computer-readable tangible recording medium” – this recites a generic computer-readable medium. There is no recitation of any specialized or improved memory technology or any particular configuration that improves computer functioning. Rather, this is a typical storage medium used to hold software instructions and is thus a generic implementation environment. See MPEP § 2106.05(f) (mere instructions to apply an abstract idea on a generic computer / generic computer components).
“one or more processors” – this recites components used in their ordinary ways to implement the abstract method (no specialized memory, no improved hardware, no particular machine tie, no transformation). It does not integrate the abstract idea into a practical application. See MPEP § 2106.04(d); MPEP § 2106.05(a)-(c), (f).
Claim 14 – Step 2B – Does the claim recite additional elements that amount to significantly more than the judicial exception?
No. There are no additional elements that amount to significantly more than the judicial exception.
“A non-transitory computer-readable tangible recording medium” – this recites well-understood, routine, and conventional (WURC) computing components. The claim does not recite any non-generic structure or improvement tied to this medium. It therefore does not provide an inventive concept. (MPEP § 2106.05(d), 2106.05(f)).
“one or more processors” – having generic processors execute the stored program to perform the abstract data generation method is well-understood, routine, and conventional (WURC) computer activity. No unconventional processor design or non-routine use is claimed. The element does not add “significantly more”. (MPEP § 2106.05(d), 2106.05(f)).
Regarding claim 15
Claim 15 – Step 1 – Is the claim to a process, machine, manufacture or composition of matter?
Yes, the claim is to a manufacture.
Claim 15 – Step 2A – Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes, the claim recites an abstract idea.
A non-transitory computer-readable tangible recording medium storing a program designed to cause one or more processors to perform the decision method of claim 13 – this limitation merely requires a program that, when executed, causes processors to perform the same abstract decision method of claim 13. The recited “program designed to cause one or more processors to perform the decision method of claim 13” does not alter the underlying abstract idea; it simply embodies that abstract method as executable instructions on a medium. (MPEP § 2106.04(a)(2)(III).
Claim 15 – Step 2A – Prong 2 – Does the claim recite additional elements that integrate the judicial exception into a practical application?
No. There are no additional elements that integrate the judicial exception into a practical application.
The additional elements:
“A non-transitory computer-readable tangible recording medium” – this recites a generic computer-readable medium. There is no recitation of any specialized or improved memory technology or any particular configuration that improves computer functioning. Rather, this is a typical storage medium used to hold software instructions and is thus a generic implementation environment. See MPEP § 2106.05(f) (mere instructions to apply an abstract idea on a generic computer / generic computer components).
“one or more processors” – this recites components used in their ordinary ways to implement the abstract method (no specialized memory, no improved hardware, no particular machine tie, no transformation). It does not integrate the abstract idea into a practical application. See MPEP § 2106.04(d); MPEP § 2106.05(a)-(c), (f).
Claim 15 – Step 2B – Does the claim recite additional elements that amount to significantly more than the judicial exception?
No. There are no additional elements that amount to significantly more than the judicial exception.
The additional elements:
“A non-transitory computer-readable tangible recording medium” – this recites well-understood, routine, and conventional (WURC) computing components. The claim does not recite any non-generic structure or improvement tied to this medium. It therefore does not provide an inventive concept. (MPEP § 2106.05(d), 2106.05(f)).
“one or more processors” – having generic processors execute the stored program to perform the abstract data generation method is well-understood, routine, and conventional (WURC) computer activity. No unconventional processor design or non-routine use is claimed. The element does not add “significantly more”. (MPEP § 2106.05(d), 2106.05(f)).
Regarding claim 16
Claim 16 – Step 1 – Is the claim to a process, machine, manufacture or composition of matter?
Yes, the claim is to a machine.
Claim 16 – Step 2A – Prong 1 – Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Yes, the claim recites an abstract idea.
a first acquirer configured to acquire result information about a result of a classification executed by a living being on a target; - this limitation is collecting information about a human’s classification decision. A person could observe the target, ask the human inspector for their classification, and write down the result on paper. These are activities that can be performed mentally or with pen and paper and thus, fall under “mental processes” (including observation/evaluation/judgment) that can be performed in the human mind or by a human with pen and paper, per MPEP § 2106.04(a)(2)(III).
a second acquirer configured to acquire execution information about execution of the classification; - this limitation recites collecting meta-information (“execution information”) about how the human performed the classification (e.g., decision time, condition, etc.). A person could manually note timing, environment, or subjective condition while the human classifies the target, i.e., achievable mentally or with pen and paper. As such, this limitation is a part of the judicial exception, falling under MPEP § 2106.04(a)(2)(III).
and a generator configured to generate data for machine learning based on the result information and the execution information, - this limitation uses the previously collected information to generate “data for machine learning”. A person could take the recorded result and execution information, and manually construct a table or set of records representing training examples (with fields for input data, labels, and meta-data). This is merely organizing and annotating information according to rules and, as such, constitutes a mental process under MPEP § 2106.04(a)(2)(III).
the data for machine learning including learning data and evaluation information about evaluation of the learning data. - this limitation recites conceptual data structuring/annotation that can be done by a human following rules: assign each record both a label and an evaluation value/score and write them into a table. Accordingly, this limitation falls under MPEP § 2106.04(a)(2)(III) (mental processes; human organization and evaluation of data) and remains a part of the abstract idea.
Claim 16 – Step 2A – Prong 2 – Does the claim recite additional elements that integrate the judicial exception into a practical application?
No. There are no additional elements that integrate the judicial exception into a practical application. The additional elements:
“A data generation system” – this merely characterizes the collection of components performing the abstract data-processing functions. No specific hardware structure or improvement to machine is recited. This is a generic system label and does not integrate the abstract idea into a practical application. (MPEP § 2106.04(d); 2106.05(a), (b), (f)).
Claim 16 – Step 2B – Does the claim recite additional elements that amount to significantly more than the judicial exception?
No. There are no additional elements that amount to significantly more than the judicial exception. The additional elements are:
“A data generation system” – standard computer system (interfaces, storage, processor) executing a program on conventional hardware is well-understood, routine, and conventional (WURC); no inventive concept. (MPEP § 2106.05(d), 2106.05(f)).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-11, and 13-16 are rejected under 35 U.S.C. 103 as being unpatentable over Cox et al., (US20160148077A1) in view of Zadeh et al., (US20180204111Al).
Regarding claim 1, Cox in view of Zadeh, teach a data generation method comprising:
a first acquisition step including acquiring result information about a result of a classification executed by a living being on a target; - Cox teaches this limitation. Cox discloses human annotators that classify and store training objects:
“training objects are classified by human annotators, psychometric data characterizing the annotation of the training objects is acquired…” (Cox, §Abstract)
“Classification data is received… training objects annotated… by at least some of the annotators with classifications for features thereof.” (Cox, p. 1, ¶[0008])
“classification data comprising annotations received from a plurality of human annotators.” (Cox, p. 2, ¶[0010])
a second acquisition step including acquiring execution information about execution of the classification; - Cox teaches this limitation. Cox discloses “psychometric data” characterizing how the human made the classification:
“psychometric data characterizing the annotation of the training objects acquired…” (Cox, §Abstract)
“The psychometric data may include or consist essentially of (i) response time for classifying one or more features, (ii) accuracy of feature classification, and/or (iii) presentation time…” (Cox, p. 2, ¶[0009])
“accuracy of image characterization, response time, presentation time, etc., is recorded in the database…” (Cox, p. 5, ¶[0045]; FIG. 3)
and a generation step including generating data for machine learning based on the result information and the execution information, - Cox teaches this limitation. Cox combines classification data (“result information”) + psychometric data (“execution information”) and uses them for training:
“Psychometric data characterizing the annotation of the training objects by the annotators is acquired. A human-weighted loss function based at least in part on the classification data and the psychometric data is computationally derived.” (Cox, p. 1, ¶[0008])
“Computationally deriving the human-weighted loss function may include or consist essentially of… generating training data, (ii) comparing the training data to the classification data… assigning the penalties for misclassification… incorporating the penalties… to generate the human-weighted loss function.” (Cox, p. 1-2, ¶[0009])
“The human-weighted loss function includes penalties for misclassification, magnitudes of the penalties increasing with increasing deviation from the classification data.” (Cox, p. 1-2, ¶[0009])
the data for machine learning including learning data
“assigning the penalties for misclassification… incorporating the penalties… to generate the human-weighted loss function.” (Cox, p. 1-2, ¶[0009])
“one or more features of a query object are computationally classified based at least in part on the human weighted loss function.” (Cox, §Abstract)
Cox does not teach:
“… and evaluation information about evaluation of the learning data.”
Zadeh, however, teaches this limitation:
“… and evaluation information about evaluation of the learning data.” – Zadeh discloses per-sample reliability factors for training samples:
“Z-factors include reliability factor, confidence factor, expertise factor, bias factor, and the like, which is associated with each Z-node in the Z-web.” (Zadeh, p. 74, ¶[1588])
“an error function (to be minimized by training) defined over the training sample space… accounts for data sample reliability by including sample reliability factor as a weight in the contribution of the data sample to the batch error function…” (Zadeh, p. 85, ¶[1718])
These disclosures show that each training sample (“learning data”) has an associated reliability factor and that factor is explicitly used as evaluation information in the training error function.
Cox teaches generating machine-learning training data from human annotations, including both classification data and psychometric data (e.g., accuracy, response time) to characterize how reliably each training example was produced. Zadeh likewise operates in supervised machine learning and teaches associating per-sample reliability/confidence factors with training samples and using those factors in the learning process. Because both references address the same problem (variable reliability in human-generated training data), and because Zadeh’s solution of assigning explicit evaluation values to each training sample is a known technique for improving model training, it would have been obvious to a POSITA to incorporate Zadeh’s per-sample reliability factors into Cox’s existing framework. Doing so merely applies a known improvement (explicit evaluation information for each training example) to Cox’s system in the same field and for the same purpose, yielding the predictable result of more informed weighting of human-supplied training data. (MPEP § 2143; “use of known techniques to improve similar device in same way”).
Regarding claim 2, Cox in view of Zadeh, teach the data generation method of claim 1, wherein
the evaluation information includes an evaluation value indicating a degree of accuracy of the learning data. – Cox teaches this limitation. Cox discloses that a description of accuracy is one of the psychometric measures:
“The psychometric data may include or consist essentially of (i) response time for classifying one or more features, (ii) accuracy of feature classification, and/or (iii) presentation time…” (Cox, p. 2, ¶[0009])
The “accuracy of feature classification” is an evaluation value that numerically (or at least quantitatively) reflects how accurately the human annotators classified that particular training object/feature. That is exactly a “degree of accuracy” for the underlying learning data (training examples).
“Computationally deriving the human-weighted loss Function may include… generating training data, (ii) comparing the training data to the classification data to identify… misclassified… features… assigning the penalties… incorporating the penalties… to generate the human-weighted loss function.” (Cox, p. 1-2, ¶[0009])
This discloses that each training object (“learning data”) has classification data (result information), and psychometric data including accuracy of feature classification (an evaluation value indicating a degree of accuracy).
Regarding claim 3, Cox in view of Zadeh, teach the data generation method of claim 1, wherein
the learning data is data representing correspondence between target information about the target and the result information. – Cox teaches this limitation. Cox discloses storing both the training object and its classification together in a database:
“The classification and psychometric data may be stored in a database 175 of training objects accessible by the server 150.” (Cox, p. 4, ¶[0034])
And describes standard supervised learning:
“a human annotator assigns labels to a set of training objects, in a step 405, the labeled data is utilized to train a machine classifier,.” (Cox, p. 6, ¶[0047])
Thus, each training record in Cox’s database embodies:
target information (training object, e.g., image), and
result information (label / classification data)
stored together as labeled training data, exactly as the claim requires.
Cox explicitly teaches labeled training data where each training object (target information) is paired with its human-supplied label (result information) and stored in a training database, and that labeled data is used to train a classifier. This is the standard supervised-learning representation of “learning data” as a correspondence between target and label.
Regarding claim 4, Cox in view of Zadeh, teach the data generation method of claim1, wherein
the learning data includes supervisor data. – Cox explicitly teaches this limitation. Cox describes a human annotator assigning labels to training objects, and those labels are sued to train a classifier:
“a human annotator assigns labels to a set of training objects, in a step 405, the labeled data is utilized to train a machine classifier,.” (Cox, p. 6, ¶[0047])
Cox also distinguishes training corpus and test corpus, each of which “can include pre-classified human-labeled data”, i.e., supervised data.
Cox already teaches learning data that consists of supervised, human-labeled training examples used to train a machine classifier. Treating such labeled training data as “supervisor data” is simply applying the conventional supervised-learning notion in different words.
Regarding claim 5, Cox in view of Zadeh, teach the data generation method of claim 1, wherein
the execution information includes time information about a time it has taken for the living being to have the classification done. – Cox teaches this limitation. Cox explicitly defines psychometric data to include response time and presentation time:
“The psychometric data may include or consist essentially of (i) response time for classifying one or more features, (ii) accuracy of feature classification, and/or (iii) presentation time of one or more training objects.” (Cox, pg. 2, ¶[0009])
Cox also reiterates that the database stores this psychometric data along with the classification data for each training object.
Mapping:
“living being” [Wingdings font/0xE0] human annotator (Cox: “human annotators”),
“time information about a time it has taken … to have the classification done” [Wingdings font/0xE0] “response time for classifying one or more features” and/or “presentation time of one or more training objects”, both of which are explicit time measures of how long the annotator took to make the classification.
Cox discloses that its execution-related psychometric data includes “response time for classifying one or more features” and “presentation time of one or more training objects”, which is time information about how long the human annotator (a living being) takes to perform the classification, corresponding to the claimed “execution information includ[ing] time information about a time it has taken for the living being to have the classification done.”
Regarding claim 6, Cox in view of Zadeh, teach the data generation method of claim 1, wherein
the execution information includes condition information about a condition of the living being. – Cox does not teach this limitation. Zadeh, however, teaches this limitation. Zadeh introduces Z-factors expressly including subjective attributes of a person (speaker/annotator):
“Z-factors include reliability factor, confidence factor, expertise factor, bias factor, truth factor, trust factor, validity factor, "trustworthiness of speaker", "sureness of speaker", "statement helpfulness", "expertise of speaker", "speaker's truthfulness", "perception of speaker (or source of information)", "apparent confidence of speaker", "broadness of statement", and the like which is associated with each Z-node in the Z-web.” (Zadeh, pg. 7, ¶[0223])
These are explicitly subjective properties of a person (e.g., “confidence factor”, “sureness of speaker”, “apparent confidence of speaker”, “trustworthiness of speaker”), and Zadeh explains that such Z-factors are associated with nodes representing people and used in analytics and machine learning contexts. Thus, Zadeh teaches subjective information (confidence, sureness, perceived trustworthiness, etc.) about the state of the person providing information/annotations. This corresponds to “subjective information about a subjective state of the living being when the classification is executed”.
A person of ordinary skill in the art, faced with Cox’s problem of evaluating the reliability of human-generated data, would have been motivated to augment Cox’s execution information with subjective-state information as taught by Zadeh, e.g., confidence factor, sureness, trustworthiness, so that each classification by a human annotator is accompanied not only by objective timing/accuracy but also by subjective information about the annotator’s state or confidence when ethe classification is made. This is a straightforward application of a known technique (per-person subjective reliability/ confidence factors) to improve a similar system (Cox’s human-annotated ML training) in the same field, yielding a predictable improvement in assessing training-example quality. (MPEP § 2143).
Regarding claim 7, Cox in view of Zadeh, teach the data generation method of claim 6, wherein
the condition of the living being includes at least one of mental and physical conditions of the living being. – Cox does not teach this limitation. Zadeh, however, teaches this limitation. Zadeh discloses that Z-nodes may represent people, emotions, mood, etc., and that Z-factors associated with such nodes include explicitly mental/subjective attributes:
“Z-factors include reliability factor, confidence factor, expertise factor, bias factor, truth factor, trust factor, validity factor, "trustworthiness of speaker", "sureness of speaker", "statement helpfulness", "expertise of speaker", "speaker's truthfulness", "perception of speaker (or source of information)", "apparent confidence of speaker", "broadness of statement", and the like which is associated with each Z-node in the Z-web.” (Zadeh, pg. 7, ¶[0223])
“and Z-nodes, for the understanding of relationships between objects, subjects, abstract ideas, concepts, or the like, including face, car, images, people, emotions, mood, …” (Zadeh, pg. 7, ¶[0223])
These are mental/psychological conditions (emotions, mood, confidence, sureness, trustworthiness, etc.) associated with a person who provides information. Thus Zadeh teaches that the “condition of the living being”: includes mental conditions of the living being.
Physical condition of the living being – obvious variation. Once the practitioner is taught by Zadeh to model and record mental conditions of the annotator (emotions, mood, confidence, sureness, etc.) as part of the annotator’s “condition” used in machine learning, it would have been an obvious design choice to also include physical conditions (e.g., fatigue, health, other physiological states affect reliability of human judgments. Extending the already-used “condition” field from mental attributes to also encompass physical attributes is therefore a straightforward and predictable variation to further refine the same reliability modeling goal. See MPEP § 2143.
The added limitation that “the condition of the living being includes at least one of mental and psychical conditions of the living being” would have been obvious in view of Zadeh’s teaching of mental conditions of the annotator (emotions, mood, confidence, sureness, trustworthiness, etc.) and the routine extension to include physical conditions.
Regarding claim 8, Cox in view of Zadeh, teach the data generation method of claim 1, wherein
the result information includes results of classifications executed by a plurality of the living beings, - Cox teaches this limitation. Cox discloses that multiple human annotators classify the training objects and their classification results are collected:
“Data corresponding to a plurality of training objects is provided, over a computer network, to a plurality of training devices each associated with one of a plurality of human annotators. … Classification data is received via communication interfaces of at least some of the training devices. The classification data includes or consists essentially of at least some of the training objects annotated, via annotation interfaces of the training devices, by at least some of the annotators with classifications for features thereof.” (Cox, pg. 1, ¶[0008])
and the execution information includes relative information about the respective classifications executed by the plurality of the living beings. – Cox teaches this limitation. Cox further discloses that, for each training object, psychometric data is acquired and stored for the plurality of human annotators:
“for each of a plurality of training objects, (i) classification data … and (ii) psychometric data characterizing the annotation of the training object by the plurality of human annotators.” (Cox, pg. 2, ¶[0010])
“psychometric data is also acquired that characterizes the annotation of the training objects by the annotators… such psychometric data may include or consist essentially of response times for classifying one or more features, the accuracy of feature classification, and/or the presentation time…” (Cox, pg. 4, ¶[0034])
Cox already teaches using multiple human annotators to classify the same training objects and storing both their classification data and psychometric data (response time, accuracy, presentation time), per annotator, thereby inherently providing relative information between annotators for the same target.
Regarding claim 9, Cox in view of Zadeh, teach the data generation method of claim 1, wherein
the execution information includes statistical information about statistics of the results of classifications executed by a plurality of the living beings on the target. – Cox teaches this limitation. Cox teachers that a plurality of human annotators classify the same training objects and that statistics of their classification performance (accuracy, response time, presentation time) are computed and used:
“for each of a plurality of training objects, (i) classification data … and (ii) psychometric data characterizing the annotation of the training object by the plurality of human annotators.” (Cox, pg. 2, ¶[0010])
Cox then explicitly refers to “individual and combined statistics” over those human results:
“The weights are determined by the classification data collected from the annotators 160, including the individual and combined statistics of accuracy, response time and presentation time, over a sampling of images across varying degrees of difficulty.” (Cox, pg. 6, ¶[0052])
Cox also explains that accuracy and response time themselves are analyzed statistically across subjects and stimuli:
“mean accuracy provides a measure of how difficult a given trial is over the subject population. Such data may be analyzed even more finely; for example, various embodiments utilize mean accuracy per presentation time, presentation location, per condition, condition by subject, and so on.” (Cox, pg. 5, ¶[0044])
These passages show that Cox’s execution information includes statistical information (individual and combined statistics, mean accuracy, etc.) about statistics of the results of classifications (accuracy, response time, presentation time) executed by a plurality of human annotators on each image/target. This corresponds directly to “statistical information about statistics of the results of classifications executed by a plurality of the living beings on the target.”
Regarding claim 10,