Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of the Claims
Claims 1, 12, and 13 are amended. Claims 2 and 5 are canceled. Claims 1, 3, 4, and 6-13 are pending.
Response to Arguments
Applicant's arguments filed 04/16/2026 regarding 35 U.S.C. 101 have been fully considered but they are not persuasive.
Applicant argues that the claim is no longer directed to an abstract idea but instead to a specific computer-implemented data processing architecture that operates on structured data stored in a database and uses the data to control subsequent system execution. Examiner disagrees. The claims recite an abstract process or business process that is implemented via computer. Even with alleged “structured databases” and conditional logic, this does not automatically result in patent eligible subject matter. Alice Corp. Pty. Ltd. v. CLS Bank Int'l, 573 U.S. 208 (2014). First, starting with the argument that the claim is no longer directed to an abstract idea, that argument is invalid. The Federal Circuit has explained that "the 'directed to' inquiry applies a stage-one filter to claims, considered in light of the specification, based on whether 'their character as a whole is directed to excluded subject matter."' Enfish, LLC v. Microsoft Corp., 822 F.3d 1327, 1335 (Fed. Cir. 2016) (quoting Internet Patents Corp. v. Active Network, Inc., 790 F.3d 1343, 1346 (Fed. Cir. 2015)). It asks whether the focus of the claims is on a specific improvement in relevant technology or on a process that itself qualifies as an "abstract idea" for which computers are invoked merely as a tool. See id. at 1335-36. Here, it is clear from the Specification (including the claim language) that the independent claims focus on an abstract idea, and not on an improvement to technology and/or a technical field. The specification is titled “Method and System for Automatically Evaluating A Candidate”, and observes in the Description that most of the recruiting processes in any industry requires an evaluation of the technical abilities and knowledge of the candidate regarding the position they are applying for and the level of experience. This evaluation usually requires asking the candidate a list of technical questions about the field of the position and its difficulty to assess their skill level based on their answers. This process can be time consuming, as most of the time it takes a person to orally ask the questions to the candidate. Depending on the position and the number of candidates, it can take a large amount of time for the interviewer. Another drawback of this process is that the subjectivity of the interviewer regarding the candidate may alter their evaluation and lead to reject a candidate, who may however be suitable for the position or, on the contrary, hire a candidate, which will eventually prove to be unsuitable for the position. A known solution to try to remedy these problems is to use an evaluation software to run through a set of questions and collect the corresponding answers of the candidate, thus avoiding the time loss for the interviewers. However, this solution still requires a human analysis of the collected answers as such a software is usually not able to assess by itself the validity of the technical answers of the candidates. A known solution to try to address this issue is to use a multiple-choice questionnaire, but this requires questions that are simple enough and that may thus not be accurate enough for a complex position with a specific and thorough set of skills, such as e.g., a technical position. It is therefore an object of the present invention to provide an easy, reliable and efficient method and device to solve at least partly the drawbacks of the prior art. (see Spec. [0002]-[0009]). That is, as indicated in the applicant’s specification, the evaluation of the technical abilities and knowledge of the candidate regarding the position they are applying for and the level of experience in a manner that is not as time consuming for the interviewer, reduce subjectivity of the interviewer, and increase analysis of validity of the answers to the questions. The invention and claims are drawn towards automatically evaluating a candidate using a set of questions, and the claims recite limitations the directly correspond to certain methods of organizing human activity (managing personal interactions or relationships), as shown by limitations detailing selecting questions stored having associated difficulty level, keywords, model answer; retrieving the question selected and a model answer; determining the associated difficulty level based on a previously received accuracy score and automatically selecting a subsequent question having a difficulty level; sending the question to the user; receiving words corresponding to an answer; processing the answer from the user/candidate by comparing words with stored answers, comparing keywords associated with the question; generating an accuracy score based on the processing, etc. The claim limitations also directly correspond to mental processes (observation, evaluation, judgment, opinion), as evidenced by the claim limitations detailing steps that observes and analyzes data and making a decision (judgment or option) based on the observed and evaluated data (e.g., evaluating the answer for a user and scoring the answer based on accuracy, as well scoring an overall candidate using the accuracy sore). The claims recite an abstract idea. The claimed operations of presenting questions, receiving answers, comparing answers to stored model answers, scoring responses, and adjusting difficulty levels are fundamentally the abstract concept of interviewing and evaluating candidates, which directly corresponds to both certain methods of organizing human activity and mental processes. The use of generic computer components such as databases to store questions and answers, and using what appears to be conditional logic to select subsequent questions based on the scores are conventional data management techniques and do noes not transform the nature of the claim. Further, the use of generic computer components does not take the claims out of the judicial exception groupings nor render the claims patent eligible. Claims can recite a mental process even if they are claimed as being performed on a computer. If the claimed invention is described as a concept that is performed in the human mind and applicant is merely claiming that concept performed 1) on a generic computer, or 2) in a computer environment, or 3) is merely using a computer as a tool to perform the concept, the claim is considered to recite a mental process. This is the case in the applicant’s invention. The claims recite a judicial exception merely being performed or implemented via a computer.
Applicant argues that the operations cannot be performed in the human mind because they involve stored datasets and coordinate comparisons across stored data. This argument is unpersuasive. The argument is conflating the volume of data with technical complexity. Claims are directed to an abstract idea when the recited steps reflect mental prosses that humans perform, regardless of whether they are merely being implement via computer. A human interviewer routinely asks candidates questions, compares the candidate’s answer to a known or correct model answer, assigns a score or evaluation, selects harder or easier follow-up questions based on the candidate’s answer to pervious questions. These are the operations recited in the claims. The fact that the claims recite storing data (questions, answers, etc.) and using a computer or computing software to perform comparisons does snot elevate the claims beyond the mental process grouping. Further, the sub-groupings of certain methods of organizing human activity encompass both activity of a single person (and activity that involves multiple people, and thus, certain activity between a person and a computer may fall within the "certain methods of organizing human activity" grouping. (MPEP §2106.04(a)(2)(II)). Additionally, as indicated above, implement the limitations that correspond to mental processes via computer components (databases, etc.) does not take the claims out of the mental processes grouping. A human mind very well can perform the observation and evaluation of data, and implementing a decision (judgment or opinion) based on the observed and evaluated data. It is not required that the human mind be able to perform the claimed steps in the same manner, at the same scale, or with the same speed as a computer.
Applicant argues that the claims define a particular machine-based implementation rather than genialized human activity. Examiner disagrees. It is noted that while the application of a judicial exception by or with a particular machine is an important clue, it is not a stand-alone test for eligibility. It is also important to note that a general purpose computer that applies a judicial exception, such as an abstract idea, by use of conventional computer functions does not qualify as a particular machine. Ultramercial, Inc. v. Hulu, LLC, 772 F.3d 709, 716-17, 112 USPQ2d 1750, 1755-56 (Fed. Cir. 2014). Applicant’ claims describe a generic computer performing conventional computing functions or operations—storing data, executing comparisons, outputting results, etc. Further, as indicated previously, the computer is merely performing the limitations corresponding to the judicial exception groupings. This is precisely the type of claims that the Supreme Court cautioned against in Alice: simply implementing a fundamental practice (in this case, interviewing and assessing candidates) on a computer.
Applicant further argues that their claims are similar to McRO, Inc. v. Bandai Namco Games America Inc., 837 F.3d 1299 (Fed. Cir. 2016). Examiner disagrees. Applicant’s claims and invention are in no way analogous to McRO. The basis for the court’s decision in McRO was that the claims improved a computer-related technology by enabling the computer to perform functions that previously could not be performed by a computer and that required the subjective judgement of a human. The court emphasized both the specific claiming of the rules and the specification’s explanation of how the claimed rules enabled the automation of these specific animation tasks that previously could not be automated. This enabling of functionality that could not previously be performed by a computer was what amounted to the improvement in computer-related technology, not the simple recitation of a set of particular rules. The Federal Circuit found that the combined order of the specific rules in McRO achieved an improved technological result in the field of computer animation. Examiner notes that the claim limitations are not analogous to a computer-related technology that enables new functions that a computer could not have previously performed. It is indisputable that interviewing candidates, in the same manner of the applicant’s claims, are not a new function that a computer could not have previously performed. The claims do not define particular riles or algorithmic structure that is different from how an interviewer would conduct the same evaluation. The comparison of answers to model answers and adjustment of difficulty level of the questions are computational steps analogous to human mental steps. The claims do not give a specific rule-based structure that produces a result not achievable by humans. The claims lack the specificity and unconventional rule-based approach of McRO.
Applicant argues that the claims define how a language model processes data such as by comparing responses to stored keywords. Applicant’s argument is unpersuasive regarding patent eligibility. Keyword comparison is a conventional data processing technique, and merely invoking a language model module as the component performing the keywords comparison does not confer patent eligibility. The claims as a whole still does not recite something beyond the abstract idea. The comparison as claimed also still corresponds to mental processes.
Applicant argues that using scores to update difficulty levels establishes a stateful feedback loop that integrates the judicial exception into a practical application. Examiner disagrees. The feedback loop described—storing accuracy scores, using the scores to update difficulty levels, and selecting subsequent question based on the scores and difficulty levels—is further directed to adaptive interviewing which correspond to the judicial exception and indicated above. The ”stateful feedback loop” is not a technical improvement to computer functionality but a description of the abstract idea itself: to evaluate performance and adjust accordingly. It is important to note, the judicial exception alone cannot provide the improvement. The improvement can be provided by one or more additional elements. In addition, the improvement can be provided by the additional element(s) in combination with the recited judicial exception. See MPEP §2106.05(a). At best, the alleged improvement is an improvement in the judicial exception itself and not an improvement in computers or technology. It is important to keep in mind that an improvement in the judicial exception itself (e.g., a recited fundamental economic concept) is not an improvement in technology (emphasis added). For example, in Trading Technologies Int’l v. IBG LLC, the court determined that the claim simply provided a trader with more information to facilitate market trades, which improved the business process of market trading but did not improve computers or technology. Similarly, the Applicant’s claim recitations are an improvement in the judicial exception, not an improvement in technology. Reducing reliance on human intervention consistent with applying the judicial exception via computer is not an improvement in computers or technology. Improving the performance of an abstract process does not confer eligibility even when being implemented via computer. Here, the improvement to interviewing is not an improvement to the operation of the computer itself. The computer performs no differently, it simply executes the adaptive steps in service of the abstract concept.
Applicant argues that their claims are similar to Enfish, LLC v. Microsoft Corp., 822 F.3d 1327 (Fed. Cir. 2016). Examiner disagrees. Enfish involved claims directed to improvements in how computers process, store, or retrieve data. Improvements that changed the technical operation of the computer itself (claims directed to a self-referential table that improved database efficiency). In applicant’s invention, the claims do not alter how a database stores data, how a processor executes instructions, nor how a network communicates information. The computer architecture in the claims is not unconventional and it is the interviewing and evaluation process that is allegedly improved, not the technology used to perform the interviewing and evaluation.
Under Step 2B, applicant argues that the claims recite a closed-loop, data-driven, processing architecture. This argument is unpersuasive. No additional element or combination of additional elements of the claim amounts to significantly more than the abstract idea itself. The characterization of the arrangement as a closed-loop architecture is a functional label, not a structural distinction that reflects unconventional computer implementation. The independent claims recite the additional elements of an automatic evaluation system comprising a control module, a database, a user interface, a language model module, a non-transitory computer program (claim 12), and computer (claim 12) . The additional elements are computer components recited at a high-level of generality performing the above-mentioned limitations. The combination of the additional elements are no more than mere instructions to apply the judicial exception using a generic computer. Additionally, the language model module amounts to generally linking the judicial exception to a particular field of use (generating accuracy scores in evaluating candidate answers). The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. The additional elements amount to no more than mere instructions to apply the exception using a generic computer, and generally linking the judicial exception to a particular field of use. Mere instructions to apply an exception using a generic computer cannot provide an inventive concept. Thus, when viewed as an ordered combination, nothing in the claims add significantly more (i.e. an inventive concept) to the abstract idea. The claims are not patent eligible.
Applicant’s claims are also not analogous to BASCOM, 827 F.3d at 1345. In BASCOM the court found an inventive concept in the specific technical arrangement of the filtering software installed at a remote ISP server location, allowing individualized filtering that neither generic nor remote filter could achieve. The claims in BASCOM recite a specific unconventional technological arrangement, not merely an abstract idea implement via computer. The applicant has not identified any analogous non-generic structural arrangement of computer components. There is no specific technical deployment or arrangement that provides functionality unavailable from normal computing operations.
The 35 U.S.C. 101 rejection is maintained.
Prior art rejections have been added to this Office Action in light of the claim amendments.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1, 3, 4, and 6-13 are rejected under 35 U.S.C. 101 because the claimed invention is directed to a judicial exception (i.e. an abstract idea) without significantly more.
Claims 1, 3, 4, and 6-11 recite a method (i.e. process), claim 12 recites non-transitory computer-readable medium (i.e. machine or article of manufacture), and claim 13 recites a system (i.e. machine). Therefore claims 1, 3, 4, and 6-13 within one of the four statutory categories of invention.
Independent claims 1, 12, and 13 recite the limitations: a first phase being iterative and comprising, at each iteration, steps of selecting, from the set of questions stored, a question having an associated difficulty level, a set of expected keywords and at least one stored model answer, said question that is selected being different from questions selected at previous iterations, if any; retrieving the question that is selected with the associated difficulty level, the set of expected keywords and the at least one stored model answer thereof; determining the associated difficulty level based on at least one previously received accuracy score and automatically selecting a subsequent question having a difficulty level corresponding to the difficulty level that is determined, sending the question that is selected to said user; receiving a set of words corresponding to an answer from a candidate to the question that is selected and sent to said user; processing the set of words corresponding to the answer from the candidate by comparing the set of words with the set of expected keywords associated with the question that is selected; analyzing a semantic relevance of the set of words relative to the at least one stored model answer associated with the question that is selected; comparing the set of words with stored answers corresponding to the question that is selected; generating an accuracy score based on said processing; storing the accuracy score that is generated in association with the question that is selected; updating the difficulty level based on the accuracy score that is stored; selecting a subsequent question from the set of questions stored based on the accuracy score that is stored and associated difficulty levels, wherein the first phase is carried out until a predeterminate condition has been reached, wherein the predetermined condition comprisesdetermining that the accuracy score that is stored satisfies a predefined threshold condition corresponding to a target accuracy level, thereby terminating the first phase; executing a second phase comprising computing an evaluation score of the candidate using the accuracy score from all of each question of the set of questions. The invention and claims are drawn towards automatically evaluating a candidate using a set of questions, and the claims recite limitations the directly correspond to certain methods of organizing human activity (managing personal interactions or relationships), as shown by limitations detailing selecting questions stored having associated difficulty level, keywords, model answer; retrieving the question selected and a model answer; determining the associated difficulty level based on a previously received accuracy score and automatically selecting a subsequent question having a difficulty level; sending the question to the user; receiving words corresponding to an answer; processing the answer from the user/candidate by comparing words with stored answers, comparing keywords associated with the question; generating an accuracy score based on the processing, etc. The claim limitations also directly correspond to mental processes (observation, evaluation, judgment, opinion), as evidenced by the claim limitations detailing steps that observes and analyzes data and making a decision (judgment or option) based on the observed and evaluated data (e.g., evaluating the answer for a user and scoring the answer based on accuracy, as well scoring an overall candidate using the accuracy sore). The claims recite an abstract idea.
The judicial exception is not integrated into a practical application simply because the claims recite the additional elements of: an automatic evaluation system comprising a control module, a database, a user interface, a language model module, a non-transitory computer program (claim 12), and computer (claim 12) . The additional elements are computer components recited at a high-level of generality performing the above-mentioned limitations. The combination of the additional elements are no more than mere instructions to apply the judicial exception using a generic computer. Additionally, the language model module amounts to generally linking the judicial exception to a particular field of use (generating accuracy scores in evaluating candidate answers). Accordingly, in combination, these additional elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea. The claims are directed to an abstract idea.
The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional elements amount to no more than mere instructions to apply the exception using a generic computer, and generally linking the judicial exception to a particular field of use. Mere instructions to apply an exception using a generic computer cannot provide an inventive concept. Thus, when viewed as an ordered combination, nothing in the claims add significantly more (i.e. an inventive concept) to the abstract idea. The claims are not patent eligible.
Dependent claim 8 recites the limitation: prior to submitting the question that is selected to the [user interface], a step of converting words of the question that is selected into an audio stream by a [text conversion module], providing said audio stream to the [user interface] and diffusing said audio stream that is received by the [user interface] to the candidate. The claim is further directed to the abstract idea analyzed above. The claim also recites the additional elements of the user interface and a text conversion module. The additional elements amount to “apply it” or merely using a computer as a tool to implement the judicial exception. The text conversion module also amounts to generally linking the judicial exception to a particular field of use. Accordingly, in combination, these additional elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea. Further, when viewed as an ordered combination, nothing in the claim adds significantly more (i.e. an inventive concept) to the abstract idea. The claim is not patent eligible.
Dependent claim 9 recites the limitation: when the answer has been given orally by the candidate, a step of recording said answer that has been given orally as an audio stream and converting said audio stream into a set of words by an [audio conversion module]. The claim is further directed to the abstract idea analyzed above. The claim also recites the additional element of the audio conversion module. The additional element amounts to “apply it” or merely using a computer as a tool to implement the judicial exception, and generally linking the judicial exception to a particular field of use. Accordingly, in combination, the additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. Further, when viewed as an ordered combination, nothing in the claim adds significantly more (i.e. an inventive concept) to the abstract idea. The claim is not patent eligible.
Dependent claims 3, 4, 6, 7, 10, and 11 recite additional limitations that are further directed to the abstract idea analyzed in the rejected claims above. The claims also recite additional elements that have been analyzed in the rejected claims above. Thus, claims 3, 4, 6, 7, 10, and 11 are also rejected under 35 U.S.C. 101. The claims are not patent eligible.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1, 3, 4, 6-13 is/are rejected under 35 U.S.C. 103 as being unpatentable over Wu (2018/0150739) in view of Thibodeaux (2019/0370719).
Claim 1: Wu discloses: A computer-implemented method for automatically evaluating a candidate using a set of questions, said computer-implemented method being executed by an automatic evaluation system comprising a control module, a database storing said set of questions, a user interface and a language model module, and the computer-implemented method comprising: (Wu ¶0014 disclosing a system for automated interviewing of candidates; the system includes at least one processor and a memory; ¶0074 disclosing the server computing device (control module) and the database; the question-answer index located on server or database; ¶0075 disclosing the interface; ¶0125 disclosing a language model; ¶0147 disclosing a processing unit (also control module); ¶0175 a device control application program module (also control module)
executing, by the control module, a first phase being iterative and comprising, at each iteration, steps of selecting, by the control module, from the set of questions stored in the database, a question having an associated difficulty level, a set of expected keywords and at least one stored model answer, said question that is selected being different from questions selected at previous iterations, if any, (Wu ¶0069 disclosing preparing technical questions; the questions may be from a chat domain or technical domain; ¶0071 disclosing the bot is capable of selecting the next technical question (thus, iterative) based on the relevance score of the last answer; ¶0069 further discloses that interview chat bot may utilize the determined relevance score and/or emotion state to determine whether the next provided response from the chat bot should be in the chat domain or the technical question domain; ¶0074 disclosing the question-answer index may be located on server or database; ¶0069 also discloses that there is a level of difficulty associated with the questions; the difficulty of subsequent questions may change based on a relevance score of an answer given; ¶0133 and Fig, 10 disclosing the process of generating the technical question including parsing sentences from the technical knowledge with a trained sentence parser, e.g., "In computer science, cycle detection or cycle finding is the algorithmic problem of finding a cycle in a sequence of iterated function values," (keywords); ¶0134 disclosing a knowledge graph is extracted from the parsed sentence; once the sentence is parsed, rules may be applied to the parsed sentence: ¶0135-0139 discloses as phrase level entities, and single nouns as word level entities, such as ‘computer science’, ‘cycle detection’, ‘cycle finding’, ‘problem’, ‘cycle’, ‘sequence’, ‘function values’; prepositions, verbs, etc.; ¶0140 disclosing the system generates technical question/answer pairs from the knowledge graph; ¶0143 disclosing example questions above in ¶0141-¶0142, the entity “cycle detection”, is taken as the keyword and web pages which return textbook/paper contents are retrieved by the question and answer generation system which contain the following sentences, “Several algorithms for finding cycles quickly and with little memory are known. Floyd's tortoise and the hare algorithm move two pointers at different speeds through the sequence of values until they both point to equal values.”; ¶0144 disclosing the question and answer generation system 124 matches entity “cycles” and action “finding” in FIG. 12 with FIG. 11; ¶0125 disclosing the communication skill classifier may utilize an n-gram language model trained using reference answers of prepared questions by the classifier; the classifier may calculate an averaged language model score of the reference answers)
Note: Additionally, in regard to claim 1, the examiner further notes the recited "if any" does not move to distinguish the claimed invention from the cited art. These phrases are conditional/contingent limitations with the noted "said question that is selected being different from questions selected at previous iterations" step not necessarily performed. The broadest reasonable interpretation of a method (or process) claim having contingent limitations requires only those steps that must be performed and does not include steps that are not required to be performed because the condition(s) precedent are not met. Language that suggests or makes optional but does not require steps to be performed or does not limit a claim to a particular structure does not limit the scope of a claim or claim limitation. [See Ex parte Schulhauser, Appeal 2013-007847 (PTAB April 28, 2016) for an analysis of contingent claim limitations in the context of both method claims and system claims.; MPEP §2111.04 II].
retrieving, from the database, the question that is selected with the associated difficulty level, the set of expected keywords and the at least one stored model answer thereof, (Wu ¶0074 disclosing the question-answer index may be located on server or database; ¶0113 the question selection system selects the next technical question from the collection of technical question-answer pairs based on a determined level of difficulty (see above previous citations indicating the questions have associated difficulty level, keywords, and model answer)
determining, by the control module, the associated difficulty level based on at least one previously received accuracy score and automatically selecting a subsequent question having a difficulty level corresponding to the difficulty level that is determined, (Wu ¶0069 disclosing the chat bot is capable of changing the level of difficulty provided in the next technical question based on a relevance score for one or more previous answers; see also ¶0113 disclosing the question selection system determines a level of difficulty for the next question based on the relevance score; ¶0156 relevance score may be a negative score for incorrect answers, a normal score for correct answers, a good score for correct answer)
sending the question that is selected to said user interface, (Wu ¶0075 disclosing displaying the question to the user interface)
receiving a set of words corresponding to an answer from a candidate to the question that is selected and sent to said user interface, (Wu ¶0075 disclosing the user inputting their answer to the question, and the user or candidate may provide his answer as text, video, audio, and/or any other known method for gathering user input; see also Fig. 3)
processing, by the language model module, the set of words corresponding to the answer from the candidate by comparing the set of words with the set of expected keywords associated with the question that is selected, (Wu ¶0133 and Fig, 10 disclosing the process of generating the technical question including parsing sentences from the technical knowledge with a trained sentence parser, e.g., "In computer science, cycle detection or cycle finding is the algorithmic problem of finding a cycle in a sequence of iterated function values," (keywords); ¶0134 disclosing a knowledge graph is extracted from the parsed sentence; once the sentence is parsed, rules may be applied to the parsed sentence: ¶0135-0139 discloses as phrase level entities, and single nouns as word level entities, such as ‘computer science’, ‘cycle detection’, ‘cycle finding’, ‘problem’, ‘cycle’, ‘sequence’, ‘function values’; prepositions, verbs, etc.; ¶0140 disclosing the system generates technical question/answer pairs from the knowledge graph; ¶0143 disclosing example questions above in ¶0141-¶0142, the entity “cycle detection”, is taken as the keyword and web pages which return textbook/paper contents are retrieved by the question and answer generation system which contain the following sentences, “Several algorithms for finding cycles quickly and with little memory are known. Floyd's tortoise and the hare algorithm move two pointers at different speeds through the sequence of values until they both point to equal values.”; ¶0144 disclosing the question and answer generation system 124 matches entity “cycles” and action “finding” in FIG. 12 with FIG. 11; ¶0085 disclosing the answer evaluation system may utilize any known system for evaluating a candidate's answers; answer evaluation system evaluate the soundness and completeness of the candidate answer based on the answers similarity to a reference answer; ¶0121 disclosing the bot prepares a basic collection of <question, reference answers> for teamwork interview questions; the system computes the similarity between the reference good answers and the candidate's answers and determines a relevance score for the provided teamwork questions; ¶0124 disclosing classifier analyzes any received input signals form the candidate, such as text (words); ¶0125 disclosing a language model; the communication skill classifier may utilize an n-gram language model trained using reference answers of prepared questions by the classifier; ¶0060 disclosing hat bot may utilize sophisticated natural language processing systems or scan for keywords from a user input and then pull a reply with the most matching keywords or the most similar wording pattern from a database; see also ¶0079 disclosing the language understanding system; ¶0145 if the question and answer generation system determines questions and corresponding answers for the technical space of computer science, the answer can be separated into two parts: 1) the text answer (comment) part in natural language sentences; and 2) the code part written in some programming languages (such as Java, c, and so on); FIGS. 9A and 9B illustrate examples of question-answer pairs that have a text answer and coding answer from existing coding question-answer websites; ¶0154 disclosing the text answer is compared to the referenced answer utilizing a deep semantic similarity model and a recurrent neural network with gated recurrent units)
analyzing a semantic relevance of the set of words relative to the at least one stored model answer associated with the question that is selected, (Wu ¶0154 the text answer is compared to the referenced answer utilizing a deep semantic similarity model)
comparing the set of words with stored answers corresponding to the question that is selected in the database, (Wu ¶0133 and Fig, 10 disclosing the process of generating the technical question including parsing sentences from the technical knowledge with a trained sentence parser, e.g., "In computer science, cycle detection or cycle finding is the algorithmic problem of finding a cycle in a sequence of iterated function values," (keywords); ¶0134 disclosing a knowledge graph is extracted from the parsed sentence; once the sentence is parsed, rules may be applied to the parsed sentence: ¶0135-0139 discloses as phrase level entities, and single nouns as word level entities, such as ‘computer science’, ‘cycle detection’, ‘cycle finding’, ‘problem’, ‘cycle’, ‘sequence’, ‘function values’; prepositions, verbs, etc.; ¶0140 disclosing the system generates technical question/answer pairs from the knowledge graph; ¶0143 disclosing example questions above in ¶0141-¶0142, the entity “cycle detection”, is taken as the keyword and web pages which return textbook/paper contents are retrieved by the question and answer generation system which contain the following sentences, “Several algorithms for finding cycles quickly and with little memory are known. Floyd's tortoise and the hare algorithm move two pointers at different speeds through the sequence of values until they both point to equal values.”; ¶0144 disclosing the question and answer generation system 124 matches entity “cycles” and action “finding” in FIG. 12 with FIG. 11; ¶0085 disclosing the answer evaluation system may utilize any known system for evaluating a candidate's answers; answer evaluation system evaluate the soundness and completeness of the candidate answer based on the answers similarity to a reference answer; ¶0121 disclosing the bot prepares a basic collection of <question, reference answers> for teamwork interview questions; the system computes the similarity between the reference good answers and the candidate's answers and determines a relevance score for the provided teamwork questions; ¶0124 disclosing classifier analyzes any received input signals form the candidate, such as text (words); ¶0145 if the question and answer generation system determines questions and corresponding answers for the technical space of computer science, the answer can be separated into two parts: 1) the text answer (comment) part in natural language sentences; and 2) the code part written in some programming languages (such as Java, c, and so on); FIGS. 9A and 9B illustrate examples of question-answer pairs that have a text answer and coding answer from existing coding question-answer websites; ¶0154 disclosing the text answer is compared to the referenced answer utilizing a deep semantic similarity model and a recurrent neural network with gated recurrent units)
generating, by the language model module, an accuracy score based on said processing, (Wu ¶0084 disclosing the answer evaluation system analyzes or evaluates the user's answer to determine a relevance score for the answer; ¶0091; ¶0121 disclosing comparing answers to reference answers to determine a relevance score)
storing, by the control module, the accuracy score that is generated in the database in association with the question that is selected, (Wu ¶0121 chat bot computes the similarity between the reference good answers and the candidate's answers and determines a relevance score for the provided teamwork questions; these features are analyzed and put into a teamwork score by the summary system; ¶0175 a number of program modules and data files may be stored in the system memory; the program modules 506 (e.g., LU system 110, answer evaluation system 112, question selection system 114, summary system 120, sentiment system 122, and/or the question and answer generation system 124))
updating, by the control module, the difficulty level based on the accuracy score that is stored, (Wu ¶0003 disclosing the system changing the level of difficulty in the nest question based on the relevance score; see also ¶0069)
Wu in view of Thibodeaux discloses:
selecting a subsequent question from the set of questions stored in the database based on the accuracy score that is stored and associated difficulty levels, wherein the first phase is carried out until a predeterminate condition has been reached,
Wu discloses selecting a subsequent question from the set of questions stored in the database based on the accuracy score that is stored and associated difficulty levels: (Wu ¶0071 selecting the next question based on the candidate's relevance score of the last answer, and adjusting the level of difficulty of the next technical questions based on the candidate's relevance score), but does not appear to explicitly disclose that the first phase is carried out until a predeterminate condition has been reached. Thibodeaux suggests or discloses this limitation/concept: (Thibodeaux ¶0024 disclosing that each level (with each level having questions) will have a cut score that determines if the individual can move on to the next level; a candidate may have to achieve a score of at least 20 in level 1 to move on to level 2; when the candidate has reached their maximum level that is their score; ¶0030 candidate works in level 1 (204), completes level 1 (206), and the ACA algorithm scores the candidate's performance in level 1 (208). The system presents a score report (210) and, if the candidate's score for level 1 is above a predefined cut score (212), the candidate begins work in level 2 (214). The candidate completes level 2 (216), the ACA algorithm scores the candidate's performance in level 2 (218), and the system again presents a score report (220); his time, the score for level 2 did not meet the level 2 cut score (222), and the candidate is not moving on to level 3. The system displays the final score (224), and the competency evaluation ends (predetermined condition met)). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Wu to include that the first phase is carried out until a predeterminate condition has been reached as taught by Thibodeaux. One of ordinary skill in the art before the effective filing date of the claimed invention would have been motivated to modify Wu in order to develop a detailed analysis of the knowledge and skills an individual needs in order to carry out the most important aspects of any given job role (see ¶0017 of Thibodeaux).
wherein the predetermined condition determining that the accuracy score that is stored satisfies a predefined threshold condition corresponding to a target accuracy level, thereby terminating the first phase,
Wu discloses determining an accuracy score (called a relevance score that may be stored in/with the summary), but does not appear to explicitly disclose that the predetermined condition determining that the accuracy score that is stored satisfies a predefined threshold condition corresponding to a target accuracy level, thereby terminating the first phase. Thibodeaux suggests or discloses this limitation/concept: (Thibodeaux ¶0024 disclosing that each level (with each level having questions) will have a cut score that determines if the individual can move on to the next level; a candidate may have to achieve a score of at least 20 in level 1 to move on to level 2; when the candidate has reached their maximum level that is their score; ¶0030 candidate works in level 1 (204), completes level 1 (206), and the ACA algorithm scores the candidate's performance in level 1 (208). The system presents a score report (210) and, if the candidate's score for level 1 is above a predefined cut score (212), the candidate begins work in level 2 (214). The candidate completes level 2 (216), the ACA algorithm scores the candidate's performance in level 2 (218), and the system again presents a score report (220); his time, the score for level 2 did not meet the level 2 cut score (222), and the candidate is not moving on to level 3. The system displays the final score (224), and the competency evaluation ends). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Wu to include that the predetermined condition determining that the accuracy score that is stored satisfies a predefined threshold condition corresponding to a target accuracy level, thereby terminating the first phase as taught by Thibodeaux. One of ordinary skill in the art before the effective filing date of the claimed invention would have been motivated to modify Wu in order to develop a detailed analysis of the knowledge and skills an individual needs in order to carry out the most important aspects of any given job role (see ¶0017 of Thibodeaux).
Wu, as modified above, discloses the following:
executing, by the control module, a second phase comprising computing, by the control module, an evaluation score of the candidate using the accuracy score from all of each question of the set of questions. (Wu ¶0167 disclosing the determination on whether the interview is complete is made; and ¶0138 disclosing the candidate’s evaluation based on the overall interview including scores from various skills such as communication, interpersonal, technical competency, team collaboration; see also ¶0119-¶0120)
Claims 12 and 13 are directed to a non-transitory computer program and system, respectively. Claims 12 and 13 recite limitations that are parallel in nature as those addressed above for claim 1, which is directed towards a method. Claims 12 and 13 are therefore rejected for the same reasons as set forth above for claim 1. Furthermore, claim 12 recites:
(Claim 12): A non-transitory computer program comprising instructions which, when the non- transitory computer program is executed by a computer, cause the computer to carry out a method for automatically evaluating a candidate using a set of questions, said method being executed by an automatic evaluation system, said automatic evaluation system comprising a control module, a database storing said set of questions, a user interface and a language model module, the method comprising: (Wu ¶0014 disclosing a system for automated interviewing of candidates; the system includes at least one processor and a memory; ¶0074 disclosing the server computing device (control module) and the database; the question-answer index located on server or database; ¶0075 disclosing the interface; ¶0125 disclosing a language model; ¶0147 disclosing a processing unit (also control module); ¶0175 a device control application program module (also control module); ¶0179-¶0180 disclosing communication media may be embodied by computer readable instructions, data structures, program module; computer readable media or storage media as used herein may include computer storage media; computer readable instructions)
Claim 3: The method according to claim 1, said method further comprising a step of generating the accuracy score by the language model module, said step comprising analyzing a structure and a relevance of the set of words that is received. (Wu ¶0092 a recurrent neural network (RNN) with gated recurrent units (GRUs) to learn the similarity among a question and good/bad answers; the embedding layer maps these input one-hot expressions into dense vector representations. Then the hidden layer will further make use of GRU to compute the sequence level representations for the question and two answers; the output layer will compute the margin between the similarity of <question, answer+> and <question, answer−>; the benefit of this network is that a sparse space of variant sentences can be projected into some dense spaces and then some vector-based computing can be performed to simply compute the “similarity” among questions as well as answers; ¶0094 disclosing the answer evaluation system compares the time and space complexity (structure) of the answer to a time and space complexity of the reference answer; a time and space complexity for the referenced answer is provided with the reference answer from the collection of technical question-answer pairs in the question-answer index; ¶0095 a relevance score for the answer is determined based on the comparison of the time and space complexity for the answer to the time and space complexity for the reference answer; ¶0156 disclosing a relevance score for the answer is determined; the relevance score may indicate whether the answer provided by the candidate was excellent, good, normal, or negative; an answer provided by the candidate is normal, good, and/or excellent if the relevance score meets one or more predetermined thresholds, e.g., the relevance score may be a negative score for incorrect answers, a normal score for correct answers with the worst time and space costs, a good score for correct answer with time and space costs that are better than the worst time space costs but that are not the best possible time and space costs, and is an excellent score for correct answer with best time and space costs)
Claim 4: The method according to claim 1, said method further comprising before the first phase, a preliminary step of building at least one model using large data of prior interviews responses and scores or data, said at least one model allowing to predict a quality of an answer based on features comprising one or more of correctness, completeness, clarity, during the first phase, a step of generating the accuracy score by the language model module by using said at least one model that is built. (Wu ¶0085 disclosing the answer evaluation system evaluate the soundness and completeness of the candidate answer based on the answers similarity to a reference answer; ¶0125 disclosing communication skill classifier may utilize an n-gram language model trained using reference answers of prepared questions by the classifier; the classifier utilizes a simple heuristic rule to determine when a candidate's answer is good or bad, such as when a candidate answer's language model score is higher than an average answer is considered good and when a candidate answer's language model score is lower than the average the answer is considered bad; ¶0126 disclosing classifier analyzes the text from the candidate for duplicate words utilizing a duplication system; the duplication system determines and catalogs duplicated words, duplicated phrases, and modal words as features; e.g., a simple heuristic rule for these features is that: when the number of duplicated words/phrases/modal words occurs more than 10% of the answer, then it is a bad answer; otherwise, a good answer; ¶0132 disclosing the question-answer index collects updates of new question-answer pairs from a question and answer generation system; the chat bot updates previously stored question-answer pairs or adds new question-answer pairs in the question-answer index based on the information received from the question and answer generation system; ¶0156 disclosing a relevance score for the answer is determined; the relevance score may indicate whether the answer provided by the candidate was excellent, good, normal, or negative; an answer provided by the candidate is normal, good, and/or excellent if the relevance score meets one or more predetermined thresholds, e.g., the relevance score may be a negative score for incorrect answers, a normal score for correct answers with the worst time and space costs, a good score for correct answer with time and space costs that are better than the worst time space costs but that are not the best possible time and space costs, and is an excellent score for correct answer with best time and space costs)
Claim 6: The method according to claim 1, said method further comprising, after the second phase, a step of evaluating, by the control module, the candidate based on said evaluation score that is computed, or a step of evaluating the candidate, based on said evaluation score that is computed, by an interviewer or any human user. (Wu ¶0118 disclosing after the chat bot has finished interviewing a candidate, the summary system of the chat bot evaluates and summarizes the candidate's results from the interview; the chat bot provides these results or makes these results available to the interviewer; ¶0119 disclosing the summary system evaluates the candidate after completion of an interview to determine a communication skills score, an interpersonal skills score, a technical competency score, and/or a team collaboration score for the candidate; ¶0120 technical competency can be assessed by analyzing the relevance scores for technical questions asked at different levels of difficulty during the chat bot interview; ¶0170 disclosing the results of the evaluation, including any generated scores and/or graphics are made available to the interviewer)
Claim 7: The method according to claim 1, said method further comprising a step of submitting the question that is selected to the candidate via the user interface. (Wu ¶0075 disclosing the client computing device provides any response from the chat bot utilizing any known visual, audio, tactile, and/or other sensory mechanisms; in this example, the user interface of the client computing device displays the startup predetermined reply and the question 108B from the chat bot as text; see also ¶0077)
Claim 8: The method according to claim 1, said method further comprising, prior to submitting the question that is selected to the user interface, a step of converting words of the question that is selected into an audio stream by a text conversion module, providing said audio stream to the user interface and diffusing said audio stream that is received by the user interface to the candidate. (Wu ¶0071 disclosing interview chat bot on a client computing device 104B of an interviewer for performing an automated interview of a technical candidate; see also ¶0074; ¶0075 discloses the client computing device 104 provides any response from the chat bot 100 utilizing any known visual, audio, tactile, and/or other sensory mechanisms; ¶0080 discloses the LU system may need to convert a generated response from text to voice to provide a voice response to the candidate)
Claim 9: The method according to claim 1, said method further comprising, when the answer has been given orally by the candidate, a step of recording said answer that has been given orally as an audio stream and converting said audio stream into a set of words by an audio conversion module. (Wu ¶0079 disclosing the LU system converts the user's answers into text and/or annotated text; the LU system includes application programming interfaces (APIs) for text understanding, speech recognition, and/or image/video recognition for processing user answers into text and/or annotated text form; ¶0080 disclosing sounds need to be recognized and decoded as texts; a speech recognition API may be necessary for the speech-to-text conversion task and is part of the LU system; ¶0124 disclosing any received voice input is converted into text utilizing a language understanding system; ¶0189 discloses the system includes a video interface to record video stream; and has a microphone to receive audible input; ¶0109 disclosing receiving recorded voice of the applicant)
Claim 10: The method according to claim 1, said method further comprising,
Wu discloses that the questions have varying levels of difficulty, but does not explicitly disclose before the first phase, a preliminary step of attributing a difficulty level to said each question of the set of questions. Thibodeaux suggests or discloses this limitation/concept:
before the first phase, a preliminary step of attributing a difficulty level to said each question of the set of questions. (Thibodeaux ¶0017 disclosing the model starting with SMEs developing an analysis of the knowledge and skills for the job and building questions that can be organized within a question bank, with rankings for each question across multiple jobs and multiple competency categories; e.g., a set of questions may be specific to a certain job or competency, and within that set of questions the questions may be organized according to difficulty; within the set of questions, the questions may be further organized into respective tiers based on the associated difficulty; e.g., a portion of the set of questions can be categorized as level “1” questions, another portion as level “2” questions, and so on, where each respective increase in level signifies an increase in difficulty in answering the questions; in some cases, a single question can be ranked as difficult for a first job or job type while easy for a second job type; ¶0019 disclosing using the collected data from the SMEs, the system generating a path for answering questions, and ¶0020 discloses the candidate starting the interview process and answering questions)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Wu to include before the first phase, a preliminary step of attributing a difficulty level to said each question of the set of questions as taught by Thibodeaux. One of ordinary skill in the art before the effective filing date of the claimed invention would have been motivated to modify Wu in order to developing a detailed analysis of the knowledge and skills an individual needs in order to carry out the most important aspects of any given job role (see ¶0017 of Thibodeaux).
Claim 11: The method according to claim 10, wherein the selecting the question comprises determining a question difficulty level based on the accuracy of at least one of previous answers given by the candidate and selecting a question having said question difficulty level that is determined. (Wu ¶0069 disclosing the chat bot is capable of changing the level of difficulty provided in the next technical question based on a relevance score for one or more previous answers; see also ¶0113 disclosing the question selection system determines a level of difficulty for the next question based on the relevance score; ¶0156 relevance score may be a negative score for incorrect answers, a normal score for correct answers, a good score for correct answer)
Additional Relevant Prior Art References
Additional prior art references that are relevant to the applicant’s invention, but is not used in a prior art rejection includes:
Danson (2015/0046357): discloses a system that identifying keywords associated with one or more categories relevant to a job position, receiving the candidate's answers to interview questions, reviewing the answers for the identified keywords, and outputting a score for the one or more categories based on a density of the keywords in the answers.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DIONE N SIMPSON whose telephone number is (571)272-5513. The examiner can normally be reached M-F; 7:30 a.m.-4:30 p.m..
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Sarah Monfeldt can be reached at (571) 270-1833. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
DIONE N. SIMPSON
Primary Examiner
Art Unit 3628
/DIONE N. SIMPSON/Primary Examiner, Art Unit 3629
Prosecution Insights