DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Acknowledgements
Applicant elected claims 1-7 in response to election/restriction filed on 5/5/2026.
Applicant filed information disclosure statement.
Claims 1-7 are pending.
Claim Objections
Claim 7 is objected to because of the following informalities: Claim 7 recites compute devices when it should recite computer devices. Appropriate correction is required.
Allowable Subject Matter
Claims 2 is allowable if rewritten to include all of the limitations of the base claim and any intervening claims, and if the independent claims were amended in such a way as to overcome the rejection(s) under 35 U.S.C. 101, set forth in this Office action. The closest prior art to these claims include Kenthapadi (US20200372472A1) in further view of Mather (US20160055457A1) in further view of Baughman (20160307111) who teaches true positive and false positive rates with respect to bias determination. However, with respect to exemplary claim 2, the closest prior art of record, either alone or taken in combination with any other references of record, do not anticipate or render obvious the claimed functionality of claim 2.
Claims 5 is allowable if rewritten to include all of the limitations of the base claim and any intervening claims, and if the independent claims were amended in such a way as to overcome the rejection(s) under 35 U.S.C. 101, set forth in this Office action. The closest prior art to these claims include Kenthapadi (US20200372472A1) in further view of Mather (US20160055457A1) in further view of Skelton (US8682837B2) who teaches a bias formula. However, with respect to exemplary claim 5, the closest prior art of record, either alone or taken in combination with any other references of record, do not anticipate or render obvious the claimed functionality of claim 5.
Claims 6 is allowable if rewritten to include all of the limitations of the base claim and any intervening claims, and if the independent claims were amended in such a way as to overcome the rejection(s) under 35 U.S.C. 101, set forth in this Office action. The closest prior art to these claims include Kenthapadi (US20200372472A1) in further view of Mather (US20160055457A1) in further view of Liu (US8818910B1) who teaches bagging which corresponds to bootstrap aggregating. However, with respect to exemplary claim 6, the closest prior art of record, either alone or taken in combination with any other references of record, do not anticipate or render obvious the claimed functionality of claim 6.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-7 are rejected under 35 U.S.C. 101 because the claimed invention is directed to a judicial exception (i.e., a law of nature, a natural phenomenon, or an abstract idea) without significantly more than the judicial exception itself.
Regarding Step 1 of subject matter eligibility for whether the claims fall within a statutory category (See MPEP 2106.03), claims 1-7 are directed to a method.
Regarding step 2A-1, Claims 1-7 recite a Judicial Exception. Exemplary independent claim 1 recites the limitations of
Executing…model to identify a first subset of candidate profiles from a set of candidate profiles that satisfy a fit metric to a job description and a second subset of candidate profiles from the set of candidate profiles that does not satisfy the fit metric to the job description; calculating a bias metric based on the first subset of candidate profiles and the second subset of candidate profiles…causing, to generate an updated …model, …model to be retrained based on the first subset of candidate profiles and the second subset of candidate profiles after calculating the bias metric ;updating the job description in response to the bias metric being outside a predetermined acceptable range to generate an updated job description; receiving, based on the updated …model and after generating the updated job description, an updated set of candidate profiles different than the set of candidate profiles; and calculating, after causing the …model to be retrained, an updated bias metric for the updated set of candidate profiles, the updated bias metric being within the predetermined acceptable range.
These limitations, as drafted, are a process that, under its broadest reasonable interpretation cover concepts of executing, generating, calculating, retraining, and updating data. The claim limitations fall under the abstract idea grouping of mental process, because the limitations can be performed in the human mind, or by a human using a pen and paper. For example, but for the language of machine learning models, the claim language encompasses simply executing a model, calculating metrics, generating a model based on training, updating data, and receiving data. These are mere data manipulation steps that do not require a computer. For example, a user is able to execute a model then train the model by picking the best variables for the model without a computer. In addition, a user is also able to calculate metrics and update data without the use of a computer.
The claims also recite job candidates, job descriptions, and fit metrics which correspond to matching job candidates with jobs (See para 0003 of Applicant’s Specification). These make the claims fall in the abstract idea grouping of certain methods of organizing human activity (fundamental economic principles or practices; business relations, interactions between people). It is clear the limitations recite these abstract idea groupings, but for the recitations of generic computer components. The mere nominal recitations of generic computer components does not take the limitations out of the mental process and certain methods of organizing human activity grouping. The claims are focused on the combination of these abstract idea processes.
Regarding step 2A-2- This judicial exception is not integrated into a practical application, and the claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception.
The claim recites the additional elements of machine learning models and remote computing devices.
These components are recited at a high level of generality, and merely automate the steps. Each of the additional limitations is no more than mere instructions to apply the exception using a generic computer component.
The combination of these additional elements is no more than mere instructions to apply the exception using a generic computer components or software. Accordingly, even in combination, these additional elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea.
Further, the claims do not provide for recite any improvements to the functioning of a computer, or to any other technology or technical field; applying or using a judicial exception to effect a particular treatment or prophylaxis for a disease or medical condition; applying the judicial exception with, or by use of, a particular machine; effecting a transformation or reduction of a particular article to a different state or thing; or applying or using the judicial exception in some other meaningful way beyond generally linking the use of the judicial exception to a particular technological environment, such that the claim as a whole is more than a drafting effort designed to monopolize the exception.
The dependent claims have the same deficiencies as their parent claims as being directed towards an abstract idea, as the dependent claims merely narrow the scope of their parent claims. For example, the dependent claims further describe how biases are calculated such as providing a formula. In addition, the dependent claims further recite additional variables for the candidate profiles such as true positive and false positive value.
Regarding step 2B the claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception because claim 1 recites
Method, however method is not considered an additional limitation.
Claim 1 further recites machine learning models
Claim 7 recites remote computer devices.
When looking at these additional elements individually, the additional elements are purely functional and generic the Applicant specification states a general purpose computer in para 0018.
When looking at the additional elements in combination, the Applicant’s specification merely states a general purpose computer as seen in para 0018. The computer components add nothing that is not already present when the steps are considered separately. See MPEP 2106.05
Looking at these limitations as an ordered combination and individually adds nothing additional that is sufficient to amount to significantly more than the recited abstract idea because they simply provide instructions to use generic computer components, recitations of generic computer structure to perform generic computer functions that are used to "apply" the recited abstract idea. Thus, the elements of the claims, considered both individually and as an ordered combination, are not sufficient to ensure that the claim as a whole amounts to significantly more than the abstract idea itself.
Since there are no limitations in these claims that transform the exception into a patent eligible application such that these claims amount to significantly more than the exception itself, claims 1-7 are rejected under 35 U.S.C. 101.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 3, 4, and 7 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kenthapadi (US20200372472A1) in further view of Mather (US20160055457A1).
Regarding claim 1, Kenthapadi teaches
A method (See para 0023-The disclosed embodiments provide a method, apparatus, and system for detecting and/or managing bias in machine learning models.) This teaches a method.
executing a machine learning (ML) model to identify a first subset of candidate profiles from a set of candidate profiles that satisfy a fit metric to a job description and a second subset of candidate profiles from the set of candidate profiles that does not satisfy the fit metric to the job description (See para 0041-After qualified candidates 132 are identified, profile and/or activity data of qualified candidates 132 may be inputted into machine learning models 114, along with features and/or characteristics of the corresponding opportunity (e.g., required or desired skills, education, previous positions, current position, years of experience, industry, title, location, etc.). Machine learning models 114 may then output scores representing the strengths of qualified candidates 132 with respect to the opportunity and/or qualifications related to the opportunity. For example, machine learning models 114 may generate scores based on similarities between the candidates' profile data and descriptions of the opportunities) This shows a machine learning model identifies a first subset of qualified metrics that satisfies a fit metric (i.e. similarities between profiles and opportunities) such as a candidates with high scores as seen here (See para 0047-As a result, a candidate that is higher in the ranking may represent a stronger candidate than a candidate that is lower in the ranking.). The machine learning model may also generate second subset such as candidates that do not satisfy the fit metric such as lower ranked candidates as seen in para 0047.
calculating a bias metric based on the first subset of candidate profiles and the second subset of candidate profiles (See para 0049-Next, monitoring system 112 uses distributions 120 to calculate a set of metrics 122 for detecting and/or quantifying bias in machine learning models 114. In particular, monitoring system 112 may calculate metrics 122 from a first distribution of an attribute in a ranking (e.g., rankings 116) of recommended candidates from a machine learning model (e.g., machine learning models 114) and a second distribution of the attribute in a corresponding set of qualified candidates 132.) This shows a bias metric 122 is calculated and is based on the rankings (i.e. first/second subset of candidates).
the bias metric being a measure of equalized odds of selection of the first subset of candidate profiles across at least one protected class from a plurality of protected classes; (See para 0048-To detect and/or quantify bias in machine learning models 114, monitoring system 112 calculates distributions 120 of attributes 118 for qualified candidates 132 and rankings 116. Each distribution may include probabilities or proportions of different values of an attribute in the corresponding set of users. For example, a distribution of gender in a “top 100” ranking of candidates for a job may be represented as 0.4 male, 0.3 female, and 0.3 unknown. In another example, a distribution of age in all qualified candidates 132 for a job may be represented as 0.05 for an age range of 18-25, 0.35 for an age range of 26-35, 0.25 for an age range of 36-45, 0.2 for an age range of 46-55, 0.10 for an age range of 56-65, and 0.05 for an age range of 66 and over) This teaches odds of selection in a subset of candidates across a protected class such as gender and age.
causing, to generate an updated machine learning model, the machine learning model to be retrained based on the first subset of candidate profiles and the second subset of candidate profiles after calculating the bias metric (See para 0125-In turn, model-training apparatus 246 uses result 228 and/or response 230 to update machine learning models 208-210, features used by machine learning models 208-210, and/or other components used to generate scores 216-218, rankings 220-222, and/or rerankings 224-226.) This teaches updating machine learning models. Updating machine learning models corresponds to retraining the models based on results. Results include a bias metric as seen here (See para 0125- Model-training apparatus 246 may then use outcomes 232 and features and scores 216-218 used to produce the corresponding results as training data for updating parameters 230 of one or both machine learning models 208.)
Even though Kenthapadi teaches bias metric exceeding a threshold (See para 0061), it is not clear that it teaches updating a job description, however Mather teaches
updating the job description in response to the bias metric being outside a predetermined acceptable range to generate an updated job description…the updated bias metric being within the predetermined acceptable range. (See para 0090- For example, if a job description received a grade of “C” on January 1 and was modified on January 2 and then received a grade of “B”) (See para 0049- The score generated by Bias Scoring Logic 235 can optionally displayed in real-time and/or be displayed based on a previous calculation. Bias Scoring Logic 235 is optionally configured to calculate a grade based on a score. A grade is a representation of a score normalized to a grading scale such as A to F, 1 to 10, one star to five stars, “Very Good” to “Very Bad,” etc.) This teaches that job descriptions are updated in response to a bias metric being outside an acceptable range. For example, a user can make a job description and receives a grade of F which is outside an acceptable range. The user would then be able to modify the description to get a grade of A. This also teaches the bias metric is within an acceptable range such as having an A. Kenthapadi and Mather both teach bias metric.
Kenthapadi and Mather are analogous art because they are from the same problem solving area of job management and bias metrics and both belong to G06Q10 classification. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined Kenthapadi invention by incorporating the method of Mather because Kenthapadi would also be able to modify the job description when the user sends a request. This would allow the user to have more control over the types of candidates they want. For example, a user would be able to add skills and other traits in a job request if they are not satisfied with the initial pool of candidates.
Kenthapadi further teaches receiving, based on the updated machine learning model…an updated set of candidate profiles different than the set of candidate profiles (See para 0125- Model-training apparatus 246 may then use outcomes 232 and features and scores 216-218 used to produce the corresponding results as training data for updating parameters 230 of one or both machine learning models 208. Model-training apparatus 246 may also store updated parameter values and/or other data associated with machine learning models 208-210 in a model repository 244 for subsequent retrieval and use. In turn, one or both rankers 202-204 may obtain the latest parameter values for the corresponding machine learning models 208-210 from model repository 244 and use the parameter values to generate subsequent sets of scores 216-218, rankings 220-222, and/or rerankings 224-226.) This shows that the updated machine learning models make subsequent rankings and re-rankings. The subsequent rankings and re-rankings correspond to updated set of candidate profiles.
However Kenthapadi doesn’t teach updated job description, however Mather teaches after generating the updated job description (See para 0090- For example, if a job description received a grade of “C” on January 1 and was modified on January 2 and then received a grade of “B”) (See para 0049- The score generated by Bias Scoring Logic 235 can optionally displayed in real-time and/or be displayed based on a previous calculation. Bias Scoring Logic 235 is optionally configured to calculate a grade based on a score. A grade is a representation of a score normalized to a grading scale such as A to F, 1 to 10, one star to five stars, “Very Good” to “Very Bad,” etc.) This teaches that job descriptions are updated in response to a bias being outside an acceptable range.
Kenthapadi and Mather are analogous art because they are from the same problem solving area of job management and both belong to G06Q10 classification. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined Kenthapadi invention by incorporating the method of Mather because Kenthapadi would also be able to modify the job description when the user sends a request. This would allow the user to have more control over the types of candidates they want. For example, a user would be able to add skills and other traits they are looking for if they are not satisfied with the initial pool of candidates for a job request.
Kenthapadi further teaches and calculating, after causing the ML model to be retrained, an updated bias metric for the updated set of candidate profiles (See para 0125- Model-training apparatus 246 may then use outcomes 232 and features and scores 216-218 used to produce the corresponding results as training data for updating parameters 230 of one or both machine learning models 208. Model-training apparatus 246 may also store updated parameter values and/or other data associated with machine learning models 208-210 in a model repository 244 for subsequent retrieval and use. In turn, one or both rankers 202-204 may obtain the latest parameter values for the corresponding machine learning models 208-210 from model repository 244 and use the parameter values to generate subsequent sets of scores 216-218, rankings 220-222, and/or rerankings 224-226.) (See para 0122-For example, ranker 204 may use the same reranking techniques as ranker 202 and/or different reranking techniques from ranker 202 to produce reranking 226 from ranking 222. As with reranking 224, reranking 226 may be performed to reduce bias in scores 218) This teaches that after an updated machine learning model is generated updated scores are also generated which include bias scores such as the distribution scores as seen in para 0048 as taught above.
In addition, Kenthapadi teaches updated bias metric, the art teaches that it can be fair as seen here the updated bias metric being within the predetermined acceptable range. (See para 0045- Conversely, a machine learning model may be unbiased or “fair” when the proportion of the attribute in one or more sets of top-ranked results (e.g., the top 5, 10, 25, 50 candidates from rankings 116) generated in response to a request (e.g., requests 130) substantially matches the proportion of the attribute in qualified candidates 132 that match the same request.) This shows a bias metric being fair which would mean the machine learning model is unbiased (i.e. acceptable range).
Regarding claim 3, Kenthapadi and Mather and Mather teach the limitations of claim 1, however Kenthapadi further teaches
further comprising: identifying, using the updated machine learning model, a subset of candidate profiles from the updated set of candidate profiles that match the updated job description. (See para 0125- Model-training apparatus 246 may then use outcomes 232 and features and scores 216-218 used to produce the corresponding results as training data for updating parameters 230 of one or both machine learning models 208. Model-training apparatus 246 may also store updated parameter values and/or other data associated with machine learning models 208-210 in a model repository 244 for subsequent retrieval and use. In turn, one or both rankers 202-204 may obtain the latest parameter values for the corresponding machine learning models 208-210 from model repository 244 and use the parameter values to generate subsequent sets of scores 216-218, rankings 220-222, and/or rerankings 224-226.) This shows that the updated machine learning model produces new scores, rankings, and re-rankings. New rankings and re-rankings corresponds to new set of candidates that match the job description. For example John might have initial ranking of 200 (i.e. not fit the job description) but the new ranking might be 2 (i.e. fits the job description).
Regarding claim 4, Kenthapadi and Mather and Mather teach the limitations of claim 1, however Kenthapadi further teaches
further comprising: identifying, using the updated machine learning model, a subset of candidate profiles from the updated set of candidate profiles that do not match the updated job description. (See para 0125- Model-training apparatus 246 may then use outcomes 232 and features and scores 216-218 used to produce the corresponding results as training data for updating parameters 230 of one or both machine learning models 208. Model-training apparatus 246 may also store updated parameter values and/or other data associated with machine learning models 208-210 in a model repository 244 for subsequent retrieval and use. In turn, one or both rankers 202-204 may obtain the latest parameter values for the corresponding machine learning models 208-210 from model repository 244 and use the parameter values to generate subsequent sets of scores 216-218, rankings 220-222, and/or rerankings 224-226.) This shows that the updated machine learning model produces new scores, rankings, and re-rankings. New rankings and re-rankings corresponds to new set of candidates that match the job description. For example John might have initial ranking of 2 (fits the job description) but the new ranking might be 200 (i.e. does not fit the job description).
Regarding claim 7, Kenthapadi and Mather and Mather teach the limitations of claim 1, however Kenthapadi further teaches
wherein the set of candidate profiles are received from a set of remote compute devices associated with the set of candidate profiles. (See para 0036- In one or more embodiments, data received from electronic devices 102-108 is aggregated into a data repository 134 for subsequent retrieval and use. For example, each profile update) (See para 0038- Data in data repository 134 may also, or instead, be used with one or more machine learning models 114 to produce rankings 116 of recommended candidates for opportunities) This shows remote devices 102-108. The data received from them include profile data on candidates.
Conclusion
The prior art made of record and not relied upon considered pertinent to Applicant’s disclosure.
Baughman (20160307111) who teaches true positive and false positive rates with respect to bias determination.
Skelton (US8682837B2) who teaches a bias formula.
Liu (US8818910B1) who teaches bagging which corresponds to bootstrap aggregating.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MUSTAFA IQBAL whose telephone number is (469)295-9241. The examiner can normally be reached Monday Thru Friday 9:30am-7:30 CST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Beth Boswell can be reached at (571) 272-6737. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MUSTAFA IQBAL/Primary Examiner, Art Unit 3625
Prosecution Insights