Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of claims
This final office action on merits is in response to the communication received on 04/09/2026. Claim 16 is cancelled. Amendments to claims 1 and 7 are acknowledged and have been carefully considered. Claims 1-15, and 17-20 are pending and considered below.
Drawings
The drawings were received on 04/09/2026. These drawings are acceptable.
Claim Rejections - 35 USC § 112
Applicant’s amendment to claim 7 is persuasive. The recitation of specific security features, including role-based access controls, encryption of PHI, local encryption using secret encryption keys, and two factor authentication, clarifies the scope of the claim. Accordingly, the rejection under 35 U.S.C. 112(b) is withdrawn.
Subject Matter Free of Art
Claims 1-8 include subject matter that is free of prior art. The cited prior art of record fails to expressly teach or suggest, either alone or in combination, the features found within independent claim 1. For claim 1, the cited prior art of record fails to expressly teach or suggest, either alone or in combination, collecting performance evaluations for a target healthcare professional and a matched peer group formed by clustering performance evaluation ratings across institutions, producing learning curves from the performance evaluations and performance parameters, accounting for evaluator rating bias and case complexity through estimation of evaluator bias parameters, performing MCMC statistical sampling to estimate posterior distributions of learning curve parameters, and determining competency scores based on comparisons between the target healthcare professional and the matched peer group using the generated learning curves.
The closest prior art of record includes 1) Wolf et al. (U.S. Patent Publication 2021/0307864 A1), referred to hereinafter as Wolf, and 2) Brown et al. (U.S. Patent Publication 2020/0411170 A1), referred to hereinafter as Brown.
Wolf teaches evaluating surgical performance using computer analysis, generating competency scores, assessing surgical skill, comparing practitioner performance, and employing machine learning and artificial intelligence techniques in connection with healthcare performance assessment. However, Wolf fails to teach or suggest collecting performance evaluations for a matched peer group formed by clustering performance evaluation ratings across institutions, generating learning curves from performance parameters for a target healthcare professional and the matched peer group, accounting for evaluator rating bias and case complexity through estimation of evaluator bias parameters, performing Markov Chain Monte Carlo statistical sampling, estimating posterior distributions of learning curve parameters, or determining competency using such bias-corrected learning curve models.
Brown teaches healthcare worker performance metrics, performance ratings, worker qualifications, task complexity considerations, machine learning techniques including clustering algorithms, and healthcare information obtained from multiple healthcare organizations. However, Brown fails to teach or suggest generating learning curves from healthcare worker performance parameters, forming a matched peer group by clustering performance evaluation ratings across institutions for competency assessment purposes, accounting for evaluator rating bias as a confounding variable through estimation of evaluator bias parameters, performing Markov Chain Monte Carlo statistical sampling, estimating posterior distributions of learning curve parameters, or determining competency scores based on comparisons of bias-corrected learning curves between a target healthcare professional and a matched peer group.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-15, and 17-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Step 1
Under step 1, the analysis is based on MPEP 2106.03, and claims 1-8 and 20 are drawn to a data management platform, and claims 9-15, and 17-19 are drawn to a method. Thus, each claim, on its face, is directed to one of the statutory categories (i.e., useful process, machine, manufacture, or composition of matter) of 35 U.S.C. §101.
Step 2A Prong One
Claim 1 recites the limitations of listing component tasks and required skills for each procedure; assessing task complexity; standardizing clinical schedule and performance evaluation records obtained from a plurality of disparate healthcare data source systems, including identifying and removing duplicate encounter records and mapping procedures and tasks to a medical procedure ontology; collecting performance evaluations for the target healthcare professional and a matched peer group of the target healthcare professional, wherein the matched peer group is formed by clustering performance evaluation ratings across institutions, for the performance of one or more selected procedures, each procedure having one or more tasks and an assigned clinical complexity value for the procedure and the one or more tasks thereof; compiling the evaluations versus predetermined standards for the successful completion of each task and one or more steps thereof to provide performance parameters; calculating a competency score for the target healthcare professional for the procedure and each task thereof; and comparing the learning curves and skill levels for the procedure and each task thereof for the target healthcare professional to that of the matched peer group of the target healthcare professional to determine a competency score for the target healthcare professional. These limitations, as drafted, are processes that, under their broadest reasonable interpretations, cover performance of the limitations in the mind or by using a pen and paper. Even when considering “a computer, a server or data storage system, a user interface, a non-transitory computer-readable medium storing computer program instructions, software for analyzing input data and providing an output, and a data array” language, the claim encompasses a user observing and reviewing clinical schedules and evaluation records, identifying duplicate records, categorizing procedures and tasks according to an ontology, assessing task complexity, collecting and compiling performance evaluations to predetermined standards, grouping individuals into peer groups based on performance characteristics, evaluating learning progress and skill levels, and determining competency scores through observation, evaluation, and judgement, and comparison performed in their mind or by using a pen and paper. The mere nominal recitation of a computer, a server or data storage system, a user interface, a non-transitory computer-readable medium storing computer program instructions, software for analyzing input data and providing an output, and a data array does not take the claim limitations out of the mental processes grouping. Thus, the claim recites a mental process which is an abstract idea.
Claim 1 recites the limitation of performing a computation to produce learning curves from the performance parameters for the target healthcare professional and the matched peer group of the target healthcare professional, wherein the computation is selected from the group consisting of statistical modeling, deep learning modeling, and machine learning modeling, and wherein the computation accounts for evaluator rating bias and case complexity as confounders by estimating evaluator bias parameters and performing a Markov Chain Monte Carlo statistical sampling method to estimate posterior distributions of learning curve parameters. This limitation, as drafted, is a process that, under its broadest reasonable interpretation, covers mathematical relationships, mathematical formulas or equations, and mathematical calculations. The mere nominal recitation of a computer, a server or data storage system, a user interface, a non-transitory computer-readable medium storing computer program instructions, software for analyzing input data and providing an output, and a data array does not take the claim limitation out of the mathematical concept grouping. Thus, the claim recites mathematical concepts which are abstract ideas.
The types of identified abstract ideas are considered together as a single abstract idea for analysis purposes.
Independent claims 9 and 20 recite identical or nearly identical steps with respect to claim 1 (and therefore also recite limitations that fall within this subject matter grouping of abstract ideas), and these claims are therefore determined to recite an abstract idea under the same analysis.
Under Step 2A Prong Two
The claimed limitations, as per claim 1, include:
a computer, a server or data storage system, a user interface, a non-transitory computer-readable medium storing computer program instructions, software for analyzing input data and providing an output, and a data array,
wherein the platform is configured to perform steps comprising:
acquiring clinical schedules indicating clinical procedures to be performed;
listing component tasks and required skills for each procedure;
assessing task complexity;
standardizing clinical schedule and performance evaluation records obtained from a plurality of disparate healthcare data source systems, including identifying and removing duplicate encounter records and mapping procedures and tasks to a medical procedure ontology;
collecting performance evaluations for the target healthcare professional and a matched peer group of the target healthcare professional, wherein the matched peer group is formed by clustering performance evaluation ratings across institutions, for the performance of one or more selected procedures, each procedure having one or more tasks and an assigned clinical complexity value for the procedure and the one or more tasks thereof;
compiling the evaluations versus predetermined standards for the successful completion of each task and one or more steps thereof to provide performance parameters;
performing a computation to produce learning curves from the performance parameters for the target healthcare professional and the matched peer group of the target healthcare professional, wherein the computation is selected from the group consisting of statistical modeling, deep learning modeling, and machine learning modeling, and wherein the computation accounts for evaluator rating bias and case complexity as confounders by estimating evaluator bias parameters and performing a Markov Chain Monte Carlo statistical sampling method to estimate posterior distributions of learning curve parameters;
from the learning curves for the target healthcare professional, calculating a competency score for the target healthcare professional for the procedure and each task thereof; and
comparing the learning curves and skill levels for the procedure and each task thereof for the target healthcare professional to that of the matched peer group of the target healthcare professional to determine a competency score for the target healthcare professional.
Examiner Note: underlined elements indicate additional elements of the claimed invention identified as performing the steps of the claimed invention.
The judicial exception expressed in claim 1 is not integrated into a practical application. The claim as a whole merely describes how to generally “apply” the concept of evaluating and determining competency of healthcare professionals based on clinical activities, performance evaluations, and peer comparisons in a computer environment. The claimed computer components (i.e., a computer, a server or data storage system, a user interface, a non-transitory computer-readable medium storing computer program instructions, software for analyzing input data and providing an output, and a data array) are recited at a high level of generality and are merely invoked as tools to perform an existing process of collecting information, evaluating practitioner performance, comparing practitioners against predetermined standards and peer groups, and determining competency scores. Simply implementing the abstract idea on a generic computer is not a practical application of the abstract idea. Accordingly, alone and in combination, these additional elements do not integrate the abstract idea into a practical application.
The judicial exception expressed in claim 1 is not integrated into a practical application. The claim recites the additional element of acquiring clinical schedules indicating clinical procedures to be performed. This limitation is recited at a high level of generality (i.e., as a general means of gathering or retrieving data for use in the claimed analysis), and amounts to merely data collection or data gathering, which is a form of insignificant extra-solution activity. Accordingly, even in combination, this additional element does not integrate the abstract idea into a practical application. The claim is directed to an abstract idea.
Therefore, under step 2A, the claims are directed to the abstract idea, and require further analysis under Step 2B.
Under step 2B
Claim 1 does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed with respect to Step 2A, the claim as a whole merely describes how to generally “apply” the concept evaluating and determining competency of healthcare professionals based on clinical activities, performance evaluations, and peer comparisons in a computer environment. Thus, even when viewed as a whole, nothing in the claim adds significantly more (i.e., an inventive concept) to the abstract idea.
For claim 1, under step 2B, the additional element of acquiring clinical schedules indicating clinical procedures to be performed has been evaluated. The data management platform comprising a computer performs a general function of receiving clinical and procedure information for subsequent analysis and evaluation, which represents a well-understood, routine, and conventional activity in the field of computer implemented data management and healthcare information systems. The specification discloses that the computer is used in its ordinary capacity as a data input device and does not describe any improvement to the computer itself or to the functioning of the overall computer system (see page 32-33). Also noted in Electric Power Group, LLC v. Alstom S.A., 830 F.3d 1350, 1354, 119 USPQ2d 1739, 1742 (Fed. Cir. 2016), merely collecting information for analysis without a technological improvement does not add significantly more to an abstract idea. The use of the data management platform is no more than collecting information before evaluating practitioner performance, generating competency assessments, and determining competency scores, and does not integrate the abstract idea into a practical application. Therefore, the claim does not recite an inventive concept and is not patent eligible.
Claims 2-4, 6, 8, 10-12, and 14-15, 17-19 recite no further additional elements, and only further narrow the abstract idea. The previously identified additional elements, individually and as a combination, do not integrate the narrowed abstract idea into a practical application for reasons similar to those explained above, and do not amount to significantly more than the narrowed abstract idea for reasons similar to those explained above.
Claims 5, 7, and 13 recite the additional elements of a graphical user interface (claim 5 and 13), a first element showing a staff assignment for a clinical encounter (claim 5 and 13), and a client device (claim 7). However, these additional elements amount to implementing an abstract idea on a generic computing device or mere displaying an output (i.e., an insignificant extra-solution activity). As such, these additional elements, when considered individually or in combination with the prior devices, do not integrate the abstract idea into a practical application or amount to significantly more than the abstract idea.
Thus, as the dependent claims remain directed to a judicial exception, and as the additional elements of the claims do not amount to significantly more, the dependent claims are not patent eligible.
Therefore, the claims here fail to contain any additional element(s) or combination of additional elements that can be considered as significantly more and the claims are rejected under 35 U.S.C. 101 for lacking eligible subject matter.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 9-15, and 17-20 are rejected under 35 U.S.C. 103 as being unpatentable over Wolf et al. (U.S. Patent Publication 2021/0307864 A1), referred to hereinafter as Wolf, in view of Brown et al. (U.S. Patent Publication 2020/0411170 A1), referred to hereinafter as Brown.
Regarding claim 9, Wolf teaches a method for determining a competency score for a target healthcare professional comprising the following steps, the platform comprising: (Wolf [0253] “In some embodiments, the stored data based on prior surgical procedures may include a machine learning model trained using a data set based on prior surgical procedures. For example, a machine learning model may be trained to process video frames and generate competency-related scores, as described below.” and Wolf [0121] At step 2460, process 2450 may include receiving a plurality of additional surgical videos from a plurality of surgical procedures performed by other medical professionals. Different portions of the plurality of additional surgical videos may correspond to at least one of intraoperative surgical events, surgical outcomes, patient characteristics, surgeon characteristics, and intraoperative surgical event characteristics. As described above, other medical professionals may be associated with a particular location, hospital, department, specialty, or residency class.”)
acquiring clinical schedules indicating clinical procedures to be performed (Wolf [0353] “FIG. 21 shows an example schedule 2100 that may include a listing of procedures such as procedures A-C (e.g., surgical procedures, or any other suitable medical procedures that may be performed in an operating room for which schedule 2100 is used). For each procedure A-C, a corresponding starting and finishing times may be determined. For example, for a past procedure A, a starting time 2121A and a finishing time 2121B may be the actual starting and finishing times. (Since procedure A is completed, the schedule 2100 may be automatically updated to reflect actual times). FIG. 21 shows that for a current procedure B, a starting time 2123A may be actual and a finishing time 2123B may be estimated (and recorded as an estimated time). Additionally, for procedure C, that is scheduled to be performed in the future, a starting time 2125A and a finishing time 2125B may be estimated and recorded. It should be noted that schedule 2100 is not limited to displaying and/or holding listings of procedures and starting/finishing times for the procedures, but may include various other data associated with an example surgical procedure.”);
listing component tasks and required skills for each procedure (Wolf [0353] “FIG. 21 shows an example schedule 2100 that may include a listing of procedures such as procedures A-C (e.g., surgical procedures, or any other suitable medical procedures that may be performed in an operating room for which schedule 2100 is used). For each procedure A-C, a corresponding starting and finishing times may be determined. For example, for a past procedure A, a starting time 2121A and a finishing time 2121B may be the actual starting and finishing times. (Since procedure A is completed, the schedule 2100 may be automatically updated to reflect actual times). FIG. 21 shows that for a current procedure B, a starting time 2123A may be actual and a finishing time 2123B may be estimated (and recorded as an estimated time). Additionally, for procedure C, that is scheduled to be performed in the future, a starting time 2125A and a finishing time 2125B may be estimated and recorded. It should be noted that schedule 2100 is not limited to displaying and/or holding listings of procedures and starting/finishing times for the procedures, but may include various other data associated with an example surgical procedure. For example, schedule 2100 may be configured to allow a user of schedule 2100 to interact with various elements of schedule 2100 (for cases when schedule 2100 is represented by a computer based interface such as a webpage, a software application, and/or another interface). For example, a user may be allowed to click over or otherwise select areas 2113, 2115, or 2117 to obtain details for procedures A, B or C respectively. Such details may include patient information (e.g., patient's name, age, medical history, etc.), surgical procedure information (e.g., a type of surgery, type of tools used for the surgery, type of anesthesia used for the surgery, and/or other characteristics of a surgical procedure), and healthcare provider information (e.g., a name of a surgeon, a name of an anesthesiologist, an experience of the surgeon, a success rate of the surgeon, a surgeon rating based on surgical outcomes for the surgeon, and/or other data relating to a surgeon). Some or all of the forgoing information may already appear in areas 2113, 2115, and 2117, without the need for further drill down.”);
assessing task complexity (Wolf [0105] “In some embodiments, the presentation of at least part of the frames assigned to the particular surgical event-related category includes a grouping of video frames from different surgical videos. The video frames may be complied into a common file for presentation or may be extracted at the time of playback from differing files. The video footage may be stored in the same location or may be selected from a plurality of storage locations. Although not necessarily so, videos within a set may be related in some way. For example, video footage within a set may include videos, recorded by the same capture device, recorded at the same facility, recorded at the same time or within the same timeframe, depicting surgical procedures performed on the same patient or group of patients, depicting the same or similar surgical procedures, depicting surgical procedures sharing a common characteristic (such as similar complexity level, including similar events, including usages of similar techniques, including usages of similar medical instruments, etc.), or sharing any other properties or characteristics.”);
performing a computation to produce learning curves from the performance parameters for the target healthcare professional and the matched peer group of the target healthcare professional, wherein the computation is selected from the group consisting of statistical modeling, deep learning modeling, and machine learning modeling (Wolf [0092] “Further embodiments may include presenting an interface enabling the specific physician to self-compare with the average skill. The interface may be a graphical user interface, e.g., user interface 700, such as on a display of a computing device. Presenting an interface may include outputting code from at least one processor, wherein the code may be configured to cause the interface to be presented. Consistent with the disclosure above, a skill score or other measure may be calculated for the specific physician and may be displayed alongside an average score or measure for the category of physicians. For example, a specific score may be calculated for a surgeon's skill in hiatal repair, wrap creation, fundus mobilization, esophageal mobilization, or other type of surgical procedure. The specific physician score in one or more surgical categories may be displayed alongside the average score for the category of physicians. In some embodiments, the specific physician score and the average score may be displayed via alphanumeric text. In other embodiments, the specific physician score and the average score may be displayed graphically. One or more scores may be displayed simultaneously either through alphanumeric text or graphically”, Wolf [0101] “In some embodiments, displaying the surgical event-related categories for selection together with the aggregate statistic for each surgical event-related category may include displaying in a juxtaposed manner, statistics of the specific medical professional and statistics of at least one of the other medical professionals. A juxtaposed manner may include displaying information from two medical professionals in a side-by-side fashion, in a table, in a graph, or in another visual arrangement that compares data between the professionals. This display allows a medical professional to compare his or her statistics against statistics of other medical professionals. For example, the display may depict a quantity of fluid leaks associated with a particular surgeon alongside a quantity of fluid leaks associated with a different surgeon. Comparisons can be made for any surgical-event related category. Other medical professionals may be associated with a particular location, hospital, department, specialty, or residency class. In some embodiments, an interface for permitting comparison of video frames captured from the specific medical professional and the at least one other medical professional may be provided. The interface may be a graphical user interface such as on a display of a computing device, e.g., user interface 700, or may include any other mechanism for providing the user with information.” and Wolf [0188] “In one example, a model (such as a statistical model, a machine learning model, a deep learning model, etc.) may be generated based on the prior surgical procedures, and the stored data may include the generated model and/or an indication of at least part of the generated model. For example, a machine learning model and/or a deep learning model may be trained using training examples based on the prior surgical procedures. While a host of correlation models may be used for prediction as discussed throughout this disclosure, exemplary predictive models may include a statistical model fit to historical image-related data (e.g., information relating to remedial actions) and outcomes; and a machine learning models trained to predict outcomes based on image-related data using training data based on historical examples.”);
from the learning curves for the target healthcare professional, calculating a competency score for the target healthcare professional for the procedure and each task thereof (Wolf [0092] “Further embodiments may include presenting an interface enabling the specific physician to self-compare with the average skill. The interface may be a graphical user interface, e.g., user interface 700, such as on a display of a computing device. Presenting an interface may include outputting code from at least one processor, wherein the code may be configured to cause the interface to be presented. Consistent with the disclosure above, a skill score or other measure may be calculated for the specific physician and may be displayed alongside an average score or measure for the category of physicians. For example, a specific score may be calculated for a surgeon's skill in hiatal repair, wrap creation, fundus mobilization, esophageal mobilization, or other type of surgical procedure. The specific physician score in one or more surgical categories may be displayed alongside the average score for the category of physicians. In some embodiments, the specific physician score and the average score may be displayed via alphanumeric text. In other embodiments, the specific physician score and the average score may be displayed graphically. One or more scores may be displayed simultaneously either through alphanumeric text or graphically.”); and
comparing the learning curves and skill levels for the procedure and each task thereof for the target healthcare professional to that of the matched peer group of the target healthcare professional to determine a competency score for the target healthcare professional (Wolf [0092] “Further embodiments may include presenting an interface enabling the specific physician to self-compare with the average skill. The interface may be a graphical user interface, e.g., user interface 700, such as on a display of a computing device. Presenting an interface may include outputting code from at least one processor, wherein the code may be configured to cause the interface to be presented. Consistent with the disclosure above, a skill score or other measure may be calculated for the specific physician and may be displayed alongside an average score or measure for the category of physicians. For example, a specific score may be calculated for a surgeon's skill in hiatal repair, wrap creation, fundus mobilization, esophageal mobilization, or other type of surgical procedure. The specific physician score in one or more surgical categories may be displayed alongside the average score for the category of physicians. In some embodiments, the specific physician score and the average score may be displayed via alphanumeric text. In other embodiments, the specific physician score and the average score may be displayed graphically. One or more scores may be displayed simultaneously either through alphanumeric text or graphically”, Wolf [0101] “In some embodiments, displaying the surgical event-related categories for selection together with the aggregate statistic for each surgical event-related category may include displaying in a juxtaposed manner, statistics of the specific medical professional and statistics of at least one of the other medical professionals. A juxtaposed manner may include displaying information from two medical professionals in a side-by-side fashion, in a table, in a graph, or in another visual arrangement that compares data between the professionals. This display allows a medical professional to compare his or her statistics against statistics of other medical professionals. For example, the display may depict a quantity of fluid leaks associated with a particular surgeon alongside a quantity of fluid leaks associated with a different surgeon. Comparisons can be made for any surgical-event related category. Other medical professionals may be associated with a particular location, hospital, department, specialty, or residency class. In some embodiments, an interface for permitting comparison of video frames captured from the specific medical professional and the at least one other medical professional may be provided. The interface may be a graphical user interface such as on a display of a computing device, e.g., user interface 700, or may include any other mechanism for providing the user with information.”, and Wolf [0092] “Further embodiments may include presenting an interface enabling the specific physician to self-compare with the average skill. The interface may be a graphical user interface, e.g., user interface 700, such as on a display of a computing device. Presenting an interface may include outputting code from at least one processor, wherein the code may be configured to cause the interface to be presented. Consistent with the disclosure above, a skill score or other measure may be calculated for the specific physician and may be displayed alongside an average score or measure for the category of physicians. For example, a specific score may be calculated for a surgeon's skill in hiatal repair, wrap creation, fundus mobilization, esophageal mobilization, or other type of surgical procedure. The specific physician score in one or more surgical categories may be displayed alongside the average score for the category of physicians. In some embodiments, the specific physician score and the average score may be displayed via alphanumeric text. In other embodiments, the specific physician score and the average score may be displayed graphically. One or more scores may be displayed simultaneously either through alphanumeric text or graphically.”).
Wolf fails to explicitly teach collecting performance evaluations for the target healthcare professional and a matched peer group of the target healthcare professional, for the performance of one or more selected procedures, each procedure having one or more tasks and an assigned clinical complexity value for the procedure and the one or more tasks thereof; and
compiling the evaluations versus predetermined standards for the successful completion of each task and one or more steps thereof to provide performance parameters.
Brown teaches collecting performance evaluations for the target healthcare professional and a matched peer group of the target healthcare professional, for the performance of one or more selected procedures, each procedure having one or more tasks and an assigned clinical complexity value for the procedure and the one or more tasks thereof (Brown [0068] “In some implementations of these embodiments, the healthcare worker information 204 can further include preference information for the respective tasks each worker is capable/qualified to performed regarding a preference of performance of the respective tasks. For example, the preference information can include system defined preference information for each task/worker combination that reflects the healthcare system or operating entity's preference for utilization the healthcare worker for the specific task relative to other tasks the healthcare worker is capable/qualified to perform. For instance, assume a surgeon is capable of performing a highly complex procedure as well as various general clinical tasks. With this example, the system can prefer the surgeon perform the highly complex procedure over the general clinical tasks, as this would in most scenarios be the most useful application of the surgeon's skills. Thus, the system can associate a higher preference rating with the complex procedure relative to the general clinical tasks. The preference information can also include worker preference rating information that provides a worker defined preference rating for performing different task reflective of the workers' personal preferences for performing certain tasks over others that the worker is capable/qualified to perform.” and Brown [0069] “The task capability information can further include information associated with the respective tasks that a worker is capable/qualified to perform regarding historically tracked and/or scored performance metrics for the respective tasks. For example, the performance metrics can include a general performance rating provided by the healthcare system that reflects the performance quality (e.g., measured by system review, employee feedback, patient feedback, etc.), efficiency, and proficiency of the healthcare worker in association with performance of each task. The performance metrics can also include information regarding the number and/or frequency of performance of the respective tasks. The performance metrics can also include information regarding historical error/complication rate. The task capability information can also include information regarding compensation schemes for compensating the healthcare worker for performing the different tasks. For example, in some implementations, a healthcare worker can be paid the same rate regardless of the specific task that the worker performs. In other embodiments, the healthcare worker can be paid different rates depending on the type of task and/or the specific task.”);
compiling the evaluations versus predetermined standards for the successful completion of each task and one or more steps thereof to provide performance parameters (Brown [0068] “In some implementations of these embodiments, the healthcare worker information 204 can further include preference information for the respective tasks each worker is capable/qualified to performed regarding a preference of performance of the respective tasks. For example, the preference information can include system defined preference information for each task/worker combination that reflects the healthcare system or operating entity's preference for utilization the healthcare worker for the specific task relative to other tasks the healthcare worker is capable/qualified to perform.”, Brown [0101] “In some embodiments, the attribute defining component 312 can also associate grouping and/or ordering attributes with two or more tasks regarding the grouping and/or ordering constraints determined by the task grouping component 308 and/or the task ordering component 310. In some embodiments, the attribute defining component 312 can also determine and associate resource attributes with the healthcare tasks that identify or indicate requirements for resources to be used for the tasks, including requirements for healthcare workers authorized to perform the tasks (e.g., determined using the healthcare worker information 204) as well as requirements for non-human resources, such as required medications, medical supplies, devices, equipment, technology and the like for use in association with performing the respective tasks (e.g., determined using the task definitions/requirements data 202 and/or the regulatory information 208).”, Brown [0071] The regulatory information 208 can include defined rules or regulations that provide guidelines regarding how to perform specific task and/or procedures circumstances. These rules or regulations are generally referred to as standard operating procedures (SOPs). For example, emergency room physicians have SOPs for patients who are brought in an unconscious state; nurses in an operating theater have SOPs for the forceps and swabs that they hand over to the operating surgeons; and laboratory technicians have SOPs for handling, testing, and subsequently discarding body fluids obtained from patients. Medical procedures can also be associated with SOPs that provide guidelines that define how to perform the procedure (e.g., steps to perform and how to perform them), how to respond to different patient conditions in association with performance of the procedure, how to respond to complications that arise, and other type of events that may arise over the course of the procedure, and the like. Some healthcare organizations can also establish or adopt SOPs for medical conditions that can define standard medical practices for treating patient's having the medical condition and the respective medical conditions. Some healthcare organizations can also have SOPs regarding providing healthcare to patients having two or more medical conditions (e.g., referred to as comorbidity). In this regard, the regulatory information 208 can include information that identifies and/or defines one or more standardized or defined protocols for following in association with performance of a procedure, treating a patient with a condition, and/or responding to a clinical scenario.”).
It would have been obvious to a person having ordinary skill in the art (PHOSITA) before the effective filing date of the invention to modify the competency assessment techniques of Wolf with the healthcare worker performance evaluation framework of Brown. Wolf teaches evaluating healthcare professionals by generating competency scores, comparing practitioners against other medical professionals, and employing statistical, machine learning, and deep learning models to assess practitioner performance. Brown teaches collecting and maintaining performance ratings, performance metrics, task capability information, worker qualifications, and information regarding the complexity of healthcare procedures and tasks. A PHOSITA would have recognized that both references are directed to assessing and improving healthcare professional performance using objective performance data.
Further, a PHOSITA would have been motivated to incorporate Brown's performance evaluation and task assessment information into Wolf's competency scoring system in order to improve the contextual accuracy and reliability of practitioner competency assessments. Brown provides additional information regarding worker performance quality, proficiency, task performance history, and task complexity, which would have predictably enhanced Wolf's ability to evaluate practitioner competency across different procedures and tasks. Combining these known techniques would have involved using one known performance assessment technique to improve another similar performance evaluation system, yielding the predictable result of a more comprehensive assessment of healthcare professional competency.
Additionally, both Wolf and Brown rely on computerized analysis of healthcare performance information to support evaluation and decision making. The combination applies known data sources, performance metrics, and modeling techniques in their expected manner to assess practitioner skill and competency. A PHOSITA would have had a reasonable expectation of success in combining the references because the teachings are complementary, address similar problems relating to healthcare professional performance assessment, and employ compatible data driven evaluation methodologies. The combination would have predictably provided competency scores based on performance evaluations, task information, and comparisons among healthcare professionals.
Regarding claim 10, Wolf and Brown teach the invention in claim 9, as discussed above, and further teach wherein the computation to produce learning curves is a deep learning curve modeling comprising the step of performing a statistical sampling method calculation to produce one or more learning curves for the target healthcare professional and the matched peer group of the target healthcare professional (Wolf [0092] “Further embodiments may include presenting an interface enabling the specific physician to self-compare with the average skill. The interface may be a graphical user interface, e.g., user interface 700, such as on a display of a computing device. Presenting an interface may include outputting code from at least one processor, wherein the code may be configured to cause the interface to be presented. Consistent with the disclosure above, a skill score or other measure may be calculated for the specific physician and may be displayed alongside an average score or measure for the category of physicians. For example, a specific score may be calculated for a surgeon's skill in hiatal repair, wrap creation, fundus mobilization, esophageal mobilization, or other type of surgical procedure. The specific physician score in one or more surgical categories may be displayed alongside the average score for the category of physicians. In some embodiments, the specific physician score and the average score may be displayed via alphanumeric text. In other embodiments, the specific physician score and the average score may be displayed graphically. One or more scores may be displayed simultaneously either through alphanumeric text or graphically”, Wolf [0101] “In some embodiments, displaying the surgical event-related categories for selection together with the aggregate statistic for each surgical event-related category may include displaying in a juxtaposed manner, statistics of the specific medical professional and statistics of at least one of the other medical professionals. A juxtaposed manner may include displaying information from two medical professionals in a side-by-side fashion, in a table, in a graph, or in another visual arrangement that compares data between the professionals. This display allows a medical professional to compare his or her statistics against statistics of other medical professionals. For example, the display may depict a quantity of fluid leaks associated with a particular surgeon alongside a quantity of fluid leaks associated with a different surgeon. Comparisons can be made for any surgical-event related category. Other medical professionals may be associated with a particular location, hospital, department, specialty, or residency class. In some embodiments, an interface for permitting comparison of video frames captured from the specific medical professional and the at least one other medical professional may be provided. The interface may be a graphical user interface such as on a display of a computing device, e.g., user interface 700, or may include any other mechanism for providing the user with information.” and Wolf [0188] “In one example, a model (such as a statistical model, a machine learning model, a deep learning model, etc.) may be generated based on the prior surgical procedures, and the stored data may include the generated model and/or an indication of at least part of the generated model. For example, a machine learning model and/or a deep learning model may be trained using training examples based on the prior surgical procedures. While a host of correlation models may be used for prediction as discussed throughout this disclosure, exemplary predictive models may include a statistical model fit to historical image-related data (e.g., information relating to remedial actions) and outcomes; and a machine learning models trained to predict outcomes based on image-related data using training data based on historical examples.”), Wolf [0054] “Machine learning algorithms (also referred to artificial intelligence) may be employed for the purposes of analyzing the video to identify surgical events. Such algorithms be trained using training examples, such as described below. Some non-limiting examples of such machine learning algorithms may include classification algorithms, data regressions algorithms, image segmentation algorithms, visual detection algorithms (such as object detectors, face detectors, person detectors, motion detectors, edge detectors, etc.), visual recognition algorithms (such as face recognition, person recognition, object recognition, etc.), speech recognition algorithms, mathematical embedding algorithms, natural language processing algorithms, support vector machines, random forests, nearest neighbors algorithms, deep learning algorithms, artificial neural network algorithms, convolutional neural network algorithms, recursive neural network algorithms, linear machine learning models, non-linear machine learning models, ensemble algorithms, and so forth. For example, a trained machine learning algorithm may comprise an inference model, such as a predictive model, a classification model, a regression model, a clustering model, a segmentation model, an artificial neural network (such as a deep neural network, a convolutional neural network, a recursive neural network, etc.), a random forest, a support vector machine, and so forth. In some examples, the training examples may include example inputs together with the desired outputs corresponding to the example inputs. Further, in some examples, training machine learning algorithms using the training examples may generate a trained machine learning algorithm, and the trained machine learning algorithm may be used to estimate outputs for inputs not included in the training examples. In some examples, engineers, scientists, processes and machines that train machine learning algorithms may further use validation examples and/or test examples. For example, validation examples and/or test examples may include example inputs together with the desired outputs corresponding to the example inputs, a trained machine learning algorithm and/or an intermediately trained machine learning algorithm may be used to estimate outputs for the example inputs of the validation examples and/or test examples, the estimated outputs may be compared to the corresponding desired outputs, and the trained machine learning algorithm and/or the intermediately trained machine learning algorithm may be evaluated based on a result of the comparison. In some examples, a machine learning algorithm may have parameters and hyper parameters, where the hyper parameters may be set manually by a person or automatically by a process external to the machine learning algorithm (such as a hyper parameter search algorithm), and the parameters of the machine learning algorithm may be set by the machine learning algorithm according to the training examples. In some implementations, the hyper-parameters may be set according to the training examples and the validation examples, and the parameters may be set according to the training examples and the selected hyper-parameters.”).
Therefore, it would be obvious to a PHOSITA before the effective filing date of the invention to implement the computation to produce learning curves using a deep learning modeling approach that includes statistical sampling methods, as taught by Wolf, because machine learning and statistical modeling techniques were well known in the art for analyzing performance data and generating predictive learning curves. A PHOSITA would have been motivated to use deep learning and sampling to improve model accuracy and generalization across practitioners, a predictable enhancement to existing competency analysis systems.
Regarding claim 11, Wolf and Brown teach the invention in claim 9, as discussed above, and further teach wherein the healthcare professional is selected from the group consisting of medical students, interns, residents, fellows, doctors, physician assistants, nurses, nurses' aides, and medical technicians (Wolf [0077] “Aspects of the present disclosure may involve medical professionals performing surgical procedures. A medical professional may include, for example, a surgeon, a surgical technician, a resident, a nurse, a physician's assistant, an anesthesiologist, a doctor, a veterinarian surgeon, and so forth.”).
Therefore, it would be obvious to a PHOSITA before the effective filing date of the invention to apply the competency scoring platform to any healthcare professional type (such as medical students, residents, or nurses) because Wolf teaches that the same performance assessment framework can be used across various medical professionals performing surgical procedures. It would have been a routine design choice to extend the system to all practitioner categories engaged in clinical procedures, yielding predictable results.
Regarding claim 12, Wolf and Brown teach the invention in claim 9, as discussed above, and further teach involving a teaching situation involving an evaluator healthcare professional and a target healthcare professional (Wolf [0239] In surgical procedures, it is important that the surgeon is competent. Thus, evaluation of the performance of health care providers (such as interns, residents, attendings, etc.) in surgeries is an import part in training of physicians and in the management of health care organizations.” and Brown [0069] “The task capability information can further include information associated with the respective tasks that a worker is capable/qualified to perform regarding historically tracked and/or scored performance metrics for the respective tasks. For example, the performance metrics can include a general performance rating provided by the healthcare system that reflects the performance quality (e.g., measured by system review, employee feedback, patient feedback, etc.), efficiency, and proficiency of the healthcare worker in association with performance of each task.”).
Therefore, it would be obvious to a PHOSITA before the effective filing date of the invention to implement the competency assessment in a teaching or training environment involving evaluators and target healthcare professionals, as Wolf and Brown both teach the use of evaluation metrics and feedback in physician training programs. A PHOSITA would have been motivated to combine these teachings to enable real time assessment of trainees by evaluators, a predictable and widely used application in medical education systems.
Regarding claim 13, Wolf and Brown teach the invention in claim 9, as discussed above, and further teach wherein the user interface is a graphical user interface configured to augment a clinical schedule with case-based actions; the graphical user interface comprising (Wolf [0277] “By way of example, FIG. 17 illustrates an exemplary interface for a system for assessing surgical competency of a subject, consistent with disclosed embodiments. As shown in FIG. 17, interface 1700 may display a variety of information to a user. For example, interface 1700 may include a name or other indication of a surgical procedure 1701 and a name or other identifier of a subject 1703. In some embodiments, interface 1700 may indicate a procedure type 1709 (e.g., a laparoscopic surgery, an open surgery, a robotic surgery, an appendectomy, a cataract surgery, or other type of surgical procedure). Interface 1700 may also provide a link 1711 to video clips related to the procedure type 1709. Such video clips may be related to different instances of the same procedure type in which the same subject participated. Additionally, or alternatively, such video clips linked by link 1711 may be related to procedures having procedure type 1709 that were performed by other subjects. For example, such video clips may be model or reference clips showing a highly skilled surgeon properly executing the procedure or specific actions related to the procedure.”):
a first element showing a staff assignment for a clinical encounter (Wolf [0342] “Aspects of this disclosure may relate to assigning surgical teams to prospective surgeries. A surgical team, in its broadest sense, may include one or more surgeons, and may also include one or more medical assistants such as nurses, technicians, or other physicians or support staff who may be involved in a surgical procedure. Assigning a surgical team may include selecting or scheduling one or more individuals to perform prospective surgeries or aid a user such as a scheduler in performing the same. Assigning surgical teams or individual members of a surgical team to prospective surgeries may involve publishing a schedule or otherwise notifying a surgical team or individual surgical team member of an assignment to a prospective surgery, providing instruction and/or information to a system that facilitate scheduling, such as a calendar. A prospective surgery may refer to a future or potential surgical procedure, as defined herein.”); and
a second element juxtaposed to the first element and showing a button, a tag, a status label, or an actionable link for an encounter-related activity, such as case logging, performance evaluation, data quality control, and accessing medical educational content (Wolf [0292] “At step 1970, process 1900 may include presenting in association with the at least one score, a link to the at least one video clip. As described herein, a link may be presented such that when the link is selected, the corresponding video clip may be presented for viewing. As described herein, the score and link may be presented, for example, in an interface in which a user may view the score and actuate the link to view the corresponding video. According to disclosed embodiments, the method may further include classifying a surgical procedure type associated with the video clip and presenting a control enabling a viewer to access other video clips in which the subject is presented sharing the surgical procedure type.” Wolf [0279] “Interface 1700 may include an indication of competency-related scores 1713. As described herein, competency-related scores may include an overall (e.g., composite) score, as well as individual scores related to certain aspects of a procedure or surgical skills (e.g., tissue handling, economy of motion, etc.). Interface 1700 may provide an indication of the type of score, the score itself, or a link to a video related to the score. For example, interface 1700 shows an overall score of 15 out of a maximum 20 and a corresponding video link. As described above, the link may include an activatable icon. When the activatable icon is selected, a window may open for playback and controls enabling playback may be presented in the graphical user interface for viewing by the user.” and Wolf [0280] “Interface 1700 may also include additional data links 1715. Data links 1715 may correspond to a variety of other data related to the surgical procedure, the type of surgical procedure, the subject, actions or events within the surgical procedure, previous assessments or performance statistics of the subject, previous video clips of the subject, or other suitable information. For example, links 1715 may provide a link to a previous assessment for the subject related to a different surgical procedure of the same type. As another example, links 1715 may display graphs, charts, or other visuals indicating the subject's performance over time (e.g., a line graph showing a change in a particular competency score of the subject over multiple procedures).”).
Therefore, it would be obvious to a PHOSITA before the effective filing date of the invention to configure the user interface as a graphical user interface (GUI) augmenting clinical schedules with case-based actions, staff assignments, and interactive links for performance evaluation or educational content, as taught by Wolf. A PHOSITA would have been motivated to adopt a GUI format because graphical, interactive dashboards were known to improve usability, data accessibility, and workflow efficiency in clinical and educational information systems, leading to predictable and desirable results.
Regarding claim 14, Wolf and Brown teach the invention in claim 9, as discussed above, and further teach wherein the performance evaluations are provided manually (Brown [0069] “The task capability information can further include information associated with the respective tasks that a worker is capable/qualified to perform regarding historically tracked and/or scored performance metrics for the respective tasks. For example, the performance metrics can include a general performance rating provided by the healthcare system that reflects the performance quality (e.g., measured by system review, employee feedback, patient feedback, etc.), efficiency, and proficiency of the healthcare worker in association with performance of each task.”).
Therefore, it would be obvious to a PHOSITA before the effective filing date of the invention to incorporate manually provided performance evaluations (such as supervisor, peer, or patient feedback_ into the competency scoring platform, because Brown teaches the use of human entered ratings and reviews to capture performance quality, efficiency, and proficiency. A PHOSITA would have been motivated to combine manual and automated data inputs to ensure a comprehensive and accurate record of healthcare worker performance, a predictable improvement to competency assessment accuracy.
Regarding claim 15, Wolf and Brown teach the invention in claim 9, as discussed above, and further teach wherein the platform is embedded in a health record data system in a hospital (Wolf [0129] “Accessing the video of the surgical procedure may be performed via communication to a computer system through a network. For example, FIG. 4 shows an example system 401 that may include a computer system 410, a network 418, and image sensors 421 (e.g., cameras positioned within the operating room), and 423 (e.g., image sensors being part of a surgical instrument) connected via network 418 to computer system 401. System 401 may include a database 411 for storing various types of data related to previously conducted surgeries (i.e., historical surgical data that may include historical image, video or audio data, text data, doctors' notes, data obtained by analyzing historical surgical data, and other data relating to historical surgeries). In various embodiments, historical surgical data may be any surgical data related to previously conducted surgical procedures. Additionally, system 401 may include one or more audio sensors 425, wireless transmitters 426, light emitting devices 427, and a schedule 430.”, Wolf [0130] “Computer system 410 may include one or more processors 412 for analyzing the visual data collected by the image sensors, a data storage 413 for storing the visual data and/or other types of information, an input module 414 for entering any suitable input for computer system 410, and software instructions 416 for controlling various aspects of operations of computer system 410.” and Wolf [0354] “FIG. 4 is a network diagram that may include a computer system 410, a network 418, and a database 411 storing patient characteristics. Computer system 410 may access database 411 via network 418 in order to obtain patient characteristics to be included in or used to produce schedule 430. In another example, patient characteristics may be obtained from an Electronic Medical Record software.”).
Therefore, it would be obvious to a PHOSITA before the effective filing date of the invention to embed the competency assessment platform within a hospital’s electronic health record (EHR) or data management system, because Wolf discloses networked hospital systems accessing EMR databases and patient data. Integrating the platform with existing EHR infrastructure would have been an obvious and predictable modification to enable automated data retrieval and improve clinical workflow efficiency.
Regarding claim 17, Wolf and Brown teach the invention in claim 9, as discussed above, and further teach further configured to comprise a step of determining a risk score, wherein the risk score indicates a probability of a clinical event achieving a predetermined patient outcome (Wolf [0206] “Aspects of this disclosure may include predicting, based on the plurality of video frames and the stored data based on prior surgical procedures, at least one expected future event in the ongoing surgical procedure. For example, a data structure may include stored data representing relationships between intraoperative events and predicted outcomes. Such data structures may be used to obtain a predicted outcome associated with a specific surgical procedure. For example, FIG. 12 shows an example graph 1200 of intraoperative events E1-E3 connected to possible outcomes C1-C3 using connections n11-n32. Connection n11 may include information indicating a probability of an outcome C1 (i.e., information indicating how often outcome C1 happens in surgical procedures that includes event E1). In some aspects, connection n11 may indicate that given an occurrence of intraoperative event E1, outcome C1 may happen 30 percent of the time, connection n12 may indicate that outcome C2 may happen 50 percent of the time, and connection n13 may indicate that outcome C3 may happen 20 percent of the time. Similarly, connection n22 may indicate a probability of outcome C2, given an occurrence of intraoperative event E2, and connection n23 may indicate a probability of outcome C3, given an occurrence of intraoperative event E2. A connection n32 may indicate a probability of outcome C2, given an occurrence of intraoperative event E3. Thus, once an intraoperative event is known, using information obtained from historical data (e.g., using information from graph C100), a most probable outcome (e.g., outcome C2) may be determined based on probability assigned to connections n11-n13. In another example, the historical information may include a hypergraph, a hyperedge of the hypergraph may connect a plurality of intraoperative events with an outcome and may indicate a particular probability of the outcome in surgical procedures that included the plurality of events. Thus, once a plurality of intraoperative events is known, using information obtained from historical data (e.g., from the hypergraph), a most probable outcome may be determined based on probability assigned to the hyperedges. In some examples, probabilities assigned to edges of graph C100 or to the hyperedges of the hypergraph may be based on an analysis of historical surgical procedures, for example by calculating the statistical probability of an outcome in a group of historical surgical procedures that include particular group of intraoperative events corresponding to a particular edge or a particular hyperedge. In some other examples, the historical information may include a trained machine learning model for predicting outcome based on intraoperative events, and the trained machine learning model may be used to predict the outcome associated with the specific surgical procedure based on the identified at least one intraoperative event. In one example, the trained machine learning model may be obtained by training a machine learning algorithm using training examples, and the training examples may be based on historical surgical procedure. An example of such training example may include a list of intraoperative surgical events, together with a label indicating an outcome corresponding to the list of intraoperative surgical events. In one example, two training examples may have the same list of intraoperative surgical events, while having different label indicating different outcomes.”).
Therefore, it would be obvious to a PHOSITA before the effective filing date of the invention to extend the competency scoring platform to include computation of a risk score indicating the probability of achieving a patient outcome, because Wolf teaches predictive modeling of surgical events and associated outcome probabilities. A PHOSITA would have been motivated to integrate outcome prediction with competency assessment to provide a fuller measure of practitioner performance and patient safety risk, a predictable enhancement using known statistical and machine learning techniques.
Regarding claim 18, Wolf and Brown teach the invention in claim 17, as discussed above, and further teach wherein the risk score is calculated for an individual practitioner to perform a specific procedure (Wolf [0277] “By way of example, FIG. 17 illustrates an exemplary interface for a system for assessing surgical competency of a subject, consistent with disclosed embodiments. As shown in FIG. 17, interface 1700 may display a variety of information to a user. For example, interface 1700 may include a name or other indication of a surgical procedure 1701 and a name or other identifier of a subject 1703. In some embodiments, interface 1700 may indicate a procedure type 1709 (e.g., a laparoscopic surgery, an open surgery, a robotic surgery, an appendectomy, a cataract surgery, or other type of surgical procedure). Interface 1700 may also provide a link 1711 to video clips related to the procedure type 1709. Such video clips may be related to different instances of the same procedure type in which the same subject participated. Additionally, or alternatively, such video clips linked by link 1711 may be related to procedures having procedure type 1709 that were performed by other subjects. For example, such video clips may be model or reference clips showing a highly skilled surgeon properly executing the procedure or specific actions related to the procedure.” and Wolf [0279] “Interface 1700 may include an indication of competency-related scores 1713. As described herein, competency-related scores may include an overall (e.g., composite) score, as well as individual scores related to certain aspects of a procedure or surgical skills (e.g., tissue handling, economy of motion, etc.). Interface 1700 may provide an indication of the type of score, the score itself, or a link to a video related to the score. For example, interface 1700 shows an overall score of 15 out of a maximum 20 and a corresponding video link. As described above, the link may include an activatable icon. When the activatable icon is selected, a window may open for playback and controls enabling playback may be presented in the graphical user interface for viewing by the user.”).
Therefore, it would be obvious to a PHOSITA before the effective filing date of the invention to calculate a personalized risk score for an individual practitioner performing a specific procedure, because Wolf teaches practitioner specific scoring and predictive outcome modeling based on prior surgical data. A PHOSITA would have recognized that applying these predictive techniques to individual clinicians would yield a useful, predictable improvement in individualized performance tracking and risk management.
Regarding claim 19, Wolf and Brown teach the invention in claim 17, as discussed above, and further teach further comprising determining a multi-task aggregate competency score based on individual task scores for an overall procedure (Wolf [0279] “Interface 1700 may include an indication of competency-related scores 1713. As described herein, competency-related scores may include an overall (e.g., composite) score, as well as individual scores related to certain aspects of a procedure or surgical skills (e.g., tissue handling, economy of motion, etc.). Interface 1700 may provide an indication of the type of score, the score itself, or a link to a video related to the score. For example, interface 1700 shows an overall score of 15 out of a maximum 20 and a corresponding video link. As described above, the link may include an activatable icon. When the activatable icon is selected, a window may open for playback and controls enabling playback may be presented in the graphical user interface for viewing by the user.” and Wolf [0289] “At step 1940, process 1900 may include, based on the assessment of at least one of tissue handling, economy of motion, depth perception and surgical procedure flow, generating a competency-related score for a subject. As described herein, a competency-related score may indicate a relative competency or skill level of a subject. Scores may be generated for individual skills, actions, etc. of the subject. In some embodiments, an overall score indicating the subject's general overall competency may be generated. Composite scores including an aggregation or weighted average of individual skill scores may also be generated.”).
Therefore, it would be obvious to a PHOSITA before the effective filing date of the invention to generate a multi-task aggregate competency score derived from individual task scores, because Wolf discloses combining multiple performance measures into composite or weighted overall competency indicators. A PHOSITA would have been motivated to aggregate task level results into an overall score to simplify reporting and enable overall evaluation of practitioner proficiency, a predictable and routine step in performance assessment systems.
Claim 20 is analogous to claim 9, thus claim 20 is similarly analyzed and rejected in a manner consistent with the rejection of claim 9.
Response to Arguments
Applicant’s arguments and amendments, see Remarks/Amendments submitted on 04/09/2026 with respect to the rejection of the claims have been carefully considered and is addressed below.
Drawings
The drawings were received on 04/09/2026. These drawings are acceptable.
Claim Rejections - 35 USC § 112
Applicant’s amendment to claim 7 is persuasive. The recitation of specific security features, including role-based access controls, encryption of PHI, local encryption using secret encryption keys, and two factor authentication, clarifies the scope of the claim. Accordingly, the rejection under 35 U.S.C. 112(b) is withdrawn.
Claim Rejections - 35 USC § 101
Applicant's arguments have been fully considered but are not persuasive. Applicant states that the claims are directed to a specific data management platform that aggregates and standardizes healthcare data, performs duplicate removal and ontology mapping, constructs matched peer groups, executes statistical computations, and generates competency scores and learning curves. However, the recited limitations are evaluating healthcare professionals, comparing their performance against peers and predetermined standards, and determining competency scores. These activities constitute observation, evaluation, judgment, comparison, and mathematical analysis, which fall within the mental process and mathematical concept groupings of abstract ideas. The additional data processing steps relied upon by Applicant collect and analyze information used to perform the claimed evaluation and scoring process.
Applicant's argument that the claims cannot practically be performed in the human mind because they involve large volumes of data, cross institutional information, and Markov Chain Monte Carlo sampling is not persuasive. The claim recites evaluating performance data, comparing practitioners to standards and peer groups, determining competency, and performing mathematical calculations. Increasing the amount of data analyzed or the complexity of the calculations does not remove the claim from the abstract idea categories identified.
Applicant's reliance on Enfish, DDR Holdings, and McRO is also unpersuasive. Unlike Enfish, the claims do not recite an improvement to computer functionality, database architecture, memory structure, or any other aspect of computer technology. Unlike DDR Holdings, the claims do not solve a problem unique to computer networks or computer technology, but instead address the evaluation and assessment of healthcare professional competency. Further, unlike McRO, the claims do not recite a specific set of technological rules that improve a technological process. Rather, the claims use statistical modeling, machine learning, and mathematical analysis to evaluate human performance and generate competency assessments. Similarly, the recited standardization, duplicate removal, ontology mapping, peer group clustering, evaluator bias correction, and learning curve generation manipulate, classify, organize, and analyze information and do not constitute improvements to computer technology itself.
Applicant's arguments regarding practical application and inventive concept are also unpersuasive. The claimed computer, server, data storage system, user interface, software, and data array perform their ordinary and expected functions of receiving, storing, processing, and presenting information. The claims do not improve the functioning of a computer, improve another technology or technical field, or otherwise apply the abstract idea in a meaningful way beyond generally linking it to a technological environment. Further, the claimed statistical modeling, machine learning, evaluator bias estimation, Markov Chain Monte Carlo sampling, peer group clustering, and competency score calculations form part of the abstract idea itself and therefore cannot provide the inventive concept. Viewed individually and as an ordered combination, the claim automates the collection, organization, analysis, and evaluation of information relating to healthcare professional competency and therefore does not recite significantly more than the abstract idea.
Claim Rejections - 35 USC § 103
Applicant's arguments have been fully considered. In view of Applicant’s amendments and arguments, the rejection under 35 U.S.C. 103 previously applied to claims 1-8 has been withdrawn. Applicant's arguments focus extensively on limitations relating to structured evaluator completed performance evaluations, matched peer groups formed by clustering evaluation ratings, learning curves, evaluator bias correction, and Markov Chain Monte Carlo (MCMC) statistical sampling. The Examiner agrees that the amendments to claims 1-8 introduce limitations that are not taught or suggested by the prior art combination previously relied upon and, accordingly, the rejection of claims 1-8 has been withdrawn.
However, the rejection maintained against claims 9-15 and 17-20 is directed to the limitations expressly recited in those claims and the teachings relied upon in Wolf and Brown. Accordingly, arguments directed to structured evaluator completed performance evaluations, matched peer groups formed by clustering evaluation ratings, learning curves, evaluator bias correction, and Markov Chain Monte Carlo (MCMC) statistical sampling, which are not recited in the rejected claims, are not persuasive.
Applicant states that Wolf is directed to video surgical analysis and therefore cannot teach competency assessment based on performance information. The Examiner disagrees, because Wolf expressly teaches generating competency scores for healthcare professionals, calculating skill scores, comparing a physician's performance to that of other medical professionals, and employing statistical models, machine learning models, and deep learning models to evaluate practitioner performance. Wolf is relied upon for competency scoring, practitioner comparison, procedure information, and statistical, machine learning, and deep learning based performance assessment. Therefore, Applicant's arguments attacking features not relied upon in the rejection are not persuasive.
Applicant also states that Brown is directed to workforce scheduling rather than competency assessment. However, Brown teaches healthcare worker performance ratings, performance quality metrics, proficiency information, task capability information, worker qualifications, and complexity information associated with healthcare procedures and tasks. Brown's disclosure of performance ratings, worker qualifications, task requirements, and standardized operating procedures provides relevant evidence regarding the evaluation of healthcare worker performance and task requirements.
Applicant's arguments regarding lack of motivation to combine are also unpersuasive. Both Wolf and Brown are directed to the analysis and evaluation of healthcare professional performance using computerized systems and performance data. Wolf evaluates practitioner competency and skill using data driven modeling techniques, while Brown collects and utilizes worker performance metrics, qualifications, task requirements, and complexity information to evaluate and optimize healthcare worker utilization. A person of ordinary skill in the art would have recognized that incorporating Brown's performance evaluation and task information into Wolf's competency assessment framework would predictably improve the contextual accuracy and reliability of practitioner competency scoring. The combination applies known performance assessment information to an existing competency evaluation system according to known methods and would have yielded no more than predictable results. Accordingly, the rejection under 35 U.S.C. 103 is maintained.
Conclusion
The prior art made of record and not relied upon is considered pertinent to Applicant's disclosure.
Myers et al. (U.S. Patent Publication 2008/0059292 A1) teaches a system for continuously assessing a user’s professional performance against benchmarks, standards, goals, and peer groups, identifying performance deficiencies in real time, and providing targeted improvement pathways and credential tracking tools to support ongoing professional development.
Tashiro et al. (U.S. Patent Publication 2013/0029300) teaches a method for evaluating healthcare practitioner’s competency by using a virtual clinical environment, tracks the practitioner’s interactions and patient data, and generates performance levels across multiple assessment categories.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KYRA R LAGOY whose telephone number is (703)756-1773. The examiner can normally be reached Monday - Friday, 8:00 am - 5:00 pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kambiz Abdi can be reached at (571)272-6702. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/K.R.L./Examiner, Art Unit 3685
/KAMBIZ ABDI/Supervisory Patent Examiner, Art Unit 3685
Prosecution Insights