Detailed Notice
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Claims
Claims 1-20 are currently pending.
Claims 1-20 are rejected.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to a judicial exception (i.e., a law of nature, a natural phenomenon, or an abstract idea) without significantly more.
Step 1:
In the instant case, claims 1-12 and 14-20 are directed toward a system (i.e. machine), claim 13 is directed toward a method (i.e. a process). Thus, each of the claims falls within one of the four statutory categories. Nevertheless, the claims fall within the judicial exception of an abstract idea.
Step 2A—Prong 1:
Independent claims 1, 13, and 14 recites steps that, under their broadest reasonable interpretations, cover performance of the limitations of a certain method of organizing human activity but for the recitation of generic computer components.
Claim 1 recites: “A system, comprising: one or more processors, coupled with memory, to: determine multimodal data, the multimodal data comprises two or more of three- dimensional point cloud data, video data, robotic system data, instrument data, or metadata for at least one medical procedure; determine at least one indication using the multimodal data, wherein the at least one indication comprises at least one of: one or more metric values; or statistical information; identify at least one potential cause of efficiency or inefficiency for the at least one indication using a mapping between a plurality of potential causes and a plurality of indications; and cause a display device to display a message comprising the at least one potential cause”.
The limitations of determine multimodal data, the multimodal data comprises two or more of three- dimensional point cloud data, video data, robotic system data, instrument data, or metadata for at least one medical procedure; determine at least one indication using the multimodal data, wherein the at least one indication comprises at least one of: one or more metric values; or statistical information; identify at least one potential cause of efficiency or inefficiency for the at least one indication using a mapping between a plurality of potential causes and a plurality of indications; and… display a message comprising the at least one potential cause, given the broadest reasonable interpretation, cover the abstract idea of a certain method of organizing human activity because they recite managing personal behavior or relationships or interactions between people (i.e. social activities, teaching, and following rules or instructions—in this case the aforementioned steps recite a process of determine, identify, and display, which is properly interpreted as a “personal behavior”), but instead automates the process via a computer model, e.g. see MPEP 2106.04(a)(2). Any limitations not identified above as part of the abstract idea are deemed “additional elements”, and will be discussed in further detail below.
Further, the abstract idea of claim 13 is identical as the abstract idea of claim 1. This limitation, given the broadest reasonable interpretation, also falls under the abstract idea of a certain method of organizing human activity because it recites managing personal behavior or relationships or interactions between people.
Additionally, claim 14 recites: “A system, comprising: one or more processors, coupled with memory, to: determine a dataset of information of a plurality of the first medical procedures in a plurality of first medical environments, wherein the dataset comprises: metadata of the plurality of the first medical procedures and the plurality of first medical environments; a timeline of a plurality of phases and a plurality of tasks within each of the plurality of phases determined for the plurality of the first medical procedures; and three-dimensional point cloud data for the plurality of the first medical procedures and the plurality of first medical environments during at least portions of the plurality of phases and the plurality of tasks within each of the phases; determine a mapping among data within the dataset; receive a first attribute of at least one second medical procedure or at least one second medical environment; and determine, using the mapping and first attribute, a second attribute of the at least one second medical procedure and the at least one second medical environment”.
The limitations of determine a dataset of information of a plurality of the first medical procedures in a plurality of first medical environments, wherein the dataset comprises: metadata of the plurality of the first medical procedures and the plurality of first medical environments; a timeline of a plurality of phases and a plurality of tasks within each of the plurality of phases determined for the plurality of the first medical procedures; and three-dimensional point cloud data for the plurality of the first medical procedures and the plurality of first medical environments during at least portions of the plurality of phases and the plurality of tasks within each of the phases; determine a mapping among data within the dataset; and determine, using the mapping and first attribute, a second attribute of the at least one second medical procedure and the at least one second medical environment, given the broadest reasonable interpretation, cover the abstract idea of a certain method of organizing human activity because they recite managing personal behavior or relationships or interactions between people (i.e. social activities, teaching, and following rules or instructions—in this case the aforementioned steps recite a process of determine, which is properly interpreted as a “personal behavior”), but instead automates the process via a computer model or machine learning, e.g. see MPEP 2106.04(a)(2). Any limitations not identified above as part of the abstract idea are deemed “additional elements”, and will be discussed in further detail below.
Dependent claims 2-12 and 15-20 include other limitations, as well as specific step of data to be processed, received, and applied, but these only serve to further limit the abstract idea and do not add and additional elements, and hence are nonetheless directed towards fundamentally the same abstract idea as independent claims 1, 13, and 14. However, recitation of an abstract idea is not the end of the 35 U.S.C. 101 analysis. Each of the claims must be analyzed for additional elements that indicate the abstract idea is integrated into a practical application to determine whether the claim is considered to be “directed to” an abstract idea.
Step 2A—Prong 2:
Claims 1-20 are not integrated into a practical application because the additional elements (i.e. any limitations that are not identified as part of the abstract idea) amount to no more than limitations which:
Amount to mere instructions to apply an exception—for example, the recitation of “system”, “processors”, “memory”, “display”, which amount to merely invoking a computer as a tool to perform the abstract idea, e.g. see FIG. 30 and [0073], of the present specification, and see further MPEP 2106.05(f);
Generally linking the abstract idea to a particular technological environment or field of use, for example, “one or more processors, coupled with memory, to” and “cause a display device to”, which amounts to limiting the abstract idea to the field of technology/the environment of computers, see MPEP 2106.05(h); and/or
Merely acquiring information for further analysis by the system and the particular manner of acquisition is not described or shown to be important, for example, “receive a first attribute of at least one second medical procedure or at least one second medical environment”, which amounts to insignificant extra-solution activity in the form of mere data gathering because it merely functions tangentially to the main idea of the invention and serves only to bring in the data necessary for the inventions main analysis, see MPEP 2106.05(g).
Additionally, dependent claims 2-12 and 15-20 include other limitations, for example:
Claim 2 recites the additional elements of a “depth acquiring sensors” and “visual image sensor”;
Claim 3 recites the additional element of a robotic system;
Claim 15 recites the additional elements of a plurality of robotic systems;
Claim 19 recites the additional element of a robotic systems;
but as stated above, the limitations recited by these claims also do not integrate the aforementioned abstract idea into a practical application.
Step 2B:
The claims do not include additional elements (i.e., “system”, “processors”, “memory”, “display”) that are sufficient to amount to “significantly more” than the judicial exception because the additional elements (i.e. the elements other than the abstract idea), as stated above, are directed towards no more than limitations that amount to mere instructions to apply the exception, and/or generally link the abstract idea to a particular technological environment or field of use, which even when reevaluated under the considerations of Step 2B of the analysis, do not amount to “significantly more” than the abstract idea.
Dependent claims 2-12 and 15-20 include other limitations, but none of these limitations are deemed significantly more than the abstract idea because, as stated above, the aforementioned dependent claims do not recite any additional elements not already recited in independent claims 1, 13, and 14, and hence do not amount to “significantly more” than the abstract idea.
Additionally, the additional elements (i.e., “receive a first attribute of at least one second medical procedure or at least one second medical environment”), add extra solution activity, which comprises limitations which amount to elements that have been recognized as well-understood, routine, and conventional activity in a particular field as demonstrated by:
Relevant court decisions (See MPEP 2106.05(d)(II)):
Receiving or transmitting data over a network, e.g., using the Internet to gather data, Symantec, 838 F.3d at 1321, 120 USPQ2d at 1362 (utilizing an intermediary computer to forward information); TLI Communications LLC v. AV Auto. LLC, 823 F.3d 607, 610, 118 USPQ2d 1744, 1745 (Fed. Cir. 2016) (using a telephone for image transmission); OIP Techs., Inc., v. Amazon.com, Inc., 788 F.3d 1359, 1363, 115 USPQ2d 1090, 1093 (Fed. Cir. 2015) (sending messages over a network); buySAFE, Inc. v. Google, Inc., 765 F.3d 1350, 1355, 112 USPQ2d 1093, 1096 (Fed. Cir. 2014) (computer receives and sends information over a network); but see DDR Holdings, LLC v. Hotels.com, L.P., 773 F.3d 1245, 1258, 113 USPQ2d 1097, 1106 (Fed. Cir. 2014) (“Unlike the claims in Ultramercial, the claims at issue here specify how interactions with the Internet are manipulated to yield a desired result‐‐a result that overrides the routine and conventional sequence of events ordinarily triggered by the click of a hyperlink.” (emphasis added)).
Thus, taken alone, the additional elements do not amount to significantly more than the abstract idea identified above. Furthermore, looking at the limitations as an ordered combination adds nothing that is not already present when looking at the elements taken individually, and there is no indication that the combination of elements improves the functioning of a computer or improves any other technology, and their collective functions merely provide conventional computer implementation.
Therefore, whether taken individually or as an ordered combination, claims 1-20 are nonetheless rejected under 35 U.S.C. 101 as being directed to non-statutory subject matter.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over Shelton et al (US 20220384019 A1) hereinafter Shelton, in view of Wolf et al. (US 20200268457 A1), hereinafter Wolf.
Regarding claim 1 Shelton teaches a system, comprising: one or more processors, coupled with memory, to (Shelton, [0105]: “The computer system 20063 may comprise a processor and a network interface 20100. The processor may be coupled to a communication module, storage, memory, non-volatile memory, and input/output (I/O) interface via a system bus”): determine multimodal data, the multimodal data comprises two or more of three-dimensional point cloud data, video data, robotic system data, instrument data, or metadata for at least one medical procedure (Shelton, [0081]: “Example HCP monitoring systems 20002, 20003, or 20004 may include a wearable sensing system 20011, an environmental sensing system 20015, a robotic system 20013, one or more intelligent instruments 20014, human interface system 20012, etc.”, [0335]: “Example mobile imaging system may be fluoroscopy, such as a C-Arm X-ray, a 3D imaging machine, and/or an ultrasonic imaging”, and [0336]: “A virtual selective boundary associated with an imaging device described herein may be outlined to a floor of an OR. The virtual selective boundary associated with an imaging device may include vertical boundaries, or three-dimensional boundaries”); determine at least one indication using the multimodal data, wherein the at least one indication comprises at least one of (Shelton, [0010]: “The computing system may communicate an indication to prepare the identified surgical instrument to one or more HCPs inside and/or outside the OR”, [0099]: “The imaging device may employ multi-spectrum monitoring to discriminate topography and underlying structures”, [0111]: “The image processor may employ parallel computing with single instruction, multiple data (SIMD) or multiple instruction, multiple data (MIMD) technologies to increase speed and efficiency”, and [0117]: “As illustrated in FIG. 4 , the surgical hub system 20060 may be expanded by interconnecting multiple network hubs 20061 and/or multiple network switches 20062 with multiple network routers 20066. The modular communication hub 20065 may be contained in a modular control tower configured to receive multiple devices 1 a-1 n/2 a-2 m”): one or more metric values (Shelton, [0041]: “For example, a computing system may monitor HCP motion and interactions, and perform an analysis of HCP motion and interactions throughout a procedure. The computing system may perform an analysis of HCP motion and interactions throughout a procedure to identify improvements for positioning, OR layout, surgical instrument mix, access to the surgical site, and/or the like”, [0197], and [0422]: “The HCP monitoring system may be adapted to perform object detection. The HCP monitoring system may perform object detection to track objects, for example, within an OR. Moving object detection may be performed to recognize the physical movement of a person or an object in a given place or region. By acting segmentation among moving objects and stationary area or region, the moving objects motion could be tracked and thus could be analyzed later”); identify at least one potential cause of efficiency or inefficiency for the at least one indication (Shelton, FIG. 15, FIG. 16, [0012]: “HCP efficiency may be analyzed based on the aggregated OR utilization data, the aggregated OR turnover data, the aggregated HCP reposition data, and/or the aggregated instrument exchange data associated with the plurality of surgical procedures”, and [0226]: “The computing system may obtain surgical resource monitoring data associated with multiple surgical procedures, determine surgical resource efficiency based on the surgical resource monitoring data, and generate an output based on the determined surgical resource efficiency. The output may include but not limited to, a control signal for improving efficiency”) using a mapping between a plurality of potential causes and a plurality of indications (Shelton, [0102]: “The hub 20006 includes a display 20048, an imaging module 20049, a generator module 20050, a communication module 20056, a processor module 20057, a storage array 20058, and an operating-room mapping module 20059” and [0218]: “A mapping or evaluation of the bounds of the operating room may be performed. For example, the surgical hub 20006 may maintain spatial awareness during operation by periodically mapping its operating room, which can be helpful in determining if the surgical hub 20006 has been moved”); and cause a display device to display a message comprising the at least one potential cause (Shelton, [0030]: “ For example, if the computing system blocks a control input by the HCP, such as a scrub nurse, the computing system may send a message to other HCP in the OR, such as a surgeon. The message may be or may include an access control level adjustment message”, [0154]: “The data processing and communication unit 20236 may send a notification message to the HID 20242 indicating that a measurement data value has crossed the threshold value. The notification message may include the measurement data associated with the monitored biomarker”, [0232]: “The recommendation may be sent via a display described herein, a speaker, an earpiece (e.g., ear bud, headset), a message board, etc.”, and [0373]: “The computing system may send an access control level adjustment message to another HCP, such as a surgeon. The access control level adjustment message may inquire whether the access control level associated with the HCP needs an adjustment based on the proximity of the HCP to the operating table”).
Shelton does not teach statistical information.
However, Wolf teaches statistical information (Wolf, [0125]: “The list may also include images (e.g., depicting alternative actions), flow diagrams, statistics (e.g., success rates, failure rates, usage rates, or other statistical information), detailed descriptions, hyperlinks, or other information associated with the alternative possible decisions that may be relevant to the surgeon viewing the playback”, [0186]: “Statistical information may refer to any information that may be useful to analyze multiple surgical procedures together. Statistical information may include, but is not limited to, average values, data trends, standard deviations, variances, correlations, causal relations, test statistics (including t statistics, chi-squared statistics, f statistics, or other forms of test statistics), order statistics (including sample maximum and minimum), graphical representations (e.g., charts, graphs, plots, or other visual or graphical representations), or similar data. As an illustrative example, in embodiments where the user selects an event characteristic including the identity of a particular surgeon, the statistical information may include the average duration in which the surgeon performs the surgical operation (or phase or event of the surgical operation), the rate of adverse or other outcomes the surgeon, the average skill level at which the surgeon performs an intraoperative event, or similar statistical information”, [0199]: “ In some examples, a measure of motion of the surgical tool may be calculated, and the calculated measure of motion may be compared with a selected threshold to distinguish the surgical activity from non-surgical activity. For example, the threshold may be selected based on a type of surgical procedure, based on time of or within the surgical procedure, based on a phase of the surgical procedure, based on parameters determined by analyzing video footage of the surgical procedure, based on parameters determined by analyzing the historical data, and so forth”, [0205]: “In one example, in response to a first relative position in a group of frames, it may be determined that the group of frames includes surgical activity, while in response to a detection of a second relative position in the group of frames, the group of frames may be identified as non surgical activity frames. In another example, the distance between the medical instrument and the anatomical structure may be compared with a selected threshold, and distinguishing the first group of frames from the second group of frames may further be based on a result of the comparison. For example, the threshold may be selected based on the type of the medical instrument, the type of the anatomical structure, the type of the surgical procedure, and so forth”, and [0232]: “In some embodiments, intraoperative events may be identified based on comparing the likelihood to a threshold”).
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify Shelton to incorporate the teachings of Wolf and account for unconventional approaches that efficiently and effectively analyze surgical videos to enable a surgeon to view surgical events, provide decision support, and/or facilitate postoperative activity (Wolf, Abstract and [0004]).
Regarding claim 2 Shelton further teaches the three-dimensional point cloud data is determined using depth acquiring sensors located in or around at least one medical environment in which the at least one medical procedure is performed (Shelton, FIG. 22, [0335]: “Example mobile imaging system may be fluoroscopy, such as a C-Arm X-ray, a 3D imaging machine, and/or an ultrasonic imaging. As illustrated in FIG. 22 , a stable arm of a mobile imaging device may have a selective virtual boundary area when the image device is being used”, and [0336]: “A virtual selective boundary associated with an imaging device described herein may be outlined to a floor of an OR. The virtual selective boundary associated with an imaging device may include vertical boundaries, or three-dimensional boundaries”); and the video data is received from visual image sensors located in or around at least one medical environment in which the at least one medical procedure is performed (Shelton, FIG. 22, [0335]: “Example mobile imaging system may be fluoroscopy, such as a C-Arm X-ray, a 3D imaging machine, and/or an ultrasonic imaging. As illustrated in FIG. 22 , a stable arm of a mobile imaging device may have a selective virtual boundary area when the image device is being used”, and [0336]: “A virtual selective boundary associated with an imaging device described herein may be outlined to a floor of an OR. The virtual selective boundary associated with an imaging device may include vertical boundaries, or three-dimensional boundaries”).
Regarding claim 3 Shelton further teaches the robotic system data comprises at least one of kinematics data of a robotic system, system events data of the robotic system, input received by a console of the robotic system from a user, or timestamps for the kinematics data, the system events data and the received input (Shelton, [0081]: “The robotic system 20013 may include a plurality of devices used for performing a surgical procedure, for example, as further described in FIG. 2”, [0091]: “A robotic system 20034 may be used in the surgical procedure as a part of the HCP monitoring system 20002… The robotic hub 20033 can be used to process the images of the surgical site for subsequent display to the surgeon through the surgeon's console 20036”, and [0420]); the instrument data comprises at least one of instrument imaging data or instrument kinematics data (Shelton, [0081]: “The robotic system 20013 may include a plurality of devices used for performing a surgical procedure, for example, as further described in FIG. 2”, [0091]: “A robotic system 20034 may be used in the surgical procedure as a part of the HCP monitoring system 20002… The robotic hub 20033 can be used to process the images of the surgical site for subsequent display to the surgeon through the surgeon's console 20036”, and [0420]); and the metadata comprises at least one of identifying information of the at least one medical procedures, identifying information of at least one medical environment in which the at least one medical procedure is performed, identifying information of medical staff by which the at least one medical procedure is performed, experience level of the medical staff, patient complexity, patient health parameters or indicators, or identifying information of at least one robotic system or at least one instrument used in the at least one medical procedure (Shelton, [0271]-[0272]: “Aggregated historic surgical procedure data may be used to recommend HCP assignment and team combinations. Historic surgical procedure data may include staffing information (e.g., HCP team combinations, HCP experience level, HCP skill set and/or the like), time, OR turnover, complication rate, patient outcomes and/or surgical resource utilization associated with surgical procedures carried out in the past. For example, surgical outcomes (e.g., complications, success rating(s), surgery duration, or the like) may be correlated with HCP team combinations, HCP experience level, HCP skill set and/or the like. The computing system may generate HCP assignment recommendations, such as recommending specific HCP for a specific procedure, recommending a specific HCP for a specific task, and/or recommending a team combination for a procedure, based on the surgical outcome-HCP correlation data”).
Regarding claim 4 Shelton further teaches each of the at least one indication comprises intermediate information, at least one condition, at least one criteria, or at least one trigger to identify the at least one potential cause (Shelton, [0120]: “At least some of the sensing systems 20069 may be employed to assess physiological conditions of a surgeon operating on a patient or a patient being prepared for a surgical procedure or a patient recovering after a surgical procedure”, [0261]: “The notification for replacement instrument and/or medical supplies may be sent to a device outside of the OR, such as a notification device associated with the instrument supply room. This may enable timely instrument replacement, thus prevent, minimize, or reduce delay in the surgery caused by inadvertent errors”, and [0507]: “The surgical computing system may send an indication of the adjustment parameter(s). The computing system may communicate the adjustment parameter(s) to one or more systems associated with the OR”).
Regarding claim 5 Shelton further teaches the one or more metric values are determined by: determining a duration of each of one or more intervals based on the multimodal data (Shelton, [0017]: “The computing system may predict that the task is to be performed during a time period. An HCP's energy level and/or fatigue level during the time period may be projected based on the biomarker measurement data associated with the HCP, and the availability of the HCP during time period may be determined based on the surgical procedure planning data and the surgical procedure progress data. Whether to assign the task to the HCP may be determined based on the projected fatigue level and the availability of the HCP during the time period”); determining the metric values based on a number of medical staff members in a medical environment during a nonoperative period and motion in the medical environment during the nonoperative period (Shelton, [0041]: “For example, a computing system may monitor HCP motion and interactions, and perform an analysis of HCP motion and interactions throughout a procedure. The computing system may perform an analysis of HCP motion and interactions throughout a procedure to identify improvements for positioning, OR layout, surgical instrument mix, access to the surgical site, and/or the like”, [0197], and [0422]: “The HCP monitoring system may be adapted to perform object detection. The HCP monitoring system may perform object detection to track objects, for example, within an OR. Moving object detection may be performed to recognize the physical movement of a person or an object in a given place or region. By acting segmentation among moving objects and stationary area or region, the moving objects motion could be tracked and thus could be analyzed later”); and determining the metric values based on the determined duration of each of the one or more intervals, wherein the metric values comprise one or more of (Shelton, [0017]: “The computing system may predict that the task is to be performed during a time period. An HCP's energy level and/or fatigue level during the time period may be projected based on the biomarker measurement data associated with the HCP, and the availability of the HCP during time period may be determined based on the surgical procedure planning data and the surgical procedure progress data. Whether to assign the task to the HCP may be determined based on the projected fatigue level and the availability of the HCP during the time period”): a metric value associated with temporal workflow (Shelton, [0197]: “Moving object detection may be performed via background subtraction, frame differencing, temporal differencing, and or optical flow analysis”); a metric value associated with scheduling (Shelton, [0011]: “Resource allocation adjustment(s) may include, but not limited to, healthcare personnel (HCP) assignment adjustment(s), surgery scheduling adjustment(s), surgical instrument allocation adjustment(s), OR layout adjustment(s), and/or medical facility layout adjustment(s), etc.” and [0228]: “As shown in FIG. 12 , planned data may be updated based on surgical monitoring data. Using OR1 35550 as an example, planned surgical data may include, but not limited to, planned surgical procedure 35512 (e.g., procedure steps, scheduled timing and expected duration associated with the procedure steps, and/or the like)”); and a metric value associated with human resources (Shelton, [0271]: “Aggregated historic surgical procedure data may be used to recommend HCP assignment and team combinations. Historic surgical procedure data may include staffing information (e.g., HCP team combinations, HCP experience level, HCP skill set and/or the like), time, OR turnover, complication rate, patient outcomes and/or surgical resource utilization associated with surgical procedures carried out in the past” and [0275]-[0276]).
Regarding claim 6 Shelton teaches the at least one medical procedure comprises a plurality of medical procedures (Shelton, [0011]: “A computing system may be configured to obtain and aggregate surgical monitoring data associated with multiple surgical procedures. The multiple surgical procedures may be associated with one or more operating rooms (ORs), and the surgical monitoring data may be obtained via one or more surgical hubs in the OR(s)”, [0012]: “For example, the surgical monitoring data may include surgical resource monitoring data, HCP monitoring data, surgical instrument utilization data, and/or surgical procedure progression data associated with multiple surgical procedures that may take place in multiple ORs. The surgical monitoring data may include instrument stock and utilization data, OR turnover data and/or cost data associated with the surgical procedures”, and [0080]: “The robotic system 20013 may include a plurality of devices used for performing a surgical procedure, for example, as further described in FIG. 2”); the one or more metric values comprises a plurality of metric values, each of the plurality of metric values is determined for a corresponding one of the plurality of medical procedures (Shelton, [0011]: “A computing system may be configured to obtain and aggregate surgical monitoring data associated with multiple surgical procedures. The multiple surgical procedures may be associated with one or more operating rooms (ORs), and the surgical monitoring data may be obtained via one or more surgical hubs in the OR(s)”, [0012]: “For example, the surgical monitoring data may include surgical resource monitoring data, HCP monitoring data, surgical instrument utilization data, and/or surgical procedure progression data associated with multiple surgical procedures that may take place in multiple ORs. The surgical monitoring data may include instrument stock and utilization data, OR turnover data and/or cost data associated with the surgical procedures”, and [0080]: “The robotic system 20013 may include a plurality of devices used for performing a surgical procedure, for example, as further described in FIG. 2”).
Shelton does not teach each of the at least one indication comprises the statistical information of the plurality of metric values and each of the at least one indication further comprises indication that the statistical information crosses a threshold.
However, Wolf teaches each of the at least one indication comprises the statistical information of the plurality of metric values (Wolf, [0125]: “The list may also include images (e.g., depicting alternative actions), flow diagrams, statistics (e.g., success rates, failure rates, usage rates, or other statistical information), detailed descriptions, hyperlinks, or other information associated with the alternative possible decisions that may be relevant to the surgeon viewing the playback” and [0186]: “Statistical information may refer to any information that may be useful to analyze multiple surgical procedures together. Statistical information may include, but is not limited to, average values, data trends, standard deviations, variances, correlations, causal relations, test statistics (including t statistics, chi-squared statistics, f statistics, or other forms of test statistics), order statistics (including sample maximum and minimum), graphical representations (e.g., charts, graphs, plots, or other visual or graphical representations), or similar data. As an illustrative example, in embodiments where the user selects an event characteristic including the identity of a particular surgeon, the statistical information may include the average duration in which the surgeon performs the surgical operation (or phase or event of the surgical operation), the rate of adverse or other outcomes the surgeon, the average skill level at which the surgeon performs an intraoperative event, or similar statistical information”); and each of the at least one indication further comprises indication that the statistical information crosses a threshold (Wolf, [0125]: “The list may also include images (e.g., depicting alternative actions), flow diagrams, statistics (e.g., success rates, failure rates, usage rates, or other statistical information), detailed descriptions, hyperlinks, or other information associated with the alternative possible decisions that may be relevant to the surgeon viewing the playback”, [0186]: “Statistical information may refer to any information that may be useful to analyze multiple surgical procedures together. Statistical information may include, but is not limited to, average values, data trends, standard deviations, variances, correlations, causal relations, test statistics (including t statistics, chi-squared statistics, f statistics, or other forms of test statistics), order statistics (including sample maximum and minimum), graphical representations (e.g., charts, graphs, plots, or other visual or graphical representations), or similar data. As an illustrative example, in embodiments where the user selects an event characteristic including the identity of a particular surgeon, the statistical information may include the average duration in which the surgeon performs the surgical operation (or phase or event of the surgical operation), the rate of adverse or other outcomes the surgeon, the average skill level at which the surgeon performs an intraoperative event, or similar statistical information”, [0199]: “ In some examples, a measure of motion of the surgical tool may be calculated, and the calculated measure of motion may be compared with a selected threshold to distinguish the surgical activity from non-surgical activity. For example, the threshold may be selected based on a type of surgical procedure, based on time of or within the surgical procedure, based on a phase of the surgical procedure, based on parameters determined by analyzing video footage of the surgical procedure, based on parameters determined by analyzing the historical data, and so forth”, [0205]: “In one example, in response to a first relative position in a group of frames, it may be determined that the group of frames includes surgical activity, while in response to a detection of a second relative position in the group of frames, the group of frames may be identified as non surgical activity frames. In another example, the distance between the medical instrument and the anatomical structure may be compared with a selected threshold, and distinguishing the first group of frames from the second group of frames may further be based on a result of the comparison. For example, the threshold may be selected based on the type of the medical instrument, the type of the anatomical structure, the type of the surgical procedure, and so forth”, and [0232]: “In some embodiments, intraoperative events may be identified based on comparing the likelihood to a threshold”).
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify Shelton to incorporate the teachings of Wolf and account for unconventional approaches that efficiently and effectively analyze surgical videos to enable a surgeon to view surgical events, provide decision support, and/or facilitate postoperative activity (Wolf, Abstract and [0004]).
Regarding claim 7 Shelton further teaches one of the at least one indication comprises one of: a metric value for an adverse event (Shelton, [0261]: “This may enable timely instrument replacement, thus prevent, minimize, or reduce delay in the surgery caused by inadvertent errors” and [0333]: “the selective virtual boundary area may be generated to prevent an HCP from an inadvertent interaction with the imaging device and/or any hazards associated with the imaging device”); a metric value for a headcount in a medical environment (Shelton, [0271]: “Historic surgical procedure data may include staffing information (e.g., HCP team combinations, HCP experience level, HCP skill set and/or the like), time, OR turnover, complication rate, patient outcomes and/or surgical resource utilization associated with surgical procedures carried out in the past”, [0275]: “The computing system may predict staff shortages or overages based on the aggregated data. The aggregated data may be used to optimize HCP work”, and [0276]: “The computing system may, based on the aggregated surgical procedure monitoring data, generate procedure plan(s) based on staff availability, instrument availability, and specialized OR equipment”); a metric value for traffic in the medical environment (Shelton, [0462]: “Referring to FIG. 27 , at section C, example adaptive control of lighting systems in response to healthcare professional traffic in the operating room is depicted… For example, movements within the operating room may correspond to traffic into and out of the operating room, movement of a surgeon into and out of the position to use the robotic surgical instrument, handing off of surgical instruments, etc.”).
Regarding claim 8 Shelton further teaches the one or more processors determines the message using a mapping between a plurality of messages and the plurality of indications (Shelton, [0218]-[0220]: “A mapping or evaluation of the bounds of the operating room may be performed. For example, the surgical hub 20006 may maintain spatial awareness during operation by periodically mapping its operating room, which can be helpful in determining if the surgical hub 20006 has been moved… The operating-room mapping module may be included in the surgical hub 20006 as described herein. The operating-room mapping module may be in operative communication with the surgical hub 20006 as described herein… The operating-room mapping module may map the physical location of device(s) and/or surgical modules that resides within the operating room. This information could be used by the user interface to display a virtual map of the room, enabling the user to more easily identify which modules are present and enabled, as well as their current status”); the message comprises at least a portion of the at least one potential cause and at least a portion of the at least one indication (Shelton, [0218]-[0220]: “A mapping or evaluation of the bounds of the operating room may be performed. For example, the surgical hub 20006 may maintain spatial awareness during operation by periodically mapping its operating room, which can be helpful in determining if the surgical hub 20006 has been moved… The operating-room mapping module may be included in the surgical hub 20006 as described herein. The operating-room mapping module may be in operative communication with the surgical hub 20006 as described herein… The operating-room mapping module may map the physical location of device(s) and/or surgical modules that resides within the operating room. This information could be used by the user interface to display a virtual map of the room, enabling the user to more easily identify which modules are present and enabled, as well as their current status”).
Regarding claim 9 Shelton further teaches the one or more processors to: generate support analytics corresponding to the message, the at least one potential cause, and the at least one indication using the multimodal data used to determine the message, the at least one potential cause, and the at least one indication (Shelton, [0030]: “For example, if the computing system blocks a control input by the HCP, such as a scrub nurse, the computing system may send a message to other HCP in the OR, such as a surgeon. The message may be or may include an access control level adjustment message”, [0083]: “The surgical hub 20006 may be configured to gather measurement data from the one or more sensing systems 20011 and send notifications or control messages to the one or more sensing systems 20011”, and [0154]: “The data processing and communication unit 20236 may send a notification message to the HID 20242 indicating that a measurement data value has crossed the threshold value. The notification message may include the measurement data associated with the monitored biomarker”); and cause the display device to display the support analytics, wherein the support analytics comprises at least one of a charts, a list, a graph, a video rendered using the three-dimensional point cloud data, the video data, the robotic system data, the instrument data, the metadata, the one or more metric values, or the statistical information (Shelton, FIG. 28, [0073]: “FIG. 28 illustrates an example graph of performed surgical procedures plotted based on outcome success and efficiency”, [0457]-[0458]: “In a top graph portion of section A, use of robotic surgical controls over time is depicted, with time extending from left to right. In the bottom graph portion of section A, operation of lighting systems, which may be measured in lumens, across time and in response to parameters received from the surgical computing system is depicted”, [0463]: “In the top graph of section D, ambient light levels measured near a display monitor are graphed over time”, and [0510]-[0512]: “FIG. 28 illustrates an example analysis associated with surgical procedures. Surgical procedures may be plotted on a graph 37600 based on outcome success and efficiency”).
Regarding claim 10 Shelton teaches the one or more processors: determine a score for one or more of the at least one indication, the at least one potential cause, or at least one message (Shelton, [0541]: “Surgical procedures may be compared with historic surgical data to determine a comparison with a baseline score”).
Shelton does not teach cause the message to be displayed using the score for the one or more of the at least one indication, the at least one potential cause, or the at least one message; and wherein the score is determined based on at least one of: a frequency or a number of occurrences of the one or more of the at least one indication, the at least one potential cause, or at least one message; uncertainty, margin of error, or inconsistency in one or more of the multimodal data, the at least one indication, the at least one potential cause, or at least one message; deviations or differences between each of the one or more metric values and a corresponding threshold.
However, Wolf teaches cause the message to be displayed using the score for the one or more of the at least one indication, the at least one potential cause, or the at least one message (Wolf, [0254]: “For example, the process of viewing surgical video clips based on complexity may be accelerated by automatically tagging portions of surgical video with a complexity score, thereby permitting a surgeon to quickly find the frames of interest based on complexity”, [0264]: “The surgical complexity level may be represented in various manners. In some embodiments, the complexity level may be represented as a value. For example, the surgical complexity level may be a value within a range of values corresponding to a scale of complexity (e.g., 0-5, 0-10, 0-100, or any other suitable scale). A percentage or other score may also be used. Generally, a higher value may indicate a higher complexity level, however, in some embodiments, the surgical complexity may be an inverse of the value”, [0563]: “A skill level may be based on a historical performance score, a number of surgeries performed, overall time spent as a surgeon (e.g., a number of years; number of hours spent in surgery), an indication of a level of training, a classification of a surgeon's skill, and/or any other assessment of a surgeon's skill whether derived from manual input, data analysis, or video image analysis”, and [0630]: “In some embodiments, a change in a confidence, probability, and/or score may cause a predicted outcome to drop below a threshold (e.g., a threshold confidence, a threshold probability, a threshold score)”); and wherein the score is determined based on at least one of: a frequency or a number of occurrences of the one or more of the at least one indication, the at least one potential cause, or at least one message (Wolf, [0254]: “For example, the process of viewing surgical video clips based on complexity may be accelerated by automatically tagging portions of surgical video with a complexity score, thereby permitting a surgeon to quickly find the frames of interest based on complexity”, [0264]: “The surgical complexity level may be represented in various manners. In some embodiments, the complexity level may be represented as a value. For example, the surgical complexity level may be a value within a range of values corresponding to a scale of complexity (e.g., 0-5, 0-10, 0-100, or any other suitable scale). A percentage or other score may also be used. Generally, a higher value may indicate a higher complexity level, however, in some embodiments, the surgical complexity may be an inverse of the value”, [0563]: “A skill level may be based on a historical performance score, a number of surgeries performed, overall time spent as a surgeon (e.g., a number of years; number of hours spent in surgery), an indication of a level of training, a classification of a surgeon's skill, and/or any other assessment of a surgeon's skill whether derived from manual input, data analysis, or video image analysis”, and [0630]: “In some embodiments, a change in a confidence, probability, and/or score may cause a predicted outcome to drop below a threshold (e.g., a threshold confidence, a threshold probability, a threshold score)”); uncertainty, margin of error, or inconsistency in one or more of the multimodal data, the at least one indication, the at least one potential cause, or at least one message (Wolf, [0120]: “The icons (or other visual properties) may be used to distinguish between unplanned events and planned events, types of errors (e.g., miscommunication errors, judgment errors, or other forms of errors), specific adverse events that occurred, types of techniques being performed, the surgical phase being performed, locations of intraoperative surgical events (e.g., in the abdominal wall, etc.), a surgeon performing the procedure, an outcome of the surgical procedure, or various other information”, [0161]: “The intraoperative event may include other errors, including technical errors, communication errors, management errors, judgment errors, decision making errors, errors related to medical equipment utilization, miscommunication, and so forth”, [0461]: “If a difference is noted, a computer-based software application may determine the source of the error, may note the error, may send a notification of the error, and/or may automatically correct the error”, and [0693]: “In various embodiments, the event identifying string may be compared with a known historical name of a corresponding intraoperative event to evaluate an associated error for the model. If the error is below a predetermined threshold value, the model may be trained using other input data. Alternatively, if the error is above the threshold value, model parameters may be modified, and a training step may be repeated using the first training data”); deviations or differences between each of the one or more metric values and a corresponding threshold (Wolf, [0186]: “Statistical information may refer to any information that may be useful to analyze multiple surgical procedures together. Statistical information may include, but is not limited to, average values, data trends, standard deviations, variances, correlations, causal relations, test statistics (including t statistics, chi-squared statistics, f statistics, or other forms of test statistics), order statistics (including sample maximum and minimum), graphical representations (e.g., charts, graphs, plots, or other visual or graphical representations), or similar data. As an illustrative example, in embodiments where the user selects an event characteristic including the identity of a particular surgeon, the statistical information may include the average duration in which the surgeon performs the surgical operation (or phase or event of the surgical operation), the rate of adverse or other outcomes the surgeon, the average skill level at which the surgeon performs an intraoperative event, or similar statistical information”). It would have been obvious to one of ordinary skill in the art before the effective filing date to modify Shelton to incorporate the teachings of Wolf and account for unconventional approaches that efficiently and effectively analyze surgical videos to enable a surgeon to view surgical events, provide decision support, and/or facilitate postoperative activity (Wolf, Abstract and [0004]).
Regarding claim 11 Shelton further teaches the one or more processors: receive feedback from a user for the message (Shelton, [0041]: “The parameters may include recommendations, adjustments, feedback, and/or control signals”, [0044]: “The parameters may be indicated in control signals, recommendations, adjustments, and/or feedback”, and [0131]: “One or more of sensors 20225, 20226, 20227, for example, provide real-time feedback to the processor 20222”); in response to receiving the feedback, modify at least one of (Shelton, [0045]: “The surgical display may present the ergonomic adjustment parameters. The system configured to control surgical display(s) may modify the positioning or settings of one or more surgical display(s), for example, based on the ergonomic adjustment parameters” and [0445]: “At 37128, the parameters may be received at the operating room systems and at 37132 the parameters may be used to modify operation of the systems. For example, the parameters may be used by a display system to modify aspects of a display. Biomarker data associated with the patient may be prioritized on the display system”): the mapping between the plurality of potential causes and the plurality of indications (Shelton, [0218]-[0220]: “The operating-room mapping module may map the physical location of device(s) and/or surgical modules that resides within the operating room. This information could be used by the user interface to display a virtual map of the room, enabling the user to more easily identify which modules are present and enabled, as well as their current status”); a mapping between the plurality of potential causes and a plurality of messages (Shelton, [0218]-[0220]: “The operating-room mapping module may map the physical location of device(s) and/or surgical modules that resides within the operating room. This information could be used by the user interface to display a virtual map of the room, enabling the user to more easily identify which modules are present and enabled, as well as their current status”); a score for one or more of the at least one indication, the at least one potential cause, or at least one message (Shelton, [0541]: “Surgical procedures may be compared with historic surgical data to determine a comparison with a baseline score”).
Regarding claim 12 Shelton further teaches a user to whom the message is displayed has a role: at least one of: the mapping between the plurality of potential causes and the plurality of indications or a mapping between the plurality of potential causes and a plurality of messages is selected based on the role of the user (Shelton, [0030]: “The computing system may adjust a control access level of an HCP based on a request from other HCP(s) in the OR. For example, if the computing system blocks a control input by the HCP, such as a scrub nurse, the computing system may send a message to other HCP in the OR, such as a surgeon. The message may be or may include an access control level adjustment message” and [0218]-[0220]: “The operating-room mapping module may map the physical location of device(s) and/or surgical modules that resides within the operating room. This information could be used by the user interface to display a virtual map of the room, enabling the user to more easily identify which modules are present and enabled, as well as their current status”); the multimodal data is selected based on the role of the user (Shelton, [0020]: “The computing system may determine an access authorization for the HCP based on an HCP role during a surgical procedure”); a score for one or more of the at least one indication, the at least one potential cause, or at least one message is updated according to the role of the user (Shelton, [0020]: “The computing system may determine an access authorization for the HCP based on an HCP role during a surgical procedure”); or at least one of the mapping between the plurality of potential causes and the plurality of indications, the mapping between the plurality of potential causes and a plurality of messages, the score for one or more of the at least one indication, the at least one potential cause, or at least one message is modified according to the feedback based on the role of the user (Shelton, [0030]: “The computing system may adjust a control access level of an HCP based on a request from other HCP(s) in the OR. For example, if the computing system blocks a control input by the HCP, such as a scrub nurse, the computing system may send a message to other HCP in the OR, such as a surgeon. The message may be or may include an access control level adjustment message” and [0218]-[0220]: “The operating-room mapping module may map the physical location of device(s) and/or surgical modules that resides within the operating room. This information could be used by the user interface to display a virtual map of the room, enabling the user to more easily identify which modules are present and enabled, as well as their current status”).
Regarding claim 13 Shelton teaches a method, comprising: determining multimodal data, the multimodal data comprises two or more of three-dimensional point cloud data, video data, robotic system data, instrument data, or metadata for at least one medical procedure (Shelton, [0081]: “Example HCP monitoring systems 20002, 20003, or 20004 may include a wearable sensing system 20011, an environmental sensing system 20015, a robotic system 20013, one or more intelligent instruments 20014, human interface system 20012, etc.”, [0335]: “Example mobile imaging system may be fluoroscopy, such as a C-Arm X-ray, a 3D imaging machine, and/or an ultrasonic imaging”, and [0336]: “A virtual selective boundary associated with an imaging device described herein may be outlined to a floor of an OR. The virtual selective boundary associated with an imaging device may include vertical boundaries, or three-dimensional boundaries”); determining at least one indication using the multimodal data, wherein the at least one indication comprises at least one of: (Shelton, [0010]: “The computing system may communicate an indication to prepare the identified surgical instrument to one or more HCPs inside and/or outside the OR”, [0099]: “The imaging device may employ multi-spectrum monitoring to discriminate topography and underlying structures”, [0111]: “The image processor may employ parallel computing with single instruction, multiple data (SIMD) or multiple instruction, multiple data (MIMD) technologies to increase speed and efficiency”, and [0117]: “As illustrated in FIG. 4 , the surgical hub system 20060 may be expanded by interconnecting multiple network hubs 20061 and/or multiple network switches 20062 with multiple network routers 20066. The modular communication hub 20065 may be contained in a modular control tower configured to receive multiple devices 1 a-1 n/2 a-2 m”): one or more metric values (Shelton, [0041]: “For example, a computing system may monitor HCP motion and interactions, and perform an analysis of HCP motion and interactions throughout a procedure. The computing system may perform an analysis of HCP motion and interactions throughout a procedure to identify improvements for positioning, OR layout, surgical instrument mix, access to the surgical site, and/or the like”, [0197], and [0422]: “The HCP monitoring system may be adapted to perform object detection. The HCP monitoring system may perform object detection to track objects, for example, within an OR. Moving object detection may be performed to recognize the physical movement of a person or an object in a given place or region. By acting segmentation among moving objects and stationary area or region, the moving objects motion could be tracked and thus could be analyzed later”); identifying at least one potential cause of efficiency or inefficiency for the at least one indication (Shelton, FIG. 15, FIG. 16, [0012]: “HCP efficiency may be analyzed based on the aggregated OR utilization data, the aggregated OR turnover data, the aggregated HCP reposition data, and/or the aggregated instrument exchange data associated with the plurality of surgical procedures”, and [0226]: “The computing system may obtain surgical resource monitoring data associated with multiple surgical procedures, determine surgical resource efficiency based on the surgical resource monitoring data, and generate an output based on the determined surgical resource efficiency. The output may include but not limited to, a control signal for improving efficiency”) using a mapping between a plurality of potential causes and a plurality of indications (Shelton, [0102]: “The hub 20006 includes a display 20048, an imaging module 20049, a generator module 20050, a communication module 20056, a processor module 20057, a storage array 20058, and an operating-room mapping module 20059” and [0218]: “A mapping or evaluation of the bounds of the operating room may be performed. For example, the surgical hub 20006 may maintain spatial awareness during operation by periodically mapping its operating room, which can be helpful in determining if the surgical hub 20006 has been moved”); and causing a display device to display a message comprising the at least one potential cause (Shelton, [0030]: “ For example, if the computing system blocks a control input by the HCP, such as a scrub nurse, the computing system may send a message to other HCP in the OR, such as a surgeon. The message may be or may include an access control level adjustment message”, [0154]: “The data processing and communication unit 20236 may send a notification message to the HID 20242 indicating that a measurement data value has crossed the threshold value. The notification message may include the measurement data associated with the monitored biomarker”, [0232]: “The recommendation may be sent via a display described herein, a speaker, an earpiece (e.g., ear bud, headset), a message board, etc.”, and [0373]: “The computing system may send an access control level adjustment message to another HCP, such as a surgeon. The access control level adjustment message may inquire whether the access control level associated with the HCP needs an adjustment based on the proximity of the HCP to the operating table”).
Shelton does not teach statistical information.
However, Wolf teaches statistical information (Wolf, [0125]: “The list may also include images (e.g., depicting alternative actions), flow diagrams, statistics (e.g., success rates, failure rates, usage rates, or other statistical information), detailed descriptions, hyperlinks, or other information associated with the alternative possible decisions that may be relevant to the surgeon viewing the playback”, [0186]: “Statistical information may refer to any information that may be useful to analyze multiple surgical procedures together. Statistical information may include, but is not limited to, average values, data trends, standard deviations, variances, correlations, causal relations, test statistics (including t statistics, chi-squared statistics, f statistics, or other forms of test statistics), order statistics (including sample maximum and minimum), graphical representations (e.g., charts, graphs, plots, or other visual or graphical representations), or similar data. As an illustrative example, in embodiments where the user selects an event characteristic including the identity of a particular surgeon, the statistical information may include the average duration in which the surgeon performs the surgical operation (or phase or event of the surgical operation), the rate of adverse or other outcomes the surgeon, the average skill level at which the surgeon performs an intraoperative event, or similar statistical information”, [0199]: “ In some examples, a measure of motion of the surgical tool may be calculated, and the calculated measure of motion may be compared with a selected threshold to distinguish the surgical activity from non-surgical activity. For example, the threshold may be selected based on a type of surgical procedure, based on time of or within the surgical procedure, based on a phase of the surgical procedure, based on parameters determined by analyzing video footage of the surgical procedure, based on parameters determined by analyzing the historical data, and so forth”, [0205]: “In one example, in response to a first relative position in a group of frames, it may be determined that the group of frames includes surgical activity, while in response to a detection of a second relative position in the group of frames, the group of frames may be identified as non surgical activity frames. In another example, the distance between the medical instrument and the anatomical structure may be compared with a selected threshold, and distinguishing the first group of frames from the second group of frames may further be based on a result of the comparison. For example, the threshold may be selected based on the type of the medical instrument, the type of the anatomical structure, the type of the surgical procedure, and so forth”, and [0232]: “In some embodiments, intraoperative events may be identified based on comparing the likelihood to a threshold”).
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify Shelton to incorporate the teachings of Wolf and account for unconventional approaches that efficiently and effectively analyze surgical videos to enable a surgeon to view surgical events, provide decision support, and/or facilitate postoperative activity (Wolf, Abstract and [0004]).
Regarding claim 14 Shelton teaches a system, comprising: one or more processors, coupled with memory, to (Shelton, [0171]: “Each of the central servers 20272 may comprise one or more processors 20273 coupled to suitable memory devices 20274 which can include volatile memory such as random-access memory (RAM) and non-volatile memory such as magnetic storage devices’): determine a dataset of information of a plurality of the first medical procedures in a plurality of first medical environments (Shelton, FIG. 14, [0015]: “For example, a set of repetitive trips may be identified based on the aggregated HCP monitoring data, and surgical tasks associated with the set of repetitive trips may be identified based on the aggregated surgical procedure progression data associated with the surgical procedures”, [0016]: “Surgical procedure planning data associated with multiple surgical procedures may be from multiple surgical hubs and may be used to generate updated surgical procedure plans based on the surgical procedure progression data with the surgical procedures”, and [0059]: “FIG. 14 shows example display summarizing surgical procedures, HCP, surgical device stock status, highlighting deficiencies and recommending remediations across multiple operating rooms”), wherein the dataset comprises: metadata of the plurality of the first medical procedures and the plurality of first medical environments (Shelton, FIG. 14, [0015]: “For example, a set of repetitive trips may be identified based on the aggregated HCP monitoring data, and surgical tasks associated with the set of repetitive trips may be identified based on the aggregated surgical procedure progression data associated with the surgical procedures”, [0016]: “Surgical procedure planning data associated with multiple surgical procedures may be from multiple surgical hubs and may be used to generate updated surgical procedure plans based on the surgical procedure progression data with the surgical procedures”, [0059]: “FIG. 14 shows example display summarizing surgical procedures, HCP, surgical device stock status, highlighting deficiencies and recommending remediations across multiple operating rooms”, and [0117]: “The modular communication hub 20065 may be connected to a display 20068 to display images obtained by some of the devices 1 a-1 n/2 a-2 m, for example during surgical procedures”); a timeline of a plurality of phases and a plurality of tasks within each of the plurality of phases determined for the plurality of the first medical procedures (Shelton, FIG. 8, [0053]: “FIG. 8 illustrates an exemplary timeline of an illustrative surgical procedure indicating adjusting operational parameters of a surgical device based on a surgeon biomarker level”, [0159]: “FIG. 8 illustrates a timeline 20265 of an illustrative surgical procedure and the contextual information that a surgical hub can derive from data received from one or more surgical devices, one or more HCP sensing systems, and/or one or more environmental sensing systems at each step in the surgical procedure”, and [0245]); and three-dimensional point cloud data for the plurality of the first medical procedures and the plurality of first medical environments during at least portions of the plurality of phases and the plurality of tasks within each of the phases (Shelton, [0087]: “The HCP sensing systems 20020 and the environmental sensing systems may be in communication with a surgical hub 20006, which in turn may be in communication with one or more cloud servers 20009 of the cloud computing system 20008, as shown in FIG. 1”, [0100]: “The surgical hub 20006, alone or in communication with the cloud computing system, may use the surgeon biomarker measurement data and/or environmental sensing information to modify the control algorithms of hand-held instruments or the averaging delay of a robotic interface, for example, to minimize tremors”, [0104], and [0113]-[0116]); determine a mapping among data within the dataset (Shelton, [0218]-[0220]: “A mapping or evaluation of the bounds of the operating room may be performed. For example, the surgical hub 20006 may maintain spatial awareness during operation by periodically mapping its operating room, which can be helpful in determining if the surgical hub 20006 has been moved”); receive a first attribute of at least one second medical procedure or at least one second medical environment (Shelton, [0081]: “The environmental sensing system 20015 may include one or more devices, for example, used for measuring one or more environmental attributes, for example, as further described in FIG. 2”, [0087]: “The environmental sensing systems may be used for measuring one or more environmental attributes, for example, HCP position in the surgical theater, HCP movements, ambient noise in the surgical theater, temperature/humidity in the surgical theater, etc.”, [0159]: “The environmental sensing system may include systems for measuring one or more of the environmental attributes, for example, cameras for detecting a surgeon's position/movements/breathing pattern, spatial microphones, for example to measure ambient noise in the surgical theater and/or the tone of voice of a healthcare provider, temperature/humidity of the surroundings, etc.”, and [0169]: “Environmental sensing systems 20267 may be paired with surgical hubs 20270 measuring environmental attributes associated with an HCP”).
Shelton does not teach determine, using the mapping and first attribute, a second attribute of the at least one second medical procedure and the at least one second medical environment.
However, Wolf teaches determine, using the mapping and first attribute, a second attribute of the at least one second medical procedure and the at least one second medical environment (Wolf, [0125]: “The list may also include images (e.g., depicting alternative actions), flow diagrams, statistics (e.g., success rates, failure rates, usage rates, or other statistical information), detailed descriptions, hyperlinks, or other information associated with the alternative possible decisions that may be relevant to the surgeon viewing the playback”, [0186]: “Statistical information may refer to any information that may be useful to analyze multiple surgical procedures together. Statistical information may include, but is not limited to, average values, data trends, standard deviations, variances, correlations, causal relations, test statistics (including t statistics, chi-squared statistics, f statistics, or other forms of test statistics), order statistics (including sample maximum and minimum), graphical representations (e.g., charts, graphs, plots, or other visual or graphical representations), or similar data. As an illustrative example, in embodiments where the user selects an event characteristic including the identity of a particular surgeon, the statistical information may include the average duration in which the surgeon performs the surgical operation (or phase or event of the surgical operation), the rate of adverse or other outcomes the surgeon, the average skill level at which the surgeon performs an intraoperative event, or similar statistical information”, [0199]: “ In some examples, a measure of motion of the surgical tool may be calculated, and the calculated measure of motion may be compared with a selected threshold to distinguish the surgical activity from non-surgical activity. For example, the threshold may be selected based on a type of surgical procedure, based on time of or within the surgical procedure, based on a phase of the surgical procedure, based on parameters determined by analyzing video footage of the surgical procedure, based on parameters determined by analyzing the historical data, and so forth”, [0205]: “In one example, in response to a first relative position in a group of frames, it may be determined that the group of frames includes surgical activity, while in response to a detection of a second relative position in the group of frames, the group of frames may be identified as non surgical activity frames. In another example, the distance between the medical instrument and the anatomical structure may be compared with a selected threshold, and distinguishing the first group of frames from the second group of frames may further be based on a result of the comparison. For example, the threshold may be selected based on the type of the medical instrument, the type of the anatomical structure, the type of the surgical procedure, and so forth”, and [0232]: “In some embodiments, intraoperative events may be identified based on comparing the likelihood to a threshold”).
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify Shelton to incorporate the teachings of Wolf and account for unconventional approaches that efficiently and effectively analyze surgical videos to enable a surgeon to view surgical events, provide decision support, and/or facilitate postoperative activity (Wolf, Abstract and [0004]).
Regarding claim 15 Shelton further teaches the dataset further comprises: cost data of costs of at least one of a plurality of robotic systems used in the plurality of the first medical procedures (Shelton, [0278]: “Such variables may include, but not limited to, HCP availability, HCP stress level, HCP fatigue level, various HCP biomarkers, OR capacity, budget, patient volume, surgical outcomes, time of day, weather, and/or surgical device or surgical equipment availability, across multiple operating rooms. For example, availability for the HCPs in the surgical department of a hospital having multiple operating rooms may be considered when the computing system generates surgical planning recommendations.”), a plurality of instruments used in the plurality of the first medical procedures (Shelton, [0010]: “The surgical computing system may obtain monitored data associated with movement of surgical instruments in the OR, and may determine parameter(s) associated with a recommended surgical instrument within the OR based at least in part on the monitored data” and [0118]: “The cloud computing services can perform a large number of calculations based on the data gathered by smart surgical instruments, robots, sensing systems, and other computerized devices located in the operating theater”), medical staff by which the plurality of the first medical procedures is performed, or the plurality of first medical environments in which the plurality of the first medical procedures is performed (Shelton, [0011]: “The aggregated data may be used to generate resource allocation adjustment(s) for the ORs. Resource allocation adjustment(s) may include, but not limited to, healthcare personnel (HCP) assignment adjustment(s), surgery scheduling adjustment(s), surgical instrument allocation adjustment(s), OR layout adjustment(s), and/or medical facility layout adjustment(s), etc.”, [0043]: “The computing system may determine one or more adjustment parameters associated with HCP actions, HCP staffing, and/or the like”, and [0228]: “As shown in FIG. 12 , planned data may be updated based on surgical monitoring data. Using OR1 35550 as an example, planned surgical data may include, but not limited to, planned surgical procedure 35512 (e.g., procedure steps, scheduled timing and expected duration associated with the procedure steps, and/or the like…”); room layout data indicating layout of the plurality of first medical environments (Shelton, FIG. 30, FIG. 31, [0041]: “The parameters may be associated with positioning, such as operating room (OR) layout, surgical equipment positioning, and/or the like”, [0043]: “The monitored data may include data associated with one or more of HCP motion, HCP interactions, OR layout, surgical equipment location, surgical instrument mix, or surgical site access”, and [0193]: “The surgical hub 5104 can compare the relative positions of the devices to a recommended or anticipated layout for the particular surgical procedure. If there are any discontinuities between the layouts, the surgical hub 5104 can be configured to provide an alert indicating that the current layout for the surgical procedure deviates from the recommended layout”); metric values determined for one or more of: the plurality of phases and the plurality of tasks; the medical staff; the layout of the plurality of first medical environments; or scheduling the medical staff and the plurality of first medical environments (Shelton, [0011]: “The aggregated data may be used to generate resource allocation adjustment(s) for the ORs. Resource allocation adjustment(s) may include, but not limited to, healthcare personnel (HCP) assignment adjustment(s), surgery scheduling adjustment(s), surgical instrument allocation adjustment(s), OR layout adjustment(s), and/or medical facility layout adjustment(s), etc.” and [0228]: “As shown in FIG. 12 , planned data may be updated based on surgical monitoring data. Using OR1 35550 as an example, planned surgical data may include, but not limited to, planned surgical procedure 35512 (e.g., procedure steps, scheduled timing and expected duration associated with the procedure steps, and/or the like)…”).
Regarding claim 16 Shelton further teaches he dataset further comprises metric values comprising at least one of: a metric value associated with temporal workflow of the plurality of first medical environments (Shelton, [0197]: “Moving object detection may be performed via background subtraction, frame differencing, temporal differencing, and or optical flow analysis”); a metric value associated with scheduling for the plurality of first medical environments in the plurality of first medical environments (Shelton, [0011]: “The aggregated data may be used to generate resource allocation adjustment(s) for the ORs. Resource allocation adjustment(s) may include, but not limited to, healthcare personnel (HCP) assignment adjustment(s), surgery scheduling adjustment(s), surgical instrument allocation adjustment(s), OR layout adjustment(s), and/or medical facility layout adjustment(s), etc.” and [0228]: “As shown in FIG. 12 , planned data may be updated based on surgical monitoring data. Using OR1 35550 as an example, planned surgical data may include, but not limited to, planned surgical procedure 35512 (e.g., procedure steps, scheduled timing and expected duration associated with the procedure steps, and/or the like)…”); and a metric value associated with human resources for the plurality of first medical environments in the plurality of first medical environments (Shelton, [0271]: “Aggregated historic surgical procedure data may be used to recommend HCP assignment and team combinations. Historic surgical procedure data may include staffing information (e.g., HCP team combinations, HCP experience level, HCP skill set and/or the like), time, OR turnover, complication rate, patient outcomes and/or surgical resource utilization associated with surgical procedures carried out in the past” and [0275]-[0276]).
Regarding claim 17 Shelton teaches the first attribute is received from a user via a user interface (UI); the second attribute is displayed to the user via the UI (Shelton, [0220]: “The operating-room mapping module may map the physical location of device(s) and/or surgical modules that resides within the operating room. This information could be used by the user interface to display a virtual map of the room, enabling the user to more easily identify which modules are present and enabled, as well as their current status”); and the one or more processor to: receive updated first attribute (Shelton, FIG. 12, [0016]: “Surgical procedure planning data associated with multiple surgical procedures may be from multiple surgical hubs and may be used to generate updated surgical procedure plans based on the surgical procedure progression data with the surgical procedures… Surgical instrument allocation data associated with the surgical procedures may be obtained and, predicted or projected surgical instrument utilization data associated with the surgical procedures may be determined based on the aggregated surgical instrument utilization monitoring data and the updated surgical procedure plans”, [0180]: “he data signals of the plurality of sensors 20286 may be stored within or be used to update the adapter data stored within the adapter identification device 20284”, and [0229]-[0230]).
Shelton does not teach the second attribute comprises a two-dimensional image of a layout of the at least one second medical environment.
However, Wolf teaches the second attribute comprises a two-dimensional image of a layout of the at least one second medical environment (Wolf, [0082]: “analyzing image data (for example by the methods, steps and modules described herein) may comprise analyzing the image data to obtain a preprocessed image data, and subsequently analyzing the image data and/or the preprocessed image data to obtain the desired outcome. Some non-limiting examples of such image data may include one or more images, videos, frames, footages, 2D image data, 3D image data, and so forth”).
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify Shelton to incorporate the teachings of Wolf and account for unconventional approaches that efficiently and effectively analyze surgical videos to enable a surgeon to view surgical events, provide decision support, and/or facilitate postoperative activity (Wolf, Abstract and [0004]).
Regarding claim 18 Shelton teaches the first attribute comprises: at least one existing attribute of at least one existing medical procedure and at least one existing medical environment (Shelton, [0010]: “The surgical computing system may obtain monitored data associated with movement of surgical instruments in the OR, and may determine parameter(s) associated with a recommended surgical instrument within the OR based at least in part on the monitored data”, [0034]-[0036]: “ The surgical computing system may obtain the data, which may be referred to as monitored data, and may determine, based upon the monitored data, parameters for controlling various systems associated with the operating room. The surgical computing system may communicate the parameters to the systems which may modify their operation based upon the received parameters… The surgical computing system, based upon the monitored data and in view of the surgical event being undertaken, may determine parameters to modify operation of operating room systems”, and [0039]: “The surgical computing system may be configured to analyze monitored data and may determine, based upon the monitored data, parameters for adaptively controlling operating room systems to implement room conditions associated with a particular healthcare provider who may be participating in the ongoing surgical procedure”); at least one goal attribute of at least one simulated medical procedure and at least one simulated medical environment (Shelton, [0540]: “As those skilled in art may appreciate, various optimization models, simulation models and/or process models may be used in the determination. For example, the surgical computing system may determine adjustments parameters for a surgical instrument mix, surgical access location, OR layout, and/or HCP staffing based on data and/or metadata associated with previous procedures”).
Shelton does not teach the at least one second medical procedure comprises the at least one existing medical procedure and the at least one simulated medical procedure and the at least one second medical environment comprises the at least one existing medical environment and the at least one simulated medical environment.
However, Wolf teaches the at least one second medical procedure comprises the at least one existing medical procedure and the at least one simulated medical procedure (Wolf, [0125]: “The list may also include images (e.g., depicting alternative actions), flow diagrams, statistics (e.g., success rates, failure rates, usage rates, or other statistical information), detailed descriptions, hyperlinks, or other information associated with the alternative possible decisions that may be relevant to the surgeon viewing the playback”, [0186]: “Statistical information may refer to any information that may be useful to analyze multiple surgical procedures together. Statistical information may include, but is not limited to, average values, data trends, standard deviations, variances, correlations, causal relations, test statistics (including t statistics, chi-squared statistics, f statistics, or other forms of test statistics), order statistics (including sample maximum and minimum), graphical representations (e.g., charts, graphs, plots, or other visual or graphical representations), or similar data. As an illustrative example, in embodiments where the user selects an event characteristic including the identity of a particular surgeon, the statistical information may include the average duration in which the surgeon performs the surgical operation (or phase or event of the surgical operation), the rate of adverse or other outcomes the surgeon, the average skill level at which the surgeon performs an intraoperative event, or similar statistical information”, [0199]: “ In some examples, a measure of motion of the surgical tool may be calculated, and the calculated measure of motion may be compared with a selected threshold to distinguish the surgical activity from non-surgical activity. For example, the threshold may be selected based on a type of surgical procedure, based on time of or within the surgical procedure, based on a phase of the surgical procedure, based on parameters determined by analyzing video footage of the surgical procedure, based on parameters determined by analyzing the historical data, and so forth”, [0205]: “In one example, in response to a first relative position in a group of frames, it may be determined that the group of frames includes surgical activity, while in response to a detection of a second relative position in the group of frames, the group of frames may be identified as non surgical activity frames. In another example, the distance between the medical instrument and the anatomical structure may be compared with a selected threshold, and distinguishing the first group of frames from the second group of frames may further be based on a result of the comparison. For example, the threshold may be selected based on the type of the medical instrument, the type of the anatomical structure, the type of the surgical procedure, and so forth”, and [0232]: “In some embodiments, intraoperative events may be identified based on comparing the likelihood to a threshold”); and the at least one second medical environment comprises the at least one existing medical environment and the at least one simulated medical environment (Wolf, [0125]: “The list may also include images (e.g., depicting alternative actions), flow diagrams, statistics (e.g., success rates, failure rates, usage rates, or other statistical information), detailed descriptions, hyperlinks, or other information associated with the alternative possible decisions that may be relevant to the surgeon viewing the playback”, [0186]: “Statistical information may refer to any information that may be useful to analyze multiple surgical procedures together. Statistical information may include, but is not limited to, average values, data trends, standard deviations, variances, correlations, causal relations, test statistics (including t statistics, chi-squared statistics, f statistics, or other forms of test statistics), order statistics (including sample maximum and minimum), graphical representations (e.g., charts, graphs, plots, or other visual or graphical representations), or similar data. As an illustrative example, in embodiments where the user selects an event characteristic including the identity of a particular surgeon, the statistical information may include the average duration in which the surgeon performs the surgical operation (or phase or event of the surgical operation), the rate of adverse or other outcomes the surgeon, the average skill level at which the surgeon performs an intraoperative event, or similar statistical information”, [0199]: “ In some examples, a measure of motion of the surgical tool may be calculated, and the calculated measure of motion may be compared with a selected threshold to distinguish the surgical activity from non-surgical activity. For example, the threshold may be selected based on a type of surgical procedure, based on time of or within the surgical procedure, based on a phase of the surgical procedure, based on parameters determined by analyzing video footage of the surgical procedure, based on parameters determined by analyzing the historical data, and so forth”, [0205]: “In one example, in response to a first relative position in a group of frames, it may be determined that the group of frames includes surgical activity, while in response to a detection of a second relative position in the group of frames, the group of frames may be identified as non surgical activity frames. In another example, the distance between the medical instrument and the anatomical structure may be compared with a selected threshold, and distinguishing the first group of frames from the second group of frames may further be based on a result of the comparison. For example, the threshold may be selected based on the type of the medical instrument, the type of the anatomical structure, the type of the surgical procedure, and so forth”, and [0232]: “In some embodiments, intraoperative events may be identified based on comparing the likelihood to a threshold”).
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify Shelton to incorporate the teachings of Wolf and account for unconventional approaches that efficiently and effectively analyze surgical videos to enable a surgeon to view surgical events, provide decision support, and/or facilitate postoperative activity (Wolf, Abstract and [0004]).
Regarding claim 19 Shelton teaches the first attribute comprises at least one of a budget, a number of robotic systems, types of robotic systems, a number of the at least one second medical environment, a size of the at least one second medical environment, at least one procedure type of the at least one second medical procedure, at least one modality of the at least one second medical procedure, or a number of medical staff member (Shelton, [0034]-[0036]: “The surgical computing system may obtain the data, which may be referred to as monitored data, and may determine, based upon the monitored data, parameters for controlling various systems associated with the operating room. The surgical computing system may communicate the parameters to the systems which may modify their operation based upon the received parameters” and [0278]: “Such variables may include, but not limited to, HCP availability, HCP stress level, HCP fatigue level, various HCP biomarkers, OR capacity, budget, patient volume, surgical outcomes, time of day, weather, and/or surgical device or surgical equipment availability, across multiple operating rooms”).
Shelton does not teach the second attribute comprises at least one of a number of the at least one second medical environment, layout of the at least one second medical environment comprising at least one robotic system, a schedule of the at least one second medical environment, a schedule of a medical staff.
However, Wolf teaches the second attribute comprises at least one of a number of the at least one second medical environment, layout of the at least one second medical environment comprising at least one robotic system, a schedule of the at least one second medical environment, a schedule of a medical staff (Wolf, [0015]: “Some embodiments may further include analyzing frames of the surgical footage to identify in a second set of frames a medical tool, the anatomical structure, and an interaction between the medical tool and the anatomical structure. The disclosed embodiments may include accessing second historical data, the second historical data being based on an analysis of a second frame data captured from a second group of prior surgical procedures. The second set of frames may be analyzed using the second historical data and using the identified interaction to determine a second surgical complexity level associated with the second set of frames”, [0098]: “It should be noted camera 311A may have a first set of parameters and camera 311B may have a second set of parameters that is different from the first set of parameters, and these parameters may be selected using appropriate input controls. Similarly, light source 313A may have a first set of parameters and light source 313B may have a second set of parameters that is different from the first set of parameters, and these parameters may be selected using appropriate input controls”, and [0274]).
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify Shelton to incorporate the teachings of Wolf and account for unconventional approaches that efficiently and effectively analyze surgical videos to enable a surgeon to view surgical events, provide decision support, and/or facilitate postoperative activity (Wolf, Abstract and [0004]).
Regarding claim 20 Shelton does not teach the one or more processor: determine a plurality of candidate second attributes; and selectively display the candidate second attributes according to respective metric values of the candidate second attributes.
However, Wolf teaches the one or more processor: determine a plurality of candidate second attributes (Wolf, [0015]: “Some embodiments may further include analyzing frames of the surgical footage to identify in a second set of frames a medical tool, the anatomical structure, and an interaction between the medical tool and the anatomical structure. The disclosed embodiments may include accessing second historical data, the second historical data being based on an analysis of a second frame data captured from a second group of prior surgical procedures. The second set of frames may be analyzed using the second historical data and using the identified interaction to determine a second surgical complexity level associated with the second set of frames”, [0098]: “It should be noted camera 311A may have a first set of parameters and camera 311B may have a second set of parameters that is different from the first set of parameters, and these parameters may be selected using appropriate input controls. Similarly, light source 313A may have a first set of parameters and light source 313B may have a second set of parameters that is different from the first set of parameters, and these parameters may be selected using appropriate input controls”, [0274], and [0641]: “Thus, if a predicted outcome change impacts a duration of surgery, a surgical schedule may be automatically updated to inform succeeding medical staff of change in operating room schedule. The update may be automatically displayed on an electronic operating room scheduling board”); and selectively display the candidate second attributes according to respective metric values of the candidate second attributes (Wolf, [0015]: “Some embodiments may further include analyzing frames of the surgical footage to identify in a second set of frames a medical tool, the anatomical structure, and an interaction between the medical tool and the anatomical structure. The disclosed embodiments may include accessing second historical data, the second historical data being based on an analysis of a second frame data captured from a second group of prior surgical procedures. The second set of frames may be analyzed using the second historical data and using the identified interaction to determine a second surgical complexity level associated with the second set of frames”, [0098]: “It should be noted camera 311A may have a first set of parameters and camera 311B may have a second set of parameters that is different from the first set of parameters, and these parameters may be selected using appropriate input controls. Similarly, light source 313A may have a first set of parameters and light source 313B may have a second set of parameters that is different from the first set of parameters, and these parameters may be selected using appropriate input controls”, [0274], and [0641]: “Thus, if a predicted outcome change impacts a duration of surgery, a surgical schedule may be automatically updated to inform succeeding medical staff of change in operating room schedule. The update may be automatically displayed on an electronic operating room scheduling board”).
It would have been obvious to one of ordinary skill in the art before the effective filing date to modify Shelton to incorporate the teachings of Wolf and account for unconventional approaches that efficiently and effectively analyze surgical videos to enable a surgeon to view surgical events, provide decision support, and/or facilitate postoperative activity (Wolf, Abstract and [0004]).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to RACHAEL SOJIN STONE whose telephone number is (571)272-8798. The examiner can normally be reached Monday-Friday 7 AM - 7 PM (EST).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Peter Choi can be reached at (469) 295-9171. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/R.S.S./Examiner, Art Unit 3681
/PETER H CHOI/Supervisory Patent Examiner, Art Unit 3681
Prosecution Insights