DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
All Information Disclosure Statements filed on or before 08/04/2026 have been considered by examiner.
Response to Amendment
Applicant’s amendments overcome the previously held 112(b) rejection, but do not overcome the previously held 101 rejection. Claims 1-15, 17-35, 37-55, and 57-60 remain pending. Claims 16, 36, and 56 have been cancelled. Claims 1, 13, 21, 33, 41, and 53 have been amended.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1,6,11,15,17, 20-21, 26, 31, 35, 37, 40-41, 46, 51, 55, 57, and 60 are rejected under 35 U.S.C. 101 because the claimed invention is directed to abstract mental process without significantly more.
Claim 1 recites, “a system (generic computer) configured for determining one or more characteristics of surgical related items and/or procedure related items present at a table during preoperative,intraoperative, and/or postoperative settings and/or simulated preoperative, intraoperative, and/or postoperative settings (mental process, a person can view items in on a table different environments and determine characteristics), comprising:one or more computer processors (generic computer); a memory (generic computer)configured to store instructions that are executable by said one or more computer processors (generic computer), wherein said one or more computer processors (generic computer) are configured to execute the instructions to: receive settings image data corresponding with the preoperative, intraoperative, and/or postoperative settings and/or simulated preoperative, intraoperative, and/or postoperative settings (mental process, a person can view an environment) at said table, wherein the settings image data is captured by one or more cameras positioned to provide an overhead-view of said table (extra solution activity), and wherein the settings image data comprises a series of images captured at intervals at a rate of multiple times per minute or per second during preoperative, intraoperative, and/or postoperative periods (mental process, a person can look at the environment in the given intervals); run a trained computer vision model (merely applied) on the received settings image data to identify and label the surgical related items and/or procedure related items present on said table, wherein the trained computer vision model is generated by a machine learning algorithm trained on annotated image data of surgical related items and/or procedure related items (extra solution activity and encompasses mathematical concepts), and wherein running the trained computer vision model comprises generating, for each image of the series of images, (i) a detection of each said item that is actively visible on said table in that image (mental process a person can visually see if each item is on the table), and (ii) a confidence score associated with each said detection (mental process a person can judge or evaluate their own opinion); interpret the surgical related items and/or procedure related items through tracking and analyzing said identified and labeled items across the series of images, to determine said one or more characteristics (mental process a person can observe, track, and analyze images visually to determine characteristics), wherein said tracking and analyzing comprises object identification and comprises: (a) determining, for each said item, a per-image detection state across the series of images, wherein an item that is actively visible on said table in a given image is detected in that image and an item that is not actively visible on said table in that image is not detected in that image (mental process a person can visually detect whether an item is on a table or not); (b) determining, for each said item, whether said item has left a field-of-view of said one or more cameras and whether said item has moved to a different spot on said table, based on the per-image detection states and on positions of said item across the series of images (mental process a person can visually detect whether an item is outside the field of view of a camera or if an item has been moved based on a previously noted position of an item); (c) aggregating the per-image detection states and the determinations of (b) across the series of images captured during said preoperative, intraoperative, and/or postoperative periods to generate aggregate data for said preoperative, intraoperative, and/or postoperative settings and/or simulated preoperative, intraoperative, and/or postoperative settings (mental process a person can note the different detected states and determinations made for the images); and (d) determining from said aggregate data, for each said item, (i) that said item was placed on said table during said preoperative, intraoperative, and/or postoperative periods (mental process a person can see or note if an item has been placed on the table), and (ii) whether said item was interacted with during preoperative, intraoperative, and/or postoperative periods, wherein said item is determined to have been interacted with based on at least one of (A) said item having left said field-of-view, or (B) said item having moved to a different spot on said table, (mental process, a person can collect and combine knowledge gathered from observations, object tracking, and various determinations, and judge whether an item has been interacted with based on that knowledge)and wherein an item determined to have been placed on said table and not interacted with is determined to have remained unused throughout said preoperative, intraoperative, and/or postoperative periods (mental process, a person can collect and combine knowledge gathered from observations, object tracking, and various determinations, and judge whether an item has been interacted with based on that knowledge and then form an opinion whether or not an item has been used); and transmit said one or more determined characteristics to a secondary source (mental process, a person can communicate findings to someone else or write them down).”
Claim 6 recites, “The system of claim 1 (mental process), wherein said trained computer vision model is generated on preliminary image data using a machine learning algorithm. (well-known and understood extra solution activity)”
Claim 11 recites, “The system of claim 1 (mental process), wherein said one or more computer processors are configured to execute the instructions for said tracking and analyzing at one or more of the following: one or more databases; cloud infrastructure; and edge-computing. (merely applied)”
Claim 15 recites, “The system of claim 1 (mental process), wherein said determined one or more characteristics includes any combination of one or more of the following:identification of the one or more of the surgical related items and/or procedure related items (mental process, a person can identify an item);usage or non-usage status of the one or more of the surgical related items and/or procedure related items (mental process, a person can identify whether a not an item has been used);opened or unopened status of the one or more of the surgical related items and/or procedure related items (mental process, a person can identify whether a not an item has been opened); moved or non-moved status of the one or more of the surgical related items and/or procedure related items (mental process, a person can identify whether a not an item has been moved); single-use or reusable status of the one or more of the surgical related items and/or procedure related items (mental process, a person can identify whether a not an item is reusable); or association of clinical events, logistical events, or operational events (mental process, a person can determine an association of items to events).”
Claim 17 recites, “The system of claim 1 (mental process), wherein said one or more computer processors (generic computer) are further configured to, based on said determined one or more characteristics, execute the instructions to:determine an actionable output to reduce unnecessary waste of the surgical related items for use in the preoperative, intraoperative, and/or postoperative settings and/or the simulated preoperative, intraoperative phase, and/or postoperative settings (mental process, a person can determine what actions to preform based on the characteristics of items in any environment); determine an actionable output to reorganize the surgical related items for use in the preoperative, intraoperative, and/or postoperative settings or the simulated preoperative, intraoperative, and/or postoperative settings (mental process, a person can determine what actions to preform based on the characteristics of items in any environment); determine an actionable output to reduce supply, storage, sterilization and disposal costs associated with use of the surgical related items for use in the preoperative, intraoperative, and/or postoperative settings and/or the simulated preoperative, intraoperative, and/or postoperative settings (mental process, a person can determine what actions to preform based on the characteristics of items in any environment);determine an actionable output to reduce garbage and unnecessary re- sterilization associated with use of the surgical related items for use in the preoperative, intraoperative, and/or postoperative settings and/or the simulated preoperative, intraoperative, and/or postoperative settings (mental process, a person can determine what actions to preform based on the characteristics of items in any environment);determine an actionable output to streamline setup of the surgical related items for use in the preoperative, intraoperative, and/or postoperative settings and/or the simulated preoperative, intraoperative, and/or postoperative settings(mental process, a person can determine what actions to preform based on the characteristics of items in any environment);determine an actionable output to improve efficiency of using the surgical related items for use in the preoperative, intraoperative, and/or postoperative settings and/or the simulated preoperative, intraoperative, and/or postoperative settings (mental process, a person can determine what actions to preform based on the characteristics of items in any environment); determine an actionable output to identify, rank, and/or recognize level of efficiency of surgeons or clinicians (mental process, a person can determine what actions to preform based on the characteristics of items in any environment); and/or determine an actionable output to improve the level of efficiency of using the surgical related items and/or procedure related items that are sterilized (mental process, a person can determine what actions to preform based on the characteristics of items in any environment).”
Claim 20 recites, “The system of claim 1 (mental processes), wherein said settings image data comprises three dimensional renderings or representation of information of the surgical related items and/or procedure related items in the preoperative, intraoperative, and/orpostoperative settings and/or simulated preoperative, intraoperative, and/or postoperative settings. (mental process, people see in three dimensions and can view three dimensional renderings)”
Claim 21 is substantially similar to claim 1, being the method version of claim 1 and the 101 rejection, with regards to claim 1, is applied mutatis mutandis.
Claim 26 is substantially similar to claim 6, being the method version of claim 6 and the 101 rejection, with regards to claim 6, is applied mutatis mutandis.
Claim 31 is substantially similar to claim 11, being the method version of claim 11 and the 101 rejection, with regards to claim 11, is applied mutatis mutandis.
Claim 35 is substantially similar to claim 15, being the method version of claim 15 and the 101 rejection, with regards to claim 15, is applied mutatis mutandis.
Claim 37 is substantially similar to claim 17, being the method version of claim 17 and the 101 rejection, with regards to claim 17, is applied mutatis mutandis.
Claim 40 is substantially similar to claim 20, being the method version of claim 20 and the 101 rejection, with regards to claim 20, is applied mutatis mutandis.
Claim 41 is substantially similar to claim 1, being directed towards a non-transitory computer readable medium storing instructions that when executed by one or more processors, cause the one or more processors to perform the processes of claim 1 and the 101 rejection, with regards to claim 1, is applied mutatis mutandis.
Claim 46 is substantially similar to claim 6, being directed towards a non-transitory computer readable medium storing instructions that when executed by one or more processors, cause the one or more processors to perform the processes of claim 6 and the 101 rejection, with regards to claim 6, is applied mutatis mutandis.
Claim 51 is substantially similar to claim 11, being directed towards a non-transitory computer readable medium storing instructions that when executed by one or more processors, cause the one or more processors to perform the processes of claim 11 and the 101 rejection, with regards to claim 11, is applied mutatis mutandis.
Claim 55 is substantially similar to claim 15, being directed towards a non-transitory computer readable medium storing instructions that when executed by one or more processors, cause the one or more processors to perform the processes of claim 15 and the 101 rejection, with regards to claim 15, is applied mutatis mutandis.
Claim 57 is substantially similar to claim 17, being directed towards a non-transitory computer readable medium storing instructions that when executed by one or more processors, cause the one or more processors to perform the processes of claim 17 and the 101 rejection, with regards to claim 17, is applied mutatis mutandis.
Claim 60 is substantially similar to claim 20, being directed towards a non-transitory computer readable medium storing instructions that when executed by one or more processors, cause the one or more processors to perform the processes of claim 20 and the 101 rejection, with regards to claim 20, is applied mutatis mutandis.
Amended Independent claims (claims 1, 21, 41) concern systems and methods [step 1], but are directed towards abstract ideas (mental processes) [step 2A Prong 1]. These claims feature additional limitations however these limitations do not integrate the judicial exception into a practical application [step 2A Prong 2]. Amended claims 1, 21, and 41 mention a processor, memory, and a computer vision model these inclusions are made generally and are merely applied or used as tools to perform the abstract idea. Similarly claims 11, 31, and 51 mention databases, cloud infrastructure, and edge-computing generally, merely applying technology as tools to perform the abstract idea. Additional limitations in claims 15, 17, 20, 35, 37,40, 55, 57, and 60 list additional mental processes. Claims 1, 6, 21, 26, 41 and 46 feature an additional limitation involving generating a trained computer vision model based on image data using a machine learning algorithm. This extra solution limitation is well known in the art (see Khan, Asharul Islam, and Salim Al-Habsi. "Machine learning in computer vision." Procedia Computer Science 167 (2020): 1444-1451.), and only tangentially related to the claimed invention. Amended claims 1, 21, and 41 also include a limitation involving receiving images captured using a camera. This extra solution activity is again insignificant (only tangentially related to the claimed invention), and amounts to mere data gathering.
Furthermore, the above claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception [step 2B] because they do not include improvements to the functioning of a computer, or any other technology or technical field, application or use any particular machines (no mentions of specific components or elements), or applications or use of technology beyond generally linking the abstract idea to a programmed computer environment. Additionally, claims 1,6, 21, 26, 41 and 46 feature an additional limitation involving generating a trained computer vision model based on image data using a machine learning algorithm. This extra solution limitation is well known, routine, and conventional in the art (see Khan, Asharul Islam, and Salim Al-Habsi. "Machine learning in computer vision." Procedia Computer Science 167 (2020): 1444-1451.), and as stated above, only tangibly related to the claimed invention.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Claims 1-9, 11-15, 18-19, 21-29, 31-35, 38-39, 41-49, 51-55, and 58-59 are rejected under 35 U.S.C. 103 as being unpatentable over Fine (US 20210327567 A1) in view of Hill (WO 2019173699 A1).
With respect to claim 1, Fine teaches a system configured for determining one or more characteristics of surgical related items and/or procedure related items present at a table during, intraoperative settings (“Building off of the success of the Phase 1 work, the AI/ML model for the instrument recognition engine 204 was optimized with the image recognition algorithms for complex image scenarios unique to the OR, and integrated within the existing ExplORer Live™ platform. One objective of Phase 2 was to deliver fully automated, role-specific workflow advancement within the context of an active OR” paragraph 0051) and simulated intraoperative settings (“The instrument recognition engine 204 was then trained by using an instrument preparation station typical of most ORs (i.e., a mayo stand) and then a video camera was mounted from above to capture the entire set of instruments in a single view. For model testing, investigators dressed in surgical scrubs and personal protective equipment (PPE) proceeded to grab and replace instruments as if using them during a surgical procedure. The RetinaNet algorithm was applied to the live video feed, detecting instruments (identified by the bounding box) and classifying instruments by identification numbers for each instrument present within the field of view (See FIGS. 6-7).” paragraph 0049), comprising:
one or more computer processors (“The disclosed embodiments may also be implemented as instructions carried by or stored on a transitory or non-transitory machine-readable (e.g., computer-readable) storage medium, which may be read and executed by one or more processors.” Paragraph 0026);
a memory configured to store instructions that are executable by said one or more computer processors (“The disclosed embodiments may also be implemented as instructions carried by or stored on a transitory or non-transitory machine-readable (e.g., computer-readable) storage medium, which may be read and executed by one or more processors.” Paragraph 0026), wherein said one or more computer processors are configured to execute the instructions to:
receive settings image data corresponding with the intraoperative (“Building off of the success of the Phase 1 work, the AI/ML model for the instrument recognition engine 204 was optimized with the image recognition algorithms for complex image scenarios unique to the OR, and integrated within the existing ExplORer Live™ platform. One objective of Phase 2 was to deliver fully automated, role-specific workflow advancement within the context of an active OR” paragraph 0051), and/or simulated intraoperative settings (“The instrument recognition engine 204 was then trained by using an instrument preparation station typical of most ORs (i.e., a mayo stand) and then a video camera was mounted from above to capture the entire set of instruments in a single view. For model testing, investigators dressed in surgical scrubs and personal protective equipment (PPE) proceeded to grab and replace instruments as if using them during a surgical procedure. The RetinaNet algorithm was applied to the live video feed, detecting instruments (identified by the bounding box) and classifying instruments by identification numbers for each instrument present within the field of view (See FIGS. 6-7).” paragraph 0049) at said table, wherein the settings image data is captured by one or more cameras positioned to provide an overhead-view of said table (“The instrument recognition engine 204 was then trained by using an instrument preparation station typical of most ORs (i.e., a mayo stand) and then a video camera was mounted from above to capture the entire set of instruments in a single view. For model testing, investigators dressed in surgical scrubs and personal protective equipment (PPE) proceeded to grab and replace instruments as if using them during a surgical procedure. The RetinaNet algorithm was applied to the live video feed, detecting instruments (identified by the bounding box) and classifying instruments by identification numbers for each instrument present within the field of view (See FIGS. 6-7).” paragraph 0049 and “Additionally, although a single video feed 104 is shown in FIG. 1, there could be multiple cameras with different camera angles feeding video from the OR depending on the circumstances.” Paragraph 0034), and wherein the settings image data comprises a series of images captured at intervals at a rate of multiple times per minute or per second during intraoperative periods (“Additionally, although a single video feed 104 is shown in FIG. 1, there could be multiple cameras with different camera angles feeding video from the OR depending on the circumstances” paragraph 0034);
run a trained computer vision model on the received settings image data to identify and label the surgical related items and/or procedure related items present on said table (“Once compiled, an artificial intelligence (AI) algorithm was developed for the instrument recognition engine 204 that could: (1) recognize and identify specific instruments; and (2) define instrument-use events based on when objects enter or leave the camera's field of view.” paragraph 0048) , wherein the trained computer vision model is generated by a machine learning algorithm trained on annotated image data of surgical related items and/or procedure related items (“In developing the instrument recognition engine 204, Phase 1-equivalent work was performed to demonstrate the potential to detect instrument use events from real-time video feeds of instrument trays on moveable carts (i.e., mayo stands), mimicking the OR environment. For this phase of the project, a small-scale version of the instrument recognition engine 204 was built. A set of 20 commonly-used surgical instruments were used for this phase. (See FIG. 4). This initial set of instruments was designed to include both: (1) tools with similar but not identical form factors; and (2) tools with identical form factors but variations in size. Each instrument was imaged in various orientations (25° rotational shift between images; varying degrees of opened/closed, as applicable; ˜20 images per instrument) (See FIG. 5). Within each image, a bounding box around the instrument was defined, and labeled each image” paragraph 0047), and wherein running the trained computer vision model comprises generating, for each image of the series of images, (i) a detection of each said item that is actively visible on said table in that image (“In the example shown in FIG. 6, there is shown a live-feed video in which the instrument recognition engine 204 detected, identified, and added bounding boxes to a surgical instrument added to a tray. In the example shown in FIG. 7, there is shown a live-feed video in which the instrument recognition engine 204 detected, identified, and added bounding boxes to a plurality of a surgical instruments added to a tray simulating a realistic scenario in the OR.” Paragraph 0049), and (ii) a confidence score associated with each said detection (“RetinaNet is the algorithm used to develop the initial prototype model, which is a single, unified network comprising one backbone network and two task-specific subnetworks. The backbone network computes a convolutional feature map of the entire image; of the two subnetworks: a Focal Loss algorithm limits cross-entropy loss (i.e., improves accuracy) by classifying the output of the backbone network; and a Feature Pyramid Network (FPN) performs convolution bounding box regression.” Paragraph 0048);
interpret the surgical related items and/or procedure related items through tracking and analyzing said identified and labeled items across the series of images to determine said one or more characteristics of the surgical related items and/or procedure related items (“The RetinaNet algorithm was applied to the live video feed, detecting instruments (identified by the bounding box) and classifying instruments by identification numbers for each instrument present within the field of view (See FIGS. 6-7). In the example shown in FIG. 6, there is shown a live-feed video in which the instrument recognition engine 204 detected, identified, and added bounding boxes to a surgical instrument added to a tray. In the example shown in FIG. 7, there is shown a live-feed video in which the instrument recognition engine 204 detected, identified, and added bounding boxes to a plurality of a surgical instruments added to a tray simulating a realistic scenario in the OR. Any time an instrument enters or exits the field of view, the instrument recognition engine 204 records this as an “instrument use event.”” Paragraph 0049), wherein said tracking and analyzing comprises object identification and comprises:
(a) determining, for each said item, a per-image detection state across the series of images, wherein an item that is actively visible on said table in a given image is detected in that image and an item that is not actively visible on said table in that image is not detected in that image (“In the example shown in FIG. 7, there is shown a live-feed video in which the instrument recognition engine 204 detected, identified, and added bounding boxes to a plurality of a surgical instruments added to a tray simulating a realistic scenario in the OR.” Paragraph 0049);
(b) determining, for each said item, whether said item has left a field-of-view of said one or more cameras (“Any time an instrument enters or exits the field of view, the instrument recognition engine 204 records this as an “instrument use event.”” Paragraph 0049), based on the per-image detection states and on positions of said item across the series of images (“The RetinaNet algorithm was applied to the live video feed, detecting instruments (identified by the bounding box) and classifying instruments by identification numbers for each instrument present within the field of view (See FIGS. 6-7). In the example shown in FIG. 6, there is shown a live-feed video in which the instrument recognition engine 204 detected, identified, and added bounding boxes to a surgical instrument added to a tray. In the example shown in FIG. 7, there is shown a live-feed video in which the instrument recognition engine 204 detected, identified, and added bounding boxes to a plurality of a surgical instruments added to a tray simulating a realistic scenario in the OR. Any time an instrument enters or exits the field of view, the instrument recognition engine 204 records this as an “instrument use event.”” Paragraph 0049);
(c) aggregating the per-image detection states and the determinations of (b) across the series of images captured during said intraoperative periods to generate aggregate data for said intraoperative settings and/or simulated intraoperative settings (see figure 11); and
(d) determining from said aggregate data, for each said item, (i) that said item was placed on said table during said intraoperative periods (“The RetinaNet algorithm was applied to the live video feed, detecting instruments (identified by the bounding box) and classifying instruments by identification numbers for each instrument present within the field of view (See FIGS. 6-7). In the example shown in FIG. 6, there is shown a live-feed video in which the instrument recognition engine 204 detected, identified, and added bounding boxes to a surgical instrument added to a tray. In the example shown in FIG. 7, there is shown a live-feed video in which the instrument recognition engine 204 detected, identified, and added bounding boxes to a plurality of a surgical instruments added to a tray simulating a realistic scenario in the OR. Any time an instrument enters or exits the field of view, the instrument recognition engine 204 records this as an “instrument use event.”” Paragraph 0049), and (ii) whether said item was interacted with during intraoperative periods, wherein said item is determined to have been interacted with based on at least one of (A) said item having left said field-of-view (“The RetinaNet algorithm was applied to the live video feed, detecting instruments (identified by the bounding box) and classifying instruments by identification numbers for each instrument present within the field of view (See FIGS. 6-7). In the example shown in FIG. 6, there is shown a live-feed video in which the instrument recognition engine 204 detected, identified, and added bounding boxes to a surgical instrument added to a tray. In the example shown in FIG. 7, there is shown a live-feed video in which the instrument recognition engine 204 detected, identified, and added bounding boxes to a plurality of a surgical instruments added to a tray simulating a realistic scenario in the OR. Any time an instrument enters or exits the field of view, the instrument recognition engine 204 records this as an “instrument use event.”” Paragraph 0049), and wherein an item determined to have been placed on said table and not interacted with is determined to have remained unused throughout said intraoperative periods (“The RetinaNet algorithm was applied to the live video feed, detecting instruments (identified by the bounding box) and classifying instruments by identification numbers for each instrument present within the field of view (See FIGS. 6-7). In the example shown in FIG. 6, there is shown a live-feed video in which the instrument recognition engine 204 detected, identified, and added bounding boxes to a surgical instrument added to a tray. In the example shown in FIG. 7, there is shown a live-feed video in which the instrument recognition engine 204 detected, identified, and added bounding boxes to a plurality of a surgical instruments added to a tray simulating a realistic scenario in the OR. Any time an instrument enters or exits the field of view, the instrument recognition engine 204 records this as an “instrument use event.”” Paragraph 0049); and
transmit said one or more determined characteristics to a secondary source (“The computing devices 102, 104, 106, 108, 110, 112 include a communication subsystem, which may be embodied as any communication circuit, device, or collection thereof, capable of enabling communications between the computing device 102, video feed 104 and other computing devices 106, 108, 110, 112 over the computer network 114. For example, the communication subsystem may be embodied as or otherwise include a network interface controller (NIC) or other network controller for sending and/or receiving network data with remote devices. The NIC may be embodied as any network interface card, network adapter, host fabric interface, network coprocessor, or other component that connects the computing device 102 and computing devices 104, 106, 108, 110, 112 to the network 106. The communication subsystem may be configured to use any one or more communication technology” paragraph 0036).
Fine does not teach determining whether an item has moved to a different spot on a table.
Hill teaches determining whether an item has moved to a different spot on a table (“The object use analyzer 148 may also track whether the surgical instrument is returned to the surgical tray. In this way, the object use analyzer 148 may track usage of surgical instruments based on whether they are detected by the RFID reader 140 attached to the Mayo stand 110.” Paragraph 0047) and wherein said item is determined to have been interacted with based on whether it has moved to a different spot on said table (“The object use analyzer 148 may also track whether the surgical instrument is returned to the surgical tray. In this way, the object use analyzer 148 may track usage of surgical instruments based on whether they are detected by the RFID reader 140 attached to the Mayo stand 110.” Paragraph 0047).
Hill is analogous art in the same field of endeavor as the claimed invention. Hill is directed towards medical equipment tracking (“The presently disclosed subject matter relates generally to healthcare. Particularly, the presently disclosed subject matter relates to electronic identification tagging systems, methods, applicators, and tapes for tracking and managing medical equipment and other objects.” Paragraph 0002). A person of ordinary skill in the art would have found it obvious to combine the teachings of Fine and Hill by utilizing the tool usage based on interactivity teachings in Hill combined with the camera and image based position determinations of Fine, with the expectation that doing so would lead to improved management and tracking of instruments and equipment (“…a system as disclosed herein can generate statistics on usage for reducing hospital cost and improving efficiency. Surgical instrument usage data collected by the system may be analyzed by the system to make recommendations as to which instruments should be supplied during specific surgeries for particular surgeons.” paragraph 0026).
With respect to claim 2, Fine and Hill teach the system of claim 1. Fine further teaches wherein said one or more computer processors are configured to execute the instructions to: retrain said trained computer vision model using said received settings image data from the intraoperative, and/or simulated intraoperative settings (“Building off of the success of the Phase 1 work, the AI/ML model for the instrument recognition engine 204 was optimized with the image recognition algorithms for complex image scenarios unique to the OR,” paragraph 0051).
With respect to claim 3, Fine and Hill teach teach the system of claim 2. Fine further teaches wherein said trained computer vision model is generated on preliminary image data using a machine learning algorithm (“RetinaNet is the algorithm used to develop the initial prototype model, which is a single, unified network comprising one backbone network and two task-specific subnetworks. The backbone network computes a convolutional feature map of the entire image; of the two subnetworks: a Focal Loss algorithm limits cross-entropy loss (i.e., improves accuracy) by classifying the output of the backbone network; and a Feature Pyramid Network (FPN) performs convolution bounding box regression.” Paragraph 0048 and “The instrument recognition engine 204 was then trained by using an instrument preparation station typical of most ORs (i.e., a mayo stand) and then a video camera was mounted from above to capture the entire set of instruments in a single view. For model testing, investigators dressed in surgical scrubs and personal protective equipment (PPE) proceeded to grab and replace instruments as if using them during a surgical procedure. The RetinaNet algorithm was applied to the live video feed, detecting instruments (identified by the bounding box) and classifying instruments by identification numbers for each instrument present within the field of view (See FIGS. 6-7).” Paragraph 0049).
With respect to claim 4, Fine and Hill teach the system of claim 3, Fine further teaches wherein: said training of said computer vision model, may be performed streaming to the cloud and in real-time (“Depending on the circumstances, the computing device 102 could be remote from the OR, such as a cloud-based platform that receives real-time video data from one or more video cameras in the OR via the network 114 and from which one or more functions of the automatic workflow management are accessible to the computing devices 106, 108, 110, 112 through the network 114.” Paragraph 0034), and locally and/or remotely on a network and/or server (“Although the system 100 is described as being a cloud-based platform accessible by the remote computing devices 104, 106, 108, 110, 112 in some embodiments one or more features of the server 102 could be performed locally on the remote computing devices 104” paragraph 0037 and figure 2).
With respect to claim 5, Fine and Hill teach the system of claim 2, Fine further teaches wherein: said retraining of said computer vision model, may be performed streaming to the cloud and in real-time (“Depending on the circumstances, the computing device 102 could be remote from the OR, such as a cloud-based platform that receives real-time video data from one or more video cameras in the OR via the network 114 and from which one or more functions of the automatic workflow management are accessible to the computing devices 106, 108, 110, 112 through the network 114.” Paragraph 0034) and locally and/or remotely on a network and/or server (“Although the system 100 is described as being a cloud-based platform accessible by the remote computing devices 104, 106, 108, 110, 112 in some embodiments one or more features of the server 102 could be performed locally on the remote computing devices 104” paragraph 0037 and figure 2).
With respect to claim 6, Fine and Hill teach the system of claim 1. Fine further teaches wherein said trained computer vision model is generated on preliminary image data using a machine learning algorithm (“RetinaNet is the algorithm used to develop the initial prototype model, which is a single, unified network comprising one backbone network and two task-specific subnetworks. The backbone network computes a convolutional feature map of the entire image; of the two subnetworks: a Focal Loss algorithm limits cross-entropy loss (i.e., improves accuracy) by classifying the output of the backbone network; and a Feature Pyramid Network (FPN) performs convolution bounding box regression.” Paragraph 0048 and “The instrument recognition engine 204 was then trained by using an instrument preparation station typical of most ORs (i.e., a mayo stand) and then a video camera was mounted from above to capture the entire set of instruments in a single view. For model testing, investigators dressed in surgical scrubs and personal protective equipment (PPE) proceeded to grab and replace instruments as if using them during a surgical procedure. The RetinaNet algorithm was applied to the live video feed, detecting instruments (identified by the bounding box) and classifying instruments by identification numbers for each instrument present within the field of view (See FIGS. 6-7).” Paragraph 0049).
With respect to claim 7, Fine and Hill teach the system of claim 6, wherein: said training of said computer vision model, may be performed streaming to the cloud and in real-time (“Depending on the circumstances, the computing device 102 could be remote from the OR, such as a cloud-based platform that receives real-time video data from one or more video cameras in the OR via the network 114 and from which one or more functions of the automatic workflow management are accessible to the computing devices 106, 108, 110, 112 through the network 114.” Paragraph 0034) and locally and/or remotely on a network and/or server (“Although the system 100 is described as being a cloud-based platform accessible by the remote computing devices 104, 106, 108, 110, 112 in some embodiments one or more features of the server 102 could be performed locally on the remote computing devices 104” paragraph 0037 and figure 2).
With respect to claim 8, Fine and Hill teach the system of claim 1, Fine further teaches wherein one or more of the following instructions: a) said receiving of said settings image data (see figure 2 202), b) said running of said trained computer vision model (see figure 2 204), and c) said interpreting of the surgical related items and/or procedure related items (see figure 2, 204 and 210 and 212), may be performed streaming to the cloud and in real-time (“Depending on the circumstances, the computing device 102 could be remote from the OR, such as a cloud-based platform that receives real-time video data from one or more video cameras in the OR via the network 114 and from which one or more functions of the automatic workflow management are accessible to the computing devices 106, 108, 110, 112 through the network 114.” Paragraph 0034) and locally and/or remotely on a network and/or server (“Although the system 100 is described as being a cloud-based platform accessible by the remote computing devices 104, 106, 108, 110, 112 in some embodiments one or more features of the server 102 could be performed locally on the remote computing devices 104” paragraph 0037 and figure 2).
With respect to claim 9, Fine and Hill teach the system of claim 1. Fine further teaches wherein said tracking and analyzing comprises object identification for tracking and analyzing (see figure 2 204 and “Example 7 includes the subject matter of Examples 1-6, and wherein: the instrument device recognition engine is to trigger the instrument use event based on detecting at least one object leaving the field of view of the real-time video feed in an operating room (OR).” Paragraph 0079).
With respect to claim 11, Fine and Hill teach the system of claim 1. Fine further teaches wherein said one or more computer processors are configured to execute the instructions for said tracking and analyzing at cloud infrastructure (“Depending on the circumstances, the computing device 102 could be remote from the OR, such as a cloud-based platform that receives real-time video data from one or more video cameras in the OR via the network 114 and from which one or more functions of the automatic workflow management are accessible to the computing devices 106, 108, 110, 112 through the network 114.” Paragraph 0034);
With respect to claim 12, Fine and Hill teach the system of claim 1. Fine further teaches wherein said secondary source includes one or more of any one of the following:
local memory (“The computing devices 102, 104, 106, 108, 110, 112 may be embodied as any type of computation or computer device capable of performing the functions described herein, including, without limitation, a computer, a server, a workstation, a desktop computer, a laptop computer, a notebook computer, a tablet computer, a mobile computing device, a wearable computing device, a network appliance, a web appliance, a distributed computing system, a processor-based system, and/or a consumer electronic device.” paragraph 0035);
remote memory (“The computing devices 102, 104, 106, 108, 110, 112 may be embodied as any type of computation or computer device capable of performing the functions described herein, including, without limitation, a computer, a server, a workstation, a desktop computer, a laptop computer, a notebook computer, a tablet computer, a mobile computing device, a wearable computing device, a network appliance, a web appliance, a distributed computing system, a processor-based system, and/or a consumer electronic device.” paragraph 0035); or display (“In the embodiment shown, the system 100 includes a computing device 102 that performs automatic workflow management in communication with one or more computing devices 104, 106, 108, 110, 112 in the OR over a network 114. For example, the computing device 104 may be one or more video cameras that stream real-time video data of a field of view in the OR to the computing device 102 over the network 114. The computing devices 106, 108, 110 could be computing devices used by one or more members of the OR team to display the steps in the workflow and/or other information specific to that stage in the surgery. For example, in some cases, at least a portion of the OR team could each have their own computing device with a role-based workflow individualized for that particular member of the OR team. Depending on the circumstances, an OR may include a computing device 112 that is shared by multiple members of the team.” Paragraph 0031) or graphical user interface (“The computing devices 106, 108, 110, 112 are configured to access one or more features of the computing device 102 over the network 114. For example, the computing device 102 may include a web-based interface or portal through which users of the computing devices 106, 108, 110, 112 can interact with features of the computing device 102 using a browser” paragraph 0037).
With respect to claim 13, Fine and Hill teach the system of claim 6. Fine further teaches wherein the machine learning algorithm includes an artificial neural network (ANN) or deep learning algorithm (“RetinaNet is a layered algorithm comprising two key sub-algorithms: a Feature Pyramid Network (FPN), which makes use of the inherent multi-scale pyramidal hierarchy of deep CNNs to create feature pyramids and a Focal Loss algorithm, which improves upon cross-entropy loss to help reduce the relative loss for well-classified examples by putting more focus on hard, misclassified examples.” Paragraph 0041).
With respect to claim 14, Fine and Hill teach the system of claim 13, wherein said artificial neural network (ANN) includes: convolutional neural network (CNN) (“RetinaNet is a layered algorithm comprising two key sub-algorithms: a Feature Pyramid Network (FPN), which makes use of the inherent multi-scale pyramidal hierarchy of deep CNNs to create feature pyramids and a Focal Loss algorithm, which improves upon cross-entropy loss to help reduce the relative loss for well-classified examples by putting more focus on hard, misclassified examples.” Paragraph 0041);
With respect to claim 15, Fine and Hill teach the system of claim 1. Fine teaches wherein said determined one or more characteristics includes any combination of one or more of the following:
identification of the one or more of the surgical related items and/or procedure related items (“According to one aspect, this disclosure provides a computing device for managing operating room workflow events. The computing device includes an instrument use event manager to: (i) define a plurality of steps of an operating room workflow for a medical procedure; and (ii) link one or more instrument use events to at least a portion of the plurality of steps in the operating room workflow. There is an instrument device recognition engine to trigger an instrument use event based an identification and classification of at least one object within a field of view of a real-time video feed in an operating room (OR).” Paragraph 0010);
usage or non-usage status of the one or more of the surgical related items and/or procedure related items (“According to one aspect, this disclosure provides a computing device for managing operating room workflow events. The computing device includes an instrument use event manager to: (i) define a plurality of steps of an operating room workflow for a medical procedure; and (ii) link one or more instrument use events to at least a portion of the plurality of steps in the operating room workflow. There is an instrument device recognition engine to trigger an instrument use event based an identification and classification of at least one object within a field of view of a real-time video feed in an operating room (OR).” Paragraph 0010);
or association of clinical events, logistical events, or operational events (“According to one aspect, this disclosure provides a computing device for managing operating room workflow events. The computing device includes an instrument use event manager to: (i) define a plurality of steps of an operating room workflow for a medical procedure; and (ii) link one or more instrument use events to at least a portion of the plurality of steps in the operating room workflow. There is an instrument device recognition engine to trigger an instrument use event based an identification and classification of at least one object within a field of view of a real-time video feed in an operating room (OR).” Paragraph 0010).
With respect to claim 18, Fine and Hill teach the system of claim 1. Fine further teaches wherein neither machine readable markings on the surgical related items and/or procedure related items nor communicable coupling between said system and the surgical related items and/or procedure related items are required by said system to provide said one or more determined characteristics (“The instrument recognition engine 204 is configured to recognize instruments, tools, materials, and/or other objects in the OR using AI/ML. For example, the instrument recognition engine 204 may go from object images to accurate object detection and classification using innovative AI/ML deep learning techniques. To accomplish this, in some embodiments, the instrument recognition engine 204 includes a convolutional neural network (CNN). A CNN “recognizes” objects by iteratively pulling out features of an object that link it to increasingly finer classification levels.” Paragraph 0040).
With respect to claim 19, Fine and Hill teach the system of claim 1. Fine further teaches wherein said settings image data comprises information from the visible light spectrum (“The instrument recognition engine 204 is configured to recognize instruments, tools, materials, and/or other objects in the OR using AI/ML. For example, the instrument recognition engine 204 may go from object images to accurate object detection and classification using innovative AI/ML deep learning techniques. To accomplish this, in some embodiments, the instrument recognition engine 204 includes a convolutional neural network (CNN). A CNN “recognizes” objects by iteratively pulling out features of an object that link it to increasingly finer classification levels.” Paragraph 0040).
With respect to claim 21, Fine and Hill render obvious all limitations in consideration of claim 1, because claim 21 is the method version of claim 1.
With respect to claim 22, Fine and Hill teach the method of claim 21 and render obvious all limitations in consideration of claim 2, because claim 22 is the method version of claim 2.
With respect to claim 23, Fine and Hill teach the method of claim 22 and render obvious all limitations in consideration of claim 3, because claim 23 is the method version of claim 3.
With respect to claim 24, Fine and Hill teach the method of claim 23 and render obvious all limitations in consideration of claim 4, because claim 24 is the method version of claim 4.
With respect to claim 25, Fine and Hill teach the method of claim 22 and render obvious all limitations in consideration of claim 5, because claim 25 is the method version of claim 5.
With respect to claim 26, Fine and Hill teach the method of claim 21 and render obvious all limitations in consideration of claim 6, because claim 26 is the method version of claim 6.
With respect to claim 27, Fine and Hill teach the method of claim 26 and render obvious all limitations in consideration of claim 7, because claim 27 is the method version of claim 7.
With respect to claim 28, Fine and Hill teach the method of claim 21 and render obvious all limitations in consideration of claim 8, because claim 28 is the method version of claim 8.
With respect to claim 29, Fine and Hill teach the method of claim 21 and render obvious all limitations in consideration of claim 9, because claim 29 is the method version of claim 9.
With respect to claim 31, Fine and Hill teach the method of claim 21 and render obvious all limitations in consideration of claim 11, because claim 31 is the method version of claim 11.
With respect to claim 32, Fine and Hill teach the method of claim 21 and render obvious all limitations in consideration of claim 12, because claim 32 is the method version of claim 12.
With respect to claim 33, Fine and Hill teach the method of claim 26 and render obvious all limitations in consideration of claim 13, because claim 33 is the method version of claim 13.
With respect to claim 34, Fine and Hill teach the method of claim 33 and render obvious all limitations in consideration of claim 14, because claim 34 is the method version of claim 14.
With respect to claim 35, Fine and Hill teach the method of claim 21 and render obvious all limitations in consideration of claim 15, because claim 35 is the method version of claim 15.
With respect to claim 38, Fine and Hill teach the method of claim 21 and render obvious all limitations in consideration of claim 18, because claim 38 is the method version of claim 18.
With respect to claim 39, Fine and Hill teach the method of claim 21 and render obvious all limitations in consideration of claim 19, because claim 39 is the method version of claim 19.
With respect to claim 41, Fine and Hill render obvious all claim limitations in consideration of claim 1 because claim 41 is directed towards a non-transitory computer readable medium storing instructions that when executed by one or more processors, cause the one or more processors to preform the processes of claim 1.
With respect to claim 42, Fine and Hill teach the non-transitory computer-readable medium of claim 41 and render obvious all claim limitations in consideration of claim 2 because claim 42 is directed towards a non-transitory computer readable medium storing instructions that when executed by one or more processors, cause the one or more processors to preform the processes of claim 2.
With respect to claim 43, Fine and Hill teach the non-transitory computer-readable medium of claim 42 and render obvious all claim limitations in consideration of claim 3 because claim 43 is directed towards a non-transitory computer readable medium storing instructions that when executed by one or more processors, cause the one or more processors to preform the processes of claim 3.
With respect to claim 44, Fine and Hill teach the non-transitory computer-readable medium of claim 43 and render obvious all claim limitations in consideration of claim 4 because claim 44 is directed towards a non-transitory computer readable medium storing instructions that when executed by one or more processors, cause the one or more processors to preform the processes of claim 4.
With respect to claim 45, Fine and Hill teach the non-transitory computer-readable medium of claim 42 and render obvious all claim limitations in consideration of claim 5 because claim 45 is directed towards a non-transitory computer readable medium storing instructions that when executed by one or more processors, cause the one or more processors to preform the processes of claim 5.
With respect to claim 46, Fine and Hill teach the non-transitory computer-readable medium of claim 41 and render obvious all claim limitations in consideration of claim 6 because claim 46 is directed towards a non-transitory computer readable medium storing instructions that when executed by one or more processors, cause the one or more processors to preform the processes of claim 6.
With respect to claim 47, Fine and Hill teach the non-transitory computer-readable medium of claim 46 and render obvious all claim limitations in consideration of claim 7 because claim 47 is directed towards a non-transitory computer readable medium storing instructions that when executed by one or more processors, cause the one or more processors to preform the processes of claim 7.
With respect to claim 48, Fine and Hill teach the non-transitory computer-readable medium of claim 41 and render obvious all claim limitations in consideration of claim 8 because claim 48 is directed towards a non-transitory computer readable medium storing instructions that when executed by one or more processors, cause the one or more processors to preform the processes of claim 8.
With respect to claim 49, Fine and Hill teach the non-transitory computer-readable medium of claim 41 and render obvious all claim limitations in consideration of claim 9 because claim 49 is directed towards a non-transitory computer readable medium storing instructions that when executed by one or more processors, cause the one or more processors to preform the processes of claim 9.
With respect to claim 51, Fine and Hill teach the non-transitory computer-readable medium of claim 41 and render obvious all claim limitations in consideration of claim 11 because claim 51 is directed towards a non-transitory computer readable medium storing instructions that when executed by one or more processors, cause the one or more processors to preform the processes of claim 11.
With respect to claim 52, Fine and Hill teach the non-transitory computer-readable medium of claim 41 and render obvious all claim limitations in consideration of claim 12 because claim 52 is directed towards a non-transitory computer readable medium storing instructions that when executed by one or more processors, cause the one or more processors to preform the processes of claim 12.
With respect to claim 53, Fine and Hill teach the non-transitory computer-readable medium of claim 46 and render obvious all claim limitations in consideration of claim 13 because claim 53 is directed towards a non-transitory computer readable medium storing instructions that when executed by one or more processors, cause the one or more processors to preform the processes of claim 13.
With respect to claim 54, Fine and Hill teach the non-transitory computer-readable medium of claim 53 and render obvious all claim limitations in consideration of claim 14 because claim 54 is directed towards a non-transitory computer readable medium storing instructions that when executed by one or more processors, cause the one or more processors to preform the processes of claim 14.
With respect to claim 55, Fine and Hill teach the non-transitory computer-readable medium of claim 41 and render obvious all claim limitations in consideration of claim 15 because claim 55 is directed towards a non-transitory computer readable medium storing instructions that when executed by one or more processors, cause the one or more processors to preform the processes of claim 15.
With respect to claim 58, Fine and Hill teach the non-transitory computer-readable medium of claim 41 and render obvious all claim limitations in consideration of claim 18 because claim 58 is directed towards a non-transitory computer readable medium storing instructions that when executed by one or more processors, cause the one or more processors to preform the processes of claim 18.
With respect to claim 59, Fine and Hill teach the non-transitory computer-readable medium of claim 41 and render obvious all claim limitations in consideration of claim 19 because claim 59 is directed towards a non-transitory computer readable medium storing instructions that when executed by one or more processors, cause the one or more processors to preform the processes of claim 19.
Claims 10, 17, 20, 30, 37, 40, 50, 57, and 60 are rejected under 35 U.S.C. 103 as being unpatentable over Fine and Hill as applied above, and further in view of Divine (US 11250947 B2).
With respect to claim 10, Fine and Hill teach the system of claim 1, but do not teach wherein said tracking and analyzing comprises specified multiple tracking and analyzing models. Divine teaches tracking and analyzing comprising specified multiple tracking and analyzing models (see fig 10 element 110, Environment Assistant component, Environment Recognition component, environment Characterization component, Descriptive feedback component, Utilization Feedback component work in conjunction with the Equipment/Supplies Assessment component to track and analyze tools in the environment).
Divine is analogous art in the same field of endeavor as the claimed invention. Divine is directed towards determining one or more characteristics of surgical related items and/or procedure related items present at intraoperative, settings (“In implementations in which the one or more objects include equipment or supplies, the environment assessment component 1006 can employ the equipment/supplies assessment component 1008 to determine various types of equipment/supplies information associated with the equipment or supplies.” Page 45 col 43 lines 53-58). A person of ordinary skill in the art before the effective filing date of the claimed invention would have found it obvious to combine the combined system of Fine and Hill’s object identification and tracking strategy with the additional contextual strategy of Divine’s multiple model method, with the expectation that doing so would lead to more improved clinical and financial strategies surround instrument use and waste (“The subject disclosure provides computer processing systems, computer-implemented methods, apparatus and/or computer program products that facilitate a user (e.g., an administrator of the healthcare organization or another suitable entity) with understanding information concerning a state of performance of a healthcare organization by providing the healthcare administrator with auxiliary information associated with a current physical area of a healthcare facility of the organization the healthcare professional is viewing or otherwise examining. In one or more implementations, the state of performance can relate to a clinical and/or financial performance state of the healthcare organization. In particular, based on captured image data and/or other input information a computing system can determine an area of a healthcare facility that the user is currently looking at, or has requested to look at. The computing system can further identify objects, equipment, supplies, people, etc., included in the area (or associated with the area), and further determine auxiliary information associated with the objects, equipment, supplies, people, etc.,” col 7 (lines 65-67 )- 8 (lines 1-17) and figure 15).
With respect to claim 17, Fine and Hill teach the system of claim 1, but do not teach any further limitations. Divine teaches wherein said one or more computer processors are further configured to, based on said determined one or more characteristics, execute the instructions to: determine an actionable output to improve efficiency of using the surgical related items for use in the intraoperative setting(“In one or more embodiments, the recommendation component 1602 can be configured to analyze the various information/data included in the one or more external information sources 138 (and/or in memory 130), as well as the information determined by the environment assessment component 1006 over time to determine recommendation information regarding how to improve an aspect of the state of performance of the healthcare organization. For example, the recommendation component 1602 can evaluate past performance information regarding the clinical and/or financial performance of the healthcare organization associated with utilization of various equipment, supplies and employees of the healthcare organization. Using one or more machine learning and/or deep learning techniques, the recommendation component 1602 can identify patterns in the data attributed to financial and/or clinical gains and losses and determine recommendation information regarding how to improve financial and/or clinical gain and minimize financial and/or clinical loss. For example, such recommendation information can include the addition or removal of resources, changes to manners in which the healthcare organization uses respective equipment, supplies and employees, and the like. Relevant recommendation information (e.g., relevant to a current area being viewed by a user and a current context of the user) can further be selected by the selection component 1016 and provided to a user as overlay data in an AR or VR experience.” Page 50 col 54 lines 43-67 and page 51 col 55 lines 1-2 and); and/or determine an actionable output to recognize level of efficiency of surgeons or clinicians (“In one or more embodiments, the recommendation component 1602 can be configured to analyze the various information/data included in the one or more external information sources 138 (and/or in memory 130), as well as the information determined by the environment assessment component 1006 over time to determine recommendation information regarding how to improve an aspect of the state of performance of the healthcare organization. For example, the recommendation component 1602 can evaluate past performance information regarding the clinical and/or financial performance of the healthcare organization associated with utilization of various equipment, supplies and employees of the healthcare organization. Using one or more machine learning and/or deep learning techniques, the recommendation component 1602 can identify patterns in the data attributed to financial and/or clinical gains and losses and determine recommendation information regarding how to improve financial and/or clinical gain and minimize financial and/or clinical loss. For example, such recommendation information can include the addition or removal of resources, changes to manners in which the healthcare organization uses respective equipment, supplies and employees, and the like. Relevant recommendation information (e.g., relevant to a current area being viewed by a user and a current context of the user) can further be selected by the selection component 1016 and provided to a user as overlay data in an AR or VR experience.” Page 50 col 54 lines 43-67 and page 51 col 55 lines 1-2, and “For example, with respect to an employee, the recommendation component 1602 can determine a suggested thing to say to a particular employee the user encounters that would improving the clinical and financial performance of the healthcare organization. According to this example, using one or more machine learning or deep learning techniques, the recommendation component 1602 can evaluate past performance information associated with an employee and determine an aspect of the employees' past performance that needs improvement. The recommendation component 1602 can also determine based on the current context of the user and the employee, a relevant suggested remark for the user to say to the employee that would encourage the employee to improve his or her performance. The suggested remark can further be provided to the user as overlay data in AR or VR when the user encounters the employee” page 51 col 55 lines 11-27);
Divine is analogous art in the same field of endeavor as the claimed invention. Divine is directed towards determining one or more characteristics of surgical related items and/or procedure related items present at intraoperative, settings (“In implementations in which the one or more objects include equipment or supplies, the environment assessment component 1006 can employ the equipment/supplies assessment component 1008 to determine various types of equipment/supplies information associated with the equipment or supplies.” Page 45 col 43 lines 53-58). A person of ordinary skill in the art before the effective filing date of the claimed invention would have found it obvious to combine the combined system of Fine and Hill’s object identification and tracking strategy with the additional contextual strategy of Divine’s multiple model method, with the expectation that doing so would lead to more improved clinical and financial strategies surround instrument use and waste (“The subject disclosure provides computer processing systems, computer-implemented methods, apparatus and/or computer program products that facilitate a user (e.g., an administrator of the healthcare organization or another suitable entity) with understanding information concerning a state of performance of a healthcare organization by providing the healthcare administrator with auxiliary information associated with a current physical area of a healthcare facility of the organization the healthcare professional is viewing or otherwise examining. In one or more implementations, the state of performance can relate to a clinical and/or financial performance state of the healthcare organization. In particular, based on captured image data and/or other input information a computing system can determine an area of a healthcare facility that the user is currently looking at, or has requested to look at. The computing system can further identify objects, equipment, supplies, people, etc., included in the area (or associated with the area), and further determine auxiliary information associated with the objects, equipment, supplies, people, etc.,” col 7 (lines 65-67 )- 8 (lines 1-17) and figure 15).
With respect to claim 20, Fine and Hill teach the system of claim 1, but do not teach the additional limitations. Divine teaches wherein said settings image data comprises three dimensional renderings or representation of information of the surgical related items and/or procedure related items in the intraoperative setting (“The image data can include live video data being captured of the selected area, still image data captured of the selected area, or model data (e.g., 2D/3D model data) including a regenerated representations or models of the selected area. With this implementation, the particular area and/or view of the environment will be known to the environment recognition component 1002 based on the input selection made by the user. In some one or more embodiments, the environment characterization component 1004 can use image analysis to identify objects and relative locations of the objects in the image data that is rendered to the user.” Page 45 col 43 line 13-24).
Divine is analogous art in the same field of endeavor as the claimed invention. Divine is directed towards determining one or more characteristics of surgical related items and/or procedure related items present at intraoperative, settings (“In implementations in which the one or more objects include equipment or supplies, the environment assessment component 1006 can employ the equipment/supplies assessment component 1008 to determine various types of equipment/supplies information associated with the equipment or supplies.” Page 45 col 43 lines 53-58). A person of ordinary skill in the art before the effective filing date of the claimed invention would have found it obvious to combine the combined system of Fine and Hill’s object identification and tracking strategy with the additional contextual strategy of Divine’s multiple model method, with the expectation that doing so would lead to more improved clinical and financial strategies surround instrument use and waste (“The subject disclosure provides computer processing systems, computer-implemented methods, apparatus and/or computer program products that facilitate a user (e.g., an administrator of the healthcare organization or another suitable entity) with understanding information concerning a state of performance of a healthcare organization by providing the healthcare administrator with auxiliary information associated with a current physical area of a healthcare facility of the organization the healthcare professional is viewing or otherwise examining. In one or more implementations, the state of performance can relate to a clinical and/or financial performance state of the healthcare organization. In particular, based on captured image data and/or other input information a computing system can determine an area of a healthcare facility that the user is currently looking at, or has requested to look at. The computing system can further identify objects, equipment, supplies, people, etc., included in the area (or associated with the area), and further determine auxiliary information associated with the objects, equipment, supplies, people, etc.,” col 7 (lines 65-67 )- 8 (lines 1-17) and figure 15).
With respect to claim 30, Fine and Hill teach the method of claim 21 and in view of Divine render obvious all limitations in consideration of claim 10, because claim 30 is the method version of claim 10.
With respect to claim 37, Fine and Hill teach the method of claim 21 and in view of Divine render obvious all limitations in consideration of claim 17, because claim 37 is the method version of claim 17.
With respect to claim 40, Fine and Hill teach the method of claim 21 and in view of Divine render obvious all limitations in consideration of claim 20, because claim 40 is the method version of claim 20.
With respect to claim 50, Fine and Hill teach the non-transitory computer-readable medium of claim 41 and in view of Divine render obvious all claim limitations in consideration of claim 10 because claim 50 is directed towards a non-transitory computer readable medium storing instructions that when executed by one or more processors, cause the one or more processors to preform the processes of claim 10.
With respect to claim 57, Fine and Hill teach the non-transitory computer-readable medium of claim 41 and in view of Divine render obvious all claim limitations in consideration of claim 17 because claim 57 is directed towards a non-transitory computer readable medium storing instructions that when executed by one or more processors, cause the one or more processors to preform the processes of claim 17.
With respect to claim 60, Fine and Hill teach the non-transitory computer-readable medium of claim 41 and in view of Divine render obvious all claim limitations in consideration of claim 20 because claim 60 is directed towards a non-transitory computer readable medium storing instructions that when executed by one or more processors, cause the one or more processors to preform the processes of claim 20.
Response to Arguments
Applicant’s arguments filed 08/06/2026 have been fully considered.
With respect to the previously made 112(b) rejections, the examiner agrees that they are overcome by the amendments made to the claims.
With respect to the previously made 101 rejections, the examiner disagrees and finds they are not overcome by the claim amendments. On page 25 of applicant’s remarks, it is argued that the claims are directed towards a specific technical implementation, instead of an abstract process. The applicant specifically cites limitations concerning the receipt of setting image data, running of a trained computer vision model, data transmission, and a four part tracking and analysis process. The examiner finds that claim 1 and the other substantially similar dependent claims (21 and 41) are directed to the tracking and analysis process. The examiner also holds that this process, as written, falls within the abstract idea grouping of mental processes. A person can view images and determine whether an item is detected based on whether it is on a table (see claim 1 (a)). A person can look at an image and see if an item has moved outside the field of view of the camera that took the picture (see claim 1 (b)). A person can then, whether through memory or the use of a pencil paper, combine, record, or otherwise aggregate that information across the series of images (see claim 1 (c)). A person can then, based off of their tracking and observation ability (how the items moved (were detected) across the series of images), come to the conclusion (determine or form an opinion) whether or not an item was interacted with and/or used (see claim 1 (d)).
The examiner also notes that the courts do not distinguish “between claims that recite mental processes performed by humans and claims that recite mental processes performed on a computer. As the Federal Circuit has explained, "[c]ourts have examined claims that required the use of a computer and still found that the underlying, patent-ineligible invention could be performed via pen and paper or in a person’s mind." Versata Dev. Group v. SAP Am., Inc., 793 F.3d 1306, 1335, 115 USPQ2d 1681, 1702 (Fed. Cir. 2015). See also Intellectual Ventures I LLC v. Symantec Corp., 838 F.3d 1307, 1318, 120 USPQ2d 1353, 1360 (Fed. Cir. 2016) (‘‘[W]ith the exception of generic computer-implemented steps, there is nothing in the claims themselves that foreclose them from being performed by a human, mentally or with pen and paper.’’); Mortgage Grader, Inc. v. First Choice Loan Servs. Inc., 811 F.3d 1314, 1324, 117 USPQ2d 1693, 1699 (Fed. Cir. 2016) (holding that computer-implemented method for "anonymous loan shopping" was an abstract idea because it could be "performed by humans without a computer").” See MPEP 2106.04(a)(2) III and “An example of a case in which a computer was used as a tool to perform a mental process is Mortgage Grader, 811 F.3d. at 1324, 117 USPQ2d at 1699. The patentee in Mortgage Grader claimed a computer-implemented system for enabling borrowers to anonymously shop for loan packages offered by a plurality of lenders, comprising a database that stores loan package data from the lenders, and a computer system providing an interface and a grading module. The interface prompts a borrower to enter personal information, which the grading module uses to calculate the borrower’s credit grading, and allows the borrower to identify and compare loan packages in the database using the credit grading. 811 F.3d. at 1318, 117 USPQ2d at 1695. The Federal Circuit determined that these claims were directed to the concept of "anonymous loan shopping", which was a concept that could be "performed by humans without a computer." 811 F.3d. at 1324, 117 USPQ2d at 1699. Another example is Berkheimer v. HP, Inc., 881 F.3d 1360, 125 USPQ2d 1649 (Fed. Cir. 2018), in which the patentee claimed methods for parsing and evaluating data using a computer processing system. The Federal Circuit determined that these claims were directed to mental processes of parsing and comparing data, because the steps were recited at a high level of generality and merely used computers as a tool to perform the processes. 881 F.3d at 1366, 125 USPQ2d at 1652-53.” MPEP 2106.04(a)(2) III (C).
With respect to the applicant’s Step 2A Prong 2 analysis the examiner again disagrees. As written the additional limitation involving receiving setting image data and running the trained vision model do not integrate the abstract idea into a practical application. Although amendments have been made that introduce limitations on the captured images and training of the model, neither the image capturing nor the model training is done as a part of the claimed invention. The data is being received and the model is being applied, but as written, the additional limitations concerning how they were generated amount to insignificant extra solution activity. Additionally, the applicant argues that the claim represents an improvement over an existing technical process. The examiner reminds the applicant that “To show that the involvement of a computer assists in improving the technology, the claims must recite the details regarding how a computer aids the method, the extent to which the computer aids the method, or the significance of a computer to the performance of the method. Merely adding generic computer components to perform the method is not sufficient. Thus, the claim must include more than mere instructions to perform the method on a generic component or machinery to qualify as an improvement to an existing technology.” See MPEP 2106.05(a) II.
With respect to the applicant’s arguments concerning their Step 2B interpretation, the examiner again disagrees, substantially due to the above reason along with disagreeing with applicant’s interpretation of Khan. Khan et al notes the use of Machine Learning in fields like medical science (see figure 4) and also showcases the use of training images for object detection (see figure 3) as well as confidence and accuracy measurements (see table 1). Additionally, while a factual determination is required to support the conclusion that something is while understood, routine, or conventional, a prior art search is not necessary (see 2106.05(d)). Khan et al was used to illustrate this factual finding additional evidence can be found in the applicant’s own specification in which the computer vision model is stated to have been trained on image data using a machine learning algorithm (“the trained computer vision model is generated on preliminary image data using a machine learning algorithm.” See page 8 paragraph 1) and runs using received images to identify and label items (“run a trained computer vision model on the received settings image data to identify and label the surgical related items” see page 7 paragraph 3). Both of these are broad and similar to the teachings found in Khan et al.
Due to the above reasoning and the reasoning provided within the examiner’s 101 rejections, the 101 rejections have been updated to coincide with the amendments but still remains. Additionally, upon an updated search and consideration new prior art was found and used to reject the newly amended claims, Applicant’s arguments about the previously made 102 and 103 rejections are thus considered moot, with Divine alone or in combination with Buch no longer is acting as the basis of rejection for any claim. Accordingly, all pending claims remain rejected.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to REBECCA C WILLIAMS whose telephone number is (571)272-7074. The examiner can normally be reached M-F 7:30am - 4:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew W Bee can be reached at (571)270-5183. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/REBECCA COLETTE WILLIAMS/Examiner, Art Unit 2677
/ANDREW W BEE/Supervisory Patent Examiner, Art Unit 2677
