DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
Claims 1-2, 4, 6-7, 10-13, 17-18, 20-24, 26-27 and 30 remain pending. Claims 3, 5, 8-9, 14-16, 19,25, and 28-29 are cancelled. Amendments made to claims, drawings, and specification have overcome all previously held objections.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2, 4, 7, 13 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over DeLand (US 20130113929 A1, published: 2013-05-09, filed: 2011-11-08) in view of Dein (WO 2010008846 A2, published 2010-01-21), Wu (WO 2015191702 A1), Osborn (WO 2004084102 A1, published 2004-03-22), Ma (US 10874759 B2, filed: 2019-03-20) and Esterberg (US 20180368930 A1, filed: 2018-12-27).
With respect to claim 1, DeLand teaches a method for tracking a plurality of surgical tools arranged on a tray through a use cycle in a perioperative environment (see figures 4 and 6 ), the method comprising: capturing, via a camera, one or more images of the plurality of surgical tools (see figures 4 and 6 cameras) and; processing the one or more images to determine identities of each surgical tool in the plurality of surgical tools on the tray (“A recognition unit adapted to recognize identities and positions of instruments or tools in images produced by the one or more cameras.” Paragraph 0012 lines 4-6 see also figures 4 and 6);recording the determined identities of each surgical tool to form an initial list of identities (“The memory unit 330 stores the positions and the identities of surgical instruments” paragraph 0035 lines 1-2); comparing the initial list of identities of each surgical tool on the tray to the expected list of identities of surgical tools expected on the tray (“As the surgical instruments are assembled on the surgical tray, the surgical instrument tracker recognizes the surgical instruments and checks off the surgical instruments listed in the template.” Paragraph 0046 lines 7-10);based on the comparing step, determining if one or more surgical tools are missing from the tray and if one or more surgical tools are added to the tray (see figures 4 and 6 and “At step 450, the surgical instrument tracker performs a similar process to identify surgical instruments on the back tray, for example, back tray 230, to check that there are no surgical instruments on the back tray. If surgical instruments are found o the back tray these instruments have to be removed from the back tray, or an explanation has to be recorded in the surgical instrument tracker as to why these instruments are on the back tray” paragraph 44),
DeLand does not teach an image of a reference tag on the tray, wherein the reference tag provides a reference grid for obtaining dimensions of the plurality of surgical tools and a 2D code that identifies an expected list of identities of surgical tools on the tray and creating an augmented reality information comprising an identity of each of the one or more surgical tools missing from or added to the tray and outputting the augmented reality information in a field of view of a user.
Dein teaches a reference grid (“…the "mayo stand" or sterile surgical back table can have markers which delineate a grid on the table, such that markers can outline areas where a particular camera/reader may scan” page 23 SYSTEM CONFIGURATIONS paragraph 2 lines 5-6) for obtaining dimensions of the plurality of surgical tools (“In addition to the intra-operative imaging device, the systems include a surgical sharp object automated shape recognition module.” Page 6 paragraph 3 lines 1-2 and “Automated shape recognition is a process by which the identification of surgical sharp objects is performed using one or more of the size, shape, aspect ratio, --> outline, color, or other distinctive feature present in the image data of the surgical sharp object.” Paragraph 4 lines 4-6)
Dein is analogous art in the same field of endeavor as the claimed invention. Dein is directed towards a system for tracking surgical tools during an operation (“Additional components of the intra-operative identification and tracking systems of the invention…” page 12 line 7). A person of ordinary skill in the art before the effective filing date of the claimed invention could have reasoned that combining the system of DeLand with Dein by incorporating Dein’s shape and pattern recognition module could lead to improved accuracy in counts and reductions in errors leading to patient safety enhancements (“The accuracy and designation of responsibility for the counts and decisions affected by the count are all improved by the subject methods and system. --> Most importantly, the chances for errors are reduced and the likelihood of incorrect counts reduced. Safety is greatly enhanced for the patient…” page 29 lines 18-21). Therefore, it would have been obvious for a person of ordinary skill in the art before the effective filing date of the claimed invention to combine the system of DeLand with Dein by incorporating Dein’s shape and pattern recognition module with the expectation that doing so would lead to improved accuracy in counts and reductions in errors leading to patient safety enhancements (see page 29 lines 18-21).
Wu teaches capturing an image of a reference tag on the tray (“Software module 806 detects barcodes on the surface of the sample trays” paragraph 00117), wherein the reference tag provides a reference grid (“Software module 806 detects barcodes on the surface of the sample trays. This allows software to determine a tray type and/or an identity of the tray at step 810. Software module 812 provides marker detection, identifying the location of fiducial marks, such as white dots, on the surface of the tray. Software module 814 generates a slot grid onto the image based on the location of these fiducial marks, as discussed above.” Paragraph 00117 And figure 12)
Wu is analogous art in the same field of endeavor as the claimed invention. Wu is directed towards an intelligent storage solution (“FIG. 12 shows the exemplary software data flow 800 for operating a drawer vision system at run time to process images from a drawer vision system and characterize the contents of a sample tray.” Paragraph 00117). A person of ordinary skill in the art would have found it obvious to combine the system of Dein and DeLand with the intelligent storage system of Wu by utilizing Wu’s tray tags in combination with Dein’s markers, with the expectation that doing so would result in reducing the need for staff and additional equipment necessary to characterizing the contents of the tray (“It is desirable to ascertain various pieces of information relating to a tray, the tubes, and the tubes' location within the tray, such as, for example, the tray slots containing a tube; a tube's center point, diameter, and height; the tray's orientation within a drawer; whether a tube is covered with a cap or tube-top cup. It is desirable to obtain these and other pieces of information quickly, without expensive equipment, and without handling or touching the tubes. Accordingly, it is further desirable to have an efficient training and calibration scheme for the cameras and image processing used by a drawer vision system (DVS) that can accomplish these above goals.” Paragraph 0008).
Osborn teaches capturing an image of a reference tag on the tray ("It will be appreciated that each tray is associated with a predetermined set of instruments. Thus, as set out in more detail below, each tray has a barcode applied to it that it is associated with a particular set of instruments" page 9 paragraph 10 lines 1-3) and a 2D code that identifies a digital list of expected surgical tools on the tray ("It will be appreciated that each tray is associated with a predetermined set of instruments. Thus, as set out in more detail below, each tray has a barcode applied to it that it is associated with a particular set of instruments" page 9 paragraph 10 lines 1-3).
Osborne is analogous art in the same field of endeavor as the claimed invention. Osborne is directed towards a hospital equipment tracking system ("The instrument table 64 includes datastrings relating to trays of equipment" page 9 paragraph 10 line 1). A person of ordinary skill in the art would have found it obvious to combine the system of Dein, DeLand, and Wu with the administrative equipment tracking system of Osborne by utilizing Osborne's teaching of tray tags in connection with a database system in combination with Wu’s tray tagging system, with the expectation that doing so would lead to achieving consistent records of inventory and activity across an organization ("Furthermore, the invention provides a means whereby control of such activities and use of inventory is facilitated to achieve consistency across an entire organization" page 2 paragraph 8 lines 2-3).
Ma teaches wherein creating an augmented reality information that relates to the identity of each of the one or more surgical tools missing from the tray (“The computing device 138 can receive templates from the datastore 129 and display the video feed from the image capture device 102. In at least one embodiment, the computing device 138 can also have a bar code scanner. The computing device 138 can receive alerts communicated from the computing device 116. For example, if a surgical instrument is missing from an operating room” page 26 col.10 lines 34-41) is created by evaluating, via a machine learning algorithm, the image captured by the camera of the plurality of surgical tools on the tray (“…receive a test image from the image capture device corresponding to a first surgical instrument, determine an identity type of the first surgical instrument using the test image in a machine vision technique” page 22 col.1 lines 41-44) and outputting the augmented reality information in a field of view of a user (“The computing device 138 can receive templates from the datastore 129 and display the video feed from the image capture device 102. In at least one embodiment, the computing device 138 can also have a bar code scanner. The computing device 138 can receive alerts communicated from the computing device 116. For example, if a surgical instrument is missing from an operating room” page 26 col.10 lines 34-41)
Ma is analogous art in the same field of endeavor as the current invention. Ma is directed towards determining identities of surgical instruments (“…receive a test image from the image capture device corresponding to a first surgical instrument, determine an identity type of the first surgical instrument using the test image in a machine vision technique” page 22 col.1 lines 41-44). A person of ordinary skill in the art before the effective filing date of the claimed invention could have reasoned that combining the system of DeLand, Osborn, Wu and Dein with the augmented reality capabilities of Ma could lead to prompt retrieval of missing tools (“The technician can promptly retrieve the surgical instrument from one part of the hospital and deliver the instrument to the operating room” page 26 col.10 lines 43-45). Therefore, it would have been obvious for a person of ordinary skill in the art before the effective filing date of the claimed invention to combine the system of DeLand, Osborn, Wu, and Dein with the augmented reality capabilities of Ma, with the expectation that doing so could lead to prompt retrieval of missing tools (see page 26 col.10 lines 43-45)
Esterberg teaches creating an augmented reality information comprising an identity of each of the one or more surgical tools added to the tray (“In at least one example embodiment, an alert may be raised while the surgical instrument deviates from the surgical path based on pre-defined conditions stored in the memory 206. The pre-defined conditions may include, for example, a surgeon or surgery attendee holding a surgical instrument beyond a fixed step, usage of a wrong surgical instrument by a surgeon or surgery attendee, a missing surgical instrument, picking a surgical instrument not included in the plan for use at a particular step, picking up of a defective or contaminated surgical instrument, using a surgical instrument in a wrong place during the procedure, using a surgical instrument at an inappropriate time, deviation in alignment of the surgical instrument relative to a surgical path or other conditions that may cause or create an adverse event.” Paragraph 0127) and outputting the augmented reality information in a field of view of a user(“AR glasses 1116” paragraph 0105 and “At step 1504, the surgical path may be highlighted in the AR display 1161” paragraph 0129).
Esterberg is analogous art in the same field of endeavor as the claimed invention. Esterberg is directed towards a surgical instrument monitoring system (“In at least one example embodiment, an alert may be raised while the surgical instrument deviates from the surgical path based on pre-defined conditions stored in the memory 206. The pre-defined conditions may include, for example, a surgeon or surgery attendee holding a surgical instrument beyond a fixed step, usage of a wrong surgical instrument by a surgeon or surgery attendee, a missing surgical instrument, picking a surgical instrument not included in the plan for use at a particular step, picking up of a defective or contaminated surgical instrument, using a surgical instrument in a wrong place during the procedure, using a surgical instrument at an inappropriate time, deviation in alignment of the surgical instrument relative to a surgical path or other conditions that may cause or create an adverse event.” Paragraph 0127). A person of ordinary skill before the effective filing date of the claimed invention would have found it obvious to combine the system of DeLand, Osborn, Wu and Dein by using Esterberg’s additional alert teachings in combination with Ma’s AR alert system, with the expectation that doing so would result in better surgical outcomes by notifying the medical professional of or preventing adverse events (“In at least one example embodiment, an alert may be raised while the surgical instrument deviates from the surgical path based on pre-defined conditions stored in the memory 206. The pre-defined conditions may include, for example, a surgeon or surgery attendee holding a surgical instrument beyond a fixed step, usage of a wrong surgical instrument by a surgeon or surgery attendee, a missing surgical instrument, picking a surgical instrument not included in the plan for use at a particular step, picking up of a defective or contaminated surgical instrument, using a surgical instrument in a wrong place during the procedure, using a surgical instrument at an inappropriate time, deviation in alignment of the surgical instrument relative to a surgical path or other conditions that may cause or create an adverse event.” Paragraph 0127).
With respect to claim 2, DeLand, Dein, Wu, Osborne, Ma, and Esterberg teach the method of Claim 1. DeLand further teaches wherein the expected list of identities of surgical tools on the tray is based on a user input value (“In some embodiments, the instrument tracker stores a set of templates for procedures. The templates include a list of all of the surgical instruments required for a particular procedure. Before beginning to assemble the instruments for the procedure, the member of the medical staff selects the procedure from a list displayed by the surgical instrument tracker. As the surgical instruments are assembled on the surgical tray, the surgical instrument tracker recognizes the surgical instruments and checks off the surgical instruments listed in the template.” Paragraph 0046).
With respect to claim 4, DeLand, Dein, Wu, Osborne, Ma, and Esterberg teach the method of Claim 1. Dein further teaches wherein the step of processing the one or more images to determine identities of each surgical tool in the plurality of surgical tools on the tray comprises: pattern recognition of a shape and size of each surgical tool (“the shape recognition module may be configured to provide shape recognition of other surgical items, such as surgical instruments” page 8 paragraph 4 lines 2-3), a tag attached to each surgical tool (“Additional components of the intra-operative identification and tracking systems of the invention can include one or more detectors configured to recognize a unique identifier label on a surgical item (e.g., a surgical instrument)” page 12 lines 7-9), or a combination of both (“…beyond the automated shape recognition of sharp objects, e.g., as described above. These additional functionalities may include identification (and in some instances tracking) of surgical items beyond surgical sharp objects, such as instruments and sponges. Such additional functionalities may --> employ the use of unique identifiers …” page 12 lines 1-4)
With respect to claim 7, DeLand, Dein, Wu, Osborne, Ma, and Esterberg teach the method of Claim 1. Dein further teaches wherein the step of capturing one or more images of the plurality of surgical tools on the tray comprises receiving image data from the camera, wherein the camera comprises a body mounted camera (“Any suitable system for mounting the reader/cameras for the automation of counts of surgical items can be used” page 23 SYSTEM CONFIGURATIONS paragraph 2 lines 12-13, includes body mounting system) or a room mounted camera (“For example, one camera 1360 can be mounted on a control arm mount 1362 attached to the ceiling or wall above or to the side of the sterile field.” page 23 SYSTEM CONFIGURATIONS paragraph 2 lines 8-9).
With respect to claim 13, DeLand, Dein, Wu, Osborne, Ma, and Esterberg teach the method of Claim 1. DeLand further teaches wherein the steps of capturing, processing, and recording are completed substantially continuously (“The surgical instrument tracker may also store the positions of each instrument during the surgical procedure.” Paragraph 0065 lines 11-13) as the plurality of surgical tools are transported throughout the perioperative environment (“Thus, for example, the surgical instrument tracker may store a first list of all the instruments on the surgical tray, a second list of all the instruments on the back tray, and a third list of all instruments at the operating table. When a surgical instrument is removed from one of the surgical tray or the back tray, the above lists may be updated.” Paragraph 0065 lines 13-18).
With respect to claim 17, DeLand, Dein, Wu, Osborne, Ma, and Esterberg teach the method of Claim 1. Ma further teaches wherein the augmented reality information that relates to the identity of each of the one or more surgical tools missing from the tray (“The computing device 138 can receive templates from the datastore 129 and display the video feed from the image capture device 102. In at least one embodiment, the computing device 138 can also have a bar code scanner. The computing device 138 can receive alerts communicated from the computing device 116. For example, if a surgical instrument is missing from an operating room” page 26 col.10 lines 34-41) is created by evaluating, via a machine learning algorithm, the image captured by the camera of the plurality of surgical tools on the tray (“…receive a test image from the image capture device corresponding to a first surgical instrument, determine an identity type of the first surgical instrument using the test image in a machine vision technique” page 22 col.1 lines 41-44) and the method further comprises: outputting the augmented reality information(“The computing device 138 can receive templates from the datastore 129 and display the video feed from the image capture device 102. In at least one embodiment, the computing device 138 can also have a bar code scanner. The computing device 138 can receive alerts communicated from the computing device 116. For example, if a surgical instrument is missing from an operating room” page 26 col.10 lines 34-41) such that, in response to a user placing the tray in a field of view of the user, an identity, correct location, or both the identity and correct location of the one or more surgical tools missing from is perceivable in the field of view of the user (“The computing device 138 can receive alerts communicated from the computing device 116. For example, if a surgical instrument is missing from an operating room, then the computing device 116 can alert computing device 138 which is carried by a technician. The technician can promptly retrieve the surgical instrument from one part of the hospital and deliver the instrument to the operating room” page 26 col.10 lines 38-45).
Esterberg further teaches wherein the augmented reality information that relates to the identity of each of the one or more surgical tools added to the operating room (“At step 1504, the surgical path may be highlighted in the AR display 1161” paragraph 0129) is created and the method further comprises: outputting the augmented reality information such that, an identity, correct location, or both the identity and correct location of the one or more surgical tools added to the operating room (“In at least one example embodiment, an alert may be raised while the surgical instrument deviates from the surgical path based on pre-defined conditions stored in the memory 206. The pre-defined conditions may include, for example, a surgeon or surgery attendee holding a surgical instrument beyond a fixed step, usage of a wrong surgical instrument by a surgeon or surgery attendee, a missing surgical instrument, picking a surgical instrument not included in the plan for use at a particular step, picking up of a defective or contaminated surgical instrument, using a surgical instrument in a wrong place during the procedure, using a surgical instrument at an inappropriate time, deviation in alignment of the surgical instrument relative to a surgical path or other conditions that may cause or create an adverse event.” Paragraph 0127) is perceivable in the field of view of the user (“AR glasses 1116” paragraph 0105 and “At step 1504, the surgical path may be highlighted in the AR display 1161” paragraph 0129).
Claims 6, and 11-12 are rejected under 35 U.S.C. 103 as being unpatentable over DeLand, Dein, Wu, Osborne, Ma, and Esterberg as applied to claim 1 above, and further in view of Broninx (US 10614412 B2, published 2020-04-07).
With respect to claim 6, DeLand, Dein, Wu, Osborne, Ma, and Esterberg teach the method of Claim 1. DeLand further teaches wherein the step of processing the one or more images to determine identities of each surgical tool in the plurality of surgical tools on the tray comprises: comparing the one or more images of the plurality of surgical tools to stored images of a plurality of reference surgical tools (“the surgical instrument recognition unit 315 comprises a memory that stores data and images used to recognize the surgical instruments in the images or video data” paragraph 0032 lines 6-9), but does not teach generating a comparison score for each of the plurality of surgical tools based on the comparing step, wherein the identity of each of the plurality of surgical tools is assigned as an identity of the reference surgical tool having the highest comparison score.
Broninx teaches generating a comparison score for each of a plurality of surgical tools based on the comparing step (“…confidence score for the identification of the instrument” page 7 col.4 line 11), wherein the identity of each of a plurality of surgical tools is assigned as an identity of the reference surgical tool having the highest comparison score (“The determination of the identification data then consists of selecting the candidate with the smallest “distance”; that is the candidate which has the best confidence score for the identification of the instrument” page 7 col. 4 lines 9-12).
Broninx is analogous art in the same field of endeavor as the claimed invention. Broninx is directed towards the “tracking of surgical instruments in a medical environment such as a hospital setting” (page 6 col 1. Lines 9-10). A person of ordinary skill in the art before the effective filing date of the claimed invention could have reasoned that combining the multi-tray instrument tracking system of DeLand, Dein, Wu, Osborne, Ma, and Esterberg with the tracing and tracking system of Broninx by assigning information relating to the use of a tool (such as sterilization) to the specific ID and utilizing a confidence score heuristic as disclosed by Broninx, could lead to guarantees of correctness and completeness of instruments kits (“Thus, the present invention, by combining its functional and structural characteristics, enables the tracking and tracing of surgical instruments in a hospital setting while guaranteeing in particular the correctness and completeness of the instruments positioned in a kit.” Page 9 col. 7 lines 65-67 and col. 8 lines 1-2). Therefore it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the multi-tray instrument tracking system of DeLand, Dein, Wu, Osborne, Ma, and Esterberg with the tracing and tracking system of Broninx by assigning information relating to the use of a tool (such as sterilization) to the specific ID and utilizing a confidence score heuristic as disclosed by Broninx with the expectation that doing so would lead to guarantees of correctness and completeness of instruments kits (see Page 9 col. 7 lines 65-67 and col. 8 lines 1-2).
With respect to claim 11, DeLand, Dein, Wu, Osborne, Ma, and Esterberg teach the method of Claim 1, DeLand further teaches wherein the steps of capturing, processing, and recording are completed at more than one location in the perioperative environment, wherein the perioperative environment includes at least an operating room (“The additional cameras may be positioned to view any position of the operating room where surgical instruments can be placed. The additional cameras can be used, as discussed below for the cameras 235, 240, 245 to track surgical instruments in the operating room.” Paragraph 0028 lines 3-7) and an assembly area (“If a tracking mode is selected the surgical instrument tracker is placed in a mode in which the image from the camera is used to recognize and count the surgical instruments as the surgical instruments are placed on the surgical tray…” paragraph 0040 lines 6-10). DeLand does not teach the perioperative environment including a decontamination area.
Broninx teaches the steps of capturing, processing, and recording completed at a decontamination area (“…or to report an instrument when the latter has been sterilised prior to the surgical intervention.” Page 8 col. 6 lines 43-44).
Broninx is analogous art in the same field of endeavor as the claimed invention. Broninx is directed towards the “tracking of surgical instruments in a medical environment such as a hospital setting” (page 6 col 1. Lines 9-10). A person of ordinary skill in the art before the effective filing date of the claimed invention could have reasoned that combining the multi-tray instrument tracking system of DeLand, Dein, Wu, Osborne, Ma, and Esterberg with the tracing and tracking system of Broninx by assigning information relating to the use of a tool (such as sterilization) to the specific ID and utilizing a confidence score heuristic as disclosed by Broninx, could lead to guarantees of correctness and completeness of instruments kits (“Thus, the present invention, by combining its functional and structural characteristics, enables the tracking and tracing of surgical instruments in a hospital setting while guaranteeing in particular the correctness and completeness of the instruments positioned in a kit.” Page 9 col. 7 lines 65-67 and col. 8 lines 1-2). Therefore it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the multi-tray instrument tracking system of DeLand, Dein, Wu, Osborne, Ma, and Esterberg with the tracing and tracking system of Broninx by assigning information relating to the use of a tool (such as sterilization) to the specific ID and utilizing a confidence score heuristic as disclosed by Broninx with the expectation that doing so would lead to guarantees of correctness and completeness of instruments kits (see Page 9 col. 7 lines 65-67 and col. 8 lines 1-2).
With respect to claim 12, DeLand, Dein, Wu, Osborne, Ma, Esterberg and Broninx teach the method of Claim 11. DeLand further teaches the method of claim 11 wherein the steps of capturing, processing, and recording are completed at each of the operating room (“The additional cameras may be positioned to view any position of the operating room where surgical instruments can be placed. The additional cameras can be used, as discussed below for the cameras 235, 240, 245 to track surgical instruments in the operating room.” Paragraph 0028 lines 3-7) and the assembly area (“If a tracking mode is selected the surgical instrument tracker is placed in a mode in which the image from the camera is used to recognize and count the surgical instruments as the surgical instruments are placed on the surgical tray…” paragraph 0040 lines 6-10). DeLand does not teach the method of claim 11 wherein the steps of capturing, processing, and recording are completed at a decontamination area.
Broninx teaches where the steps of capturing, processing, and recording are completed at a decontamination area (“…or to report an instrument when the latter has been sterilised prior to the surgical intervention.” Page 8 col. 6 lines 43-44).
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over DeLand, Dein, Wu, Osborne, Ma, and Esterberg as applied to claim 1 and further in view of Dor (US 10187742 B2, published: 2019-01-22, filed: 2015-01-19).
With respect to claim 10, DeLand, Dein, Wu, Osborne, Ma, and Esterberg teach the method of Claim 1. Dein further teaches wherein the step of processing the one or more images to determine the identity of each surgical tool in the plurality of surgical tools on the tray comprises: recognizing a reference tag positioned on each of the surgical tools (“Additional components of the intra-operative identification and tracking systems of the invention can include one or more detectors configured to recognize a unique identifier label on a surgical item (e.g., a surgical instrument….” Page 12 lines 7-8), but does not teach accessing a reference list database comprising a surgical tool identity related to the reference tag.
Dor teaches accessing a reference list database comprising a surgical tool identity related to a reference tag (“In step 412, central server 110 may identify or determine a matching data record in a tag database, according to the tag identification data received with the flag event. The flag event comprises a unique tag identification which enables the central server 110 to identify the tag 104 and the tool 102 or tool set 107 for which the flag event is received.” Page 14 col. 12 lines 52-57).
Dor is analogous art in the same field of endeavor as the claimed invention. Dor is directed towards “a system and method for real time tracking of one or more surgical tools across one or more locations” (page 9 col. 1 lines 6-8). A person of ordinary skill in the art could have reasoned that combining the system of DeLand, Dein, Wu, Osborne, Ma, and Esterberg with Dor by utilizing Dor’s tag database to store Dein’s tags, could lead to efficient and accurate monitoring and tracking of tools (“Thus, it is desirable to provide a solution for efficient monitoring and tracking of tools, in case an infection control or disease control event is detected.” Page 9 col.1 lines 34-36), improving patient health outcomes by avoiding infections and other hazardous events arising from improperly maintaining and managing tools (“Thus monitoring and tracking becomes even more critical when the tools are improperly maintained and then moved around both within an institution and between multiple institutions. When the tools are improperly cleaned, washed, sterilized, etc., they risk causing contaminations, infections, and other hazardous events.” Page 10 col.4 lines 1-6). Therefore it would have been obvious for a person of ordinary skill in the art before the effective filing date of the claimed invention to combine the system of DeLand, Dein, Wu, Osborne, Ma, and Esterberg with Dor by utilizing Dor’s tag database to store Dein’s tags, with the expectation that doing so would lead to efficient and accurate monitoring and tracking of tools (see Page 9 col.1 lines 34-36), improving patient health outcomes by avoiding infections and other hazardous events arising from improperly maintaining and managing tools (see Page 10 col.4 lines 1-6).
Claims 18 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over DeLand, Osborn, Wu, Dein, Ma, and Esterberg in view of Ronen (US-20190328460-A1, published: 2019-10-31) and White (US 20180082480 A1, published: 2018-03-22, filed: 2016-09-16).
With respect to claim 18, DeLand, Dein, Wu, Osborne, Ma, and Esterberg teach the method of claim 17, but do not teach wherein the identity, correct location, or both the identity and correct location of the one or more surgical tools missing from or added to the tray is provided as an overlaid image on the assembled tray in the field of view of the user, wherein the identity, correct location, or both the identity and correct location of the one or more surgical tools missing from or added to the tray is based on a surgeon preference list, a procedure type, or both .
Ronen teaches wherein the identity, correct location, or both the identity and correct location of the one or more surgical tools missing from or added to the tray is based on a surgeon preference list, a procedure type, or both (“Further, a machine learning system may be used to identify an instrument set based upon various inputs such as the procedure and surgeon, as selected surgeons may select different instruments and/or a surgeon's preference (e.g. pedicle screw size) may vary or change a selected instrument set” paragraph 0090 lines 13-18)
Ronen is analogous art in the same field of endeavor as the claimed invention. Ronen is directed toward the tracking of surgical instruments or objects during surgical procedures or assembly (“It is understood, however, that the instrument 16 may be tracked and/or navigated relative to any subject for any appropriate procedure. Tracking or navigating an instrument for a procedure, such as a surgical procedure, on a human or living subject is merely exemplary.” Paragraph 0029 lines 13-18). A person of ordinary skill in the art before the effective filing date of the claimed invention could have reasoned that combining the multi-tray tracking capabilities of DeLand and Ma’s augmented reality system with the machine learning tracking system of Ronen by implementing Ronen’s machine learning model into DeLand, Osborn, Dein, Wu, Esterberg and Ma’s system utilizing DeLand’s multi-camera set up and Ma’s augmented reality capabilities, could lead to reduced or higher speed user interactions (“Accordingly, a workflow may be automated or have selected user interaction, such as reduced or faster, in performing selected and/or standard portions of a selected procedure.” Paragraph 0011 lines 14-16) that allow professionals to efficiently and effectively perform a procedure (“A selected workflow may be used to efficiently and effectively perform a procedure.” Paragraph 0011 lines 1-2). Therefore, it would have been obvious before the effective filing date of the claimed invention for a person of ordinary skill in the art, to combine the multi-tray tracking capabilities of DeLand and Ma’s augmented reality system with the machine learning tracking system of Ronen by implementing Ronen’s machine learning model into DeLand’s, Osborn, Dein, Wu, Esterberg and Ma’s system utilizing DeLand’s multi-camera set up and Ma’s augmented reality capabilities, with the expectation that doing so would lead to reduced or higher speed user interactions (see Paragraph 0011 lines 14-16) that allow professionals to efficiently and effectively perform a procedure (see Paragraph 0011 lines 1-2).
White teaches wherein the identity, correct location, or both the identity and [[/or]] correct location of the one or more surgical tools missing from (“The AR display 300 may display likely locations of items that may be missing.” Paragraph 0049 lines 13-14) or added to the tray (“An alert may be displayed to the surgeon (or the surgical team in general) when a count does not match up. The AR display 300 may display the alert or the alert” paragraph 0049) is provided as an overlaid image on the assembled tray in the field of view of the user (“In an example, the virtual representation of a surgical instrument 202 may include a particular location in the AR display 200A. For example, the virtual representation of a surgical instrument 202 may hover over the case 205 or tray containing the physical surgical instrument (corresponding to the virtual representation of the surgical instrument 202).” Paragraph 0029 lines 1-7 )
White is analogous art in the same field of endeavor as the claimed invention. White is directed toward system for using augmented reality during surgery, having a display that can highlight recognized tools (“…the AR display 200A may direct a surgical assistant to a specific virtual surgical instrument 220 by highlighting the virtual instrument 220…” paragraph 0030 lines 1-3). A person of ordinary skill in the art before the effective filing date of the claimed invention could have reasoned that combing the system of DeLand, Dein, Wu, Osborne, Ma, Esterberg and Ronen with White by utilizing White’s overlays on Ma’s augmented reality devices, could lead to a more accurate surgical procedure by allowing the surgeon to focus on the surgery as opposed to otherwise having to split focus between the procedure and outside information (“…the AR display 1102 presents a mixed reality or augmented reality for the surgeon to more accurately perform the surgical procedure. By using the AR display 1102, the surgeon is allowed to focus on the surgical procedure without needing to direct attention away from the surgical procedure to consult information.” Paragraph 0085 lines 16-22). Therefore, it would have been obvious for a person of ordinary skill in the art before the effective filing date of the claimed invention to combine the system of DeLand, Dein, Wu, Osborne, Ma, Esterberg and Ronen with White by utilizing White’s overlays on Ma’s augmented reality devices, with the expectation that doing so would lead to a more accurate surgical procedure by allowing the surgeon to focus on the surgery as opposed to otherwise having to split focus between the procedure and outside information (See Paragraph 0085 lines 16-22)
With respect to claim 20, DeLand, Dein, Wu, Osborne, Ma, Esterberg, Ronen, and White teach the method of claim 18. DeLand further teaches the method of claim 18, wherein the surgeon preference list includes a list of surgical tools selected by a surgeon (“Before beginning to assemble the instruments for the procedure, the member of the medical staff selects the procedure from a list displayed by the surgical instrument tracker” Paragraph 0046 lines 4-7). Ronen further teaches the method of claim 18, wherein the surgeon preference list includes a list of surgical tools determined by the machine learning algorithm (“Further, a machine learning system may be used to identify an instrument set based upon various inputs such as the procedure and surgeon, as selected surgeons may select different instruments and/or a surgeon's preference (e.g. pedicle screw size) may vary or change a selected instrument set” paragraph 0090 lines 13-18).
Claim 21 is rejected under 35 U.S.C. 103 as being unpatentable over DeLand Osborn, Dein, Wu, Ma, Esterberg, Ronen, and White as applied to claim 18 above, and further in view of Chin (Chin CJ, Sowerby LJ, John-Baptiste A, Rotenberg BW. Reducing otolaryngology surgical inefficiency via assessment of tray redundancy. Journal of Otolaryngology - Head & Neck Surgery. 2014;43(1).).
With respect to claim 21, DeLand, Dein, Wu, Osborne, Ma, Esterberg, Ronen, and White teach the method of claim 18. Ronen further teaches the method of claim 18, wherein the list of surgical tools determined by the machine learning algorithm (“Further, a machine learning system may be used to identify an instrument set based upon various inputs such as the procedure and surgeon, as selected surgeons may select different instruments and/or a surgeon's preference (e.g. pedicle screw size) may vary or change a selected instrument set” paragraph 0090 lines 13-18). However, DeLand, Ma, Ronen and White do not teach wherein the list of surgical tools determined by the machine learning algorithm is identified by a use rate of individual surgical tools in the plurality of surgical tools during a procedure in an aseptic environment.
Chin teaches where a list of surgical tools is identified by a use rate of individual surgical tools in the plurality of surgical tools during a procedure in an aseptic environment (“The total IUR for all identical cases was then calculated. If an instrument was used at least one out of every five times (20% of the time) the tray was opened, it was felt this would merit inclusion on a new optimized tray for that particular procedure.” Page 2 col.2 lines 20-24).
Chin is analogous art in the same field of endeavor as the claimed invention. Chin is directed toward surgical tool selection (“Many parts of the sterilization process are fixed times, regardless of the number of instruments to be processed. The most variable part, which is when the instrument
trays are rebuilt, was the focus of this study.” Page 2 lines 17-20). A person of ordinary skill in the art before the effective filing date of the claimed invention could have reasoned that incorporating the tray optimization strategy of Chin into the instrument image recognition and display system of DeLand, Dein, Wu, Osborne, Ma, Esterberg, Ronen, and White could lead to lowered cost and improved operating room efficiency (“The evidence suggests that substantial room exists to optimize tray efficiency as a novel means of improving Operating Room efficiency. Systematically assessing the utility of individual instruments could allow for a subsequent reduction in the number of instruments on surgical trays without impacting the surgery or surgeon. This could translate into direct and impactful cost containment in the CPD” page 2 lines 6-13). Therefore, it would have been obvious for a person of ordinary skill in the art before the effective filing date of the claimed invention to combine DeLand, Dein, Wu, Osborne, Ma, Esterberg, Ronen, and White, by incorporating the tray optimization strategy of Chin into the instrument image recognition and display system of DeLand, Osborn, Dein, Wu, Ma, Esterberg, Ronen, and White with the expectation that doing so would lead to lowered cost and improved operating room efficiency (See page 2 lines 6-13)
Claims 22-23 are rejected under 35 U.S.C. 103 as being unpatentable over DeLand, Dein, Wu, Osborne, Ma, and Esterberg as applied to claim 1, and further in view of Aldridge (US 12144136 B2, filed: 2019-09-05).
With respect to claim 22, DeLand, Dein, Wu, Osborne, Ma, and Esterberg teach the method of Claim 1. Deland further teaches the method of Claim 1, wherein the step of capturing one or more images of the plurality of surgical tools is performed in an operating room prior to initiation of an aseptic procedure (“If a tracking mode is selected the surgical instrument tracker is placed in a mode in which the image from the camera is used to recognize and count the surgical instruments as the surgical instruments are placed on the surgical tray and the method proceeds to step 420.” Paragraph 0040 lines 6-10), and the method further comprises: during the aseptic procedure, repeating the capturing and processing steps substantially in real-time (“Thus, for example, the surgical instrument tracker may store a first list of all the instruments on the surgical tray, a second list of all the instruments on the back tray, and a third list of all instruments at the operating table. When a surgical instrument is removed from one of the surgical tray or the back tray, the above lists may be updated.” Paragraph 0065 lines 13-18), and recording an identity of each surgical tool removed from the plurality of surgical tools on the tray (“When the surgeon is finished with a particular instrument 110, the surgical instrument 110 may be placed on a back tray if contaminated or not required again in the surgical procedure” paragraph 0023 lines 7-10 and “…the surgical instrument tracker also records video from all the cameras in the operating room, along with the time of each frame in the videos. Moreover, each time a surgical instrument is placed back on the surgical tray, or the back tray, for example, back tray 230, the surgical instrument tracker identifies the surgical instrument and records the time that the surgical instrument was placed on the surgical tray or the back tray” paragraph 0057 lines 16-23). DeLand does not teach recording an identity of each surgical tool removed from the plurality of surgical tool to form a list of surgical tools used during the aseptic procedure.
Aldridge teaches recording an identity of each surgical tool removed from the plurality of surgical tool to form a list of surgical tools used during the aseptic procedure (“…compare the steps being performed or the equipment being used during the course of the surgical procedure to the expected steps or equipment for the type of surgical procedure that the surgical hub 5104 determined is being performed.” Page 74 col.51 lines 42-46).
Aldridge is analogous art in the same field of endeavor as the claimed invention. Aldridge is directed towards surgical systems (“The present disclosure relates to various surgical systems, including modular electrosurgical and/or ultrasonic surgical systems. Operating rooms (ORs) are in need of streamlined capital solutions” page 49 col.1 lines 34-37). A person of ordinary skill in the art before the effective filing date of the claimed invention could have reasoned that combining, DeLand’s multi-tray instrument tracking system, from the combined system of DeLand, Dein, Wu, Osborne, Ma, and Esterberg, with Aldridge’s used and unused tool comparison could lead to improved “surgical staff efficiency during a surgical procedure by reducing the number of devices that surgical staff members need to interact with” (page 49 col.1 lines 16-18). Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of DeLand and Aldridge by utilizing DeLand’s multi-tray instrument tracking system from the combined system of DeLand, Dein, Wu, Osborne, Ma, and Esterberg , with Aldridge’s used and unused tool comparison with the expectation that doing so would lead to more efficient surgical procedures (see page 49 col.1 lines 16-18)
With respect for claim 23, DeLand, Dein, Wu, Osborne, Ma, and Esterberg and Aldridge teach the method of claim 22. Aldridge further teaches the method of claim 22, further comprising: after the aseptic procedure, comparing the list of surgical tools used during the aseptic procedure to the initial list to form a list of unused surgical tools (“…compare the steps being performed or the equipment being used during the course of the surgical procedure to the expected steps or equipment for the type of surgical procedure that the surgical hub 5104 determined is being performed.” Page 74 col.51 lines 42-46).
Claim 24 is rejected under 35 U.S.C. 103 as being unpatentable over DeLand in view of Aldridge, Dein, Wu, and Osborn.
With respect to claim 24, DeLand teaches A method for optimizing tool usage in a perioperative environment,the method comprising:prior to initiation of an aseptic procedure, capturing one or more images of a plurality of surgical tools on a tray (“images produced by the one or more cameras” paragraph 0012 line 6), processing the one or more images to determine an identity of each surgical tool in the plurality of surgical tools on the tray (“A recognition unit adapted to recognize identities and positions of instruments or tools in images produced by the one or more cameras.” Paragraph 0012 lines 4-6), and recording the determined identity of each surgical tool in the plurality of surgical tools to create an initial list of surgical tools (“The memory unit 330 stores the positions and the identities of surgical instruments” paragraph 0035 lines 1-2); during the aseptic procedure, repeating the capturing and processing steps substantially in real-time (“Thus, for example, the surgical instrument tracker may store a first list of all the instruments on the surgical tray, a second list of all the instruments on the back tray, and a third list of all instruments at the operating table. When a surgical instrument is removed from one of the surgical tray or the back tray, the above lists may be updated.” Paragraph 0065 lines 13-18), and recording an identity of each surgical tool removed from the plurality of surgical tools to form a list of surgical tools used during the aseptic procedure (“When the surgeon is finished with a particular instrument 110, the surgical instrument 110 may be placed on a back tray if contaminated or not required again in the surgical procedure” paragraph 0023 lines 7-10 and “…the surgical instrument tracker also records video from all the cameras in the operating room, along with the time of each frame in the videos. Moreover, each time a surgical instrument is placed back on the surgical tray, or the back tray, for example, back tray 230, the surgical instrument tracker identifies the surgical instrument and records the time that the surgical instrument was placed on the surgical tray or the back tray” paragraph 0057 lines 16-23). DeLand does not teach after the aseptic procedure, comparing the list of surgical tools used during the aseptic procedure to the initial list to form a list of unused surgical tools.
DeLand does not teach after the aseptic procedure, comparing the list of surgical tools used during the aseptic procedure to the initial list to form a list of unused surgical tools and wherein the step of capturing the one or more images further comprises capturing an image of a reference tag on the tray, wherein the reference tag provides a reference grid for obtaining dimensions of the plurality of surgical tools, and a 2D code that identifies a digital list of expected surgical tools on the tray.
Aldridge teaches after the aseptic procedure, comparing the list of surgical tools used during the aseptic procedure to the initial list to form a list of unused surgical tools (“…compare the steps being performed or the equipment being used during the course of the surgical procedure to the expected steps or equipment for the type of surgical procedure that the surgical hub 5104 determined is being performed.” Page 74 col.51 lines 42-46).
Aldridge is analogous art in the same field of endeavor as the claimed invention. Aldridge is directed towards surgical systems (“The present disclosure relates to various surgical systems, including modular electrosurgical and/or ultrasonic surgical systems. Operating rooms (ORs) are in need of streamlined capital solutions” page 49 col.1 lines 34-37). A person of ordinary skill in the art before the effective filing date of the claimed invention could have reasoned that combining DeLand’s multi-tray instrument tracking system with Aldridge’s used and unused tool comparison could lead to improved “surgical staff efficiency during a surgical procedure by reducing the number of devices that surgical staff members need to interact with” (page 49 col.1 lines 16-18). Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of DeLand and Aldridge by utilizing DeLand’s multi-tray instrument tracking system with Aldridge’s used and unused tool comparison with the expectation that doing so would lead to more efficient surgical procedures (see page 49 col.1 lines 16-18)
Dein teaches a reference grid (“…the "mayo stand" or sterile surgical back table can have markers which delineate a grid on the table, such that markers can outline areas where a particular camera/reader may scan” page 23 SYSTEM CONFIGURATIONS paragraph 2 lines 5-6) for obtaining dimensions of the plurality of surgical tools (“In addition to the intra-operative imaging device, the systems include a surgical sharp object automated shape recognition module.” Page 6 paragraph 3 lines 1-2 and “Automated shape recognition is a process by which the identification of surgical sharp objects is performed using one or more of the size, shape, aspect ratio, --> outline, color, or other distinctive feature present in the image data of the surgical sharp object.” Paragraph 4 lines 4-6)
Dein is analogous art in the same field of endeavor as the claimed invention. Dein is directed towards a system for tracking surgical tools during an operation (“Additional components of the intra-operative identification and tracking systems of the invention…” page 12 line 7). A person of ordinary skill in the art before the effective filing date of the claimed invention could have reasoned that combining the system of DeLand with Dein by incorporating Dein’s shape and pattern recognition module could lead to improved accuracy in counts and reductions in errors leading to patient safety enhancements (“The accuracy and designation of responsibility for the counts and decisions affected by the count are all improved by the subject methods and system. --> Most importantly, the chances for errors are reduced and the likelihood of incorrect counts reduced. Safety is greatly enhanced for the patient…” page 29 lines 18-21). Therefore, it would have been obvious for a person of ordinary skill in the art before the effective filing date of the claimed invention to combine the system of DeLand with Dein by incorporating Dein’s shape and pattern recognition module with the expectation that doing so would lead to improved accuracy in counts and reductions in errors leading to patient safety enhancements (see page 29 lines 18-21).
Wu teaches capturing an image of a reference tag on the tray (“Software module 806 detects barcodes on the surface of the sample trays” paragraph 00117), wherein the reference tag provides a reference grid (“Software module 806 detects barcodes on the surface of the sample trays. This allows software to determine a tray type and/or an identity of the tray at step 810. Software module 812 provides marker detection, identifying the location of fiducial marks, such as white dots, on the surface of the tray. Software module 814 generates a slot grid onto the image based on the location of these fiducial marks, as discussed above.” Paragraph 00117 And figure 12)
Wu is analogous art in the same field of endeavor as the claimed invention. Wu is directed towards an intelligent storage solution (“FIG. 12 shows the exemplary software data flow 800 for operating a drawer vision system at run time to process images from a drawer vision system and characterize the contents of a sample tray.” Paragraph 00117). A person of ordinary skill in the art would have found it obvious to combine the system of Dein and DeLand with the intelligent storage system of Wu by utilizing Wu’s tray tags in combination with Dein’s markers, with the expectation that doing so would result in reducing the need for staff and additional equipment necessary to characterizing the contents of the tray (“It is desirable to ascertain various pieces of information relating to a tray, the tubes, and the tubes' location within the tray, such as, for example, the tray slots containing a tube; a tube's center point, diameter, and height; the tray's orientation within a drawer; whether a tube is covered with a cap or tube-top cup. It is desirable to obtain these and other pieces of information quickly, without expensive equipment, and without handling or touching the tubes. Accordingly, it is further desirable to have an efficient training and calibration scheme for the cameras and image processing used by a drawer vision system (DVS) that can accomplish these above goals.” Paragraph 0008).
Osborn teaches capturing an image of a reference tag on the tray ("It will be appreciated that each tray is associated with a predetermined set of instruments. Thus, as set out in more detail below, each tray has a barcode applied to it that it is associated with a particular set of instruments" page 9 paragraph 10 lines 1-3) and a 2D code that identifies a digital list of expected surgical tools on the tray ("It will be appreciated that each tray is associated with a predetermined set of instruments. Thus, as set out in more detail below, each tray has a barcode applied to it that it is associated with a particular set of instruments" page 9 paragraph 10 lines 1-3).
Osborne is analogous art in the same field of endeavor as the claimed invention. Osborne is directed towards a hospital equipment tracking system ("The instrument table 64 includes datastrings relating to trays of equipment" page 9 paragraph 10 line 1). A person of ordinary skill in the art would have found it obvious to combine the system of Dein, DeLand, and Wu with the administrative equipment tracking system of Osborne by utilizing Osborne's teaching of tray tags in connection with a database system in combination with Wu’s tray tagging system, with the expectation that doing so would lead to achieving consistent records of inventory and activity across an organization ("Furthermore, the invention provides a means whereby control of such activities and use of inventory is facilitated to achieve consistency across an entire organization" page 2 paragraph 8 lines 2-3).
Claims 26-27 are rejected under 35 U.S.C. 103 as being unpatentable over DeLand, Aldridge, Dein, Wu, Osborn, and Broninx.
With respect to claim 26, DeLand, Aldridge, Dein, Wu, and Osborn teach the method of claim 24. Deland further teaches the method of Claim 24, wherein the step of processing the one or more images to determine the identity of each surgical tool in the plurality of surgical tools comprises: comparing the one or more images of the plurality of surgical tools to stored images of a plurality of reference surgical tools (“the surgical instrument recognition unit 315 comprises a memory that stores data and images used to recognize the surgical instruments in the images or video data” paragraph 0032 lines 6-9). DeLand does not teach generating a comparison score for each of the plurality of surgical tools based on the comparing step, wherein the identity of each of the plurality of surgical tools is assigned as an identity of the reference surgical tool having the highest comparison score.
Broninx teaches generating a comparison score for each of the plurality of surgical tools based on the comparing step (“…confidence score for the identification of the instrument” page 7 col.4 line 11), wherein the identity of each of a plurality of surgical tools is assigned as an identity of the reference surgical tool having the highest comparison score (“The determination of the identification data then consists of selecting the candidate with the smallest “distance”; that is the candidate which has the best confidence score for the identification of the instrument” page 7 col. 4 lines 9-12).
Broninx is analogous art in the same field of endeavor as the claimed invention. Broninx is directed towards the “tracking of surgical instruments in a medical environment such as a hospital setting” (page 6 col 1. Lines 9-10). A person of ordinary skill in the art before the effective filing date of the claimed invention could have reasoned that combining the multi-tray instrument tracking system of DeLand, Aldridge, Dein, Wu, and Osborn, with the tracing and tracking system of Broninx by assigning information relating to the use of a tool (such as sterilization) to the specific ID and utilizing a confidence score heuristic as disclosed by Broninx, could lead to guarantees of correctness and completeness of instruments kits (“Thus, the present invention, by combining its functional and structural characteristics, enables the tracking and tracing of surgical instruments in a hospital setting while guaranteeing in particular the correctness and completeness of the instruments positioned in a kit.” Page 9 col. 7 lines 65-67 and col. 8 lines 1-2). Therefore it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the multi-tray instrument tracking system of DeLand, Aldridge, Dein, Wu, and Osborn, with the tracing and tracking system of Broninx by assigning information relating to the use of a tool (such as sterilization) to the specific ID and utilizing a confidence score heuristic as disclosed by Broninx with the expectation that doing so would lead to guarantees of correctness and completeness of instruments kits (see Page 9 col. 7 lines 65-67 and col. 8 lines 1-2).
With respect to claim 27, DeLand, Aldridge, Dein, Wu, Osborn, and Broninx teach the method of claim 26. Dein further teaches the step of capturing one or more images of a plurality of surgical tools comprising receiving image data from a cameras wherein the camera comprises a body mounted camera (“Any suitable system for mounting the reader/cameras for the automation of counts of surgical items can be used” page 23 SYSTEM CONFIGURATIONS paragraph 2 lines 12-13, includes body mounting system) or a room mounted camera (“For example, one camera 1360 can be mounted on a control arm mount 1362 attached to the ceiling or wall above or to the side of the sterile field.” page 23 SYSTEM CONFIGURATIONS paragraph 2 lines 8-9)
Claim 30 is rejected under 35 U.S.C. 103 as being unpatentable over DeLand, Aldridge, Dein, Wu, and Osborn as applied to claim 24 above, and further in view of Dor.
With respect to claim 30, DeLand, Aldridge, Dein, Wu, and Osborn teach the method of claim 24, Dein further teaches where the step of processing the one or more images to determine the identity of each surgical tool in the plurality of surgical tools comprises: recognizing a reference tag positioned on each of the surgical tools (“Additional components of the intra-operative identification and tracking systems of the invention can include one or more detectors configured to recognize a unique identifier label on a surgical item (e.g., a surgical instrument)” page 12 lines 7-9), but does not teach accessing a reference list database comprising a surgical tool identity related to the reference tag
Dor teaches accessing a reference list database comprising a surgical tool identity related to the reference tag (“In step 412, central server 110 may identify or determine a matching data record in a tag database, according to the tag identification data received with the flag event. The flag event comprises a unique tag identification which enables the central server 110 to identify the tag 104 and the tool 102 or tool set 107 for which the flag event is received.” Page 14 col. 12 lines 52-57).
Dor is analogous art in the same field of endeavor as the claimed invention. Dor is directed towards “a system and method for real time tracking of one or more surgical tools across one or more locations” (page 9 col. 1 lines 6-8). A person of ordinary skill in the art could have reasoned that combining the system of Aldridge, DeLand, Osborn and Dein with Dor by utilizing Dor’s tag database to store Dein’s tags, could lead to efficient and accurate monitoring and tracking of tools (“Thus, it is desirable to provide a solution for efficient monitoring and tracking of tools, in case an infection control or disease control event is detected.” Page 9 col.1 lines 34-36), improving patient health outcomes by avoiding infections and other hazardous events arising from improperly maintaining and managing tools (“Thus monitoring and tracking becomes even more critical when the tools are improperly maintained and then moved around both within an institution and between multiple institutions. When the tools are improperly cleaned, washed, sterilized, etc., they risk causing contaminations, infections, and other hazardous events.” Page 10 col.4 lines 1-6). Therefore it would have been obvious for a person of ordinary skill in the art before the effective filing date of the claimed invention to combine the system of Aldridge, DeLand, Osborn and Dein with Dor by utilizing Dor’s tag database to store Dein’s tags, with the expectation that doing so would lead to efficient and accurate monitoring and tracking of tools (see Page 9 col.1 lines 34-36), improving patient health outcomes by avoiding infections and other hazardous events arising from improperly maintaining and managing tools (see Page 10 col.4 lines 1-6).
Response to Arguments
Applicant’s arguments filed 10/06/2025 have been fully considered. Applicant’s arguments regarding the 101 and 112 rejections are persuasive and withdrawn.
With regard to claim 1, the rejection has been updated due to the amendments of the claim necessitating a new grounds of rejection (see above claim mapping). On page 13, the applicant asserts that Dein does not teach a reference grid on the tray for obtaining dimensions of the plurality of surgical tools. The examiner disagrees because the grid is used in concert with the cameras to image/scan instruments leading to size and shape recognition (“In addition to the intra-operative imaging device, the systems include a surgical sharp object automated shape recognition module.” Page 6 paragraph 3 lines 1-2 and “Automated shape recognition is a process by which the identification of surgical sharp objects is performed using one or more of the size, shape, aspect ratio, --> outline, color, or other distinctive feature present in the image data of the surgical sharp object.” Dein Paragraph 4 lines 4-6, see the updated rejection). Also on page 13, the applicant asserts that DeLand does not teach recognizing tools added to the tray erroneously. The examiner disagrees on two points. The first reason is that the limitation mentions added, but does not mention “erroneously added”. The addition of “erroneously” narrows the scope of the claim farther than the claim language currently states. Secondly, the examiner disagrees that DeLand does not teach that limitation (see “At step 450, the surgical instrument tracker performs a similar process to identify surgical instruments on the back tray, for example, back tray 230, to check that there are no surgical instruments on the back tray. If surgical instruments are found o the back tray these instruments have to be removed from the back tray, or an explanation has to be recorded in the surgical instrument tracker as to why these instruments are on the back tray. DeLand Paragraph 0044, see the updated rejection). Additionally, the applicant notes that Ma fails to teach a similar limitation in that its AR system does not teach recognition of tools added to the tray. The examiner agrees, but considers it a moot point with the changes made in the updated reference involving the inclusion of a new source (see claim mapping). Further on, towards the end of the claim 1 arguments on page 13, the applicant argues that Dein does not teach a reference tag providing a grid on the tray. The examiner agrees, but considers it a moot point with the changes made in the updated reference involving the inclusion of a new source (see claim mapping).
With regard to claim 24, the rejection has been updated due to the amendments of the claim necessitating a new grounds of rejection (see above claim mapping). On page 15, the applicant again argues that Dein does not teach a reference grid on the tray for obtaining dimensions of the plurality of surgical tools. The examiner again disagrees because the grid is used in concert with the cameras to image/scan instruments leading to size and shape recognition (“In addition to the intra-operative imaging device, the systems include a surgical sharp object automated shape recognition module.” Page 6 paragraph 3 lines 1-2 and “Automated shape recognition is a process by which the identification of surgical sharp objects is performed using one or more of the size, shape, aspect ratio, --> outline, color, or other distinctive feature present in the image data of the surgical sharp object.” Dein Paragraph 4 lines 4-6). Further on, towards the end of the claim 24 arguments on page 15, the applicant argues that Dein does not teach a reference tag providing a grid on the tray. The examiner agrees, but considers it a moot point with the changes made in the updated reference involving the inclusion of a new source (see claim mapping).
With regard to the dependent claims (2, 4, 6, 7, 10-13, 17, 18, 20-23 [see page 14] and 26, 27, and 30 [see page 15]), due to the above claim mapping and disagreements with the above arguments, these claims have not been rendered allowable and are rejected.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Van Den Brink (CN 1842301 A)- discloses a medical instrument recognition system that utilizes an electric current to determine whether instruments are present on, added to, or missing from the tray.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to REBECCA C WILLIAMS whose telephone number is (571)272-7074. The examiner can normally be reached M-F 7:30am - 4:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew W Bee can be reached at (571)270-5183. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/REBECCA COLETTE WILLIAMS/Examiner, Art Unit 2677
/ANDREW W BEE/Supervisory Patent Examiner, Art Unit 2677