Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Claim 1 is objected to because of the following informalities:
In line 4, “wherein the is first trained” should be “wherein the first trained”.
In line 9, “determined surgical using” should be “determined surgical events using”.
Appropriate correction is required.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2, 6-7, 9-11, 13-18, and 30 are rejected under 35 U.S.C. 103 as being unpatentable over Buch et al. (U.S. 2021/0307841 A1, hereinafter Buch, from IDS), in view of Wolf et al. (U.S. 2021/0298869 A1, hereinafter Wolf, from IDS).
Regarding claim 1, Buch teaches a method for managing surgical data, the method comprising: (FIG. 1A, where “FIG. 1A is a diagram illustrating exemplary systems for AI assisted surgical guidance.” [0026])
receiving surgical video data; ([0026] “Referring to FIG. 1A, a neural network 100 may be trained from video images from past surgeries and ground truth data from surgeons”)
applying a trained first neural network to the surgical video data to determine one or more surgical events based on the surgical video data, ([0026] “Referring to FIG. 1A, a neural network 100 may be trained from video images from past surgeries and ground truth data from surgeons to identify anatomical objects, surgical objects, and tissue manipulations in the video images.”, where “Trained neural network 102 receives a live video feed from the surgery and outputs classifications of anatomical objects, surgical objects, and tissue manipulations.” [0027])
wherein the is first trained neural network is trained to recognize at least one of: a surgery stage, a surgical activity, a surgical action, a surgical tool, an implant, a suture, and an implant anchor; ([0026] “The result of the training is a trained neural network 102 that is trained to identify anatomical objects, surgical objects, and tissue manipulations.”, where “training a neural network to identify at least one of anatomical objects (i.e. tissue types and critical structures), surgical objects (i.e. instruments and items), and tissue manipulation (i.e. the precise interface of anatomical and surgical objects) in video images” [0005])
and forming a report summary including a video clip comprising highlights of one or more determined surgical ([0033] “In another example, algorithms that process and display output from neural network 102 in surgery and surgeon-specific manners may be used in combination with trained neural network 102 to provide enhanced surgical guidance. This can include a surgical roadmap with suggested next steps at each time point based on chronologically processed data for each surgery type, or a continuous extent-of-resection calculation for tumor resection surgery by utilizing volumetric analysis of the resection cavity in real-time. Surgeon-specific metrics could include movement efficiency metrics based on, for example, the amount of time a particular instrument was used for a given step in the procedure.”)
and outputting the report summary. ([0035] “Surgical guidance generator 104, in one example, may output the surgical guidance on a video screen that surgeons use during image guided surgery. The video screen may be standalone display separate from the surgical field or an augmented reality display located in or in front of the surgical field.”).
Buch fails to teach forming a report summary… using a second trained neural network wherein the second trained neural network is trained to identify a representative image or video clip from the surgical video data based on the one or more determined surgical events. However, this is known in the art as taught by Wolf.
Wolf teaches forming a report summary… using a second trained neural network wherein the second trained neural network is trained to identify a representative image or video clip from the surgical video data based on the one or more determined surgical events ([0433] “Further, aspects of a method of populating a post-operative report of a surgical procedure may include identifying a property of at least one phase of identified phases.”, where “Additionally or alternatively, a property of a phase may be non-textural data (e.g., image, audio, numerical, and/or video data) collected during a surgical procedure. For example, a representative image of an anatomical structure (or surgical instrument, or an interaction of a surgical instrument with an example anatomical structure) performed during a phase of a surgical procedure may be used as a property of a phase. In one example, a machine learning model may be trained using training examples to identify properties of surgical phases from images and/or videos. An example of such training example may include an image and/or a video of at least a portion of a surgical phase of a surgical procedure, together with a label indicating one or more properties of the surgical phase.” [0433]).
Wolf is analogous to the claimed invention, as both relate to analyzing videos of surgical events to create a report ([0005] “Embodiments consistent with the present disclosure provide systems and methods for analysis of surgical videos.”, where “Systems and methods for automatically populating a post-operative report of a surgical procedure are disclosed.” [Abstract]). Wolf further teaches that “The surgical timeline may include markers identifying at least one of a surgical phase… The surgical timeline may enable a surgeon, while viewing playback of the at least one video to select one or more markers on the surgical timeline, and thereby cause a display of the video to skip to a location associated with the selected marker.” [0006]. Therefore, it would be obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Wolf to Buch in order to have a second neural network to identify a highlight or representative image or video in order to allow the surgeon to playback in an efficient way by skipping to the desired part of the timeline.
Regarding claim 2, the combination of Buch and Wolf teaches the method of claim 1, wherein the second neural network is trained to indicate that no surgical progress was made during a related surgery in the video clip (Buch; [0037] “In step 204, trained neural network 102 identifies anatomical objects, surgical objects, and tissue manipulations from the live video feed from the surgery and outputs classifications of the anatomical objects, surgical objects, and tissue manipulations. For example, trained neural network 102 may output the classifications to surgical guidance generator 104 illustrated in FIG. 1A.”. Note: the classification step is done when the live video feed shows that no surgical progress is made, which is then used to create surgical guidance before the surgery begins.)
Regarding claim 6, the combination of Buch and Wolf teaches the method of claim 1, further comprising generating a deidentified video clip based at least in part on the surgical video data (Buch; [0020] “Once initialized, this network will be trained using hundreds of these de-identified surgical videos in which the same key structures have been segmented out. Over multiple iterations, the deep neural network will coalesce classifiers for each pixel designating the anatomical or surgical object class to which it belongs.”).
Regarding claim 7, the combination of Buch and Wolf teaches the method of claim 1, wherein the report summary includes a text summary of one or more surgical stages within the surgical video data. (Wolf; [0107] “A timeline may also be a text-based list of events arranged in chronological order. A surgical timeline may be a timeline representing events associated with a surgery. As one example, a surgical timeline may be a timeline of events or actions that occur during a surgical procedure, as described in detail above. In some embodiments, the surgical timeline may include textual information identifying portions of the surgical procedure.”).
Wolf is analogous to the claimed invention, as both relate to analyzing videos of surgical events to create a report ([0005] “Embodiments consistent with the present disclosure provide systems and methods for analysis of surgical videos.”, where “Systems and methods for automatically populating a post-operative report of a surgical procedure are disclosed.” [Abstract]). Wolf further teaches that “Therefore, there is a need for unconventional approaches that efficiently and effectively enable a surgeon to view a surgical video summary that aggregates footage of relevant surgical events while omitting other irrelevant footage.” [0006]. Therefore, it would be obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Wolf to Buch in order to allow a surgeon to effectively and efficiently view and surgical summaries.
Regarding claim 9, the combination of Buch and Wolf teaches the method of claim 1, wherein outputting comprising saving the report summary to a secure cloud-based storage unit (Buch; [0017] “As a product, we would aim to launch our platform, known as TAIRIS, through either a cloud based or physical server. The end user-interface will be known as the AOA. Cloud based access would entail video feed processing in the cloud (in a HIPPA compliant manner) and results returned to the OR either through a boom audiovisual monitor or to a worn heads-up display.”).
Regarding claim 10, the combination of Buch and Wolf teaches the method of claim 1, wherein the surgical video data is a real-time video stream received from an endoscopic camera (Buch; [0017] “The computing power involved may prohibit this so we will also be prepared to create a physical processing device (local DNN CPU, with daily updates/communication between itself and the mainframe TAIRIS server) that can be integrated into an individual or suite of ORs.”, where “In its first iteration, our technology utilizes computer vision based DNNs to continuously monitor an operative field and provide feedback to a surgeon in real-time with the goal of enhancing surgical decision making. In all operations, a video feed of the operative field can be obtained with either an overhead operating room (OR) camera, microscope, or endoscope. Regardless of the video input, the feed will be analyzed on a trained, multilayer, convolutional deep neural network capable of identifying key anatomical objects, surgical objects, and tissue manipulation. This real-time feedback and output of the trained DNN is considered the Artificial Operative Assistant (AOA).” [0018]).
Regarding claim 11, the combination of Buch and Wolf teaches the method of claim 1, wherein the surgical video data is received from a secure cloud-based storage unit. (Buch; [0017] “The computing power involved may prohibit this so we will also be prepared to create a physical processing device (local DNN CPU, with daily updates/communication between itself and the mainframe TAIRIS server) that can be integrated into an individual or suite of ORs.”, where “As a product, we would aim to launch our platform, known as TAIRIS, through either a cloud based or physical server. The end user-interface will be known as the AOA. Cloud based access would entail video feed processing in the cloud (in a HIPPA compliant manner)” [0017].)
Regarding claim 13, the combination of Buch and Wolf teaches the method of claim 1, further comprising generating an alert based on a recognized surgical activity (Buch; [0030] “In another example, neural network 100 may be trained to identify changes in contour of specific tissue types. In such a case, surgical guidance generator 104 may output a warning when a change in contour for tissue type identified in the live video feed nears a damaged threshold for the tissue type. Such output may prevent the surgeon from damaging tissue during the surgery.”).
Regarding claim 14, the combination of Buch and Wolf teaches the method of claim 1, further comprising: receiving a voice input from a surgeon during the surgery; and generating context specific alerts based on the voice input (Buch; [0024] “To accomplish this, we plan to develop an intraoperative user interface based on voice recognition and special labeling probe. A surgeon would be able to interact with the AOA using voice activation to ask questions of the AOA (i.e. “How certain are you that this is the disc material?”, “How safe is it to retract this part of the thecal sac?”) or ask the AOA to label objects as the user sees fit (i.e. “Label this object as the carotid artery.”). Intraoperative feedback from the surgeon can also be fed back into the network to improve its output.”. Note: the answering of questions and labeling of objects is mapped to context specific alerts.)
Regarding claim 15, the combination of Buch and Wolf teaches the method of claim 1, wherein the first neural network is trained to determine a start and stop of a surgical action (Wolf; [0198] “The information may distinguish the portions of surgical footage in various ways. For example, in connection with historical surgical footage, frames associated with surgical and non-surgical activity may already have been distinguished. This may have previously occurred, for example, through manual flagging of surgical activity or through training of an artificial intelligence engine to distinguish between surgical and non-surgical activity. The historical information may identify, for example, a set of frames (e.g., using a starting frame number, a number of frames, an end frame number, etc.) of the surgical footage. The information may also include time information, such as a begin timestamp, an end timestamp, a duration, a timestamp range, or other information related to timing of the surgical footage.”
Wolf is analogous to the claimed invention, as both relate to analyzing videos of surgical events to create a report ([0005] “Embodiments consistent with the present disclosure provide systems and methods for analysis of surgical videos.”, where “Systems and methods for automatically populating a post-operative report of a surgical procedure are disclosed.” [Abstract]). Wolf further teaches that “Therefore, there is a need for unconventional approaches that efficiently and effectively enable a surgeon to view a surgical video summary that aggregates footage of relevant surgical events while omitting other irrelevant footage.” [0006]. Therefore, it would be obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Wolf to Buch in order to allow a surgeon to effectively and efficiently view and surgical summaries.
Regarding claim 16, claim 16 recites substantially similar limitations to claim 1, but in a system form. The combination of Buch and Wolf further teaches a system, comprising: ([0002] “More particularly, the subject matter described herein relates to methods, systems, and computer readable media for generating and providing artificial intelligence assisted surgical guidance.”)
one or more processors; (Buch; [0006] “For example, the subject matter described herein can be implemented in software executed by a processor.”)
and a memory configured to store instructions that, when executed by one of the one or more processors, cause the system to: (Buch; [0006] “In one exemplary implementation, the subject matter described herein can be implemented using a non-transitory computer readable medium having stored thereon computer executable instructions that when executed by the processor of a computer control the computer to perform steps.”).
Regarding claim 17, claim 17 recites substantially similar limitations to claim 1, but in a medium form. The combination of Buch and Wolf further teaches a non-transitory computer-readable storage medium comprising instructions that, when executed by one or more processors of a device, cause the one or more processors to perform operations comprising: (Buch; [0006] “In one exemplary implementation, the subject matter described herein can be implemented using a non-transitory computer readable medium having stored thereon computer executable instructions that when executed by the processor of a computer control the computer to perform steps.”).
Regarding claim 18, claim 18 recites substantially similar limitations to claim 2, therefore, rejected under the same rationale as claim 2.
Regarding claim 30, claim 30 recites substantially similar limitations to claim 14, therefore, rejected under the same rationale as claim 14.
Claims 3-5 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Buch, in view of Wolf, and further in view of Casas (U.S. 2016/0191887 A1, from IDS).
Regarding claim 3, the combination of Buch and Wolf teaches the method of claim 1, but fails to teach further comprising: receiving patient-specific radiological imaging information; and generating a three-dimensional (3D) navigational guidance for a surgery based, at least in part, on the patient-specific radiological imaging information. However, this is known in the art as taught by Casas.
Casas teaches receiving patient-specific radiological imaging information; ([0073] “Returning now to FIG. 1, the preoperative 102 or intraoperative image 106 information, may be received from a CT scan or an MR scan, as well as ultrasound, PET, and C-arm cone-beam computed tomography. Preoperative X-ray and stereoscopic X-ray images, intraoperative fluoroscopic and stereoscopic fluoroscopic images, and other image modalities that allow for stereoscopic and 3D representations are also used, either directly in a digital format 108, or as graphical 3D volumetric representations 104 of the image data.”)
and generating a three-dimensional (3D) navigational guidance for a surgery based, at least in part, on the patient-specific radiological imaging information ([0092] “The internal anatomic structures are therefore displayed based on the positioning of patient 118 was when the preoperative image 102 was obtained, e.g. with the patient 118 in the supine position during the CT or MR scan. Responsive to performing a medical procedure on the target anatomic structures, and e.g. bony structures or soft tissues are (partially) exposed, a new 3D scan is obtained with the 3D scanner system 110.”).
Casas is analogous to the claimed invention, as both relate to analyzing and processing real-time surgery data for surgical assistance ([0003] “Embodiments are directed towards image-guided surgery, and more particularly CT-guided, MR-guided, fluoroscopy-based or surface-based image-guided surgery, wherein images of a portion of a patient are taken in the preoperative or intraoperative setting and used during surgery for guidance.”). Casas further teaches that “Equipment has been developed by many companies to provide intraoperative interactive surgery planning and display systems, mixing live video of the external surface of the patient with interactive computer generated models of internal anatomy obtained from medical diagnostic imaging data of the patient. The computer images and the live video are coordinated and displayed to a surgeon in real time during surgery, allowing the surgeon to view internal and external structures and the relationship between them simultaneously, and adjust the surgery accordingly.” [0006]. Therefore, it would be obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Casas to the combination of Buch and Wolf in order to simultaneously display the internal and external structures in real-time in order to guide the surgeon more effectively.
Regarding claim 4, the combination of Buch, Wolf, and Casas teaches the method of claim 3, wherein the 3D navigational guidance is based at least in part on a patient-specific preoperative plan (Buch; [0023] “The surgical guidance generator will utilize the foundational information from the neural network output as described above. This data will also be post-processed in a novel hierarchical algorithm framework and combined with other input data such as pre-operative imaging, patient demographic and co-morbidity factors, surgical object cost data, or intraoperative vital signs. These post-processed outputs of the surgical guidance generator will be clinically tailored, surgery- and even surgeon-specific including information such as a surgical roadmap of suggested next steps, movement efficiency metrics, complication avoidance warnings, procedural stepwise cost estimates, and predictive analytics that can provide novel intraoperative indices predicting post-operative outcome.”).
Regarding claim 5, the combination of Buch, Wolf, and Casas teaches the method of claim 3, wherein the 3D navigational guidance is based at least in part on generic models of an anatomical joint (Casas; [0095] “In that manner, instead of blending a 3D surface with a static 3D volume image directly, the pose change 318 in the 3D surface is interpreted by computer means 100 and applied 320 to a predefined virtual anatomical model 306, to which the 3D volume of the patient is registered, obtaining more precise positions of the internal anatomic parts, e.g. location and orientation of joints and bones by using an anatomic skeletal model.”).
Casas is analogous to the claimed invention, as both relate to analyzing and processing real-time surgery data for surgical assistance ([0003] “Embodiments are directed towards image-guided surgery, and more particularly CT-guided, MR-guided, fluoroscopy-based or surface-based image-guided surgery, wherein images of a portion of a patient are taken in the preoperative or intraoperative setting and used during surgery for guidance.”). Casas further teaches that “obtaining more precise positions of the internal anatomic parts, e.g. location and orientation of joints and bones by using an anatomic skeletal model.” [0095]. Therefore, it would be obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Casas to the combination of Buch and Wolf in order to obtain more precise positions of internal anatomic parts by using a generic model.
Regarding claim 19, claim 19 recites substantially similar limitations to claim 3, therefore, is rejected under the same rationale as claim 19.
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Buch, in view of Wolf, and further in view of Morvan et al. (U.S. 2025/0352269 A1, hereinafter Morvan).
The combination of Buch and Wolf teaches the method of claim 1, wherein forming the report summary comprises generating a log of… measurements taken during a surgery associated with the surgical video data (Wolf; [0400] “A condition of an anatomical structure may be determined based on observed visual characteristics of the anatomical structure such as a size, color, shape, translucency, reflectivity of a surface, fluorescence, and/or other image features. A condition may be based on one or more of the anatomical structure, temporal characteristics (motion, shape change, etc.) for the anatomical structure, sound characteristics (e.g., transmission of sound through the anatomical structure, sound generated by the anatomical structure, and/or other aspects of sound), imaging of the anatomical structure (e.g., imaging using x-rays, using magnetic resonance, and/or other means), or electromagnetic measurements of the structure (e.g., electrical conductivity of the anatomical structure, and/or other properties of the structure).”).
Wolf is analogous to the claimed invention, as both relate to analyzing videos of surgical events to create a report ([0005] “Embodiments consistent with the present disclosure provide systems and methods for analysis of surgical videos.”, where “Systems and methods for automatically populating a post-operative report of a surgical procedure are disclosed.” [Abstract]). Wolf further teaches that “Procedures performed on anatomical structures in poor condition, for example, may justify higher reimbursement than procedures performed on anatomical structures in better condition.” [0400]. Therefore, it would be obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Wolf to Buch in order to determine the condition of the anatomical structure that the surgery is being done on.
The combination of Buch and Wolf fails to teach wherein forming the report summary comprises generating a log of pinless landmarks used. However, this is known in the art as taught by Morvan.
Morvan teaches wherein forming the report summary comprises generating a log of pinless landmarks used ([0004] “This disclosure describes example techniques for automated estimation of landmarks (e.g., bony landmarks) on one or more bones of a patient to aid in orthopedic surgery planning. In particular, this disclosure describes techniques that include the application of a neural network to a 3D representation of one or more bones, where the neural network outputs a 3D representation including labels that identify the locations of the landmarks.”)
Morvan is analogous to the claimed invention, as both relate to using neural networks in order to assist surgeons with surgical planning. Morvan further teaches that “he output point cloud may be used in orthopedic surgery planning, including to visualize bones that are to be operated on, design and/or select surgical guides, design and/or select a prosthesis, design and/or select implant components that closely match the patient's anatomy, determine tool alignments, determine surgical cut locations, and for other surgical planning needs.” [0004]. Therefore, it would be obvious for one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teachings of Morvan into the combination of Buch and Wolf in order to provide the surgeon assistance with visualizing the patient’s anatomy before surgery.
Claim 12 is rejected under 35 U.S.C. 103 as being unpatentable over Buch, in view of Wolf, and further in view of Fouts et al. (U.S. 2022/0207896 A1, hereinafter Fouts, from IDS).
The combination of Buch and Wolf teaches the method of claim 1, but fails to teach wherein the surgical video data is received via an HDMI or USB connector. However, this is known in the art as taught by Fouts.
Fouts teaches wherein the surgical video data is received via an HDMI or USB connector ([0117] “In one or more examples of the disclosure, the process 300 illustrated in FIG. 3 can begin at step 302 wherein video data from an endoscopic device or other type of imaging device is received. In one or more examples, the video data can be transmitted to one or more processors configured to implement process 300 using a High-Definition Multimedia Interface (HDMI), Digital Visual Interface (DVI) or other interface capable of connecting a video source (such as an endoscopic camera) to a display device or graphics processor.”).
Fouts is analogous to the claimed invention, as both relate to analyzing surgical procedure videos to assist with creating a surgical report. Fouts further teaches “Furthermore, the systems and methods described herein can further reduce the time spent generating a post-surgical report by automatically generating the annotations and laying them out vis-à-vis the image in a way that clearly conveys the context of the image so that the patient can understand what is being shown in the image.” [0008]. Therefore, it would be obvious for one of ordinary skill in the art to incorporate the teachings of Fouts to the combination of Buch and Wolf in order to reduce the time spent on generating a post-surgical report.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ALICIA HA whose telephone number is (571)272-3601. The examiner can normally be reached Mon-Thurs 9:30 AM - 6:30 PM, and Fri 9:30 AM - 1:30 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kee Tung can be reached at (571) 272-7794. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KEE M TUNG/Supervisory Patent Examiner, Art Unit 2611
/ALICIA HA/Examiner, Art Unit 2611
Prosecution Insights