DETAILED ACTION
Notice of Pre-AIA or AIA Status
1. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
2. Acknowledgement is made of amendment filed on April 30, 2026, in which claim 1 is amended, claim 9 is canceled, claim 21 is new, and claims 1-8 and 10-21 are still pending.
Response to Arguments
3. Applicant's arguments, filed on April 30, 2026, with respect to Claims 1-8 and 10-21 have been fully considered but they are not persuasive.
4. With regards to arguments for independent claim 1, applicants argue that Lang (US 2021/0137634 A1) and Verard et al. (US 2014/0282008 A1) fail to disclose obtaining a patient intervention site dataset of a previously operated upon patient intervention site at which surgery was previously completed. However, the examiner respectfully disagrees and maintains in the grounds of rejections regarding claim 1, since in Lang (US 2021/0137634 A1) teaches (One or more optical imaging systems or 3D scanners can, for example, be used to image and/or monitor, e.g. the coordinates, position, orientation, alignment, direction of movement, speed of movement of, Anatomic landmarks, patient surface(s), organ surface(s), tissue surface(s), pathologic tissues and/or surface(s), e.g. for purposes of registration, e.g. of the patient and/or the surgical site, e.g. one or more bones or cartilage, and/or one or more OHMD's, e.g. in a common coordinate system The surgeon or interventionalist's hands and/or fingers, e.g. for Monitoring steps in an interventional procedure. Select hand and/or finger movements can be associated with corresponding surgical steps. When the 3D scanner system detects a particular hand and/or finger movement, it can trigger the display of the corresponding surgical step or the next surgical step, e.g. by displaying a predetermined virtual path, e.g. for a catheter, a virtual instrument, a virtual device etc. … One or more OHMDs, e.g. registered in a common coordinate system, e.g. with the surgical site and/or the surgeon or interventionalist's hands and/or fingers.” [0235-0240] “Placement rules, selection rules or design rules of a virtual surgical plan can include demographic information of the patient, e.g. weight, height, age, gender, other information such as bone mineral density or structure, clinical history, history of prior fractures, or functional information, e.g. on motion of a surgical site, an organ or a tissue, or metabolic information, e.g. for certain organs or pathologic tissues.” [0642] “The one or more of these anatomic landmarks, distances, dimensions, surfaces, edges, angles, axes, curvatures, shapes, lengths, widths, depths and/or other features, e.g. external or internal, can be used to identify a standard bone shape and/or a standard cartilage shape by comparing the one or more landmarks, distances, dimensions, surfaces, edges, angles, axes, curvatures, shapes, lengths, widths, depths and/or other external or internal features with data in a reference database of reference patients and/or reference bone and/or cartilage shapes and by selecting a 3D model that most closely matches the selected landmarks, distances, dimensions, surfaces, edges, angles, axes, curvatures, shapes, lengths, widths, depths and/or other features.” [0675] “The surgeon or interventionalist can optionally store each surgical instrument or implantable that has been scanned in this manner in a virtual library of surgical instruments or implantables. The virtual surgical instruments or implantables stored in this manner can be named and stored for future use in subsequent surgical procedures in other patients. … the surgeon or interventionalist can use the virtual data of the surgical instrument or implantables that were previously generated in a new surgical plan for another, new patient.” [0789-0790] “Pre-operative, intra-operative or post-operative images of the patient can be acquired 240. The image data can optionally be segmented 241. 3D reconstructions of the patient's anatomy or pathology including multiple different tissues, e.g. using different colors or shading, can be generated 242. Virtual 3D models of surgical instruments and devices components can be generated which can include their predetermined position, location, rotation, orientation, alignment and/or direction 243. The virtual 3D models can be registered, for example in relationship to the OHMD and the patient 244. The virtual 3D models can be registered relative to the live patient data 245.” [0796]) Lang teaches selecting a 3D model that most closely matches the selected features in a reference database of reference patients and/or reference bone, that the data were previously generated, which the dataset can consider a previously operated upon patient intervention site at which surgery was previously completed. Lang also teaches the OHMDs registered in a common coordinate system with the surgical site and/or the surgeon or interventionalist's hands and/or fingers, the virtual 3D models can be registered, in relationship to the OHMD and the Pre-operative, intra-operative or post-operative images of the patient, which applying the surgeon hand positional information to the patient intervention site dataset. Therefore, Lang teaches the arguments of the limitations for claim 1 as it is recited.
Claim Rejections - 35 USC § 103
5. In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
6. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
7. The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
8. Claim(s) 1-8 and 10-21 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lang (US 2021/0137634 A1) in view of Verard et al. (US 2014/0282008 A1).
9. With reference to claim 1, Lang teaches A method of surgical analysis, comprising obtaining surgeon hand positional information, (“The present disclosure relates to devices and methods for performing various interventional or surgical procedures with visual guidance using an optical head mounted display with cardiac and/or respiratory gating of the displayed virtual data.” [0002] “the virtual image is a three-dimensional digital representation corresponding to at least one portion of a physical anatomical target for surgical or other medical intervention.” [0006] “3D scanning can be used for imaging of the patient and/or the surgical site and/or anatomic landmarks and/or pathologic structures and/or tissues (e.g. damaged or diseased cartilage or exposed subchondral bone) and/or the surgeon or interventionalist's hands and/or fingers and/or the OR table and/or reference areas or points and/or marker, e.g. optical markers, in the operating room and/or on the patient and/or on the surgical field. … One or more optical imaging systems or 3D scanners can, for example, be used to image and/or monitor, e.g. the coordinates, position, orientation, alignment, direction of movement, speed of movement of, Anatomic landmarks, patient surface(s), organ surface(s), tissue surface(s), pathologic tissues and/or surface(s), e.g. for purposes of registration, e.g. of the patient and/or the surgical site, e.g. one or more bones or cartilage, and/or one or more OHMD's, e.g. in a common coordinate system The surgeon or interventionalist's hands and/or fingers, e.g. for Monitoring steps in an interventional procedure. Select hand and/or finger movements can be associated with corresponding surgical steps. When the 3D scanner system detects a particular hand and/or finger movement, it can trigger the display of the corresponding surgical step or the next surgical step, e.g. by displaying a predetermined virtual path, e.g. for a catheter, a virtual instrument, a virtual device etc. … One or more OHMDs, e.g. registered in a common coordinate system, e.g. with the surgical site and/or the surgeon or interventionalist's hands and/or fingers.” [0235-0240]) Lang also teaches obtaining a patient intervention site dataset of a previously operated upon patient intervention site at which surgery was previously completed, (“Placement rules, selection rules or design rules of a virtual surgical plan can include demographic information of the patient, e.g. weight, height, age, gender, other information such as bone mineral density or structure, clinical history, history of prior fractures, or functional information, e.g. on motion of a surgical site, an organ or a tissue, or metabolic information, e.g. for certain organs or pathologic tissues.” [0642] “The one or more of these anatomic landmarks, distances, dimensions, surfaces, edges, angles, axes, curvatures, shapes, lengths, widths, depths and/or other features, e.g. external or internal, can be used to identify a standard bone shape and/or a standard cartilage shape by comparing the one or more landmarks, distances, dimensions, surfaces, edges, angles, axes, curvatures, shapes, lengths, widths, depths and/or other external or internal features with data in a reference database of reference patients and/or reference bone and/or cartilage shapes and by selecting a 3D model that most closely matches the selected landmarks, distances, dimensions, surfaces, edges, angles, axes, curvatures, shapes, lengths, widths, depths and/or other features.” [0675] “The surgeon or interventionalist can optionally store each surgical instrument or implantable that has been scanned in this manner in a virtual library of surgical instruments or implantables. The virtual surgical instruments or implantables stored in this manner can be named and stored for future use in subsequent surgical procedures in other patients. … the surgeon or interventionalist can use the virtual data of the surgical instrument or implantables that were previously generated in a new surgical plan for another, new patient.” [0789-0790] “Pre-operative, intra-operative or post-operative images of the patient can be acquired 240. The image data can optionally be segmented 241. 3D reconstructions of the patient's anatomy or pathology including multiple different tissues, e.g. using different colors or shading, can be generated 242. Virtual 3D models of surgical instruments and devices components can be generated which can include their predetermined position, location, rotation, orientation, alignment and/or direction 243. The virtual 3D models can be registered, for example in relationship to the OHMD and the patient 244. The virtual 3D models can be registered relative to the live patient data 245.” [0796] “Any of the 2D or 3D virtual data in any of the embodiments throughout the specification can include an arterial run-off, e.g. a peripheral run-off or an aortic run-off, optionally including bolus chasing. Bolus chasing can permit real-time visualization of the contrast bolus so that it can be followed, for example, peripherally with images, e.g. digital images, being acquired at a suitable frame rate. If the run-off and/or bolus chasing includes movement of the table on which the patient is positioned during the image acquisition, any 2D or 3D virtual data, e.g. from a pre-operative imaging study, e.g. a CTA or MRA or ultrasound, can be moved with the patient and/or table movement to maintain superimposition and/or alignment of a first 2D or 3D virtual data set, e.g. from the pre-operative imaging study, displayed by one or more OHMDs with a second or additional 2D or 3D virtual data sets, e.g. images obtained during the run-off and/or bolus chasing, displayed by the one or more OHMDs or a separate, standalone computer monitor using any of the combinations outlined in exemplary form in Tables 11, 13 and 14. The moving of the 2D or 3D virtual data, e.g. from a first set of virtual data from a pre-operative imaging test such as a CTA, MRA or ultrasound, displayed, using a computer processor, by the one or more OHMDs can be accomplished using the image registration including 3D-2D transformation matrices or 3D-3D registration techniques and/or the known table movement, e.g. in x, y, and/or z-direction, and/or the measured movement of one or more markers applied to the table or the patient. The image registration can include selection and/or matching and/or superimposition and/or alignment of a different volume of interest from a pre-operative vascular imaging study, e.g. an ultrasound, echocardiogram, CTA or MRA, to match the run-off and/or bolus chasing images and to maintain the superimposition and/or alignment of the first virtual dataset displayed by the one or more OHMDs with the second virtual data set, i.e. the run-off and/or bolus chasing images, e.g. displayed by the one or more OHMDs or a standalone or separate computer monitor or display.” [1043]) Lang further teaches applying the surgeon hand positional information to the patient intervention site dataset (“Any of the 2D or 3D virtual data in any of the embodiments throughout the specification can include an arterial run-off, e.g. a peripheral run-off or an aortic run-off, optionally including bolus chasing. Bolus chasing can permit real-time visualization of the contrast bolus so that it can be followed, for example, peripherally with images, e.g. digital images, being acquired at a suitable frame rate. If the run-off and/or bolus chasing includes movement of the table on which the patient is positioned during the image acquisition, any 2D or 3D virtual data, e.g. from a pre-operative imaging study, e.g. a CTA or MRA or ultrasound, can be moved with the patient and/or table movement to maintain superimposition and/or alignment of a first 2D or 3D virtual data set, e.g. from the pre-operative imaging study, displayed by one or more OHMDs with a second or additional 2D or 3D virtual data sets, e.g. images obtained during the run-off and/or bolus chasing, displayed by the one or more OHMDs or a separate, standalone computer monitor using any of the combinations outlined in exemplary form in Tables 11, 13 and 14. The moving of the 2D or 3D virtual data, e.g. from a first set of virtual data from a pre-operative imaging test such as a CTA, MRA or ultrasound, displayed, using a computer processor, by the one or more OHMDs can be accomplished using the image registration including 3D-2D transformation matrices or 3D-3D registration techniques and/or the known table movement, e.g. in x, y, and/or z-direction, and/or the measured movement of one or more markers applied to the table or the patient. The image registration can include selection and/or matching and/or superimposition and/or alignment of a different volume of interest from a pre-operative vascular imaging study, e.g. an ultrasound, echocardiogram, CTA or MRA, to match the run-off and/or bolus chasing images and to maintain the superimposition and/or alignment of the first virtual dataset displayed by the one or more OHMDs with the second virtual data set, i.e. the run-off and/or bolus chasing images, e.g. displayed by the one or more OHMDs or a standalone or separate computer monitor or display.” [1043] “A combination of data can be beneficial for more accurate measurement of changes in position or orientation of the surgeon or interventionalist's head, body, operating arm, hand, or the patient.” [0291] “3D scanning can be used for imaging of the patient and/or the surgical site and/or anatomic landmarks and/or pathologic structures and/or tissues (e.g. damaged or diseased cartilage or exposed subchondral bone) and/or the surgeon or interventionalist's hands and/or fingers and/or the OR table and/or reference areas or points and/or marker, e.g. optical markers, in the operating room and/or on the patient and/or on the surgical field. … One or more optical imaging systems or 3D scanners can, for example, be used to image and/or monitor, e.g. the coordinates, position, orientation, alignment, direction of movement, speed of movement of, Anatomic landmarks, patient surface(s), organ surface(s), tissue surface(s), pathologic tissues and/or surface(s), e.g. for purposes of registration, e.g. of the patient and/or the surgical site, e.g. one or more bones or cartilage, and/or one or more OHMD's, e.g. in a common coordinate system The surgeon or interventionalist's hands and/or fingers, e.g. for Monitoring steps in an interventional procedure. Select hand and/or finger movements can be associated with corresponding surgical steps. When the 3D scanner system detects a particular hand and/or finger movement, it can trigger the display of the corresponding surgical step or the next surgical step, e.g. by displaying a predetermined virtual path, e.g. for a catheter, a virtual instrument, a virtual device etc. … One or more OHMDs, e.g. registered in a common coordinate system, e.g. with the surgical site and/or the surgeon or interventionalist's hands and/or fingers.” [0235-0240] “Pre-operative, intra-operative or post-operative images of the patient can be acquired 240. The image data can optionally be segmented 241. 3D reconstructions of the patient's anatomy or pathology including multiple different tissues, e.g. using different colors or shading, can be generated 242. Virtual 3D models of surgical instruments and devices components can be generated which can include their predetermined position, location, rotation, orientation, alignment and/or direction 243. The virtual 3D models can be registered, for example in relationship to the OHMD and the patient 244. The virtual 3D models can be registered relative to the live patient data 245.” [0796])
Lang does not explicitly teach assess impact of surgical hand position on a patient intervention site. This is what Verard teaches (“a fellow/practitioner could first select a program called "HIP" by activating a trigger point 504 to display a 3D CT image of a patient's (subject's) hip, and then select different "HIP IMPLANTS" from different manufacturers to see and "feel" which implant would fit best for the particular patient. It is also possible to use (e.g., physically hold and manipulate) the actual implant in the air and position it within the 3D holographic display to see, feel and assess fit (e.g., if and how well such implant may fit the particular patient).” [0059] “Robotics via a master/slave configuration can be used, where a shape sensed analog 604 of the device 602 moving within the display 158 is employed to actuate the motion of the actual device 602 within a target region 606. A practitioner's (surgeon's, physician's, etc.) hands 610 or voice can be tracked by sensor-based and/or voice-based techniques, such as by, e.g., tracking a physician's hands using a shape-sensing device 608 and shape sensing system 614 in the 3D holographic display. Accordingly, a practitioner's movements (including, e.g., (re)positioning, orientation, etc. of their hands) performed in the holographic display 158 can be transmitted to the device 602, such as a robot 612 (e.g., robotically controlled instruments) inside the patient to replicate such movements within the patient's body, and thereby perform the actual surgery, procedure or task inside the patient's body. Thus, a surgeon can see, touch and feel a 3D holographic display of an organ, perform a procedure thereon (i.e., within the 3D holographic display), causing such procedure to be performed inside of a patient on the actual organ via, or simply to move instruments, e.g., robotically controlled instruments.” [0062]) Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Verard into Lang, in order to improve accuracy and effectiveness in medical applications.
PNG
media_image1.png
542
415
media_image1.png
Greyscale
10. With reference to claim 2, Lang teaches the patient intervention site dataset is malleable such that a depiction of the patient intervention site may be computationally adjusted to simulate an effect of the surgeon hand positional information without reobtaining patient intervention site dataset data. (“tissue deformation, a shape change or removal of tissue caused by the surgery or surgical instruments can be simulated in the virtual data. The resultant simulated virtual data can then be registered related to the live patient data, either before and/or after deformation, alteration of shape or removal of tissue of the live patient. The tissue deformation, shape change or removal of tissue caused by the surgery or surgical instruments can include the shape alteration or removal of one or more osteophytes or bone spurs or other bony anatomy or deformity. The virtual data of the patient and the live data of the patient can be registered in a common coordinate system, for example with one or more OHMDs. … the registration of virtual patient data and live patient data using the techniques described herein can be repeated after one or more surgical steps have been performed. In this case, the surgically altered tissue or tissue surface or tissue contour or tissue perimeter or tissue volume or other tissue features in the live patient can be matched to, superimposed onto and/or registered with the surgically altered tissue or tissue surface or tissue contour or tissue perimeter or tissue volume or other tissue features in the virtual data of the patient, e.g. in a virtual surgical plan developed for the patient. The matching, superimposing and/or registering of the live data of the patient and the virtual data of the patient after the surgical tissue alteration can be performed using the same techniques described in the foregoing or any of the other registration techniques described in the specification or any other registration technique known in the art. Re-registration of live patient data and virtual patient data can be particularly helpful if the surgical alteration or surgical step has led to some tissue deformation. For example, the re-registration can be performed by matching, superimposing, and/or registering tissues that have not been performed by the surgical step or surgical alteration. Alternatively, the re-registration can be performed by matching, superimposing and/or registering deformed live patient data, e.g. from surgically deformed tissue, with virtual patient data that simulate the same tissue deformation after the virtual surgical step, e.g. an osteophyte or tissue removal. “ [0498-0499] “Registration can be performed, for example, based on motion data, kinematic data in the live data which can then be registered to an estimate or simulated center of rotation in the virtual data of the patient). Registration can be performed using metabolic data, for example using an area of high 18 FDG-PET uptake in a PET scan or PET-MRI or PET CT, which can be, for example matched to an area of increased body temperature in a target surgical site. Registration can be performed using functional data, e.g. using functional MRI studies.” [0510] “The OHMD can display virtual alterations to a surgical site superimposed onto the live surgical site prior to the physical alteration of the live surgical site. The virtual alterations to a surgical can be simulated using a virtual surgical plan.” [0877] “Any of the 2D or 3D virtual data in any of the embodiments throughout the specification can include an arterial run-off, e.g. a peripheral run-off or an aortic run-off, optionally including bolus chasing. Bolus chasing can permit real-time visualization of the contrast bolus so that it can be followed, for example, peripherally with images, e.g. digital images, being acquired at a suitable frame rate. If the run-off and/or bolus chasing includes movement of the table on which the patient is positioned during the image acquisition, any 2D or 3D virtual data, e.g. from a pre-operative imaging study, e.g. a CTA or MRA or ultrasound, can be moved with the patient and/or table movement to maintain superimposition and/or alignment of a first 2D or 3D virtual data set, e.g. from the pre-operative imaging study, displayed by one or more OHMDs with a second or additional 2D or 3D virtual data sets, e.g. images obtained during the run-off and/or bolus chasing, displayed by the one or more OHMDs or a separate, standalone computer monitor using any of the combinations outlined in exemplary form in Tables 11, 13 and 14. The moving of the 2D or 3D virtual data, e.g. from a first set of virtual data from a pre-operative imaging test such as a CTA, MRA or ultrasound, displayed, using a computer processor, by the one or more OHMDs can be accomplished using the image registration including 3D-2D transformation matrices or 3D-3D registration techniques and/or the known table movement, e.g. in x, y, and/or z-direction, and/or the measured movement of one or more markers applied to the table or the patient. The image registration can include selection and/or matching and/or superimposition and/or alignment of a different volume of interest from a pre-operative vascular imaging study, e.g. an ultrasound, echocardiogram, CTA or MRA, to match the run-off and/or bolus chasing images and to maintain the superimposition and/or alignment of the first virtual dataset displayed by the one or more OHMDs with the second virtual data set, i.e. the run-off and/or bolus chasing images, e.g. displayed by the one or more OHMDs or a standalone or separate computer monitor or display.” [1043] “Catheter or device tracking methods that work independently from the x-ray fluoroscopy imaging, e.g. MRI-based tracking or electromagnetic tracking, do not require continuous updates of the x-ray image. Information about 3D position and orientation of the catheter and/or catheter tip or other device (e.g. a guidewire, sheath, stent, coil, instrument, implant, vascular prosthesis or other intra-vascular or endoluminal instrument and/or device) can be available continuously and in real-time.” [1049])
11. With reference to claim 3, Lang teaches obtaining surgeon hand positional information comprises recording surgeon hand position pursuant to performance of a surgery on a patient. (“While the OHMD is placed in the fixed position, live data can be viewed by the surgeon or interventionalist and they can be, optionally recorded with a camera and/or displayed on a monitor. Virtual data can then be superimposed and the matching and registration of virtual data and live data can be performed.” [0269] “3D scanning can be used for imaging of the patient and/or the surgical site and/or anatomic landmarks and/or pathologic structures and/or tissues (e.g. damaged or diseased cartilage or exposed subchondral bone) and/or the surgeon or interventionalist's hands and/or fingers and/or the OR table and/or reference areas or points and/or marker, e.g. optical markers, in the operating room and/or on the patient and/or on the surgical field. … One or more optical imaging systems or 3D scanners can, for example, be used to image and/or monitor, e.g. the coordinates, position, orientation, alignment, direction of movement, speed of movement of, Anatomic landmarks, patient surface(s), organ surface(s), tissue surface(s), pathologic tissues and/or surface(s), e.g. for purposes of registration, e.g. of the patient and/or the surgical site, e.g. one or more bones or cartilage, and/or one or more OHMD's, e.g. in a common coordinate system The surgeon or interventionalist's hands and/or fingers, e.g. for Monitoring steps in an interventional procedure. Select hand and/or finger movements can be associated with corresponding surgical steps. When the 3D scanner system detects a particular hand and/or finger movement, it can trigger the display of the corresponding surgical step or the next surgical step, e.g. by displaying a predetermined virtual path, e.g. for a catheter, a virtual instrument, a virtual device etc. … One or more OHMDs, e.g. registered in a common coordinate system, e.g. with the surgical site and/or the surgeon or interventionalist's hands and/or fingers.” [0235-0240])
12. With reference to claim 4, Lang teaches obtaining surgeon hand positional information comprises obtaining positional information for positional markers that mark surgeon hand position. (“3D scanning can be used for imaging of the patient and/or the surgical site and/or anatomic landmarks and/or pathologic structures and/or tissues (e.g. damaged or diseased cartilage or exposed subchondral bone) and/or the surgeon or interventionalist's hands and/or fingers and/or the OR table and/or reference areas or points and/or marker, e.g. optical markers, in the operating room and/or on the patient and/or on the surgical field. … One or more optical imaging systems or 3D scanners can, for example, be used to image and/or monitor, e.g. the coordinates, position, orientation, alignment, direction of movement, speed of movement of, Anatomic landmarks, patient surface(s), organ surface(s), tissue surface(s), pathologic tissues and/or surface(s), e.g. for purposes of registration, e.g. of the patient and/or the surgical site, e.g. one or more bones or cartilage, and/or one or more OHMD's, e.g. in a common coordinate system The surgeon or interventionalist's hands and/or fingers, e.g. for Monitoring steps in an interventional procedure. Select hand and/or finger movements can be associated with corresponding surgical steps. When the 3D scanner system detects a particular hand and/or finger movement, it can trigger the display of the corresponding surgical step or the next surgical step, e.g. by displaying a predetermined virtual path, e.g. for a catheter, a virtual instrument, a virtual device etc. … One or more OHMDs, e.g. registered in a common coordinate system, e.g. with the surgical site and/or the surgeon or interventionalist's hands and/or fingers.” [0235-0240])
13. With reference to claim 5, Lang teaches obtaining the patient intervention site dataset comprises obtaining at least one x-ray. (“This can also be accomplished by attaching one or more markers (e.g. retroreflective markers or markers with RF emitters for surgical navigation, or markers, e.g. with geometric patterns, for video imaging) to the x-ray imaging system that can also be tracked by the OHMD. Any relative movement between the markers attached to the table and the markers attached to the x-ray imaging system can then be measured and used for updating the registration parameters. The markers can optionally be radiopaque or include radiopaque elements. Any of the foregoing techniques, e.g. image based registration of 3D models and 2D or 3D angiograms and registration using table movement and related coordinates and/or marker movement can be combined.” [1038] “If the run-off and/or bolus chasing includes movement of the table on which the patient is positioned during the image acquisition, any 2D or 3D virtual data, e.g. from a pre-operative imaging study, e.g. a CTA or MRA or ultrasound, can be moved with the patient and/or table movement to maintain superimposition and/or alignment of a first 2D or 3D virtual data set, e.g. from the pre-operative imaging study, displayed by one or more OHMDs with a second or additional 2D or 3D virtual data sets, e.g. images obtained during the run-off and/or bolus chasing, displayed by the one or more OHMDs or a separate, standalone computer monitor using any of the combinations outlined in exemplary form in Tables 11, 13 and 14. The moving of the 2D or 3D virtual data, e.g. from a first set of virtual data from a pre-operative imaging test such as a CTA, MRA or ultrasound, displayed, using a computer processor, by the one or more OHMDs can be accomplished using the image registration including 3D-2D transformation matrices or 3D-3D registration techniques and/or the known table movement, e.g. in x, y, and/or z-direction, and/or the measured movement of one or more markers applied to the table or the patient. The image registration can include selection and/or matching and/or superimposition and/or alignment of a different volume of interest from a pre-operative vascular imaging study, e.g. an ultrasound, echocardiogram, CTA or MRA, to match the run-off and/or bolus chasing images and to maintain the superimposition and/or alignment of the first virtual dataset displayed by the one or more OHMDs with the second virtual data set, i.e. the run-off and/or bolus chasing images, e.g. displayed by the one or more OHMDs or a standalone or separate computer monitor or display.” [1043] “a 3D-2D registration can be performed to match a 3D model of the marker(s) on the catheter and/or catheter tip or other device (e.g. a guidewire, sheath, stent, coil, instrument, implant, vascular prosthesis or other intra-vascular or endoluminal instrument and/or device) with the projected marker of the catheter and/or catheter tip or other device (e.g. a guidewire, sheath, stent, coil, instrument, implant, vascular prosthesis or other intra-vascular or endoluminal instrument and/or device) on the x-ray image. The virtual 3D model of the marker(s) and/or the catheter and/or catheter tip or other device, e.g. a guidewire, sheath, stent, coil, instrument, implant, vascular prosthesis or other intra-vascular or endoluminal instrument and/or device, can then be displayed by one or more OHMDs, for example superimposed on a vascular structure or vascular tree or other structures of the patient displayed by a standalone or separate computer monitor or displayed by the OHMD. Alternatively, a search for the projected marker of the catheter and/or catheter tip or other device (e.g. a guidewire, sheath, stent, coil, instrument, implant, vascular prosthesis or other intra-vascular or endoluminal instrument and/or device) on the x-ray image can be performed by cross correlation of a tip template with the pixels in the x-ray image. After this step, the position and direction of the catheter tip in the 2D coordinate system of the x-ray image can be known. Biplanar angiography systems can utilize two separate x-ray sources in a fixed and known spatial configuration and allow for simultaneous visualization of two imaging planes.” [1045])
14. With reference to claim 6, Lang teaches obtaining the patient intervention site dataset comprises obtaining MRI data. (“A 3D model of the vasculature and/or the heart can be generated using pre-operative 3D imaging data, for example a computed tomography (CT) scan or magnetic resonance imaging (MRI) scan. Preferably, the pre-operative 3D images are acquired using vascular enhancement such as CT angiography, e.g. spiral CT angiography, (with use of contrast media) or MR angiography with injection of a contrast agent or use of contrast enhancing MRI pulse sequences.” [1027] “If the run-off and/or bolus chasing includes movement of the table on which the patient is positioned during the image acquisition, any 2D or 3D virtual data, e.g. from a pre-operative imaging study, e.g. a CTA or MRA or ultrasound, can be moved with the patient and/or table movement to maintain superimposition and/or alignment of a first 2D or 3D virtual data set, e.g. from the pre-operative imaging study, displayed by one or more OHMDs with a second or additional 2D or 3D virtual data sets, e.g. images obtained during the run-off and/or bolus chasing, displayed by the one or more OHMDs or a separate, standalone computer monitor using any of the combinations outlined in exemplary form in Tables 11, 13 and 14. The moving of the 2D or 3D virtual data, e.g. from a first set of virtual data from a pre-operative imaging test such as a CTA, MRA or ultrasound, displayed, using a computer processor, by the one or more OHMDs can be accomplished using the image registration including 3D-2D transformation matrices or 3D-3D registration techniques and/or the known table movement, e.g. in x, y, and/or z-direction, and/or the measured movement of one or more markers applied to the table or the patient. The image registration can include selection and/or matching and/or superimposition and/or alignment of a different volume of interest from a pre-operative vascular imaging study, e.g. an ultrasound, echocardiogram, CTA or MRA, to match the run-off and/or bolus chasing images and to maintain the superimposition and/or alignment of the first virtual dataset displayed by the one or more OHMDs with the second virtual data set, i.e. the run-off and/or bolus chasing images, e.g. displayed by the one or more OHMDs or a standalone or separate computer monitor or display.” [1043])
15. With reference to claim 7, Lang teaches obtaining the patient intervention site dataset comprises obtaining at least one patient image. (“Pre-operative, intra-operative or post-operative images of the patient can be acquired 240. The image data can optionally be segmented 241. 3D reconstructions of the patient's anatomy or pathology including multiple different tissues, e.g. using different colors or shading, can be generated 242. Virtual 3D models of surgical instruments and devices components can be generated which can include their predetermined position, location, rotation, orientation, alignment and/or direction 243. The virtual 3D models can be registered, for example in relationship to the OHMD and the patient 244. The virtual 3D models can be registered relative to the live patient data 245.” [0796] “If the run-off and/or bolus chasing includes movement of the table on which the patient is positioned during the image acquisition, any 2D or 3D virtual data, e.g. from a pre-operative imaging study, e.g. a CTA or MRA or ultrasound, can be moved with the patient and/or table movement to maintain superimposition and/or alignment of a first 2D or 3D virtual data set, e.g. from the pre-operative imaging study, displayed by one or more OHMDs with a second or additional 2D or 3D virtual data sets, e.g. images obtained during the run-off and/or bolus chasing, displayed by the one or more OHMDs or a separate, standalone computer monitor using any of the combinations outlined in exemplary form in Tables 11, 13 and 14. The moving of the 2D or 3D virtual data, e.g. from a first set of virtual data from a pre-operative imaging test such as a CTA, MRA or ultrasound, displayed, using a computer processor, by the one or more OHMDs can be accomplished using the image registration including 3D-2D transformation matrices or 3D-3D registration techniques and/or the known table movement, e.g. in x, y, and/or z-direction, and/or the measured movement of one or more markers applied to the table or the patient. The image registration can include selection and/or matching and/or superimposition and/or alignment of a different volume of interest from a pre-operative vascular imaging study, e.g. an ultrasound, echocardiogram, CTA or MRA, to match the run-off and/or bolus chasing images and to maintain the superimposition and/or alignment of the first virtual dataset displayed by the one or more OHMDs with the second virtual data set, i.e. the run-off and/or bolus chasing images, e.g. displayed by the one or more OHMDs or a standalone or separate computer monitor or display.” [1043])
16. With reference to claim 8, Lang teaches obtaining the patient intervention site dataset comprises imaging a patient prior to a surgery on the patient. (“Pre-operative, intra-operative or post-operative images of the patient can be acquired 240. The image data can optionally be segmented 241. 3D reconstructions of the patient's anatomy or pathology including multiple different tissues, e.g. using different colors or shading, can be generated 242. Virtual 3D models of surgical instruments and devices components can be generated which can include their predetermined position, location, rotation, orientation, alignment and/or direction 243. The virtual 3D models can be registered, for example in relationship to the OHMD and the patient 244. The virtual 3D models can be registered relative to the live patient data 245.” [0796] “If the run-off and/or bolus chasing includes movement of the table on which the patient is positioned during the image acquisition, any 2D or 3D virtual data, e.g. from a pre-operative imaging study, e.g. a CTA or MRA or ultrasound, can be moved with the patient and/or table movement to maintain superimposition and/or alignment of a first 2D or 3D virtual data set, e.g. from the pre-operative imaging study, displayed by one or more OHMDs with a second or additional 2D or 3D virtual data sets, e.g. images obtained during the run-off and/or bolus chasing, displayed by the one or more OHMDs or a separate, standalone computer monitor using any of the combinations outlined in exemplary form in Tables 11, 13 and 14. The moving of the 2D or 3D virtual data, e.g. from a first set of virtual data from a pre-operative imaging test such as a CTA, MRA or ultrasound, displayed, using a computer processor, by the one or more OHMDs can be accomplished using the image registration including 3D-2D transformation matrices or 3D-3D registration techniques and/or the known table movement, e.g. in x, y, and/or z-direction, and/or the measured movement of one or more markers applied to the table or the patient. The image registration can include selection and/or matching and/or superimposition and/or alignment of a different volume of interest from a pre-operative vascular imaging study, e.g. an ultrasound, echocardiogram, CTA or MRA, to match the run-off and/or bolus chasing images and to maintain the superimposition and/or alignment of the first virtual dataset displayed by the one or more OHMDs with the second virtual data set, i.e. the run-off and/or bolus chasing images, e.g. displayed by the one or more OHMDs or a standalone or separate computer monitor or display.” [1043])
17. With reference to claim 10, Lang teaches the patient intervention site dataset is suitable for a three dimensional depiction. (“If the run-off and/or bolus chasing includes movement of the table on which the patient is positioned during the image acquisition, any 2D or 3D virtual data, e.g. from a pre-operative imaging study, e.g. a CTA or MRA or ultrasound, can be moved with the patient and/or table movement to maintain superimposition and/or alignment of a first 2D or 3D virtual data set, e.g. from the pre-operative imaging study, displayed by one or more OHMDs with a second or additional 2D or 3D virtual data sets, e.g. images obtained during the run-off and/or bolus chasing, displayed by the one or more OHMDs or a separate, standalone computer monitor using any of the combinations outlined in exemplary form in Tables 11, 13 and 14. The moving of the 2D or 3D virtual data, e.g. from a first set of virtual data from a pre-operative imaging test such as a CTA, MRA or ultrasound, displayed, using a computer processor, by the one or more OHMDs can be accomplished using the image registration including 3D-2D transformation matrices or 3D-3D registration techniques and/or the known table movement, e.g. in x, y, and/or z-direction, and/or the measured movement of one or more markers applied to the table or the patient. The image registration can include selection and/or matching and/or superimposition and/or alignment of a different volume of interest from a pre-operative vascular imaging study, e.g. an ultrasound, echocardiogram, CTA or MRA, to match the run-off and/or bolus chasing images and to maintain the superimposition and/or alignment of the first virtual dataset displayed by the one or more OHMDs with the second virtual data set, i.e. the run-off and/or bolus chasing images, e.g. displayed by the one or more OHMDs or a standalone or separate computer monitor or display.” [1043])
18. With reference to claim 11, Lang teaches applying the surgeon hand positional information to the patient intervention site dataset comprises calculating a 3-dimensional image of the patient intervention site. (“Calculate 3D transformation of first 3D vascular tree representation using transformation T” [1398] “Any of the 2D or 3D virtual data in any of the embodiments throughout the specification can include an arterial run-off, e.g. a peripheral run-off or an aortic run-off, optionally including bolus chasing. Bolus chasing can permit real-time visualization of the contrast bolus so that it can be followed, for example, peripherally with images, e.g. digital images, being acquired at a suitable frame rate. If the run-off and/or bolus chasing includes movement of the table on which the patient is positioned during the image acquisition, any 2D or 3D virtual data, e.g. from a pre-operative imaging study, e.g. a CTA or MRA or ultrasound, can be moved with the patient and/or table movement to maintain superimposition and/or alignment of a first 2D or 3D virtual data set, e.g. from the pre-operative imaging study, displayed by one or more OHMDs with a second or additional 2D or 3D virtual data sets, e.g. images obtained during the run-off and/or bolus chasing, displayed by the one or more OHMDs or a separate, standalone computer monitor using any of the combinations outlined in exemplary form in Tables 11, 13 and 14. The moving of the 2D or 3D virtual data, e.g. from a first set of virtual data from a pre-operative imaging test such as a CTA, MRA or ultrasound, displayed, using a computer processor, by the one or more OHMDs can be accomplished using the image registration including 3D-2D transformation matrices or 3D-3D registration techniques and/or the known table movement, e.g. in x, y, and/or z-direction, and/or the measured movement of one or more markers applied to the table or the patient. The image registration can include selection and/or matching and/or superimposition and/or alignment of a different volume of interest from a pre-operative vascular imaging study, e.g. an ultrasound, echocardiogram, CTA or MRA, to match the run-off and/or bolus chasing images and to maintain the superimposition and/or alignment of the first virtual dataset displayed by the one or more OHMDs with the second virtual data set, i.e. the run-off and/or bolus chasing images, e.g. displayed by the one or more OHMDs or a standalone or separate computer monitor or display.” [1043] “A combination of data can be beneficial for more accurate measurement of changes in position or orientation of the surgeon or interventionalist's head, body, operating arm, hand, or the patient.” [0291] “A processor receives the electrical signals and calculates three-dimensional position information of tissue surfaces based on changes in the relative phase of the emitted optical radiation and the received optical radiation scattered by the surfaces.” [0252] “3D scanning can be used for imaging of the patient and/or the surgical site and/or anatomic landmarks and/or pathologic structures and/or tissues (e.g. damaged or diseased cartilage or exposed subchondral bone) and/or the surgeon or interventionalist's hands and/or fingers and/or the OR table and/or reference areas or points and/or marker, e.g. optical markers, in the operating room and/or on the patient and/or on the surgical field. … One or more optical imaging systems or 3D scanners can, for example, be used to image and/or monitor, e.g. the coordinates, position, orientation, alignment, direction of movement, speed of movement of, Anatomic landmarks, patient surface(s), organ surface(s), tissue surface(s), pathologic tissues and/or surface(s), e.g. for purposes of registration, e.g. of the patient and/or the surgical site, e.g. one or more bones or cartilage, and/or one or more OHMD's, e.g. in a common coordinate system The surgeon or interventionalist's hands and/or fingers, e.g. for Monitoring steps in an interventional procedure. Select hand and/or finger movements can be associated with corresponding surgical steps. When the 3D scanner system detects a particular hand and/or finger movement, it can trigger the display of the corresponding surgical step or the next surgical step, e.g. by displaying a predetermined virtual path, e.g. for a catheter, a virtual instrument, a virtual device etc. … One or more OHMDs, e.g. registered in a common coordinate system, e.g. with the surgical site and/or the surgeon or interventionalist's hands and/or fingers.” [0235-0240])
Lang does not explicitly teach assess impact of surgical hand position on a patient intervention site. This is what Verard teaches (“a fellow/practitioner could first select a program called "HIP" by activating a trigger point 504 to display a 3D CT image of a patient's (subject's) hip, and then select different "HIP IMPLANTS" from different manufacturers to see and "feel" which implant would fit best for the particular patient. It is also possible to use (e.g., physically hold and manipulate) the actual implant in the air and position it within the 3D holographic display to see, feel and assess fit (e.g., if and how well such implant may fit the particular patient).” [0059] “Robotics via a master/slave configuration can be used, where a shape sensed analog 604 of the device 602 moving within the display 158 is employed to actuate the motion of the actual device 602 within a target region 606. A practitioner's (surgeon's, physician's, etc.) hands 610 or voice can be tracked by sensor-based and/or voice-based techniques, such as by, e.g., tracking a physician's hands using a shape-sensing device 608 and shape sensing system 614 in the 3D holographic display. Accordingly, a practitioner's movements (including, e.g., (re)positioning, orientation, etc. of their hands) performed in the holographic display 158 can be transmitted to the device 602, such as a robot 612 (e.g., robotically controlled instruments) inside the patient to replicate such movements within the patient's body, and thereby perform the actual surgery, procedure or task inside the patient's body. Thus, a surgeon can see, touch and feel a 3D holographic display of an organ, perform a procedure thereon (i.e., within the 3D holographic display), causing such procedure to be performed inside of a patient on the actual organ via, or simply to move instruments, e.g., robotically controlled instruments.” [0062]) Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Verard into Lang, in order to improve accuracy and effectiveness in medical applications.
19. With reference to claim 12, Lang teaches applying the surgeon hand positional information to the patient intervention site dataset (“Any of the 2D or 3D virtual data in any of the embodiments throughout the specification can include an arterial run-off, e.g. a peripheral run-off or an aortic run-off, optionally including bolus chasing. Bolus chasing can permit real-time visualization of the contrast bolus so that it can be followed, for example, peripherally with images, e.g. digital images, being acquired at a suitable frame rate. If the run-off and/or bolus chasing includes movement of the table on which the patient is positioned during the image acquisition, any 2D or 3D virtual data, e.g. from a pre-operative imaging study, e.g. a CTA or MRA or ultrasound, can be moved with the patient and/or table movement to maintain superimposition and/or alignment of a first 2D or 3D virtual data set, e.g. from the pre-operative imaging study, displayed by one or more OHMDs with a second or additional 2D or 3D virtual data sets, e.g. images obtained during the run-off and/or bolus chasing, displayed by the one or more OHMDs or a separate, standalone computer monitor using any of the combinations outlined in exemplary form in Tables 11, 13 and 14. The moving of the 2D or 3D virtual data, e.g. from a first set of virtual data from a pre-operative imaging test such as a CTA, MRA or ultrasound, displayed, using a computer processor, by the one or more OHMDs can be accomplished using the image registration including 3D-2D transformation matrices or 3D-3D registration techniques and/or the known table movement, e.g. in x, y, and/or z-direction, and/or the measured movement of one or more markers applied to the table or the patient. The image registration can include selection and/or matching and/or superimposition and/or alignment of a different volume of interest from a pre-operative vascular imaging study, e.g. an ultrasound, echocardiogram, CTA or MRA, to match the run-off and/or bolus chasing images and to maintain the superimposition and/or alignment of the first virtual dataset displayed by the one or more OHMDs with the second virtual data set, i.e. the run-off and/or bolus chasing images, e.g. displayed by the one or more OHMDs or a standalone or separate computer monitor or display.” [1043] “A combination of data can be beneficial for more accurate measurement of changes in position or orientation of the surgeon or interventionalist's head, body, operating arm, hand, or the patient.” [0291] “3D scanning can be used for imaging of the patient and/or the surgical site and/or anatomic landmarks and/or pathologic structures and/or tissues (e.g. damaged or diseased cartilage or exposed subchondral bone) and/or the surgeon or interventionalist's hands and/or fingers and/or the OR table and/or reference areas or points and/or marker, e.g. optical markers, in the operating room and/or on the patient and/or on the surgical field. … One or more optical imaging systems or 3D scanners can, for example, be used to image and/or monitor, e.g. the coordinates, position, orientation, alignment, direction of movement, speed of movement of, Anatomic landmarks, patient surface(s), organ surface(s), tissue surface(s), pathologic tissues and/or surface(s), e.g. for purposes of registration, e.g. of the patient and/or the surgical site, e.g. one or more bones or cartilage, and/or one or more OHMD's, e.g. in a common coordinate system The surgeon or interventionalist's hands and/or fingers, e.g. for Monitoring steps in an interventional procedure. Select hand and/or finger movements can be associated with corresponding surgical steps. When the 3D scanner system detects a particular hand and/or finger movement, it can trigger the display of the corresponding surgical step or the next surgical step, e.g. by displaying a predetermined virtual path, e.g. for a catheter, a virtual instrument, a virtual device etc. … One or more OHMDs, e.g. registered in a common coordinate system, e.g. with the surgical site and/or the surgeon or interventionalist's hands and/or fingers.” [0235-0240])
Lang does not explicitly teach assess impact of surgical hand position on a patient intervention site comprises projecting into space the 3-dimensional image of the impact of surgical hand position. This is what Verard teaches (“A localization system 120 includes a coordinate system 122 to which a holographic image or hologram 124 is registered. The localization system 120 may also be employed to register a monitored object 128, which may include virtual instruments, which are separately created and controlled, real instruments, a physician's hands, fingers or other anatomical parts, etc. The localization system 120 may include an electromagnetic tracking system, a shape sensing system, such as a fiber optic based shape sensing system, an optical sensing system, including light sensors and arrays, or other sensing modality, etc. The localization system 120 is employed to define spatial regions in and around the hologram or the holographic image 124 to enable a triggering of different functions or actions as a result of movement in the area of the hologram 124. For example, dynamic locations of a physician's hands may be tracked using a fiber optic shape sensing device. When the physician's hands enter the same space, e.g., a monitored space 126 about a projected hologram 124, the intensity of the hologram may be increased. In another example, the physician's hand movements may be employed to spatially alter the position or orientation of the hologram 124 or to otherwise interact with the hologram 124.” [0034] “a fellow/practitioner could first select a program called "HIP" by activating a trigger point 504 to display a 3D CT image of a patient's (subject's) hip, and then select different "HIP IMPLANTS" from different manufacturers to see and "feel" which implant would fit best for the particular patient. It is also possible to use (e.g., physically hold and manipulate) the actual implant in the air and position it within the 3D holographic display to see, feel and assess fit (e.g., if and how well such implant may fit the particular patient).” [0059] “Robotics via a master/slave configuration can be used, where a shape sensed analog 604 of the device 602 moving within the display 158 is employed to actuate the motion of the actual device 602 within a target region 606. A practitioner's (surgeon's, physician's, etc.) hands 610 or voice can be tracked by sensor-based and/or voice-based techniques, such as by, e.g., tracking a physician's hands using a shape-sensing device 608 and shape sensing system 614 in the 3D holographic display. Accordingly, a practitioner's movements (including, e.g., (re)positioning, orientation, etc. of their hands) performed in the holographic display 158 can be transmitted to the device 602, such as a robot 612 (e.g., robotically controlled instruments) inside the patient to replicate such movements within the patient's body, and thereby perform the actual surgery, procedure or task inside the patient's body. Thus, a surgeon can see, touch and feel a 3D holographic display of an organ, perform a procedure thereon (i.e., within the 3D holographic display), causing such procedure to be performed inside of a patient on the actual organ via, or simply to move instruments, e.g., robotically controlled instruments.” [0062]) Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Verard into Lang, in order to improve accuracy and effectiveness in medical applications.
20. With reference to claim 13, Lang teaches obtaining surgeon hand positional information further comprises obtaining surgical insert positional information. (“3D scanning can be used for imaging of the patient and/or the surgical site and/or anatomic landmarks and/or pathologic structures and/or tissues (e.g. damaged or diseased cartilage or exposed subchondral bone) and/or the surgeon or interventionalist's hands and/or fingers and/or the OR table and/or reference areas or points and/or marker, e.g. optical markers, in the operating room and/or on the patient and/or on the surgical field. … One or more optical imaging systems or 3D scanners can, for example, be used to image and/or monitor, e.g. the coordinates, position, orientation, alignment, direction of movement, speed of movement of, Anatomic landmarks, patient surface(s), organ surface(s), tissue surface(s), pathologic tissues and/or surface(s), e.g. for purposes of registration, e.g. of the patient and/or the surgical site, e.g. one or more bones or cartilage, and/or one or more OHMD's, e.g. in a common coordinate system The surgeon or interventionalist's hands and/or fingers, e.g. for Monitoring steps in an interventional procedure. Select hand and/or finger movements can be associated with corresponding surgical steps. When the 3D scanner system detects a particular hand and/or finger movement, it can trigger the display of the corresponding surgical step or the next surgical step, e.g. by displaying a predetermined virtual path, e.g. for a catheter, a virtual instrument, a virtual device etc. … One or more OHMDs, e.g. registered in a common coordinate system, e.g. with the surgical site and/or the surgeon or interventionalist's hands and/or fingers.” [0235-0240] “With guidance in mixed reality environment, e.g. with stereoscopic display like an electronic holographic environment, a virtual surgical guide, tool, instrument or implant can be superimposed onto the surgical site, e.g. an organ or a tumor. Further, the physical guide, tool, instrument or implant can be aligned with the 2D or 3D representation of the virtual surgical guide, tool, instrument or implant.” [0305])
21. With reference to claim 14, Lang teaches applying the surgeon hand positional information to the patient intervention site dataset further comprises applying the surgical insert positional information to the patient intervention site dataset (“Any of the 2D or 3D virtual data in any of the embodiments throughout the specification can include an arterial run-off, e.g. a peripheral run-off or an aortic run-off, optionally including bolus chasing. Bolus chasing can permit real-time visualization of the contrast bolus so that it can be followed, for example, peripherally with images, e.g. digital images, being acquired at a suitable frame rate. If the run-off and/or bolus chasing includes movement of the table on which the patient is positioned during the image acquisition, any 2D or 3D virtual data, e.g. from a pre-operative imaging study, e.g. a CTA or MRA or ultrasound, can be moved with the patient and/or table movement to maintain superimposition and/or alignment of a first 2D or 3D virtual data set, e.g. from the pre-operative imaging study, displayed by one or more OHMDs with a second or additional 2D or 3D virtual data sets, e.g. images obtained during the run-off and/or bolus chasing, displayed by the one or more OHMDs or a separate, standalone computer monitor using any of the combinations outlined in exemplary form in Tables 11, 13 and 14. The moving of the 2D or 3D virtual data, e.g. from a first set of virtual data from a pre-operative imaging test such as a CTA, MRA or ultrasound, displayed, using a computer processor, by the one or more OHMDs can be accomplished using the image registration including 3D-2D transformation matrices or 3D-3D registration techniques and/or the known table movement, e.g. in x, y, and/or z-direction, and/or the measured movement of one or more markers applied to the table or the patient. The image registration can include selection and/or matching and/or superimposition and/or alignment of a different volume of interest from a pre-operative vascular imaging study, e.g. an ultrasound, echocardiogram, CTA or MRA, to match the run-off and/or bolus chasing images and to maintain the superimposition and/or alignment of the first virtual dataset displayed by the one or more OHMDs with the second virtual data set, i.e. the run-off and/or bolus chasing images, e.g. displayed by the one or more OHMDs or a standalone or separate computer monitor or display.” [1043] “With guidance in mixed reality environment, e.g. with stereoscopic display like an electronic holographic environment, a virtual surgical guide, tool, instrument or implant can be superimposed onto the surgical site, e.g. an organ or a tumor. Further, the physical guide, tool, instrument or implant can be aligned with the 2D or 3D representation of the virtual surgical guide, tool, instrument or implant.” [0305] “A combination of data can be beneficial for more accurate measurement of changes in position or orientation of the surgeon or interventionalist's head, body, operating arm, hand, or the patient.” [0291] “3D scanning can be used for imaging of the patient and/or the surgical site and/or anatomic landmarks and/or pathologic structures and/or tissues (e.g. damaged or diseased cartilage or exposed subchondral bone) and/or the surgeon or interventionalist's hands and/or fingers and/or the OR table and/or reference areas or points and/or marker, e.g. optical markers, in the operating room and/or on the patient and/or on the surgical field. … One or more optical imaging systems or 3D scanners can, for example, be used to image and/or monitor, e.g. the coordinates, position, orientation, alignment, direction of movement, speed of movement of, Anatomic landmarks, patient surface(s), organ surface(s), tissue surface(s), pathologic tissues and/or surface(s), e.g. for purposes of registration, e.g. of the patient and/or the surgical site, e.g. one or more bones or cartilage, and/or one or more OHMD's, e.g. in a common coordinate system The surgeon or interventionalist's hands and/or fingers, e.g. for Monitoring steps in an interventional procedure. Select hand and/or finger movements can be associated with corresponding surgical steps. When the 3D scanner system detects a particular hand and/or finger movement, it can trigger the display of the corresponding surgical step or the next surgical step, e.g. by displaying a predetermined virtual path, e.g. for a catheter, a virtual instrument, a virtual device etc. … One or more OHMDs, e.g. registered in a common coordinate system, e.g. with the surgical site and/or the surgeon or interventionalist's hands and/or fingers.” [0235-0240])
Lang does not explicitly teach assess impact of surgical hand position on a patient intervention site comprises assess impact of the surgical insert on the patient intervention site. This is what Verard teaches (“a fellow/practitioner could first select a program called "HIP" by activating a trigger point 504 to display a 3D CT image of a patient's (subject's) hip, and then select different "HIP IMPLANTS" from different manufacturers to see and "feel" which implant would fit best for the particular patient. It is also possible to use (e.g., physically hold and manipulate) the actual implant in the air and position it within the 3D holographic display to see, feel and assess fit (e.g., if and how well such implant may fit the particular patient).” [0059] “Robotics via a master/slave configuration can be used, where a shape sensed analog 604 of the device 602 moving within the display 158 is employed to actuate the motion of the actual device 602 within a target region 606. A practitioner's (surgeon's, physician's, etc.) hands 610 or voice can be tracked by sensor-based and/or voice-based techniques, such as by, e.g., tracking a physician's hands using a shape-sensing device 608 and shape sensing system 614 in the 3D holographic display. Accordingly, a practitioner's movements (including, e.g., (re)positioning, orientation, etc. of their hands) performed in the holographic display 158 can be transmitted to the device 602, such as a robot 612 (e.g., robotically controlled instruments) inside the patient to replicate such movements within the patient's body, and thereby perform the actual surgery, procedure or task inside the patient's body. Thus, a surgeon can see, touch and feel a 3D holographic display of an organ, perform a procedure thereon (i.e., within the 3D holographic display), causing such procedure to be performed inside of a patient on the actual organ via, or simply to move instruments, e.g., robotically controlled instruments.” [0062]) Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Verard into Lang, in order to improve accuracy and effectiveness in medical applications.
22. With reference to claim 15, Lang teaches the applying the surgical insert positional information to the patient intervention site dataset (“Any of the 2D or 3D virtual data in any of the embodiments throughout the specification can include an arterial run-off, e.g. a peripheral run-off or an aortic run-off, optionally including bolus chasing. Bolus chasing can permit real-time visualization of the contrast bolus so that it can be followed, for example, peripherally with images, e.g. digital images, being acquired at a suitable frame rate. If the run-off and/or bolus chasing includes movement of the table on which the patient is positioned during the image acquisition, any 2D or 3D virtual data, e.g. from a pre-operative imaging study, e.g. a CTA or MRA or ultrasound, can be moved with the patient and/or table movement to maintain superimposition and/or alignment of a first 2D or 3D virtual data set, e.g. from the pre-operative imaging study, displayed by one or more OHMDs with a second or additional 2D or 3D virtual data sets, e.g. images obtained during the run-off and/or bolus chasing, displayed by the one or more OHMDs or a separate, standalone computer monitor using any of the combinations outlined in exemplary form in Tables 11, 13 and 14. The moving of the 2D or 3D virtual data, e.g. from a first set of virtual data from a pre-operative imaging test such as a CTA, MRA or ultrasound, displayed, using a computer processor, by the one or more OHMDs can be accomplished using the image registration including 3D-2D transformation matrices or 3D-3D registration techniques and/or the known table movement, e.g. in x, y, and/or z-direction, and/or the measured movement of one or more markers applied to the table or the patient. The image registration can include selection and/or matching and/or superimposition and/or alignment of a different volume of interest from a pre-operative vascular imaging study, e.g. an ultrasound, echocardiogram, CTA or MRA, to match the run-off and/or bolus chasing images and to maintain the superimposition and/or alignment of the first virtual dataset displayed by the one or more OHMDs with the second virtual data set, i.e. the run-off and/or bolus chasing images, e.g. displayed by the one or more OHMDs or a standalone or separate computer monitor or display.” [1043] “With guidance in mixed reality environment, e.g. with stereoscopic display like an electronic holographic environment, a virtual surgical guide, tool, instrument or implant can be superimposed onto the surgical site, e.g. an organ or a tumor. Further, the physical guide, tool, instrument or implant can be aligned with the 2D or 3D representation of the virtual surgical guide, tool, instrument or implant.” [0305] “A combination of data can be beneficial for more accurate measurement of changes in position or orientation of the surgeon or interventionalist's head, body, operating arm, hand, or the patient.” [0291] “3D scanning can be used for imaging of the patient and/or the surgical site and/or anatomic landmarks and/or pathologic structures and/or tissues (e.g. damaged or diseased cartilage or exposed subchondral bone) and/or the surgeon or interventionalist's hands and/or fingers and/or the OR table and/or reference areas or points and/or marker, e.g. optical markers, in the operating room and/or on the patient and/or on the surgical field. … One or more optical imaging systems or 3D scanners can, for example, be used to image and/or monitor, e.g. the coordinates, position, orientation, alignment, direction of movement, speed of movement of, Anatomic landmarks, patient surface(s), organ surface(s), tissue surface(s), pathologic tissues and/or surface(s), e.g. for purposes of registration, e.g. of the patient and/or the surgical site, e.g. one or more bones or cartilage, and/or one or more OHMD's, e.g. in a common coordinate system The surgeon or interventionalist's hands and/or fingers, e.g. for Monitoring steps in an interventional procedure. Select hand and/or finger movements can be associated with corresponding surgical steps. When the 3D scanner system detects a particular hand and/or finger movement, it can trigger the display of the corresponding surgical step or the next surgical step, e.g. by displaying a predetermined virtual path, e.g. for a catheter, a virtual instrument, a virtual device etc. … One or more OHMDs, e.g. registered in a common coordinate system, e.g. with the surgical site and/or the surgeon or interventionalist's hands and/or fingers.” [0235-0240])
Lang does not explicitly teach assess impact of the surgical insert on the patient intervention site further comprises selecting the surgical insert based upon assessing the impact of the surgical insert. This is what Verard teaches (“a fellow/practitioner could first select a program called "HIP" by activating a trigger point 504 to display a 3D CT image of a patient's (subject's) hip, and then select different "HIP IMPLANTS" from different manufacturers to see and "feel" which implant would fit best for the particular patient. It is also possible to use (e.g., physically hold and manipulate) the actual implant in the air and position it within the 3D holographic display to see, feel and assess fit (e.g., if and how well such implant may fit the particular patient).” [0059] “Robotics via a master/slave configuration can be used, where a shape sensed analog 604 of the device 602 moving within the display 158 is employed to actuate the motion of the actual device 602 within a target region 606. A practitioner's (surgeon's, physician's, etc.) hands 610 or voice can be tracked by sensor-based and/or voice-based techniques, such as by, e.g., tracking a physician's hands using a shape-sensing device 608 and shape sensing system 614 in the 3D holographic display. Accordingly, a practitioner's movements (including, e.g., (re)positioning, orientation, etc. of their hands) performed in the holographic display 158 can be transmitted to the device 602, such as a robot 612 (e.g., robotically controlled instruments) inside the patient to replicate such movements within the patient's body, and thereby perform the actual surgery, procedure or task inside the patient's body. Thus, a surgeon can see, touch and feel a 3D holographic display of an organ, perform a procedure thereon (i.e., within the 3D holographic display), causing such procedure to be performed inside of a patient on the actual organ via, or simply to move instruments, e.g., robotically controlled instruments.” [0062]) Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Verard into Lang, in order to improve accuracy and effectiveness in medical applications.
23. With reference to claim 16, Lang teaches applying the surgeon insert positional information to the patient intervention site dataset comprises calculating a 3-dimensional image of the patient intervention site. (“Calculate 3D transformation of first 3D vascular tree representation using transformation T” [1398] “Any of the 2D or 3D virtual data in any of the embodiments throughout the specification can include an arterial run-off, e.g. a peripheral run-off or an aortic run-off, optionally including bolus chasing. Bolus chasing can permit real-time visualization of the contrast bolus so that it can be followed, for example, peripherally with images, e.g. digital images, being acquired at a suitable frame rate. If the run-off and/or bolus chasing includes movement of the table on which the patient is positioned during the image acquisition, any 2D or 3D virtual data, e.g. from a pre-operative imaging study, e.g. a CTA or MRA or ultrasound, can be moved with the patient and/or table movement to maintain superimposition and/or alignment of a first 2D or 3D virtual data set, e.g. from the pre-operative imaging study, displayed by one or more OHMDs with a second or additional 2D or 3D virtual data sets, e.g. images obtained during the run-off and/or bolus chasing, displayed by the one or more OHMDs or a separate, standalone computer monitor using any of the combinations outlined in exemplary form in Tables 11, 13 and 14. The moving of the 2D or 3D virtual data, e.g. from a first set of virtual data from a pre-operative imaging test such as a CTA, MRA or ultrasound, displayed, using a computer processor, by the one or more OHMDs can be accomplished using the image registration including 3D-2D transformation matrices or 3D-3D registration techniques and/or the known table movement, e.g. in x, y, and/or z-direction, and/or the measured movement of one or more markers applied to the table or the patient. The image registration can include selection and/or matching and/or superimposition and/or alignment of a different volume of interest from a pre-operative vascular imaging study, e.g. an ultrasound, echocardiogram, CTA or MRA, to match the run-off and/or bolus chasing images and to maintain the superimposition and/or alignment of the first virtual dataset displayed by the one or more OHMDs with the second virtual data set, i.e. the run-off and/or bolus chasing images, e.g. displayed by the one or more OHMDs or a standalone or separate computer monitor or display.” [1043] “With guidance in mixed reality environment, e.g. with stereoscopic display like an electronic holographic environment, a virtual surgical guide, tool, instrument or implant can be superimposed onto the surgical site, e.g. an organ or a tumor. Further, the physical guide, tool, instrument or implant can be aligned with the 2D or 3D representation of the virtual surgical guide, tool, instrument or implant.” [0305] “A combination of data can be beneficial for more accurate measurement of changes in position or orientation of the surgeon or interventionalist's head, body, operating arm, hand, or the patient.” [0291] “A processor receives the electrical signals and calculates three-dimensional position information of tissue surfaces based on changes in the relative phase of the emitted optical radiation and the received optical radiation scattered by the surfaces.” [0252] “3D scanning can be used for imaging of the patient and/or the surgical site and/or anatomic landmarks and/or pathologic structures and/or tissues (e.g. damaged or diseased cartilage or exposed subchondral bone) and/or the surgeon or interventionalist's hands and/or fingers and/or the OR table and/or reference areas or points and/or marker, e.g. optical markers, in the operating room and/or on the patient and/or on the surgical field. … One or more optical imaging systems or 3D scanners can, for example, be used to image and/or monitor, e.g. the coordinates, position, orientation, alignment, direction of movement, speed of movement of, Anatomic landmarks, patient surface(s), organ surface(s), tissue surface(s), pathologic tissues and/or surface(s), e.g. for purposes of registration, e.g. of the patient and/or the surgical site, e.g. one or more bones or cartilage, and/or one or more OHMD's, e.g. in a common coordinate system The surgeon or interventionalist's hands and/or fingers, e.g. for Monitoring steps in an interventional procedure. Select hand and/or finger movements can be associated with corresponding surgical steps. When the 3D scanner system detects a particular hand and/or finger movement, it can trigger the display of the corresponding surgical step or the next surgical step, e.g. by displaying a predetermined virtual path, e.g. for a catheter, a virtual instrument, a virtual device etc. … One or more OHMDs, e.g. registered in a common coordinate system, e.g. with the surgical site and/or the surgeon or interventionalist's hands and/or fingers.” [0235-0240])
Lang does not explicitly teach assess impact of surgical hand position on a patient intervention site. This is what Verard teaches (“a fellow/practitioner could first select a program called "HIP" by activating a trigger point 504 to display a 3D CT image of a patient's (subject's) hip, and then select different "HIP IMPLANTS" from different manufacturers to see and "feel" which implant would fit best for the particular patient. It is also possible to use (e.g., physically hold and manipulate) the actual implant in the air and position it within the 3D holographic display to see, feel and assess fit (e.g., if and how well such implant may fit the particular patient).” [0059] “Robotics via a master/slave configuration can be used, where a shape sensed analog 604 of the device 602 moving within the display 158 is employed to actuate the motion of the actual device 602 within a target region 606. A practitioner's (surgeon's, physician's, etc.) hands 610 or voice can be tracked by sensor-based and/or voice-based techniques, such as by, e.g., tracking a physician's hands using a shape-sensing device 608 and shape sensing system 614 in the 3D holographic display. Accordingly, a practitioner's movements (including, e.g., (re)positioning, orientation, etc. of their hands) performed in the holographic display 158 can be transmitted to the device 602, such as a robot 612 (e.g., robotically controlled instruments) inside the patient to replicate such movements within the patient's body, and thereby perform the actual surgery, procedure or task inside the patient's body. Thus, a surgeon can see, touch and feel a 3D holographic display of an organ, perform a procedure thereon (i.e., within the 3D holographic display), causing such procedure to be performed inside of a patient on the actual organ via, or simply to move instruments, e.g., robotically controlled instruments.” [0062]) Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Verard into Lang, in order to improve accuracy and effectiveness in medical applications.
24. With reference to claim 17, Lang teaches obtaining surgeon hand positional information further comprises obtaining surgical tool positional information. (“3D scanning can be used for imaging of the patient and/or the surgical site and/or anatomic landmarks and/or pathologic structures and/or tissues (e.g. damaged or diseased cartilage or exposed subchondral bone) and/or the surgeon or interventionalist's hands and/or fingers and/or the OR table and/or reference areas or points and/or marker, e.g. optical markers, in the operating room and/or on the patient and/or on the surgical field. … One or more optical imaging systems or 3D scanners can, for example, be used to image and/or monitor, e.g. the coordinates, position, orientation, alignment, direction of movement, speed of movement of, Anatomic landmarks, patient surface(s), organ surface(s), tissue surface(s), pathologic tissues and/or surface(s), e.g. for purposes of registration, e.g. of the patient and/or the surgical site, e.g. one or more bones or cartilage, and/or one or more OHMD's, e.g. in a common coordinate system The surgeon or interventionalist's hands and/or fingers, e.g. for Monitoring steps in an interventional procedure. Select hand and/or finger movements can be associated with corresponding surgical steps. When the 3D scanner system detects a particular hand and/or finger movement, it can trigger the display of the corresponding surgical step or the next surgical step, e.g. by displaying a predetermined virtual path, e.g. for a catheter, a virtual instrument, a virtual device etc. … One or more OHMDs, e.g. registered in a common coordinate system, e.g. with the surgical site and/or the surgeon or interventionalist's hands and/or fingers.” [0235-0240] “With guidance in mixed reality environment, e.g. with stereoscopic display like an electronic holographic environment, a virtual surgical guide, tool, instrument or implant can be superimposed onto the surgical site, e.g. an organ or a tumor. Further, the physical guide, tool, instrument or implant can be aligned with the 2D or 3D representation of the virtual surgical guide, tool, instrument or implant.” [0305])
25. With reference to claim 18, Lang teaches applying the surgeon hand positional information to the patient intervention site dataset further comprises applying the surgical tool positional information to the patient intervention site dataset (“Any of the 2D or 3D virtual data in any of the embodiments throughout the specification can include an arterial run-off, e.g. a peripheral run-off or an aortic run-off, optionally including bolus chasing. Bolus chasing can permit real-time visualization of the contrast bolus so that it can be followed, for example, peripherally with images, e.g. digital images, being acquired at a suitable frame rate. If the run-off and/or bolus chasing includes movement of the table on which the patient is positioned during the image acquisition, any 2D or 3D virtual data, e.g. from a pre-operative imaging study, e.g. a CTA or MRA or ultrasound, can be moved with the patient and/or table movement to maintain superimposition and/or alignment of a first 2D or 3D virtual data set, e.g. from the pre-operative imaging study, displayed by one or more OHMDs with a second or additional 2D or 3D virtual data sets, e.g. images obtained during the run-off and/or bolus chasing, displayed by the one or more OHMDs or a separate, standalone computer monitor using any of the combinations outlined in exemplary form in Tables 11, 13 and 14. The moving of the 2D or 3D virtual data, e.g. from a first set of virtual data from a pre-operative imaging test such as a CTA, MRA or ultrasound, displayed, using a computer processor, by the one or more OHMDs can be accomplished using the image registration including 3D-2D transformation matrices or 3D-3D registration techniques and/or the known table movement, e.g. in x, y, and/or z-direction, and/or the measured movement of one or more markers applied to the table or the patient. The image registration can include selection and/or matching and/or superimposition and/or alignment of a different volume of interest from a pre-operative vascular imaging study, e.g. an ultrasound, echocardiogram, CTA or MRA, to match the run-off and/or bolus chasing images and to maintain the superimposition and/or alignment of the first virtual dataset displayed by the one or more OHMDs with the second virtual data set, i.e. the run-off and/or bolus chasing images, e.g. displayed by the one or more OHMDs or a standalone or separate computer monitor or display.” [1043] “With guidance in mixed reality environment, e.g. with stereoscopic display like an electronic holographic environment, a virtual surgical guide, tool, instrument or implant can be superimposed onto the surgical site, e.g. an organ or a tumor. Further, the physical guide, tool, instrument or implant can be aligned with the 2D or 3D representation of the virtual surgical guide, tool, instrument or implant.” [0305] “A combination of data can be beneficial for more accurate measurement of changes in position or orientation of the surgeon or interventionalist's head, body, operating arm, hand, or the patient.” [0291] “3D scanning can be used for imaging of the patient and/or the surgical site and/or anatomic landmarks and/or pathologic structures and/or tissues (e.g. damaged or diseased cartilage or exposed subchondral bone) and/or the surgeon or interventionalist's hands and/or fingers and/or the OR table and/or reference areas or points and/or marker, e.g. optical markers, in the operating room and/or on the patient and/or on the surgical field. … One or more optical imaging systems or 3D scanners can, for example, be used to image and/or monitor, e.g. the coordinates, position, orientation, alignment, direction of movement, speed of movement of, Anatomic landmarks, patient surface(s), organ surface(s), tissue surface(s), pathologic tissues and/or surface(s), e.g. for purposes of registration, e.g. of the patient and/or the surgical site, e.g. one or more bones or cartilage, and/or one or more OHMD's, e.g. in a common coordinate system The surgeon or interventionalist's hands and/or fingers, e.g. for Monitoring steps in an interventional procedure. Select hand and/or finger movements can be associated with corresponding surgical steps. When the 3D scanner system detects a particular hand and/or finger movement, it can trigger the display of the corresponding surgical step or the next surgical step, e.g. by displaying a predetermined virtual path, e.g. for a catheter, a virtual instrument, a virtual device etc. … One or more OHMDs, e.g. registered in a common coordinate system, e.g. with the surgical site and/or the surgeon or interventionalist's hands and/or fingers.” [0235-0240])
Lang does not explicitly teach assess impact of surgical hand position on a patient intervention site comprises assess impact of the surgical tool on the patient intervention site. This is what Verard teaches (“Using this video capability, the object 402 (e.g., a computer aided design rendering, model, scan, etc. for an instrument, medical device, implant, etc.) may be independently manipulated relative to the hologram 124 on the display 158 or in the air. In this way, the object 402 can be placed in or around the hologram 124 to determine whether the object will fit within a portion of the hologram 124, etc. For example, an implant may be placed through a blood vessel to test the fit visually.” [0056] “a fellow/practitioner could first select a program called "HIP" by activating a trigger point 504 to display a 3D CT image of a patient's (subject's) hip, and then select different "HIP IMPLANTS" from different manufacturers to see and "feel" which implant would fit best for the particular patient. It is also possible to use (e.g., physically hold and manipulate) the actual implant in the air and position it within the 3D holographic display to see, feel and assess fit (e.g., if and how well such implant may fit the particular patient).” [0059] “Robotics via a master/slave configuration can be used, where a shape sensed analog 604 of the device 602 moving within the display 158 is employed to actuate the motion of the actual device 602 within a target region 606. A practitioner's (surgeon's, physician's, etc.) hands 610 or voice can be tracked by sensor-based and/or voice-based techniques, such as by, e.g., tracking a physician's hands using a shape-sensing device 608 and shape sensing system 614 in the 3D holographic display. Accordingly, a practitioner's movements (including, e.g., (re)positioning, orientation, etc. of their hands) performed in the holographic display 158 can be transmitted to the device 602, such as a robot 612 (e.g., robotically controlled instruments) inside the patient to replicate such movements within the patient's body, and thereby perform the actual surgery, procedure or task inside the patient's body. Thus, a surgeon can see, touch and feel a 3D holographic display of an organ, perform a procedure thereon (i.e., within the 3D holographic display), causing such procedure to be performed inside of a patient on the actual organ via, or simply to move instruments, e.g., robotically controlled instruments.” [0062]) Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Verard into Lang, in order to improve accuracy and effectiveness in medical applications.
26. With reference to claim 19, Lang teaches the applying the surgical insert positional information to the patient intervention site dataset (“Any of the 2D or 3D virtual data in any of the embodiments throughout the specification can include an arterial run-off, e.g. a peripheral run-off or an aortic run-off, optionally including bolus chasing. Bolus chasing can permit real-time visualization of the contrast bolus so that it can be followed, for example, peripherally with images, e.g. digital images, being acquired at a suitable frame rate. If the run-off and/or bolus chasing includes movement of the table on which the patient is positioned during the image acquisition, any 2D or 3D virtual data, e.g. from a pre-operative imaging study, e.g. a CTA or MRA or ultrasound, can be moved with the patient and/or table movement to maintain superimposition and/or alignment of a first 2D or 3D virtual data set, e.g. from the pre-operative imaging study, displayed by one or more OHMDs with a second or additional 2D or 3D virtual data sets, e.g. images obtained during the run-off and/or bolus chasing, displayed by the one or more OHMDs or a separate, standalone computer monitor using any of the combinations outlined in exemplary form in Tables 11, 13 and 14. The moving of the 2D or 3D virtual data, e.g. from a first set of virtual data from a pre-operative imaging test such as a CTA, MRA or ultrasound, displayed, using a computer processor, by the one or more OHMDs can be accomplished using the image registration including 3D-2D transformation matrices or 3D-3D registration techniques and/or the known table movement, e.g. in x, y, and/or z-direction, and/or the measured movement of one or more markers applied to the table or the patient. The image registration can include selection and/or matching and/or superimposition and/or alignment of a different volume of interest from a pre-operative vascular imaging study, e.g. an ultrasound, echocardiogram, CTA or MRA, to match the run-off and/or bolus chasing images and to maintain the superimposition and/or alignment of the first virtual dataset displayed by the one or more OHMDs with the second virtual data set, i.e. the run-off and/or bolus chasing images, e.g. displayed by the one or more OHMDs or a standalone or separate computer monitor or display.” [1043] “With guidance in mixed reality environment, e.g. with stereoscopic display like an electronic holographic environment, a virtual surgical guide, tool, instrument or implant can be superimposed onto the surgical site, e.g. an organ or a tumor. Further, the physical guide, tool, instrument or implant can be aligned with the 2D or 3D representation of the virtual surgical guide, tool, instrument or implant.” [0305] “A combination of data can be beneficial for more accurate measurement of changes in position or orientation of the surgeon or interventionalist's head, body, operating arm, hand, or the patient.” [0291] “3D scanning can be used for imaging of the patient and/or the surgical site and/or anatomic landmarks and/or pathologic structures and/or tissues (e.g. damaged or diseased cartilage or exposed subchondral bone) and/or the surgeon or interventionalist's hands and/or fingers and/or the OR table and/or reference areas or points and/or marker, e.g. optical markers, in the operating room and/or on the patient and/or on the surgical field. … One or more optical imaging systems or 3D scanners can, for example, be used to image and/or monitor, e.g. the coordinates, position, orientation, alignment, direction of movement, speed of movement of, Anatomic landmarks, patient surface(s), organ surface(s), tissue surface(s), pathologic tissues and/or surface(s), e.g. for purposes of registration, e.g. of the patient and/or the surgical site, e.g. one or more bones or cartilage, and/or one or more OHMD's, e.g. in a common coordinate system The surgeon or interventionalist's hands and/or fingers, e.g. for Monitoring steps in an interventional procedure. Select hand and/or finger movements can be associated with corresponding surgical steps. When the 3D scanner system detects a particular hand and/or finger movement, it can trigger the display of the corresponding surgical step or the next surgical step, e.g. by displaying a predetermined virtual path, e.g. for a catheter, a virtual instrument, a virtual device etc. … One or more OHMDs, e.g. registered in a common coordinate system, e.g. with the surgical site and/or the surgeon or interventionalist's hands and/or fingers.” [0235-0240])
Lang does not explicitly teach assess impact of the surgical tool on the patient intervention site further comprises selecting the surgical tool based upon assessing the impact of the surgical tool. This is what Verard teaches (“Using this video capability, the object 402 (e.g., a computer aided design rendering, model, scan, etc. for an instrument, medical device, implant, etc.) may be independently manipulated relative to the hologram 124 on the display 158 or in the air. In this way, the object 402 can be placed in or around the hologram 124 to determine whether the object will fit within a portion of the hologram 124, etc. For example, an implant may be placed through a blood vessel to test the fit visually.” [0056] “a fellow/practitioner could first select a program called "HIP" by activating a trigger point 504 to display a 3D CT image of a patient's (subject's) hip, and then select different "HIP IMPLANTS" from different manufacturers to see and "feel" which implant would fit best for the particular patient. It is also possible to use (e.g., physically hold and manipulate) the actual implant in the air and position it within the 3D holographic display to see, feel and assess fit (e.g., if and how well such implant may fit the particular patient).” [0059] “Robotics via a master/slave configuration can be used, where a shape sensed analog 604 of the device 602 moving within the display 158 is employed to actuate the motion of the actual device 602 within a target region 606. A practitioner's (surgeon's, physician's, etc.) hands 610 or voice can be tracked by sensor-based and/or voice-based techniques, such as by, e.g., tracking a physician's hands using a shape-sensing device 608 and shape sensing system 614 in the 3D holographic display. Accordingly, a practitioner's movements (including, e.g., (re)positioning, orientation, etc. of their hands) performed in the holographic display 158 can be transmitted to the device 602, such as a robot 612 (e.g., robotically controlled instruments) inside the patient to replicate such movements within the patient's body, and thereby perform the actual surgery, procedure or task inside the patient's body. Thus, a surgeon can see, touch and feel a 3D holographic display of an organ, perform a procedure thereon (i.e., within the 3D holographic display), causing such procedure to be performed inside of a patient on the actual organ via, or simply to move instruments, e.g., robotically controlled instruments.” [0062]) Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Verard into Lang, in order to improve accuracy and effectiveness in medical applications.
27. With reference to claim 20, Lang teaches applying the surgeon tool positional information to the patient intervention site dataset comprises calculating a 3-dimensional image of the patient intervention site. (“Calculate 3D transformation of first 3D vascular tree representation using transformation T” [1398] “Any of the 2D or 3D virtual data in any of the embodiments throughout the specification can include an arterial run-off, e.g. a peripheral run-off or an aortic run-off, optionally including bolus chasing. Bolus chasing can permit real-time visualization of the contrast bolus so that it can be followed, for example, peripherally with images, e.g. digital images, being acquired at a suitable frame rate. If the run-off and/or bolus chasing includes movement of the table on which the patient is positioned during the image acquisition, any 2D or 3D virtual data, e.g. from a pre-operative imaging study, e.g. a CTA or MRA or ultrasound, can be moved with the patient and/or table movement to maintain superimposition and/or alignment of a first 2D or 3D virtual data set, e.g. from the pre-operative imaging study, displayed by one or more OHMDs with a second or additional 2D or 3D virtual data sets, e.g. images obtained during the run-off and/or bolus chasing, displayed by the one or more OHMDs or a separate, standalone computer monitor using any of the combinations outlined in exemplary form in Tables 11, 13 and 14. The moving of the 2D or 3D virtual data, e.g. from a first set of virtual data from a pre-operative imaging test such as a CTA, MRA or ultrasound, displayed, using a computer processor, by the one or more OHMDs can be accomplished using the image registration including 3D-2D transformation matrices or 3D-3D registration techniques and/or the known table movement, e.g. in x, y, and/or z-direction, and/or the measured movement of one or more markers applied to the table or the patient. The image registration can include selection and/or matching and/or superimposition and/or alignment of a different volume of interest from a pre-operative vascular imaging study, e.g. an ultrasound, echocardiogram, CTA or MRA, to match the run-off and/or bolus chasing images and to maintain the superimposition and/or alignment of the first virtual dataset displayed by the one or more OHMDs with the second virtual data set, i.e. the run-off and/or bolus chasing images, e.g. displayed by the one or more OHMDs or a standalone or separate computer monitor or display.” [1043] “With guidance in mixed reality environment, e.g. with stereoscopic display like an electronic holographic environment, a virtual surgical guide, tool, instrument or implant can be superimposed onto the surgical site, e.g. an organ or a tumor. Further, the physical guide, tool, instrument or implant can be aligned with the 2D or 3D representation of the virtual surgical guide, tool, instrument or implant.” [0305] “A combination of data can be beneficial for more accurate measurement of changes in position or orientation of the surgeon or interventionalist's head, body, operating arm, hand, or the patient.” [0291] “A processor receives the electrical signals and calculates three-dimensional position information of tissue surfaces based on changes in the relative phase of the emitted optical radiation and the received optical radiation scattered by the surfaces.” [0252] “3D scanning can be used for imaging of the patient and/or the surgical site and/or anatomic landmarks and/or pathologic structures and/or tissues (e.g. damaged or diseased cartilage or exposed subchondral bone) and/or the surgeon or interventionalist's hands and/or fingers and/or the OR table and/or reference areas or points and/or marker, e.g. optical markers, in the operating room and/or on the patient and/or on the surgical field. … One or more optical imaging systems or 3D scanners can, for example, be used to image and/or monitor, e.g. the coordinates, position, orientation, alignment, direction of movement, speed of movement of, Anatomic landmarks, patient surface(s), organ surface(s), tissue surface(s), pathologic tissues and/or surface(s), e.g. for purposes of registration, e.g. of the patient and/or the surgical site, e.g. one or more bones or cartilage, and/or one or more OHMD's, e.g. in a common coordinate system The surgeon or interventionalist's hands and/or fingers, e.g. for Monitoring steps in an interventional procedure. Select hand and/or finger movements can be associated with corresponding surgical steps. When the 3D scanner system detects a particular hand and/or finger movement, it can trigger the display of the corresponding surgical step or the next surgical step, e.g. by displaying a predetermined virtual path, e.g. for a catheter, a virtual instrument, a virtual device etc. … One or more OHMDs, e.g. registered in a common coordinate system, e.g. with the surgical site and/or the surgeon or interventionalist's hands and/or fingers.” [0235-0240])
Lang does not explicitly teach assess impact of surgical tool position on a patient intervention site. This is what Verard teaches (“Using this video capability, the object 402 (e.g., a computer aided design rendering, model, scan, etc. for an instrument, medical device, implant, etc.) may be independently manipulated relative to the hologram 124 on the display 158 or in the air. In this way, the object 402 can be placed in or around the hologram 124 to determine whether the object will fit within a portion of the hologram 124, etc. For example, an implant may be placed through a blood vessel to test the fit visually.” [0056] “a fellow/practitioner could first select a program called "HIP" by activating a trigger point 504 to display a 3D CT image of a patient's (subject's) hip, and then select different "HIP IMPLANTS" from different manufacturers to see and "feel" which implant would fit best for the particular patient. It is also possible to use (e.g., physically hold and manipulate) the actual implant in the air and position it within the 3D holographic display to see, feel and assess fit (e.g., if and how well such implant may fit the particular patient).” [0059] “Robotics via a master/slave configuration can be used, where a shape sensed analog 604 of the device 602 moving within the display 158 is employed to actuate the motion of the actual device 602 within a target region 606. A practitioner's (surgeon's, physician's, etc.) hands 610 or voice can be tracked by sensor-based and/or voice-based techniques, such as by, e.g., tracking a physician's hands using a shape-sensing device 608 and shape sensing system 614 in the 3D holographic display. Accordingly, a practitioner's movements (including, e.g., (re)positioning, orientation, etc. of their hands) performed in the holographic display 158 can be transmitted to the device 602, such as a robot 612 (e.g., robotically controlled instruments) inside the patient to replicate such movements within the patient's body, and thereby perform the actual surgery, procedure or task inside the patient's body. Thus, a surgeon can see, touch and feel a 3D holographic display of an organ, perform a procedure thereon (i.e., within the 3D holographic display), causing such procedure to be performed inside of a patient on the actual organ via, or simply to move instruments, e.g., robotically controlled instruments.” [0062]) Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Verard into Lang, in order to improve accuracy and effectiveness in medical applications.
28. With reference to claim 21, Lang teaches applying the surgeon tool positional information to the patient intervention site dataset comprises generating a 3-dimensional depiction of the patient intervention site. (“Any of the 2D or 3D virtual data in any of the embodiments throughout the specification can include an arterial run-off, e.g. a peripheral run-off or an aortic run-off, optionally including bolus chasing. Bolus chasing can permit real-time visualization of the contrast bolus so that it can be followed, for example, peripherally with images, e.g. digital images, being acquired at a suitable frame rate. If the run-off and/or bolus chasing includes movement of the table on which the patient is positioned during the image acquisition, any 2D or 3D virtual data, e.g. from a pre-operative imaging study, e.g. a CTA or MRA or ultrasound, can be moved with the patient and/or table movement to maintain superimposition and/or alignment of a first 2D or 3D virtual data set, e.g. from the pre-operative imaging study, displayed by one or more OHMDs with a second or additional 2D or 3D virtual data sets, e.g. images obtained during the run-off and/or bolus chasing, displayed by the one or more OHMDs or a separate, standalone computer monitor using any of the combinations outlined in exemplary form in Tables 11, 13 and 14. The moving of the 2D or 3D virtual data, e.g. from a first set of virtual data from a pre-operative imaging test such as a CTA, MRA or ultrasound, displayed, using a computer processor, by the one or more OHMDs can be accomplished using the image registration including 3D-2D transformation matrices or 3D-3D registration techniques and/or the known table movement, e.g. in x, y, and/or z-direction, and/or the measured movement of one or more markers applied to the table or the patient. The image registration can include selection and/or matching and/or superimposition and/or alignment of a different volume of interest from a pre-operative vascular imaging study, e.g. an ultrasound, echocardiogram, CTA or MRA, to match the run-off and/or bolus chasing images and to maintain the superimposition and/or alignment of the first virtual dataset displayed by the one or more OHMDs with the second virtual data set, i.e. the run-off and/or bolus chasing images, e.g. displayed by the one or more OHMDs or a standalone or separate computer monitor or display.” [1043] “With guidance in mixed reality environment, e.g. with stereoscopic display like an electronic holographic environment, a virtual surgical guide, tool, instrument or implant can be superimposed onto the surgical site, e.g. an organ or a tumor. Further, the physical guide, tool, instrument or implant can be aligned with the 2D or 3D representation of the virtual surgical guide, tool, instrument or implant.” [0305] “A combination of data can be beneficial for more accurate measurement of changes in position or orientation of the surgeon or interventionalist's head, body, operating arm, hand, or the patient.” [0291] “The virtual 2D, 3D, and multi-dimensional data can be generated or acquired by different data acquisition technologies, e.g. different imaging tests etc.” [0283] “3D scanning can be used for imaging of the patient and/or the surgical site and/or anatomic landmarks and/or pathologic structures and/or tissues (e.g. damaged or diseased cartilage or exposed subchondral bone) and/or the surgeon or interventionalist's hands and/or fingers and/or the OR table and/or reference areas or points and/or marker, e.g. optical markers, in the operating room and/or on the patient and/or on the surgical field. … One or more optical imaging systems or 3D scanners can, for example, be used to image and/or monitor, e.g. the coordinates, position, orientation, alignment, direction of movement, speed of movement of, Anatomic landmarks, patient surface(s), organ surface(s), tissue surface(s), pathologic tissues and/or surface(s), e.g. for purposes of registration, e.g. of the patient and/or the surgical site, e.g. one or more bones or cartilage, and/or one or more OHMD's, e.g. in a common coordinate system The surgeon or interventionalist's hands and/or fingers, e.g. for Monitoring steps in an interventional procedure. Select hand and/or finger movements can be associated with corresponding surgical steps. When the 3D scanner system detects a particular hand and/or finger movement, it can trigger the display of the corresponding surgical step or the next surgical step, e.g. by displaying a predetermined virtual path, e.g. for a catheter, a virtual instrument, a virtual device etc. … One or more OHMDs, e.g. registered in a common coordinate system, e.g. with the surgical site and/or the surgeon or interventionalist's hands and/or fingers.” [0235-0240])
Lang does not explicitly teach assess impact of surgical hand position on a patient intervention site that shows the impact. This is what Verard teaches (“Using this video capability, the object 402 (e.g., a computer aided design rendering, model, scan, etc. for an instrument, medical device, implant, etc.) may be independently manipulated relative to the hologram 124 on the display 158 or in the air. In this way, the object 402 can be placed in or around the hologram 124 to determine whether the object will fit within a portion of the hologram 124, etc. For example, an implant may be placed through a blood vessel to test the fit visually.” [0056] “a fellow/practitioner could first select a program called "HIP" by activating a trigger point 504 to display a 3D CT image of a patient's (subject's) hip, and then select different "HIP IMPLANTS" from different manufacturers to see and "feel" which implant would fit best for the particular patient. It is also possible to use (e.g., physically hold and manipulate) the actual implant in the air and position it within the 3D holographic display to see, feel and assess fit (e.g., if and how well such implant may fit the particular patient).” [0059] “Robotics via a master/slave configuration can be used, where a shape sensed analog 604 of the device 602 moving within the display 158 is employed to actuate the motion of the actual device 602 within a target region 606. A practitioner's (surgeon's, physician's, etc.) hands 610 or voice can be tracked by sensor-based and/or voice-based techniques, such as by, e.g., tracking a physician's hands using a shape-sensing device 608 and shape sensing system 614 in the 3D holographic display. Accordingly, a practitioner's movements (including, e.g., (re)positioning, orientation, etc. of their hands) performed in the holographic display 158 can be transmitted to the device 602, such as a robot 612 (e.g., robotically controlled instruments) inside the patient to replicate such movements within the patient's body, and thereby perform the actual surgery, procedure or task inside the patient's body. Thus, a surgeon can see, touch and feel a 3D holographic display of an organ, perform a procedure thereon (i.e., within the 3D holographic display), causing such procedure to be performed inside of a patient on the actual organ via, or simply to move instruments, e.g., robotically controlled instruments.” [0062]) Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the teachings of Verard into Lang, in order to improve accuracy and effectiveness in medical applications.
Conclusion
29. THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Michelle Chin whose telephone number is (571)270-3697. The examiner can normally be reached on Monday-Friday 8:00 AM-4:30 PM.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kent Chang can be reach on (571)272-7667. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MICHELLE CHIN/
Primary Examiner, Art Unit 2614
Prosecution Insights