DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Drawings
The drawings are objected to because the texts in certain figures are unreadable. For instance, FIG.2, FIG.3, FIG.5, FIG.15 are too blurry to read. Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. The figure or figure number of an amended drawing should not be labeled as “amended.” If a drawing figure is to be canceled, the appropriate figure must be removed from the replacement sheet, and where necessary, the remaining figures must be renumbered and appropriate changes made to the brief description of the several views of the drawings for consistency. Additional replacement sheets may be necessary to show the renumbering of the remaining figures. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-12, 15-22 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lang (US 20210192759) in view of JONES et al. (US 20160225192).
Regarding claim 1, Lang discloses A system for assisting in a surgical procedure, the system comprising: one or more processors; a memory coupled to the one or more processors, the memory storing computer-program instructions, that, when executed by the one or more processors, perform a computer- implemented method comprising:
receiving a real-time video feed of a surgical field of view (“[0181] Non see through optical head mounted displays can be used, for example, with one or more image or video capture systems (e.g. cameras) or 3D scanners to image the live data of the patient, e.g. a skin, a subcutaneous tissue, a surgical site, an anatomic landmark, an organ, or an altered tissue, e.g. a surgically altered tissue, as well as any physical surgical tools, instruments, devices and/or implants, or portions of the surgeon's body, e.g. his or her fingers, hands or arms.”);
matching a field of view specified in a surgical procedure template to the surgical field of view to generate a 3D model of the surgical field of view (“[1160] The one or more of these anatomic landmarks, distances, dimensions, surfaces, edges, angles, axes, curvatures, shapes, lengths, widths, depths and/or other features, e.g. external and/or internal, can be used to identify a standard bone shape and/or a standard cartilage shape by comparing the one or more landmarks, distances, dimensions, surfaces, edges, angles, axes, curvatures, shapes, lengths, widths, depths and/or other external and/or internal features with data in a reference database of reference patients and/or reference bone and/or cartilage shapes and by selecting a 3D model that most closely matches the selected landmarks, distances, dimensions, surfaces, edges, angles, axes, curvatures, shapes, lengths, widths, depths and/or other features. [1182] Steps and processes in 168, 169, 170, and 171 can optionally be combined 172. Steps and processes 168, 169, 170, 171, and 172 can be used to generate a 2D to 3D morphed model 173, which can be used to generate pre-operative virtual surgical plan 174.”);
transferring overlays from the activated template to the 3D model, wherein the overlays comprise one or more of: offsets, measurements, or zones to avoid; displaying the 3D model with the overlays to the user (FIG.15, “[0831] including same or different tissues) and/or adjacent or subjacent or opposing or articulating or connected medical devices (including, for example, their dimensions, radius, radii, curvature, geometry, shape, placement position, orientation and/or alignment, e.g. relative to landmarks and/or anatomic or biomechanical axis/axes), and/or virtual data, e.g. pre- or intra-operative imaging studies, 2D, 3D images, graphical representations, CAD files, optionally registered, optionally superimposed, and/or other pre-or intra-operative data, optionally registered, optionally superimposed, e.g. pressure measurements, flow measurements, time of flight studies, metabolic data, functional data. [1182] Interconnecting or fill features can be determined 168, a closely fitting or matching model can be selected from a library or database of models 169, a standard model can be selected and optionally be deformed 170 using the shapes, geometries or features 166, a closely fitting or matching model can be selected from a library or database of models 171 and deformed using the information in 166. The morphed model 173 and the pre-operative virtual surgical plan 174 can be displayed by one or more OHMDs 175, optionally stereoscopic 176 or non-stereoscopic 177. An intra-operative scan or probe data 180 can be used to generate a model of the patient using intra-operative data, e.g. 2D, 2D to 3D morphed, 3D 181, which can optionally be superimposed, matched, merged or aligned 173 with the morphed model of the patient using pre-operative data 173 or the pre-operative virtual surgical plan 174”. Therefore, the virtual data is selected and transferred from a database and then superimposed and displayed on OHMD);
updating, in real time, the displayed 3D model with the overlays as the surgical field of view changes from the received real-time video feed (“[0171] Any type of live data of the patient captured by the OHMD including image and/or video capture systems attached to, integrated with or coupled to the OHMD. [1764] The optical markers attached to humerus and scapula can be continuously or intermittently tracked and, using, for example, the spatial relationship between the virtual model and the optical markers as described in the preceding sections, the position of the display of the virtual model, e.g. a 3D reconstruction of a CT or MRI scan or other virtual data, e.g. data display or highlighting sensitive anatomic structures, overlaid on the live, physical anatomy of the patient can be updated in real time”);
extracting procedure data from the real-time video feed in real time; identifying, by the one or more processors in real time, a match between a predefined procedural landmark from the surgical procedure template and the extracted procedure data (“[0378] Image and/or video capture and/or a 3D scanner, for example integrated into or attached to or coupled to the OHMD, can be used to identify the location of the light on the patient's tissue or the patient's anatomic landmark. Once the light has been directed to the desired location on the live data of the patient, specifically, the live landmark of the patient, registration can be performed by executing a registration command, registering the live data of the patient with the virtual data of the patient, e.g. the live landmark with the laser or other light being reflected of it and the corresponding virtual landmark of the patient. [0427] In some embodiments, an optical marker can be used to determine or identify the position, location, orientation, alignment, dimensions, axis or axes, plane or planes of a surgical alteration. [1256] The landmarks can be identified using, for example, an image and/or video capture system and/or 3D scanner integrated into, attached to or separate from an OHMD. The landmarks can be identified by attaching optionally one or more optical markers. [1285] Standard image processing techniques can, for example, recognize anatomic landmarks or features on the data or images displayed on the standalone or separate computer or display monitor and match these with the corresponding anatomic landmarks or features in the data and/or images available for display by the OHMD. The OHMD can then display the corresponding data and/or images, optionally superimposing the data based on landmark matching”); and
displaying visual confirmation on the display that user has matched the predefined procedural landmark (“[0272] Once the data have been superimposed, the surgeon can optionally register the virtual data with the live data. This registration can be as simple as described here, e.g. a visual confirmation from the surgeon that virtual and live data are substantially matching or substantially superimposed. [1287] Image processing techniques can, for example, recognize anatomic landmarks or features on the data or images acquired by the real-time imaging system and match these with the corresponding anatomic landmarks or features in the data and/or images available for display by the OHMD.”).
On the other hand, Lang fails to explicitly disclose but JONES discloses activating the surgical procedure template after a user confirms that the field of view of the surgical procedure template matches the surgical field of view (“[0095] the image generator 918 determines whether any portion of the patient's body is presently within the field of view of what the surgeon sees through the display screen 110 that corresponds to any portion of the transformed patient model. When a portion of the transformed patient model corresponds to a portion of the patient's body within the surgeon's field of view through the display screen 110, the image generator 918 generates video for display on the display screen 110 based on the corresponding portion of the transformed patient model, while translating the portion of the transformed patient model and scaling size of the portion of the transformed patient model to provide an accurately scaled graphical representation of the object that was imaged from the patient or modeled from another source such as an anatomical database”. Therefore, the image generator activates the data source in a database to generate video for display).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined Lang and JONES, to include all limitations of claim 1. That is, adding the image generator of JONES to process the data from the database of Lang. The motivation/ suggestion would have been to provide an accurately scaled graphical representation of the object that was imaged from the patient or modeled from another source such as an anatomical database (JONES, [0095]).
Regarding claim(s) 17, it is interpreted and rejected for the same reasons set forth in claim(s) 1. Particularly, claim 17 further recites A non-transitory computer-readable storage medium comprising instructions that, when executed by one or more processors of a device, cause the device to perform operations.
Lang further discloses “[1504] The system can comprise the OHMD display unit, one or more processors or computer chips, software, memory, e.g. RAM memory, sensors, e.g. depth sensors or acoustic sensors, and/or one or more cameras. Optionally, the data can also reside or be imported into a standalone computer, e.g. a PC, or server”.
Regarding claim 2, Lang in view of JONES discloses The system of claim 1.
Lang further discloses wherein the computer-implemented method further comprises: matching one or more pre-surgical patient scans to the 3D model (“[1182] An intra-operative scan or probe data 180 can be used to generate a model of the patient using intra-operative data, e.g. 2D, 2D to 3D morphed, 3D 181, which can optionally be superimposed, matched, merged or aligned 173 with the morphed model of the patient using pre-operative data 173 or the pre-operative virtual surgical plan 174.”).
Regarding claim 3, Lang in view of JONES discloses The system of claim 2.
Lang further discloses wherein the one or more pre-surgical patient scan comprises MRI scans (“[0507] In some embodiments, pre-operative imaging is performed to acquire 3D data of the patient. The pre-operative imaging can, for example, entail ultrasound, CT or MRI, any of the foregoing, optionally with administration of a contrast agent.”).
Regarding claim 4, Lang in view of JONES discloses The system of claim 1.
Lang further discloses wherein updating the displayed 3D model with the overlays as the surgical field of view changes from the received real-time video feed further comprises updating the overlays (“[0171] Any type of live data of the patient captured by the OHMD including image and/or video capture systems attached to, integrated with or coupled to the OHMD. [1764] The optical markers attached to humerus and scapula can be continuously or intermittently tracked and, using, for example, the spatial relationship between the virtual model and the optical markers as described in the preceding sections, the position of the display of the virtual model, e.g. a 3D reconstruction of a CT or MRI scan or other virtual data, e.g. data display or highlighting sensitive anatomic structures, overlaid on the live, physical anatomy of the patient can be updated in real time.”).
Regarding claim 5, Lang in view of JONES discloses The system of claim 1.
Lang further discloses wherein the procedure data comprises the surgical field of view or a modified version of the field of view (“[0686] The change in perspective view and visualized size and/or shape of the calibration or registration phantom can be measured and can be used to determine the change in position, location or orientation of the surgeon's or operator's head or body, which can then be used to maintain registration between the virtual patient data and the live patient data. [1256] One or more of the same landmarks that have been/are being used for registration of virtual data and live data of the patient can be used for defining or identifying a target area or a smaller portion of the field of view visible through the OHMD or displayable by the display of the OHMD”).
Regarding claim 6, Lang in view of JONES discloses The system of claim 1.
Lang further discloses wherein the procedure data comprises one or more of: visual data of implant position, surgical tool position, or anatomy orientation (“[0151] Any of a position, location, orientation, alignment, direction, speed of movement, force applied of a surgical instrument or tool, virtual and/or physical, can be predetermined using, for example, pre-operative imaging studies, pre-operative data, pre-operative measurements, intra-operative imaging studies, intra-operative data, and/or intra-operative measurements”).
Regarding claim(s) 18-22, they are interpreted and rejected for the same reasons set forth in claim(s) 2-6, respectively.
Regarding claim 7, Lang in view of JONES discloses The system of claim 1.
Lang further discloses identifying, by the one or more processors in real time, a mismatch between the predefined procedural landmark from the surgical procedure template and the extracted procedure data (“[1325] If a size and/or shape mismatch is detected, the software can send an alert or alarm to the surgeon, e.g. visual or audible, that indicates a mismatch. A mismatch can indicate to the surgeon that the accuracy of registration of virtual data and live data has been compromised and that re-registration may be required. A mismatch can also indicate to the surgeon that the wrong physical surgical instrument or implantable has been selected in comparison to the previously identified virtual surgical instrument or implantable. [1182] at least one or more of the same landmarks, distances, dimensions, surfaces, edges, angles, axes, curvatures, shapes, lengths, widths, depths and/or other features used for 2D to 3D tissue morphing, e.g. bone morphing, can be used for intra-operative registration of live data and virtual data”).
Regarding claim 8, Lang in view of JONES discloses The system of claim 7.
Lang further discloses displaying visual confirmation on the display that user has not matched the predefined procedural landmark (“[1325] If a size and/or shape mismatch is detected, the software can send an alert or alarm to the surgeon, e.g. visual or audible, that indicates a mismatch”).
Regarding claim 16, it is covered by claims 5 and 8, thus is rejected under similar rationale as claims 5 and 8.
Regarding claim 9, Lang in view of JONES discloses The system of claim 1.
Lang further discloses scaling physical dimensions in the 3D model using a pre-surgical scan for the patient (“[0667] so that the magnification of the patient or anatomic data resulting for a given intraoperative film-focus and optionally object distance will be matched or reflected in the re-projected pre-operative data. [0764] One or more of the intra-operative x-ray projections can be matched with pre-operative imaging data of the patient or virtual data of the patient including, optionally, a virtual surgical plan, using, for example, pattern recognition algorithms, image processing algorithms, or manual/visual matching by the surgeon or operator, optionally with magnification adjustment for a given film/detector focus distance, with magnification or de-magnification of either the intraoperative x-ray data, the pre-operative data, the virtual data of the patient, including, optionally, the virtual surgical plan, with the aim that all data used have similar or the same magnification. [1916] the OHMD can display a virtual 3D model of the patient's distal femur and/or the proximal tibia, for example generated from 2D x-ray images with 3D model generation, e.g. bone morphing, and/or derived from a CT or MRI scan, registered with and superimposed on the physical distal femur and/or the physical proximal tibia. [1941] Co-Display of Pre-Operative or Intra-Operative X-ray Images Including Implant Sizing and/or Templating Information with Live Data of the Patient”).
Regarding claim 10, Lang in view of JONES discloses The system of claim 1.
Lang further discloses adjusting the template based on one or more structures from pre-surgical scan for the patient (“[0664] An adjustment for the difference in phantom location from the pre-operative to the intraoperative data can be performed; this adjustment can optionally be defined as a phantom offset between pre-operative and intra-operative data. [0912] a virtual pedicle screw can be fitted, sized, and/or aligned by virtually placing it inside the pedicle of the patient; the pedicle anatomy can be projected by the OHMD simultaneously, e.g. using a pre-operative or intra-operative CT scan or O-arm scan that is registered in a common coordinate system with the patient's spine and one/or more OHMDs.”).
Regarding claim 11, Lang in view of JONES discloses The system of claim 10.
Lang further discloses wherein adjusting the template comprises adjusting based on a physical dimension from a corresponding reference anatomical structure from a matched pre-surgical scan for the patient (“[0912] virtual spinal cage can be fitted, sized, and/or aligned by virtually placing it between two vertebral endplates of the patient; the vertebral anatomy can be projected by the OHMD simultaneously, e.g. using a pre-operative or intra-operative CT scan or O-arm scan that is registered in a common coordinate system with the patient's spine and one/or more OHMDs. [0935] Since the spatial coordinates of the one or more optical markers are known in relationship to the one or more radiopaque elements and in relationship to the teeth or underlying bone based on the scan or imaging data, by including the optical markers and the radiopaque elements in the pre- or intra-operative scan, pre- and intra-operative data including live data, e.g. the intended implantation site, can be referenced and registered in the same coordinate system”).
Regarding claim 12, Lang in view of JONES discloses The system of claim 10.
Lang further discloses wherein adjusting the template comprises one or more of: scaling, referencing, labeling, or measuring (“[0912] a virtual pedicle screw can be fitted, sized, and/or aligned by virtually placing it inside the pedicle of the patient; the pedicle anatomy can be projected by the OHMD simultaneously, e.g. using a pre-operative or intra-operative CT scan or O-arm scan that is registered in a common coordinate system with the patient's spine and one/or more OHMDs.”).
Regarding claim 15, Lang in view of JONES discloses The system of claim 1.
Lang further discloses wherein the computer- implemented method further comprises: importing the surgical procedure template (“[1109] The surgeon can then incorporate or import data from one or more pre-operative and/or intra-operative knee x-rays, for example femoral, tibial or patellar component size and/or desired varus or valgus correction and/or desired femoral and/or tibial component rotation and/or desired femoral component flexion and/or desired tibial slope, and/or desired femoral, tibial and/or patellar component position and/or orientation and/or alignment into the virtual surgical plan”).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to GRACE Q LI whose telephone number is (571)270-0497. The examiner can normally be reached Monday - Friday, 8:00 am-5:00 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, DEVONA FAULK can be reached at 571-272-7515. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/GRACE Q LI/Primary Examiner, Art Unit 2618 9/30/2026
