DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claims 1-20 are pending.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over JONES et al. (hereinafter “JONES”) (US 20160225192 A1) in view of Kang et al. (hereinafter “Kang”) (US 20160175054 A1).
As to claims 1 and 13, JONES teaches a surgical system and method for facilitating intraoperative surgical planning relative to a target site, the surgical system comprising:
a head-mounted device configured to be worn by a user and comprising a display that is positionable directly in front of the eyes of the user ([Figs. 1-4] [0023] an augmented reality surgical system that includes a head mounted display (HMD) apparatus that can be worn by a surgeon, physician, or other personnel during a medical procedure…The HMD includes a display screen that can be positioned within the line-of-sight and/or periphery Field Of View (FOV) of the wearer to provide visual information that can be organized and displayed as a single virtual display or as a collection of virtual displays that a wearer can navigate between to view using head movement, hand gestures, voice commands, eye control, and/or other operations disclosed herein);
a digitization device comprising a pointer tip configured to contact the target site ([Fig. 14 shows a tool with pointer tip in contacts with the target surgical site] [0091, 0099, 0122-0123] while a surgeon is seeking to position and orient the drill bit 1310 to intersect or to avoid intersecting various anatomical structure illustrated by one or more of the slices 1330, 1332, 1334 and/or other slices selected by the surgeon, and while accomplishing the objective of guiding the drill bit 1310 to contact the target location 1316 on the bone. Through a head movement, hand gesture, or other command the surgeon may move a pointer along the virtual trajectory 1312 to select a location where a perpendicular slice is to be generated and displayed on the HMD 100….FIG. 22 illustrates the slice 1334 along plane 22-22 in FIG. 21 rotated to provide a front view, and illustrates the tool impact location 1314 where the drill bit 1310 would impact the graphically displayed bone model 1302 based on a present location and orientation of the drill bit 1310 relative to the patient site 1300 and further illustrates the target location 1316. FIG. 23 illustrates the slice 1330 along plane 23-23 in FIG. 21 rotated to provide a front view, and illustrates the tool impact location 1314 where the drill bit 1310 would impact the graphically displayed bone model 1302 based on a present location and orientation of the drill bit 1310 relative to the patient site 1300 and further illustrates the target location 1316);
a navigation system configured to track states of the digitization device and to track states of the target site ([0091, 0093] the position tracking system 810 can include a camera system 902 that tracks the location tracking markers 904 attached to a surgery table, tracking markers 906 attached to patient adjacent to a surgical or other target site, tracking markers 908 attached to a surgery tool and/or prosthetic, and tracking markers 910 attached to the HMD 100… A relative positioning component 916 identifies the relative position and angular orientation of each of the tracked markers 904-910 and the combined HMD position data. The component 916 may perform coordinate transformations of the relative coordinate systems of the surgery table, the patient, the surgery tool, and the HMD 100 to a unified (common) coordinate system); and
one or more controllers coupled to the navigation system and the head-mounted device and being configured to:
receive the tracked states of the digitization device and the target site from the navigation system ([0082-0084, 0091-0093, 0116] a position tracking system 810 (e.g., cameras spaced apart in the operating room) that track the location of a surgical tool 800 and/or prosthetic 802, the HMD 100, and a surgical site 804 or other target location on a patient… a tracking system 1208 tracks the relative location and orientation of reference markers attached to the drill bit 1310, the patient site 1300, and the HMD 100, as illustrated in FIG. 14, and computes the relative distance and orientation therebetween. The tracking system 1208 may operate based on the position tracking system 810 described above in FIG. 14);
register landmarks to the target site ([0058, 0083, 0086, 0116-0119] a surgeon can virtually mark a location within a surgical site using a graphical indicia and can subsequently track that marked location based on the location of the graphical indicia within the display screen 608 while the surgeon's head moves… The model can include reference markers or other references that assist with performing correlations between virtual locations in the patient model and physical locations on the patient's body...and uses the reference markers or other references contained in the patient model to transform the patient model to a present perspective view of a wearer of the HMD 100… to provide the surgeon with a graphical overlay that is precisely oriented and scaled on the surgical site 804 or other target location on the patient);
create a virtual object that is registered to the target site, wherein the virtual object comprises a location and geometry that is defined, at least in part, based on the landmarks ([0094-0098] The image generator 918 provides video generated based on the transformed patient model to the HMD 100 for display as a visual model that is dynamically oriented and scaled as a graphical overlay on the surgical site 804… When a portion of the transformed patient model corresponds to a portion of the patient's body within the surgeon's field of view through the display screen 110, the image generator 918 generates video for display on the display screen 110 based on the corresponding portion of the transformed patient model, while translating the portion of the transformed patient model and scaling size of the portion of the transformed patient model to provide an accurately scaled graphical representation of the object that was imaged from the patient or modeled from another source such as an anatomical database… the image generator 918 transforms and scales the patient model of the bone to generate a graphical representation of the bone that is displayed in the display screen 110 as a graphical overlay that matches the orientation and size of the bone from the perspective of the surgeon as-if the surgeon could view the bone through intervening layers of tissue and/or organs... a leg bone model that has been generated, e.g., based on a CT scan of the leg, is transformed and displayed on the display screen 110 to have the accurate six degree of freedom orientation and size relative to the leg bone when viewed as a graphically illustrated representation 922 of the leg bone superimposed on a skin surface of the leg… Moreover, the surgeon may control the size of the displayed graphical representation 922 relative to the leg. The surgeon may, for example, temporarily magnify the displayed graphical representation 922 to view certain details and then return the displayed graphical representation 922 to be scaled and aligned with the leg); and
present, on the display of the head-mounted device, the virtual object being overlaid onto a real-world image of the target site or real-world view of the target site ([0094-0098, 0115] a leg bone model that has been generated, e.g., based on a CT scan of the leg, is transformed and displayed on the display screen 110 to have the accurate six degree of freedom orientation and size relative to the leg bone when viewed as a graphically illustrated representation 922 of the leg bone superimposed on a skin surface of the leg… Moreover, the surgeon may control the size of the displayed graphical representation 922 relative to the leg. The surgeon may, for example, temporarily magnify the displayed graphical representation 922 to view certain details and then return the displayed graphical representation 922 to be scaled and aligned with the leg…FIG. 18 illustrates a graphical image generated on the HMD 100 that shows a virtual trajectory 1312 of a drill bit 1310 extending from a drill or other surgical tool (e.g., scalpel, etc.) into a patient's anatomical bone model 1302 and other selected intervening structure, and which is overlaid at a patient site 1300).
JONES teaches a surgical system and method for facilitating intraoperative surgical planning relative to a target site, more particular, to generating anatomical models and other information to be displayed as augmenting reality in surgical operating rooms [Abstract, 0002]. JONES also teaches using different methods to interact with the surgical system to provide command to perform control operations, such as using head movement, hand gestures, voice commands, etc., which are sensed by the HMD [0118-0120]. JONES does not explicitly teach using a pointer tip contacting the target site to perform control operations such as register landmarks to the target site.
However, Kang teaches a system and method for providing a virtual environment to assist surgeon for intraoperative surgical planning. Especially, Kang teaches using a digitization device comprising a pointer tip to register landmarks to the target site by using a pointer tip contacting the target site ([0045-0046, 0067, 0075] a virtual representation of a portion of the patient's anatomy is created based on patient-specific characteristics (such as anatomical landmarks obtained by physically touching the patient's anatomy using a probe tool)… CAS system 200 may be configured to create a virtual representation of a surgical site that includes, for example, virtual representations of a patient's anatomy, a surgical instrument to be used during a surgical procedure, a probe tool for registering other objects within the surgical site, and any other such object associated with a surgical site… information indicative of anatomical landmarks may be received based on a user input. For example, a surgeon may designate one or more points, lines, or areas of the patient's anatomy as anatomical landmarks manually by physically touching the points, lines, or areas of the patient's anatomy using a probe tool that has been registered with the virtual coordinate space… After the virtual model of the patient's anatomy is generated, it may be registered with the actual anatomy of the patient so that CAS system 200 can virtually track the position and orientation of the actual anatomy of the patient in virtual software space. According to one embodiment, this registration process involves associating a plurality of points of the patient's anatomy with corresponding points on the virtual model. As explained above, such associations can be made using a probe tool that has been registered in the virtual coordinate space, whereby a plurality of points on the patient's anatomy gathered by touching or “exploring” one or more surfaces of the patient's anatomy using the tip of the probe tool. Once the virtual model is registered with the patient's anatomy, CAS system 200 may be able to track the position and orientation of the patient's anatomy in the virtual coordinate space).
JONES and Kang are analogous art because they are from the same field of endeavor of generating a virtual intraoperative surgical planning environment relative to a target site. At the time before the effective filing date of the invention it would have been obvious to a person of ordinary skill in the art to use different input and tracking method to recognize user’s control operations. Therefore, it would have been obvious to an ordinary person skilled in the art before the effective filing date of the invention to incorporate the teachings of Kang with the teachings of JONES for the purpose of using a pointer tip contacting the target site to perform control operations such as register landmarks to the target site as specified in the claims 1 and 13.
As to claim 2, Kang teaches the digitization device comprises one or more control inputs that are configured to be actuated [0034, 0038-0039, 0042, 0045-0049, 0058, 0067].
As to claims 3 and 14, Kang teaches the one or more controllers register the landmarks to the target site in response to actuation of the one or more control inputs of the digitization device [0034, 0038-0039, 0042, 0045-0049, 0058, 0067].
As to claims 4 and 15, Kang teaches the one or more controllers create the virtual object in response to actuation of the one or more control inputs of the digitization device [0034, 0038-0039, 0042, 0045-0049, 0058, 0067].
As to claim 5, Kang teaches to create the virtual object, the one or more controllers are further configured to enable arrangement of one or more geometrical design objects relative to one or more of the landmarks in response to actuation of the one or more control inputs of the digitization device [0034, 0038-0039, 0042, 0045-0049, 0057-0059, 0067].
As to claim 6, Kang teaches in response to actuation of the one or more control inputs of the digitization device, the one or more controllers are further configured to enable selection of a type of the one or more geometrical design objects from a list [0045-0049, 0057-0060, 0067, 0077].
As to claim 7, JONES teaches the one or more geometrical design objects are presented on the display of the head-mounted device during arrangement of the one or more geometrical design objects relative to the one or more of the landmarks [0093-0099, 0110-0114, 0116-0119].
As to claim 8, JONES teaches the one or more controllers are further configured to temporarily fix a graphical representation of the one or more geometrical design objects to the pointer tip such that the graphical representation of the one or more geometrical design objects that are temporarily fixed correspondingly follow changes in pose of the pointer tip [0064-0065].
As to claim 9, JONES teaches the virtual object is a virtual target trajectory, a virtual cutting plane, a virtual resection volume, or a virtual implant model [0115-0118, 0120-0123].
As to claim 10, Kang teaches the virtual object is a virtual boundary that defines a region of the target site to be avoided and is configured to be utilized to constrain movement and/or operation of a surgical tool in response to a tracked state of the surgical tool meeting, exceeding, or potentially exceeding the virtual boundary [0007-0008, 0035, 0048].
As to claim 11, JONES teaches the head-mounted device comprises tracking sensors to detect the eyes of the user and the head-mounted device is configured to present the virtual object based on measurements from the tracking sensors [0033, 0051].
As to claim 12, JONES teaches the head-mounted device comprises sensors to detect hand gestures and the head-mounted device enables a pose of the virtual object to be modified based detected hand gestures [0033, 0038, 0051-0053, 0056, 0062, 0072].
As to claim 16, Kang teaches creating the virtual object comprises the one or more controllers: enabling selection of one or more geometrical design objects, in response to detecting actuation of the one or more control inputs of the digitization device [0045-0049, 0057-0060, 0067, 0077]; enabling arrangement of the selected one or more geometrical design objects relative to one or more of the landmarks in response to detecting actuation of the one or more control inputs of the digitization device [0034, 0038-0039, 0042, 0045-0049, 0057-0059, 0067]. and JONES teaches temporarily fixing a graphical representation of the selected one or more geometrical design objects to the pointer tip such that the graphical representation is temporarily fixed correspondingly follows changes in pose of the pointer tip [0064-0065]; presenting the selected one or more geometrical design objects on the display of the head-mounted device during arrangement of the selected one or more geometrical design objects relative to the one or more landmarks, wherein the virtual object is derived from arrangement of the selected one or more geometrical design objects [0093-0099, 0110-0114, 0116-0119].
As to claim 17, JONES teaches a head-mounted device for facilitating intraoperative surgical planning relative to a target site, the head-mounted device comprising:
a support configured to mounted to a head of a user ([Figs. 1-5 showing a headband 120] [0027];
a display provided on the support and being positionable directly in front of the eyes of the user ([Figs. 1-4] [0023] an augmented reality surgical system that includes a head mounted display (HMD) apparatus that can be worn by a surgeon, physician, or other personnel during a medical procedure…The HMD includes a display screen that can be positioned within the line-of-sight and/or periphery Field Of View (FOV) of the wearer to provide visual information that can be organized and displayed as a single virtual display or as a collection of virtual displays that a wearer can navigate between to view using head movement, hand gestures, voice commands, eye control, and/or other operations disclosed herein); and
one or more controllers being configured to:
provide, with the display, a real-world image or real-world view of the target site ([0028, 0044, 0083, 0097-0098, 0105] It may be preferable for the display screen 608 to be a see-through display device allowing the user to see displayed video superimposed on what is viewed through the display screen 608…the display screen (DIS) 110 can be a see-through display device that is fed video which displays in real-time portions of the transformed patient model and desired symbology, which can be accurately superimposed on the target surgical site or other target area of the patient within the FOV of the surgeon) and
a digitization device comprising a pointer tip ([Fig. 14 shows a tool with pointer tip in contacts with the target surgical site] [0091, 0099, 0122-0123] while a surgeon is seeking to position and orient the drill bit 1310 to intersect or to avoid intersecting various anatomical structure illustrated by one or more of the slices 1330, 1332, 1334 and/or other slices selected by the surgeon, and while accomplishing the objective of guiding the drill bit 1310 to contact the target location 1316 on the bone. Through a head movement, hand gesture, or other command the surgeon may move a pointer along the virtual trajectory 1312 to select a location where a perpendicular slice is to be generated and displayed on the HMD 100….FIG. 22 illustrates the slice 1334 along plane 22-22 in FIG. 21 rotated to provide a front view, and illustrates the tool impact location 1314 where the drill bit 1310 would impact the graphically displayed bone model 1302 based on a present location and orientation of the drill bit 1310 relative to the patient site 1300 and further illustrates the target location 1316. FIG. 23 illustrates the slice 1330 along plane 23-23 in FIG. 21 rotated to provide a front view, and illustrates the tool impact location 1314 where the drill bit 1310 would impact the graphically displayed bone model 1302 based on a present location and orientation of the drill bit 1310 relative to the patient site 1300 and further illustrates the target location 1316);
enable selection of one or more geometrical design objects ([0096-0098, 0118-0120] For example, the surgeon can select among various visualization modes that control which one or more of the following are displayed on the HMD 100: 1) bone; 2) skin; 3) muscle; 4) organ; 5) vessel; 6) virtual tool trajectory; and 7) cross sectional slice. Another visualization mode can control which cross sectional slices are displayed and/or the orientation of the slice(s) relative to the surgeon's point-of-view. Some visualization modes cause the system subcomponent 1200 to graphically render anatomical structure of the patient for display as wireframes, polygons, or smoothed surfaces, and which can be selectively displayed in monochrome or false colors);
present, on the display, a graphical representation of the one or more geometrical design objects being temporarily fixed to user input such as commands, gestures, and eye/head motion such that the graphical representation correspondingly follows changes in pose of the user input such as gestures and eye/head motion ([0051-0053, 0062, 0064-0065, 0072] the computer equipment 620 may enable the user to input a command (e.g., “Lock”) which causes whichever virtual display panel is presently most closely spaced to the user's line-of-sight to be displayed full screen and held statically in-place not responding to head movement. The virtual display panel may remain statically locked as-displayed until the user deactivates the command… provide an automatic-lock and/or unlock of a virtual display panel relative to the display screen 608. When an automatic mode is enabled, a virtual display panel becomes automatically locked relative to the display screen 608 when the user's line-of-sight (e.g., yaw and pitch) indicated by the head motion signal is within a first threshold amount (e.g., 2°) offset from a defined location (e.g., yaw and pitch) of the virtual display panel. The computer equipment 620 may be configured to automatically unlock the virtual display panel from the display screen 608 when the user's line-of-sight (e.g., yaw and pitch) indicated by the head motion signal becomes at least a second threshold amount…);
enable arrangement of the one or more geometrical design objects relative to the target site in response to interaction of user input with the target site; based on the arrangement of the one or more geometrical design objects, create a virtual object that is registered to the target site ([0094-0098] The image generator 918 provides video generated based on the transformed patient model to the HMD 100 for display as a visual model that is dynamically oriented and scaled as a graphical overlay on the surgical site 804… When a portion of the transformed patient model corresponds to a portion of the patient's body within the surgeon's field of view through the display screen 110, the image generator 918 generates video for display on the display screen 110 based on the corresponding portion of the transformed patient model, while translating the portion of the transformed patient model and scaling size of the portion of the transformed patient model to provide an accurately scaled graphical representation of the object that was imaged from the patient or modeled from another source such as an anatomical database… the image generator 918 transforms and scales the patient model of the bone to generate a graphical representation of the bone that is displayed in the display screen 110 as a graphical overlay that matches the orientation and size of the bone from the perspective of the surgeon as-if the surgeon could view the bone through intervening layers of tissue and/or organs... a leg bone model that has been generated, e.g., based on a CT scan of the leg, is transformed and displayed on the display screen 110 to have the accurate six degree of freedom orientation and size relative to the leg bone when viewed as a graphically illustrated representation 922 of the leg bone superimposed on a skin surface of the leg… Moreover, the surgeon may control the size of the displayed graphical representation 922 relative to the leg. The surgeon may, for example, temporarily magnify the displayed graphical representation 922 to view certain details and then return the displayed graphical representation 922 to be scaled and aligned with the leg); and
present, on the display, the virtual object being overlaid onto the real-world image or real-world view of the target site ([0094-0098, 0115] a leg bone model that has been generated, e.g., based on a CT scan of the leg, is transformed and displayed on the display screen 110 to have the accurate six degree of freedom orientation and size relative to the leg bone when viewed as a graphically illustrated representation 922 of the leg bone superimposed on a skin surface of the leg… Moreover, the surgeon may control the size of the displayed graphical representation 922 relative to the leg. The surgeon may, for example, temporarily magnify the displayed graphical representation 922 to view certain details and then return the displayed graphical representation 922 to be scaled and aligned with the leg…FIG. 18 illustrates a graphical image generated on the HMD 100 that shows a virtual trajectory 1312 of a drill bit 1310 extending from a drill or other surgical tool (e.g., scalpel, etc.) into a patient's anatomical bone model 1302 and other selected intervening structure, and which is overlaid at a patient site 1300).
JONES teaches a surgical system and method for facilitating intraoperative surgical planning relative to a target site, more particular, to generating anatomical models and other information to be displayed as augmenting reality in surgical operating rooms [Abstract, 0002]. JONES also teaches using different methods to interact with the surgical system to provide command to perform control operations, such as using head movement, hand gestures, voice commands, etc., which are sensed by the HMD [0118-0120]. JONES does not explicitly teach using a pointer tip contacting the target site to perform control operations.
However, Kang teaches a system and method for providing a virtual environment to assist surgeon for intraoperative surgical planning. Especially, Kang teaches using a digitization device comprising a pointer tip to register landmarks to the target site by using a pointer tip contacting the target site ([0045-0046, 0067, 0075] a virtual representation of a portion of the patient's anatomy is created based on patient-specific characteristics (such as anatomical landmarks obtained by physically touching the patient's anatomy using a probe tool)… CAS system 200 may be configured to create a virtual representation of a surgical site that includes, for example, virtual representations of a patient's anatomy, a surgical instrument to be used during a surgical procedure, a probe tool for registering other objects within the surgical site, and any other such object associated with a surgical site… information indicative of anatomical landmarks may be received based on a user input. For example, a surgeon may designate one or more points, lines, or areas of the patient's anatomy as anatomical landmarks manually by physically touching the points, lines, or areas of the patient's anatomy using a probe tool that has been registered with the virtual coordinate space… After the virtual model of the patient's anatomy is generated, it may be registered with the actual anatomy of the patient so that CAS system 200 can virtually track the position and orientation of the actual anatomy of the patient in virtual software space. According to one embodiment, this registration process involves associating a plurality of points of the patient's anatomy with corresponding points on the virtual model. As explained above, such associations can be made using a probe tool that has been registered in the virtual coordinate space, whereby a plurality of points on the patient's anatomy gathered by touching or “exploring” one or more surfaces of the patient's anatomy using the tip of the probe tool. Once the virtual model is registered with the patient's anatomy, CAS system 200 may be able to track the position and orientation of the patient's anatomy in the virtual coordinate space).
JONES and Kang are analogous art because they are from the same field of endeavor of generating a virtual intraoperative surgical planning environment relative to a target site. At the time before the effective filing date of the invention it would have been obvious to a person of ordinary skill in the art to use different input and tracking method to recognize user’s control operations. Therefore, it would have been obvious to an ordinary person skilled in the art before the effective filing date of the invention to incorporate the teachings of Kang with the teachings of JONES for the purpose of using a pointer tip contacting the target site to perform control operations such as register landmarks to the target site as specified in the claim 17.
As to claim 18, JONES teaches the virtual object is a virtual target trajectory, a virtual cutting plane, a virtual resection volume, or a virtual implant model [0115-0118, 0120-0123].
As to claim 19, Kang teaches the virtual object is a virtual boundary that defines a region of the target site to be avoided and is configured to be utilized to constrain movement and/or operation of a surgical tool in response to a tracked state of the surgical tool meeting, exceeding, or potentially exceeding the virtual boundary [0007-0008, 0035, 0048].
As to claim 20, JONES teaches the one or more controllers being configured to: present, on the display, the virtual object being overlaid onto the real-world image or real-world view of the target site using augmented reality or mixed reality [0094-0098, 0115]; present, on the display, the virtual object based on measurements from tracking sensors that are configured to detect the eyes of the user [0033, 0051]; and enable a pose of the virtual object to be modified based on measurements from sensors being configured to detect hand gestures of the user [0033, 0038, 0051-0053, 0056, 0062, 0072].
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ZHIPENG WANG whose telephone number is (571)272-5437. The examiner can normally be reached Monday-Friday 10-7.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kamini Shah can be reached at 5712722279. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ZHIPENG WANG/Primary Examiner, Art Unit 2115
Prosecution Insights