DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 4/3/2026 has been entered.
Response to Amendment
The amendment filed on 4/3/2026 has been entered. Claims 1-20 remain pending the application. Of the above claims, claims 13-20 are withdrawn.
Response to Arguments
Applicant's arguments filed on 9/15/2025 have been fully considered but they are moot or not persuasive.
Applicant argues on pages 8-11 that Bradski fails to disclose the newly added limitations to the claims related to dual-modality markers. This argument is moot in view of the new grounds rejection necessitated by amendment which relies on Lang to disclose these limitations in the claims. Accordingly, this argument is moot.
Applicant argues on pages 11-12 that Bradski fails to disclose the newly added limitations to the claims related to detecting and extracting from both x-ray and spatial data. This argument is moot in view of the new grounds rejection necessitated by amendment which relies on Lang to disclose these limitations in the claims. Accordingly, this argument is moot.
Applicant argues on pages 12-13 that Bradski fails to disclose the newly added limitations to the claims related to computing a coordinate transformation matrix. This argument is moot in view of the new grounds rejection necessitated by amendment which relies on Lee to disclose these limitations in the claims. Accordingly, this argument is moot.
Applicant argues on pages 13-14 that Bradski fails to disclose the newly added limitations to the claims related to computing a coordinate transformation matrix. The Examiner respectfully disagrees. Newly cited portions of Bradski disclose these limitations as can be seen in the new grounds rejection necessitated by amendment below. Accordingly, this argument is not persuasive.
Applicant argues on pages 14-16 that there is no motivation to combine Lang with Bradski. In response to applicant’s argument that there is no teaching, suggestion, or motivation to combine the references, the examiner recognizes that obviousness may be established by combining or modifying the teachings of the prior art to produce the claimed invention where there is some teaching, suggestion, or motivation to do so found either in the references themselves or in the knowledge generally available to one of ordinary skill in the art. See In re Fine, 837 F.2d 1071, 5 USPQ2d 1596 (Fed. Cir. 1988), In re Jones, 958 F.2d 347, 21 USPQ2d 1941 (Fed. Cir. 1992), and KSR International Co. v. Teleflex, Inc., 550 U.S. 398, 82 USPQ2d 1385 (2007). As can be seen in the previous rejection and the rejection below, motivations have been provided. Accordingly, this argument is not persuasive.
Applicant argues on pages 14-16 that there was no reasonable expectation of success for combining Lang and Bradski. However, the Applicant is reminded that attorney’s arguments are not evidence. See MPEP 2145.I. Additionally, these arguments fail to comply with 37 CFR 1.111(b) because they amount to a general allegation that the claims define a patentable invention without specifically pointing out how the language of the claims patentably distinguishes them from the references. Accordingly, this argument is not persuasive.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 1-12 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention.
Regarding claim 1, the claim recites computing a transformation matrix. The specification does not provide sufficient support for this limitation in the claims. While the specification does disclose performing spatial transformations such as in [0069-0074] [0082][0104] the specification does not specifically disclose using matrix transformations, which is a more specific and narrower method of performing spatial transformations. Accordingly, this claim is rejected under 112a.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-11 are rejected under 35 U.S.C. 103 as being unpatentable over Bradski et al. (WO2015192117, hereafter Bradski), Lang (US20170258526), and Lee (US20170273595).
Regarding claim 1, Bradski discloses a system for image to world registration using a world spatial map (Bradski, Para 1114; “Referring to Fig. 64, a wearable system may capture image information and extract fiducials and recognized points 6452. The wearable local system may calculate pose using one of the pose calculation techniques mentioned below. The cloud 6454 may use images and fiducials to segment 3-D objects from more static 3-D background. Images may provide textures maps for objects and the world (textures may be real-time videos). The cloud resources may store and make available static fiducials and textures for world registration”) comprising:
a spatial mapping apparatus (user device 12) having a visual field for display of information (Bradski, Para 365-366; “Users interact with one or more digital worlds using some type of a local computing device, which is generally designated as a user device 12. Examples of such user devices include, but are not limited to, a smart phone, tablet device, heads-mounted display (HMD), gaming console, or any other device capable of communicating data and providing an interface or display to the user, as well as combinations of such devices […] An example of a user device 12 for interacting with the system 10 is illustrated in Fig. 2. In the example embodiment shown in Fig. 2, a user 21 may interface one or more digital worlds through a smart phone 22. The gateway is implemented by a software application 23 stored on and running on the smart phone 22. In this particular example, the data network 13 includes a wireless mobile network connecting the user device (e.g., smart phone 22) to the computer network 5”), wherein the spatial mapping apparatus continuously (Bradski, Para 765; “At 3010, the AR system may synchronize the parametric geometry of the objects to the cloud. Next, at 3012, the object recognizer may re-insert the geometric and parametric information into the passable world model. For example, the object recognizer may dynamically estimate the angle of the door, and insert it into the world. Thus, it can be appreciated that using the object recognizer allows the system to save computational power because, rather than constantly requiring real-time capture of information about the angle of the door or movement of the door, the object recognizer uses the stored parametric information to estimate the movement or angle of the door. This allows the system to function independently based on computational capabilities of the individual AR system without necessarily relying on information in the cloud servers. It should be appreciated that this information may be updated to the cloud, and transmitted to other AR systems such that virtual content may be appropriately displayed in relation to the recognized door.”) generates a world frame of reference (Bradski, Para 11-13; “In another example, an object may be world-centric, in which case it may be presented as though it is part of the real world coordinate system, such that the user may move his head or eyes without moving the position of the object relative to the real world. Thus when placing virtual content into the augmented reality world presented with an AR system, choices are made as to whether the object should be presented as world centric, body-centric, head-centric or eye centric.”) (Bradski, Para 85; “a method for generating a virtual user interface, comprises detecting a manipulation of a predefined physical object, recognizing, based on the detected manipulation, a command to create a virtual user interface, determining, from a virtual world model, a set of map points associated with a position of the predefined physical object, and rendering, in real-time, the virtual user interface at the determined map points associated with the position of the totem such that the virtual user interface, when viewed by the user, appears to be stationary at the position of the predefined physical object”) (Bradski, Para 90; “a database comprising a passable world model, the passable world model comprising a set of map points associated with physical objects of the world, and a processor communicatively coupled to the image capturing device to recognize, based on the one or more images, a command to create a virtual user interface, to determine, based at least in part on the passable world model, map points corresponding to the predefined physical object, and to control the display in a manner such that the virtual user interface is generated at the determined map points corresponding to the predefined physical object, such that the virtual user interface appears to be stationary at the position of the predefined physical object”) of an environment surrounding an area of interest (Bradski, Para 135; “generating virtual content based on the task data, creating a virtual user interface in a field of view of the user, retrieving a set of map points corresponding to a location of the user, and displaying the virtual content on the virtual user interface, while the user is performing the task, such that the virtual user interface, when viewed by the user, appears to be fixed at the set of map points”) (Bradski, Para 125; "retrieving a first set of map points corresponding to a first location of the first user, creating a first virtual user interface in a field of view of a first user, based at least in part on the retrieved set of map points, and displaying the virtual content on the first virtual user interface, such that the first virtual user interface, when viewed by the first user, appears to be fixed at the first set of map points") (Bradski, Para 11-13; “In another example, an object may be world-centric, in which case it may be presented as though it is part of the real world coordinate system, such that the user may move his head or eyes without moving the position of the object relative to the real world. Thus when placing virtual content into the augmented reality world presented with an AR system, choices are made as to whether the object should be presented as world centric”) in real-time during a medical procedure (Bradski, Para 126-128; “In one or more embodiments, the virtual content is selected from the group consisting of a three-dimensional image of a target of the surgery, patient identification information, a medical image, vital patient sign information, and a medical chart […] n one or more embodiments, the virtual content is displayed to the first user during the surgical procedure, and the method further comprises receiving user input, generating additional virtual content based on the user input, and displaying the additional virtual content on the first virtual user interface, while the first user is performing the surgical procedure”);
a non-transitory computer readable medium (Bradski, Para 721; “The control system may include one or more non-transitory computer- or processor readable medium that stores executable logic or instructions and/or data or information”) programmed for:
receiving medical image information (Bradski, Para 126; “the virtual content is selected from the group consisting of a three-dimensional image of a target of the surgery [...] a medical image”) (Bradski, Para 761-763; “At 3002, when a user walks into a room, the user's individual AR system captures information (e.g., images, sensor information, pose tagged images, etc.) about the user's surroundings from multiple points of view […] It should be appreciated that the object recognizer takes 2D images of the object (2D color information, etc.), 3D images (depth information) and also takes 3D sparse points to recognize the object in a geometric coordinate frame of the world”) related to an area of interest (Bradski, Para 126; “the virtual content is selected from the group consisting of a three-dimensional image of a target of the surgery [...] a medical image”) (Bradski, Para 135; “generating virtual content based on the task data, creating a virtual user interface in a field of view of the user, retrieving a set of map points corresponding to a location of the user, and displaying the virtual content on the virtual user interface, while the user is performing the task, such that the virtual user interface, when viewed by the user, appears to be fixed at the set of map points”) (Bradski, Para 125; "retrieving a first set of map points corresponding to a first location of the first user, creating a first virtual user interface in a field of view of a first user, based at least in part on the retrieved set of map points, and displaying the virtual content on the first virtual user interface, such that the first virtual user interface, when viewed by the first user, appears to be fixed at the first set of map points") (Bradski, Para 11-13; “In another example, an object may be world-centric, in which case it may be presented as though it is part of the real world coordinate system, such that the user may move his head or eyes without moving the position of the object relative to the real world. Thus when placing virtual content into the augmented reality world presented with an AR system, choices are made as to whether the object should be presented as world centric”);
labeling and linking positional information related to the area of interest in an image frame of reference to corresponding positional information in the world frame of reference (Bradski, Para 135; “generating virtual content based on the task data, creating a virtual user interface in a field of view of the user, retrieving a set of map points corresponding to a location of the user, and displaying the virtual content on the virtual user interface, while the user is performing the task, such that the virtual user interface, when viewed by the user, appears to be fixed at the set of map points”) (Bradski, Para 974; “For example, as a user walks through an environment, the user's individual AR system captures information (e.g., images) and saves the information as posed tagged images, which form the core of the passable world model. The passable world model is a combination of raster imagery, point + descriptors clouds, and/or polygonal/geometric definitions (referred to herein as parametric geometry)”) (Bradski, Para 125; "retrieving a first set of map points corresponding to a first location of the first user, creating a first virtual user interface in a field of view of a first user, based at least in part on the retrieved set of map points, and displaying the virtual content on the first virtual user interface, such that the first virtual user interface, when viewed by the first user, appears to be fixed at the first set of map points") (Bradski, Para 11-13; “In another example, an object may be world-centric, in which case it may be presented as though it is part of the real world coordinate system, such that the user may move his head or eyes without moving the position of the object relative to the real world. Thus when placing virtual content into the augmented reality world presented with an AR system, choices are made as to whether the object should be presented as world centric, body-centric, head-centric or eye centric.”) (Bradski, Para 85; “a method for generating a virtual user interface, comprises detecting a manipulation of a predefined physical object, recognizing, based on the detected manipulation, a command to create a virtual user interface, determining, from a virtual world model, a set of map points associated with a position of the predefined physical object, and rendering, in real-time, the virtual user interface at the determined map points associated with the position of the totem such that the virtual user interface, when viewed by the user, appears to be stationary at the position of the predefined physical object”) (Bradski, Para 90; “a database comprising a passable world model, the passable world model comprising a set of map points associated with physical objects of the world, and a processor communicatively coupled to the image capturing device to recognize, based on the one or more images, a command to create a virtual user interface, to determine, based at least in part on the passable world model, map points corresponding to the predefined physical object, and to control the display in a manner such that the virtual user interface is generated at the determined map points corresponding to the predefined physical object, such that the virtual user interface appears to be stationary at the position of the predefined physical object”) (Bradski, Para 761; “At 3002, when a user walks into a room, the user's individual AR system captures information (e.g., images, sensor information, pose tagged images, etc.) about the user's surroundings from multiple points of view. At 3004, a set of 3D points may be extracted from the one or more captured images. For example, by the time the user walks into a section of a room, the user's individual AR system has already captured numerous keyframes and pose tagged images about the surroundings (similar to the embodiment shown in Fig. 28)”) using a coordinate transformation (Bradski, Para 73; “the processor performs a transformation between the reference frame associated with the identified virtual user interface and the obtained location of the user in relation to the world to determine the set of coordinate points of the virtual user interface.”) (Bradski, Para 761-764; describing a system in which captured images/objects are correlated to a world model) (Bradski, Para 1868; “In other words, if the values of matrix A is known, the exact geometry of the image plane may also be known. In other words, the geometry of the image plane is encoded by matrix A.”) and;
displaying a visual representation of the positional information in the image frame of reference linked to the corresponding positional information in the world frame of reference in the visual field displayed by the spatial mapping apparatus (Bradski, Para 761; “At 3002, when a user walks into a room, the user's individual AR system captures information (e.g., images, sensor information, pose tagged images, etc.) about the user's surroundings from multiple points of view. At 3004, a set of 3D points may be extracted from the one or more captured images. For example, by the time the user walks into a section of a room, the user's individual AR system has already captured numerous keyframes and pose tagged images about the surroundings (similar to the embodiment shown in Fig. 28)”) (Bradski, Para 973; “The passable world model provides the ability to efficiently communicate or pass information that essentially encompasses at least a field of view of a user.”), such that the visual representation shows the area of interest along with corresponding positional information from both the image frame of reference and the world frame of reference (Bradski, Para 126; “the virtual content is selected from the group consisting of a three-dimensional image of a target of the surgery [...] a medical image”) (Bradski, Para 135; “generating virtual content based on the task data, creating a virtual user interface in a field of view of the user, retrieving a set of map points corresponding to a location of the user, and displaying the virtual content on the virtual user interface, while the user is performing the task, such that the virtual user interface, when viewed by the user, appears to be fixed at the set of map points”) (Bradski, Para 125; "retrieving a first set of map points corresponding to a first location of the first user, creating a first virtual user interface in a field of view of a first user, based at least in part on the retrieved set of map points, and displaying the virtual content on the first virtual user interface, such that the first virtual user interface, when viewed by the first user, appears to be fixed at the first set of map points") (Bradski, Para 11-13; “In another example, an object may be world-centric, in which case it may be presented as though it is part of the real world coordinate system, such that the user may move his head or eyes without moving the position of the object relative to the real world. Thus when placing virtual content into the augmented reality world presented with an AR system, choices are made as to whether the object should be presented as world centric”) (Bradski, Para 765; “At 3010, the AR system may synchronize the parametric geometry of the objects to the cloud. Next, at 3012, the object recognizer may re-insert the geometric and parametric information into the passable world model. For example, the object recognizer may dynamically estimate the angle of the door, and insert it into the world”) (Bradski, Para 853; “The AR system may render virtual content locked to various reference frames, as discussed above. For example, where the AR system includes a head worn component, a view locked reference head-mounted (HMD) frame may be useful. That is, the reference frame stays locked to a reference frame of the head, turning and/or tilting with movement of the head. A body locked reference frame is locked to a reference frame of the body, essentially moving around (e.g., translating, rotating) with the movement of the user's body. A world locked reference frame is fixed to a reference frame of the environment and remains stationary within environment. For example, a world locked reference frame may be fixed to a room, wall or table”), wherein the visual representation is dynamically updated as the spatial mapping apparatus moves relative to the area of interest (Bradski, Para 98; “the method further comprises transmitting the first updated information corresponding to the first portion of the virtual world model to the first user, wherein the first updated information is configured to indicate whether any portion of the first updated information needs to be displayed to the first user. In one or more embodiments, the method further comprises transmitting the first updated information corresponding to the first portion of the virtual world model to a plurality of other users, wherein the first updated information is configured to indicate whether any portion of the first updated information needs to be displayed to each of the plurality of other users.”) (Bradski, Para 376; “The part of the dynamic object that is changing can be updated by a real-time, threaded high priority data stream from a server 1 1 , through computing network 5, managed by the gateway component 14.”) (Bradski, Para 1506; “the passable world allows the first user to transmit information about the room to the second user, and simultaneously allows the second user to create an avatar to place himself/herself in the physical environment of the first user. Thus, both users are continuously updating, transmitting and receiving information from the cloud, giving both users the experience of being in the same room at the same time”) (Bradski, Para 1643; “The AR system renders a patient's pre-mapped anatomy (e.g., heart) in virtual form 91 12 for the team to analyze during the planning. The AR system may render the anatomy using a light field, which allows viewing from any angle or orientation. For example, the surgeon could walk around the heart to see a back side thereof.”) (Bradski, Para 853; “The AR system may render virtual content locked to various reference frames, as discussed above. For example, where the AR system includes a head worn component, a view locked reference head-mounted (HMD) frame may be useful. That is, the reference frame stays locked to a reference frame of the head, turning and/or tilting with movement of the head. A body locked reference frame is locked to a reference frame of the body, essentially moving around (e.g., translating, rotating) with the movement of the user's body. A world locked reference frame is fixed to a reference frame of the environment and remains stationary within environment. For example, a world locked reference frame may be fixed to a room, wall or table”).
Bradski does not clearly and explicitly disclose a dual-modality radiovisible marker, wherein the dual-modality radiovisible marker is both radiopaque for visualization in x-ray imaging and comprises optically-detectable features for visualization by the spatial mapping apparatus, wherein the dual-modality radiovisible marker is positioned such that it is in view of the spatial mapping apparatus; wherein the medical image information includes x-ray image information and wherein the x-ray image information includes the dual-modality radiovisible marker; automatically detecting and extracting geometric characteristics of the dual-modality radiovisible marker from both the x-ray image information and from spatial mapping data captured by the spatial mapping apparatus, computing a coordinate transformation matrix between an image frame of reference and the world frame of reference based on correlation of the geometric characteristics of the dual-modality radiovisible marker detected in both frames of reference; and creating a visual combination of the x-ray image information and the world frame of reference using the dual-modality radiovisible marker as a guidepost in the x-ray image and the world frame of reference for the area of interest.
In an analogous visually augmented surgical display field of endeavor Lang discloses a dual-modality radiovisible marker, wherein the dual-modality radiovisible marker is both radiopaque for visualization in x-ray imaging and comprises optically-detectable features for visualization by a spatial mapping apparatus(Lang, Para 451; “portions of the optical marker or the entire optical marker can be radiopaque, so that the optical marker can also be visible on a radiograph or other imaging studies that utilize ionizing radiation including, for example, fluoroscopy, digital tomosynthesis, cone beam CT, and/or computed tomography”) (Lang, Para 453; "Multiple partially or completely radiopaque optical markers can be used. The radiopaque optical markers can be applied at different locations and in different planes around the surgical site."), wherein the dual-modality radiovisible marker is positioned such that it is in view of the spatial mapping apparatus (Lang, Para 453; "By using multiple radiopaque optical markers in multiple different locations and in different planes around the surgical site, the accuracy of any three-dimensional spatial registration and cross-reference of the optical markers in different modalities, e.g. radiographs, image capture, can be increased, for example by obtaining multiple x-rays at different angles, e.g. AP, lateral and/or oblique, and/or by imaging the radiopaque optical markers from multiple view angles using an image and/or video capture system integrated into [...] the OHMD" emphasis added);
wherein received medical image information is x-ray image information (Lang, Para 652-655; "In some embodiments of the invention, intraoperative imaging, for example using x-ray imaging or CT imaging and/or ultrasound imaging, can be performed. [...] The intraoperative x-ray images can then be registered and, optionally, superimposed onto the preoperative data of the patient or the live data of the patient in the projection by the OHMD. [...] intraoperative imaging [...] used for enhancing the accuracy of the registration of preoperative virtual data of the patient and live data of the patient and, for example, preoperative virtual data of the patient and/or a medical device intended for placement in a surgical site will be displayed by the OHMD together with the view of the live data of the patient or the surgical site") and wherein the x-ray image information includes the dual-modality radiovisible marker, automatically detecting and extracting geometric characteristics of the dual-modality radiovisible marker from both the x-ray image information and from spatial mapping data captured by the spatial mapping apparatus, computing a coordinate transformation between an image frame of reference and the world frame of reference based on correlation of the geometric characteristics of the dual-modality radiovisible marker detected in both frames of reference (Lang, Para 451; "portions of the optical marker or the entire optical marker can be radiopaque, so that the optical marker can also be visible on a radiograph or other imaging studies that utilize ionizing radiation including, for example, fluoroscopy, digital tomosynthesis, cone beam CT, and/or computed tomography") (Lang, Para 453; "Multiple partially or completely radiopaque optical markers can be used. [...] By using multiple radiopaque optical markers in multiple different locations and in different planes around the surgical site, the accuracy of any three-dimensional spatial registration and cross-reference of the optical markers in different modalities, e.g. radiographs, image capture, can be increased, for example by obtaining multiple x-rays at different angles, e.g. AP, lateral and/or oblique, and/or by imaging the radiopaque optical markers from multiple view angles using an image and/or video capture system integrated into, attached to or separate from the OHMD") (Lang, Para 658; "skin markers and soft-tissue markers, calibration or registration phantoms or devices can be used for registering preoperative virtual data, optionally intraoperative virtual data such as data obtained from intraoperative x-ray imaging, and live data seen through the OHMD in a common coordinate system with one or more OHMD's. Virtual and physical surgical instruments and implant components can also be registered in the common coordinate system. For example, an initial registration between preoperative virtual data and live data of the patient can happen at the beginning of the procedure. The initial registration can, for example, be performed using corresponding anatomic landmarks, surfaces or shapes, or using intraoperative imaging resulting in intraoperative virtual data or any of the other embodiments described in this invention. The registration can be used, for example, to place the virtual data and the live data and the optical head mounted display into a common coordinate system. Skin markers, calibration or registration phantoms or devices can then be applied. Virtual and physical surgical instruments and implant components can also be registered in the common coordinate system") (Lang, Para 458; “radiopaque and non-radiopaque optical markers can optionally be attached to or applied to extenders that increase the distance of the optical marker from the patient's skin. Such extenders can, for example, be anchored in a spinous process, a pedicle or other spinal element via a pin, drill or screw. The use of extenders with attached radiographic optical markers can increase the accuracy of registration between radiographic data and image capture data, for example when AP and lateral radiographs are used.”); and
creating a visual combination of the x-ray image information and a frame of reference using the dual-modality radiovisible marker as a guidepost in the x-ray image and the frame of reference (Lang, Para 1514; “the radiopaque portions of the optical marker detected on the imaging test are cross-referenced with the visible portions of the optical marker detected with the camera, image capture or video system and wherein the information is used to register the internal structures of the patient or the surgical site in the common coordinate system”) (Lang, Para 453; “By using multiple radiopaque optical markers in multiple different locations and in different planes around the surgical site, the accuracy of any three-dimensional spatial registration and cross-reference of the optical markers in different modalities, e.g. radiographs, image capture, can be increased, for example by obtaining multiple x-rays at different angles, e.g. AP, lateral and/or oblique, and/or by imaging the radiopaque optical markers from multiple view angles using an image and/or video capture system integrated into [...] the OHMD […] In addition, the accuracy of the registration can be better maintained as the view angle or radiographic angle changes, for example during the course of the surgical procedure or due to patient movement.”) (Lang, Para 658; "skin markers and soft-tissue markers, calibration or registration phantoms or devices can be used for registering preoperative virtual data, optionally intraoperative virtual data such as data obtained from intraoperative x-ray imaging, and live data seen through the OHMD in a common coordinate system with one or more OHMD's. Virtual and physical surgical instruments and implant components can also be registered in the common coordinate system. For example, an initial registration between preoperative virtual data and live data of the patient can happen at the beginning of the procedure. The initial registration can, for example, be performed using corresponding anatomic landmarks, surfaces or shapes, or using intraoperative imaging resulting in intraoperative virtual data or any of the other embodiments described in this invention. The registration can be used, for example, to place the virtual data and the live data and the optical head mounted display into a common coordinate system. Skin markers, calibration or registration phantoms or devices can then be applied. Virtual and physical surgical instruments and implant components can also be registered in the common coordinate system").
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Bradski to include a dual-modality radiovisible marker, wherein the dual-modality radiovisible marker is both radiopaque for visualization in x-ray imaging and comprises optically-detectable features for visualization by the spatial mapping apparatus, wherein the dual-modality radiovisible marker is positioned such that it is in view of the spatial mapping apparatus; wherein the medical image information includes x-ray image information and wherein the x-ray image information includes the dual-modality radiovisible marker; automatically detecting and extracting geometric characteristics of the dual-modality radiovisible marker from both the x-ray image information and from spatial mapping data captured by the spatial mapping apparatus, computing a coordinate transformation between an image frame of reference and the world frame of reference based on correlation of the geometric characteristics of the dual-modality radiovisible marker detected in both frames of reference; and creating a visual combination of the x-ray image information and the world frame of reference using the dual-modality radiovisible marker as a guidepost in the x-ray image and the world frame of reference for the area of interesting order to increase the accuracy of image registration as taught by Lang (Lang, Para 458 and 453) and in order to allow for better identification of spinal level for surgical accuracy as taught by Lang (Lang, Para 507)
Bradski as modified by Lang above is interpreted as disclosing creating a visual combination of the x-ray image information and the world frame of reference using the radiovisible marker as a guidepost in the x-ray image and the world frame of reference for the area of interest because Bradski shows medical images in a world frame of reference and is modified by Lang to use the radiovisible marker as a guidepost to combine x-ray medical images with a frame of reference.
In an analogous transformation between two coordinate systems field of endeavor Lee discloses using a transformation matrix to compute transformations between two coordinate systems (Lee, Para 13-14; “the coordinate transformation relationships may be expressed as coordinate transformation matrices […] where PR is a coordinate transformation matrix of the patient with respect to the reference marker unit, T1 is a coordinate transformation matrix of the reference marker unit with respect to the tracking sensor unit, T2 is a coordinate transformation matrix of the marker of the shape measurement unit with respect to the tracking sensor unit, T3 is a coordinate transformation matrix of the measurement device with respect to the marker of the shape measurement unit, and T4 is a coordinate transformation matrix of the patient with respect to the measurement device of the shape measurement unit”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Bradski as modified by Lang above to include using a transformation matrix in order to perform accurate registration in a shorter time and at reduced cost even if a patient moves or posture changes during a medical procedure as taught by Lee (Lee, Para 7-8).
Regarding claim 2, Bradski as modified by Lang and Lee above discloses all of the limitations of claim 1 as discussed above.
Bradski further discloses generating visual representations in a 2D image frame of reference that are visualized in a 3D world frame of reference (Bradski, Para 764; “Next, at 3006, the object recognizer(s) may correlate the 2D segmented image features with the sparse 3D points to derive object structures and one or more properties about the object using 2D/3D data fusion. For example, the object recognizer may identify specific geometry of the door with respect to the keyframes. Next, at 3008, the object recognizer parameterizes the geometry of the object. For example, the object recognizer may attach semantic information to the geometric primitive (e.g., the door has a hinge, the door can rotate 90 degrees, etc.) of the object. Or, the object recognizer may reduce the size of the door, to match the rest of the objects in the surroundings, etc..”).
Regarding claim 3, Bradski as modified by Lang and Lee above discloses all of the limitations of claim 1 as discussed above.
Bradski does not clearly and explicitly disclose wherein the dual-modality radiovisible marker takes the form of a dual-modality radiovisible augmented reality marker.
However, Lang further discloses a dual-modality radiovisible augmented reality marker (Lang, Para 458; “radiopaque and non-radiopaque optical markers can optionally be attached to or applied to extenders that increase the distance of the optical marker from the patient's skin. Such extenders can, for example, be anchored in a spinous process, a pedicle or other spinal element via a pin, drill or screw. The use of extenders with attached radiographic optical markers can increase the accuracy of registration between radiographic data and image capture data, for example when AP and lateral radiographs are used.”) (Lang, Para 453; "Multiple partially or completely radiopaque optical markers can be used. [...] By using multiple radiopaque optical markers in multiple different locations and in different planes around the surgical site, the accuracy of any three-dimensional spatial registration and cross-reference of the optical markers in different modalities, e.g. radiographs, image capture, can be increased, for example by obtaining multiple x-rays at different angles, e.g. AP, lateral and/or oblique, and/or by imaging the radiopaque optical markers from multiple view angles using an image and/or video capture system integrated into, attached to or separate from the OHMD") (Lang, Para 658; "skin markers and soft-tissue markers, calibration or registration phantoms or devices can be used for registering preoperative virtual data, optionally intraoperative virtual data such as data obtained from intraoperative x-ray imaging, and live data seen through the OHMD in a common coordinate system with one or more OHMD's. Virtual and physical surgical instruments and implant components can also be registered in the common coordinate system. For example, an initial registration between preoperative virtual data and live data of the patient can happen at the beginning of the procedure. The initial registration can, for example, be performed using corresponding anatomic landmarks, surfaces or shapes, or using intraoperative imaging resulting in intraoperative virtual data or any of the other embodiments described in this invention. The registration can be used, for example, to place the virtual data and the live data and the optical head mounted display into a common coordinate system. Skin markers, calibration or registration phantoms or devices can then be applied. Virtual and physical surgical instruments and implant components can also be registered in the common coordinate system").
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Bradski as modified by Lang and Lee above wherein the dual-modality radiovisible marker takes the form of a dual-modality radiovisible augmented reality marker in order to increase the accuracy of image registration as taught by Lang (Lang, Para 458).
Regarding claim 4, Bradski as modified by Lang and Lee above discloses all of the limitations of claim 1 as discussed above.
Bradski further discloses a head mounted display (HMD) (Bradski, Para 365; “Users interact with one or more digital worlds using some type of a local computing device, which is generally designated as a user device 12. Examples of such user devices include, but are not limited to, a smart phone, tablet device, heads-mounted display (HMD”).
Regarding claim 5, Bradski as modified by Lang and Lee above discloses all of the limitations of claim 4 as discussed above.
Bradski further discloses wherein the HMD uses Simultaneous Localization and Mapping (SLAM) (Bradski, Para 979; “A set of "sparse point representation" may be the output of a simultaneous localization and mapping (or "SLAM"; or "V-SLAM") 5124. This refers to a configuration wherein the input is an images/visual only) process. The system is not only determines where in the world the various components are, but also what the world comprises. Pose 5108 is a building block that achieves many goals, including populating the Map 5106 and using the data from the Map 5106”).
Regarding claim 6, Bradski as modified by Lang and Lee above discloses all of the limitations of claim 1 as discussed above.
Bradski further discloses creating a template on the imaging frame of reference in two dimensions (Bradski, Para 764; “Next, at 3006, the object recognizer(s) may correlate the 2D segmented image features with the sparse 3D points to derive object structures and one or more properties about the object using 2D/3D data fusion. For example, the object recognizer may identify specific geometry of the door with respect to the keyframes”) and subsequently allowing a user to visualize this template in three dimensions in the world space (Bradski, Para 765; “It should be appreciated that this information may be updated to the cloud, and transmitted to other AR systems such that virtual content may be appropriately displayed in relation to the recognized door”) (Bradski, Para 135; “generating virtual content based on the task data, creating a virtual user interface in a field of view of the user, retrieving a set of map points corresponding to a location of the user, and displaying the virtual content on the virtual user interface, while the user is performing the task, such that the virtual user interface, when viewed by the user, appears to be fixed at the set of map points”).
Regarding claim 7, Bradski as modified by Lang and Lee above discloses all of the limitations of claim 1 as discussed above.
Bradski further discloses scanning an environment as a whole and linking the image information to this reference frame (Bradski, Para 5; “The human visual system is not a passive sensor type of system; it actively scans the environment”) (Bradski, Para 96; “method for updating a virtual world, comprises receiving a first input from a first device of a first user, the first input corresponding to a physical environment of the first user, updating a virtual world model based on the received first input, the virtual world model corresponding to the physical environment of the first user”).
Regarding claim 8, Bradski as modified by Lang and Lee above discloses all of the limitations of claim 1 as discussed above.
Bradski further discloses generating a workflow that mimics a surgeon’s (Bradski, Para 1811; “It should be appreciated that the AR system may thus have many utilities in the health care and hospital space, and may help doctors (e.g., surgeon, radiologist etc.) accurately visualize various organs in the body to diagnose or treat their patients accordingly”) preferred workflow (Bradski, Para 783; “in order to render an avatar that properly mimics the user, the user may train the AR system, for example by moving through a desired or prescribed set of movements. In response, the AR system may generate an avatar sequence in which an avatar replicates the movements, for example, by animating the avatar. Thus, the AR system captures or receives images of a user, and generates animations of an avatar based on movements of the user in the captured images. The user may be instrumented, for example, by wearing one or more sensors”).
Regarding claim 9, Bradski as modified by Lang and Lee above discloses all of the limitations of claim 1 as discussed above.
Bradski does not clearly and explicitly disclose displaying a virtual line that is perpendicular to a plane of the image that intersects a point of interest.
However, Lang further discloses displaying a virtual line that is perpendicular to a plane of an image that intersects a point of interest (Lang, Para 385-386; “the OHMD can display a virtual arbitrary plane over the surgical site. The virtual arbitrary plane can be perpendicular to the operating table. Using a virtual interface, e.g. a touch area on the virtual surgical plane and gesture tracking, the OHMD can detect how the surgeon is moving the virtual arbitrary plane. Optionally, the virtual arbitrary plane can maintain its perpendicular (or of desired other angle) orientation relative to the OR table while the surgeon is moving and/or re-orienting the plane; a perpendicular orientation can be desirable when the surgeon intends to make a perpendicular femoral neck cut. A different angle can be desirable, when the surgeon intends to make the femoral neck cut with another orientation […] The moving of the arbitrary virtual plane can include translation and rotation or combinations thereof in any desired direction using any desired angle or vector. The surgeon can move the arbitrary virtual plane to intersect with select anatomic landmarks or to intersect with select anatomic or biomechanical axes. The surgeon can move the arbitrary virtual plane to be tangent with select anatomic landmarks or select anatomic or biomechanical axes”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Bradski as modified by Lang and Lee above to include displaying a virtual line that is perpendicular to a plane of the image that intersects a point of interest in order to guide a surgeon during a procedure (Lang, Para 21).
Regarding claim 10, Bradski as modified by Lang and Lee above discloses all of the limitations of claim 9 as discussed above.
Bradski does not clearly and explicitly disclose wherein the point of interest lies at any point along the virtual line.
However, Lang further discloses wherein the point of interest lies at any point along the virtual line (Lang, Para 385-386; “the OHMD can display a virtual arbitrary plane over the surgical site. The virtual arbitrary plane can be perpendicular to the operating table. Using a virtual interface, e.g. a touch area on the virtual surgical plane and gesture tracking, the OHMD can detect how the surgeon is moving the virtual arbitrary plane. Optionally, the virtual arbitrary plane can maintain its perpendicular (or of desired other angle) orientation relative to the OR table while the surgeon is moving and/or re-orienting the plane; a perpendicular orientation can be desirable when the surgeon intends to make a perpendicular femoral neck cut. A different angle can be desirable, when the surgeon intends to make the femoral neck cut with another orientation […] The moving of the arbitrary virtual plane can include translation and rotation or combinations thereof in any desired direction using any desired angle or vector. The surgeon can move the arbitrary virtual plane to intersect with select anatomic landmarks or to intersect with select anatomic or biomechanical axes. The surgeon can move the arbitrary virtual plane to be tangent with select anatomic landmarks or select anatomic or biomechanical axes”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Bradski as modified by Lang and Lee above wherein the point of interest lies at any point along the virtual line in order to guide a surgeon during a procedure (Lang, Para 21).
Regarding claim 11, Bradski as modified by Lang and Lee above discloses all of the limitations of claim 9 as discussed above.
Bradski does not clearly and explicitly disclose displaying the virtual line in a user's field of view in a head mounted display.
However, Lang further discloses displaying the virtual line in a user's field of view in a head mounted display (Lang, Para 1501; “the method of preparing a joint for a prosthesis in a patient comprises registering the patient's live surgical site and an optical head mounted display worn by a surgeon or surgical assistant in a common coordinate system, developing a virtual surgical plan, registering the virtual surgical plan in the common coordinate system, the virtual surgical plan including at least one virtual cut plane, and displaying or projecting the at least one virtual cut planes superimposed onto the corresponding portions of the patient's live surgical site with the optical head mounted display”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Bradski as modified by Lang and Lee above to include displaying the virtual line in a user's field of view in a head mounted display in order to guide a surgeon during a procedure (Lang, Para 21).
Claim 12 is rejected under 35 U.S.C. 103 as being unpatentable over Bradski, Lang, and Lee as applied to claim 1 above, and in further view of Chopra et al. (US20180092698, hereafter Chopra).
Regarding claim 12, Bradski as modified by Lang and Lee above discloses all of the limitations of claim 1 as discussed above.
Bradski further discloses wherein the spatial map comprises a number of world spatial maps and further comprising generating overlapping world spatial maps with the number of world spatial maps and objects in a room (Bradski, Para 794-795; “the AR system may layer topological maps on the passable world model, for example to localize nodes. The topological map can layer various types of information on the passable world model, for instance: point cloud, images, objects in space, global positioning system (GPS) data, Wi-Fi data, histograms (e.g., color histograms of a room), received signal strength (RSS) data, etc […] in order to create a complete virtual world that can be reliably passed between various users, the AR system captures different types of information about the user's surroundings(e.g., map points, features, pose tagged images, objects in a scene, etc.)”).
Bradski does not clearly and explicitly disclose applying a tracker rigidly fixed to a medical imaging system.
In an analogous enhanced reality medical system field of endeavor Chopra discloses applying a tracker rigidly fixed to a medical imaging system (Chopra, Para 64; “the term “probe” can mean a probe with sensors, energy emitters, detectors, radiopaque markers or other elements that can be detected by a sensor, or detect data or energy emissions, can perform a scanning operation (e.g. ultrasound imaging, micro x-ray detection, micro x-ray emission, or other modalities) and export detected signals to a control unit”) which are used for correlation in enhanced reality (Chopra, Para 83; “E (Enhanced reality) markers can be feature points that can be visible to the visual image camera (tablet, fixed camera, glasses/goggle mounted camera, etc.) and connect visual image with the scan image data. E markers may be visible to the visual image camera. The relative position of the E and P markers are used to determine the various positions of objects relative to the markers, thus the position of the P and E markers relative to each other is known”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Bradski as modified by Lang and Lee above to include applying a tracker rigidly fixed to a medical imaging system in order to allow for the use of markers as reference points which improves correlation of images in augmented reality (Chopra, Para 68-69).
Conclusion Any inquiry concerning this communication or earlier communications from the examiner should be directed to John Li whose telephone number is (313)446-4916. The examiner can normally be reached Monday to Thursday; 5:30 AM to 3:30 PM Eastern.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Pascal Bui-Pho can be reached at (571) 272-2714. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JOHN D LI/Primary Examiner, Art Unit 3798
Prosecution Insights