DETAILED ACTION
Response to Arguments
The amendments filed 2/25/2026 have been entered and made of record. Since Applicant amends claim 18 to recite “A non-transitory computer readable medium….”, therefore, 35 USC § 101 rejection for claim 18 has been withdrawn.
Applicant's arguments filed 2/25/2026 for 35 USC § 103 rejections have been fully considered but they are not persuasive:
Applicant asserts that cited the cited references, particularly do not disclose “a plurality of object markers arranged on at least two non-parallel surfaces or non-parallel surface portions of an object”;
However, the Examiner disagrees, because:
FRANTZ discloses a method for determining an object marker arrangement comprising a plurality of object markers arranged on at least two non-parallel surfaces or non-parallel surface portions of an object (see FRANTZ: e.g., --uses only a predetermined (optimal) sub-set of three IR object markers to detect the correspondence between the IR object markers in the IR image and the IR object markers of the set of IR object markers.--, in [0013]-[0014], Fig. 4B, --the IR sensor 12 is configured to detect light in the spectral range in which the object markers emit light and/or in which an IR light source of the HMD 10 emits light…to detect IR reflective object markers by illuminating them with the IR light source 13 and detecting in the IR image taken by the IR sensor 12 the object markers reflecting the IR light from the IR source 13. The IR light source emits the IR light also in the direction of view of the HMD 10. In a less preferred embodiment, it would also be possible to arrange the IR light source 13 in the environment to illuminate the markers 21. If the markers 21 emit actively IR light, the IR light source 13 can be completely avoided.--, in [0031],and, -- the at least four markers 21 are arranged in an asymmetric way on the object 20. This helps to retrieve the correct orientation of the object from the 3D positions of the markers 21 of the set of markers 21. The spatial/local relationship/arrangement between the markers 21 of the set of markers 21 arranged on the object 20 is known to the HMD 21 or stored in the HMD 21….[0037] The markers 21 within a set of markers 21 are preferably labelled. That is that each marker 21 of the set of markers 21 has a different label. A label is an identifier for identifying the marker 21. The identifier or label is preferably just virtual, i.e. is not visible from the image of the marker 21 in the IR image of the IR camera 12. The label of each marker 21 can be determined based on the 3D spatial relationship of the set of markers 21. A set of four markers 21 can for example be labelled from 1 to 4.--, in [0036]-[0037]; and, --[0038] The system can also comprise two or more objects 20 to be tracked, each having a (different) set of markers 21 arranged on it. For example, the first object can be a patient locator fixed on the patient to detect the pose of the patient (or a part of the patient like a certain bone) having a first set of markers 21 arranged on it as shown for example in FIG. 2. For example, the second object can be a surgical tool having a second set of markers 21 on it. The second set of markers 21 has a different spatial arrangement/relationship of markers 21 compared to the first set of markers 21 so that the two objects can be distinguished in the same IR image by the spatial relationship of their markers 21. Thus, when the HMD 10 can track both objects 20 together, it is possible to track the relative position of the second object or the surgical tool with respect to the first object or the patient. This allows to guide the surgeon with the correct application position of the surgical tool with respect to the patient based on the relative pose of the two objects 20. FIG. 4A shows for example an IR image taken with the IR sensor 12 showing two objects 20. The first object 20 is a patient locator fixed with respect to the patient or the region of interest of the patient. The second object 20′ is a surgical tool 20′ for example a drill.--, in [0038]; and, --[0055] In step S122, the pose of the object in the HMD coordinate system is determined/computed based on the markers 21 detected in the IR image, more precisely based on the 2D positions of the markers 21 in the IR image. Preferably, the pose of the object in the HMD coordinate system is determined/computed based on the markers 21 detected in the IR image and based on the known spatial relationship of the markers 21 in the set of markers 21 arranged on the object 20. Preferably, the pose of the object 20 is determined based on the 3D positions of the markers 21. The 3D positions of the markers 21 (or the pose of the object) in the HMD coordinate system are(is) determined based on the 2D positions of the markers 21 in the IR image. Preferably, in this step S122, the IR sensor coordinate system is used as HMD coordinate system. Preferably, the 3D positions of the markers 21 in the HMD coordinate system are determined based on the 2D positions of the markers 21 in the IR image and based on the known camera parameters of the IR camera 12. In the following, one preferred way to detect the pose of the object in the HMD coordinate system is determined/computed based on the markers 21 detected in the IR image.--, in [0055]);
Also see FRANTZ: Fig. 4, as reproduced below:
PNG
media_image1.png
646
695
media_image1.png
Greyscale
as demonstrated in above Fig. 4, there are two sets of markers, set 20 of markers of “the first object can be a patient locator fixed on the patient to detect the pose of the patient (or a part of the patient like a certain bone) having a first set of markers 21”, and,
set 20’ of markers of the second object 20′ is a surgical tool 20′ for example a drill;
Apparently, these two separated sets of markers read on claimed limitation “a plurality of object markers arranged on at least two non-parallel surfaces or non-parallel surface portions of an object”, because at least the second set 20’ of markers are moving three dimensionally when the tool moving during the surgery, while the first set of markers are fixed on patient’s bone, and it is apparently these two sets of markers are on two non-parallel surfaces or non-parallel surface portions of an object, even if patient bone is considered as the target object, since the tool is moving, and apparently at different places of the patient bone these two separated markers would be on at least two non-parallel surfaces or non-parallel surface portions of an object.
Therefore, claims 1-20 are still not patentably distinguishable over the prior art reference(s). Further discussions are addressed in the prior art rejection section below.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over FRANTZ (US 20240062387 A1), and in view of Minne (US 20240299100 A1),
and further in view of JOSHI (US 20230363827 A1).
Re Claim 1, FRANTZ discloses a method for determining an object marker arrangement comprising a plurality of object markers arranged on at least two non-parallel surfaces or non-parallel surface portions of an object (see FRANTZ: e.g., --uses only a predetermined (optimal) sub-set of three IR object markers to detect the correspondence between the IR object markers in the IR image and the IR object markers of the set of IR object markers.--, in [0013]-[0014], Fig. 4B, --the IR sensor 12 is configured to detect light in the spectral range in which the object markers emit light and/or in which an IR light source of the HMD 10 emits light…to detect IR reflective object markers by illuminating them with the IR light source 13 and detecting in the IR image taken by the IR sensor 12 the object markers reflecting the IR light from the IR source 13. The IR light source emits the IR light also in the direction of view of the HMD 10. In a less preferred embodiment, it would also be possible to arrange the IR light source 13 in the environment to illuminate the markers 21. If the markers 21 emit actively IR light, the IR light source 13 can be completely avoided.--, in [0031],and, -- the at least four markers 21 are arranged in an asymmetric way on the object 20. This helps to retrieve the correct orientation of the object from the 3D positions of the markers 21 of the set of markers 21. The spatial/local relationship/arrangement between the markers 21 of the set of markers 21 arranged on the object 20 is known to the HMD 21 or stored in the HMD 21….[0037] The markers 21 within a set of markers 21 are preferably labelled. That is that each marker 21 of the set of markers 21 has a different label. A label is an identifier for identifying the marker 21. The identifier or label is preferably just virtual, i.e. is not visible from the image of the marker 21 in the IR image of the IR camera 12. The label of each marker 21 can be determined based on the 3D spatial relationship of the set of markers 21. A set of four markers 21 can for example be labelled from 1 to 4.--, in [0036]-[0037]; and, --[0038] The system can also comprise two or more objects 20 to be tracked, each having a (different) set of markers 21 arranged on it. For example, the first object can be a patient locator fixed on the patient to detect the pose of the patient (or a part of the patient like a certain bone) having a first set of markers 21 arranged on it as shown for example in FIG. 2. For example, the second object can be a surgical tool having a second set of markers 21 on it. The second set of markers 21 has a different spatial arrangement/relationship of markers 21 compared to the first set of markers 21 so that the two objects can be distinguished in the same IR image by the spatial relationship of their markers 21. Thus, when the HMD 10 can track both objects 20 together, it is possible to track the relative position of the second object or the surgical tool with respect to the first object or the patient. This allows to guide the surgeon with the correct application position of the surgical tool with respect to the patient based on the relative pose of the two objects 20. FIG. 4A shows for example an IR image taken with the IR sensor 12 showing two objects 20. The first object 20 is a patient locator fixed with respect to the patient or the region of interest of the patient. The second object 20′ is a surgical tool 20′ for example a drill.--, in [0038]; and, --[0055] In step S122, the pose of the object in the HMD coordinate system is determined/computed based on the markers 21 detected in the IR image, more precisely based on the 2D positions of the markers 21 in the IR image. Preferably, the pose of the object in the HMD coordinate system is determined/computed based on the markers 21 detected in the IR image and based on the known spatial relationship of the markers 21 in the set of markers 21 arranged on the object 20. Preferably, the pose of the object 20 is determined based on the 3D positions of the markers 21. The 3D positions of the markers 21 (or the pose of the object) in the HMD coordinate system are(is) determined based on the 2D positions of the markers 21 in the IR image. Preferably, in this step S122, the IR sensor coordinate system is used as HMD coordinate system. Preferably, the 3D positions of the markers 21 in the HMD coordinate system are determined based on the 2D positions of the markers 21 in the IR image and based on the known camera parameters of the IR camera 12. In the following, one preferred way to detect the pose of the object in the HMD coordinate system is determined/computed based on the markers 21 detected in the IR image.--, in [0055]);
Also see FRANTZ: Fig. 4, as reproduced below:
PNG
media_image1.png
646
695
media_image1.png
Greyscale
as demonstrated in above Fig. 4, there are two sets of markers, set 20 of markers of “the first object can be a patient locator fixed on the patient to detect the pose of the patient (or a part of the patient like a certain bone) having a first set of markers 21”, and,
set 20’ of markers of the second object 20′ is a surgical tool 20′ for example a drill;
Apparently, these two separated sets of markers read on claimed limitation “a plurality of object markers arranged on at least two non-parallel surfaces or non-parallel surface portions of an object”, because at least the second set 20’ of markers are moving three dimensionally when the tool moving during the surgery, while the first set of markers are fixed on patient’s bone, and it is apparently these two sets of markers are on two non-parallel surfaces or non-parallel surface portions of an object, even if patient bone is considered as the target object, since the tool is moving, and apparently at different places of the patient bone these two separated markers would be on at least two non-parallel surfaces or non-parallel surface portions of an object);
wherein the object marker arrangement is characterized by positions of the object markers (see FRANTZ: e.g., --uses only a predetermined (optimal) sub-set of three IR object markers to detect the correspondence between the IR object markers in the IR image and the IR object markers of the set of IR object markers.--, in [0013]-[0014], Fig. 4B, --the IR sensor 12 is configured to detect light in the spectral range in which the object markers emit light and/or in which an IR light source of the HMD 10 emits light…to detect IR reflective object markers by illuminating them with the IR light source 13 and detecting in the IR image taken by the IR sensor 12 the object markers reflecting the IR light from the IR source 13. The IR light source emits the IR light also in the direction of view of the HMD 10. In a less preferred embodiment, it would also be possible to arrange the IR light source 13 in the environment to illuminate the markers 21. If the markers 21 emit actively IR light, the IR light source 13 can be completely avoided.--, in [0031],and, -- the at least four markers 21 are arranged in an asymmetric way on the object 20. This helps to retrieve the correct orientation of the object from the 3D positions of the markers 21 of the set of markers 21. The spatial/local relationship/arrangement between the markers 21 of the set of markers 21 arranged on the object 20 is known to the HMD 21 or stored in the HMD 21….[0037] The markers 21 within a set of markers 21 are preferably labelled. That is that each marker 21 of the set of markers 21 has a different label. A label is an identifier for identifying the marker 21. The identifier or label is preferably just virtual, i.e. is not visible from the image of the marker 21 in the IR image of the IR camera 12. The label of each marker 21 can be determined based on the 3D spatial relationship of the set of markers 21. A set of four markers 21 can for example be labelled from 1 to 4.--, in [0036]-[0037]; and, --[0055] In step S122, the pose of the object in the HMD coordinate system is determined/computed based on the markers 21 detected in the IR image, more precisely based on the 2D positions of the markers 21 in the IR image. Preferably, the pose of the object in the HMD coordinate system is determined/computed based on the markers 21 detected in the IR image and based on the known spatial relationship of the markers 21 in the set of markers 21 arranged on the object 20. Preferably, the pose of the object 20 is determined based on the 3D positions of the markers 21. The 3D positions of the markers 21 (or the pose of the object) in the HMD coordinate system are(is) determined based on the 2D positions of the markers 21 in the IR image. Preferably, in this step S122, the IR sensor coordinate system is used as HMD coordinate system. Preferably, the 3D positions of the markers 21 in the HMD coordinate system are determined based on the 2D positions of the markers 21 in the IR image and based on the known camera parameters of the IR camera 12. In the following, one preferred way to detect the pose of the object in the HMD coordinate system is determined/computed based on the markers 21 detected in the IR image.--, in [0055]);
FRANTZ however does not explicitly disclose a reference device with a pre-determined reference pattern is provided;
Minne discloses a reference device with a pre-determined reference pattern is provided (see Minne: e.g., Fig. 2, and, -- a reference frame of navigation target on calibration device 964--, in [0046], and, -- the calibration device navigation target 964… calibration device navigation target 964 are used to calculate the pose.--, in [0061], and [0063]-[0064];
in addition, also see: “markers”, in -- The trackable target might consist of a geometric arrangement of markers such as retroreflective spheres capable of reflecting infrared (“IR”) light, or of a similar arrangement of IR light emitters. These are detected by a localizer device 105 with an IR-detecting camera, and when required an (IR) light source to reflect off the retroreflectors and/or to stimulate a passively-synchronized IR emitter. Actively-synchronized IR emitters also exist. Alternatively the trackable target might consist of computer-vision readable targets created from high-contrast patterns, or by patterns that reflect or emit light in various regions of the electromagnetic spectrum.--, in [0067]);
FRANTZ and MINNE are combinable as they are in the same field of endeavor: marker arrangements to identify target object’s position and pose in medical imaging and surgery navigation. Therefore it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to further modify FRANTZ’s method using Minne’s teachings by including a reference device with a pre-determined reference pattern is provided to FRANTZ’s markers arrangements in order to determine and track the locations and pose of target object (see Minne: e.g. Fig. 2, in [0046], in [0061], [0063]-[0064] and [0067]); and
FRANTZ as modified by MINNE further disclose the method comprising the following steps at least partially performed by a processing device:
receiving image data representative of a plurality of images that contain the reference pattern and at least a subset of the object markers (see Minne: e.g. Fig. 2, --, a LOCALIZER_T_CALIBRATION-DEVICE transformation 912a, a LOCALIZER_T_CAMERA-TARGET transformation 914, patient anatomy 916, a digital surgical microscope (“DSM”) CAMERA TARGET 920 (and associated reference frame), a DSM CAMERA TARGET 920a, 920b, and 920c (and associated reference frame) during camera calibration (a plurality of poses), a CAMERA OPTICAL MODEL REFERENCE FRAME viewing patient anatomy 922, a CAMERA OPTICAL MODEL REFERENCE FRAME during camera calibration (a plurality of poses) 922a, 922b, and 922c, and a LOCALIZER REFERENCE FRAME 924.--, in [0045]; and
--A computer vision readable target 954 on the calibration device is detected in the acquired or received images and the locations of its salient features located relative to the photogrammetry reference frame 960 which salient features describe the X and Y axis of a reference frame 956 the Z axis of which reference frame--, in [0061], --To track a tool of interest (such as a clamp holding patient anatomy) the systems disclosed herein may include a trackable target mounted on a tool. The trackable target might consist of a geometric arrangement of markers such as retroreflective spheres capable of reflecting infrared (“IR”) light, or of a similar arrangement of IR light emitters. These are detected by a localizer device 105 with an IR-detecting camera, and when required an (IR) light source to reflect off the retroreflectors and/or to stimulate a passively-synchronized IR emitter. Actively-synchronized IR emitters also exist. Alternatively the trackable target might consist of computer-vision readable targets created from high-contrast patterns, or by patterns that reflect or emit light in various regions of the electromagnetic spectrum.--, in [0067]; and, --[0071] The locations of the navigation trackable markers 1002, 1004, 1005 and 1006 in the reference frame of the bracket 1000 are known by design and/or by measurement and together compose a tool described by the markers' geometric relation to each other in that reference frame. This information is stored in a tool description file and used by the navigation computer 170 to discern which tool is in view of the localizer 105 and in what pose relative to the localizer's reference frame 924.
[0072] The locations of the computer-vision readable targets 1008, 1010 and 1012 are also known by design and/or by measurement, in the same reference frame as the navigation markers, namely that of the bracket. Such CV-ready targets are detectable as unique regardless of viewing angle, and encode a known origin for example the physical center of the pattern. Other embodiments may loosen such restrictions for example by allowing viewing-angle redundant individual targets as long as the group of targets is detectable as unique regardless of viewing angle.--, in [0071]-[0072]; also see: --[0236] Multiple patterns of varying sizes may be optionally used to provide accurate calibration over a wide zoom range.
[0237] Traditional camera calibration can also provide a measure of the optical distortion of the system at the optical parameter settings at which the calibration process was performed. A set of distortion coefficients can be found and can be used in some embodiments to correct such optical distortion. In some embodiments, such distortion correction can be used to improve the field of view calibration method. Furthermore, in some embodiments, such distortion correction can be used to improve the accuracy of the overlay (e.g., how it matches the live view.)
[0238] In embodiments where an explicit field of view calibration process may be used to improve on the field of view determination for the projection matrix of the computer graphics renderer, the distance to the focal surface of each camera eye of the stereoscopic digital surgical microscope may be required to be calculated. The determination of this distance for each camera eye will be discussed herein, in relation to FIG. 21.
[0239] FIG. 20 is a diagram showing an angle of view applicable to the integrated surgical navigation and visualization system, according to an example embodiment of the present disclosure. With the focus distance, the angle of view can be calculated. This angle may be needed to calculate terms in the projection matrix and can be found by trigonometry, as shown in FIG. 20:
[0240] For example, the half angle 2600 can be found by measuring the focus distance 2610 from the camera center of projection (also referred to as the camera “eye point”) 2620 to the focus surface 2630 along the optical axis 2640. The additional field of view calibration can provide a measure of the field of view (for example the horizontal width) at the focus surface. The half of such distance is shown as marker 2650. The tangent of half angle 2600 is distance 2650 divided by distance 2640. The inverse tangent function can then be used to calculate the “half field of view angle.” The half field of view angle can be used to calculate directly certain matrix elements of the combined projection matrix--, in [0236]-[0240]);
wherein at least some of the images were captured by an imaging device from different viewing angles, wherein the reference pattern and the object markers were arranged in a fixed spatial relationship relative to each other when the images were captured (see Minne: e.g. Fig. 2, --, a LOCALIZER_T_CALIBRATION-DEVICE transformation 912a, a LOCALIZER_T_CAMERA-TARGET transformation 914, patient anatomy 916, a digital surgical microscope (“DSM”) CAMERA TARGET 920 (and associated reference frame), a DSM CAMERA TARGET 920a, 920b, and 920c (and associated reference frame) during camera calibration (a plurality of poses), a CAMERA OPTICAL MODEL REFERENCE FRAME viewing patient anatomy 922, a CAMERA OPTICAL MODEL REFERENCE FRAME during camera calibration (a plurality of poses) 922a, 922b, and 922c, and a LOCALIZER REFERENCE FRAME 924.--, in [0045]; and
--A computer vision readable target 954 on the calibration device is detected in the acquired or received images and the locations of its salient features located relative to the photogrammetry reference frame 960 which salient features describe the X and Y axis of a reference frame 956 the Z axis of which reference frame--, in [0061], --To track a tool of interest (such as a clamp holding patient anatomy) the systems disclosed herein may include a trackable target mounted on a tool. The trackable target might consist of a geometric arrangement of markers such as retroreflective spheres capable of reflecting infrared (“IR”) light, or of a similar arrangement of IR light emitters. These are detected by a localizer device 105 with an IR-detecting camera, and when required an (IR) light source to reflect off the retroreflectors and/or to stimulate a passively-synchronized IR emitter. Actively-synchronized IR emitters also exist. Alternatively the trackable target might consist of computer-vision readable targets created from high-contrast patterns, or by patterns that reflect or emit light in various regions of the electromagnetic spectrum.--, in [0067]; and, --[0071] The locations of the navigation trackable markers 1002, 1004, 1005 and 1006 in the reference frame of the bracket 1000 are known by design and/or by measurement and together compose a tool described by the markers' geometric relation to each other in that reference frame. This information is stored in a tool description file and used by the navigation computer 170 to discern which tool is in view of the localizer 105 and in what pose relative to the localizer's reference frame 924.
[0072] The locations of the computer-vision readable targets 1008, 1010 and 1012 are also known by design and/or by measurement, in the same reference frame as the navigation markers, namely that of the bracket. Such CV-ready targets are detectable as unique regardless of viewing angle, and encode a known origin for example the physical center of the pattern. Other embodiments may loosen such restrictions for example by allowing viewing-angle redundant individual targets as long as the group of targets is detectable as unique regardless of viewing angle.--, in [0071]-[0072]; also see: --[0236] Multiple patterns of varying sizes may be optionally used to provide accurate calibration over a wide zoom range.
[0237] Traditional camera calibration can also provide a measure of the optical distortion of the system at the optical parameter settings at which the calibration process was performed. A set of distortion coefficients can be found and can be used in some embodiments to correct such optical distortion. In some embodiments, such distortion correction can be used to improve the field of view calibration method. Furthermore, in some embodiments, such distortion correction can be used to improve the accuracy of the overlay (e.g., how it matches the live view.)
[0238] In embodiments where an explicit field of view calibration process may be used to improve on the field of view determination for the projection matrix of the computer graphics renderer, the distance to the focal surface of each camera eye of the stereoscopic digital surgical microscope may be required to be calculated. The determination of this distance for each camera eye will be discussed herein, in relation to FIG. 21.
[0239] FIG. 20 is a diagram showing an angle of view applicable to the integrated surgical navigation and visualization system, according to an example embodiment of the present disclosure. With the focus distance, the angle of view can be calculated. This angle may be needed to calculate terms in the projection matrix and can be found by trigonometry, as shown in FIG. 20:
[0240] For example, the half angle 2600 can be found by measuring the focus distance 2610 from the camera center of projection (also referred to as the camera “eye point”) 2620 to the focus surface 2630 along the optical axis 2640. The additional field of view calibration can provide a measure of the field of view (for example the horizontal width) at the focus surface. The half of such distance is shown as marker 2650. The tangent of half angle 2600 is distance 2650 divided by distance 2640. The inverse tangent function can then be used to calculate the “half field of view angle.” The half field of view angle can be used to calculate directly certain matrix elements of the combined projection matrix--, in [0236]-[0240]);
FRANTZ as modified by MINNE however do not explicitly disclose determining the positions of the object markers relative to the reference pattern, wherein the position of an individual one of the object markers is determined based on at least two images that contain the individual object marker and based on geometrical information about the reference pattern,
JOSHI discloses determining the positions of the object markers relative to the reference pattern, wherein the position of an individual one of the object markers is determined based on at least two images that contain the individual object marker and based on geometrical information about the reference pattern (see JOSHI: e.g., Fig. 5, Fig. 15, and Fig. 17, Fig. 18 and Fig. 19, -- The reference element arrays are typically small in size (e.g., on a few centimeters wide) to minimize obstruction of the surgical area. The number of markers is also usually limited to optimize costs and workflow. A larger array with more markers can improve the accuracy of divot position.--, in [0007], and, -- include obtaining a first image of the tracked instrument while it is positioned at the first physical position between the emitter and the detector. The operations further include determining a second virtual position within the virtual space of the emitter of the imaging device. The operations further include determining a second virtual position within the virtual space of the detector of the imaging device. The operations further include determining a second virtual position within the virtual space of the tracked instrument while the tracked instrument is at a second physical position between the emitter and the detector. The operations further include determining a second expected image of the tracked instrument based on the second virtual position of the emitter, the second virtual position of the detector, and the second virtual position of the tracked instrument. The operations further include obtaining a second image of the tracked instrument while it is positioned between the emitter and the detector, the second image being different than the first image. The operations further include determining whether the tracked instrument is accurate based on the first expected image, the second expected image, the first image, and the second image.--, [0012]);
FRANTZ (as modified by MINNE) and JOSHI are combinable as they are in the same field of endeavor: marker arrangements to identify target object’s position and pose in medical imaging and surgery navigation. Therefore it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to further modify FRANTZ (as modified by MINNE)’s method using JOSHI’s teachings by including determining the positions of the object markers relative to the reference pattern, wherein the position of an individual one of the object markers is determined based on at least two images that contain the individual object marker and based on geometrical information about the reference pattern to FRANTZ (as modified by MINNE)’s markers arrangements in order to determine position of the tracked object markers (see JOSHI: e.g., Fig. 5, Fig. 15, and Fig. 17, Fig. 18 and Fig. 19, and in [0007], and [0012]).
Re Claim 2, FRANTZ as modified by MINNE and JOSHI further disclose wherein the different viewing angles comprise at least (i) a first viewing angle in which the individual object marker is visible, and (ii) a second viewing angle in which the individual object marker is not visible but at least another one of the object markers is visible (see FRANTZ: e.g., Fig. 2, and, -- [0037] The markers 21 within a set of markers 21 are preferably labelled. That is that each marker 21 of the set of markers 21 has a different label. A label is an identifier for identifying the marker 21. The identifier or label is preferably just virtual, i.e. is not visible from the image of the marker 21 in the IR image of the IR camera 12. The label of each marker 21 can be determined based on the 3D spatial relationship of the set of markers 21. A set of four markers 21 can for example be labelled from 1 to 4. [0038] The system can also comprise two or more objects 20 to be tracked, each having a (different) set of markers 21 arranged on it. For example, the first object can be a patient locator fixed on the patient to detect the pose of the patient (or a part of the patient like a certain bone) having a first set of markers 21 arranged on it as shown for example in FIG. 2. For example, the second object can be a surgical tool having a second set of markers 21 on it. The second set of markers 21 has a different spatial arrangement/relationship of markers 21 compared to the first set of markers 21 so that the two objects can be distinguished in the same IR image by the spatial relationship of their markers 21. Thus, when the HMD 10 can track both objects 20 together, it is possible to track the relative position of the second object or the surgical tool with respect to the first object or the patient.--, in [0036]-[0038], and, -- [0054] In step S121, the markers 21 are detected in the IR image. The markers 21 detected in the IR image in step S121 can be described as candidate markers 21 which fulfill certain requirements to be considered as candidate marker 21. The requirements can depend on the luminosity (above a certain threshold), the size and/or the form of the (candidate) marker 21 in the IR image. In an optimal case, the same number of markers 21 are detected in the IR image as the set of markers 21 of the object 20. However, it is also possible that a lower number of markers 21 are detected, if some of the markers 21 are occluded e.g. by the specific pose of the object 20 itself or by another object in front of the object 20 to be detected. It is further possible that a higher number of (candidate) markers 21 is detected, if something else than the IR object marker 21 of the set of markers 21 emits or reflects IR light which leads to a false positive candidate marker 21 or if more than one object 20 is detected in the IR image. Preferably, the (candidate) markers 21 in the IR image and their 2D positions in the IR image are detected in step S121. The IR image has preferably a 2D coordinate system expressed in pixels. A first dimension extends in the image plane in a first direction, often called the x-axis or the row direction. A second dimension extends in the image plane in a second direction, often called the y-axis or the column direction. A 2D position of the IR image is a point in the IR image with two coordinates. The 2D position can also be expressed with a sub-pixel precision. Since the markers 21 emit or better reflect IR light and the IR image detect the intensity of IR light in the field of view, the markers 21 have a high IR intensity. So, one possible way to detect the markers 21 in the IR image is to detect all pixels above a certain IR light intensity as points of the markers 21.--, in [0054]).
Re Claim 3, FRANTZ as modified by MINNE and JOSHI further disclose wherein the image data have been taken by the imaging device while the imaging device was moving relative to the object marker arrangement (see FRANTZ: e.g., -- the at least four markers 21 are arranged in an asymmetric way on the object 20. This helps to retrieve the correct orientation of the object from the 3D positions of the markers 21 of the set of markers 21. The spatial/local relationship/arrangement between the markers 21 of the set of markers 21 arranged on the object 20 is known to the HMD 21 or stored in the HMD 21….[0037] The markers 21 within a set of markers 21 are preferably labelled. That is that each marker 21 of the set of markers 21 has a different label. A label is an identifier for identifying the marker 21. The identifier or label is preferably just virtual, i.e. is not visible from the image of the marker 21 in the IR image of the IR camera 12. The label of each marker 21 can be determined based on the 3D spatial relationship of the set of markers 21. A set of four markers 21 can for example be labelled from 1 to 4.--, in [0036]-[0037]; and, -- FIG. 4A shows an exemplary IR image taken by the IR camera 12. The IR image is continuously, preferably periodically taken/received. Preferably, the IR image is taken/received with an IR image sampling rate. Preferably, the IR image sampling rate is equal to the object pose detection sampling rate so that for each IR image obtained by the IR camera 12, the object pose detection step S12 is performed.--, in [0050]-[0052], and, --[0055] In step S122, the pose of the object in the HMD coordinate system is determined/computed based on the markers 21 detected in the IR image, more precisely based on the 2D positions of the markers 21 in the IR image. Preferably, the pose of the object in the HMD coordinate system is determined/computed based on the markers 21 detected in the IR image and based on the known spatial relationship of the markers 21 in the set of markers 21 arranged on the object 20. Preferably, the pose of the object 20 is determined based on the 3D positions of the markers 21. The 3D positions of the markers 21 (or the pose of the object) in the HMD coordinate system are(is) determined based on the 2D positions of the markers 21 in the IR image. Preferably, in this step S122, the IR sensor coordinate system is used as HMD coordinate system. Preferably, the 3D positions of the markers 21 in the HMD coordinate system are determined based on the 2D positions of the markers 21 in the IR image and based on the known camera parameters of the IR camera 12. In the following, one preferred way to detect the pose of the object in the HMD coordinate system is determined/computed based on the markers 21 detected in the IR image.--, in [0055]).
Re Claim 4, FRANTZ as modified by MINNE and JOSHI further disclose wherein the imaging device is a video camera capturing video data comprising the image data (see MINNE: e.g., --a stereoscopic video (or separate two-dimensional left and right videos) of a target surgical site imaged by the stereoscopic camera 130.--, in [0109], and, --This image acquisition module may generate surgical site image data stream 410, which may be communicated to microscope processing unit 420 and the associated surgical site image processing module 430. Images may be captured and processed at a frame rate high enough to be perceived as video by the user, for example, 60 frames per second (fps.). Thus, images may be considered to be “image data stream.” It is to be understood that, where a two-camera stereoscopic digital surgical microscope is described, the concept may be extendible to an N-camera digital surgical microscope where N is 2 or greater.--, in [0137]).
Re Claim 5, FRANTZ as modified by MINNE and JOSHI further disclose wherein the step of determining the position of the individual object marker comprises determining, based on at least a first image and a second image of the plurality of images that contain the reference pattern and the individual object marker, a position of the individual object marker relative to the reference pattern (see JOSHI: e.g., Fig. 5, Fig. 15, and Fig. 17, Fig. 18 and Fig. 19, -- The reference element arrays are typically small in size (e.g., on a few centimeters wide) to minimize obstruction of the surgical area. The number of markers is also usually limited to optimize costs and workflow. A larger array with more markers can improve the accuracy of divot position.--, in [0007], and, -- include obtaining a first image of the tracked instrument while it is positioned at the first physical position between the emitter and the detector. The operations further include determining a second virtual position within the virtual space of the emitter of the imaging device. The operations further include determining a second virtual position within the virtual space of the detector of the imaging device. The operations further include determining a second virtual position within the virtual space of the tracked instrument while the tracked instrument is at a second physical position between the emitter and the detector. The operations further include determining a second expected image of the tracked instrument based on the second virtual position of the emitter, the second virtual position of the detector, and the second virtual position of the tracked instrument. The operations further include obtaining a second image of the tracked instrument while it is positioned between the emitter and the detector, the second image being different than the first image. The operations further include determining whether the tracked instrument is accurate based on the first expected image, the second expected image, the first image, and the second image.--, [0012]).
Re Claim 6, FRANTZ as modified by MINNE and JOSHI further disclose wherein another one of the individual object markers is not contained in at least one of the first image and the second image, and further comprising determining, based on at least a third image and one of the first, the second and a fourth image of the plurality of images that contain the reference pattern and the other one of the object markers, a position of the other one of the object markers relative to the reference pattern (see FRANTZ: e.g., Fig. 2, and, -- [0037] The markers 21 within a set of markers 21 are preferably labelled. That is that each marker 21 of the set of markers 21 has a different label. A label is an identifier for identifying the marker 21. The identifier or label is preferably just virtual, i.e. is not visible from the image of the marker 21 in the IR image of the IR camera 12. The label of each marker 21 can be determined based on the 3D spatial relationship of the set of markers 21. A set of four markers 21 can for example be labelled from 1 to 4. [0038] The system can also comprise two or more objects 20 to be tracked, each having a (different) set of markers 21 arranged on it. For example, the first object can be a patient locator fixed on the patient to detect the pose of the patient (or a part of the patient like a certain bone) having a first set of markers 21 arranged on it as shown for example in FIG. 2. For example, the second object can be a surgical tool having a second set of markers 21 on it. The second set of markers 21 has a different spatial arrangement/relationship of markers 21 compared to the first set of markers 21 so that the two objects can be distinguished in the same IR image by the spatial relationship of their markers 21. Thus, when the HMD 10 can track both objects 20 together, it is possible to track the relative position of the second object or the surgical tool with respect to the first object or the patient.--, in [0036]-[0038], and, -- [0054] In step S121, the markers 21 are detected in the IR image. The markers 21 detected in the IR image in step S121 can be described as candidate markers 21 which fulfill certain requirements to be considered as candidate marker 21. The requirements can depend on the luminosity (above a certain threshold), the size and/or the form of the (candidate) marker 21 in the IR image. In an optimal case, the same number of markers 21 are detected in the IR image as the set of markers 21 of the object 20. However, it is also possible that a lower number of markers 21 are detected, if some of the markers 21 are occluded e.g. by the specific pose of the object 20 itself or by another object in front of the object 20 to be detected. It is further possible that a higher number of (candidate) markers 21 is detected, if something else than the IR object marker 21 of the set of markers 21 emits or reflects IR light which leads to a false positive candidate marker 21 or if more than one object 20 is detected in the IR image. Preferably, the (candidate) markers 21 in the IR image and their 2D positions in the IR image are detected in step S121. The IR image has preferably a 2D coordinate system expressed in pixels. A first dimension extends in the image plane in a first direction, often called the x-axis or the row direction. A second dimension extends in the image plane in a second direction, often called the y-axis or the column direction. A 2D position of the IR image is a point in the IR image with two coordinates. The 2D position can also be expressed with a sub-pixel precision. Since the markers 21 emit or better reflect IR light and the IR image detect the intensity of IR light in the field of view, the markers 21 have a high IR intensity. So, one possible way to detect the markers 21 in the IR image is to detect all pixels above a certain IR light intensity as points of the markers 21.--, in [0054]).
Re Claim 7, FRANTZ as modified by MINNE and JOSHI further disclose wherein at least one of the object markers has a rotational-symmetric shape (see FRANTZ: e.g., --uses only a predetermined (optimal) sub-set of three IR object markers to detect the correspondence between the IR object markers in the IR image and the IR object markers of the set of IR object markers.--, in [0013]-[0014], Fig. 4B, --the IR sensor 12 is configured to detect light in the spectral range in which the object markers emit light and/or in which an IR light source of the HMD 10 emits light…to detect IR reflective object markers by illuminating them with the IR light source 13 and detecting in the IR image taken by the IR sensor 12 the object markers reflecting the IR light from the IR source 13. The IR light source emits the IR light also in the direction of view of the HMD 10. In a less preferred embodiment, it would also be possible to arrange the IR light source 13 in the environment to illuminate the markers 21. If the markers 21 emit actively IR light, the IR light source 13 can be completely avoided.--, in [0031],and, -- the at least four markers 21 are arranged in an asymmetric way on the object 20. This helps to retrieve the correct orientation of the object from the 3D positions of the markers 21 of the set of markers 21. The spatial/local relationship/arrangement between the markers 21 of the set of markers 21 arranged on the object 20 is known to the HMD 21 or stored in the HMD 21….[0037] The markers 21 within a set of markers 21 are preferably labelled. That is that each marker 21 of the set of markers 21 has a different label. A label is an identifier for identifying the marker 21. The identifier or label is preferably just virtual, i.e. is not visible from the image of the marker 21 in the IR image of the IR camera 12. The label of each marker 21 can be determined based on the 3D spatial relationship of the set of markers 21. A set of four markers 21 can for example be labelled from 1 to 4.--, in [0036]-[0037]; and, --[0055] In step S122, the pose of the object in the HMD coordinate system is determined/computed based on the markers 21 detected in the IR image, more precisely based on the 2D positions of the markers 21 in the IR image. Preferably, the pose of the object in the HMD coordinate system is determined/computed based on the markers 21 detected in the IR image and based on the known spatial relationship of the markers 21 in the set of markers 21 arranged on the object 20. Preferably, the pose of the object 20 is determined based on the 3D positions of the markers 21. The 3D positions of the markers 21 (or the pose of the object) in the HMD coordinate system are(is) determined based on the 2D positions of the markers 21 in the IR image. Preferably, in this step S122, the IR sensor coordinate system is used as HMD coordinate system. Preferably, the 3D positions of the markers 21 in the HMD coordinate system are determined based on the 2D positions of the markers 21 in the IR image and based on the known camera parameters of the IR camera 12. In the following, one preferred way to detect the pose of the object in the HMD coordinate system is determined/computed based on the markers 21 detected in the IR image.--, in [0055]).
Re Claim 8, FRANTZ as modified by MINNE and JOSHI further disclose wherein the object marker arrangement is determined in an object marker coordinate system, and wherein an origin of the object marker coordinate system has a predefined geometric relationship to one of the object markers (see FRANTZ: e.g., --uses only a predetermined (optimal) sub-set of three IR object markers to detect the correspondence between the IR object markers in the IR image and the IR object markers of the set of IR object markers.--, in [0013]-[0014], Fig. 4B, --the IR sensor 12 is configured to detect light in the spectral range in which the object markers emit light and/or in which an IR light source of the HMD 10 emits light…to detect IR reflective object markers by illuminating them with the IR light source 13 and detecting in the IR image taken by the IR sensor 12 the object markers reflecting the IR light from the IR source 13. The IR light source emits the IR light also in the direction of view of the HMD 10. In a less preferred embodiment, it would also be possible to arrange the IR light source 13 in the environment to illuminate the markers 21. If the markers 21 emit actively IR light, the IR light source 13 can be completely avoided.--, in [0031],and, -- the at least four markers 21 are arranged in an asymmetric way on the object 20. This helps to retrieve the correct orientation of the object from the 3D positions of the markers 21 of the set of markers 21. The spatial/local relationship/arrangement between the markers 21 of the set of markers 21 arranged on the object 20 is known to the HMD 21 or stored in the HMD 21….[0037] The markers 21 within a set of markers 21 are preferably labelled. That is that each marker 21 of the set of markers 21 has a different label. A label is an identifier for identifying the marker 21. The identifier or label is preferably just virtual, i.e. is not visible from the image of the marker 21 in the IR image of the IR camera 12. The label of each marker 21 can be determined based on the 3D spatial relationship of the set of markers 21. A set of four markers 21 can for example be labelled from 1 to 4.--, in [0036]-[0037]; and, --[0055] In step S122, the pose of the object in the HMD coordinate system is determined/computed based on the markers 21 detected in the IR image, more precisely based on the 2D positions of the markers 21 in the IR image. Preferably, the pose of the object in the HMD coordinate system is determined/computed based on the markers 21 detected in the IR image and based on the known spatial relationship of the markers 21 in the set of markers 21 arranged on the object 20. Preferably, the pose of the object 20 is determined based on the 3D positions of the markers 21. The 3D positions of the markers 21 (or the pose of the object) in the HMD coordinate system are(is) determined based on the 2D positions of the markers 21 in the IR image. Preferably, in this step S122, the IR sensor coordinate system is used as HMD coordinate system. Preferably, the 3D positions of the markers 21 in the HMD coordinate system are determined based on the 2D positions of the markers 21 in the IR image and based on the known camera parameters of the IR camera 12. In the following, one preferred way to detect the pose of the object in the HMD coordinate system is determined/computed based on the markers 21 detected in the IR image.--, in [0055]).
Re Claim 9, FRANTZ as modified by MINNE and JOSHI further disclose herein at least one of the object 15 markers comprises a reflective material configured to reflect light of at least one of the visible and infrared spectrum (see FRANTZ: e.g., --uses only a predetermined (optimal) sub-set of three IR object markers to detect the correspondence between the IR object markers in the IR image and the IR object markers of the set of IR object markers.--, in [0013]-[0014], Fig. 4B, --the IR sensor 12 is configured to detect light in the spectral range in which the object markers emit light and/or in which an IR light source of the HMD 10 emits light…to detect IR reflective object markers by illuminating them with the IR light source 13 and detecting in the IR image taken by the IR sensor 12 the object markers reflecting the IR light from the IR source 13. The IR light source emits the IR light also in the direction of view of the HMD 10. In a less preferred embodiment, it would also be possible to arrange the IR light source 13 in the environment to illuminate the markers 21. If the markers 21 emit actively IR light, the IR light source 13 can be completely avoided.--, in [0031],and, -- the at least four markers 21 are arranged in an asymmetric way on the object 20. This helps to retrieve the correct orientation of the object from the 3D positions of the markers 21 of the set of markers 21. The spatial/local relationship/arrangement between the markers 21 of the set of markers 21 arranged on the object 20 is known to the HMD 21 or stored in the HMD 21….[0037] The markers 21 within a set of markers 21 are preferably labelled. That is that each marker 21 of the set of markers 21 has a different label. A label is an identifier for identifying the marker 21. The identifier or label is preferably just virtual, i.e. is not visible from the image of the marker 21 in the IR image of the IR camera 12. The label of each marker 21 can be determined based on the 3D spatial relationship of the set of markers 21. A set of four markers 21 can for example be labelled from 1 to 4.--, in [0036]-[0037]; and, --[0055] In step S122, the pose of the object in the HMD coordinate system is determined/computed based on the markers 21 detected in the IR image, more precisely based on the 2D positions of the markers 21 in the IR image. Preferably, the pose of the object in the HMD coordinate system is determined/computed based on the markers 21 detected in the IR image and based on the known spatial relationship of the markers 21 in the set of markers 21 arranged on the object 20. Preferably, the pose of the object 20 is determined based on the 3D positions of the markers 21. The 3D positions of the markers 21 (or the pose of the object) in the HMD coordinate system are(is) determined based on the 2D positions of the markers 21 in the IR image. Preferably, in this step S122, the IR sensor coordinate system is used as HMD coordinate system. Preferably, the 3D positions of the markers 21 in the HMD coordinate system are determined based on the 2D positions of the markers 21 in the IR image and based on the known camera parameters of the IR camera 12. In the following, one preferred way to detect the pose of the object in the HMD coordinate system is determined/computed based on the markers 21 detected in the IR image.--, in [0055]).
Re Claim 10, FRANTZ as modified by MINNE and JOSHI further disclose removing the reference device from the fixed spatial relationship with the object markers; and tracking the object based on the object marker arrangement after the reference device has been removed (see FRANTZ: e.g., Fig. 2, and, -- [0037] The markers 21 within a set of markers 21 are preferably labelled. That is that each marker 21 of the set of markers 21 has a different label. A label is an identifier for identifying the marker 21. The identifier or label is preferably just virtual, i.e. is not visible from the image of the marker 21 in the IR image of the IR camera 12. The label of each marker 21 can be determined based on the 3D spatial relationship of the set of markers 21. A set of four markers 21 can for example be labelled from 1 to 4. [0038] The system can also comprise two or more objects 20 to be tracked, each having a (different) set of markers 21 arranged on it. For example, the first object can be a patient locator fixed on the patient to detect the pose of the patient (or a part of the patient like a certain bone) having a first set of markers 21 arranged on it as shown for example in FIG. 2. For example, the second object can be a surgical tool having a second set of markers 21 on it. The second set of markers 21 has a different spatial arrangement/relationship of markers 21 compared to the first set of markers 21 so that the two objects can be distinguished in the same IR image by the spatial relationship of their markers 21. Thus, when the HMD 10 can track both objects 20 together, it is possible to track the relative position of the second object or the surgical tool with respect to the first object or the patient.--, in [0036]-[0038], and, -- [0054] In step S121, the markers 21 are detected in the IR image. The markers 21 detected in the IR image in step S121 can be described as candidate markers 21 which fulfill certain requirements to be considered as candidate marker 21. The requirements can depend on the luminosity (above a certain threshold), the size and/or the form of the (candidate) marker 21 in the IR image. In an optimal case, the same number of markers 21 are detected in the IR image as the set of markers 21 of the object 20. However, it is also possible that a lower number of markers 21 are detected, if some of the markers 21 are occluded e.g. by the specific pose of the object 20 itself or by another object in front of the object 20 to be detected. It is further possible that a higher number of (candidate) markers 21 is detected, if something else than the IR object marker 21 of the set of markers 21 emits or reflects IR light which leads to a false positive candidate marker 21 or if more than one object 20 is detected in the IR image. Preferably, the (candidate) markers 21 in the IR image and their 2D positions in the IR image are detected in step S121. The IR image has preferably a 2D coordinate system expressed in pixels. A first dimension extends in the image plane in a first direction, often called the x-axis or the row direction. A second dimension extends in the image plane in a second direction, often called the y-axis or the column direction. A 2D position of the IR image is a point in the IR image with two coordinates. The 2D position can also be expressed with a sub-pixel precision. Since the markers 21 emit or better reflect IR light and the IR image detect the intensity of IR light in the field of view, the markers 21 have a high IR intensity. So, one possible way to detect the markers 21 in the IR image is to detect all pixels above a certain IR light intensity as points of the markers 21.--, in [0054]).
Re Claim 11, FRANTZ as modified by MINNE and JOSHI further disclose receiving tracking image data representative of the object markers; and tracking the object based on the tracking image data and information about the object marker arrangement (see FRANTZ: e.g., Fig. 2, and, -- [0037] The markers 21 within a set of markers 21 are preferably labelled. That is that each marker 21 of the set of markers 21 has a different label. A label is an identifier for identifying the marker 21. The identifier or label is preferably just virtual, i.e. is not visible from the image of the marker 21 in the IR image of the IR camera 12. The label of each marker 21 can be determined based on the 3D spatial relationship of the set of markers 21. A set of four markers 21 can for example be labelled from 1 to 4. [0038] The system can also comprise two or more objects 20 to be tracked, each having a (different) set of markers 21 arranged on it. For example, the first object can be a patient locator fixed on the patient to detect the pose of the patient (or a part of the patient like a certain bone) having a first set of markers 21 arranged on it as shown for example in FIG. 2. For example, the second object can be a surgical tool having a second set of markers 21 on it. The second set of markers 21 has a different spatial arrangement/relationship of markers 21 compared to the first set of markers 21 so that the two objects can be distinguished in the same IR image by the spatial relationship of their markers 21. Thus, when the HMD 10 can track both objects 20 together, it is possible to track the relative position of the second object or the surgical tool with respect to the first object or the patient.--, in [0036]-[0038], and, -- [0054] In step S121, the markers 21 are detected in the IR image. The markers 21 detected in the IR image in step S121 can be described as candidate markers 21 which fulfill certain requirements to be considered as candidate marker 21. The requirements can depend on the luminosity (above a certain threshold), the size and/or the form of the (candidate) marker 21 in the IR image. In an optimal case, the same number of markers 21 are detected in the IR image as the set of markers 21 of the object 20. However, it is also possible that a lower number of markers 21 are detected, if some of the markers 21 are occluded e.g. by the specific pose of the object 20 itself or by another object in front of the object 20 to be detected. It is further possible that a higher number of (candidate) markers 21 is detected, if something else than the IR object marker 21 of the set of markers 21 emits or reflects IR light which leads to a false positive candidate marker 21 or if more than one object 20 is detected in the IR image. Preferably, the (candidate) markers 21 in the IR image and their 2D positions in the IR image are detected in step S121. The IR image has preferably a 2D coordinate system expressed in pixels. A first dimension extends in the image plane in a first direction, often called the x-axis or the row direction. A second dimension extends in the image plane in a second direction, often called the y-axis or the column direction. A 2D position of the IR image is a point in the IR image with two coordinates. The 2D position can also be expressed with a sub-pixel precision. Since the markers 21 emit or better reflect IR light and the IR image detect the intensity of IR light in the field of view, the markers 21 have a high IR intensity. So, one possible way to detect the markers 21 in the IR image is to detect all pixels above a certain IR light intensity as points of the markers 21.--, in [0054]).
Re Claim 12, FRANTZ as modified by MINNE and JOSHI further disclose selecting the position of one of the object markers as a reference position for an object marker coordinate system and transforming the positions of the other object markers in the object marker coordinate system (see Minne: e.g. Fig. 2, --, a LOCALIZER_T_CALIBRATION-DEVICE transformation 912a, a LOCALIZER_T_CAMERA-TARGET transformation 914, patient anatomy 916, a digital surgical microscope (“DSM”) CAMERA TARGET 920 (and associated reference frame), a DSM CAMERA TARGET 920a, 920b, and 920c (and associated reference frame) during camera calibration (a plurality of poses), a CAMERA OPTICAL MODEL REFERENCE FRAME viewing patient anatomy 922, a CAMERA OPTICAL MODEL REFERENCE FRAME during camera calibration (a plurality of poses) 922a, 922b, and 922c, and a LOCALIZER REFERENCE FRAME 924.--, in [0045]; and
--A computer vision readable target 954 on the calibration device is detected in the acquired or received images and the locations of its salient features located relative to the photogrammetry reference frame 960 which salient features describe the X and Y axis of a reference frame 956 the Z axis of which reference frame--, in [0061], --To track a tool of interest (such as a clamp holding patient anatomy) the systems disclosed herein may include a trackable target mounted on a tool. The trackable target might consist of a geometric arrangement of markers such as retroreflective spheres capable of reflecting infrared (“IR”) light, or of a similar arrangement of IR light emitters. These are detected by a localizer device 105 with an IR-detecting camera, and when required an (IR) light source to reflect off the retroreflectors and/or to stimulate a passively-synchronized IR emitter. Actively-synchronized IR emitters also exist. Alternatively the trackable target might consist of computer-vision readable targets created from high-contrast patterns, or by patterns that reflect or emit light in various regions of the electromagnetic spectrum.--, in [0067]; and, --[0071] The locations of the navigation trackable markers 1002, 1004, 1005 and 1006 in the reference frame of the bracket 1000 are known by design and/or by measurement and together compose a tool described by the markers' geometric relation to each other in that reference frame. This information is stored in a tool description file and used by the navigation computer 170 to discern which tool is in view of the localizer 105 and in what pose relative to the localizer's reference frame 924.
[0072] The locations of the computer-vision readable targets 1008, 1010 and 1012 are also known by design and/or by measurement, in the same reference frame as the navigation markers, namely that of the bracket. Such CV-ready targets are detectable as unique regardless of viewing angle, and encode a known origin for example the physical center of the pattern. Other embodiments may loosen such restrictions for example by allowing viewing-angle redundant individual targets as long as the group of targets is detectable as unique regardless of viewing angle.--, in [0071]-[0072]; also see: --[0236] Multiple patterns of varying sizes may be optionally used to provide accurate calibration over a wide zoom range.
[0237] Traditional camera calibration can also provide a measure of the optical distortion of the system at the optical parameter settings at which the calibration process was performed. A set of distortion coefficients can be found and can be used in some embodiments to correct such optical distortion. In some embodiments, such distortion correction can be used to improve the field of view calibration method. Furthermore, in some embodiments, such distortion correction can be used to improve the accuracy of the overlay (e.g., how it matches the live view.)
[0238] In embodiments where an explicit field of view calibration process may be used to improve on the field of view determination for the projection matrix of the computer graphics renderer, the distance to the focal surface of each camera eye of the stereoscopic digital surgical microscope may be required to be calculated. The determination of this distance for each camera eye will be discussed herein, in relation to FIG. 21.
[0239] FIG. 20 is a diagram showing an angle of view applicable to the integrated surgical navigation and visualization system, according to an example embodiment of the present disclosure. With the focus distance, the angle of view can be calculated. This angle may be needed to calculate terms in the projection matrix and can be found by trigonometry, as shown in FIG. 20:
[0240] For example, the half angle 2600 can be found by measuring the focus distance 2610 from the camera center of projection (also referred to as the camera “eye point”) 2620 to the focus surface 2630 along the optical axis 2640. The additional field of view calibration can provide a measure of the field of view (for example the horizontal width) at the focus surface. The half of such distance is shown as marker 2650. The tangent of half angle 2600 is distance 2650 divided by distance 2640. The inverse tangent function can then be used to calculate the “half field of view angle.” The half field of view angle can be used to calculate directly certain matrix elements of the combined projection matrix--, in [0236]-[0240]).
Re Claim 13, FRANTZ as modified by MINNE and JOSHI further disclose removing the reference device from the fixed spatial relationship with the object markers; tracking the object based on the object marker arrangement after the reference device has been removed; and tracking the object based on the object marker coordinate system (see FRANTZ: e.g., Fig. 2, and, -- [0037] The markers 21 within a set of markers 21 are preferably labelled. That is that each marker 21 of the set of markers 21 has a different label. A label is an identifier for identifying the marker 21. The identifier or label is preferably just virtual, i.e. is not visible from the image of the marker 21 in the IR image of the IR camera 12. The label of each marker 21 can be determined based on the 3D spatial relationship of the set of markers 21. A set of four markers 21 can for example be labelled from 1 to 4. [0038] The system can also comprise two or more objects 20 to be tracked, each having a (different) set of markers 21 arranged on it. For example, the first object can be a patient locator fixed on the patient to detect the pose of the patient (or a part of the patient like a certain bone) having a first set of markers 21 arranged on it as shown for example in FIG. 2. For example, the second object can be a surgical tool having a second set of markers 21 on it. The second set of markers 21 has a different spatial arrangement/relationship of markers 21 compared to the first set of markers 21 so that the two objects can be distinguished in the same IR image by the spatial relationship of their markers 21. Thus, when the HMD 10 can track both objects 20 together, it is possible to track the relative position of the second object or the surgical tool with respect to the first object or the patient.--, in [0036]-[0038], and, -- [0054] In step S121, the markers 21 are detected in the IR image. The markers 21 detected in the IR image in step S121 can be described as candidate markers 21 which fulfill certain requirements to be considered as candidate marker 21. The requirements can depend on the luminosity (above a certain threshold), the size and/or the form of the (candidate) marker 21 in the IR image. In an optimal case, the same number of markers 21 are detected in the IR image as the set of markers 21 of the object 20. However, it is also possible that a lower number of markers 21 are detected, if some of the markers 21 are occluded e.g. by the specific pose of the object 20 itself or by another object in front of the object 20 to be detected. It is further possible that a higher number of (candidate) markers 21 is detected, if something else than the IR object marker 21 of the set of markers 21 emits or reflects IR light which leads to a false positive candidate marker 21 or if more than one object 20 is detected in the IR image. Preferably, the (candidate) markers 21 in the IR image and their 2D positions in the IR image are detected in step S121. The IR image has preferably a 2D coordinate system expressed in pixels. A first dimension extends in the image plane in a first direction, often called the x-axis or the row direction. A second dimension extends in the image plane in a second direction, often called the y-axis or the column direction. A 2D position of the IR image is a point in the IR image with two coordinates. The 2D position can also be expressed with a sub-pixel precision. Since the markers 21 emit or better reflect IR light and the IR image detect the intensity of IR light in the field of view, the markers 21 have a high IR intensity. So, one possible way to detect the markers 21 in the IR image is to detect all pixels above a certain IR light intensity as points of the markers 21.--, in [0054]).
Re Claim 14, FRANTZ as modified by MINNE and JOSHI further disclose eceiving tracking image data representative of the object markers; tracking the object based on the tracking image data and information about the object marker arrangement; and tracking the object based on the object marker coordinate system (see FRANTZ: e.g., Fig. 2, and, -- [0037] The markers 21 within a set of markers 21 are preferably labelled. That is that each marker 21 of the set of markers 21 has a different label. A label is an identifier for identifying the marker 21. The identifier or label is preferably just virtual, i.e. is not visible from the image of the marker 21 in the IR image of the IR camera 12. The label of each marker 21 can be determined based on the 3D spatial relationship of the set of markers 21. A set of four markers 21 can for example be labelled from 1 to 4. [0038] The system can also comprise two or more objects 20 to be tracked, each having a (different) set of markers 21 arranged on it. For example, the first object can be a patient locator fixed on the patient to detect the pose of the patient (or a part of the patient like a certain bone) having a first set of markers 21 arranged on it as shown for example in FIG. 2. For example, the second object can be a surgical tool having a second set of markers 21 on it. The second set of markers 21 has a different spatial arrangement/relationship of markers 21 compared to the first set of markers 21 so that the two objects can be distinguished in the same IR image by the spatial relationship of their markers 21. Thus, when the HMD 10 can track both objects 20 together, it is possible to track the relative position of the second object or the surgical tool with respect to the first object or the patient.--, in [0036]-[0038], and, -- [0054] In step S121, the markers 21 are detected in the IR image. The markers 21 detected in the IR image in step S121 can be described as candidate markers 21 which fulfill certain requirements to be considered as candidate marker 21. The requirements can depend on the luminosity (above a certain threshold), the size and/or the form of the (candidate) marker 21 in the IR image. In an optimal case, the same number of markers 21 are detected in the IR image as the set of markers 21 of the object 20. However, it is also possible that a lower number of markers 21 are detected, if some of the markers 21 are occluded e.g. by the specific pose of the object 20 itself or by another object in front of the object 20 to be detected. It is further possible that a higher number of (candidate) markers 21 is detected, if something else than the IR object marker 21 of the set of markers 21 emits or reflects IR light which leads to a false positive candidate marker 21 or if more than one object 20 is detected in the IR image. Preferably, the (candidate) markers 21 in the IR image and their 2D positions in the IR image are detected in step S121. The IR image has preferably a 2D coordinate system expressed in pixels. A first dimension extends in the image plane in a first direction, often called the x-axis or the row direction. A second dimension extends in the image plane in a second direction, often called the y-axis or the column direction. A 2D position of the IR image is a point in the IR image with two coordinates. The 2D position can also be expressed with a sub-pixel precision. Since the markers 21 emit or better reflect IR light and the IR image detect the intensity of IR light in the field of view, the markers 21 have a high IR intensity. So, one possible way to detect the markers 21 in the IR image is to detect all pixels above a certain IR light intensity as points of the markers 21.--, in [0054]).
Re Claim 15, FRANTZ as modified by MINNE and JOSHI further disclose registering a geometrical attribute of the object with the object marker arrangement or an object marker coordinate system (see FRANTZ: e.g., -- The set of markers 21 comprises at least four markers 21. Preferably, the at least four markers 21 are arranged in an asymmetric way on the object 20. This helps to retrieve the correct orientation of the object from the 3D positions of the markers 21 of the set of markers 21. The spatial/local relationship/arrangement between the markers 21 of the set of markers 21 arranged on the object 20 is known to the HMD 21 or stored in the HMD 21. This can be achieved by pre-stored spatial relationships of sets of markers 21 for standard objects 20. This can also be achieved by registering an object 20 as will be explained in more detail below. The object 20 is preferably a surgical tool like a drill, a catheter, an implant, etc. The surgical tool can be for example a surgical tool which is applied mainly in one direction of application along an application axis of the surgical tool. This can be a drill, a screwdriver, etc. FIG. 2 shows a patient locator as an example object 20 which is fixed on a part of the patient to detect with the HMD 20 the pose of the patient. The set of markers 21 are arranged in a fixed position on the object 20 so that the spatial relationship between the markers 21 of the set of markers 21 on the object 20 cannot change. Different objects 20 have preferably different spatial relationships between their markers 21 and/or different number of markers 21 so that the HMD 20 can detect from the spatial relationship and/or number of the markers 21 the (type of) object 20. The object 20 can also be part of the augmented reality system.--, in [0036], and, -- the correspondence detection algorithm relates each marker 21 detected in the IR image to a corresponding marker 21 of the object to be detected and/or of the set of markers 21. Thus, it identifies for each detected marker 21 the corresponding label of the marker 21 of the set of markers 21 of the object 20. Once, the correspondences of the markers 21 in the 2D image with respect to the corresponding markers 21 of the set of markers 21 of the object 20 are known, the exact 3D positions of all markers 21 in the HMD coordinate system (more precisely in the IR sensor coordinate system) can be calculated by the pose solver 116 based on their 2D positions in the IR image, their correspondences, the spatial relationship of the set of markers 21 of the registered object 114 and based on the camera model 112. The camera model 112 and the spatial relationship of the set of markers 21 (registered object 114) are stored in the HMD 10 to perform the described computations. Normally, the correspondences of at least 4 (candidate) markers 21 in the IR image must be detected to find a solution in the pose solver 116.--, in [0056], [0074]; also see MINNE: e.g., --registering the photogrammetry frame to a navigation frame (also called a “patient target reference frame”), which patient target reference frame is trackable by a surgical navigation system tracking camera such as an NDI Polaris Vega™ localizer camera. It is understood that the disclosed system is adaptable to use any existing surgical navigation systems.--, in [0041], and [0051], and, --0059] The patient data 910 is then registered to the patient anatomy 916 represented in the photogrammetry model 926 (block 5316). This amounts to determining the transformation 946 between the two reference frames 926 and 928. This transformation 946 is called PHOTOGRAMMETRY_T_PATIENT-DATA.--, in [0059], and, -- When feasible, patient anatomy features are used as navigation targets, which further eliminates the need for the determination of the transformation PATIENT-TARGET_T_PATIENT-DATA since the features are already coincident. When this is not feasible, fiducials or other targets are added to the patient anatomy and registered--, in [0079]).
Re Claim 16, FRANTZ as modified by MINNE and JOSHI further disclose manually arranging the object markers on the object at not-predefined positions before
capturing the plurality of images (see FRANTZ: e.g., Fig. 2, and, -- [0037] The markers 21 within a set of markers 21 are preferably labelled. That is that each marker 21 of the set of markers 21 has a different label. A label is an identifier for identifying the marker 21. The identifier or label is preferably just virtual, i.e. is not visible from the image of the marker 21 in the IR image of the IR camera 12. The label of each marker 21 can be determined based on the 3D spatial relationship of the set of markers 21. A set of four markers 21 can for example be labelled from 1 to 4. [0038] The system can also comprise two or more objects 20 to be tracked, each having a (different) set of markers 21 arranged on it. For example, the first object can be a patient locator fixed on the patient to detect the pose of the patient (or a part of the patient like a certain bone) having a first set of markers 21 arranged on it as shown for example in FIG. 2. For example, the second object can be a surgical tool having a second set of markers 21 on it. The second set of markers 21 has a different spatial arrangement/relationship of markers 21 compared to the first set of markers 21 so that the two objects can be distinguished in the same IR image by the spatial relationship of their markers 21. Thus, when the HMD 10 can track both objects 20 together, it is possible to track the relative position of the second object or the surgical tool with respect to the first object or the patient.--, in [0036]-[0038], and, -- [0054] In step S121, the markers 21 are detected in the IR image. The markers 21 detected in the IR image in step S121 can be described as candidate markers 21 which fulfill certain requirements to be considered as candidate marker 21. The requirements can depend on the luminosity (above a certain threshold), the size and/or the form of the (candidate) marker 21 in the IR image. In an optimal case, the same number of markers 21 are detected in the IR image as the set of markers 21 of the object 20. However, it is also possible that a lower number of markers 21 are detected, if some of the markers 21 are occluded e.g. by the specific pose of the object 20 itself or by another object in front of the object 20 to be detected. It is further possible that a higher number of (candidate) markers 21 is detected, if something else than the IR object marker 21 of the set of markers 21 emits or reflects IR light which leads to a false positive candidate marker 21 or if more than one object 20 is detected in the IR image. Preferably, the (candidate) markers 21 in the IR image and their 2D positions in the IR image are detected in step S121. The IR image has preferably a 2D coordinate system expressed in pixels. A first dimension extends in the image plane in a first direction, often called the x-axis or the row direction. A second dimension extends in the image plane in a second direction, often called the y-axis or the column direction. A 2D position of the IR image is a point in the IR image with two coordinates. The 2D position can also be expressed with a sub-pixel precision. Since the markers 21 emit or better reflect IR light and the IR image detect the intensity of IR light in the field of view, the markers 21 have a high IR intensity. So, one possible way to detect the markers 21 in the IR image is to detect all pixels above a certain IR light intensity as points of the markers 21.--, in [0054]).
Re Claim 17, FRANTZ as modified by MINNE and JOSHI further disclose wherein each object marker is arranged via an adhesive or a magnetic force on the object (see Frantz: e.g., --a set of IR object markers attached in a known spatial relationship to an object to be detected--, claim 1; also see Joshi: e.g., -- the surveillance marker 500 is affixed to the patient to provide information on whether the patient reference array 116 has shifted. For example, during a spinal fusion procedure with planned placement of pedicle screw fixation, two small incisions are made over the posterior superior iliac spine bilaterally. The DRB and the surveillance marker are then affixed to the posterior superior iliac spine bilaterally. If the surveillance marker's 500 location changes relative to the patient reference array 116, the camera tracking system 200 may display a meter indicating the amount of movement and/or may display a pop-up warning message to inform the user that the patient reference array may have been bumped. If the patient reference array has indeed been bumped, the registration of the patient reference array to the tracked coordinate system may be invalid and could result in erroneous navigation which is off target.--, in [0046]).
Re Claim 18, claim 18 is corresponding medium claim to claim 1, respectively. Claim 18 thus is rejected for the similar reasons for claim 1. See above discussions with regard to claim 1 respectively. FRANTZ as modified by MINNE and JOSHI further disclose further disclose a non-transitory computer-readable medium storing instructions that, when executed on at least one processor, cause the at least one processor to carry out the method (see FRANTZ: e.g., -- [0029] The processing means 11 is configured to execute the subsequently described methods/algorithms according to the invention. The processing means 11 comprises preferably a general processing means like a CPU or programmable microprocessor. Preferably, the processing means 11 comprises a programmable processor. Preferably, the subsequently described functions/methods/algorithms of the invention are realized as software programs executed on the processing means 11 or in the processor…. The processing means 11 or the HMD 10 comprises preferably a storage for storing the software programs according to the invention to be executed on the HMD 10. The storage stores preferably further a library of objects and associated to each object the spatial relationship of the set of object markers 21 as arranged on the object 20.--, in [0029]).
Re Claims 19-20, claims 19-20 are corresponding processing device claim to claims 1-2, respectively. Claims 19-20 thus are rejected for the similar reasons for claims 1-2. See above discussions with regard to claims 1-2 respectively. FRANTZ as modified by MINNE and JOSHI further disclose further disclose processing device for determining an object marker arrangement to carry out the method (see FRANTZ: e.g., -- [0029] The processing means 11 is configured to execute the subsequently described methods/algorithms according to the invention. The processing means 11 comprises preferably a general processing means like a CPU or programmable microprocessor. Preferably, the processing means 11 comprises a programmable processor. Preferably, the subsequently described functions/methods/algorithms of the invention are realized as software programs executed on the processing means 11 or in the processor…. The processing means 11 or the HMD 10 comprises preferably a storage for storing the software programs according to the invention to be executed on the HMD 10. The storage stores preferably further a library of objects and associated to each object the spatial relationship of the set of object markers 21 as arranged on the object 20.--, in [0029]).
Conclusion
Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to WEI WEN YANG whose telephone number is (571)270-5670. The examiner can normally be reached on 8:00 - 5:00 pm.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Amandeep Saini can be reached on 571-272-3382. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/WEI WEN YANG/Primary Examiner, Art Unit 2662
Prosecution Insights