DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Allowable Subject Matter
Claims 3, 6-8 and 14 are objected to as being dependent upon a rejected base claim but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2, 4-5, 9-13, 15-17, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over “Gregerson et al.,” US 2018/0185113 (hereinafter Gregerson), “ “Navab et al.,” US 2010/0210902 (hereinafter Navab), “Riedel,” US 2017/0329402 (hereinafter Riedel), and “Kassis et al.,” US 2021/0104063 (hereinafter Kassis).
Regarding to claim 1, Gregerson teaches a system for aiding in interaction with a physical object, the system comprising:
a display device defining a plane, wherein the physical object is located in front of the plane (display device Figure 1 has a screen with a plane and subject is located on front side of the display screen 121 [0040]); and
a navigation system (motion tracking system [0077]) coupled to a control system (coupled to computer [0036]) and being configured to:
register computer images to the physical object (register the image data [0036] and register objects or tools with the image data [0053]);
track a pose of the physical object and the display device in a common coordinate system (common coordinate system using motion tracking [0036], tracking the position/orientation of the display device [0077]); and
control the display device (computer controlling the display device [0040]) for rendering the registered computer images according to the tracked pose of the physical object and the display device (at least a portion of the image data displayed on the display device may be modified in response to a change in at least one of the position and orientation of a display device with respect to the patient [0077]); and
In regards to limitations of “wherein the physical object is located in front of the plane,” and “wherein a perspective of the rendering is based on a field of view of a virtual camera that has a virtual position behind the plane,” the examiner notes that Gregerson teaches moving a display device over the patient or within a predetermined proximity to the patient surgical site ([0078]), and Figures 1 and 7A visually show that the display device is on the other side of the patient while the operator stands on the opposite side looking at the display device.
However, in the analogous field of endeavor in visualizing medical procedures, Navab discloses configuration that the physical object is located in front of the plane of the display device (Figure 1 shows that display device 21 and patient body and instrument (labelled as 5) in such an arrangement and discloses that mirror reflection of the scene is displayed ([0031]-[0032]). Thus, mirror reflection has a FOV from behind the display, as it is nature and property of mirror reflection to have a view potin from behind the mirror to create a reflection image in them mirror and “Riedel” supports this since it teaches that mirror showing reflected image and user view position perceives the virtual environment reflected from the mirror as originating from the target viewing volume behind the mirror ([0010]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify display device as taught by Gregerson to incorporate teaching of Navab, since mirror reflective image was well known in the art as taught by Navab. One of ordinary skill in the art could have combined the elements as claimed by Gregerson with no change in their respective functions, Gregerson can move the display device to be placed behind the physical object, and have a virtual view from the opposite side with respect to the patient and surgical scene, and the combination would have yielded nothing more than predictable results to one of ordinary skill in the art before the effective filing date of the claimed invention. The motivation would have been to allow user to better perceive the augmented 3D volumes, its depth, exact dimension and complex structures ([0016]), and there was reasonable expectation of success.
Gregerson further discloses a virtual view ([0060] and [0082]; virtual view works as virtual camera as claimed) but does not explicitly disclose user’s viewpoint tracking and updating the virtual position of the virtual camera in response to movement of the tracked pose of the user’s viewpoint.
However, in the analogous field of endeavor in displaying with virtual camera, Kassis teaches rendering image by established virtual camera position and details on the virtual camera position as follows:
track a user’s viewpoint sensor tracking eye of the user [0028]) and control the display device for rendering the images according to the user’s viewpoint (rendering image in the direction of the user’s gaze [0018])
automatically update the virtual position of the virtual camera in response to movement of the tracked pose of the user’s viewpoint ([0041]-[0042] virtual position updated/redefined periodically, receive gaze vector and body feature information, automatically [0044], using distance [0066]-[0068] Figure 4).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify virtual view as taught by Gregerson to incorporate teaching of Kassis, since a virtual camera with a virtual position that can be updated with respect to the tracked eye movement was well known in the art as taught by Kassis. One of ordinary skill in the art could have combined the elements as claimed by Gregerson with no change in their respective functions, but providing a virtual camera positioned to use distance and angle with respect to display device, and the combination would have yielded nothing more than predictable results to one of ordinary skill in the art before the effective filing date of the claimed invention. The motivation would have been to provide rendering of the image displayed by following gaze movement ([0018]), and there was reasonable expectation of success.
Regarding to claim 2, Gregerson, Navab, Riedel, and Kassis together teach all limitations of claim 1 as discussed above.
Navab teaches wherein the perspective of the rendering emulates a mirror reflection relative to the tracked pose of the user’s viewpoint ([0033]).
Regarding to claims 4-5, 12, and 15-16, Gregerson, Navab, Riedel, and Kassis together teach all limitations of claim 1 as discussed above.
Kassis further teaches following limitations:
Of claim 4, Wherein the perspective of the rendering is automatically rotated to align with the tracked pose of the user’s viewpoint relative to the plane of the display (virtual camera is not tied to the position or angle of any physical camera, positioned to follow a gaze of the user, by following the gaze of the user, the rendered image appears as though the user is looking into the camera [0018]).
Of claim 5, Wherein the field of view of the virtual camera is automatically updated in response to movement of the tracked pose of the user’s viewpoint (virtual camera is positioned to follow a gaze of a user [0018])
Of claim 12, wherein a perspective of the one or more slices is render based on the field-of-view of the virtual camera that has the virtual position automatically updated in response to movement of the tracked pose of the user’s viewpoint (automatically updating/changing the virtual position of the virtual camera in response to the pose of the display device ([0044]).
Of claim 15, a camera directed towards the user and wherein the navigation system is configured to track the pose of the user’s viewpoint by being configured to track the user’s head, face, and/or eyes using the camera ([0057]) and
of claim 16, the camera is mounted the display device (cameras mounted, integrated on a tablet, laptop ad smartphone, cameras arranged around a display [0025]).
Regarding to claims 9-11 and 13, Gregerson, Navab, Riedel, and Kassis together teach all limitations of claim 1 as discussed above.
Gregerson further teaches following limitations:
Of claim 9, wherein the physical object is a patient anatomy ( patient’s anatomy image dataset [0043]) and the computer images of the patient anatomy are derived from a 3D model (slice with a 3D volume rendering [0062], software model of the anatomical region of interest [0135]), and
of claim 10, the control system is configured to control the display device to display one or more slices of the 3D model of the patient anatomy depending upon the tracked pose of the physical object and the display device (display screen updated show the slices corresponding to the current position of the tip end [0064], [0079]).
Of claim 11, wherein the 3D model and the one or more slices are derived from medical imaging data of the patient’s anatomy ([0043]).
Of claim 13, wherein the control system is configured to control the display device to change the one or more slices to other slices in response to one or more of the following: the tracked pose of the display device; the tracked pose of the physical object; a tracked pose of a surgical instrument; and a tracked pose of a user’s viewpoint (updating slices/images as the display device is moved [0088]).
Regarding to claim 17, Gregerson teaches a method of operating a system for aiding in interaction with a physical object, the system comprising:
a display device defining a plane, wherein the physical object is located in front of the plane (display device Figure 1 has a screen with a plane and subject is located on front side of the display screen 121 [0040]); and
a navigation system (motion tracking system [0077]) coupled to a control system, the method comprising the navigation system performing the following:
registering computer images to the physical object (register the image data [0036] and register objects or tools with the image data [0053]);
tracking a pose of the physical object and the display device in a common coordinate system (common coordinate system using motion tracking [0036]); and
controlling the display device (computer controlling the display device [0040]) for rendering the registered computer images according to the tracked pose of the physical object and the display device (at least a portion of the image data displayed on the display device may be modified in response to a change in at least one of the position and orientation of a display device with respect to the patient [0077]); and
In regards to limitations of “wherein the physical object is located in front of the plane,” and “wherein a perspective of the rendering is based on a field of view of a virtual camera that has a virtual position behind the plane,” the examiner notes that Gregerson teaches moving a display device over the patient or within a predetermined proximity to the patient surgical site ([0078]), and Figures 1 and 7A visually show that the display device is on the other side of the patient while the operator stands on the opposite side looking at the display device.
However, in the analogous field of endeavor in visualizing medical procedures, Navab discloses configuration that the physical object is located in front of the plane of the display device (Figure 1 shows that display device 21 and patient body and instrument (labelled as 5) in such an arrangement and discloses that mirror reflection of the scene is displayed ([0031]-[0032]). Thus, mirror reflection has a FOV from behind the display, as it is nature and property of mirror reflection to have a view potin from behind the mirror to create a reflection image in them mirror and “Riedel” supports this since it teaches that mirror showing reflected image and user view position perceives the virtual environment reflected from the mirror as originating from the target viewing volume behind the mirror ([0010]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify display device as taught by Gregerson to incorporate teaching of Navab, since mirror reflective image was well known in the art as taught by Navab. One of ordinary skill in the art could have combined the elements as claimed by Gregerson with no change in their respective functions, Gregerson can move the display device to be placed behind the physical object, and have a virtual view from the opposite side with respect to the patient and surgical scene, and the combination would have yielded nothing more than predictable results to one of ordinary skill in the art before the effective filing date of the claimed invention. The motivation would have been to allow user to better perceive the augmented 3D volumes, its depth, exact dimension and complex structures ([0016]), and there was reasonable expectation of success.
Gregerson further discloses a virtual view ([0060] and [0082]; virtual view works as virtual camera as claimed), but does not explicitly disclose user’s viewpoint tracking and updating the virtual position of the virtual camera in response to movement of the tracked pose of the user’s viewpoint.
However, in the analogous field of endeavor in displaying with virtual camera, Kassis teaches rendering image by established virtual camera position and details on the virtual camera position as follows:
tracking a user’s viewpoint ( sensor tracking eye of the user [0028]) and controlling the display device for rendering the images according to the user’s viewpoint (rendering image in the direction of the user’s gaze [0018])
automatically updating the virtual position of the virtual camera in response to movement of the tracked pose of the user’s viewpoint ([0041]-[0042] virtual position updated/redefined periodically, receive gaze vector and body feature information, automatically [0044], using distance [0066]-[0068] Figure 4).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify virtual view as taught by Gregerson to incorporate teaching of Kassis, since a virtual camera with a virtual position that can be updated with respect to the tracked eye movement was well known in the art as taught by Kassis. One of ordinary skill in the art could have combined the elements as claimed by Gregerson with no change in their respective functions, but providing a virtual camera positioned to use distance and angle with respect to display device, and the combination would have yielded nothing more than predictable results to one of ordinary skill in the art before the effective filing date of the claimed invention. The motivation would have been to provide rendering of the image displayed by following gaze movement ([0018]), and there was reasonable expectation of success.
Regarding to claim 19, Gregerson teaches a non-transitory computer-readable medium ([0141]) configured for use with a system to aid interaction with a physical object, the system including
a display device defining a plane, wherein the physical object is located in front of the plane (display device Figure 1 has a screen with a plane and subject is located on front side of the display screen 121 [0040]); and
a navigation system (motion tracking system [0077]), and one or more processors ([0140]), the non-transitory computer readable medium comprising instructions, which when execute4d by the one or more processors ([0141]), are configured to:
register computer images to the physical object (register the image data [0036] and register objects or tools with the image data [0053]);
track a pose of the physical object and the display device in a common coordinate system (common coordinate system using motion tracking [0036]); and
control the display device (computer controlling the display device [0040]) for rendering the registered computer images according to the tracked pose of the physical object and the display device (at least a portion of the image data displayed on the display device may be modified in response to a change in at least one of the position and orientation of a display device with respect to the patient [0077]); and
In regards to limitations of “wherein the physical object is located in front of the plane,” and “wherein a perspective of the rendering is based on a field of view of a virtual camera that has a virtual position behind the plane,” the examiner notes that Gregerson teaches moving a display device over the patient or within a predetermined proximity to the patient surgical site ([0078]), and Figures 1 and 7A visually show that the display device is on the other side of the patient while the operator stands on the opposite side looking at the display device.
However, in the analogous field of endeavor in visualizing medical procedures, Navab discloses configuration that the physical object is located in front of the plane of the display device (Figure 1 shows that display device 21 and patient body and instrument (labelled as 5) in such an arrangement and discloses that mirror reflection of the scene is displayed ([0031]-[0032]). Thus, mirror reflection has a FOV from behind the display, as it is nature and property of mirror reflection to have a view potin from behind the mirror to create a reflection image in them mirror and “Riedel” supports this since it teaches that mirror showing reflected image and user view position perceives the virtual environment reflected from the mirror as originating from the target viewing volume behind the mirror ([0010]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify display device as taught by Gregerson to incorporate teaching of Navab, since mirror reflective image was well known in the art as taught by Navab. One of ordinary skill in the art could have combined the elements as claimed by Gregerson with no change in their respective functions, Gregerson can move the display device to be placed behind the physical object, and have a virtual view from the opposite side with respect to the patient and surgical scene, and the combination would have yielded nothing more than predictable results to one of ordinary skill in the art before the effective filing date of the claimed invention. The motivation would have been to allow user to better perceive the augmented 3D volumes, its depth, exact dimension and complex structures ([0016]), and there was reasonable expectation of success.
Gregerson further discloses a virtual view ([0060] and [0082]; virtual view works as virtual camera as claimed), but does not explicitly disclose user’s viewpoint tracking and updating the virtual position of the virtual camera in response to movement of the tracked pose of the user’s viewpoint.
However, in the analogous field of endeavor in displaying with virtual camera, Kassis teaches rendering image by established virtual camera position and details on the virtual camera position as follows:
track a user’s viewpoint ( sensor tracking eye of the user [0028]) and control the display device for rendering the images according to the user’s viewpoint (rendering image in the direction of the user’s gaze [0018])
automatically update the virtual position of the virtual camera in response to movement of the tracked pose of the user’s viewpoint ([0041]-[0042] virtual position updated/redefined periodically, receive gaze vector and body feature information, automatically [0044], using distance [0066]-[0068] Figure 4).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify virtual view as taught by Gregerson to incorporate teaching of Kassis, since a virtual camera with a virtual position that can be updated with respect to the tracked eye movement was well known in the art as taught by Kassis. One of ordinary skill in the art could have combined the elements as claimed by Gregerson with no change in their respective functions, but providing a virtual camera positioned to use distance and angle with respect to display device, and the combination would have yielded nothing more than predictable results to one of ordinary skill in the art before the effective filing date of the claimed invention. The motivation would have been to provide rendering of the image displayed by following gaze movement ([0018]), and there was reasonable expectation of success.
Claims 18 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Gregerson, Navab, Riedel, and Kassis as applied to claims 17 and 19 above, and further in view of “Engel,” US 2022/0022839 (hereinafter Engel).
Regarding to claims 18 and 20, Gregerson, Navab, Riedel, and Kassis together teach all limitations of claims 17 and 19 as discussed above.
Gregerson further teaches following limitations:
Obtaining a 3D model of a patient anatomy, wherein the 3D model is based on medical image data set (3D reconstruction of image data [0032]; patient’s anatomy image dataset [0043]), and deriving the computer images of the patient anatomy from a 3D model (slice with a 3D volume rendering [0062], software model of the anatomical region of interest [0135]), and
Controlling the display device to display one or more slices of the 3D model of the patient anatomy depending upon the tracked pose of the physical object and the display device (display screen updated show the slices corresponding to the current position of the tip end [0064], [0079], updating slice/images as the display device is moved [0088]).
Gregerson does not teach obtaining a 3D model of the patient anatomy based on the medical image data.
However, in the analogous field of endeavor in image guidance of medical procedures, Engel teaches a 3D model of the subject based on pre-acquired image data of the subject ([0044] and [0046]).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify 3D model as taught by Gregerson to incorporate teaching of Engel, since a 3D model of the subject is based on pre-acquired image data of the subject was well known in the art as taught by Engel. One of ordinary skill in the art could have combined the elements as claimed by Gregerson with no change in their respective functions, and the combination would have yielded nothing more than predictable results to one of ordinary skill in the art before the effective filing date of the claimed invention. The motivation would have been to provide 3D model of the specific subject ([0046]), and there was reasonable expectation of success.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PATRICIA J PARK whose telephone number is (571)270-1788. The examiner can normally be reached Monday-Thursday 8 am - 3 pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Pascal Bui-Pho can be reached on 571-272-2714. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PATRICIA J PARK/Primary Examiner, Art Unit 3798
Prosecution Insights