DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Claims
Claims 1-6, 8-17, and 19-22 are pending. No new claim has been added. Claims 7 and 18 have been cancelled. . Claims 1, 12, and 20 have been amended.
Response to Arguments
The Examiner finds Applicant’s arguments are unpersuasive.
(1) Applicant states,
PNG
media_image1.png
439
653
media_image1.png
Greyscale
Remarks p. 10.
The Examiner disagrees.
(a) Regarding “a virtual indication of the landmark at the location representing the surgeon-validated placement and a virtual indication of the landmark at the planned location representing the pre- planned state, simultaneously displayed as two visually distinct virtual objects overlaid on a real bone of the patient, the location and the planned location being different locations.”
The combination of Slagmolen and Wollowick teaches the limitation in question.
Slagmolen already teaches “a virtual indication of the landmark at the location representing the surgeon-validated placement and a virtual indication of the landmark at the planned location representing the pre- planned state,” the mappings of which Applicant did not dispute.
Slagmolen already teaches these virtual indications could be “displayed as two visually distinct virtual objects overlaid on a real bone of the patient” with augmented reality display during surgery, the mappings of which Applicant did not dispute.
Slagmolen already teaches that with respect to the same landmark, the two virtual indications could be distinct because of changed condition, stating “Automatic detection of the exposed bone or soft-tissue surface can allow creating of a 3D representation of this anatomy. This may be used for example to detect if the bone shape still corresponds to the original image data (e.g. to check that a recent scan was used) or to identify if soft-tissue changes have occurred that may influence the outcome (e.g. weight loss).” Slagmolen ¶ 88.
Wollowick teaches Slagmolen’s pre-operative plan image, similar to Wollowick’s pre-operative image, and Slagmolen’s intra-operative imagery information, similar to Wollowick’s intra-operative image, could be displayed simultaneously and compared.
(b) Regarding “receiving, at the processor, a confirmation of the location of the landmark, the confirmation validating the virtual indication at the location over the virtual indication at the planned location; removing, by the processor and in response to receiving the confirmation of the location of the landmark, the virtual indication of the landmark at the planned location.”
The combination of Slagmolen and Wollowick and Kuck teaches the limitation in question.
Applicant did not provide specific argument except for the statement: “Wollowick’s processing is performed on image data for analysis purposes, not within a live augmented reality surgical field in which the surgeon views the real anatomy through the display.” However, the feature is taught by Slagmolen, the Examiner’s primary reference, the teaching of which is not disputed by Applicant.
(2) Applicant states,
PNG
media_image2.png
387
660
media_image2.png
Greyscale
Remarks pp. 10-11.
The Examiner disagrees.
Applicant made general allegations except for the statement: “Kuck relies on manual user-driven editing of virtual objects, not on a processor-executed resolution of competing landmark states in response to a confirmation input.”
Applicant appears to allege that the removing process is a manual editing process, instead of an automated one.
Even if Applicant’s characterization is correct, according to MPEP 2144.04.III AUTOMATING A MANUAL ACTIVITY, it would have been obvious to automate a process or selected steps of the process.
Claim Rejections - 35 USC § 101
The rejection of Claim 20 under 35 U.S.C. 101 is withdrawn in view of Applicant’s amendment to the claim.
Claim Rejections - 35 USC § 112
Claims 1-6, 8-17, 19, and 21-22 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention.
Claim 1 recites: “a virtual indication of the landmark at the location representing the surgeon-validated placement and a virtual indication of the landmark at the planned location representing the pre-planned state, simultaneously displayed as two visually distinct virtual objects overlaid on a real bone of the patient;” However, the specification states, “The landmarks 302B and 304B may greater than the threshold distance in some examples, but less than the threshold distance in some other examples. In some examples, only landmarks that have a distance between surgeon generated and planned greater than the threshold may trigger a warning or require confirmation input from the surgeon.” Spec. ¶ 36.
Here, the specification appear to support “a virtual indication of the landmark at the location representing the surgeon-generated placement.” The distinction is significant, because Claim 1 later recites “a confirmation of the location of the landmark,validating the virtual indication at the location over the virtual indication at the planned location,” wherein “the indication of the location representing a surgeon-validated placement.” Why validate a location that has already been validated? More importantly, there is no support for validating an already-validated location over a planned location. For the purpose of art rejection. The Examiner is reading the related limitations as “the indication of the location representing a surgeon-generated placement” and “a virtual indication of the landmark at the location representing the surgeon-generated placement.”
Independent claim 12 shares the same feature as Claim 1 and has the same deficiency. Therefore, Claim 12 is rejected. All claims that depend on Claim 1 or 12 are also rejected because they inherit the deficiency.
Claim 20 is not rejected because it is differentiated from other independent claims.
Recommendations/suggestions from the Examiner:
The support issue and confusion of the wording in the claim could be addressed by clarifying language for the claim.
Applicant states, “Applicant's claimed solution addresses a problem specific to intraoperative augmented reality guidance, namely how to concurrently present both pre-operative planning information and surgeon-validated intraoperative information, and then resolve discrepancies between those states in a
controlled and automated manner.” Remarks p. 9.
Claim 1 states, “an indication of a location of a landmark on a bone of a patient in the surgical field, the indication of the location representing a surgeon-validated placement.” Claim 1 does not recite that the “indication of a location of a landmark” is intraoperative information. Therefore, Claim 1 does not sufficiently recite the problem or the solution as Applicant described.
Further, in order to address the concern of claiming validating an already-validated location over a planned location, he Examiner would modify Applicant’s characterization of the Application, “Applicant's claimed solution addresses a problem specific to intraoperative augmented reality guidance, namely how to concurrently present both pre-operative planning information and surgeon-generatedvalidation process
The specification states, “The landmarks 302B and 304B may greater than the threshold distance in some examples, but less than the threshold distance in some other examples. In some examples, only landmarks that have a distance between surgeon generated and planned greater than the threshold may trigger a warning or require confirmation input from the surgeon.” Spec. ¶ 36. If Applicant’s amendment incorporate this feature, it will overcome the existing art rejections on the record. However, further search and consideration will be needed to determine allowability.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-6, 8, 12-17, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Slagmolen et al. (US 20220125519 A1) in view of Wollowick et al. (US 20160128654 A1) and Kuck et al. (US 20160331262 A1).
Regarding Claim 1, Slagmolen teaches A method for using an augmented reality device in a surgical field (“Systems and methods of providing augmented reality assisted surgery are disclosed.” Slagmolen Abstract.) comprising:
receiving, at a processor, an indication of a location of a landmark on a bone of a patient in the surgical field, the indication of the location representing a surgeon-validated placement (
Slagmolen teaches an indication of a location of an anatomical landmark, stating “One form of interaction with the system is indicating, such as via I/O module 122, certain points in the scene, such as anatomical landmarks. In some embodiments, indicating points on the anatomy can be performed by using a tracked pointing device, such as a stylus (e.g., an input device coupled to I/O module 122). In other embodiments, points may be indicated without using a pointing device. For example, after entering into an ‘indicate point’ mode, a target symbol, such as cross hairs, may be displayed to the user in the display unit. These cross hairs may be then be moved by head or eye movement, until they point to the desired anatomical point, at which time the user can give the command to lock the position. In yet other embodiments, such a target symbol may be moved manually by the user. In such embodiments, the system may detect hand or finger movements in the user's field of view and translate those movements to movements of the target symbol. By varying the speed at which the target symbol is moved relative to the speed of the hand or finger movements, the system may allow anything from rough indication to fine-tuning.” Slagmolen ¶ 154.
Slagmolen further teaches the anatomical landmark could be a bone of a patient in a surgical field, stating “In some embodiments, the surgical scene may also be augmented with anatomical landmarks or structures directly derived from those anatomical landmarks. These may include individual locations in space (points) such as malleoli, ligament insertion points, joint rotation centers, ankle or hip tuberositas, etc. These may also include lines that represent either anatomical or mechanical axes of the individual bones or rotation axes of the joint or the mechanical axis of the entire joint system (HKA).” Slagmolen ¶ 105.
Slagmolen teaches “surgeon-generated” because of disclosed “surgeon choice,” stating “During planning, the surgeon may choose the landmarks that he wishes to be highlighted and overlaid on the patient during surgery.” Slagmolen ¶ 47.);
retrieving, using the processor, a planned location of the landmark corresponding to a known location on the bone of the patient based on a pre-operative image of the bone of the patient, the planned location representing a pre-planned state ( Slagmolen teaches pre-operative planning:
Fig. 5:
PNG
media_image3.png
392
406
media_image3.png
Greyscale
. Fig. 5 502 “Pre-operative Planning.” “At block 502, pre-operative planning is performed beforehand on a conventional planning workstation. This defines the implant type, size and its position in relation to the anatomy. From this, the number, size and location of apertures are derived for the glenoid cavity.” Slagmolen ¶ 184.
Slagmolen teaches “planned location of the landmark” of a patient in the context of “pre-operative” planning for a patient, stating “The computing environment 100 may also include a planning module 108. The planning module 108 may be configured to perform surgical planning for an AR system. This may be done pre-operatively or intra-operatively. In both cases, anatomical landmarks, any derived coordinate systems and/or axes, and/or 3D anatomical shapes may be used by planning module 108 to determine (e.g., the most) suitable implant components, their sizes and their positions (e.g. locations and orientations) in relation to said anatomical landmarks. . . . During planning, the surgeon may choose the landmarks that he wishes to be highlighted and overlaid on the patient during surgery.” Slagmolen ¶ 47.
The claimed “landmark” is mapped to disclosed “anatomical landmark” or implant locations.
When the disclosed “planning” is done, e.g., “pre-operatively,” the planned location representing a pre-planned state.
Slagmolen teaches pre-operative planning based on “image of the bone of the patient,” stating “Automatic detection of the exposed bone or soft-tissue surface can allow creating of a 3D representation of this anatomy. This may be used for example to detect if the bone shape still corresponds to the original image data (e.g. to check that a recent scan was used) or to identify if soft-tissue changes have occurred that may influence the outcome (e.g. weight loss).” Slagmolen ¶ 88. See Slagmolen ¶ 145.);
presenting within the surgical field, using an augmented reality display of the augmented reality device, while permitting the surgical field to be viewed through the augmented reality display (
“Systems and methods of providing augmented reality assisted surgery are disclosed.” Slagmolen Abstract. Slagmolen states, “Accordingly, certain aspects of the disclosure provide novel systems and methods for using mixed-reality (e.g., augmented (AR)), to allow the translation of the virtual surgical plan to the operating room (OR) by blending the surgical scene with a virtual environment (augmented environment'), and using a display device/unit, such as a portable device or a headset, to visualize this blended environment, thereby assisting the surgeon during surgery.” Slagmolen ¶ 20.
Here, the “mixed reality” allows the surgical field to be viewed through, because the surgical scene and the virtual environment are blended.),
a virtual indication of the landmark at the location representing the surgeon-validated placement and a virtual indication of the landmark at the planned location representing the preplanned state,
PNG
media_image4.png
510
678
media_image4.png
Greyscale
(a) Slagmolen teaches displaying a surgical plan (fig. 2 210), e.g., pre-operative plan (Slagmolen ¶¶ 19, 47), during a surgery, stating “. . . using mixed-reality (e.g., augmented (AR)), to allow the translation of the virtual surgical plan to the operating room (OR) by blending the surgical scene with a virtual environment (augmented environment'), and using a display device/unit, such as a portable device or a headset, to visualize this blended environment, thereby assisting the surgeon during surgery.” Slagmolen ¶ 20. Slagmolen further teaches the surgery plan may include highlighted/indicated landmark at the planned location, stating “During planning, the surgeon may choose the landmarks that he wishes to be highlighted and overlaid on the patient during surgery.” Slagmolen ¶ 47.
The claimed “virtual indication” is mapped to the disclosed highlighting.
(b) Slagmolen teaches the pre-operative plan may be updated and displayed during the surgery, stating “The plan 210 may be updated by planning module 108 during surgery when additional information (such as complications that could not be predicted pre-operatively) becomes available (cf. FIG. 6 for a workflow).” Slagmolen ¶ 47. Slagmolen teaches the update could be due to bone shape’s failed correspondence during surgery, stating “Automatic detection of the exposed bone or soft-tissue surface can allow creating of a 3D representation of this anatomy. This may be used for example to detect if the bone shape still corresponds to the original image data (e.g. to check that a recent scan was used) or to identify if soft-tissue changes have occurred that may influence the outcome (e.g. weight loss).” Slagmolen ¶ 88.
Slagmolen teaches marking location of bone landmark during surgery, stating “The user may receive guidance 214, by guidance module 116, for intra-operative annotation of landmarks, surface or any other geometric feature, either via a physical object (e.g. a 3D printed model with marked registration landmarks) or via a virtual object displayed in the augmented scene. This guidance may show which landmarks need to be marked and in which order, it may show which parts of the patient anatomy are reliable for registration during marking . . .. This guidance may be aligned and updated with the user's steps and as he moves forward through the workflow (e.g. showing consequential landmarks after the previous have been annotated successfully). . . . If the user (surgeon in the OR) then wishes for the plan to be updated as per the suggestion, the plan is updated or else the surgeon can go back to the initial plan (as it was the locked scene) i.e., the initial data/plan is not lost and navigating is still easy.” Slagmolen ¶ 96.
Slagmolen teaches marking landmarks with virtual indication, e.g., cross hair, stating “One form of interaction with the system is indicating, such as via I/O module 122, certain points in the scene, such as anatomical landmarks. . . . For example, after entering into an ‘indicate point’ mode, a target symbol, such as cross hairs, may be displayed to the user in the display unit. These cross hairs may be then be moved by head or eye movement, until they point to the desired anatomical point, at which time the user can give the command to lock the position.” Slagmolen ¶ 154.);
However, Slagmolen does not explicitly disclose
visual indications of the landmark simultaneouslydisplayed for comparison,
receiving, at the processor, a confirmation of the location of the landmark, the confirmation validating the virtual indication at the location over the virtual indication at the planned location;
removing, by the processor and in response to receiving the confirmation of the location of the landmark, the virtual indication of the landmark at the planned location.
Wollowick teaches:
visual indications of the landmark simultaneouslydisplayed for comparison (
PNG
media_image5.png
462
664
media_image5.png
Greyscale
“In this construction of Step 3018, the Image Comparison Module 3036, FIG. 18 superimposes the preop and intraop images by aligning pelvic anatomy, with the images displayed with some transparency so that both can be visualized in the overlay, such as illustrated in FIG. 20. In a preferred construction the overlaid images will contain the identified femoral landmarks (generally placed on the greater trochanter) generated in step 3008 so that location differences between the two points can be visualized. The system will maintain the location of the generated greater trochanter points and the femoral axis lines, relative to the preop and intraop images, as the images are manipulated to create the image overlay.” Wollowick ¶ 97.
“In Step 3022, FIG. 17, Landmark Correction Module 3038, FIG. 18 uses data gathered in previous steps to generate an additional ‘corrected’ or ‘phantom’ landmark point that accounts for differences in femoral position between the preop and intraop images. A corrected landmark point 3082 is shown in FIG. 20, positioned along circle 3083 from intraoperative landmark point 3074′, which is similar to corrected landmark point 3116, FIG. 21, along circle 3124 as described in more detail below.” Wollowick ¶ 101.
Here, for example, 3074’, 3054,’ and 3082 are landmark simultaneouslydisplayed.);
receiving, at the processor, a confirmation of the location of the landmark, the confirmation validating the virtual indication at the location over the virtual indication at the planned location (
“This anatomic feature, in conjunction with the stationary base, will depict any differences or errors in pelvic anatomy or the overlay which will enable the physician to validate, or to have more confidence in, the output of the present system. As generally utilized herein: (i) a ‘stationary point’ refers to a point on a relatively stationary bone such as on the pelvis; (ii) a ‘landmark point’ is located on an articulating bone such as a femur; (iii) an ‘error point’ is preferably on pelvis and spaced from other points; and (iv) a ‘fixed point’ is located on an implant, such as the shoulder of a femoral stem prosthesis.” Wollowick ¶ 52.
After Slagmolen and Wollowick are combined, the output of the present system becomes Slagmolen’s planned location.
Slagmolen in view of Wollowick’s validation could help Slagmolen’s method to update its surgery plan/guidance, Slagmolen stating “The user may receive guidance 214, by guidance module 116, for intra-operative annotation of landmarks, surface or any other geometric feature, either via a physical object (e.g. a 3D printed model with marked registration landmarks) or via a virtual object displayed in the augmented scene. This guidance may show which landmarks need to be marked and in which order, it may show which parts of the patient anatomy are reliable for registration during marking . . .. This guidance may be aligned and updated with the user's steps and as he moves forward through the workflow (e.g. showing consequential landmarks after the previous have been annotated successfully). . . . If the user (surgeon in the OR) then wishes for the plan to be updated as per the suggestion, the plan is updated or else the surgeon can go back to the initial plan (as it was the locked scene) i.e., the initial data/plan is not lost and navigating is still easy.” Slagmolen ¶ 96.
This may be needed because of Slagmolen’s observation:
Slagmolen teaches the update could be due to bone shape’s failed correspondence during surgery, stating “Automatic detection of the exposed bone or soft-tissue surface can allow creating of a 3D representation of this anatomy. This may be used for example to detect if the bone shape still corresponds to the original image data (e.g. to check that a recent scan was used) or to identify if soft-tissue changes have occurred that may influence the outcome (e.g. weight loss).” Slagmolen ¶ 88.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Wollowick’s simultaneously displaying visual indications with Slagmolen. One of ordinary skill in the art would be motivated to visually compare landmarks to understand the change in location and/or to validate landmarks. Wollowick states, “In this construction of Step 3018, the Image Comparison Module 3036, FIG. 18 superimposes the preop and intraop images by aligning pelvic anatomy, with the images displayed with some transparency so that both can be visualized in the overlay, such as illustrated in FIG. 20. In a preferred construction the overlaid images will contain the identified femoral landmarks (generally placed on the greater trochanter) generated in step 3008 so that location differences between the two points can be visualized.” Wollowick ¶ 97.
Slagmolen in view of Wollowick does not explicitly teach:
removing, by the processor and in response to receiving the confirmation of the location of the landmark, the virtual indication of the landmark at the planned location.
Kuck teaches removing, by the processor and in response to receiving the confirmation of the location of the landmark, the virtual indication of the landmark at the planned location (
Kuck states “The physician is instructed to confirm the locations of the labels and landmarks that are displayed. Once the locations have been confirmed by the physician, the repeat loop terminates. Else, the loop continues and the physician is repetitively instructed to correct, add or remove landmarks, until he or she has confirmed them. In such a way, the computer visualizes on display 520 an interactive graphical user interface tool for repositioning and/or removal/addition of labels and landmarks.” Kuck ¶ 144.
Here, Kuck teaches removing a virtual indication of a landmark, if it is not confirmed.
Slagmolen in view of Wollowick teaches that the virtual indication of the landmark at the planned location is a preoperative marking (Slagmolen ¶ 47); and Slagmolen in view of Wollowick teaches the virtual indication of the landmark at the location is intraoperative or postoperative marking (Slagmolen ¶¶ 47, 96, 153). Both markings refer to the same landmark. The markings could be different because of i) incorrect marking, ii) insufficient registration, iii) a patient’s changed condition, and/or iv) changes effected by the surgery.
The preoperative marking is provided for comparison. See Wollowick ¶ 97. When the current location for the landmark is confirmed/validated/locked (Slagmolen ¶ 154; Wollowick ¶ 52; Kuck ¶ 144), the preoperative marking is unconfirmed/invalidated, because it is outdated. The preoperative marking is provided for comparison (Wollowick ¶ 97).
When the intraoperative or postoperative marking for the landmark is confirmed/validated, the preoperative marking for the same landmark is unconfirmed/invalidated. Accordingly, according to Kuck, confirmed intraoperative or postoperative marking for a landmark, validated by a surgeon, will be maintained, while the outdated preoperative marking for the same landmark will be removed.
Further, according to MPEP 2144.04.III AUTOMATING A MANUAL ACTIVITY, it would have been obvious to automate a process or selected steps of the process.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Kuck’s removing unconfirmed landmark(s) with Slagmolen in view of Wollowick. One of ordinary skill in the art would be motivated to maintain information that is more relevant/validated/accurate while removing information that is unrelated/invalidated/inaccurate. It may be less likely that the unrelated/invalidated/inaccurate information may be misused by mistake. In addition, it would also make the interface less cluttered.
Regarding Claim 2, Slagmolen in view of Wollowick and Kuck teaches The method of claim 1,
wherein the indication of the location of the landmark is stored in a database (
Slagmolen states, “In certain embodiments, patient data may be loaded into the virtual 3D model creation module 106 from a file, a storage medium, cloud database or entered manually into the virtual 3D model creation module 106 in the OR.” Slagmolen ¶ 43. Slagmolen states, “The non-volatile memory 313 may also be used to store non-executable data, such database files and the like.” Slagmolen ¶ 169.
Slagmolen does not explicitly disclose that its indication of the location of the landmark are stored in the disclosed database.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Slagmolen’s database with Slagmolen’s indication of the location of the landmark. One of ordinary skill in the art would be motivated to a user to reuse the landmark data later, which makes it more convenient for the user.).
Regarding Claim 3, Slagmolen in view of Wollowick and Kuck teaches The method of claim 1,
wherein the indication of the location of the landmark is received directly from a landmark generation device (
Slagmolen teaches landmark generation device that generates an indication of a location of an anatomical landmark, stating “One form of interaction with the system is indicating, such as via I/O module 122, certain points in the scene, such as anatomical landmarks. In some embodiments, indicating points on the anatomy can be performed by using a tracked pointing device, such as a stylus (e.g., an input device coupled to I/O module 122). In other embodiments, points may be indicated without using a pointing device. For example, after entering into an ‘indicate point’ mode, a target symbol, such as cross hairs, may be displayed to the user in the display unit. These cross hairs may be then be moved by head or eye movement, until they point to the desired anatomical point, at which time the user can give the command to lock the position. In yet other embodiments, such a target symbol may be moved manually by the user. In such embodiments, the system may detect hand or finger movements in the user's field of view and translate those movements to movements of the target symbol. By varying the speed at which the target symbol is moved relative to the speed of the hand or finger movements, the system may allow anything from rough indication to fine-tuning.” Slagmolen ¶ 154.).
Regarding Claim 4, Slagmolen in view of Wollowick and Kuck teaches The method of claim 1,
wherein the confirmation indicates the location of the landmark is correct (
Slagmolen teaches locking/confirming the landmark location, stating “For example, after entering into an ‘indicate point’ mode, a target symbol, such as cross hairs, may be displayed to the user in the display unit. These cross hairs may be then be moved by head or eye movement, until they point to the desired anatomical point, at which time the user can give the command to lock the position.” Slagmolen ¶ 154.
Wollowick states, “This anatomic feature, in conjunction with the stationary base, will depict any differences or errors in pelvic anatomy or the overlay which will enable the physician to validate, or to have more confidence in, the output of the present system. As generally utilized herein: (i) a “stationary point” refers to a point on a relatively stationary bone such as on the pelvis; (ii) a “landmark point” is located on an articulating bone such as a femur; (iii) an “error point” is preferably on pelvis and spaced from other points; and (iv) a “fixed point” is located on an implant, such as the shoulder of a femoral stem prosthesis.” Wollowick ¶ 52.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Wollowick’s validation by a surgeon with Slagmolen in view of Wollowick and Kuck. One of ordinary skill in the art would be motivated to increase data accuracy by confirmation made by an expert.
Regarding Claim 5, Slagmolen in view of Wollowick and Kuck teaches The method of claim 1, further comprising,
for a second landmark having a second location and a second planned location, receiving a change for the second landmark from the second location to the second planned location, and outputting information corresponding to the change to the augmented reality display (
Slagmolen teaches marking landmarks with virtual indication, e.g., cross hair, stating “One form of interaction with the system is indicating, such as via I/O module 122, certain points in the scene, such as anatomical landmarks. . . . For example, after entering into an ‘indicate point’ mode, a target symbol, such as cross hairs, may be displayed to the user in the display unit. These cross hairs may be then be moved by head or eye movement, until they point to the desired anatomical point, at which time the user can give the command to lock the position.” Slagmolen ¶ 154.
Note, there are multiple landmarks here.
The user may control and place the cross hair to any location on the display.
In addition, it would have been obvious to try (KSR) to change the second landmark from the second location to the second planned location, because one is choosing from a finite number of identified, predictable solutions (planned locations), with a reasonable expectation of success.).
Regarding Claim 6, Slagmolen in view of Wollowick and Kuck teaches The method of claim 1, further comprising,
for a second landmark having a second location and a second planned location, receiving a change for the second landmark from the second location to a new location other than the second location and the second planned location, and outputting information corresponding to the new location to the augmented reality display for presenting (
Slagmolen teaches marking landmarks with virtual indication, e.g., cross hair, stating “One form of interaction with the system is indicating, such as via I/O module 122, certain points in the scene, such as anatomical landmarks. . . . For example, after entering into an ‘indicate point’ mode, a target symbol, such as cross hairs, may be displayed to the user in the display unit. These cross hairs may be then be moved by head or eye movement, until they point to the desired anatomical point, at which time the user can give the command to lock the position.” Slagmolen ¶ 154.
Note, there are multiple landmarks here.
The user may control and place the cross hair to any location on the display. ).
Regarding Claim 8, Slagmolen in view of Wollowick and Kuck teaches The method of claim 1, further comprising
registering the bone using a 3D model before receiving the indication of the landmark (
PNG
media_image4.png
510
678
media_image4.png
Greyscale
Slagmolen teaches receiving the indication of the landmark at fig. 2 214 and 216 through annotation, stating “The user may receive guidance 214, by guidance module 116, for intra-operative annotation of landmarks, surface or any other geometric feature, either via a physical object (e.g. a 3D printed model with marked registration landmarks) or via a virtual object displayed in the augmented scene.” Slagmolen ¶ 96.
Slagmolen teaches registering the bone using a 3D model at fig. 2 218, stating “Optical systems may use physical markers to perform registration 218 and tracking by registration module 114. These markers may be positioned in a fixed location in the actual scene 230, or be attached to any independently moving objects in the actual scene 230, including the patient, any individually moving anatomical part of the patient (e.g. bones, soft-tissue, limbs, etc.), and objects such as instruments (e.g. drills, saws, bone pins, etc.), anatomical 3D patient-specific models, implants or implant components. . . . Any physical 3D object with a known geometry may directly serve as a physical marker. The marker may be a (3D printed) guide, which is (partially) patient-specific and may be configurable intra-operatively to a certain degree or an anatomical 3D patient-specific model used for registration. Markers may also be included in wearable devices such as eyewear, elastic bands, etc.” Slagmolen ¶ 73.
Here, fig. 2 218 (bone registration) could occur before fig. 2 214 and 216 (landmark annotation) according to fig. 2 as shown by the arrows.).
Claims 12-17, 19 are rejected with similar reasons as set forth in claims 1-6, 8, respectively above. In addition, Claim 12 recites, “An augmented reality device in a surgical field comprising: a processor: memory including instructions, which when executed by the processor, cause the processor to perform operations to: . . .” (Slagmolen ¶¶ 168-169).
Claims 9 and 21 are rejected under 35 U.S.C. 103 as being unpatentable over Slagmolen in view of Wollowick and Kuck as applied to Claims 1 and , in further view of Kim et al. (US 20210176383 A1).
Regarding Claim 9, Slagmolen in view of Wollowick and Kuck teaches The method of claim 1.
Slagmolen in view of Wollowick and Kuck does not explicitly disclose further comprising
receiving an indication to move a virtual navigation menu presented in the augmented reality display.
Kim teaches receiving an indication to move a virtual navigation menu presented in the augmented reality display (
“The eye tracking information may also permit a user to control an electronic system. For example, when a user moves his or her eyes, blinks, or makes a gesture using his or her eyes, the user may be able to move a cursor, navigate menus, and/or launch and interact with various applications. . . . Moreover, the eye tracking information may be used in applications involving virtual reality, augmented reality and/or mixed reality, and the eye tracking information may be used in any device including an electronic display such a mobile phone, tablet, phablet, laptop, automobiles (e.g., consoles), or other consumer or commercial goods.” Kim ¶ 59.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Kim’s navigation menu with Slagmolen in view of Wollowick and Kuck. One of ordinary skill in the art would be motivated to provide more convenience to a augmented reality user, so that the user may move a menu to a preferred location for easy interaction.
Regarding Claim 21, Slagmolen in view of Wollowick and Kuck teaches The method of claim 1,
wherein the confirmation is received through gesture recognition detected by
Slagmolen teaches marking landmarks with virtual indication, e.g., cross hair, stating “One form of interaction with the system is indicating, such as via I/O module 122, certain points in the scene, such as anatomical landmarks. . . . For example, after entering into an ‘indicate point’ mode, a target symbol, such as cross hairs, may be displayed to the user in the display unit. These cross hairs may be then be moved by head or eye movement, until they point to the desired anatomical point, at which time the user can give the command to lock the position.” Slagmolen ¶ 154. “For example, head motion tracking, eye tracking or gaze tracking, etc. may all instruct the system to display, hide or alter the position or scale of certain augmentation elements. One or more combinations may be used.” Slagmolen ¶ 150.
“The system, such as via I/O module 122, may provide ways for the user to interact with it to change location, orientation and/or scale of the displayed elements, for example, by means of gesture-based controls.” Slagmolen ¶ 125.).
Slagmolen in view of Wollowick and Kuck does not explicitly disclose that eye movement is detected by a camera of the augmented reality device.
Kim teaches that eye movement is detected by a camera of a device (“The eye tracking unit may include cameras, mirrors, and lenses.” Kim ¶ 2.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Kim’s eye tracking with Slagmolen in view of Wollowick and Kuck. One of ordinary skill in the art would be motivated to allow a user to communicate and interact with the computing system with eyes, which is useful when other ways of communication, e.g., hand gesture, are inconvenient or unavailable.
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Slagmolen in view of Wollowick and Kuck as applied to Claims 1, and in further view of Couture et al. (US 20170360512 A1).
Regarding Claim 10, Slagmolen in view of Wollowick and Kuck teaches The method of claim 1.
Slagmolen in view of Wollowick and Kuck does not explicitly disclose wherein a position and orientation of the bone is determined using bone tracking via a passive robotic arm.
Couture teaches wherein a position and orientation of the bone is determined using bone tracking via a passive robotic arm (“The tracking apparatus 70 may be used to track the bones of the patient, and the robot arm 20 when present. For example, the tracking apparatus 70 may assist in performing the calibration of the patient bone with respect to the robot arm, for subsequent navigation in the X, Y, Z coordinate system.” Couture ¶ 44. “Alternatively, the foot support 30 could be supported by a passive mechanism, with the robot arm 20 connecting to the foot support 30 to actuate its displacements in a controlled manner in the coordinate system.” Couture ¶ 47. “Example 1 is a robot-aided surgical system comprising: a tracking system including a first tracker affixed to a bone of a patient, the tracking system configured to track movement of the bone; a robotic controller to: monitor a position and orientation of an end effector coupled to an end of a robotic arm; apply a force to the bone using the end effector; determine soft tissue balance using information from the tracking system including a position of the first tracker affixed to the bone; and output the soft tissue balance.” Couture ¶ 150.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Couture’s bone tracking system with Slagmolen in view of Wollowick and Kuck. One of ordinary skill in the art would be motivated to tracking precision. Couture states, “However, when performing orthopedic surgery on joints, soft tissue evaluations are conventionally done by hand, with the surgeon qualitatively assessing the limits of patient's range of motion. The conventional technique may result in errors or lack precision.” Couture ¶ 3.
Claim 11 is rejected under 35 U.S.C. 103 as being unpatentable over Slagmolen in view of Wollowick and Kuck as applied to Claim 1, in further view of Zhang et al. (US 20220313375 A1).
Regarding Claim 11, Slagmolen in view of Wollowick and Kuck teaches The method of claim 1 further comprising
displaying a live video, using the augmented reality display, of the bone using a camera in a surgical theater where a robotic arm is used (
Slagmolen teaches the live video of augmented reality could be stored, stating “The augmented environment (and/or the individual virtual and physical scene components) may be recorded 234 for archiving, either as a video stream or a sub-selection of individual frames may be taken at any point during the surgery. In certain aspects, computing environment 100 includes a recording/streaming module 124 be configured to perform such recording.” Slagmolen ¶ 161.
Slagmolen teaches the augmented reality is live, stating “The augmented environment may also be used to keep additional staff informed on surgical decisions/status of surgery so they can prepare themselves better, for example by highlighting the next instrument that will be needed or the screw length that should be taken out of the inventory. The augmented environment may also be stored for future references wherein it can later be played as a movie, as needed. The system also enables the surgeon to pause, play, capture any part (s) of the augment environment in the form of snapshots and/or live motion pictures at any time during its use.” Slagmolen ¶ 163.
Slagmolen teaches an augmented reality display and camera affixed to a surgical theater, stating “AR systems typically comprise at least one camera. This camera may be embedded in the device that also comprises the display (handheld device, headmounted display), but may also be external to the display system (e.g. a wireless or wired camera attached to the surgical theater).” Slagmolen ¶ 24.
Slagmolen further teaches the anatomical landmark could be a bone of a patient in a surgical field, stating “In some embodiments, the surgical scene may also be augmented with anatomical landmarks or structures directly derived from those anatomical landmarks. These may include individual locations in space (points) such as malleoli, ligament insertion points, joint rotation centers, ankle or hip tuberositas, etc. These may also include lines that represent either anatomical or mechanical axes of the individual bones or rotation axes of the joint or the mechanical axis of the entire joint system (HKA).” Slagmolen ¶ 105.
Slagmolen teaches a robotic arm, stating “In some embodiments, the I/O module 122 for example, may be able to interact with a robot system, e.g. by actively positioning a robot arm or providing instructions for manual positioning of the robot arm, e.g. by showing a target position or providing force-based guidance to the robot. In some embodiments, the system is used to control the robot, e.g. by integrating the robot control user interface in the AR system.” Slagmolen ¶ 149.).
However, Slagmolen in view of Wollowick and Kuck does not explicitly disclose that the camera is affixed to an end effector of a robotic arm.
Zhang teaches the camera is affixed to an end effector of a robotic arm (“The robotic endoscopic apparatus comprises: a disposable elongate member comprising: a proximal end and a distal end, wherein the proximal end is removably attached to a robotic arm via a handle, wherein the distal end is integrated with an imaging device, a position sensor and an illumination device; and a bending section that is articulated by one or more pull wires.” Zhang ¶ 6. ).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Zhang’s camera placement with Slagmolen in view of Wollowick and Kuck. One of ordinary skill in the art would be motivated to provide high quality images that follow the robotic arm, which allows better control of the robotic camera.
Claim 20 is rejected under 35 U.S.C. 103 as being unpatentable over Slagmolen in view of Wollowick.
Regarding Claim 20, Slagmolen teaches At least one non-transitory machine-readable medium including instructions for operating an augmented reality device in a surgical field, which when executed by a processor, cause the processor to perform operations (“Systems and methods of providing augmented reality assisted surgery are disclosed.” Slagmolen Abstract. Slagmolen ¶¶ 168-169.) to:
retrieving a plurality of planned locations corresponding to each of a plurality of landmarks on a bone of a patient based on pre-operative imaging of the bone of the patient, each planned location representing a pre-planned state (
Slagmolen teaches indications of locations of anatomical landmarks, stating “One form of interaction with the system is indicating, such as via I/O module 122, certain points in the scene, such as anatomical landmarks. In some embodiments, indicating points on the anatomy can be performed by using a tracked pointing device, such as a stylus (e.g., an input device coupled to I/O module 122). In other embodiments, points may be indicated without using a pointing device. For example, after entering into an ‘indicate point’ mode, a target symbol, such as cross hairs, may be displayed to the user in the display unit. These cross hairs may be then be moved by head or eye movement, until they point to the desired anatomical point, at which time the user can give the command to lock the position. In yet other embodiments, such a target symbol may be moved manually by the user. In such embodiments, the system may detect hand or finger movements in the user's field of view and translate those movements to movements of the target symbol. By varying the speed at which the target symbol is moved relative to the speed of the hand or finger movements, the system may allow anything from rough indication to fine-tuning.” Slagmolen ¶ 154.
Slagmolen further teaches the anatomical landmark could be a bone of a patient in a surgical field, stating “In some embodiments, the surgical scene may also be augmented with anatomical landmarks or structures directly derived from those anatomical landmarks. These may include individual locations in space (points) such as malleoli, ligament insertion points, joint rotation centers, ankle or hip tuberositas, etc. These may also include lines that represent either anatomical or mechanical axes of the individual bones or rotation axes of the joint or the mechanical axis of the entire joint system (HKA).” Slagmolen ¶ 105.
Slagmolen teaches “planned location of the landmark” of a patient in the context of “pre-operative” planning for a patient, stating “The computing environment 100 may also include a planning module 108. The planning module 108 may be configured to perform surgical planning for an AR system. This may be done pre-operatively or intra-operatively. In both cases, anatomical landmarks, any derived coordinate systems and/or axes, and/or 3D anatomical shapes may be used by planning module 108 to determine (e.g., the most) suitable implant components, their sizes and their positions (e.g. locations and orientations) in relation to said anatomical landmarks. . . . During planning, the surgeon may choose the landmarks that he wishes to be highlighted and overlaid on the patient during surgery.” Slagmolen ¶ 47.);
presenting within the surgical field, using an augmented reality display of the augmented reality device, while permitting the surgical field to be viewed through the augmented reality display (
“Systems and methods of providing augmented reality assisted surgery are disclosed.” Slagmolen Abstract. Slagmolen states, “Accordingly, certain aspects of the disclosure provide novel systems and methods for using mixed-reality (e.g., augmented (AR)), to allow the translation of the virtual surgical plan to the operating room (OR) by blending the surgical scene with a virtual environment (augmented environment'), and using a display device/unit, such as a portable device or a headset, to visualize this blended environment, thereby assisting the surgeon during surgery.” Slagmolen ¶ 20.
Here, the “mixed reality” allows the surgical field to be viewed through, because the surgical scene and the virtual environment are blended.),
virtual indications of the plurality of landmarks at the plurality of planned locations simultaneously displayed as at least two visually distinct virtual objects overlaid on a real bone of the patient, each of the plurality of planned locations being different locations (
PNG
media_image4.png
510
678
media_image4.png
Greyscale
(a) Slagmolen teaches displaying a surgical plan (fig. 2 210), e.g., pre-operative plan (Slagmolen ¶¶ 19, 47), during a surgery, stating “. . . using mixed-reality (e.g., augmented (AR)), to allow the translation of the virtual surgical plan to the operating room (OR) by blending the surgical scene with a virtual environment (augmented environment'), and using a display device/unit, such as a portable device or a headset, to visualize this blended environment, thereby assisting the surgeon during surgery.” Slagmolen ¶ 20. Slagmolen further teaches the surgery plan may include highlighted/indicated landmarks at the planned locations, stating “During planning, the surgeon may choose the landmarks that he wishes to be highlighted and overlaid on the patient during surgery.” Slagmolen ¶ 47.
The claimed “virtual indication” is mapped to the disclosed highlighting.
Slagmolen teaches marking landmarks with virtual indication, e.g., cross hair, stating “One form of interaction with the system is indicating, such as via I/O module 122, certain points in the scene, such as anatomical landmarks. . . . For example, after entering into an ‘indicate point’ mode, a target symbol, such as cross hairs, may be displayed to the user in the display unit. These cross hairs may be then be moved by head or eye movement, until they point to the desired anatomical point, at which time the user can give the command to lock the position.” Slagmolen ¶ 154.);
receiving, for a first landmark of the plurality of landmarks, an indication of a location on the bone representing a surgeon-validated placement (Slagmolen teaches locking/confirming the landmark location, stating “For example, after entering into an ‘indicate point’ mode, a target symbol, such as cross hairs, may be displayed to the user in the display unit. These cross hairs may be then be moved by head or eye movement, until they point to the desired anatomical point, at which time the user can give the command to lock the position.” Slagmolen ¶ 154.
Slagmolen teaches a “surgeon choice,” stating “During planning, the surgeon may choose the landmarks that he wishes to be highlighted and overlaid on the patient during surgery.” Slagmolen ¶ 47.
The secondary reference also teaches the limitation.);
receiving a change to a second planned location on the bone as presented using the augmented reality display for a second landmark of the plurality of landmarks; and removing, in response to receiving the change to the second planned location, a virtual indication of the second landmark at the second planned location (
Slagmolen teaches a change to a second planned location of a second landmark location, stating “For example, after entering into an ‘indicate point’ mode, a target symbol, such as cross hairs, may be displayed to the user in the display unit. These cross hairs may be then be moved by head or eye movement, until they point to the desired anatomical point, at which time the user can give the command to lock the position.” Slagmolen ¶ 154.
There is removing a virtual indication of the second landmark, because the cross hair the represent the desired anatomical point has been moved.
The new virtual indication is displayed, when the cross hair is moved to a new location.
The Examiner has already explained that the landmarks could be ones on the bone.).
Slagmolen teaches/suggests virtual indications are simultaneously displayed (Slagmolen further teaches the surgery plan may include highlighted/indicated landmarks at the planned locations, stating “During planning, the surgeon may choose the landmarks that he wishes to be highlighted and overlaid on the patient during surgery.” Slagmolen ¶ 47. ).
However, Slagmolen does not provide a figure that shows virtual indications are simultaneously displayed.
Wollowick teaches virtual indications can be simultaneously displayed (
PNG
media_image5.png
462
664
media_image5.png
Greyscale
PNG
media_image6.png
474
368
media_image6.png
Greyscale
).
Wollowick also teaches an indication of a location on the bone representing a surgeon-validated placement (“This anatomic feature, in conjunction with the stationary base, will depict any differences or errors in pelvic anatomy or the overlay which will enable the physician to validate, or to have more confidence in, the output of the present system. As generally utilized herein: (i) a “stationary point” refers to a point on a relatively stationary bone such as on the pelvis; (ii) a “landmark point” is located on an articulating bone such as a femur; (iii) an “error point” is preferably on pelvis and spaced from other points; and (iv) a “fixed point” is located on an implant, such as the shoulder of a femoral stem prosthesis.” Wollowick ¶ 52.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Wollowick’s simultaneously displaying visual indications and validation with Slagmolen. One of ordinary skill in the art would be motivated to visually compare landmarks to understand the change in location and/or to validate landmarks to enhance accuracy. Wollowick states, “In this construction of Step 3018, the Image Comparison Module 3036, FIG. 18 superimposes the preop and intraop images by aligning pelvic anatomy, with the images displayed with some transparency so that both can be visualized in the overlay, such as illustrated in FIG. 20. In a preferred construction the overlaid images will contain the identified femoral landmarks (generally placed on the greater trochanter) generated in step 3008 so that location differences between the two points can be visualized.” Wollowick ¶ 97.
Claim 22 is rejected under 35 U.S.C. 103 as being unpatentable over Slagmolen in view of Wollowick and Kuck as applied to Claim 1, in further view of Forutanpour et al. (US 20140078175 A1).
Regarding Claim 22, Slagmolen in view of Wollowick and Kuck teaches The method of claim 1.
Slagmolen in view of Wollowick and Kuck does not explicitly disclose wherein at least one of the two visually distinct virtual objects includes a visual effect including at least one of moving lights, flashing lights, or changing color lights.
Forutanpour teaches wherein at least one of the two visually distinct virtual objects includes a visual effect including at least one of moving lights, flashing lights, or changing color lights (“In some embodiments, rather than superimposing the virtual objects over a second face, and alert may be displayed to the user to remind the user that he is been looking at a particular virtual object for an extended period of time. For instance, a flashing light may be displayed to the user, the virtual object itself may be flashed or changed in appearance, the virtual object may be made more transparent, the virtual object may shrink in size, the virtual object may be hidden, etc. In some embodiments, throughout the period of time in which the user is viewing a virtual object, the virtual objects may become more transparent.” Forutanpour ¶ 94.).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Forutanpour’s alert with Slagmolen in view of Wollowick and Kuck. One of ordinary skill in the art would be motivated to invite a user’s attention. “In some embodiments, rather than superimposing the virtual objects over a second face, and alert may be displayed to the user to remind the user that he is been looking at a particular virtual object for an extended period of time. For instance, a flashing light may be displayed to the user, the virtual object itself may be flashed or changed in appearance, the virtual object may be made more transparent, the virtual object may shrink in size, the virtual object may be hidden, etc. In some embodiments, throughout the period of time in which the user is viewing a virtual object, the virtual objects may become more transparent.” Forutanpour ¶ 94.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Park et al. (US 20200337734 A1):
PNG
media_image7.png
710
528
media_image7.png
Greyscale
, where a physician is consulted on medical images, similar to Applicant’s claimed invention. However, Park does not disclose augmented-reality intraoperative decisions or details about virtual indications as claimed.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ZHENGXI LIU whose telephone number is (571)270-7509. The examiner can normally be reached M-F 9 AM - 5 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kee Tung can be reached at 571-272-7794. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ZHENGXI LIU/Primary Examiner, Art Unit 2611
Prosecution Insights