DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment / Arguments
Claim Objections. Applicant’s amendment overcomes the objection to the claims.
103 Rejections. The amendments to the dependent claims, and some of those to the independent claims, are respectfully stylistic. Regarding the substantive amendments to the independent claims, Applicant’s arguments have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-15 and 26-30 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lang (U.S. Patent App. Pub. No. 2020/0305980 A1; cited in IDS) in view of Nett (U.S. Patent App. Pub. No. 2019/0287239 A1), and further in view of Lang 484 (U.S. Patent App. Pub. No. 2022/0133484 A1) and Harris (U.S. Patent App. Pub. No. 2023/0005232 A1).
Regarding claim 1:
Lang teaches: a system (claim 1, a system) for image-guided surgery (claim 1, the system can “generate a virtual surgical guide), comprising:
a near-eye unit (claim 1, the system comprises a “see-through optical head mounted display” (hereinafter: OHMD)), comprising;
a see-through augmented-reality display (para 77, the see-through OHMD has a display, in combination with para. 116, the OHMDs can be for augmented reality), which is configured to display graphical information with respect to a region of interest (ROI) on a body of a patient, including a bone inside the body (para. 1649, bones can be a ROI. The examiner is interpreting this claim feature such that the ROI includes, or comprises, a bone), that is viewed through the see-through augmented reality display by a user wearing the near- eye unit (see for example Fig. 3 and related description. The “digital hologram” of Fig. 3 corresponds to a display of graphical info with respect to a ROI on a body on a patient (the digital hologram/graphical info is superimposed on a live patient), can be viewed by a user wearing the OHMD:
Quoting para. 263, in part (which is part of the description of Fig. 3): “The operator can move the OHMD until digital hologram of the virtual data or virtual data of the patient is superimposed and aligned with the live data of the patient, e.g. the surgical site 57. The digital hologram of the virtual data or virtual data can then be registered using the same or similar coordinates as those of the live data with which the digital hologram is superimposed 58. The surgeon can then perform one or more predetermined surgical steps, e.g. bone cuts 59. A digital hologram of the virtual data or virtual data can optionally be registered or re-registered after the surgical alteration with the live data 60. The digital hologram of the virtual data or virtual data after the surgical alteration can optionally be displayed by the OHMD 61. The digital hologram of the virtual data or virtual data after the surgical alteration can optionally be fixed relative to the OHMD so that it will move with the movement of the OHMD 62.”); and
a depth sensor, which is configured to generate depth data with respect to the ROI (para. 33, depth sensors for depth data with respect to the ROI, which can be any one of the surgical site, joint, spine or implant components. See also para. 255 and 1366); and
a processor (claim 1, system has a processor), which is configured to, upon execution of software stored on tangible, non-transitory computer-readable media (paras. 203-04, can have software stored in memory):
access three-dimensional (3D) image data with respect to the bone (para. 236, user can access 3D image data, such as of a bone (e.g. hip, spine, knee)).
Regarding: identify, by processing the 3D image data using a convolutional neural network a first 3D shape of the bone prior to a surgical procedure on the bone, the first 3D shape including a portion of the bone to be removed, consider the following.
Nett teaches that it is known to use a convolutional neural network (CNN) (see paras. 8, 9) to process 3D image data to identify a first 3D shape of an anatomical object. See Nett, Fig. 5, which illustrates a flow chart for utilizing a trained neural network to detect or remove objects from 3D CTA (computed tomography angiography) data. In terms of the 3D image data being that of a bone prior to a surgical procedure,
the first 3D shape including a portion of the bone to be removed, Harris teaches using a CNN (Harris, para. 78), to process 3D image data (i.e. data can be from an MRI or ultrasound, see para. 80, either image modality comprises or includes “3D image data”), to generate a 3D model (first 3D shape of a bone) including a portion of the bone removed (para. 81, remove the detected bone aberration from the 3D model). See also Harris, paras. 78-81 and Fig. 8. Likewise, Lang and Lang 484 both respectively teach using neural networks for pre-imaging studies. (Lang, paras. 201-202) (Lang 484, paras. 49-50). Lang 484 is also relevant to bone removal procedures (see e.g. para. 40).
Modifying the applied references, in view of Nett, such to include a trained CNN to process 3D image data prior to a surgical procedure, per both Lang references, such to identify a first 3D shape including a portion of a bone to be removed, as taught/suggested by both Lang 484 and Harris, is all of taught and suggested by the prior art, and would have been obvious and predictable to one of ordinary skill.
Regarding the remaining features of claim 1, consider the following.
process a second 3D shape of the bone following the surgical procedure (Lang 484, para. 86, the second shape following procedure can be postoperative data. The above mapping the previous process function also applies here),
measure, based on the depth data, a volume of the bone that was removed during the surgical procedure (Lang, para. 242, which teaches 3D reconstruction of image data or other patient data can be done intraoperative and/or postoperatively. Lang also teaches that the system can uses images to perform volume measurements (para. 456). Modifying Lang, in view of itself, such that a volume measurement is done using depth data (i.e. image data), whereby the volume is measured post-operatively (i.e. volume of bone removed), whereby Lang also teaches bone removal surgical procedures (para. 769, 818), is all of taught and suggested by Lang, and would have been obvious and predictable to one of ordinary skill),
generate, based on the first 3D shape and the second 3D shape and using the depth data, an image showing the volume of the bone that was removed during the surgical procedure (see Lang, Figs. 3: 60, 61, 69, 62, 63, 64, 65, 66, which teach/illustrate generating holograms to superimpose post-operatively, or after bone cut, in this example (Lang is not limited to one specific medical procedure). Modifying Lang, such that the holograms correspond to volume of bone removed as holographic virtual data, is all of taught and suggested by Lang, and would have been obvious and predictable to one of ordinary skill), and
present the image on the see-through augmented-reality display (Fig. 3).
Fig. 3 of Lang is reproduced below for convenience:
PNG
media_image1.png
730
552
media_image1.png
Greyscale
It would have been obvious for one of ordinary skill in the art to have combined and modified the applied reference(-s), in view of same, to have obtained the above, and the results of the modification would have been obvious and predictable to one of ordinary skill in the art as of the effective filing date of the claimed invention. See MPEP §2143(A).
The prior art included each element recited in claim 1, although not necessarily in a single embodiment, with the only difference being between the claimed element and the prior art being the lack of actual combination of certain elements in a single prior art embodiment, as described above.
One of ordinary skill in the art could have combined the elements as claimed by known methods, and in that combination, each element merely performs the same function as it does separately. One of ordinary skill in the art would have also recognized that the results of the combination were predictable as of the effective filing date of the claimed invention.
Regarding claim 2:
Lang teaches: the system according to claim 1, wherein the processor is further configured to access a plan of the surgical procedure (Fig. 11: 141 access virtual surgical plan) and present, based on the plan, a guide for cutting the bone on the see- through augmented-reality display (see Fig. 11, reproduced below. Applying this to a procedure for cutting the bone (i.e. “virtual cut planes”, see Fig. 11: 141, or para. 146, a “bone cut” is also taught by Lang).
PNG
media_image2.png
544
736
media_image2.png
Greyscale
It would have been obvious for one of ordinary skill in the art, as of the effective filing date of Applicant’s claims, to have further modified the applied references, in view of Lang, to have obtained the above, motivated to improve patient procedures with access to plan and/or available data.
Regarding claim 3:
Lang teaches: the system according to claim 2, wherein the processor is further configured to compare the second 3D shape to the plan and present, on the see-through augmented-reality display, an indication of a deviation between the volume of the bone that was removed and the plan (Lang teaches that intra-operative and post-operative assessments can be made, and compared to intended results (see e.g. para. 738, in part: “The surgeon can perform a surgical step 80. The surgeon can then assess the actual changes induced in the live patient 81. The surgeon can compare the actual changes induced in the live patient with the predetermined changes in the virtual data of the patient, e.g. in a virtual surgical plan or in a virtual 3D display 82. The magnitude of the difference(s) between the actual and the predetermined changes can be determined 83. If they are acceptable 84, the surgeon can perform the next surgical step 85”, Also, while the example here is for tissue removal, Lang isn’t limited to one specific medical procedure. Applying this to bone removal would have been an obvious modification and/or embodiment taught by Lang). This is also illus. in Fig. 6, reproduced below:
PNG
media_image3.png
534
728
media_image3.png
Greyscale
It would have been obvious for one of ordinary skill in the art, as of the effective filing date of Applicant’s claims, to have further modified the applied references, in view of Lang, to have obtained the above, motivated to improve patient procedures with access to plan and/or available data in order to make changes as needed.
Regarding claim 4:
It would have been obvious for one of ordinary skill in the art to have further modified the applied reference(-s), in view of same, to have obtained: the system according to claim 2, wherein the processor is further configured to present, the see-through augmented-reality display, the guide as an outline of an area of the bone that is to be removed, wherein the outline is superimposed on the bone in the see-through augmented-reality display, and the results of the modification would have been obvious and predictable to one of ordinary skill in the art as of the effective filing date of the claimed invention. See MPEP §2143(A).
Lang teaches that displaying outlines is known (para. 741-42). Example: Fig. 7B: shows a cross-section or top view of the intended virtual femoral neck cut (broken outline) 97, for example as developed in the virtual surgical plan. Other examples: Figs. 7D and 7E. Modifying the applied references, in view of Lang, such that the guide is an outline area, per Lang, of a bone to be removed, also procedures taught by Lang, and superimposed, as mapped above, is all of taught and suggested by Lang, and would have been obvious and predictable to one of ordinary skill in the art.
One of ordinary skill in the art could have combined the elements as claimed by known methods, and in that combination, each element merely performs the same function as it does separately. One of ordinary skill in the art would have also recognized that the results of the combination were predictable as of the effective filing date of the claimed invention.
Regarding claim 5:
It would have been obvious for one of ordinary skill in the art to have further modified the applied reference(-s), in view of same, to have obtained: the system according to claim 2, wherein the processor is further configured to present on the see-through augmented-reality display an icon indicating a position of a tool used in cutting the bone and a line showing a trajectory that the tool is to take in cutting the bone according to the plan (Lang, para. 5, virtual representation of a virtual surgical tool teaches the icon See also para 70: “Aspects of the invention relates to devices, systems and methods for positioning a virtual path, virtual plane, virtual tool, virtual surgical instrument or virtual implant component in a mixed reality environment using a head mounted display device….”
And para. 72: “the OHMD can display one or more of a virtual surgical tool, virtual surgical instrument including a virtual surgical guide or virtual cut block, virtual trial implant, virtual implant component, virtual implant or virtual device, predetermined start point, predetermined start position, predetermined start orientation or alignment, predetermined intermediate point(s), predetermined intermediate position(s), predetermined intermediate orientation or alignment, predetermined end point, predetermined end position, predetermined end orientation or alignment, predetermined path, predetermined plane, predetermined cut plane, predetermined contour or outline or cross-section or surface features or shape or projection, predetermined depth marker or depth gauge, predetermined angle or orientation or rotation marker,….” Applying this teaching of Lang to a bone cutting, also taught by Lang and mapped above, is all of taught and suggested by Lang, and would have been obvious and predictable to one of ordinary skill in the art.
One of ordinary skill in the art could have combined the elements as claimed by known methods, and in that combination, each element merely performs the same function as it does separately. One of ordinary skill in the art would have also recognized that the results of the combination were predictable as of the effective filing date of the claimed invention.
Regarding claim 6:
Lang teaches: the system according to claim 2, wherein the processor is further configured to present the guide on the see-through augmented-reality display together with an image of the bone, wherein the guide and the image of the bone are overlaid on an actual location of the bone in the body (see Fig. 11, a virtual hologram of a bone overlaid on live data (bone in the body), as virtual patient data, is also taught by Lang. See also para. 267, the virtual data can be of patient body parts, a bone being the body part is one embodiment taught by Lang).
It would have been obvious for one of ordinary skill in the art, as of the effective filing date of Applicant’s claims, to have further modified the applied references, in view of Lang, to have obtained the above, motivated to improve patient procedures with access to plan and/or available data in order to make changes as needed.
Regarding claim 7:
Lang teaches: the system according to claim 2, wherein the processor is further configured to present the guide on the see-through augmented-reality display wherein the guide is overlaid on actual bone that is to be cut in open surgery (e.g. Figs. 6 and 11, as mapped or shown above. See also paras. 4-6).
It would have been obvious for one of ordinary skill in the art, as of the effective filing date of Applicant’s claims, to have further modified the applied references, in view of Lang, to have obtained the above, motivated to improve patient procedures with access to plan and/or available data in order to make changes as needed.
Regarding claim 8:
Lang teaches: the system according to claim 1, wherein the processor is further configured to identify by processing the 3D image data at one or more times during the surgical procedure, one or more intermediate 3D shapes of the bone during the surgical procedure, and present, on the see-through augmented-reality display, the volume of the bone removed at each of the one or more times (see para. 958,968, surgeon can perform intra-operative measurements (done during the procedure), using intra-operative imaging. See also Figs. 2 and 14. Modifying Lang, in view of same, such to perform intra-operative measurements to identify one or more intermediate 3D shapes of bone, and present volume of bone removed (see also mapping to claim 3 above), is all of taught and suggested by Lang, and would have been obvious and predictable to one of ordinary skill in the art.
One of ordinary skill in the art could have combined the elements as claimed by known methods, and in that combination, each element merely performs the same function as it does separately. One of ordinary skill in the art would have also recognized that the results of the combination were predictable as of the effective filing date of the claimed invention. Additional motivation can be found in a desire to manage surgical procedures and be able to correct or adjust for changes.
Regarding claim 9:
Lang teaches: the system according to claim 1, wherein the depth sensor comprises a pattern projector and a camera configured for performing structured-light depth mapping (paras. 255-56, a structured light depth sensor includes a pattern projector and camera).
It would have been obvious for one of ordinary skill in the art, as of the effective filing date of Applicant’s claims, to have further modified the applied references, in view of Lang, to have obtained the above, motivated to make use of known equipment for imaging.
Regarding claim 10:
Lang teaches: the system according to claim 1, wherein the depth sensor comprises a pair of cameras configured for stereoscopic depth mapping (para. 1183, stereoscopic 3D is one embodiment. See also para. 1294-96, 1300).
It would have been obvious for one of ordinary skill in the art, as of the effective filing date of Applicant’s claims, to have further modified the applied references, in view of Lang, to have obtained the above, motivated to make use of known equipment for imaging.
Regarding claim 11:
Lang teaches: the system according to claim 1 wherein the depth sensor comprises a beam projector and one or more detectors configured for time-of-flight measurement (para. 255-56, time of flight 3D sensor; these sensors emit pulses of light (i.e. a beam projector) and measure the time it takes for the light to reflect off the object and return to the sensor (detector)).
It would have been obvious for one of ordinary skill in the art, as of the effective filing date of Applicant’s claims, to have further modified the applied references, in view of Lang, to have obtained the above, motivated to make use of known equipment for imaging.
Regarding claim 12:
Lang teaches: the system according to claim 1, wherein the processor is further configured to access 3D tomographic data with respect to the body of the patient and generate, using the 3D tomographic data, the image showing the volume of the bone that was removed (Lang, para. 690, 3D CT data is known for patient imaging). Modifying the applied references, in view of same, such to include 3D CT data to show volume of the bone removed as input data to the neural network of Harris (which, per para. 80 of Harris, can receive CT imaging data), would have been obvious and predictable to one of ordinary skill. See MPEP 2143(A)).
One of ordinary skill in the art could have combined the elements as claimed by known methods, and in that combination, each element merely performs the same function as it does separately. One of ordinary skill in the art would have also recognized that the results of the combination were predictable as of the effective filing date of the claimed invention.
Regarding claim 13: see also claim 1.
a method for image-guided surgery (functions performed by system of claim 1), comprising:
identifying by processing first three-dimensional (3D) image data with respect to a bone inside a body of a patient using a convolutional neural network (CNN) a first 3D shape of the bone prior to a surgical procedure on the bone ,the first 3D shape including a portion of the bone to be removed (claim 1);
identifying by processing second 3D image data a second 3D shape of the bone following the surgical procedure (claim 1);
generating, based on the first 3D shape and the second 3D shape, an image showing a volume part of the bone that was removed in the surgical procedure (claim 1); and
presenting the image on a see-through augmented-reality display (claim 1), wherein the image is overlaid on a region of interest (ROI) on the body of the patient that contains the bone inside the body and is viewed through the see-through augmented0reality display (e.g. Lang, Fig. 3),
wherein the first 3D image data and the second 3D image data are acquired by a depth sensor (depth sensor of claim 1), and
generating the image showing the volume of the bone that was removed in the surgical procedure comprises measuring, based on depth data acquired by the depth sensor, the volume of the bone that was remove and using the depth data to generate the image (claim 1).
Regarding claim 14: see claim 2.
These claims are similar; the same rationale for rejection applies.
Regarding claim 15: see claim 3.
These claims are similar; the same rationale for rejection applies.
Regarding claim 26: see also claim 1.
a system for image-guided surgery comprising (Claim 1):
a near-eye unit comprising: a see-through augmented-reality display (claim 1) configured to be worn on a head of a user (Lang, OHMD of claim 1) and to display a bone with respect to a region of interest (ROI) on a body of a patient, the bone being disposed inside the patient (claim 1); and
a depth sensor (claim 1); and
a processor and a memory for storing instructions that, when executed by the processor (claim 1), cause the system to:
access three-dimensional (3D) image data related to the bone ( claim1)(;
determine, by processing the 3D image data using a convolutional neural network, a first 3D shape of the bone prior to removing a portion of the bone, the first 3D shape including the portion to be removed, (first process step of claim 1);
determine a second 3D shape of the bone after the portion is removed (second process step of claim 1) ;
measure, based on depth data acquired by the depth sensor, a volume of the bone that was removed (Claim 1);
generate, based on the depth data, an image of the volume of the bone that was removed (Claim 1); and
display the image on the see-through augmented-reality display wherein the image is viewable by the user in the ROI on the body of the patient (claim 1 and Lang, Fig. 3).
Regarding claim 27: see claim 8.
the system according to claim 26, wherein the instructions are further configured to cause the system to determine one or more intermediate 3D shapes [clam 8: one or more times during the procedure corresponds to between the first and second 3D shape, since the first was before removal of the bone portion, and the second was after the bone was removed] at one or more times, and to display, on the see-through augmented- reality display, an image of a volume of the bone removed at each of the one or more times (claim 8).
Regarding claim 28: see any one of claims 9-11.
Claim 28 corresponds to any one of the above enumerated claims.
Regarding claim 29: see claim 12.
These claims are similar; the same rationale for rejection applies.
Regarding claim 30:
Lang 484 teaches: the system according to claim 1, wherein the convolutional neural network is trained based on a type of the surgical procedure and using 3D computed tomography scans (Lang 484, para. 49, training can be done with pre, intra and post-operative data including CT scans. Lang teaches that 3D CT scans are known (see para. 690) as scans and patient data. Training data that includes types of surgical procedures, types being categories or classifications, and training data can include treatment plans (Lang 484, paras. 68, 86, 87), in combination with para. 52: “treatment plan” includes surgical procedures).
Modifying the applied references, in view of same, to have included the above, is all of taught and suggested by Lang, and would have been obvious and predictable to one of ordinary skill in the art. MPEP 2143(A).
One of ordinary skill in the art could have combined the elements as claimed by known methods, and in that combination, each element merely performs the same function as it does separately. One of ordinary skill in the art would have also recognized that the results of the combination were predictable as of the effective filing date of the claimed invention.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
* * * * *
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Sarah Lhymn whose telephone number is (571)270-0632. The examiner can normally be reached M-F, 9:00 AM to 6:00 PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Xiao Wu can be reached at 571-272-7761. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
Sarah Lhymn
Primary Examiner
Art Unit 2613
/Sarah Lhymn/Primary Examiner, Art Unit 2613