DETAILED ACTION
This action is in response to communications: Preliminary-Amendment filed November 4, 2024.
Claims 1-12, 15, and 16 are pending in this case. Claims 1-3, 5-12, and 15 have been newly amended. Claims 13 and 14 have been newly cancelled. Claim 16 has been newly added. This action is made Non-Final.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Information Disclosure Statement
The information disclosure statement (IDS) submitted on November 4, 2024 was filed on the filing date of the application on November 4, 2024. The submission is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Drawings
The drawings were received on November 4, 2024. These drawings are accepted.
Claim Objections
Claim 10 is objected to because of the following informalities:
Claim 10, currently depends upon claim 1, recites, “…wherein segmenting the X-ray image is performed…” where claim 1 does not provide proper antecedent basis for “segmenting the X-ray image” but claim 2 provides proper antecedent basis. Therefore, it is considered claim 10 should depend upon claim 2.
Appropriate correction is required.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 6-9, 11, 12, 15, and 16 is/are rejected under 35 U.S.C. 103 as being unpatentable over JUSTIN et al. (US 2018/0325618) in view of May et al. (US 2018/0256256).
As to claim 1, JUSTIN et al. disclose a computer-implemented method for rendering soft tissue into an X-ray image, the method comprising: providing an articulated three-dimensional anatomical model of at least one joint of a body (Figure 5, anatomical 3D model 236, [0043] notes anatomical 3D models 236 may be models of anatomical features of the surgical site, e.g. one or more models of bone surfaces involved in the surgical procedure, the 3D models 236 may be obtained from other data, such as image data 232, depth data 233, spatial data 234, and/or anatomical data 235 ([0042]), e.g. obtained via one or more image capture devices 330, where [0067] further notes techniques of generating/constructing the 3D models further include (1) Digital Image Correlation ([0068],[0069]); (2) Structured Illumination ([0070]-[0072]); (3) 3D Cameras ([0073]-[0076]); Time-of-Flight Sensors ([0077],[0078]); (5) Modulated Light LiDAR systems ([0079],[0080); (6) Polarized Light Imaging ([0081]-[0083]); and (7) Quality Control Systems ([0084]-[0086])), wherein the model comprises a rigid structure and at least one type of soft tissue, and wherein the rigid structure comprises at least two bones connected by the joint ([0105], [0133] notes the anatomical 3D models 236 constructed based on the anatomical features, e.g. defined by anatomical data 235, which may include hard and/or soft tissues, e.g. for knee replacement surgery, only the distal end of the femur and/or the proximal end of the tibia may be represented in the 3D models 236, e.g. knee joint); providing an X-ray image of a joint of a patient (Figure 5, image data 232 via one or more capture devices 330), wherein the joint of the patient corresponds to the joint of the anatomical model ([0033] notes images data 232 may include one or more images captured by one or more image capture devices, where [0059] notes the image capture device 330 may capture one or more images of an exposed anatomical features of the patient at the surgical sire 320, e.g. the exposed anatomical feature of the patient may include an exposed portion of a bone of the patient, such as an exposed portion of a tibial bone, a femoral bone, and/or a patellar bone of the patient, where [0066] further notes the one or more image capture devices 330 may include, but are not limited to: visual light cameras, photographic video cameras, light-field cameras, plenoptic light-field cameras, 3D cameras, depth sensing cameras, environment mapping cameras, LiDAR sensors, time of flight sensors, infrared cameras, X-ray imaging devices, and/or any combination thereof); registering the anatomical model to the X-ray image ([0116], [0128] notes the anatomical 3D model 236 may be registered to the exposed anatomical feature of the patient at the surgical site 320, 620 based on one or more images of the exposed anatomical feature, e.g. the x-ray images captured as image data 232 via one or more image capture devices 330, where the morphology of the anatomical 3D model 236 of the patient’s anatomical feature may be matched up with the actual morphology of the patent’s anatomical feature exposed at the surgical site 320, 620 during surgery); projecting…into the image domain of the X-ray image (e.g. via projector 340)([0090], [0091], [0117] notes projector 340,640 may be any device that can present data, such as guidance information 238, to a viewer, such as a surgeon, in the form of light projected on the patient or in the form of an augmented reality experience, e.g. guidance information 238 directly onto the surfaces of one or more tissues or bones, such as a tibia, a femur, and/or a patella, involved in the surgery, e.g. as overlaid projected images, which include 2D projections, 3D projections, holograms, partially translucent projections, opaque projections, simulated 3D projections and the like).
As noted above, JUSTIN et al. disclose its method for rendering into an X-ray image includes projecting images, e.g. guidance information, into the image domain of the X-ray image, but do not explicitly disclose “…projecting the at least one type of soft tissue into the image domain of the X-ray image; and visualizing the at least one type of soft tissue in the X-ray image.”
May et al. disclose a computer-implemented method for rendering soft tissue into an X-ray image, the method comprising…projecting the at least one type of soft tissue into the image domain of the X-ray image (e.g. projecting virtual representations of soft tissue anatomy, e.g. as virtual soft tissue overlaid on the surgical field of the patient); and visualizing the at least one type of soft tissue in the X-ray image (e.g. visualizing the soft tissue as an augmented reality (AR) display)(e.g. Figure 3, [0033]-[0036] notes the augmented reality (AR) display allows, for example, a femur bone 302 and a tibia bone 304 to be viewed, where the AR display 300 further includes, for example, virtual representation of an anterior cruciate (ACL) 306, a posterior cruciate ligament (PCL) 308, a lateral collateral ligament (LCL) 312, and a medial collateral ligament (MCL) 310, where other examples such as tendons, muscles, skin, blood vessels, or the like may also be displayed as virtual representations, the virtual soft tissue provided as an overlay in the AR display 300, where [0051] further notes the virtual representation may have a virtual position or orientation corresponding to a physical position or orientation of the physical feature, the relative physical position or orientation of the physical feature to the anatomical aspect may be determined preoperatively, such as using an x-ray, an MRI, or ultrasound).
It would have been obvious to one of ordinary skill in the art at the time of the invention to modify JUSTIN et al.’s system and method of projecting data/images over X-ray images, e.g. as augmented reality overlay, with May et al.’s method of projecting virtual representations of soft tissue anatomy as an augmented reality overlay to enhance visualization of hidden or hard to see anatomy and/or improve minimally invasive surgery (see at least [0013]-[0017] of May et al.).
As to claim 6, JUSTIN et al. modified with May et al. disclose the X-ray image is a three-dimensional X-ray image (JUSTIN, [0120] notes image capture devices 630 may include X-ray image devices, where [0042], [0132] notes 2D x-ray data, 3D x-ray data, and/or 2D x-ray data converted to 3D x-ray data).
As to claim 7, JUSTIN et al. modified with May et al. disclose the joint is a hip joint, a knee joint, an ankle joint, a shoulder joint, an elbow joint, or a wrist joint (JUSTIN, e.g. as noted in claim 1, example of knee joint, e.g. femur and tibia bones; modified with May, e.g. as noted in claim 1, example of knee joint, where [0017], [0060] further notes total or partial knee arthroplasty, shoulder replacement, hip arthroplasty, or other orthopedic implant procedures, thus joint may further include at least shoulder joint and hip joint), and/or wherein the soft tissue comprises a ligament, a muscle, a cartilage, a tendon, or a skin (modified with May, e.g. as noted in claim 1, soft tissue may include tendons, muscles, skin, blood vessels, or the like).
As to claim 8, JUSTIN et al. modified with May et al. disclose the articulated three-dimensional anatomical model is built from a dedicated or routine examination of a study subject and/or from an anatomical atlas (JUSTIN, e.g. as noted in claim 1, the anatomical 3D model may be constructed from image data 232, depth data 233, spatial data 234, and/or anatomical data 235, and additional techniques (1)-(7) described, where at least [0042] notes the anatomical data 235 may provide details regarding the anatomy of the patient, e.g. dimensions of one or more bones involved in a surgical procedure, data regarding quality, porosity, or other characteristics of the bone and/or the like, the anatomical data 235 may be obtained by processing other data, such as the image data 232, depth data 233, and/or spatial data 234, or may be obtained directly by scans of the surgical site, such as CT scans, MRI scans, MRT scans, 2D x-ray scans, 3D x-ray scans fluoroscopy, and the like, thus may be considered at least a “dedicated examination of a study subject,” e.g. the patient).
As to claim 9, JUSTIN et al. modified with May et al. disclose the articulated three-dimensional anatomical model comprises a position and a shape of the rigid structure (JUSTIN, e.g. as noted in claim 1, anatomical 3D models 236 may be models of anatomical features of the surgical site, e.g. one or more models of bone surfaces involved in the surgical procedure, the 3D models 236 may be obtained from other data, such as image data 232, depth data 233, spatial data 234, and/or anatomical data 235, where further techniques of generating/constructing the 3D models may include at least (2) Structured Illumination ([0070]-[0072]) which notes contouring, positions, and/or orientations of pattern elements may be measured to ascertain the shape of the underlaying surfaces on which the pattern is projected, and (3) 3D camera ([0073]-[0076]) which notes the shapes, positions, and/or orientations of objects in the scene may be obtained by comparing images) and the at least one type of soft tissue over an articulation range of the joint (modified with May, e.g. as noted in claim 1, Figure 3, the virtual representation of the soft tissue may be projected over the surgical field of the patient, e.g. over an articulation range of the joint, e.g. knee joint as described).
As to claim 11, JUSTIN et al. modified with May et al. disclose visualizing the at least one type of soft tissue in the X-ray image comprises determining an intensity change in the X-ray image and adapting the visualization of the at least one type of soft tissue based on the determined intensity change (modified with May, [0034] notes the virtual soft tissue may augment the actual tissue, e.g. the virtual soft tissue may be color coded or include a high contrast color (e.g. ligaments in a first color and tendons in a second, or each ligament may be individually color coded), the virtual soft tissue may outlined or highlight the actual soft tissue, [0036] further notes the AR display may show the virtual soft tissue when a device (e.g. a surgical instrument such as a saw blade, burr, rasp, reamer, etc.) approaches or enters a danger zone surrounding the actual soft tissue, where the AR display 300 may alert the surgeon, such as by flashing a warning or increasing intensity of light or changing color of the virtual soft tissue when the danger zone is approached or entered).
As to claim 12, JUSTIN et al. modified with May et al. disclose visualizing the at least one type of soft tissue in the X-ray image comprises rendering the soft tissue as an additional contrast, tone mapping, color overlay, or hue modulation (modified with May, see claim 11).
As to claim 15, JUSTIN et al. modified with May et al. disclose a non-transitory computer-readable storage medium (JUSTIN, e.g. Figure 1, computing device 120 implemented as a desktop computer 122 of Figure 2, comprising memory 220; modified with May, Figure 9, memory 904, static memory 906, and/or mass storage 916) comprising executable instructions (JUSTIN, [0031] notes desktop computer may include memory 220 which may further includes one or more memory modules and executable instructions, data referenced by such executable instructions, and/or any other data that may be beneficially be made readily accessible to processor 210; modified with May, each of memories 904, 906, and 916 storing instructions 924) which, when executed by at least one processor (JUSTIN, processor 210, [0030] notes processor 210 designed to execute instructions on data, e.g. the instruction and data stored in memory 220; modified with May, processor 902 to execute instructions 924, [0071]-[0073]), cause the at least one processor to perform a method for rendering soft tissue into an X-ray image (JUSTIN, modified with May, rendering soft tissue into an X-ray image), the method comprising the steps as performed by the method as outlined in claim 1. Please see the rejection and rationale of claim 1 above.
As to claim 16, JUSTIN et al. modified with May et al. disclose a system (JUSTIN, e.g. Figure 1, computing device 120 implemented as a desktop computer 122 of Figure 2; modified with May, machine 900) for rendering soft tissue into an X-ray image (JUSTIN, modified with May, rendering soft tissue into an X-ray image), comprising: a memory that stores a plurality of instructions (JUSTIN, memory 220, [0031] notes desktop computer may include memory 220 which may further includes one or more memory modules and executable instructions, data referenced by such executable instructions, and/or any other data that may be beneficially be made readily accessible to processor 210; modified with May, Figure 9, memory 904, static memory 906, and/or mass storage 916 storing instructions 924); and a processor coupled to the memory and configured to execute the plurality of instructions (JUSTIN, processor 210, [0030] notes processor 210 designed to execute instructions on data, e.g. the instruction and data stored in memory 220; modified with May, processor 902 to execute instructions 924, [0071]-[0073]) to perform the steps of the method as outlined in claim 1. Please see the rejection and rationale of claim 1 above.
Allowable Subject Matter
Claims 2-5 and 10 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter:
Regarding dependent claim 2, the prior art of record fails to teach or suggest, singly or combined, the limitations of the claim as recited. Dependent claims 3-5 and 10 (as noted in the claim objection above) are indicated allowable subject matter for directly or indirectly depending upon dependent claim 2.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Yi et al. (US 11,373,308) disclose a system and method of obtaining a first X-ray image of an object including first material, e.g. bone, and second material, tissue; obtaining three-dimensional (3D) information about the object; obtaining first information about a thickness of the object based on the 3D information; and obtaining second information related to a stereoscopic structure of the first material by decomposing the first material from the object using the first information and the first X-ray image;
Gibby et al. (US 2023/0169740) disclose a system and method of aligning an image data set with a patient using augmented reality; and
Amanatullah (US 2019/0231432) disclose a system and method of augmenting a surgical field with virtual guidance and tracking and adapting deviation from a surgical plan.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JACINTA M CRAWFORD whose telephone number is (571)270-1539. The examiner can normally be reached 8:30a.m. to 4:30p.m.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, King Y. Poon can be reached at (571)272-7440. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JACINTA M CRAWFORD/Primary Examiner, Art Unit 2617
Prosecution Insights