DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on March 30, 2026 has been entered.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are: “2D image device configured to acquire” and “machine learning engine configured to generate” in claim 118.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 107-109, 111-114, 118, 135, 136, 138-141 and 143 are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Tolkowsky et al. (WO 2017/158592) and Oved (US 10,867,436).
Regarding claim 107, Tolkowsky et al. discloses a method for performing a procedure with respect to a skeletal portion within a body of a subject, the method comprising:
(A) acquiring with a 2D x-ray imaging device two skeletal-portion 2D x-ray images of at least the skeletal portion (“For some applications, the 3D imaging that is used is CT imaging, and the following explanation of the registration of the 3D image data to the 2D images will focus on CT images. However, the scope of the present invention includes applying the techniques describe herein to other 3D imaging modalities, such as MRI and 3D x-ray” at page 55, line 26; 3D x-ray implies at least two 2D x-ray acquisition);
(B) generating from the skeletal-portion 2D x-ray images 3D image data of at least the skeletal portion (the 3D x-ray is generated using the at least two x-ray images);
(C) while a portion of a tool is disposed at a location with respect to the skeletal portion, sequentially:
acquiring a first tool 2D x-ray image of at least the portion of the tool and the skeletal portion from a first view, the first view being similar to the view of one of the skeletal-portion 2D x-ray images used for generating the 3D image data, the acquisition of the first tool 2D x-ray image being performed by the 2D x-ray imaging device while the 2D imaging device is disposed at a first pose with respect to the subject's body (“In a fifth step 78, the first tool in the sequence of tools (which is typically a needle, e.g., a Jamshidi™ needle) is typically inserted into the subject (e.g., in the subject's back), and is slightly fixated in the vertebra. In a sixth step 80, two or more 2D radiographic images are acquired from respective views that typically differ by at least 10 degrees (and further typically by 30 degrees or more), and one of which is typically from the direction of insertion of the tool. Typically, generally-AP and generally-lateral images are acquired. Alternatively or additionally, images from different views are acquired” at page 43, line 23),
moving the 2D x-ray imaging device to a second pose with respect to the subject's body (as described above, the device is moved to differ the degree of the view between images), and
while the 2D x-ray imaging device is at the second pose, acquiring with the 2D x-ray imaging device a second tool 2D x-ray image of at least the portion of the tool and the skeletal portion from a second view, the second view being similar to the view of the other one of the skeletal-portion 2D x-ray images used for generating the 3D image data (“It is noted, that, as described in further detail hereinbelow, for some applications, in order to perform step 84, the computer processor need acquire one or more 2D x-ray images of a tool at a first location inside the vertebra from only a single x-ray image view, and the one or more 2D x-ray images are registered to the 3D image data by generating a plurality of 2D projections from the 3D image data, and identifying a 2D projection that matches the 2D x-ray images of the vertebra” at page 44, line 7),
wherein during the acquisition of the skeletal-portion 2D x-ray images used for generating the 3D image data the portion of the tool was either (i) not visible in the skeletal-portion 2D x-ray images (“Applying pre-operative 3D visibility (e.g., from CT and/or MRI) during the intervention. It is noted that 3D visibility provides desired cross-sectional images (as described in further detail hereinbelow), and is typically more informative and/or of better quality than that provided by intraoperative 2D images” at page 42, line 27; as the 3D data is acquired pre-operatively, it is understood that this prior to any tools being introduced to the patient), or (ii) disposed at a different location with respect to the skeletal portion than the location in which the tool is disposed during the acquisition of the first and second tool 2D x-ray images; and
(D) using at least one computer processor:
registering the first and second tool 2D x-ray images to the 3D image data (“In a seventh step 82, computer processor 22 of system 20 typically registers the 3D image data to the 2D images” at page 43, line 30),
identifying a location of the portion of the tool with respect to the skeletal portion, within the first and second tool 2D x-ray images (“In response to registering the one or more 2D x-ray images acquired from the single x-ray image view to the 3D image data, the computer processor drives a display to display a cross-section derived from the 3D image data at a the first location of a tip of the tool, as identified from the one or more 2D x-ray images, and optionally to show a vertical line on the cross-sectional image indicating a line within the cross-sectional image somewhere along which the first location of the tip of the tool is disposed. Typically, when the tip of the tool is disposed at an additional location with respect to the vertebra, further 2D x-ray images of the tool at the additional location are acquired from the same single x-ray image view, or a different single x-ray image view, and the above-described steps are repeated. Typically, for each location of the tip of the tool to which the above-described technique is applied, 2D x-ray images need only be acquired from a single x-ray image view, which may stay the same for the respective locations of the tip of the tool, or may differ for respective locations of the tip of the tool” at page 44, line 12), and
based upon the identified location of the portion of the tool within the first and second tool 2D x-ray images, and the registration of the first and second tool 2D x-ray images to the 3D image data, determining the location of the portion of the tool with respect to the 3D image data (“In step 90, the computer processor overlays an image of the tool, a representation thereof, and/or a representation of the tool path upon the 3D image data (e.g., a 3D image, a 2D cross-section derived from 3D image data, and/or a 2D projection image derived from 3D image data), the location of the tool or tool path having been derived from current 2D images” at page 44, line 31).
Tolkowsky et al. does not explicitly disclose acquiring with a 2D x-ray imaging device only two skeletal-portion 2D x-ray images of at least the skeletal portion and generating from the only two skeletal-portion 2D x-ray images 3D image data of the at least skeletal portion using machine learning techniques.
Oved teaches a method in the same field of endeavor of 3D x-ray image generation, the method comprising:
(A) acquiring with a 2D x-ray imaging device only two skeletal-portion 2D x-ray images of at least the skeletal portion (“Reference is now made to FIG. 6, which is a diagram depicting an exemplary architecture of the neural network for mapping two 2D anatomical images into a 3D point cloud, in accordance with some embodiments of the present invention. The two 2D images may be x-ray images. The two 2D images may be of different views of the target anatomical image, for example, an anterior-posterior (AP) and lateral views” at col. 18, line 44; “For example, the 2D anatomical image(s) are x-ray image(s) outputted by an x-ray machine, for example, AP and/or lateral views (e.g., of the chest, femur)” at col. 19, line 12);
B) generating from the skeletal-portion 2D x-ray images 3D image data of at least the skeletal portion, using machine learning techniques (“At 306, the 2D anatomical image(s) are fed into the neural network” at col. 19, line 20; “At 308, a reconstructed 3D point cloud model of the target anatomical structure is outputted by the trained neural network, as described herein” at col. 19, line 22).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to utilize a machine learning algorithm as taught by Oved to construct the pre-operative 3D data of Tolkowsky et al. as the 2D images have “relatively lower radiation, lower cost, faster acquisition time, and provides high quality treatment planning for patients where 3D imaging devices are unavailable” (Oved at col. 6, line 48).
Regarding claim 118, Tolkowsky et al. discloses an apparatus for performing a procedure with respect to a skeletal portion within a body of a subject, the apparatus for use with a 2D x-ray imaging device configured to acquire 2D x-ray images of at least the skeletal portion (“3D x-ray data are acquired” at page 32, line 25), the apparatus comprising:
an engine configured to generate from two skeletal-portion 2D x-ray images (“For some applications, the 3D imaging that is used is CT imaging, and the following explanation of the registration of the 3D image data to the 2D images will focus on CT images. However, the scope of the present invention includes applying the techniques describe herein to other 3D imaging modalities, such as MRI and 3D x-ray” at page 55, line 26; 3D x-ray implies at least two 2D x-ray acquisition) 3D image data of at least the skeletal portion (the 3D x-ray is generated using the at least two x-ray images); and
at least one computer processor configured to:
A) receive the 3D image data from the engine (“In a first step 70, targeted vertebra(e) are marked by an operator with respect to 3D image data (e.g., a 3D image, a 2D cross-section derived from 3D image data, and/or a 2D projection image derived from 3D image data) of the subject's spine” at page 43, line 13),
B) receive from the 2D x-ray imaging device, while the 2D x-ray imaging device is at a first pose with respect to the subject's body, a first tool 2D x-ray image, from a first view of the skeletal portion and of at least a portion of a tool disposed at a location with respect to the skeletal portion, the first view being similar to the view of one of the skeletal-portion 2D x-ray images used for generating the 3D image data (“In a fifth step 78, the first tool in the sequence of tools (which is typically a needle, e.g., a Jamshidi™ needle) is typically inserted into the subject (e.g., in the subject's back), and is slightly fixated in the vertebra. In a sixth step 80, two or more 2D radiographic images are acquired from respective views that typically differ by at least 10 degrees (and further typically by 30 degrees or more), and one of which is typically from the direction of insertion of the tool. Typically, generally-AP and generally-lateral images are acquired. Alternatively or additionally, images from different views are acquired” at page 43, line 23),
(C) receive from the 2D x-ray imaging device, while the 2D x-ray imaging device is at a second pose with respect to the subject's body, a second tool 2D x-ray image, from a second view, of the skeletal portion and of at least the portion of the tool disposed at the location with respect to the skeletal portion, the second view being similar to the view of the other one of the skeletal-portion 2D x-ray images used for generating the 3D image data (“It is noted, that, as described in further detail hereinbelow, for some applications, in order to perform step 84, the computer processor need acquire one or more 2D x-ray images of a tool at a first location inside the vertebra from only a single x-ray image view, and the one or more 2D x-ray images are registered to the 3D image data by generating a plurality of 2D projections from the 3D image data, and identifying a 2D projection that matches the 2D x-ray images of the vertebra” at page 44, line 7; as described above, the device is moved to differ the degree of the view between images),
wherein during the acquisition of the skeletal-portion 2D x-ray images used for generating the 3D image data the portion of the tool was either (i) not visible in the skeletal-portion 2D x-ray images (“Applying pre-operative 3D visibility (e.g., from CT and/or MRI) during the intervention. It is noted that 3D visibility provides desired cross-sectional images (as described in further detail hereinbelow), and is typically more informative and/or of better quality than that provided by intraoperative 2D images” at page 42, line 27; as the 3D data is acquired pre-operatively, it is understood that this prior to any tools being introduced to the patient), or (ii) disposed at a different location with respect to the skeletal portion than the location in which the tool is disposed during the acquisition of the first and second tool 2D x-ray images;
(D) register the first and second tool 2D x-ray images to the 3D image data (“In a seventh step 82, computer processor 22 of system 20 typically registers the 3D image data to the 2D images” at page 43, line 30),
(E) identify a location of the portion of the tool with respect to the skeletal portion, within the first and second tool 2D x-ray images (“In response to registering the one or more 2D x-ray images acquired from the single x-ray image view to the 3D image data, the computer processor drives a display to display a cross-section derived from the 3D image data at a the first location of a tip of the tool, as identified from the one or more 2D x-ray images, and optionally to show a vertical line on the cross-sectional image indicating a line within the cross-sectional image somewhere along which the first location of the tip of the tool is disposed. Typically, when the tip of the tool is disposed at an additional location with respect to the vertebra, further 2D x-ray images of the tool at the additional location are acquired from the same single x-ray image view, or a different single x-ray image view, and the above-described steps are repeated. Typically, for each location of the tip of the tool to which the above-described technique is applied, 2D x-ray images need only be acquired from a single x-ray image view, which may stay the same for the respective locations of the tip of the tool, or may differ for respective locations of the tip of the tool” at page 44, line 12), and
(F) based upon the identified location of the portion of the tool within the first and second tool 2D x-ray images, and the registration of the first and second tool 2D x-ray images to the 3D image data, determining the location of the portion of the tool with respect to the 3D image data (“In step 90, the computer processor overlays an image of the tool, a representation thereof, and/or a representation of the tool path upon the 3D image data (e.g., a 3D image, a 2D cross-section derived from 3D image data, and/or a 2D projection image derived from 3D image data), the location of the tool or tool path having been derived from current 2D images” at page 44, line 31).
Tolkowsky et al. does not explicitly disclose acquiring with a 2D x-ray imaging device only two skeletal-portion 2D x-ray images of at least the skeletal portion and receiving the 3D image data from the machine learning engine.
Oved teaches an apparatus in the same field of endeavor of 3D x-ray image generation, comprising:
a machine learning engine configured to generate from only two skeletal-portion 2D x-ray images (“Reference is now made to FIG. 6, which is a diagram depicting an exemplary architecture of the neural network for mapping two 2D anatomical images into a 3D point cloud, in accordance with some embodiments of the present invention. The two 2D images may be x-ray images. The two 2D images may be of different views of the target anatomical image, for example, an anterior-posterior (AP) and lateral views” at col. 18, line 44; “For example, the 2D anatomical image(s) are x-ray image(s) outputted by an x-ray machine, for example, AP and/or lateral views (e.g., of the chest, femur)” at col. 19, line 12) 3D image data of at least the skeletal portion (“At 306, the 2D anatomical image(s) are fed into the neural network” at col. 19, line 20; “At 308, a reconstructed 3D point cloud model of the target anatomical structure is outputted by the trained neural network, as described herein” at col. 19, line 22); and
(A) receive the 3D image data from the machine learning engine (implied that the reconstructed 3D data is sent for further processing or visualization).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to utilize a machine learning algorithm as taught by Oved to construct the pre-operative 3D data of Tolkowsky et al. as the 2D images have “relatively lower radiation, lower cost, faster acquisition time, and provides high quality treatment planning for patients where 3D imaging devices are unavailable” (Oved at col. 6, line 48).
Regarding claim 143, Tolkowsky et al. discloses a non-transitory computer software product for facilitating performing a procedure with respect to a skeletal portion within a body of a subject, the non-transitory computer software product for use with (i) a 2D x-ray imaging device configured to acquire 2D x-ray images of at least the skeletal portion (“3D x-ray data are acquired” at page 32, line 25) and (ii) an engine configured to generate from two skeletal-portion 2D x-ray images (“For some applications, the 3D imaging that is used is CT imaging, and the following explanation of the registration of the 3D image data to the 2D images will focus on CT images. However, the scope of the present invention includes applying the techniques describe herein to other 3D imaging modalities, such as MRI and 3D x-ray” at page 55, line 26; 3D x-ray implies at least two 2D x-ray acquisition) 3D image data of at least the skeletal portion (the 3D x-ray is generated using the at least two x-ray images), the non-transitory computer software product comprising a non-transitory computer-readable medium in which program instructions are stored, which instructions, when read by a computer cause the computer (“Applications of the invention described herein can take the form of a computer program product accessible from a computer-usable or computer-readable medium (e.g., a non-transitory computer-readable medium) providing program code for use by or in connection with a computer or any instruction execution system, such as computer processor 22” at page 88, line 4) to perform the steps of:
A) receiving the 3D image data from the engine (“In a first step 70, targeted vertebra(e) are marked by an operator with respect to 3D image data (e.g., a 3D image, a 2D cross-section derived from 3D image data, and/or a 2D projection image derived from 3D image data) of the subject's spine” at page 43, line 13),
B) receiving from the 2D x-ray imaging device, while the 2D x-ray imaging device is at a first pose with respect to the subject's body, a first tool 2D x-ray image, from a first view of the skeletal portion and of at least a portion of a tool disposed at a location with respect to the skeletal portion, the first view being similar to the view of one of the skeletal-portion 2D x-ray images used for generating the 3D image data (“In a fifth step 78, the first tool in the sequence of tools (which is typically a needle, e.g., a Jamshidi™ needle) is typically inserted into the subject (e.g., in the subject's back), and is slightly fixated in the vertebra. In a sixth step 80, two or more 2D radiographic images are acquired from respective views that typically differ by at least 10 degrees (and further typically by 30 degrees or more), and one of which is typically from the direction of insertion of the tool. Typically, generally-AP and generally-lateral images are acquired. Alternatively or additionally, images from different views are acquired” at page 43, line 23),
(C) receiving from the 2D x-ray imaging device, while the 2D x-ray imaging device is at a second pose with respect to the subject's body, a second tool 2D x-ray image, from a second view, of the skeletal portion and of at least the portion of the tool disposed at the location with respect to the skeletal portion, the second view being similar to the view of the other one of the first and second skeletal-portion 2D x-ray images used for generating the 3D image data (“It is noted, that, as described in further detail hereinbelow, for some applications, in order to perform step 84, the computer processor need acquire one or more 2D x-ray images of a tool at a first location inside the vertebra from only a single x-ray image view, and the one or more 2D x-ray images are registered to the 3D image data by generating a plurality of 2D projections from the 3D image data, and identifying a 2D projection that matches the 2D x-ray images of the vertebra” at page 44, line 7; as described above, the device is moved to differ the degree of the view between images),
wherein during the acquisition of the skeletal-portion 2D x-ray images used for generating the 3D image data the portion of the tool was either (i) not visible in the skeletal-portion 2D x-ray images (“Applying pre-operative 3D visibility (e.g., from CT and/or MRI) during the intervention. It is noted that 3D visibility provides desired cross-sectional images (as described in further detail hereinbelow), and is typically more informative and/or of better quality than that provided by intraoperative 2D images” at page 42, line 27; as the 3D data is acquired pre-operatively, it is understood that this prior to any tools being introduced to the patient), or (ii) disposed at a different location with respect to the skeletal portion than the location in which the tool is disposed during the acquisition of the first and second tool 2D x-ray images;
(D) registering the first and second tool 2D x-ray images to the 3D image data (“In a seventh step 82, computer processor 22 of system 20 typically registers the 3D image data to the 2D images” at page 43, line 30),
(E) identifying a location of the portion of the tool with respect to the skeletal portion, within the first and second tool 2D x-ray images (“In response to registering the one or more 2D x-ray images acquired from the single x-ray image view to the 3D image data, the computer processor drives a display to display a cross-section derived from the 3D image data at a the first location of a tip of the tool, as identified from the one or more 2D x-ray images, and optionally to show a vertical line on the cross-sectional image indicating a line within the cross-sectional image somewhere along which the first location of the tip of the tool is disposed. Typically, when the tip of the tool is disposed at an additional location with respect to the vertebra, further 2D x-ray images of the tool at the additional location are acquired from the same single x-ray image view, or a different single x-ray image view, and the above-described steps are repeated. Typically, for each location of the tip of the tool to which the above-described technique is applied, 2D x-ray images need only be acquired from a single x-ray image view, which may stay the same for the respective locations of the tip of the tool, or may differ for respective locations of the tip of the tool” at page 44, line 12), and
(F) based upon the identified location of the portion of the tool within the first and second tool 2D x-ray images, and the registration of the first and second tool 2D x-ray images to the 3D image data, determining the location of the portion of the tool with respect to the 3D image data (“In step 90, the computer processor overlays an image of the tool, a representation thereof, and/or a representation of the tool path upon the 3D image data (e.g., a 3D image, a 2D cross-section derived from 3D image data, and/or a 2D projection image derived from 3D image data), the location of the tool or tool path having been derived from current 2D images” at page 44, line 31).
Tolkowsky et al. does not explicitly disclose acquiring with a 2D x-ray imaging device only two skeletal-portion 2D x-ray images of at least the skeletal portion and receiving the 3D image data from the machine learning engine.
Oved teaches a non-transitory computer software product in the same field of endeavor of 3D x-ray image generation, for facilitating performing a procedure with respect to a skeletal portion within a body of a subject, the non-transitory computer software product for use with (ii) a machine learning engine configured to generate from only two skeletal-portion 2D x-ray images (“Reference is now made to FIG. 6, which is a diagram depicting an exemplary architecture of the neural network for mapping two 2D anatomical images into a 3D point cloud, in accordance with some embodiments of the present invention. The two 2D images may be x-ray images. The two 2D images may be of different views of the target anatomical image, for example, an anterior-posterior (AP) and lateral views” at col. 18, line 44; “For example, the 2D anatomical image(s) are x-ray image(s) outputted by an x-ray machine, for example, AP and/or lateral views (e.g., of the chest, femur)” at col. 19, line 12) 3D image data of at least the skeletal portion (“At 306, the 2D anatomical image(s) are fed into the neural network” at col. 19, line 20; “At 308, a reconstructed 3D point cloud model of the target anatomical structure is outputted by the trained neural network, as described herein” at col. 19, line 22) comprising a non-transitory computer-readable medium in which program instructions are stored, which instructions, when read by a computer cause the computer (“An aspect of some embodiments of the present invention relates to systems, methods, an apparatus, and/or code instructions (i.e., stored in a memory and executable by hardware processor(s)) for surgical treatment of a target anatomical structure of a patient based on a 3D reconstruction of a target anatomical structure depicted in a 2D anatomical image(s), for example, an x-ray” at col. 5, line 13) to perform the steps of:
(A) receiving the 3D image data from the machine learning engine (implied that the reconstructed 3D data is sent for further processing or visualization).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to utilize a machine learning algorithm as taught by Oved to construct the pre-operative 3D data of Tolkowsky et al. as the 2D images have “relatively lower radiation, lower cost, faster acquisition time, and provides high quality treatment planning for patients where 3D imaging devices are unavailable” (Oved at col. 6, line 48).
Regarding claims 108 and 135, Tolkowsky et al. discloses a method and apparatus wherein the at least one computer processor is further configured to display the location of the portion of the tool with respect to the 3D image data (“In step 90, the computer processor overlays an image of the tool, a representation thereof, and/or a representation of the tool path upon the 3D image data (e.g., a 3D image, a 2D cross-section derived from 3D image data, and/or a 2D projection image derived from 3D image data), the location of the tool or tool path having been derived from current 2D images” at page 44, line 31).
Regarding claims 109 and 136, Tolkowsky et al. discloses a method and apparatus wherein the at least one computer processor is configured to identify by means of image processing the location of the portion of the tool with respect to the skeletal portion, within the first and second tool 2D x-ray images (“In response to registering the one or more 2D x-ray images acquired from the single x-ray image view to the 3D image data, the computer processor drives a display to display a cross-section derived from the 3D image data at a the first location of a tip of the tool, as identified from the one or more 2D x-ray images, and optionally to show a vertical line on the cross-sectional image indicating a line within the cross-sectional image somewhere along which the first location of the tip of the tool is disposed. Typically, when the tip of the tool is disposed at an additional location with respect to the vertebra, further 2D x-ray images of the tool at the additional location are acquired from the same single x-ray image view, or a different single x-ray image view, and the above-described steps are repeated. Typically, for each location of the tip of the tool to which the above-described technique is applied, 2D x-ray images need only be acquired from a single x-ray image view, which may stay the same for the respective locations of the tip of the tool, or may differ for respective locations of the tip of the tool” at page 44, line 12).
Regarding claims 111 and 138, Tolkowsky et al. discloses a method and apparatus wherein the at least one computer processor is further configured to, subsequently to step (B):
designate at least one locational element selected from the group consisting of (a) a longitudinal insertion path with respect to the skeletal portion, (b) a skin-level incision point corresponding to the skeletal portion (“In a fourth step 76, an incision site (in the case of minimally-invasive surgery) or a tool entry point (in the case of open surgery) is determined” at page 43, line 22), (c) a skeletal-portion-level entry point within the body of the subject, (d) an intermediate point along an insertion path between a skin-level incision point and a target point, and (e) a target point within the skeletal portion (“In a first step 70, targeted vertebra(e) are marked by an operator with respect to 3D image data (e.g., a 3D image, a 2D cross-section derived from 3D image data, and/or a 2D projection image derived from 3D image data) of the subject's spine” at page 43, line 13), and
associate the at least one designated locational element with the 3D image data for the skeletal portion (the designation of these markers and points is done with respect to the 3D pre-operative data).
Regarding claims 112 and 139, Tolkowsky et al. discloses a method and apparatus wherein the at least one computer processor is further configured to determine the location of the portion of the tool with respect to the at least one designated locational element based on (a) the determined location of the portion of the tool with respect to the 3D image data, and (b) the association of the at least one designated locational element with the 3D image data. (“For some applications, a surgeon places a radiopaque knife 116 (or another radiopaque tool or object) at a prospective incision site (and/or places a tool at a prospective tool insertion location) and verifies the location of the incision site (and/or tool insertion location) by observing the location of the tip of the knife (or portion of another tool) with respect to the x-ray (e.g., via cursor 117), by means of the bi-directional mapping between the optical image and the x-ray image” at page 53, line 20; as previously mentioned, the marker and target designation is done with respect to the pre-operative 3D data).
Regarding claims 113 and 140, Tolkowsky et al. discloses a method and apparatus wherein the at least one computer processor is further configured to display the location of the portion of the tool with respect to the at least one designated locational element, with respect to the 3D image data (“Reference is now made to FIG. 15B, which is a schematic illustration of the location of the tool tip 168 denoted by cross-hairs upon cross-sections 160 and 162 of the vertebra corresponding, respectively, to first and second locations of a tip 164 of a tool that is advanced into the vertebra along a longitudinal insertion path (as shown in FIG. 15A)” at page 66, line 6).
Regarding claims 114 and 141, Tolkowsky et al. discloses a method and apparatus wherein the at least one computer processor is configured to receive at least one 2D x-ray image selected from the group consisting of: (a) the first tool 2D x-ray image, from the first view, and (b) the second tool 2D x-ray image, from the second view, the at least one 2D x-ray image having been acquired using one or more acquisition parameters having values that are similar to the respective values of a corresponding one or more acquisition parameters from the acquisition of the corresponding one of the skeletal-portion 2D x-ray images that were used for generating the 3D image data (“Alternatively or additionally, during the deep-learning phase, a large database of 2D x-ray images and (at least some of) their known parameters relative to vertebra are inputted to a deep-learning engine. Such parameters typically include viewing angle, viewing distance, and optionally additional camera parameters. For some applications, the aforementioned parameters are exact. Alternatively, the parameters are approximate parameters. The parameters may be recorded originally when generating the images, or annotated by a radiologist. Thus, the engine learns, given a certain 2D projection image, to suggest simulated camera viewing distances and angles that correspond to that projection image. Subsequently, the deep-learning data is fed as an input to computer processor 22 of system 20. Intraprocedurally, in order to register the 2D x-ray images to the 3D image data, computer processor uses the deep-learning data in order to limit the search space in which DRRs of the 3D image data that match the x-ray images should be searched for. Computer processor 22 then searches for the matching DRRs only within the search space that was prescribed by the deep-learning data” at page 61, line 8).
Claim(s) 110 and 137 are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Tolkowsky et al. and Oved as applied to claims 107 and 118 above, and further in view of Boddington et al. (US 2021/0015560).
The Tolkowsky et al. and Oved combination discloses a method and apparatus as described in claims 107 and 118 above.
The Tolkowsky et al. and Oved combination does not explicitly disclose the at least one computer processor is further configured to train a machine learning engine to detect the presence of a tool within a 2D x-ray image, and the at least one computer processor is configured to identify the location of the portion of the tool with respect to the skeletal portion within the first and second 2D tool x-ray images using the machine learning engine.
Boddington et al. teaches a method and apparatus in the same field of endeavor of intraoperative guidance, wherein the at least one computer processor is further configured to train a machine learning engine to detect the presence of a tool within a 2D x-ray image (“When an image is acquired, the artificial intelligence intra-operative surgical guidance system 1 uses the hip application segmentation machine learning module to identify the relevant anatomical, instrument, and implant features such as the nail alignment jig, and nail and lag-screw” at paragraph 0155, last sentence), and the at least one computer processor is configured to identify the location of the portion of the tool with respect to the skeletal portion within the first and second tool 2D x-ray images using the machine learning engine (“In addition, the artificial intelligence intra-operative surgical guidance system 1 provides an automatic determination of screw trajectories and more generally in situations of instrumentation trajectories. For example, to determine if the instrumentation is within the right plane while simultaneously tracking anatomical, implant and instrument considerations in different views. This is achieved using deep learning techniques, more specifically, a Reinforcement Learning (RL) technique. RL strategies are used to train an artificial RL agent to precisely and robustly identify/localize the optimal trajectory/path by navigating in an environment, in our case the acquired fluoroscopic images” at paragraph 0156, line 1).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to utilize the machine learning asset of Boddington et al. to localize the tool location of the Tolkowsky et al. and Oved combination “to learn an optimal policy that maximizes the intermediate rewards but also to subsequent future rewards” (Boddington et al. at paragraph 0156, last sentence) for optimal navigation during the procedure.
Allowable Subject Matter
Claims 116, 117 and 142 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter: the prior art does not teach or disclose performing iterative image comparisons of (a) the initial 2D x-ray image to (b) any of the multiple 2D x-ray images of at least the skeletal portion, and wherein, acquiring the first 2D x-ray image of at least the portion of the tool and the skeletal portion from the first view, the first view being similar to the view of the one of the multiple 2D images used for generating the 3D image data, comprises: in response to an outcome of the image comparisons, moving the 2D imaging device to an imaging view that is similar to the view of one of the multiple 2D images used for generating the 3D image data, such that in response thereto the 2D imaging device is disposed at the first pose with respect to the subject's body and acquiring the first 2D x-ray image of at least the portion of the tool and the skeletal portion as required by claim 116; performing iterative image comparisons of (a) the subsequent 2D x-ray image to (b) any of the multiple 2D x-ray images of at least the skeletal portion, and wherein moving the 2D imaging device to the second pose with respect to the subject's body comprises, in response to the outcome of the image comparisons, moving the 2D imaging device to an imaging view that is similar to the view of another of the multiple 2D x-ray images used for generating the 3D image data, as required by claim 117; perform iterative image comparisons of (a) the subsequent 2D x-ray image to (b) any of the multiple 2D x-ray images of at least the skeletal portion, and the at least one computer processor is configured to receive from the 2D imaging device the second 2D x-ray image while the 2D imaging device is disposed at the second pose, wherein, based on the outcome of the image comparisons, the second pose is an imaging view that is similar to the view of another one of the multiple 2D x-ray images used for generating the 3D image data as required by claim 142.
Response to Arguments
Summary of Remarks (@ response pages labeled 12-13): Tolkowsky et al. does
not teach or disclose “only two skeletal-portion 2D x-ray images”.
Examiner’s Response: This argument is moot in view of the newly utilized Oved reference.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KATRINA R FUJITA whose telephone number is (571)270-1574. The examiner can normally be reached Monday - Friday 9:30-5:30 pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Sumati Lefkowitz can be reached at 5712723638. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KATRINA R FUJITA/ Primary Examiner, Art Unit 2672
Prosecution Insights