DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Acknowledgement is made to Applicant’s claim to priority to U.S. Provisional App. No. 62/965,628 filed January 24, 2020.
Status of Claims
This Office Action is responsive to the claims filed on 05/16/2025. Claims 8 and 15 have been amended. Claim 13 has been cancelled. Claims 20 and 21 are newly presented (claims 20 and 21 are inadvertently marked as previously presented in the set of claims). Claims 1-12 and 14-21 are presently pending in this application. Claims 1-7 are presently withdrawn from consideration following the response to the requirement for restriction/election.
Claim Objections
Claim 15 objected to because of the following informalities:
claim 15, line 1: “calculating a pose of each of the subset” should be amended to read “calculating the pose of each medical image of the subset of the medical images”.
Appropriate correction is required.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim 8, 9, 14, 15, and 17-21 are rejected under 35 U.S.C. 103 as being unpatentable over Ali (US 20110176723) in view of Vetterli (US 20170091945).
Regarding claim 8, Ali teaches a method (Paragraph [0008]; a method of reconstructing a CBCT image using the corrected CBCT projections), comprising:
receiving a sequence of medical images captured by a medical imaging device while the medical imaging device is rotated through a rotation (Paragraph [0029]; on-board-imager OBI may be configured to generate a series of 2D radiographic projections (CBCT projections) as its conical beam is rotated around the object along a circular and/or helical trajectory), wherein the sequence of medical images show an area of interest (Paragraph [0029]; Each projection may comprise a snapshot of the X-ray beam's attenuation as it passes through the object at a unique view angle) including a landmark having a 3D shape (Paragraphs [0030]-[0031]; 3D markers; Paragraphs [0056]-[0057]);
calculating a pose of each medical image of a subset of the medical images (Paragraphs [0030]-[0035]; at a unique view angle (A), thereby projecting each patient voxel, e.g. (x, y, z), onto the flat-panel imager at a corresponding pixel location, e.g. (j, k); Paragraph [0036]; processing each CBCT projection may comprise transforming each CBCT projection based on the transformation vector and view angle, e.g. at the corresponding time-tagged angular view; Paragraph [0047]; compute a transformation vector (u, v) that may be used to map the position of an external marker at the various angular views for some or all of the CBCT projections; The transformation of the imager projection to a patient coordinate system is understood to read on the claimed limitation of calculating a pose as understood in its broadest reasonable interpretation) based on at least 3D-2D correspondence of a 2D projection of the landmark in each medical image of the subset (Paragraphs [0030]-[0035]; diagram of geometric relationships occurring between a markers' 3D position, e.g. at voxel (x, y, z), and the corresponding projection's 2D position, e.g. at pixel (j, k)); and
calculating a volumetric reconstruction of the area of interest (Paragraph [0005]; computer processing to generate a three dimensional (3D) representation (volumetric or otherwise) of the patient's internal structure from a series of two dimensional (2D) X-ray images. Hence, a CT scan may generate a 3D image of a patient's internal structure; Paragraph [0036]; At step 216, the OBI may perform CBCT reconstruction using the motion-corrected CBCT projections to generate a CBCT image) based on at least the subset of the medical images and the calculated poses of the subset of the medical images (Paragraphs [0047]-[0048]; The adjusted 2D map I’ is based on the images I. The OBI may use the transformed radiographic projections (I') as input parameters during CBCT reconstruction, e.g. based on a Feldkamp back-projection algorithm as provided by the OBI vendor; to extract a 3D motion trajectory; The process of forming the CBCT reconstruction using the adjusted I’ is considered to read on the claimed limitation of calculating a volumetric reconstruction of the area of interest based on at least the subset of the medical images and the calculated poses of the subset of the medical images as understood in its broadest reasonable interpretation); and
the volumetric reconstruction includes the landmark (Paragraph [0030]; A first 3D marker, e.g. positioned at voxel (0, 0, z), and a second 3D marker, e.g. positioned at voxel (x, y, 0), may be projected onto the flat-panel imager as marker B, e.g. located at pixel (0, k), and marker C, e.g. located at pixel (j, k), (respectively)); and wherein the at least one further one of the medical images is not included in the subset of the medical images (Paragraph [0036]; wherein the at least one further one of the medical images is not included in the subset of the medical images).
Ali does not explicitly teach calculating a pose for at least one further on of the medical images based at least on a position of the landmark in the volumetric reconstruction,
wherein the at least one further one of the medical images is not included in the subset of the medical images; and
calculating a further volumetric reconstruction of the area of interest based on:
the subset of the medical images,
the calculated poses of the subset of the medical images,
the at least one further one of the medical images, and
the calculated pose of the at least one further one of the medical images.
Vetterli, however, teaches a method (Paragraph [0009]; approach to calculate the sub-spaces of potential locations of point(s) and/or of potential sensor pose(s)) comprising calculating a pose of each image (Paragraph [0043]; locations of a plurality of point sources and/or the poses of different images of a set of images shall be determined on the basis of the set of images taken from different viewpoints or sensor poses) of a subset of images (Paragraph [0030]; a source point s shall be identified localised in a set of images taken from different viewpoints… The sensors m=1, 3 and 4 captured the point s in the pixel positions) based on at least 3D-2D correspondence of a 2D projection of the landmark in each image of the subset (Paragraph [0031]; scene 1 might show the identified point source s so that the images capturing the point source s belong to the set of images of interest; Paragraph [0040]; the point source location (six, siz), the camera parameter like f and the size of the subregion of the sensor w are known, the subspace of potential sensor poses can be computed in the solution space… A similar equation system could be computed for a three dimensional scene with the solution space (tx, ty, tz, θ1, θ2));
calculating a volumetric reconstruction of the area of interest based on at least the subset of the images (Paragraph [0045]; for each point source of a first image of the set of images the sub-space of potential locations of the point source in the scene is computed on the basis of the sub-region of the image representing the point source);
wherein the volumetric reconstruction includes the landmark (Paragraph [0045]; the subspace 10.1 of potential locations of the point source 9.1 is computed as described in the first embodiment);
calculating a pose for at least one further one of the images based at least on a position of the landmark in the volumetric reconstruction (Paragraph [0047]; the subspace of potential poses of the sensor of the other image can be calculated from this subspace of potential locations of the point source and its subregion on the other image or its sensor),
wherein the at least one further one of the images is not included in the subset of the images (Paragraph [0031]; not all images taken from the scene 1 might show the identified point source s so that the images capturing the point source s belong to the set of images of interest. Instead of identifying a point or several points in a set of images, the method could simply receive over an interface the point(s) and their pixel position in the respective images instead of actively identifying the point(s); Fig. 3 shows the imager m = 2 does not capture the point s); and
calculating a further volumetric reconstruction of the area of interest (Paragraph [0049]; a fifth step for each point source of the other image the sub-space of potential locations of the point source in the scene are calculated on the basis of the sub-region of the other image representing the point source and on the basis of the sensor intersection region of the sub-spaces of potential poses of the sensor of the other image) based on:
the subset of the images (Paragraph [0044]-[0045]; In the first image two (any other number possible) point sources 9.1, 9.2 are identified; for each point source of a first image of the set of images the sub-space of potential locations of the point source in the scene is computed on the basis of the sub-region of the image representing the point source),
the calculated poses of the subset of the images (Paragraph [0041]; the pose intersection region of the sub-spaces of potential poses corresponding to the point sources is calculated),
the at least one further one of the images (Paragraph [0044]; with different viewpoints new point sources are added to the images and some point sources are lost; Paragraph [0048]-[0049]; subspace of potential locations of the point source 9.1 from a second image (here the other image) taken from another viewpoint), and
the calculated pose of the at least one further one of the images (Paragraph [0049]-[0052]; f a point source 9.1 could be improved by intersecting it with a subspace of potential locations of the point source 9.1 from a second image (here the other image) taken from another viewpoint. Contrary to the first embodiment, the exact viewpoint or sensor pose of the other image is not known. But the subspace of potential poses of the other image is known from the sensor intersection region 13 previously calculated; if the poses of the sensors shall be estimated, a pose of each sensor intersection region 13 is selected to estimate the pose of each sensor).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have modified the method of Ali to have included calculating a pose for at least one further on of the medical images based at least on a position of the landmark in the volumetric reconstruction, wherein the at least one further one of the medical images is not included in the subset of the medical images; and calculating a further volumetric reconstruction of the area of interest based on: the subset of the medical images, the calculated poses of the subset of the medical images, the at least one further one of the medical images, and the calculated pose of the at least one further one of the medical images as taught by Vetterli because it would have been a well known and understood method of determining camera pose estimation from a set of images and further allowed estimation of points within the collection of images that would have been doable in real time and with high quality (Paragraph [0046]). Furthermore it would improve sensing and navigation capabilities of robotic devices and base control around the movement of the identified points within the scene (Paragraph [0053]).
Regarding claim 9, together Ali and Vetterli teach all of the limitations of claim 8 as noted above.
Ali further teaches the landmark is an anatomical landmark (Paragraphs [0056] and [0057]; Hence, anatomical mapping based on marker motion; by tracking various marker surrogates, e.g. anatomical surrogates, surface features, air flow of the patient, etc.; external skin surface imaging, respiratory sensor monitoring, etc.).
Regarding claim 14, together Ali and Vetterli teach all of the limitations of claim 8 as noted above.
Ali further teaches the sequence of images does not show a plurality of radiopaque markers (Paragraph [0057]; disclosed techniques may also be performed using various marker-less tracking methods e.g. external skin surface imaging, respiratory sensor monitoring, etc.; by tracking various marker surrogates, e.g. anatomical surrogates, surface features, air flow of the patient, etc.).
Regarding claim 15, together Ali and Vetterli teach all of the limitations of claim 8 as noted above.
Ali further teaches the calculating a pose of each of the at least some of the medical images is further based on a known trajectory of the rotation (Paragraph [0029]; Each projection may comprise a snapshot of the X-ray beam's attenuation as it passes through the object at a unique view angle Paragraph [0036]; The 2D mobile track may be a function of view angle (e.g. according to time-tagged angular views corresponding with the various CBCT projections); OBI may compute a plurality of 2D position shifts by subtracting the 2D stationary track from the 2D mobile track, e.g. at each corresponding view angle.).
Regarding claim 17, together Ali and Vetterli teach all of the limitations of claim 8 as noted above.
Ali further teaches the landmark is an instrument positioned within a body of a patient at the area of interest (Paragraph [0056]; In some embodiments, the motion of an internal marker implanted into the ROI may be used).
Regarding claim 18, together Ali and Vetterli teach all of the limitations of claim 8 as noted above.
Ali further teaches the landmark is an object positioned proximate to a body of a patient and outside the body of the patient (Paragraph [0025]; using an external marker attached to the patient's skin).
Regarding claim 19, together Ali and Vetterli teach teaches all of the limitations of claim 18 as noted above.
Ali further teaches the object is fixed to the body of the patient (Paragraph [0025]; using an external marker attached to the patient's skin).
Regarding claim 20, together Ali and Vetterli teach all of the limitations of claim 8 as noted above.
Ali further teaches the landmark is an anatomical landmark (Paragraphs [0056] and [0057]; Hence, anatomical mapping based on marker motion; by tracking various marker surrogates, e.g. anatomical surrogates, surface features, air flow of the patient, etc.; external skin surface imaging, respiratory sensor monitoring, etc.).
Regarding claim 21, together Ali and Vetterli teach all of the limitations of claim 8 as noted above.
Ali further teaches the subset of the medical images and the at least one further one of the medical images together constitute all of the medical images (Paragraph [0036]; In some embodiments, the internal marker may not be captured in one or more, e.g. about half, of the CBCT projections, and hence some unknown data points in the 2D mobile track may be interpolated, e.g. via polynomial interpolation, from other known data points; Paragraph [0050]; During the scans, internal marker #1 and internal marker #2 where positioned on the patient's right side, while the external marker was positioned on the patient's left side. Accordingly, each of the markers appeared in approximately half of the CBCT projections, and as a result their positions were interpolated for those CBCT projections in which they did not appear. When using FF scans (e.g. having diameters in excess of 25 cm), the entire ROI is captured at all times during the scan such that the markers show up in each projection; The scans with the marker and the scans without the marker makes up all the medical images taken, and is considered to read on the claimed limitation as understood in its broadest reasonable interpretation).
Claims 10-12 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Ali in view of Vetterli as applied to claims 9 and 8 above, respectively, and further in view of Weingarten (US 20170035379).
Regarding claim 10, together Ali and Vetterli teach all of the limitations of claim 9 as noted above.
Ali does not teach the 3D shape of the anatomical landmark is determined based at least on at least one preoperative image.
Weingarten, however, teaches a method (Paragraph [0023]; determine a pose of the fluoroscopic imaging device for each frame of the fluoroscopic video and to construct fluoroscopic-based three dimensional volumetric data of the target area in which soft tissue objects are visible) wherein the 3D shape of the anatomical landmark is determined based at least on at least one preoperative image (Paragraph [0047]; previously acquired CT image data for generating and viewing a three dimensional model of the patient's “P's” airways, enables the identification of a target on the three dimensional model; Paragraphs [0077]-[0078]; virtual fluoroscopic images are created from previously acquired CT data… fluoroscopic imaging device pose of each video frame of the captured fluoroscopic video is determined based on the registration of the fluoroscopic frame with the virtual fluoroscopic image).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have modified the method of Ali in view of Vetterli such that the 3D shape of the anatomical landmark is determined based at least on at least one preoperative image as taught by Weingarten because it would have helped ensure registration of the images with determined pathways through the patient and further facilitate identification of the target within the patient in interoperative images for navigation with the medical device (Weingarten, Paragraph [0047]).
Regarding claim 11, together Ali and Vetterli teach all of the limitations of claim 8 as noted above.
Ali does not teach the 3D shape of the landmark is determined based at least on applying a structure from motion technique to at least some of the sequence of medical images.
Weingarten, however, teaches a method (Paragraph [0023]; determine a pose of the fluoroscopic imaging device for each frame of the fluoroscopic video and to construct fluoroscopic-based three dimensional volumetric data of the target area in which soft tissue objects are visible) wherein the 3D shape of the landmark is determined based at least on applying a structure from motion technique to at least some of the sequence of medical images (Paragraphs [0012] and [0027]; determine three dimensional positions of features in the fluoroscopic video… by tracking a few visible markers (two dimensional visible features) in the fluoroscopic video, the pose and three dimensional positions may be solved together by using some structure from motion technique).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have modified the method of Ali in view of Vetterli to have had the 3D shape of the landmark determined based at least on applying a structure from motion technique to at least some of the sequence of medical images as taught by Weingarten because it would have improved the detection of the positions and orientations of those markers can be tracked along a continuous fluoroscope rotation video, and further improving the reconstruction of the markers in three dimensional position and the corresponding fluoroscopic imaging device locations can be determined (Weingarten, Paragraph [0070]).
Regarding claim 12, together Ali, Vetterli, and Weingarten teach all of the limitations of claim 11 as noted above.
Weingarten further teaches the structure from motion technique is applied to all of the sequence of medical images (Paragraph [0070]; the marker positions are constructed in three dimensional using structure-from-motion techniques and the pose of the fluoroscopic imaging device is obtained for each video frame).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have further modified the method of Ali in view of Vetterli and Weingarten such that the structure from motion technique is applied to all of the sequence of medical images because it would have improved the detection of the positions and orientations of those markers can be tracked along a continuous fluoroscope rotation video, and further improving the reconstruction of the markers in three dimensional position and the corresponding fluoroscopic imaging device locations can be determined (Weingarten, Paragraph [0070]).
Regarding claim 16, together Ali and Vetterli teach all of the limitations of claim 8 as noted above.
Ali does not teach the 3D shape of the landmark is determined based on at least one preoperative image and further based on applying a structure from motion technique to at least some of the sequence of medical images.
Weingarten, however, teaches a method (Paragraph [0012]; determine three dimensional positions of features in the fluoroscopic video) wherein the 3D shape of the landmark is determined based on at least one preoperative image (Paragraph [0047]; respect to the planning phase, computing device 125 utilizes previously acquired CT image data for generating and viewing a three dimensional model of the patient's “P's” airways, enables the identification of a target on the three dimensional model (automatically, semi-automatically, or manually); Paragraph [0050]; using an interior geometry of passages of the three dimensional model generated in the planning phase… The software aligns, or registers, an image representing a location of sensor 44 with a the three dimensional model and two dimensional images generated from the three dimension model, which are based on the recorded location data and an assumption that locatable guide 32 remains located in non-tissue space in the patient's “P's” airways) and applying a structure from motion technique to at least some of the sequence of medical images (Paragraphs [0012] and [0027]; determine three dimensional positions of features in the fluoroscopic video… by tracking a few visible markers (two dimensional visible features) in the fluoroscopic video, the pose and three dimensional positions may be solved together by using some structure from motion technique).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have modified the method of Ali in view of Vetterli such that the 3D shape of the landmark is determined based on at least one preoperative image and further based on applying a structure from motion technique to at least some of the sequence of medical images as taught by Weingarten because it would have improved the navigation of the device inside the body (Weingarten, Paragraph [0068]) and it would have improved the detection of the positions and orientations of those markers can be tracked along a continuous fluoroscope rotation video, and further improving the reconstruction of the markers in three dimensional position and the corresponding fluoroscopic imaging device locations can be determined (Weingarten, Paragraph [0070]).
Response to Arguments
Claim Objections
Applicant failed to provide amendments or arguments regarding the objection to claim 15. Claim 15 remains objected to for reciting “a pose of each of the subset” which should be amended to “the pose of each medical image of the subset of the medical images” to be consistent with the already recited “a pose of each medical image of a subset of the medical images” recited in claim 8.
Claim Rejections under – 35 U.S.C. § 112
Applicant’s arguments, see Remarks, filed 05/16/2025, with respect to “applying a structure from motion technique” have been fully considered and are persuasive. The steps of applying a structure from motion technique appears to be a general method known within the art as described in references pointed to by Applicant. For these reasons, the rejections of claims 11, 12, and 16 under 35 USC 112(a) and 112(b) have been withdrawn.
Claim Rejections under – 35 U.S.C. § 102 and 103
Applicant’s arguments with respect to the previous 35 U.S.C. § 102 and 103 rejections have been considered but are moot in view of the updated grounds of rejection necessitated by amendments.
Examiner would like to point out the new reference of Vetterli teaches imaging and camera pose estimation techniques which take into account images where the landmark is not included in the subset of images and further calculating the object points in the volume based on the subset of images, the calculated pose of the images, the further image, and the calculated pose of the further image.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Dean N Edun whose telephone number is (571)270-3745. The examiner can normally be reached M-F 8am-5:30pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Anh Tuan Nguyen can be reached at (571)272-4963. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DEAN N EDUN/Examiner, Art Unit 3797
/ANH TUAN T NGUYEN/Supervisory Patent Examiner, Art Unit 3795
08/17/2025
Prosecution Insights