Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1 and 4 are rejected under 35 U.S.C. 103 as being unpatentable over Liu (US 20180270474 A1; hereinafter Liu) in view of Ross (US 20230068745 A1; hereinafter Ross).
Regarding claim 1, Liu teaches a method of using a fiducial marker in a surgical navigation system (“It is another aspect of the optical imaging system of the present invention to be configured, whereby fluorescence imaging (such as 5-Aminolevulinic acid or Indocyanine green) is used in conjunction with preoperative surgical navigation (e.g. MRI, CT, PET, SPECT . . . ) for guiding interventions or surgeries,” (page 2, para [0017]).
“Next, a 3D model is created based on the pre-operative imaging data. In the next step, the position of the patient is tracked intra-operatively using any suitable technique, such as fiducial markers for example. Next, a transformation matrix is calculated between the pre-operative image space and the intra-operative object space (i.e. patient space),” (para [0215]).), the method comprising:
a computed tomography (CT) process for obtaining a CT image via a CT scan of a patient (“Initially, at step 1610, prior to surgery, the surgical target object 324 will undergo a PET/CT scan to acquire preoperative image data,” (page 22, para [0195]). A patient includes the surgical target object.);
CT image-based 3D reconstruction for generating, based on the CT image, a CT 3D model by using 3D reconstruction technology (“In one embodiment of the present invention, data from a preoperative PET/CT scan may be acquired prior to performing intraoperative imaging using the system 100 of the present invention. Surface-based registration will be carried out using an iterative closest point algorithm, with the use of k-d dimensional trees to minimize processing time, in accordance with the process discussed in detail below. Initially, at step 1010, the controller 130 acquires preoperative imaging data, such as PET/CT image data. Next, at step 1020, the process creates a point cloud or 3D mesh model, which is based on the preoperative imaging data…” (page 16, para [0162]).
“Briefly, a surface mesh or volumetric mesh can be generated based on preoperative tomographic imaging data (e.g. MRI, CT, SPECT, PET, 3D ultrasound, OCT, etc.). Image segmentation can be performed to isolate the organ of interest. Features and landmarks can be extracted from both the preoperative image model (based on tomographic imaging data) and the intraoperative image model (3D topography captured by our invention),” (page 29, para [0241]).
3D reconstruction includes creating a point cloud or 3D mesh model. 3D reconstruction based on the CT image includes creating a point cloud or 3D mesh model, which is based on the preoperative imaging data. 3D reconstruction technology includes mesh generation.);
a fluorescence process for generating a fluorescent image by capturing, using multiple cameras, an image of the target object “In one aspect, contrast agents, such as indocyanine green (ICG) or 5-am inolevulinic acid (5-ALA) may be used to guide brain surgery. The preoperative MRI data can enable surgical navigation using tracking technologies, such as optical tracking or electromagnetic tracking, and intraoperative fluorescence imaging, color reflectance and topography data can provide accurate image guidance. The system 100 is also enabled to integrate surgical navigation and intraoperative imaging with visualization of co-registered data. In one aspect, 5-ALA fluorescence can be used to image the brain tumor in order to correlate with images that are detected by surgical navigation based on preoperative MRI data. In another aspect, ICG fluorescence can image vascularity of tissues to provide useful information in addition to the preoperative MRI,” (pages 12-13, para [0142]).
“the imaging configuration 700A includes the light emitter module 310, the image detection module 320, the fluorescence imaging module 510, the emission filter 340, as previously discussed with regard to FIG. 6A, with the addition of the beam splitter 610A, the image detection module 320A, the fluorescence imaging module 510A and the emission filter 340A. As such, the image detection module 320 and the fluorescence imaging module 510 are positioned at a substantially right angle to each other, while the image detection module 320A and the fluorescence imaging module 510A are also positioned at a substantially right angle to each other. In addition, the beam splitter 620 is positioned at an oblique angle, such as about 45 degrees, relative to the image detection module 320 and the fluorescence imaging module 510. Similarly, the beam splitter 620A is positioned at an oblique angle, such as about 45 degrees, relative to the image detection module 320A and the fluorescence imaging module 510A. As such, during operation of the detection configuration 700A, the light emitter module 310 emits light 322 to illuminate the target object 324, whereupon the light 326 reflected and emitted by the target object 324 in response to being illuminated is received by the beam splitters 610 and 610A, where a portion 620 of the light 326 is permitted to pass through the beam splitters 610 and 610A for receipt by the image detection modules 320 and 320A to capture topography data of the target object 324. In addition, another portion 622 of the light 326 received by the beam splitters 610 and 610A is reflected by the beam splitters 610 and 610A, whereupon it is directed to pass through the emission filters 340 and 340A for receipt by the fluorescence imaging modules 510 and 510A to capture fluorescence information of the target object 324,” (page 10, para [0129]; Fig 7A).
“the configuration 500A utilizes 2 cameras/sensors/detectors, the image detection module 320 and the fluorescence detection module 510,” (pages 7-8, para [0120]).
Generating a fluorescent image includes capturing fluorescence information. From at least Figure 7A it is clear that multiple cameras are used to generate a fluorescent image. Multiple cameras include fluorescence imaging modules 510 and 510A. A fluorescent contrast agent includes any one of indocyanine green (ICG) and / or 5-am inolevulinic acid (5-ALA).);
multi-camera-based 3D reconstruction for generating a fluorescent 3D model from the fluorescent image by using 3D reconstruction technology (“…intraoperative imaging data is acquired from the optical imaging topography scanning system 100 of the present invention, such as fluorescence imaging data, in order to generate a point cloud or 3D model, as indicated at step 1030,” (page 16, para [0162]; Fig 12).
“In the present invention, 3D (three-dimensional) surface topography information combined with fluorescence imaging information is able to identify a scalar value that is associated with each point on a surface being imaged of the target object 324. This association of values is referred to as “point clouds,” which are expressed as P.sub.i=x.sub.i, y.sub.i, z.sub.i, I.sub.i; where (i=1, 2, 3 . . . N), and I.sub.i represents the fluorescence imaging value at the ith surface point in the data set,” (page 12, para [0135]-[0136]).
“The 3D scanning module 110 includes one more technologies, including but not limited to laser scanning triangulation, structured light, time-of-flight, conoscopic holography, modulated light, stereo-camera, Fourier 3D scanning, low coherence interferometry, common-path interference 3D scanning, and contact profilometers,” (page 4, para [0091]).
“…the fluorescence imaging modules 510 and 510A to capture fluorescence information of the target object 324,” (page 10, para [0129]; Fig 7A).
3D reconstruction includes generating a point cloud or 3D mesh model. 3D reconstruction from the fluorescent image includes acquiring fluorescence imaging data in order to generate a point cloud or 3D model. The 3D reconstruction is multi-camera-based because from at least Figure 7A and (page 10, para [0129]) it is clear that multiple cameras are used to generate a fluorescent image. 3D reconstruction technology includes any one or more of laser scanning triangulation, structured light, time-of-flight, stereo-camera, etc.); and registration for registering the CT 3D model and the fluorescent 3D model into one model (“Next, at step 1620, a point cloud {Pi=(xi, yi, zi, fi, ri, gi, bi), i=1, 2, . . . , N} is then obtained, where the vector (fi, ri, gi, bi) represents the fluorescence, red, green and blue data associated with individual surface points… the surface-based registration is carried out using the iterative closest point (ICP) algorithm with k-d dimensional trees for minimizing processing time as previously described. It should be appreciated that in some embodiments, the co-registration algorithm may be implemented using any suitable programming language, such as MATLAB or Python. After co-registration, the preoperative PET/CT volumetric data will be incorporated, and the updated point cloud will become {Pi=(xi, yi, zi, fi, ri, gi, bi, pi, ci,), i=1, 2, . . . , N}, where the vector (fi, ri, gi, bi, pi, ci,) represents the fluorescence, red, green, blue, PET and CT data associated with individual points in the 3D volume,” (page 22, para [0195]; Fig 19).
“Continuing, at step 1040, the process 1000 calculates a transformation matrix between the intraoperative image space and the preoperative image space. Next, at step 1050 the intraoperative imaging data, such as fluorescence image data and color reflectance data, is registered to the preoperative image space,” (page 16, para [0162]).
“the image registration can be based on point clouds or on the surface models that are rendered from the image data, or combinations thereof…It should also be appreciated that the 3D topography data may be represented using point clouds, or polygon meshes. Furthermore, the registration may be performed with point cloud representation, or polygon mesh representation, or combinations thereof,” (page 16, para [0163]).
The CT 3D model includes a point cloud / 3D mesh model based on the preoperative imaging data and / or the preoperative PET/CT volumetric data. The fluorescent 3D model includes a point cloud or 3D model generated based on fluorescence imaging data and / or a point cloud {Pi=(xi, yi, zi, fi, ri, gi, bi), i=1, 2, . . . , N} where fi represents the fluorescence. The claimed one model includes the updated point cloud {Pi=(xi, yi, zi, fi, ri, gi, bi, pi, ci,), i=1, 2, . . . , N}, where fi represents the fluorescence and ci represents the CT data associated with individual points in the 3D volume.).
Liu is not relied upon teaching but Ross teaches the image is of an anatomical landmark in which an anatomical structure near a surgical target of the patient is highlighted with a fluorescent contrast agent (“…the pulmonary artery (or more specifically, the left-pulmonary artery) may be segmented from or isolated within the 3D model of the patient's lungs. When the endoscope 100 has been navigated to a position within the patient's thoracic cavity where the left-pulmonary artery is believed to be located, the ICG dye is injected into the patient's blood stream, which then diffuses through critical structures within the patient's lungs, causing these critical structures, including the left-pulmonary artery, to illuminate when viewed by the second camera 110. Illumination of the left-pulmonary artery is then identified within the images captured by the second camera 110 by the workstation 400, which then correlates or otherwise registers the real-time location of the left-pulmonary artery within the images captured by the second camera 110 with the 3D model using the left-pulmonary artery as the mesh point or landmark to more accurately reflect the position of the endoscope 100 within the 3D model illustrated on the display 220,” (page 5, para [0055]).
“The endoscope 100 is navigated to a position adjacent the target tissue in step 610 and the workstation 400 segments or otherwise isolates critical structures, such as the left-pulmonary artery, in the 3D model in step 612. In step 614, ICG dye or any other suitable dye is injected into the patient's blood stream and diffuses within structures of the patient's anatomy. The ICG dye is illuminated using NIR light and captured by the second camera 110 in step 616 to identify critical structures within the patient's anatomy. In step 618, the targeted critical structure identified in step 612 is identified in the real-time images captured by the second camera 110 using fluorescence, and in step 620, the identified targeted critical structure in the captured real-time images is registered to the 3D model using the identified targeted critical structure as the mesh point or landmark. Once registered, the 3D model is updated in step 622 to accurately reflect the real-time position of the endoscope 100 within the 3D model and with respect to the target tissue, such that the target tissue may be treated accurately and/or a biopsy may be obtained more accurately,” (page 5, para [0060]).
An anatomical landmark includes using the left-pulmonary artery as the mesh point or landmark. An anatomical structure includes the left-pulmonary artery. Highlighted with a fluorescent contrast agent includes ICG dye causing critical structures, including the left-pulmonary artery, to illuminate. A surgical target includes target tissue.).
Before the effective filling date of the claimed invention, it would have been obvious to one having ordinary skill in the art to apply the teachings of Ross to Liu. The motivation would have been “to improve the accuracy of the registration,” (Ross; page 5, para [0058]).
Regarding claim 4, Liu in view of Ross teach the method of claim 1, wherein the fluorescence process comprises: waiting until the fluorescent contrast agent injected to target the anatomical structure is deposited (Ross; “…utilizing dyes that target or otherwise attach to specific structures within the patient's body may reduce the amount of time it takes to identify the target critical structure within the images captured by the second camera 110. In this manner, a dye that attaches to a specific type of lesion, such as a tumor, may be injected and then the patient's anatomy may be monitored until the targeted structure is illuminated by the dye in the images captured by the second camera,” (page 5, para [0054]-[0055]). Waiting until the fluorescent contrast agent is deposited includes monitoring until the targeted structure is illuminated by the dye.); and
capturing an image using multiple near-infrared cameras (Liu; “For fluorescence imaging, the fluorescence detection module 510 may comprise a NIR-sensitive CMOS sensor (MT9V032, Aptina). This sensor offers a high quantum efficiency of about 40% at 800 nm, and a high frame rate of about 60 fps. In addition, this sensor uses a NIR-sensitive monochrome sensor, which improves the detection limit to reach a pM level ICG (indocyanine green) concentration applied to the target object 324, which is used to facilitate fluorescence imaging,” (page 13, para [0145]; Fig 7A).
“…the fluorescence imaging modules 510 and 510A to capture fluorescence information of the target object 324,” (Liu; page 10, para [0129]; Fig 7A).
A near-infrared camera includes the fluorescence detection module 510 comprising a NIR-sensitive CMOS sensor. From at least Figure 7A and (Liu; page 10, para [0129]) it is clear that multiple near-infrared cameras are used to generate a fluorescent image.) when the fluorescent contrast agent is deposited (Ross; “…utilizing dyes that target or otherwise attach to specific structures within the patient's body may reduce the amount of time it takes to identify the target critical structure within the images captured by the second camera 110. In this manner, a dye that attaches to a specific type of lesion, such as a tumor, may be injected and then the patient's anatomy may be monitored until the targeted structure is illuminated by the dye in the images captured by the second camera,” (page 5, para [0054]-[0055]).
“When the ICG dye is illuminated with certain frequencies of light in the IR or NIR range, the ICG fluoresces an infrared color that can be captured by a camera such as the second camera 110. Capturing the NIR images can be triggered automatically or as directed by the clinician... Alternatively, the system can perform this task automatically by detecting sudden changes in the image,” (page 4, para [0050]).).
Before the effective filling date of the claimed invention, it would have been obvious to one having ordinary skill in the art to apply the teachings of Ross to Liu. The motivation would have been to “to improve the accuracy of the registration,” (Ross; page 5, para [0058]) and / or to “reduce the amount of time it takes to identify the target critical structure,” (page 5, para [0054]).
Claim 2 is rejected under 35 U.S.C. 103 as being unpatentable over Liu in view of Ross in further view of Kadoury et al. (US 20140193053 A1; Kadoury).
Regarding claim 2, Liu in view of Ross teach the method of claim 1, wherein the CT process comprises: performing a computed tomography (CT) scan of the patient (Liu; “Initially, at step 1610, prior to surgery, the surgical target object 324 will undergo a PET/CT scan to acquire preoperative image data,” (page 22, para [0195]).); and segmenting the CT image captured via the CT scan (Liu; “Briefly, a surface mesh or volumetric mesh can be generated based on preoperative tomographic imaging data (e.g. MRI, CT, SPECT, PET, 3D ultrasound, OCT, etc.). Image segmentation can be performed to isolate the organ of interest. Features and landmarks can be extracted from both the preoperative image model (based on tomographic imaging data) and the intraoperative image model (3D topography captured by our invention). A first image registration is performed based on the features and landmarks identified,” (page 29, para [0241]).), by using a selected surgical target and anatomical structure of the patient ((Liu; page 29, para [0241]).
Ross; “The endoscope 100 is navigated to a position adjacent the target tissue in step 610 and the workstation 400 segments or otherwise isolates critical structures, such as the left-pulmonary artery, in the 3D model in step 612,” (page 5, para [0060]).
“the software stored in the memory 404 may identify and segment out a targeted critical structure within the 3D model. It is envisioned that the segmentation may be performed automatically, manually, or a combination of both. The segmentation process isolates the targeted critical structure from the surrounding tissue in the 3D model and identifies its position within the 3D model,” (Ross; page 3, para [0045]).
A selected surgical target includes Liu’s organ of interest and / or Ross’ target tissue. An anatomical structure includes the left-pulmonary artery.).
Before the effective filling date of the claimed invention, it would have been obvious to one having ordinary skill in the art to apply the teachings of Ross to Liu. The motivation would have been “to improve the accuracy of the registration,” (Ross; page 5, para [0058]).
Liu in view of Ross is not relied upon teaching but Kadoury teaches performing a computed tomography (CT) scan of the patient including the fiducial marker attached to the patient's skin (“Attaching fiducial markers to a patient's skin before taking a high-resolution 3D image is one widely used method to facilitate spatial registration. These markers can easily be localized in the image, i.e., their spatial coordinates in an image coordinate system can be determined precisely,” (page 1, para [0005]).
“Surgery and minimally invasive therapy require reliable, precise navigation along predefined paths to predetermined target points. Many kinds of interventions are guided based on pre-procedural high-resolution three-dimensional (3D) images taken from a region of interest. Typically a computed tomography (CT) or magnetic resonance (MR) scanner acquires images before an intervention (pre-operative images). These modalities are not real-time, at least when high resolution is required,” (page 1, para [0004]).).
Before the effective filling date of the claimed invention, it would have been obvious to one having ordinary skill in the art to apply the teachings of Kadoury to Liu in view of Ross. The motivation would have been to enable “semi-automatic or fully automatic registration,” (Kadoury; para [0025]) and / or to improve the precision and / or accuracy of registration.
Claim 3 is rejected under 35 U.S.C. 103 as being unpatentable over Liu in view of Ross in further view of Zhang et al. (US 20240024011 A1; hereinafter; Zhang).
Regarding claim 3, Liu in view of Ross is not relied upon teaching but Zhang teaches the method of claim 1, wherein the CT image-based 3D reconstruction comprises reconstructing the CT image into a 3D model by using marching cube algorithm-based 3D reconstruction technology (“Perform three-dimensional reconstruction (such as Marching Cube) on preoperative (before reduction) computed tomography (CT) data, and convert the preoperative CT data into 3D model data,” (page 5, para [0052]).
“Rigidly register preoperative model data with reconstructed ultrasonic point set data (for example, by using an iterative closest point (ICP) algorithm), so as to obtain a mapping relationship between a preoperative CT image space and an intraoperative ultrasonic image 23,” (page 5, para [0053]).).
Before the effective filling date of the claimed invention, it would have been obvious to one having ordinary skill in the art to apply the teachings of Kadoury to Liu in view of Ross. The motivation would have been to improve the speed and / or efficiency of reconstruction and / or create smoother 3D surfaces.
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over Liu in view of Ross in further view of Hladio et al. (US 20190133695 A1; hereinafter Hladio).
Regarding claim 5, Liu in view of Ross teaches the method of claim 1, wherein the registration comprises: Liu; the controller 130 acquires preoperative imaging data, such as PET/CT image data. Next, at step 1020, the process creates a point cloud or 3D mesh model, which is based on the preoperative imaging data…” (page 16, para [0162]).
“…intraoperative imaging data is acquired from the optical imaging topography scanning system 100 of the present invention, such as fluorescence imaging data, in order to generate a point cloud or 3D model, as indicated at step 1030,” (Liu; page 16, para [0162]; Fig 12).
“After co-registration, the preoperative PET/CT volumetric data will be incorporated, and the updated point cloud will become {Pi=(xi, yi, zi, fi, ri, gi, bi, pi, ci,), i=1, 2, . . . , N}, where the vector (fi, ri, gi, bi, pi, ci,) represents the fluorescence, red, green, blue, PET and CT data associated with individual points in the 3D volume,” (Liu; page 22, para [0195]; Fig 19).
“the image registration can be based on point clouds or on the surface models that are rendered from the image data, or combinations thereof…It should also be appreciated that the 3D topography data may be represented using point clouds, or polygon meshes. Furthermore, the registration may be performed with point cloud representation, or polygon mesh representation, or combinations thereof,” (Liu; page 16, para [0163]).);
Liu in view of Ross is not relied upon teaching that the registering the CT 3D model and the fluorescent 3D model into one model is comprised by a coarse registration.
Liu in view of Ross is not relied upon teaching fine registration of more precisely registering the model obtained after performing the coarse registration.
Hladio teaches that the registering the CT 3D model and the fluorescent 3D model into one model is comprised by a coarse registration (“Attempting to match a plurality of points to locations on a digital model may comprise one or more of a coarse registration and a fine registration. A method of coarse registration between a digital image and the anatomical points may be performed by receiving, via the input system, in the computer anatomical points to identify gross anatomical points on the anatomy in a first coordinate frame, receiving input to identify locations on the digital image corresponding to the anatomical points in the digital image coordinate frame, and determining a transformation to map the anatomical points to the identified locations in the respective coordinate systems. After determining a coarse registration, steps to perform a fine registration may follow, in which additional anatomical points are gathered and registered to the digital image coordinate frame. Additional computations may be performed using an initial guess of registration; at least in part to determine a transformation that aligns the digital image data and the received anatomical points with higher accuracy. For example, the computations may be an optimization operation such as iterative close point (ICP), where Euclidean distance between identified points on the digital image and the plurality of anatomical points is minimized, using the coarse registration to define an initial condition for the optimization operation,” (page 4, para [0044]).
“receive an initial coarse patient registration correlating the location of the patient in three-dimensional space with the digital model,” (page 2, para [0012]).
“The digital model may be a three-dimensional digital model (e.g. based on a segmented CT or MRI Scan, an EOS 3D reconstruction, etc.),” (page 6, para [0059]).).
Hladio teaches fine registration of more precisely registering the model obtained after performing the coarse registration (“Attempting to match a plurality of points to locations on a digital model may comprise one or more of a coarse registration and a fine registration. A method of coarse registration between a digital image and the anatomical points may be performed by receiving, via the input system, in the computer anatomical points to identify gross anatomical points on the anatomy in a first coordinate frame, receiving input to identify locations on the digital image corresponding to the anatomical points in the digital image coordinate frame, and determining a transformation to map the anatomical points to the identified locations in the respective coordinate systems. After determining a coarse registration, steps to perform a fine registration may follow, in which additional anatomical points are gathered and registered to the digital image coordinate frame. Additional computations may be performed using an initial guess of registration; at least in part to determine a transformation that aligns the digital image data and the received anatomical points with higher accuracy. For example, the computations may be an optimization operation such as iterative close point (ICP), where Euclidean distance between identified points on the digital image and the plurality of anatomical points is minimized, using the coarse registration to define an initial condition for the optimization operation,” (page 4, para [0044]).
More precisely registering the model obtained after performing the coarse registration includes using the coarse registration to define an initial condition for the optimization operation.).
Before the effective filling date of the claimed invention, it would have been obvious to one having ordinary skill in the art to apply the teachings of Hladio to Liu in view of Ross. The motivation would have been to improve the accuracy of alignment and / or prevent local minima failures and / or improve convergence speed.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ERICA G THERKORN whose telephone number is (571)272-2939. The examiner can normally be reached Monday - Friday 9:00am - 5:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Devona Faulk can be reached at 571-272-7515. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ERICA G THERKORN/Examiner, Art Unit 2618
/DEVONA E FAULK/Supervisory Patent Examiner, Art Unit 2618