DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1-4, 15, 19, 20 and 22-24 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Wang et al. (US 20190287229).
Regarding claim 1, Wang et al. discloses a method of registering frames of a medical video comprising, at a computing system:
receiving a series of video frames captured by a medical imager imaging a scene of interest of a patient (“The system receives high frame-rate images captured from human gastrointestinal (GI) tract by a capsule camera in step 210” at paragraph 0048, line 4);
for a first frame of the series of video frames, determining a plurality of registration estimates for registering the first frame to a second frame of the series of video frames, wherein each registration estimate is determined according to a different one of a plurality of registration methods (“As mention previously, the images captured by capsule camera not only contain deformation, but also have global rigid transformation. Accordingly, the global transformation is introduced to work with deformation. In one embodiment, global transformation and deformable registration are applied iteratively. The global transformation can be applied first and followed by the deformable registration. Alternatively, the deformable registration can be applied first and followed by the global transformation” at paragraph 0047, line 1);
determining registration accuracies for the plurality of registration estimates (“To determine whether the registration is good, several criteria may be used. For example, the overlap between two images after registration can be computed. If the overlap percentage is below a certain threshold (e.g. 75% of the reference image), the registration is rejected. The maximum displacement of the deformation may also be calculated. If large deformation (e.g. 40 pixels) is found, the registration is rejected. In yet another example, a quality measure of the registration can be calculated. The quality measure may be related to information preserved for clinical purpose. For example, a measure that is capable of detecting a small amount of changes between two frames. This measure can be computed for three channels (e.g. RGB) of the difference image between R and warped F. An average filter of 30×30 pixels can be applied to the difference image and the maximum difference value is used as the score. The registration can be rejected if the score exceeds a threshold (e.g. 0.18)” at paragraph 0024);
selecting one of the plurality of registration estimates based on the registration accuracies (“In yet another embodiment, a pre-defined set of global transformations can be used and the deformable registration is performed at each transformation. The global transformation that achieves the best match is used as the selected global transform. Its associated deformable registration is used as the selected deformable registration” at paragraph 0047, third to last sentence); and
storing in a memory the selected registration estimate as a registration of the first frame to the second frame (“Motion models among the high frame-rate images are derived by applying image registration to the high frame-rate images in step 220. The high frame-rate images are stitched according to the motion models to generate stitching outputs comprising stitched images and non-stitched images in step 230” at paragraph 0048, third to last sentence).
Regarding claim 2, Wang et al. discloses a method wherein the series of video frames comprises at least one frame that was captured between the first frame and the second frame, and the plurality of registration methods comprises at least one registration method that includes calculating a transform from the first frame directly to the second frame (“While the registration for the high frame-rate images may take advantage of the additional bridging images to provide intermediate motion models between two regular images, the possibility to stitch two regular images has been improved. Nevertheless, it is not guaranteed that bridging images always help to improve stitching. Accordingly, in one embodiment, registration directly between two regular images is checked if the two regular images with the intervening bridging images cannot be stitched. In other words the system using bridging images to assist image stitching will never be worse than the conventional approach in any case according to this embodiment.” at paragraph 0043).
Regarding claim 3, Wang et al. discloses a method wherein the plurality of registration methods comprises at least one registration method that comprises calculating a transform from the first frame to an intermediate frame that was captured between the first and second frames (“When image registration is applied to the high frame-rate images, the registration between regular images is assisted by the intervening bridging images. For example, registration can be applied between regular image 1 and bridging image 2, between bridging images 2 and 3, and between bridging image 3 and regular image 4” at paragraph 0038, line 7).
Regarding claim 4, Wang et al. discloses a method wherein the at least one registration method comprises combining the transform from the first frame to the intermediate frame with a plurality of transforms associated with frames captured between the intermediate frame and the second frame (“When image registration is applied to the high frame-rate images, the registration between regular images is assisted by the intervening bridging images. For example, registration can be applied between regular image 1 and bridging image 2, between bridging images 2 and 3, and between bridging image 3 and regular image 4” at paragraph 0038, line 7).
Regarding claim 15, Wang et al. discloses a method wherein selecting the one of the plurality of registration estimates based on the registration accuracies comprises comparing a first registration accuracy corresponding to a first registration estimate to a threshold, and in accordance with the first registration accuracy not meeting the threshold, comparing the first registration accuracy to a second registration accuracy corresponding to a second registration estimate (“To determine whether the registration is good, several criteria may be used. For example, the overlap between two images after registration can be computed. If the overlap percentage is below a certain threshold (e.g. 75% of the reference image), the registration is rejected. The maximum displacement of the deformation may also be calculated. If large deformation (e.g. 40 pixels) is found, the registration is rejected. In yet another example, a quality measure of the registration can be calculated. The quality measure may be related to information preserved for clinical purpose. For example, a measure that is capable of detecting a small amount of changes between two frames. This measure can be computed for three channels (e.g. RGB) of the difference image between R and warped F. An average filter of 30×30 pixels can be applied to the difference image and the maximum difference value is used as the score. The registration can be rejected if the score exceeds a threshold (e.g. 0.18)” at paragraph 0024).
Regarding claim 19, Wang et al. discloses a method wherein the medical imager is an endoscopic imager (“The system receives high frame-rate images captured from human gastrointestinal (GI) tract by a capsule camera in step 210” at paragraph 0048, line 4).
Regarding claim 20, Wang et al. discloses a method comprising registering pixel data of the first frame to pixel data of the second frame based on the selected registration estimate (“In yet another embodiment, a pre-defined set of global transformations can be used and the deformable registration is performed at each transformation. The global transformation that achieves the best match is used as the selected global transform. Its associated deformable registration is used as the selected deformable registration” at paragraph 0047, third to last sentence).
Regarding claim 22, Wang et al. discloses a system for registering frames of a medical video comprising one or more processors and memory storing one or more programs for execution by the one or more processors, the one or more programs including instructions that when executed by the one or more processors (“The above method can be implemented as computer programs on a machine-readable medium to be executed on a computer” at paragraph 0051, line 1) cause the system to perform a method comprising:
receiving a series of video frames captured by a medical imager imaging a scene of interest of a patient (“The system receives high frame-rate images captured from human gastrointestinal (GI) tract by a capsule camera in step 210” at paragraph 0048, line 4);
for a first frame of the series of video frames, determining a plurality of registration estimates for registering the first frame to a second frame of the series of video frames, wherein each registration estimate is determined according to a different one of a plurality of registration methods (“As mention previously, the images captured by capsule camera not only contain deformation, but also have global rigid transformation. Accordingly, the global transformation is introduced to work with deformation. In one embodiment, global transformation and deformable registration are applied iteratively. The global transformation can be applied first and followed by the deformable registration. Alternatively, the deformable registration can be applied first and followed by the global transformation” at paragraph 0047, line 1);
determining registration accuracies for the plurality of registration estimates (“To determine whether the registration is good, several criteria may be used. For example, the overlap between two images after registration can be computed. If the overlap percentage is below a certain threshold (e.g. 75% of the reference image), the registration is rejected. The maximum displacement of the deformation may also be calculated. If large deformation (e.g. 40 pixels) is found, the registration is rejected. In yet another example, a quality measure of the registration can be calculated. The quality measure may be related to information preserved for clinical purpose. For example, a measure that is capable of detecting a small amount of changes between two frames. This measure can be computed for three channels (e.g. RGB) of the difference image between R and warped F. An average filter of 30×30 pixels can be applied to the difference image and the maximum difference value is used as the score. The registration can be rejected if the score exceeds a threshold (e.g. 0.18)” at paragraph 0024);
selecting one of the plurality of registration estimates based on the registration accuracies (“In yet another embodiment, a pre-defined set of global transformations can be used and the deformable registration is performed at each transformation. The global transformation that achieves the best match is used as the selected global transform. Its associated deformable registration is used as the selected deformable registration” at paragraph 0047, third to last sentence); and
storing in a memory the selected registration estimate as a registration of the first frame to the second frame (“Motion models among the high frame-rate images are derived by applying image registration to the high frame-rate images in step 220. The high frame-rate images are stitched according to the motion models to generate stitching outputs comprising stitched images and non-stitched images in step 230” at paragraph 0048, third to last sentence).
Regarding claim 23, Wang et al. discloses a system comprising the medical imager (“The system receives high frame-rate images captured from human gastrointestinal (GI) tract by a capsule camera in step 210” at paragraph 0048, line 4).
Regarding claim 24, Wang et al. discloses a non-transitory computer readable storage medium storing one or more programs for execution by one or more processors of a computing system, the one or more programs including instructions for causing the computing system to perform (“The above method can be implemented as computer programs on a machine-readable medium to be executed on a computer” at paragraph 0051, line 1) a method comprising:
receiving a series of video frames captured by a medical imager imaging a scene of interest of a patient (“The system receives high frame-rate images captured from human gastrointestinal (GI) tract by a capsule camera in step 210” at paragraph 0048, line 4);
for a first frame of the series of video frames, determining a plurality of registration estimates for registering the first frame to a second frame of the series of video frames, wherein each registration estimate is determined according to a different one of a plurality of registration methods (“As mention previously, the images captured by capsule camera not only contain deformation, but also have global rigid transformation. Accordingly, the global transformation is introduced to work with deformation. In one embodiment, global transformation and deformable registration are applied iteratively. The global transformation can be applied first and followed by the deformable registration. Alternatively, the deformable registration can be applied first and followed by the global transformation” at paragraph 0047, line 1);
determining registration accuracies for the plurality of registration estimates (“To determine whether the registration is good, several criteria may be used. For example, the overlap between two images after registration can be computed. If the overlap percentage is below a certain threshold (e.g. 75% of the reference image), the registration is rejected. The maximum displacement of the deformation may also be calculated. If large deformation (e.g. 40 pixels) is found, the registration is rejected. In yet another example, a quality measure of the registration can be calculated. The quality measure may be related to information preserved for clinical purpose. For example, a measure that is capable of detecting a small amount of changes between two frames. This measure can be computed for three channels (e.g. RGB) of the difference image between R and warped F. An average filter of 30×30 pixels can be applied to the difference image and the maximum difference value is used as the score. The registration can be rejected if the score exceeds a threshold (e.g. 0.18)” at paragraph 0024);
selecting one of the plurality of registration estimates based on the registration accuracies (“In yet another embodiment, a pre-defined set of global transformations can be used and the deformable registration is performed at each transformation. The global transformation that achieves the best match is used as the selected global transform. Its associated deformable registration is used as the selected deformable registration” at paragraph 0047, third to last sentence); and
storing in a memory the selected registration estimate as a registration of the first frame to the second frame (“Motion models among the high frame-rate images are derived by applying image registration to the high frame-rate images in step 220. The high frame-rate images are stitched according to the motion models to generate stitching outputs comprising stitched images and non-stitched images in step 230” at paragraph 0048, third to last sentence).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 16-18 and 21 are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Wang et al. and Kohler et al. (“Comparison of image registration methods for combining laparoscopic video and spectral image data”).
Regarding claim 16, Wang et al. discloses the elements of claim1 as described above.
Wang et al. does not explicitly disclose that at least one registration method of the plurality of registration methods comprises determining at least one homography estimate.
Kohler et al. teaches a method in the same field of endeavor of medical image registration, wherein at least one registration method of the plurality of registration methods comprises determining at least one homography estimate (“Image transformation was performed with: (a) perspective transformation using a single homography (SH) from RANSAC34 with a reprojection error threshold of 5, (b) hierarchical multi-affine (HMA) approach, (c) image deformation with affine Moving Least Squares (MLS)12, and (d) multi-grid deep homography estimation (MG-DHNN)” at page 3, Outlier rejection and image transformation, line 1).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to utilize the homography measure as taught by Kohler et al. in the registration estimation of Wang et al. to quantify the motion of feature points from one frame to the next.
Regarding claim 17, Kohler et al. discloses a method wherein the at least one homography estimate is determined using a random sample consensus algorithm (“Image transformation was performed with: (a) perspective transformation using a single homography (SH) from RANSAC with a reprojection error threshold of 5” at page 3, Outlier rejection and image transformation, line 1).
Regarding claim 18, Kohler et al. discloses method wherein the at least one homography estimate is determined using a machine learning-based algorithm (“Image transformation was performed with: (a) perspective transformation using a single homography (SH) from RANSAC with a reprojection error threshold of 5, (b) hierarchical multi-affine (HMA) approach, (c) image deformation with affine Moving Least Squares (MLS)12, and (d) multi-grid deep homography estimation (MG-DHNN)” at page 3, Outlier rejection and image transformation, line 1).
Regarding claim 21, Wang et al. discloses a method wherein the series of video frames is associated with a first imaging modality (the capsule endoscope images constitute an imaging modality).
Wang et al. does not explicitly disclose that the method comprises registering a frame of a series of video frames associated with a second imaging modality based on the selected registration estimate.
Kohler et al. teaches a method in the same field of endeavor of medical image registration, wherein the method comprises registering a frame of a series of video frames associated with a second imaging modality based on the selected registration estimate (“Two main steps are required for the augmentation of the video with the oxygenation maps obtained from HSI. First, a one-time registration based on manual annotation of both modalities has to be performed to align the video and hyperspectral images. The result of this first step is illustrated in Fig. 3 and a detailed description can be found in Köhler et al.5. This calibration does not have to be repeated before surgery because the arrangement of the color and hyperspectral image sensor is fixed inside the camera housing. Second, during the HSI record, one image of the video is saved as a start frame and will be used for the registration with all consecutive frames. The video is augmented by applying the transformations found in steps one and two to the oxygenation map and creating a semitransparent overlay with the grayscale image of the current video frame” at page 5, Qualitative evaluation: intraoperative laparoscopic hyperspectral imaging, paragraph 2, line 1).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to utilize the cross-modality registration as taught by Kohler et al. using the endoscope data of Wang et al. to be able to view the perfusion data of the localized anatomy within the video data.
Allowable Subject Matter
Claims 5-14 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter: While registration to intermediate key frames is known, the bridging image of Wang et al. cannot be replaced by a key frame; thus it is not obvious that the intermediate frame is a key frame and the at least one registration method comprises combining the transform from the first frame to the key frame with a second transform from the key frame to the second frame as required by claim 5; the prior art also does not disclose determining a location of the first frame relative to the second frame based on a first registration estimate, determining whether a key frame database includes a key frame that is sufficiently close in location to the first frame, in accordance with the key frame database not including a key frame that is sufficiently close in location to the first frame, adding the first frame to the key frame database, and in accordance with the key frame database including a key frame that is sufficiently close in location to the first frame: comparing a registration accuracy associated with the key frame and a registration accuracy associated with the first registration estimate for the first frame, and replacing the key frame with the first frame if the registration accuracy associated with the first registration estimate is greater than the registration accuracy associated with the key frame as required by claim 14.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Kumar et al. (US 20120316421) is pertinent as disclosing a capsule endoscope registration method utilizing a best matched registration from a plurality of registrations. Taruttis et al. (US 20240378735) discloses a multi-endoscope registration system. Ren et al. (US 10,987,172) discloses an adaptive image registration for ophthalmic applications wherein a plurality of registration methods are disclosed.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KATRINA R FUJITA whose telephone number is (571)270-1574. The examiner can normally be reached Monday - Friday 9:30-5:30 pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Sumati Lefkowitz can be reached at 5712723638. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KATRINA R FUJITA/Primary Examiner, Art Unit 2672
Prosecution Insights