Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
This action is in response to the applicant's communication filed on 01/29/2024. In virtue of this communication, claims 1-20 filed on 01/29/2024 are currently pending in the instant application.
Information Disclosure Statement
The information Disclosure statement (IDS) form PTO-1449, filed on 01/29/2024 are in compliance with the provisions of CFR 1.97. Accordingly, the information disclosed therein was considered by the examiner.
Drawings
The drawings were received on 01/29/2024 have been reviewed by Examiner and they are acceptable.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1, 6-7, 10-12, 15-17, and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lawrence et al. (US 2012/0172666), further in view of Shen et al. (US 2022/0319031).
As per claim 1, A method for dynamically assessing airway deformation comprising:
“receiving endoscopic image data of an airway;”(Lawrence, ¶[0023] discloses methods for the direct visualization of a body lumen are provided. Such methods generally comprise: inserting the catheter of an endoscope of the invention into a subject… directly visualizing said body lumen via the image guide. In certain embodiments, the body lumen is selected from the group consisting of lung airways, bronchial airways, and sinus cavities.)
“generating depth data from the endoscopic image data;”(Lawrence, ¶[0108] discloses In one run, a 1.0 mm nominal outside diameter of a fiber bundle, consisting of fibers approximately 4 .mu.m in diameter, supporting approximately 30,000 fibers was used for image transmission. Lenses on the fiber tip can provide a depth of field from 1 mm up to 10 mm, with an angular field of view of 55 degrees. ¶[0110] Lenses on the fiber tip can provide a depth of field from 1 mm to 10 mm, with an angular field of view of 55 degrees. This field of view can be canted to support visualization of the sides of the passage by combining a small prism with the distal lens. The bending radius of these image guides is as small as 10 mm, which is smaller than the bend radius of the conventional bronchoscope through which the MFS passes, and is suitable for navigating the branches of the many body cavities including the peripheral airways.)
“and generating a dynamic representation of the airway based on the endoscopic image data and the depth data.” (Lawrence,¶[0105] discloses a MFS of the invention may use in lung and airway applications to support direct visualization and access for diagnostic, treatment and surgical procedures. ¶[0107] discloses MFS technology offers several distinct advantages that improve the quality of care, including: support of direct visualization, access, and real-time imaging; access to previously inaccessible regions of the body; improved maneuverability and precise positioning; and supporting precise surgery and minimal trauma to the body cavities, e.g., nasal structures and airway passages.
However Lawrence does not explicitly disclose the following which would have been obvious in view of Shen from similar filed of endeavor “generating depth data from the endoscopic image data using a depth prediction model” (Shen, ¶[0135] discloses a corresponding depth map 435 is extracted from the video image data 92. The advantage of using a depth map 435 instead of a video image 92 is that the depth map 435 provides better generalizability than a corresponding video image 92. ¶[0137] discloses The localization module 95 can perform conversion from a video image 92 to a depth map 435 with a convolutional neural network (CNN) that performs domain adaptation. For example, using machine learning, the localization module 95 can extract multi-dimensional information from digital images such as the video images 92. The multi-dimensional information can be formed into the depth maps 435, with a mapping function. The mapping function Z=G(I) can be learned with supervised learning if known paired video images and corresponding depth maps are available. A dataset comprised of previously paired images and depth maps can be used to train the mapping function from the source domain (video images 92) to the target domain (depth maps 435) based on a deep feature space. Alternatively, when perfectly paired data are difficult to obtain, the mapping function can also be achieved by unsupervised learning methods such as generative adversarial learning with a cyclic consistency that does not require paired data. Further Shen also disclose a dynamic representation ¶[0149] disclose Referring to FIG. 16, in various embodiments, the process of generating second depth maps 455 and identifying one candidate depth map 465 of the plurality of second depth maps 455 is an iterative or continuous process 1600. the process 1600, for real-time localization/navigation, the imaging device 315 (e.g., camera) can continuously capture video images 92.)
Before the effective filing date of the claimed invention it would have been obvious to a person of ordinary skill in the art to combine Shen technique of Bronchoscopy camera pose estimation into Lawrence technique to provide the known and expected uses and benefits of Shen technique over Endoscope diagnostic technique of Lawrence. The proposed combination would have constituted a mere arrangement of old elements with each performing their known function, the combination yielding no more than one would expect from such an arrangement.
Therefore, it would have been obvious to a person of ordinary skill in the art to incorporate Shen to Lawrence in order to provide accurate diagnosis and treatment of lung airways. (Refer to Shen paragraph [0003].)
Claims 12 and 17 have been analyzed and are rejected for the reasons indicated in claim 1 above. Additionally, the rationale and motivation to combine the Lawrence and Shen references, presented in rejection of claim 1, apply to these claims.
As per claim 6, The method of claim 1, “wherein the dynamic representation of the airway comprises a sequence of point cloud surface representations, wherein the sequence of point cloud surface representations is generated from the endoscopic image data and the depth data.” (Shen, ¶[0018] discloses the control circuitry is further configured to represent the first depth map with a first point cloud and a second depth map with a second point cloud. The control circuitry can be further configured to determine deep geometric features based at least in part on the first and second point clouds using a first neural network. ¶[0147] discloses CNN can be employed to estimate the camera pose 475 parameters by solving the transformation matrix between a 3D point cloud of the airway model and the point cloud inverse projected from the first depth map 435. For example, as shown in FIG. 15, the control circuitry 60 can be further configured to represent the first depth map 435 with a first point cloud 710 and the plurality of second depth maps 455 with a second point cloud 720. Point clouds of the second depth maps 455 can be combined/merged into one point cloud (i.e., the second point cloud 720). The control circuitry 60 can be further configured to determine deep features 730 based at least in part on the first 710 and second 720 point clouds using a first neural network)
Claims 15 and 20 have been analyzed and are rejected for the reasons indicated in claim 6 above. Additionally, the rationale and motivation to combine the Lawrence and Shen references, presented in rejection of claim 6, apply to these claims.
As per claim 7, The method of claim 6 “further comprising determining an airway contour at an axial location of the airway from the dynamic representation of the airway.” (Shen, ¶[0135] discloses as captured by an imaging device 315 such as a video camera within a luminal network 130 (e.g., airway). The video image data 92 shows an example portion of the luminal network 130 with branches 150 as well as contours, textures, features, moisture droplets. )
Claims 16 have been analyzed and are rejected for the reasons indicated in claim 7 above.
As per claim 10, The method of claim 1 “further comprising identifying regions of the airway based on the endoscopic image data and the depth data.” (Lawrence, Abstract discloses a MicroFlex Scope (MFS). The MFS is a dexterous endoscope providing access, direct visualization, tissue sampling, and treatment, of body lumens. ¶[0023] discloses directly visualizing said body lumen via the image guide. In certain embodiments, the body lumen is selected from the group consisting of lung airways, bronchial airways. ¶[0058] discloses The MFS may also be used to enhance other procedures where a bronchoscope is used, including diagnosis and treatment of infections, diffuse lung diseases, and airway obstructions. )
As per claim 11, The method of claim 1 “further comprising identifying regions of the airway that contribute to obstructed breathing based on endoscopic image data and the depth data.” (Lawrence, Abstract discloses a MicroFlex Scope (MFS). The MFS is a dexterous endoscope providing access, direct visualization, tissue sampling, and treatment, of body lumens. ¶[0023] discloses directly visualizing said body lumen via the image guide. In certain embodiments, the body lumen is selected from the group consisting of lung airways, bronchial airways. ¶[0058] discloses The MFS may also be used to enhance other procedures where a bronchoscope is used, including diagnosis and treatment of infections, diffuse lung diseases, and airway obstructions.)
Claim(s) 2-3, 13-14, and 18-19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lawrence et al. (US 2012/0172666), in view of Shen et al. (US 2022/0319031), in view of Barak et al. (US 2019/0239838), further in view of Tong, Hon-Sing, et al. "Real-to-virtual domain transfer-based depth estimation for real-time 3D annotation in transnasal surgery: a study of annotation accuracy and stability." International Journal of Computer Assisted Radiology and Surgery 16.5 (2021).
As per claim 2, The method of claim 1, further comprising: However Lawrence as modified by Shen does not explicitly disclose the following which would have been obvious in view of Barak form similar filed of endeavor “generating the depth prediction model, wherein generating the depth prediction model comprises: generating a structural model of an airway from endoscopic image data and MRI data;” “generating data from the structural model” (Barak, ¶[0039] discloses The resulting volume generated from the MRI scan or CT scan is then utilized to create a navigation plan to facilitate the advancement of a navigation catheter (or other suitable medical device) through a bronchoscope and a branch of the bronchus of a patient to an area of interest. ¶[0072] and ¶[0079] discloses viewing a three dimensional model of the patient's “P's” airways, enables the identification of a target on the three dimensional model. the pathway plan, three dimensional model, and images derived therefrom, can be saved and exported to a navigation system for use during the navigation phase(s). The three-dimensional model may be manipulated to facilitate identification of target on the three-dimensional model or two-dimensional images, and selection of a suitable pathway through the patient's “P's” airways to access tissue located at the target can be made. ¶[0081] discloses Registration, is generally performed to coordinate locations of the three-dimensional model and two-dimensional images from the planning phase with the patient's “P's” airways as observed through the bronchoscope 830, and allow for the navigation phase to be undertaken with precise knowledge of the location of the sensor 844, even in portions of the airway where the bronchoscope 830 cannot reach. ¶[0082] discloses a shape resulting from this location data is compared to an interior geometry of passages of the three dimensional model generated in the planning phase, and a location correlation between the shape and the three dimensional model based on the comparison is determined.)
Before the effective filing date of the claimed invention it would have been obvious to a person of ordinary skill in the art to combine Barak technique of determining location of imaging medical devices into Lawrence as modified by Shen technique to provide the known and expected uses and benefits of Barak technique over Endoscope diagnostic technique of Lawrence as modified by Shen. The proposed combination would have constituted a mere arrangement of old elements with each performing their known function, the combination yielding no more than one would expect from such an arrangement.
Therefore, it would have been obvious to a person of ordinary skill in the art to incorporate Barak to Lawrence as modified by Shen in order to navigate the medical device to the target in a more safe and accurate manner (e.g., with unnecessary or no damage caused to other tissues and organs). (Refer to Barak paragraph [0004].)
However Lawrence as modified by Shen as modified by Barak does not explicitly disclose the following which would have been obvious in view of Tong form similar filed of endeavor “generating a training dataset comprising synthetic image data and synthetic depth data” (Tong, Figure 1, description discloses synthetic endoscopic images and real endoscopic images for depth estimation and image style transfer training, page 733, Col. 2, last paragraph discloses a dataset of 3600 synthetic endoscopic images and the corresponding ground truth depth maps were captured from the virtual camera while being moved inside the virtual nasal airway mode)“ and training the depth prediction model from the training dataset.” (Tong, page 734, Col. 2, section Depth estimation discloses The aim of the depth estimation model is to learn the mapping between the synthetic endoscopic image dataset and the corresponding ground truth depth. Page 735, Col. 2, Implementation of DNNs section second paragraph discloses To train the network synthetic endoscopic images and the corresponding depth maps were used as a subsequent f ine-tune training.)
Before the effective filing date of the claimed invention it would have been obvious to a person of ordinary skill in the art to combine Tong technique of use of endoscopic surgery into Lawrence as modified by Shen as modified by Barak technique to provide the known and expected uses and benefits of Tong technique over Endoscope diagnostic technique of Lawrence as modified by Shen as modified by Barak. The proposed combination would have constituted a mere arrangement of old elements with each performing their known function, the combination yielding no more than one would expect from such an arrangement.
Therefore, it would have been obvious to a person of ordinary skill in the art to incorporate Tong to Lawrence as modified by Shen as modified by Barak in order to increase surgical efficiency and aid with surgical training. (Refer to Tong paragraph page 732, Col. 1 paragraph 1.)
Claims 13 and 18 have been analyzed and are rejected for the reasons indicated in claim 2 above. Additionally, the rationale and motivation to combine the Lawrence , Shen, Barak, and Tong references, presented in rejection of claim 2, apply to these claims.
As per claim 3, The method of claim 2, “wherein the training dataset comprises at least one video sequence, wherein the at least one video sequence comprises the synthetic image data and the synthetic depth data.” (Tong, page 733, Col. 2, last paragraph discloses A dataset of 3600 synthetic endoscopic images and the corresponding ground truth depth maps were captured from the virtual camera while being moved inside the virtual nasal airway model along a pre-defined pathway. Page 735, Col1, paragraph 2 discloses RGB video frames were streamed from the endo scope during observation of the 3D-printed nasal airway phantom, which was static relative to the EM tracking field. Before passing video frames to i) Unity3D for visualization. Page 738, Col 1, discloses creating patient nasal airway video datasets .)
Claims 14 and 19 have been analyzed and are rejected for the reasons indicated in claim 3 above. Additionally, the rationale and motivation to combine the Lawrence , Shen, Barak, and Tong references, presented in rejection of claim 3, apply to these claims.
Claim(s) 4 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lawrence et al. (US 2012/0172666), in view of Shen et al. (US 2022/0319031), in view of Barak et al. (US 2019/0239838), further in view of Tong, Hon-Sing, et al. "Real-to-virtual domain transfer-based depth estimation for real-time 3D annotation in transnasal surgery: a study of annotation accuracy and stability." International Journal of Computer Assisted Radiology and Surgery (2021), further in view of Hunter et al. (US 2020/0265584).
As per claim 4, The method of claim 3, However Lawrence as modified by Shen as modified by Barak as modified by Tong does not explicitly disclose the following which would have been obvious in view of Hunter form similar filed of endeavor “wherein the at least one video sequence further comprises a parameter set, wherein the parameter set comprises one or more of camera exposure data, lighting data, airway surface data, and image artifact data.”(Hunter, ¶[0126] disclose a real-time image feed from bronchoscopic video camera 630 may be used to view the operation of the medical device. ¶[0131] discloses navigation system 70 may simulate and/or display orthogonal image slices, oblique or off-axis image slices, volume rendered images, segmented images, fused modality images, maximum intensity projection (MIPS) images, video, and video enhanced images.¶[0132] discloses navigation system 70 may be able to simulate the virtual volumetric scene using inspiration 3D airway model 410, expiration 3D airway model 412, and/or hybrid “Inspiration-Expiration” 3D airway model 414. Accordingly, navigation system 70 renders an internal view of 3D airway model(s) 410, 412, and/or 414 based on a virtual bronchoscope video camera position, for example, by applying certain surface properties (e.g., Lambertian), diffuse shading model(s), and perspective projection camera model(s). )
Before the effective filing date of the claimed invention it would have been obvious to a person of ordinary skill in the art to combine Hunter technique of use of Bronchoscopes for airway inspection into Lawrence as modified by Shen as modified by Barak as modified by Tong technique to provide the known and expected uses and benefits of Hunter technique over Endoscope diagnostic technique of Lawrence as modified by Shen as modified by Barak as modified by Barak as modified by Tong. The proposed combination would have constituted a mere arrangement of old elements with each performing their known function, the combination yielding no more than one would expect from such an arrangement.
Therefore, it would have been obvious to a person of ordinary skill in the art to incorporate Hunter to Lawrence as modified by Shen as modified by Barak as modified by Barak as modified by Tong in order to provide better visualizing, accessing, locating, real-time confirming, sampling, and manipulating a target tissue or area. (Refer to Hunter ¶[0005].)
Claim(s) 5 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lawrence et al. (US 2012/0172666), in view of Shen et al. (US 2022/0319031), in view of Ito et al. (US 2015/0272423).
As per claim 5, The method of claim 1, However Lawrence as modified by Shen does not explicitly disclose the following which would have been obvious in view of Ito form similar filed of endeavor “wherein the endoscopic image data comprises a sequence of monocular images.” (Ito, ¶[0061] The stereo measurement may be performed as explained below using the endoscope 3A including a monocular (single) image pickup apparatus 16.)
Before the effective filing date of the claimed invention it would have been obvious to a person of ordinary skill in the art to combine Ito technique of determining location of imaging medical devices into Lawrence as modified by Shen technique to provide the known and expected uses and benefits of Ito technique over Endoscope diagnostic technique of Lawrence as modified by Shen. The proposed combination would have constituted a mere arrangement of old elements with each performing their known function, the combination yielding no more than one would expect from such an arrangement.
Therefore, it would have been obvious to a person of ordinary skill in the art to incorporate Ito to Lawrence as modified by Shen in order to navigate the medical device to the target in a more safe and accurate manner (e.g., with unnecessary or no damage caused to other tissues and organs). (Refer to Ito paragraph [0004].)
Claim(s) 8 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lawrence et al. (US 2012/0172666), in view of Shen et al. (US 2022/0319031), in view of Hunter et al. (US 2020/0265584).
As per claim 8, The method of claim 7, However Lawrence as modified by Shen does not explicitly disclose the following which would have been obvious in view of Hunter form similar filed of endeavor “further comprising quantifying characteristics of the airway contour, the characteristics comprising a cross-sectional area and a deformation.” (Hunter, ¶[0082] discloses the hybrid “Inspiration-Expiration” 3D airway model 414 contains all of the structural information of the airways of patient 10 depicted in inspiration 3D airway model 410. However, this structural information is now more closely matched to the location, shape, and orientation of the airways of patient 10 depicted in expiration 3D airway model 412. Accordingly, the deformation vectors represent not only a change in location of the structure of the airway but a change in shape of the structure of the airway from inspiration to expiration. ¶[0085] discloses the deformation or vector field may also be applied to multiple image datasets or in a progressive way to create a moving underlying image dataset that matches the respiratory or cardiac motion of patient 10. ¶[0147] discloses the span of the eigenvectors corresponding to the two larger magnitude eigenvectors of the Hessian analysis (the tubular structure's cross-section plane) is used to create a gradient vector flow vector field, which is then used to assign a metric value. Alternatively, a metric value can be obtained via a circle fitting method using the gradient vector flow vector field. The analysis can be used to detect either relatively dark (e.g., airway lumens) or relatively bright (e.g., vasculature) tubular structures.)
Before the effective filing date of the claimed invention it would have been obvious to a person of ordinary skill in the art to combine Hunter technique of use of Bronchoscopes for airway inspection into Lawrence as modified by Shen technique to provide the known and expected uses and benefits of Hunter technique over Endoscope diagnostic technique of Lawrence as modified by Shen. The proposed combination would have constituted a mere arrangement of old elements with each performing their known function, the combination yielding no more than one would expect from such an arrangement.
Therefore, it would have been obvious to a person of ordinary skill in the art to incorporate Hunter to Lawrence as modified by Shen in order to provide better visualizing, accessing, locating, real-time confirming, sampling, and manipulating a target tissue or area. (Refer to Hunter ¶[0005].)
Claim(s) 9 is/are rejected under 35 U.S.C. 103 as being unpatentable over Lawrence et al. (US 2012/0172666), in view of Shen et al. (US 2022/0319031), in view of Hunter et al. (US 2020/0265584).
As per claim 9, The method of claim 6, However Lawrence as modified by Shen does not explicitly disclose the following which would have been obvious in view of Hunter form similar filed of endeavor “further comprising determining an airway deformation at an axial location of the airway.” (Hunter, ¶[0082] discloses the hybrid “Inspiration-Expiration” 3D airway model 414 contains all of the structural information of the airways of patient 10 depicted in inspiration 3D airway model 410. However, this structural information is now more closely matched to the location, shape, and orientation of the airways of patient 10 depicted in expiration 3D airway model 412. Accordingly, the deformation vectors represent not only a change in location of the structure of the airway but a change in shape of the structure of the airway from inspiration to expiration. ¶[0085] discloses the deformation or vector field may also be applied to multiple image datasets or in a progressive way to create a moving underlying image dataset that matches the respiratory or cardiac motion of patient 10. ¶[0131] discloses navigation system 70 may be able to simulate and display axial, coronal and oblique images based on the position and orientation (POSE) of localization element 610 of steerable catheter 600, as shown in panels 702, 704, and 706. To simulate these views, navigation system 70 may modify one or more images from image dataset 400 using known image manipulation techniques. Additionally, navigation system 70 may simulate and/or display orthogonal image slices, oblique or off-axis image slices, volume rendered images, segmented images, fused modality images, maximum intensity projection (MIPS) images, video, and video enhanced images.)
Before the effective filing date of the claimed invention it would have been obvious to a person of ordinary skill in the art to combine Hunter technique of use of Bronchoscopes for airway inspection into Lawrence as modified by Shen technique to provide the known and expected uses and benefits of Hunter technique over Endoscope diagnostic technique of Lawrence as modified by Shen. The proposed combination would have constituted a mere arrangement of old elements with each performing their known function, the combination yielding no more than one would expect from such an arrangement.
Therefore, it would have been obvious to a person of ordinary skill in the art to incorporate Hunter to Lawrence as modified by Shen in order to provide better visualizing, accessing, locating, real-time confirming, sampling, and manipulating a target tissue or area. (Refer to Hunter ¶[0005].)
Contact
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SHAGHAYEGH AZIMA whose telephone number is (571)272-1459. The examiner can normally be reached Monday-Friday, 9:30-6:30.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Vincent Rudolph can be reached at (571)272-8243. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SHAGHAYEGH AZIMA/Examiner, Art Unit 2671