DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) are: “a laryngeal examination device” in claim 1. This limitation includes the function language of “used to enter a larynx of a user for examination,” but provides no structural information in the claims. Thus, the laryngeal examination device is interpreted as a tube and/or scope device for intubation, such as a laryngoscope. This interpretation comes from 0029 of the specification. [0029] “The laryngeal examination device 11 is used to enter the user's larynx for examination, such as a laryngoscope, a stylet, a fiberscope, etc.”
Because this claim limitation is being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it is being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this limitation interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitations to avoid it being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation recites sufficient structure to perform the claimed function so as to avoid it being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 8, and 10 are rejected under 35 U.S.C. 103 as being unpatentable over Deng et al. (Automatic endoscopic navigation based on attention-based network for Nasotracheal Intubation. Biomedical Signal Processing and Control. Volume 86.) and further in view of Yang et al. (Endoscope Localization and Dense Surgical Scene Reconstruction for Stereo Endoscopy by Unsupervised Optical Flow and Kanade-Lucas-Tomasi Tracking. 2022 44th Annual International Conference of the IEEE Engineering in Medicine & Biology Society.), hereafter Yang.
Regarding claim 1, Deng teaches a multi-task real-time intubation assistant system (Deng teaches a machine learning system for assisting in intubation. [Introduction] “This work aims to automatically generate navigation information from endoscopic images to assist surgeons in steering the flexible endoscope during NTI.” NTI is nasotracheal intubation.), comprising:
a laryngeal examination device, used to enter a larynx of a user for examination; a photographic device, photographing the larynx from a travel direction of the laryngeal examination device to produce a laryngoscopic image (Deng teaches using an endoscope with a camera attached for capturing images during intubation. [Introduction] “This work aims to automatically generate navigation information from endoscopic images to assist surgeons in steering the flexible endoscope during NTI. To this end, a learning-based navigation method is proposed, which classifies anatomical landmarks and detects the heading target of the endoscopic tip. The core of the method is an end-to-end multi-task network, which automatically learns effective features from images. A convolutional attention module is designed to refine features along two primary dimensions of the image (i.e., the channel and spatial axis) to ensure the performance of our network.”); and
a control device, connected to the photographic device and the laryngeal examination device to receive the laryngoscopic image, wherein the control device comprises a processor and a memory, and the processor accesses positioning and navigation commands of the memory (In Section 2, Deng teaches the architecture of the attention-based CNN for identifying landmarks in images. Inherently, a computing device with at least a processor and memory is needed for running the CNN. Additionally, Deng teaches displaying navigational commands on a computer screen. [Section 2] “The center point (cx,cy) of the box denotes the heading target of the endoscopic tip. The computed navigation information is displayed on a computer screen to guide surgeons in operating the endoscope during NTI.”) to execute following modules:
a local attentive region proposal module, extracting a network output feature map of the laryngoscopic image through a feature extractor (In Section 2, Deng teaches an attention-based CNN for determining landmarks from low-level features extracted from endoscopic images. [Section 2.1] “At each convolutional layer l, the input feature map… Given the learned low-level features as input, the landmark classification component predicts one of the four landmarks.”),
generating a regional localization through an attention recurrent mechanism, scanning the regional localization through a region proposal network to generate a region of interest, and inputting the region of interest into the detector for classification and regression to generate an object-detecting output, wherein the object-detecting output corresponds to a specific organ within the larynx ([Section 2.1] “Given the learned low-level features as input, the landmark classification component predicts one of the four landmarks… The soft-max function takes a feature vector zi produced by the last layer as input and outputs the prediction probability Pi for each anatomical landmark i.” As shown in Figs. 3-4, the anatomical landmarks are the nose, throat, glottis, and trachea.);
a direction-detecting module, generating a guiding direction of the laryngeal examination device through a direction-detecting program (Fig. 5 shows the guiding direction output to the computer screen. See the crosshair which indicates the where the endoscope should go next. [Section 2.2] “Navigation information is generated from the low-level features extracted by the feature extractor… This work develops convolutional attention and incorporates it into the feature extractor (Fig. 1). The proposed attention module highlights informative features along the image’s two primary dimensions, namely the channel and spatial axis.”).
Although Deng teaches processing endoscopic images during intubation and determining a heading direction for the endoscope, Deng does not teach determining the endoscope’s spatial position using odometry. More specifically, Deng fails to teach a visual odometer module, detecting a moving distance of the photographic device through a visual odometer detecting program, and positioning the laryngeal examination device by the moving distance.
However, Yang teaches a visual odometer module, detecting a moving distance of the photographic device through a visual odometer detecting program ([Section I] “here we propose a hybrid visual odometry and dense scene reconstruction framework (END-VO).”), and
positioning the laryngeal examination device by the moving distance (Fig. 2 shows the tracking results of the examination device. The moving distance is tracked live.).
Deng and Yang are analogous in the art, because both teach methods of image-guided endoscopy. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Deng’s invention by using visual odometry for tracking the position of the endoscope. This modification would further enhance Deng’s invention by allowing for surgical scene reconstruction and tracking of the endoscope position ([Yang Introduction] “Endoscope tracking is also paramount for surgical scene reconstruction, which entails the fusion of dense depth maps and color images with estimated camera poses. The reconstruction benefits many downstream tasks, including visual analysis and image-to-patient registration. Hence, given the critical need for accurate endoscope tracking to achieve faithful surgical scene reconstruction, in this work, we target both visual odometry and scene reconstruction.”).
Regarding claim 8, Deng teaches recognizing organs in a region and determining the position of the endoscopic camera, and this information is used to determine the guiding direction to extend the tube for intubation. See Fig. 5. However, Deng fails to teach wherein the visual odometer detecting program compares feature points by an LK pyramid optical flow algorithm to calculate a rotation matrix and translation vector, and then obtains scale information by a checkerboard as an initial value to generate the moving distance.
However, Yang teaches wherein the visual odometer detecting program compares feature points by an LK pyramid optical flow algorithm to calculate a rotation matrix and translation vector ([Section II, A, 2] “This module infers the current camera location relative to the previous camera position and updates the camera pose in the map. The 3D landmarks identified in the mapping module are projected onto the previous frame using the estimated camera pose of the previous frame and then tracked on the current frame via Lucas-Kanade (L-K) optical flow 11. Given a set of 3D landmarks and their corresponding 2D projections, the current camera pose relative to the previous frame can be solved via the Perspective-n-Point (PnP) RANSAC, detailed in 2. The pose is then transformed into the world coordinate system (the first frame in the sequence) and stored in the map.”), and
then obtains scale information by a checkerboard as an initial value to generate the moving distance ([Section II, A, 5] “All camera poses, landmarks and point clouds of the map are visualized in real time. We apply the Truncated Signed Distance Function (TSDF) to the point clouds to reconstruct a global 3D mesh. The TSDF grids the space into equal voxels of the TSDF volume (Vtsdf) and sequentially averages the 3D locations and point cloud colors within each voxel.”).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Deng’s invention by utilizing deep learning-based scene reconstruction methods for depth estimation and pose estimation of the endoscopic camera. This modification allows for the use of techniques, such as Kanade-Lucas-Tomasi tracking, which account for the accumulation of tracking drift and sparse reconstruction due to the texture-less surfaces ([Yang, Section I] “To further mitigate the limitations associated with accumulated tracking drift and sparse reconstruction, here we propose a hybrid visual odometry and dense scene reconstruction framework (END-VO).”).
Regarding claim 10, Deng teaches further comprising a display device connected to the control device, wherein the display device displays the laryngoscopic image and positioning and navigation information of the laryngeal examination device ([Section 2] “The center point (cx, cy) of the box denotes the heading target of the endoscopic tip. The computed navigation information is displayed on a computer screen to guide surgeons in operating the endoscope during NTI.”).
Claim 2 is rejected under 35 U.S.C. 103 as being unpatentable over Deng in view of Yang, and further in view of Yeung et al. (US 10,646,288 B2), hereafter Yeung.
Regarding claim 2, Deng and Yang fail to teach the multi-task real-time intubation assistant system according to claim 1, wherein the direction-detecting program comprises a symmetry-detecting program or a groove edge-detecting program.
However, Yeung teaches the multi-task real-time intubation assistant system according to claim 1, wherein the direction-detecting program comprises a symmetry-detecting program or a groove edge-detecting program (Yeung teaches a method to find the lumen center by identifying contours in endoscopic images, and Yeung discloses that a Hough transform would be well known in the art for this method. The lumen center is used to find the guide line and provide steering commands for the endoscope. [Col. 60, lines 31-34] “Referring back to FIG. 15, a mucosal pattern method 1509 may be used to derive the location of the lumen center within an image and/or a confidence level for the determination of the location of the lumen center 1519.”).
Deng, Yang, and Yeung are analogous in the art, because all teach methods of image-guided endoscopy. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Deng’s invention by determining the lumen center using a symmetry or groove-edge detecting program. This modification allows for steering to be guided towards the centerline of the lumen which would minimize the contact of the endoscope to the walls of the lumen, and steering the endoscope towards the lumen centerline is well-known in the art (Yeung provides many references teaching automatically steering an endoscope towards the centerline of the lumen. [Col. 1, lines 26-39] “For example, by exploiting soft deformable structures that are capable of moving effectively through a complex environment like the inside the colon, one can significantly reduce pain and patient discomfort while minimizing the risk of colonic perforation… A variety of image processing-based approaches to identifying the center of the colon lumen and determining a navigation direction for an advancing colonoscope have been described in the literature…” See the background section of Yeung.).
Claims 3-5 are rejected under 35 U.S.C. 103 as being unpatentable over Deng in view of Yang and Yeung, and further in view of Vunjak-Novakovic (US 2020/0046943 A1).
Regarding claim 3, Deng, Yang, and Yeung fail to teach wherein the symmetry-detecting program horizontally flips the laryngoscopic image by a scale-invariant feature transformation algorithm and compares feature points to obtain a symmetry line.
However, Vunjak-Novakovic teaches wherein the symmetry-detecting program horizontally flips the laryngoscopic image by a scale-invariant feature transformation algorithm and compares feature points to obtain a symmetry line ([0133] “FIGS. 19A-19B depict the maneuvering of a steerable catheter using feature-based image registration and visual servo control methods. In FIG. 19A, specific features of airways such as edges, corners, and areas are extracted from CT scan images using feature detection algorithms such as Scale-invariant feature transform (SIFT) or Speeded up robust features (SURF). In FIG. 19B, the pose of a steerable catheter is maneuvered via a servo controller based on the location of target airways determined through matching of corresponding features (i.e., image registration) in the pre-obtained images and the images collected from the live video during navigation.”).
Deng, Yang, Yeung, and Vunjak-Novakovic are analogous in the art to the claimed invention, because all teach methods of image guided endoscopy or catheters. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to utilize a scale-invariant feature transformation algorithm for identifying features when determining the lumen center. As discussed in claim 2, Yeung renders obvious a need for determining the lumen center, and Vunjak-Novakovic discloses that a scale-invariant feature transformation algorithm is well-known in the art for determining image features. Thus, the use of a scale-invariant feature transformation algorithm for identifying features is the result of combining well-known elements in the art producing a predictable result ([Vunjak-Novakovic 0133] “specific features of airways such as edges, corners, and areas are extracted from CT scan images using feature detection algorithms such as Scale-invariant feature transform (SIFT) or Speeded up robust features (SURF).”).
Regarding claim 4, Deng and Yang fail to teach wherein the groove edge-detecting program detects edge line segments on both sides by Canny Edge Detection and Probability Hough Transform and generates a guide line from the edge line segments on both sides.
However, Yeung teaches wherein the groove edge-detecting program detects edge line segments on both sides by Canny Edge Detection and Probability Hough Transform and generates a guide line from the edge line segments on both sides (Yeung teaches a method to find the lumen center by identifying contours in endoscopic images, and Yeung discloses that a Hough transform would be well known in the art for this method. The lumen center is used to find the guide line and provide steering commands for the endoscope. [Col. 60, lines 40-47] “Any of a variety of edge detection or pattern detection algorithms known to those of skill in the art may be utilized in identifying contours, e.g., a Hough Transform and the like. The detected contours are then further analyzed to identify the location of the lumen center. In some cases, the lumen center may be identified by finding the intersection of the normal vectors for each of a series of contour segments.” Additionally, Yeung mentions Canny edge detection as a well-known method for processing the endoscopic images. [Col. 36, lines 13-24] “For example, given the particular topography or texture of the lumen wall such as the mucosal folds, features perceived in the image that indicate the presence of the folds may be recognized through image processing, and a centroid or locus of a dark region may be used to determine the location of the lumen center 501. Examples of image processing algorithms that may be suitable for use in the disclosed methods include, but are not limited to, Laplacian of Gaussian (LoG) methods, Canny edge detection methods, Canny-Deriche edge detection methods…”).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Deng’s invention by utilizing canny-edge detection and a probability Hough transform for identifying image features and determining a guide line. This modification would utilize well-known methods of feature detection in endoscopic images ([Yeung Col. 60, lines 40-43] “Any of a variety of edge detection or pattern detection algorithms known to those of skill in the art may be utilized in identifying contours…” [Yeung Col. 36, lines 19-24] “Examples of image processing algorithms that may be suitable for use in the disclosed methods include, but are not limited to, Laplacian of Gaussian (LoG) methods, Canny edge detection methods, Canny-Deriche edge detection methods…”). Also, as discussed in the rejection to claim 2, Yeung provides motivation for determining the centerline of the lumen.
Regarding claim 5, Deng teaches wherein when the local attentive region proposal module generates the object-detecting output (Fig. 5 shows the channel and spatial attention outputs of the model. Additionally, Fig. 5 shows the object outputs indicating the identified landmarks and heading direction for the endoscope.), but Deng fails to teach wherein the direction- detecting program uses the symmetry line of the symmetry-detecting program as the guiding direction.
However, Yeung teaches the direction- detecting program uses the symmetry line of the symmetry-detecting program as the guiding direction ([Col. 57, line 61 – Col. 58, line 4] “a) providing a first image data stream, wherein the first image data stream comprises image data relating to a position of a distal end of the robotic endoscope relative to the center of the lumen, relative to a wall of the lumen, relative to an obstacle, or any combination thereof; b) generating a steering control output signal based on an analysis of the first image data stream using a machine learning architecture to identify a steering direction for movement towards the center of the lumen, wherein the steering control output signal adapts to changes in the data of the first image data stream in real time;”).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Deng’s invention by considering the centerline of the lumen when determining a guiding direction. This modification allows for steering to be guided towards the centerline of the lumen which would minimize the contact of the endoscope to the walls of the lumen, and steering the endoscope towards the lumen centerline is well-known in the art (Yeung provides many references teaching automatically steering an endoscope towards the centerline of the lumen. [Col. 1, lines 26-39] “For example, by exploiting soft deformable structures that are capable of moving effectively through a complex environment like the inside the colon, one can significantly reduce pain and patient discomfort while minimizing the risk of colonic perforation… A variety of image processing-based approaches to identifying the center of the colon lumen and determining a navigation direction for an advancing colonoscope have been described in the literature…” See the background section of Yeung.).
Claims 9 is rejected under 35 U.S.C. 103 as being unpatentable over Deng in view of Yang, and further in view of Mercazini et al. (WO 2024/239125 A1), hereafter Mercanzini.
Regarding claim 9, Deng and Yang fail to teach wherein the specific organ comprises a uvula, epiglottis, arytenoid cartilage, or vocal cord.
However, Mercanzini teaches the multi-task real-time intubation assistant system according to claim 1, wherein the specific organ comprises a uvula, epiglottis, arytenoid cartilage, or vocal cord ([0036] “Important features that will guide the machine vision predictions are labeled, the epiglottis 110, the uvula 120, the pharyngal wall 130, and the cricoid cartilage 140.”).
Deng, Yang, and Mercanzini are analogous to the claimed invention, because all teach methods of image-guided intubation. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention before the effective filing date of the claimed invention to modify Deng’s invention by considering the uvula, epiglottis, arytenoid cartilage, or vocal cord as targets. This modification would further improve Deng’s invention by providing additional targets for guiding the intubation ([Mercanzini 0039] “In particular, an important number of features are present that provide a rich set of predictive points for machine vision algorithms, such as the tongue 160, epiglottis 110, trachea 100. With further detail around the epiglottis 110, the tubercle of the epiglottis 112, the vallecula 113, the median glossoepiglottic fold 114, the lateral glossoepiglottic fold 116, and the aryepiglottic fold 118 are visualized.”).
Allowable Subject Matter
Claims 6 and 7 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Regarding claim 6, the prior art found, such as Deng and Mercanzini, utilizes the object-detecting output for guiding the intubation tube. Other sources utilizing symmetry and edge detection, such as Yeung and Vunjak-Novakovic, utilize the object-detecting output and groove detection together or separately. For example, see the rejection applied to claim 5; the detected features and the centerline of the lumen are both used to determine steering directions for the endoscope along the centerline. However, these references do not mention a conditional statement where the edge detection algorithm is used when objects are detected and the symmetry line algorithm is used when no object is detected.
Regarding claim 7, Mercanzini teaches wherein the preset organ is a uvula ([0036] “Important features that will guide the machine vision predictions are labeled, the epiglottis 110, the uvula 120, the pharyngal wall 130, and the cricoid cartilage 140.”), but claim 7 contains allowable subject matter due to its dependence on claim 6. However, claim 7 is objected to since both claims 6 and 7 are dependent on a rejected base claim.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Prisco et al. (US 8,801,601 B2) teaches systems and methods for guiding an endoscope based on identified anatomical landmarks.
Deng et al. (CN 117079012 A) teaches an artificial intelligence model for recognizing key structures in the trachea and a reconstruction method. The model provides a decision-making system for guiding tracheal intubation and improves the accuracy and safety of the intubation process.
Jiang et al. (US 2022/0265360 A1) teaches a method and system for a tracheal positioning method and device based on deep learning. The method includes using a YOLOv3 network for extracting features from endoscopic images and utilizing sensors for detecting carbon dioxide concentration differences.
Chauhan et al. (US 2025/0185881 A1) teaches a method and system for automatically performing endotracheal intubation. The method involves imaging the airway and predicting an intended path for the insertion of the tube based on the recognized anatomical structure.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ERIC JAMES SHOEMAKER whose telephone number is (571)272-6605. The examiner can normally be reached Monday through Friday from 8am to 5pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner' s supervisor, JENNIFER MEHMOOD, can be reached at (571)272-2976. The fax phone number for the organization where this application or proceeding is assigned is (571)273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Eric Shoemaker/
Patent Examiner
/JENNIFER MEHMOOD/Supervisory Patent Examiner, Art Unit 2664