DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Regarding 35 U.S.C. 112
Examiner notes that new 112(b) rejections are necessitated by amendment. Examiner notes that the previously set forth 112(b) rejections are withdrawn or updated in view of the amendments to the claims (See below for more detail).
Regarding 35 U.S.C. 101
Applicant's arguments filed 07/29/2026 have been fully considered but they are not persuasive.
For example, applicant argues “the claimed device applies a specific image-processing technique in which a trained model performs segmentation to classify a region in a nasal-cavity medical image as an upper route insertion region and/or a lower route insertion region, and route information is derived from the classified insertion region. This is not merely a desired result or a subjective evaluation of the image. Such trained-model segmentation and classification, followed by calculation of route information from the classified insertion region, cannot practically be performed in the mind” (REMARKS pg. 15). Examiner respectfully disagrees in that the segmentation and calculation of route information are practically performed in the mind and mere use of a trained model which is recited with high generality amounts to merely application of the judicial exception with a generic computer.
Applicant further argues “the route information is displayed on a guide image to support movement of the distal end part toward an observation position. Claim 1 therefore applies the segmentation and classification result to support a concrete physical operation of the endoscope, thereby integrating any alleged abstract idea into a practical application”. Examiner respectfully disagrees in that the display of the route information (which is merely one of an area, a width, or a coordinate information of a center of a region) is recited with such high generality that it amounts to merely generic display of results/data, without any specific application which results in concrete physical operation of an endoscope other than an intended use that would assist an operator via visualization.
Applicants arguments against the 101 rejection are therefore not found persuasive and the 101 rejection is maintained/updated in view of the amendments to the claims.
Regarding prior art
Applicant’s arguments with respect to claims 1 and 28 have been considered but are moot in view of the new grounds of rejection necessitated by amendment. Specifically it is noted that new teachings are relied upon to teach the features regarding the output of insertion region information and calculation of insertion route information as amended.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-14, 16, 18-26 and 28 are rejected under 35 U.S.C. 101 because the claimed invention is directed to a judicial exception in the form of an abstract idea without significantly more.
In a test for patent subject matter eligibility, the claims pass Step 1 (see 2019 Revised Patent Subject Matter Eligibility), as they are related to a process, machine, manufacture, or composition of matter.
When assessed under Step2A, Prong I, Independent claims 1 and 28 are found to recite a judicial exception (i.e. abstract idea). In this instance, claims 1 and 28 recite the limitations “acquire a medical image”, “output observation target identification information indicating an observation support target part included in the medical image or indicating that the medical image is out of an observation support target, by inputting the medical image to an observation target identification algorithm”, “select, from a plurality of observation support algorithm, one specific observation support algorithm based on the observation target identification information, wherein the plurality of observation support algorithsm respectively correspond to different observation support target parts, and the specific observation support algorithm is selected according to the observation support target part indicated by the observation target identification information”, “output observation support information by inputting the medical image to the specific observation support algorithm, wherein the observation support information comprises endoscope operation information indicating an operation method of an endoscope for prompting a user to move a distal end part of the endoscope toward an observation position corresponding to the observation support target part”, “perform a control of notifying of the observation support information”, “output insertion region information indicating at least one of an upper route insertion region suitable for insertion of the endoscope and a lower route insertion nregion suitable for insertion of the endoscope”, “performs segmentation on the medical image to classify and output at least one of the upper route insertion region and the lower route insertion region”, calculate, based on the insertion region information the at least one of the upper route insertion region and the lower route insertion region classified by the segmentation”, insertion route informationfor guiding the distal end part of the endoscope in the nasal cavity, the insertion route information comprising an area, a width, and/or coordinate information of a center position of at least one of the upper route insertion region and the lower route insertion region”. The cited limitation(s), under their broadest reasonable interpretation, encompass a mental process (i.e. abstract idea) of acquiring, outputting, selecting, calculating, and notifying which can be performed in the mind or by a human using a pen and a paper (e.g. observation, evaluation, judgment, opinion). In other words, a person could reasonably acquire a medical image (via observation/evaluation), “output observation target identification information (via thought), select one specific algorithm (via evaluation), output observation support information (via thought), output insertion region information (via thought), perform segmentation (via evaluation), calculate insertion route information (via observation/evaluation) and perform a control of notifying (via thought). Examiner notes that with the exception of generic computer-implemented steps (e.g. one or more processors recited in claims 1 and 28), there is nothing in the claims that preclude the limitation from being performed by a human, mentally or with pen and paper, thus the cited limitation(s) recites a judicial exception (MPEP 2106.04(a)) and the claim must be reviewed under Step 2A, Prong II to determine patent eligibility.
Step 2A, Prong II determines whether any claim recites an additional element that integrates the judicial exception into a practical application. Independent claims recites the following additional element(s):
Acquire a medical image (alternatively)
Inputting the medical image to an observation target identification algorithm (alternatively)
Inputting the medical image to the specific observation support algorithm (alternatively)
The operation support algorithm is a trained model
Perform a control of displaying the observation support information comprising the insertion route information on a guide image
A light source device (claim 28)
An endoscope (claim 28)
The additional element(s) in the cited independent claim(s) are not found to integrate the judicial exception into a practical application. In this case, acquiring a medical image is alternatively interpreted as an additional element amounting to mere data gathering and inputting the medical image into an observation target identification algorithm and a specific observation support algorithm which is a trained model is alternatively interpreted as an additional element recited with such high generality that it amounts to merely inputting data to a generic computer/algorithm. Performing a control of displaying the observation support information comprising the insertion route information on a guide image amounts to merely insignificant extra-solution activity of displaying data/results. The light source and the endoscope are considered merely generic components of an endoscope system. These elements are seen as adding insignificant extra-solution activity to the judicial exception. They do no more than link the judicial exception to a particular technological environment or field of use. Therefore, under step 2A Prong II the Judicial exception is not integrated into a practical application by additional elements of independent claims 1 and 28 and the claims must be reviewed under Step 2B to determine patent eligibility.
Step 2B determines where a claim amounts to significantly more.
The additional element listed above do not amount to significantly more than the judicial exception for the same reasons listed above. Additionally there is no improvement in the functioning of the computer or technological field, and there is no transformation of subject matter into a different state. Therefore, under Step 2B in a test for patent subject matter eligibility, the judicial exception of the independent claim(s) do not amount to significantly more and the independent claim(s) remain patent ineligible.
Dependent claims 2-26 further limit the abstract idea of independent claim 1. When analyzed as a whole, these claims are held to be patent ineligible under 35 U.S.C. 101 because the additional recited limitations fail to establish that the claims are not directed towards an abstract idea and do not sufficiently integrate the subject matter into a practical application or recite elements which constitute significantly more than the abstract ideas identified. The dependent claims are directed toward additional elements which encompass abstract ideas
In this instance, dependent claims recite the following limitations:
Output the endoscope operation information or the observation support stop information, by inputting the medical image to the operation support algorithm (claim 8)
output nasopharyngeal position information indicating that the position of the endoscope is in an appropriate position or in an inappropriate direction position that is an inappropriate right position, an inappropriate left position, an inappropriate upper position, an inappropriate lower position, an inappropriate back position, or an inappropriate front position, by inputting the medical image to the operation support algorithm (claim 18)
output the endoscope operation information based on the nasopharyngeal position information (claim 18)
output oropharyngeal region information indicating a glottis region and/or an epiglottis region included in the medical image, by inputting the medical image to the operation support algorithm (claim 20)
calculate oropharyngeal region arithmetic information that is an area, a width, and/or coordinate information of a center position of the glottis region and the epiglottis region based on the oropharyngeal region information (claim 20)
output the endoscope operation information using the oropharyngeal region arithmetic information (claim 20)
output hypopharyngeal region information indicating a glottis region and/or a vocal fold region included in the medical image, by inputting the medical image to the operation support algorithm (claim 22)
calculate hypopharyngeal region arithmetic information that is an area, a width, and/or coordinate information of a center position of the glottis region, and that is a length of the vocal fold region based on the hypopharyngeal region information (claim 22)
output the endoscope operation information using the hypopharyngeal region arithmetic information (claim 22)
output the endoscope operation information for providing an instruction to pull out the endoscope by inputting the medical image to the operation support algorithm (claim 24)
output the observation support stop information, by inputting the medical image to the operation support algorithm (claim 25)
output that the medical image is out of the observation support target as the observation target identification information (claim 26)
The cited limitation(s), under their broadest reasonable interpretation, encompass mental processes (i.e. abstract idea) which can be performed in the mind or by a human using a pen and a paper (e.g. observation, evaluation, judgment, opinion). In other words, a human could reasonably output information via thought and perform calculations via observation/evaluation. Examiner notes that with the exception of generic computer-implemented steps (e.g. the one or more processors), there is nothing in the claims that preclude the limitation from being performed by a human, mentally or with pen and paper, thus the claimed limitation is considered to be directed towards a judicial exception (MPEP 2106.04(a)).
Under Step 2A, Prong II for dependent claims 2-26, present additional elements which only further narrow the judicial exceptions (e.g. claim 2 merely narrows the observation support target part, claim 3 merely narrows the notification to now be considered an additional element which amounts to merely insignificant extra-solution activity of displaying results/outputting results via voice, claim 3 further narrows the observation support information, claim 5 further narrows the endoscope operation information, claim 6 further recites displaying an endoscope operation support diagram which amounts to merely insignificant extra-solution activity of displaying results, claim 7 recites switching a display of the endoscope operation information with such high generality that it amounts to merely insignificant post-solution activity of displaying different results of the information, claims 9 and 12 which recite merely insignificant extra-solution activity of inputting data, claim 10 which further narrows the operation support algorithm such that it is a trained model which is recited with such high generality that it amounts to merely a generic computer, claim 11 which merely further narrows the specific observation support algorithm and recites trained models which are considered merely generic computer, claim 13 which further narrows the observation target identification algorithm such that it is a trained model that has been trained… such a recitation amounts to merely a generic computer, claim 14 which recites merely insignificant extra-solution activity of switching between providing the notification and not providing the notification, claim 16 which recites control of displaying insertion region by changing a display mode thereof based on the insertion route information which amounts to insignificant extra-solution activity of changing display data without any specificity as to how the display mode is changed or how it is based on the insertion route information, claims 19, 21, and 23, which further narrows the operation support algorithm/position determination module to be a trained model which is trained… amounting to merely use of a generic computer) and provide no additional element which are found to integrate the judicial exception into a practical application.
These dependent claims include no additional claims that are sufficient to amount to significantly more than the judicial exception. Additionally, there is no improvement in the functioning of the computer or technological field, and there is no transformation of subject matter into a different state. As discussed above with respect to integration of the abstract idea into a practical application, the additional claims do not provide any additional elements that would amount to significantly more than the judicial exception. Under Step 2B, these claims are not patent eligible.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-14, 16, 18-26 and 28 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 1 recites the limitation “not an observation support target”. It is unclear if the observation support target is the same as the observation support target part or if this is a different element. If they are different, it is unclear what an observation support target encompasses and what the difference between the observation support target part and the observation support target is. In other words, it is unclear if the observation support target part can be included in the medical image and at the same time that the medical image is not an observation support target due to them being different or if the claim is intending to define that the observation identification can only indicate one or the other. Furthermore, it is unclear what is meant by indicating that the medical image is not an observation support target as an observation support target would appear to be an anatomical target and not an image itself. For examination purposes, it has been interpreted that the observation support target may be the same or different than the observation support target part.
Claim 1 recites the limitation “the specific observation support algorithm is selected according to the observation support target part indicated by the observation target identification information”. The limitation is unclear as the limitation regarding the observation target identification information lists indicating an observation support target part included in the medical image as one of two options (i.e. output observation target identification indicating an observation support target part included in the medical image or indicating that the medical image is not an observation support target. In other words, it is unclear how the specific observation support algorithm function is selected in an instance where the observation target identification information indicates that the medical image is not an observation support target or if the claim is attempting to further define the observation target identification information to indicate the observation support target part. For examination purposes, it has been interpreted that the observation target identification information indicates an observation support target part, however, clarification is required.
Claim 1 recites the limitation “output insertion region information indicating at least one of an upper route insertion region suitable for insertion of the endoscope and a lower route insertion region suitable for insertion of the endoscope…. By inputting the medical image to the operation support algorithm of the specific observation support algorithm, wherein the operation support algorithm is a trained model that performs segmentation on the medical image to classify and output at least one of the upper route insertion region and the lower route insertion region”. The limitation is unclear as to whether the insertion region information is the same as the segmentation, classification or output of the trained model, is the same as the observation support information or endoscope operation information, or if these are different. In other words, it would appear that any of the segmentation, classification, or output of the upper route insertion region and lower route insertion region would necessarily indicate at least one of the upper route insertion region and lower route insertion region and it would further appear that insertion region information that indicates at least one of an upper route insertion region and a lower route insertion region would indicate an operation method of an endoscope for prompting a user to move a distal end part of the endoscope as required of the endoscope operation information. Furthermore, both the observation support information and the insertion region information are recited as being output by inputting the medical image to the operation support algorithm of the specific observation support algorithm, however, the claim does not clearly distinguish nor provide a relationship between each of the cited features to make clear whether there is a difference or relationship amongst them. For examination purposes, it has been interpreted that the insertion region information may be the same as the segmentation, classification, or output of the operation support algorithm, the same as the observation support information or endoscope operation information, or each of the cited elements may be different, however, clarification is required.
Claim 1 recites the limitation “calculate…insertion route information for guiding the distal end part of the endoscope in the nasal cavity”. The limitation is unclear as to whether the insertion route information is the same as or different from the observation support information and/or the endoscope operation information. Specifically, examiner notes that the endoscope operation information is recited such that it indicates an operation method of an endoscope for prompting a user to move a distal end part of the endoscope toward an observation position, such insertion route information for guiding the distal end part of the endoscope would appear to read on such endoscope operation information and thus on the observation support information, however, no such distinction nor relationship is defined between the two. For examination purposes, it has been interpreted that they may be the same or different, however, clarification is required.
Claim 1 recites the limitation “perform a control of displaying the observation support information comprising the insertion route information on a guide image”. The limitation is unclear as to whether the control of displaying the observation support information comprising the insertion route information on a guide image is the same as the control of notifying of the observation support information or if these are different notifications/displays. For examination purposes, it has been interpreted that they may be the same or different, however, clarification is required.
Claim 7 recites the limitation “wherein the observation support information is the endoscope position determination information”. Examiner notes that claim 1 explicitly discloses that the output observation support information comprises endoscope operation information and claim 4 recites that the observation support information further comprises endoscope position determination information indicating whether a position of an endoscope is appropriate or inappropriate. The limitation suggests that the observation support information comprises both endoscope operation information and endoscope position determination information. It is therefore unclear how the observation support information is the endoscope position determination information in a case where the endoscope operation information is something different from the position determination information or if the claim is intending to define the operation information and the position determination information to be the same. It is further unclear if the displaying of the endoscope operation information is the same as notifying or if these are different aspects. For examination purposes, it has been interpreted that the observation support information at least includes the endoscope operation information and the endoscope position determination information and that notifying includes displaying the endoscope operation information, however, clarification is required.
Claim 8 recites the limitation “wherein the specific observation support algorithm includes an operation support algorithm”. The limitation is unclear as to whether the operation support algorithm recited is the same as the operation support algorithm included in the specific observation support algorithm as recited in claim 1 or if this is a different observation support algorithm included in the specific observation support algorithm.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-4, 7-14, 18-19, 24, and 28 are rejected under 35 U.S.C. 103 as being unpatentable over WIPO Kamon (WO 2020162275 A1), hereinafter Kamon in view of Tata et al. (US 2023013600 A1), hereinafter Tata.
Regarding claim 1,
Kamon discloses an image processing device comprising:
one or more processors (at least fig. 2 (200) and corresponding disclosure in at least pg. 2 second to last full paragraph and/or at least fig. 12 (204A) and corresponding disclosure in at least pg. 12) configured to:
acquire a medical image (at least fig. 16 (S110) and corresponding disclosure in at least pg. 9 second to last full paragraph which discloses medical image acquisition unit 220 acquires an endoscopic image (medical image)));
output observation target identification information indicating an observation support target part included in the medical image or indicating that the medical image is out of an observation support target, by inputting the medical image to an observation target identification algorithm (at least fig. 16 (S120) and corresponding disclosure in at least pg. 9 last paragraph which discloses the part information acquisition part 222 acquires part information indicating the part in the living body where the endoscopic image was captured and can acquire site information by analyzing the endoscopic image and further discloses when analyzing an endoscopic image, the part information acquisition unit 222 (part information acquisition unit) (i.e. an observation target identification algorithm) can perform analysis using a feature amount such as the color of the subject. A trained model (CNN, SVM, etc.) for analysis may be used. Examiner notes that such output would indicate an observation support target part is included in the medical image and indicate that a medical image is out of an observation support target (i.e. in a case where the observation support target is a stomach and the site information is the esophagus, it is indicates that the medical image is out of the stomach and vice versa);
select, from a plurality of observation support algorithms, one specific observation support algorithm based on the observation target identification information (at least fig. 16 (S130) and corresponding disclosure in at least pg. 13 last full paragraph which discloses the section unit 226 selects a recognizer (i.e. one specific observation support algorithm) corresponding to the part indicated by the part information (i.e. based on the observation target identification information) from the plurality of recognizers), wherein the plurality of observation support algorithms respectively correspond to different observation support target parts, and the specific observation support algorithm is selected according to the observation support target part indicated by the observation target identification information (pg. 12 which discloses the recognizer selected from the plurality of recognizers according to the part indicated by the part information recognizes the medical image.)
output observation support information by inputting the medical image to the specific observation support algorithm (at least fig. 16 (S140) and corresponding disclosure in at least pg. 10 first full paragraph), wherein the observation support information comprises endoscope operation information indicating an operation method of an endoscope (Examiner notes that any of the recognition results would indicate an operation method of the endoscope in its broadest reasonable interpretation. In other words, output of such information would indicate to a user of a method (e.g. imaging, position, location, etc.) of the endoscope) for prompting a user to move a distal end part of the endoscope toward an observation position corresponding to the observation support target part (Examiner notes that “for prompting a user to move” is considered intended use, where examiner notes that a person having ordinary skill in the art would have recognized that the recognition results of pg. 10 could be used for prompting a user to move a distal end part of the endoscope accordingly)
and perform a control of notifying of the observation support information (at least fig. 16 (S150) and corresponding disclosure in at least pg. 10 second full paragraph which discloses the display control unit causes the monitor to display the endoscopic image and the recognition result on the monitor),
wherein the specific observation support algorithm includes an observation support algorithm,
wherein in a case in which the observation support target part output by the observation target identification algorithm is a nasal cavity, the one or more processors are configured to:
output information by inputting the medical image to the operation support algorithm of the specific observation support algorithm, wherein the operation support algorithm is a trained model that performs segmentation on the medical image (pg. 8 which discloses when performing segmentation, the output layer 562C grasps the position of the attention area shown in the image at the pixel level by the feature map obtained from the intermediate layer 562B and the target region o interest can be segmented and then measured by the image processing unit and pg. 10 which discloses recognizer 224 grasps the position of the attention area shown in the image by the "feature map" at the pixel level (that is, the attention area is determined for each pixel of the endoscopic image. Whether or not it belongs) can be output and the detection result can be output. Examples of the region of interest (region of interest) to be detected include polyps, cancer, large intestine diverticulum, inflammation, treatment scars (EMR: Endoscopic Mucosal Resection), ESD scars (ESD: Endoscopic Submucosal Dissection), clip locations, etc.) Examples include bleeding points, perforations, and vascular atypia) to classify and output at least one of the upper route insertion region and the lower route insertion region (examiner notes that the limitation “to classify an output” is directed towards intended use where it is noted that the segmentation disclosed in at least pg. 8 and 10 would allow for classifying and outputting at least one of an upper route insertion region and a lower route insertion region in its BRI. See also pg 13-14 in which the detector has a plurality of detectors respectively corresponding to different positions of the same organ (such as esophageal detector 240 has a first detector 240A and a second detector 240 B for upper and lower esophagus, respectively. Examiner thus notes that in a case where observation support target part is the nasal cavity, that the detectors would function accordingly).
Perform a control of displaying the observation support information on a guide image (at least fig. 16 (S150) and corresponding disclosure in at least pg. 10 second full paragraph which discloses the display control unit causes the monitor to display the endoscopic image and the recognition result on the monitor. See at least figs. 8-11 depicting a guide image for displaying the observation support information (i.e. recognition result)).
Kamon fails to explicitly teach wherein the one or more processors are configured to:
Output insertion region information output insertion region information indicating at least one of an upper route insertion region suitable for insertion of the endoscope and a lower route insertion region suitable for insertion of the endoscope, the upper route insertion region and the lower route insertion region being included in the medical image, by inputting the medical image to the operation support algorithm of the specific observation support algorithm, wherein the operation support algorithm is a trained model that performs segmentation on the medical image to classify and output at least one of the upper route insertion region and the lower route insertion region; calculate, based on the insertion region information indicating the at least one of the upper route insertion region and the lower route insertion region classified by the segmentation, insertion route information for guiding the distal end part of the endoscope in the nasal cavity, the insertion route information comprising an area, a width, and/or coordinate information of a center position of at least one of the upper route insertion region and the lower route insertion region; and perform a control of displaying the observation support information comprising the insertion route information on a guide image.
Tata, in a similar field of endeavor involving endoscope image analysis, teaches one or more processors configured to acquire a medical image:
Output observation support information by inputting the medical image to a specific observation support algorithm, wherein the observation support algorithm comprises endoscope operation information indicating an operation method of an endoscope for prompting a user to move a distal end part of the endoscope toward an observation position corresponding to an observation support target part ([0042] which discloses in certain cases, the image signal 50 includes multiple features that are identified by the feature identification model 54. For example, the branching of a pathway into at least two possible forward passages can be identified by the feature identification model 54 operating to identify any passages in the image signal 50. Thus, multiple features can be identified in the capture image of the image signal 50. Accordingly, block 76 may include selecting one feature as the steering target. In an embodiment, the feature selection is performed by the steering controller 60 and [0035] which discloses the feature identification model 54 may incorporate artificial intelligence or machine learning algorithms to identify one or more features 56 as generally discussed herein);
Wherein in a case in which the observation support target part is a nasal cavity, the one or more processors are configured to:
Output insertion region information indicating at least one of an upper route insertion region suitable for insertion of the endoscope and a lower route insertion region suitable for insertion of the endoscope, the upper route insertion region and the lower route insertion region being included the medical image, by inputting the medical image to the specific operation support algorithm, wherein the specific operation support algorithm is a trained model that performs segmentation on the medical image to classify and output at least one of the upper route insertion region and the lower route insertion region ([0042] which discloses in certain cases, the image signal 50 includes multiple features that are identified by the feature identification model 54. For example, the branching of a pathway into at least two possible forward passages can be identified by the feature identification model 54 operating to identify any passages in the image signal 50. Thus, multiple features can be identified in the capture image of the image signal 50. Accordingly, block 76 may include selecting one feature as the steering target. In an embodiment, the feature selection is performed by the steering controller 60 and [0035] which discloses the feature identification model 54 may incorporate artificial intelligence or machine learning algorithms to identify one or more features 56 as generally discussed herein. See also [0058] which discloses the feature identification model 54 may use segmentation, object identification, or other techniques to identify multiple candidate objects in the image signal 50);
Calculate, based on the insertion region information indicating the at least one of the upper route insertion region and the lower route insertion region classified by the segmentation insertion route information for guiding the distal end part of the endoscope in the nasal cavity, the insertion region information comprising an area, a width, and/or coordinate information of a center position of at least one of the upper route insertion region and the lower route insertion region ([0041] which discloses once identified, the controller 14, e.g., using the steering controller 60, can select a portion (such as a center, or an approximate center) of the identified passage as a steering target for the endoscope 12 (block 76) and In an embodiment, the passage has an irregularly shaped cross-section, and the approximate center of the passage is a centroid or center of cross-sectional area of the passage); and
Perform a control of displaying the observation support information comprising the insertion route information on a guide image (see at least fig. 6 and [0052] which discloses the center 126, e.g. the steering targe may be marked by an icon on the image display for navigation reference, as shown and discussed further below and [0057] which discloses However, in an embodiment, one or both of the generated bounding box or the center of the bounding box is overlaid or otherwise marked on the displayed image on the screen 24, to inform the user which objects the steering system is considering as candidate objects and selecting as the steering target. The bounding boxes (or other visual indication of a candidate object) may also be displayed to the user in the case where the system identifies multiple candidate objects and pauses for user input, as discussed above.).
It would have been obvious to a person having ordinary skill in the art before the effective filing date to have modified Kamon to include outputting insertion region information and calculating insertion route information, and performing control of displaying the observation support information comprising the insertion route information as taught by Tata in order to provide information for steering the medical device/endoscope to the appropriate target accordingly. Such a modification would provide automated or assisted steering techniques that track a steering target, such that the distal end of the endoscope may be maintained at an orientation towards a center of the desired passageways (Tata ([0031])
Regarding claim 2,
Kamon further teaches wherein the observation support target part is an esophagus (pg. 14 first paragraph which discloses the control unit 228 activates the esophageal detector 240 (the upper esophageal first detector 240A, the lower esophageal second detector 2408) when the part information indicates the esophagus).
Regarding claim 3,
Kamon further discloses wherein the notification of the observation support information is performed by the guide image for displaying the observation support information (at least fig. 16 (S150) and corresponding disclosure in at least pg. 10 second full paragraph which discloses the display control unit causes the monitor to display the endoscopic image and the recognition result on the monitor. See at least figs. 8-11 depicting a guide image for displaying the observation support information (i.e. recognition result)) and/or a voice for notifying of the observation support information (pg. 5 which discloses the voice processing unit 209 output a message (voice) regarding the recognition process and the recognition result from the speaker 209A under the control of the CPU 210 and the image processing unit 204.
Regarding claim 4,
Kamon further discloses wherein the observation support information further comprises endoscope position determination information indicating whether a position of an endoscope is appropriate or inappropriate (pg. 10 first full paragraph which discloses the recognizer 224 grasps the position of the attention area shown in the image by the "feature map" at the pixel level (that is, the attention area is determined for each pixel of the endoscopic image. Whether or not it belongs) can be output and the detection result can be output. Examples of the region of interest (region of interest) to be detected include polyps, cancer, large intestine diverticulum, inflammation, treatment scars (EMR: Endoscopic Mucosal Resection), ESD scars (ESD: Endoscopic Submucosal Dissection), clip locations, etc.). Examples include bleeding points, perforations, and vascular atypia). Examiner notes that any of the recognition results would indicate whether a position of an endoscope is appropriate or inappropriate in its broadest reasonable interpretation. In other words, output of such information would indicate to a user of the system whether the endoscope is appropriate or inappropriate), subject position information for prompting change or configuration of a position of a subject (Examiner notes that “for prompting change” is considered intended use, where examiner notes that a person having ordinary skill in the art would have recognized that the recognition results of pg. 10 could be used for prompting change or configuration of a position of a subject depending on the location of the region of interest/attention area in the image).
Regarding claim 7,
Kamon, as modified, teaches the elements of claim 4 as previously stated. Tata further teaches wherein the observation support information is the endoscope position determination information, wherein the one or more processors are configured to perform a control of switching a display of the endoscope operation information based on the endoscope position determination information (pg. 10 last paragraph to pg. 11 first paragraph which discloses further, the recognition result may be displayed or hidden depending on a predetermined condition (elapsed time or the like) other than the region. Furthermore, the display mode (changing the figure, changing the color and brightness, etc.) may be changed according to the recognition result and/or its certainty. Thus the one or more processors are configured to perform a control of switching a display of information (including recognition result information such as endoscope operation information) based on the recognition result including the endoscope position determination information)
Regarding claim 8,
Kamon further discloses wherein the specific observation support algorithm includes an operation support algorithm (examiner notes that the recognizer is considered an operation support algorithm), and
The one or more processors are configured to output the endoscope operation information, by inputting the medical image to the operation support algorithm (Examiner notes that the recognition result is considered the endoscope operation information as noted above).
Regarding claim 9,
Kamon further discloses wherein the one or more processors are configured to input a latest medical image to the operation support algorithm of the specific observation support algorithm, wherein the latest medical image is acquired at a time point after the medical image input to the observation target identification algorithm (See at least pg. 9 last full paragraph which discloses the medical image acquisition unit 220 sequentially takes images of the inside of a living body, which is the subject, at a predetermined frame rate by the imaging unit (the imaging lens 132, the imaging element 134, the AFE 138, etc.) of the endoscope 100 (medical device). Then, the endoscopic image can be acquired in real time).
Regarding claim 10,
Kamon further teaches wherein the operation support algorithm of the specific observation support algorithm is a trained model that outputs the observation support information including the endoscope operation information and/or the subject position information (pg. 13 second to last full paragraph which discloses the recognizers 224A and 225 can be configured by using a plurality of learned models such as CNN or SVM).
Regarding claim 11,
Kamon further discloses wherein the specific observation support algorithm (at least fig. 15 (225) and corresponding disclosure in at least pg. 13 second full paragraph) further includes a position determination algorithm (at least fig. 14 (240A) and corresponding disclosure in at least pg. 12 last paragraph to pg. 13 first paragraph, see also 241A and pg. 13 first paragraph which discloses the case where two (upper and lower) detectors are provided for each organ),
The position determination algorithm (240A) is a trained model (pg. 13 second to last full paragraph) that outputs the endoscope position determination information in response to an input of the medical image (Examiner notes that the upper esophageal recognizer outputs a recognition result which is considered to be the endoscope position determination information as noted above), and
Regarding claim 12,
Kamon further teaches wherein the one or more processors are configured to input a latest medical image to the operation support algorithm of the specific observation support algorithm, wherein the latest medical image is acquired at a time point after the medical image input to the observation target identification algorithm (See at least pg. 9 last full paragraph which discloses the medical image acquisition unit 220 sequentially takes images of the inside of a living body, which is the subject, at a predetermined frame rate by the imaging unit (the imaging lens 132, the imaging element 134, the AFE 138, etc.) of the endoscope 100 (medical device). Then, the endoscopic image can be acquired in real time).
Regarding claim 13,
Kamon further discloses wherein the observation target identification algorithm is a trained model that has been trained using a learning image including the medical image in which an esophagus is included in an observation target (pg. 10 which discloses when analyzing an endoscopic image, the part information acquisition unit 222 (part information acquisition unit) can perform analysis using a feature amount such as the color of the subject. A trained model (CNN, SVM, etc.) for analysis may be used and pg. 14 first paragraph which discloses when the part information indicates the esophagus. Examiner notes that a person having ordinary skill in the art would have recognized a trained model for analysis of the endoscopic image resulting in a part determination of the esophagus is necessarily trained using a learning image including the medical image in which an esophagus is included in an observation target)
Regarding claim 14,
Kamon further discloses wherein the one or more processors are configured to perform a control of switching between presence and absence of the notification of the observation support information (pg. 10 last paragraph which discloses It should be noted that the recognition result may be displayed or hidden depending on the region by the above-mentioned condition setting. Further, when the recognition result is set to be non-display, a mode in which "recognition is performed but the result is not displayed" is possible. Further, the recognition result may be displayed or hidden depending on a predetermined condition (elapsed time or the like) other than the region).
Regarding claim 16,
Kamon, as modified, teaches the elements of claim 1 as previously stated. Tata, as applied to claim 1 above, further teaches wherein the one or more processors are configured to perform a control of displaying the upper route insertion region or the lower route insertion region on the guide image by changing a display mode thereof based on the insertion route information (see at least fig. 5 10 (258) in which the distal end of the endoscope is steered towards the steering target based on the selected candidate object, thus would perform a control of displaying the upper route insertion region or lower route insertion region depending on which is selected on the guide image by changing a display move thereof (e.g. via steering) based on the insertion route information)
Regarding claim 18,
Kamon further discloses wherein the one or more processor are configured to, in a case in which the observation support target output by the observation target identification algorithm is a nasopharynx:
Output nasopharyngeal position information indicating that the position of the endoscope is in an appropriate position or an inappropriate direction position that is an inappropriate right position, an inappropriate left position, an inappropriate upper position, an inappropriate lower position, an inappropriate back position, or an inappropriate front position, by inputting the medical image to the operation support algorithm (Examiner notes that the recognition result of a lower organ detector (e.g. 240B/241B) is considered to indicate that the position of the endoscope is in an appropriate position or an inappropriate direction. The processor would function in any case including a case in which the observation support target output by the observation target identification algorithm is the nasopharynx in which case such position information is considered nasopharyngeal position information)
Output the endoscope operation information based on the nasopharyngeal position information (S150).
Regarding claim 19,
Kamon further teaches wherein the operation support algorithm is a trained model (pg. 13 second to last full paragraph) that has been trained using a learning image including the medical image associated with the nasopharyngeal position information (pg. 13 second to last full paragraph).
Regarding claim 24,
Kamon further teaches wherein the one or more processors are configured to, in a case in which the observation support target part output by the observation target identification algorithm is an esophagus, output the endoscope operation information for providing an instruction (i.e. the recognition result) to pull out the endoscope by inputting the medical image to the operation support algorithm (Examiner notes that to pull out the endoscope is considered an intended use where the recognition result is considered an instruction which may be used to pull out the endoscope).
Regarding claim 28,
Kamon, as modified, teaches the elements of claim 1 as previously stated. Kamon, as modified, further teaches, an endoscope system (at least fig. 1 (10) and corresponding disclosure in at pg. 2 last full paragraph) comprising:
the image processing device according to claim 1 (see rejection of claim 1 above);
A light source device (at least fig. 1 (300) and corresponding disclosure in at least pg. 4 first full paragraph) that emits illumination light (see pg. 4 first full paragraph which discloses illumination light source and illuminance of the observation light from the light source 310 is controlled); and
An endoscope (at least fig. 1 (100) and corresponding disclosure in at least pg. 2 last full paragraph) that images the medical image (pg. 9 last full paragraph which discloses the medical image acquisition unit 220 (medical image acquisition unit) acquires an endoscopic image (medical image) taken in the living body of the subject (step S110: medical image acquisition step) and The medical image acquisition unit 220 sequentially takes images of the inside of a living body, which is the subject, at a predetermined frame rate by the imaging unit (the imaging lens 132, the imaging element 134, the AFE 138, etc.) of the endoscope 100 (medical device))
Claims 5-6 are rejected under 35 U.S.C. 103 as being unpatentable over Kamon and Taka, as applied to claim 4 above, and further in view of WIPO Takenouchi (WO 2021157487 A1), hereinafter Takenouchi. Examiner notes that citations to Takenouchi are with respect to the translated copy provided herein.
Regarding claim 5,
Kamon, as modified, teaches the elements of claim 4 as previously stated. Kamon, as modified, fails to explicitly teach wherein the endoscope operation information is a moving direction and/or a moving amount of a distal end part of the endoscope, and the moving direction is a right direction, a left direction, an upward direction, a downward direction, a backward direction, a forward direction, a right turn, or a left turn.
Takenouchi, in a similar field of endeavor involving endoscopic image processing, teaches outputting observation support information including endoscope operation information indicating whether a position of an endoscope is appropriate or inappropriate, wherein the endoscope operation information is a moving direction and/or a moving amount of a distal end part of the endoscope (pg. 8 second-third paragraphs which discloses operation support information (first information to fifth information and the direction or amount of movement of the endoscope 100 is first information)
Wherein the moving direction is a right direction, a left direction, an upward direction, a downward direction, a backward direction, a forward direction, a right turn, or a left turn (see at least fig. 11/17 depicting an arrow and pg. 12 which discloses the user moves the endoscope scope 100 forward/ backward, bends, etc. according to the arrow 906 (one aspect of the operation support information))
It would have been obvious to a person having ordinary skill in the art before the effective filing date to have modified Kamon to include endoscope operation information as observation support information as taught by Takenouchi in order to improve determination results of the region of interest (Takenouchi Abstract). Such a modification would provide guidance to a user in order to make it possible to make the region of interest appear in the endoscopic image in a state suitable for estimating the size (Takenouchi pg. 8).
Regarding claim 6,
Kamon, as modified, teaches the elements of claim 8 as previously stated. Takenouchi, as applied to claim 5 above, further teaches wherein one or more processor are configured to perform a control of displaying an endoscope operation support diagram on a guide image as the endoscope operation information.
It would have been obvious to a person having ordinary skill in the art before the effective filing date to have modified Kamon, as currently modified, to include an endoscope operation support diagram as the endoscope operation information as taught by Takenouchi in order to improve determination results of the region of interest (Takenouchi Abstract). Such a modification would provide guidance to a user in order to make it possible to make the region of interest appear in the endoscopic image in a state suitable for estimating the size (Takenouchi pg. 8)
Claims 20-23 are rejected under 35 U.S.C. 103 as being unpatentable over Kamon and Tata, as applied ot claim 11 above, and further in view of Sun et al. (US 20250017460 A1), hereinafter Sun.
Regarding claim 20,
Kamon teaches the elements of claim 11 as previously stated. Kamon fails to explicitly teach wherein the one or more processors are configured to, in a case in which the observation support target part output by the observation target identification algorithm is an oropharynx, and the position determination algorithm to which the medical image is input outputs that the position of the endoscope is inappropriate as the endoscope position determination information:
output oropharyngeal region information indicating a glottis region and an epiglottis region included in the medical image, by inputting the medical image to the operation support algorithm;
calculate oropharyngeal region arithmetic information that is an area, a width, and/or coordinate information of a center position of the glottis region and the epiglottis region based on the oropharyngeal region information; and output the endoscope operation information using the oropharyngeal region arithmetic information.
Sun, in a similar field of endeavor involving endoscopic guidance, teaches wherein one or more processors are configured to output oropharyngeal region information indicating a glottis region included in the medical image, by inputting the medical image to the operation support algorithm ([0031] which discloses one or more anatomical features are identified in the image signal including the glottis and epiglottis);
calculate oropharyngeal region arithmetic information that is an area, a width, and/or coordinate information of a center position of the glottis region based on the oropharyngeal region information ([0050] which discloses the anatomy model can estimate passage size and feature size based on the image signal and extrapolate position s and sizes of other features and the direction of the arrow can be determined using the parameters (e.g. size and relative position estimates) determined based on the image signal that are provided to the anatomy model));
and output the endoscope operation information using the oropharyngeal region arithmetic information (at least fig. 6 (46a) and corresponding disclosure in a least [0043] and/or fig. 7 (46b) and corresponding disclosure in at least [0046]).
It would have been obvious to a person having ordinary skill in the art before the effective filing date to have modified Kamon to include outputting oropharyngeal region information, arithmetic information, and outputting endoscope operation information as taught by Sun in order to provide guidance for orienting the endoscope towards an identified anatomical feature (Sun Abstract), thereby permitting improved visualization of an airway during intubation or other procedures (Sun [0025]). Such a modification would provide for enhanced navigation guidance through airways accordingly.
Examiner notes that in the modified system, the processors would function accordingly in any case including in a case in which the observation support target part output by the observation target identification algorithm is an oropharynx, and the position determination algorithm to which the medical image is input outputs that the position of the endoscope is inappropriate as the endoscope position determination information.
Regarding claim 21,
wherein the position determination algorithm is a trained model that has been trained using a learning image in which the medical image and the endoscope position determination information are associated with each other (pg. 13 second to last full paragraph which discloses the recognizers 224A and 225 can be configured by using a plurality of learned models such as CNN or SVM).
Regarding claim 22,
Kamon teaches the elements of claim 11 as previously stated. Kamon fails to explicitly teach wherein the one or more processors are configured to, in a case in which the observation support target part output by the observation target identification algorithm is a hypopharynx, and the position determination algorithm to which the medical image is input outputs that the position of the endoscope is inappropriate as the endoscope position determination information: output hypopharyngeal region information indicating a glottis region and/or a vocal fold region included in the medical image, by inputting the medical image to the operation support algorithm; calculate hypopharyngeal region arithmetic information that is an area, a width, and/or coordinate information of a center position of the glottis region, and that is a length of the vocal fold region based on the hypopharyngeal region information; and output the endoscope operation information using the hypopharyngeal region arithmetic information.
Sun, in a similar field of endeavor involving endoscopic guidance, teaches wherein one or more processors are configured to output hypopharyngeal region information indicating a glottis region and/or a vocal fold region included in the medical image, by inputting the medical image to the operation support algorithm ([0031] which discloses one or more anatomical features are identified in the image signal including the glottis and vocal cords);
calculate hypopharyngeal region arithmetic information that is an area, a width, and/or coordinate information of a center position of the glottis region, and that is a length of the vocal fold region based on the hypopharyngeal region information ([0050] which discloses the anatomy model can estimate passage size and feature size based on the image signal and extrapolate position s and sizes of other features and the direction of the arrow can be determined using the parameters (e.g. size and relative position estimates) determined based on the image signal that are provided to the anatomy model));
and output the endoscope operation information using the hypopharyngeal region arithmetic information (at least fig. 6 (46a) and corresponding disclosure in a least [0043] and/or fig. 7 (46b) and corresponding disclosure in at least [0046])
It would have been obvious to a person having ordinary skill in the art before the effective filing date to have modified Kamon to include outputting hypopharyngeal region information, arithmetic information, and outputting endoscope operation information as taught by Sun in order to provide guidance for orienting the endoscope towards an identified anatomical feature (Sun Abstract), thereby permitting improved visualization of an airway during intubation or other procedures (Sun [0025]). Such a modification would provide for enhanced navigation guidance through airways accordingly.
Examiner notes that in the modified system, the processors would function accordingly in any case including in a case in which the observation support target part output by the observation target identification algorithm is a hypopharynx, and the position determination algorithm to which the medical image is input outputs that the position of the endoscope is inappropriate as the endoscope position determination information.
Regarding claim 23,
Kamon further teaches wherein the position determination algorithm is a trained model that has been trained using a learning image in which the medical image and the endoscope position determination information are associated with each other (pg. 13 second to last full paragraph which discloses the recognizers 224A and 225 can be configured by using a plurality of learned models such as CNN or SVM).
Claims 25-26 are rejected under 35 U.S.C. 103 as being unpatentable over Kamon and Tata, as applied ot claim 1 above and further in view of Aladahalli et al. (US 20210145411 A1), hereinafter Aladahalli.
Regarding claim 25,
Kamon teaches the elements of claim 1 as previously stated. Kamon fails to explicitly teach wherein the one or more processors are configured to, in a case in which the observation target identification algorithm outputs that the medical image is out of the observation support target as the observation target identification information, output the observation support stop information, by inputting the medical image to the operation support algorithm.
Aladahalli, in a similar field of endeavor involving medical image analysis, teaches wherein one or more processors are configured to, in a case in which an observation target identification algorithm outputs that a medical image is out of an observation support target as observation target identification information, output an observation support stop information indicating that an observation support is stopped, by inputting the medical image to an operation support algorithm (at least fig. 3A (320) and corresponding disclosure in at least [0046] where it is noted that observation support stop information is information that there is turbulents/low image quality and the one or more desired image interpretation protocols are stopped (i.e. not deployed)).
It would have been obvious to a person having ordinary skill in the art before the effective filing date to have modified Kamon to include outputting an observation support stop information as taught by Aladahalli in order to provide guidance to a user that the image quality is low and to hold the probe steady or move the probe to a position where the quality is improved so that the operation support may be continued.
Regarding claim 26,
Kamon, as modified, teaches the elements of claim 25, as previously stated. Kamon further teaches wherein the observation target identification algorithm is a trained model that has been trained to output that the observation support target is in the medical image, in a case in which the medical image including a foreign substance, which is food or saliva, shake, blurriness, or halation is input, output that the medical image is out of the observation support target as the observation target identification information (at least fig. 16 (S120) and corresponding disclosure in at least pg. 9 last paragraph which discloses the part information acquisition part 222 acquires part information indicating the part in the living body where the endoscopic image was captured and can acquire site information by analyzing the endoscopic image and further discloses when analyzing an endoscopic image, the part information acquisition unit 222 (part information acquisition unit) (i.e. an observation target identification algorithm) can perform analysis using a feature amount such as the color of the subject. A trained model (CNN, SVM, etc.) for analysis may be used. Examiner notes that outputting the esophagus as the part information is considered an output that the medical image is out of the observation support target (e.g. stomach) and vice versa and would occur in any instance including an instance in which the medical image includes a foreign substance, which is food or saliva, shake, blurriness, or halation)
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to BROOKE L KLEIN whose telephone number is (571)270-5204. The examiner can normally be reached Mon-Fri 7:30-4.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Anne Kozak can be reached at 5712700552. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/BROOKE LYN KLEIN/Primary Examiner, Art Unit 3797
Prosecution Insights