DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on July 29, 2026 has been entered.
Response to Arguments
Applicant’s amendments and arguments in the Amendment with RCE filed July 29, 2026 (herein “Amendment”) with respect to the rejection of claims 1, 3–10 and 15 under 35 U.S.C. 112(b) have been fully considered and are persuasive. The rejection of claims 1, 3–10 and 15 under 35 U.S.C. 112(b) has been withdrawn.
Applicant's arguments and amendments in the Amendment with respect to the rejection of claims 1, 8 and 15 on the ground of nonstatutory double patenting over claims 2, 10 and 14 of co-pending application no. 18/463,897 (now patented as US Patent No. 12,700,123) in view of Mori, US Patent Application Publication No. US 2021/00566095 A1 (herein “Mori”) have been fully considered but they are only persuasive in part, and therefore, the previous rejection in view of co-pending application no. 18/463,897 and Mori is withdrawn, and a new grounds of rejection is set forth herein in view of the now issued patent No. 12,700,123 in view of Mori and now additionally in view of Soper et al., US Patent Application Publication No. US 2024/0070883. Specifically, Applicant argues on page 11 of the Amendment that Mori does not teach estimating a position of a detection target “such as a lesion” from the position of the landmark, but rather that Mori only teaches estimating a position of a landmark from an artificial object. However, Claims 1, 8 and 15 do not recite “lesion” and besides, Mori was not relied upon for the claimed “calculate estimated position information of the detection target” as this is exactly claimed by the referenced claims of co-pending (now issued) application no. 18/463,897. Therefore, in view of the above, the nonstatutory double patenting rejection discussed here is withdrawn, but a new grounds is set forth to include citations to newly cited Soper et al., US Patent Application Publication No. US 2024/0070883 with rejection rationale updated below to include the new claim limitations in the present application and reflect the claims of 18/463,897 as they have been now patented, and accordingly the double patenting rejection is no longer provisional.
Applicant's arguments and amendments in the Amendment with respect to the rejection of claims 12–14 on the ground of nonstatutory double patenting over claims 11–13 of co-pending application no. 18/463,897 (now patented as US Patent No. 12,700,123) in view of Zhang et al., US Patent Application Publication No. US 2016/0235388 A1 (herein “Zhang”) have been fully considered and are persuasive. The rejection of claims 12–14 on the ground of nonstatutory double patenting over claims 11–13 of co-pending application no. 18/463,897 (now patented as US Patent No. 12,700,123) in view of Zhang is withdrawn.
Applicant's arguments and amendments in the Amendment with respect to the rejections of claims 1 and 15 on the ground of nonstatutory double patenting as being unpatentable over claims 1 and 2, 18 and 19 of co-pending Application No. 18/463,930 (herein ‘930) in view of Kamon, US Patent Application Publication No. US 2020/0008653 A1 (herein “Kamon”) have been fully considered but they are not persuasive. Specifically, Applicant argues on pages 12–13 that Kamon does not disclose “a state-dependent display control for distinguishing actual position information from estimated position information calculated based on a stored relative relationship,” however, claims 1 and 15 also do not recite these limitations. Claims 1 and 15 merely recite “a display mode … is a different from a display mode of the actual position,” and an expected result of “so that” the estimated position information is distinguishable from the actual position information, which is disclosed by Kamon’s changing of displayed markings via changed colors or marking thickness. Therefore, the rejection in view of Kamon is maintained, with updates to the rejection rationale made in view of the changes to claims 1 and 15 of the present application, and amendments made to the claims of co-pending Application no. 18/462,930.
Applicant's arguments and amendments in the Amendment with respect to the rejection under 35 USC 103 of claims 1 and 15 and various claims depending therefrom have been fully considered but they are only persuasive in part; therefore the current rejection in view of the combination of Kiyuna, Wand and Kamon is withdrawn, and a new ground of rejection in reliance upon newly cited Soper et al., US Patent Application Publication No. US 2024/0070883 A1, is set forth herein. Specifically on pages 13–14 of the Amendment, Applicant argues that Kiyuna does not teach the claimed “the position information of the landmark” in that Kiyuna does not teach a coordinate relationship between an attention part and a landmark and that Kiyuna’s position is “not position information with an accuracy that allows the position of a detection target to be specified on a display image,” but rather “a rough position in the body.” However, the claims do not recite “coordinate relationship” or that the claimed position has a certain accuracy, and therefore, to the extent Kiyuna’s position is simply a position within the body, it meets the broadest reasonable interpretation of what is actually claimed.
Applicant next argues on pages 14–15 that Kiyuna does not teach or suggest “associating the position information of the landmark with the actual position information of the detection target” in that the cited ¶67 disclosing that the output unit displays information on the landmark immediately before or immediately after the attention part when it corresponds to the attention part, is “merely a determination of whether the content of a captured image corresponds to a predetermined target.” However, in the process of displaying information on the landmark “before or after” the attention part, then the landmark information is being associated with (either before or after) the position of the attention part. The claims merely recite and require under a broadest reasonable interpretation an “association” not something more specific as Applicant contends such as geometric positional relationship. Therefore, Kiyuna does teach the “associating the position information” limitation.
Applicant’s arguments and amendments set forth on pages 15–16 of the Amendment directed towards how a detection target’s position information is calculated between a first endoscopic image and a further endoscopic image are persuasive and found to not be explicitly taught by Kiyuna. Accordingly, newly cited Soper, US Patent Application Publication No. US 2024/0070883 is relied upon in a new grounds of rejection.
Finally, Applicant’s arguments and amendments set forth on pages 17–18 of the Amendment regarding the rejection of claim 12 as being unpatentable under 35 U.S.C. 103 in view of Kiyuna in view of Wang, have been fully considered and are persuasive, and therefore the rejection is withdrawn. Upon further search and consideration of the amendments in view of the prior art, claim 12, and therefore claims 13–14 which depend therefrom, patentably distinguish over the cited art of record. Accordingly, claims 12–14 are allowed.
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claims 1, 8 and 15 are rejected on the ground of nonstatutory double patenting as being unpatentable over claim 2 which depends from 1 and includes the limitations of claim 1, and claims 9 and 13 of US Patent No. 12,700,123 B2 (herein “USPN 12,700,123”), in view of Mori, US Patent Application Publication No. US 2021/0056695 A1 (herein “Mori”) further in view of Soper et al., US Patent Application Publication No. US 2024/0070883 A1 (herein “Soper”).
Regarding claim 1, the correspondence between the limitations of claim 1 of the present application, and claim 1, of USPN 12,700,123 is as given below, with claim interpretation rationale provided additionally in parenthesis (), and deficiencies of claim 1 of USPN 12,700,123 noted in square brackets [] in the listed limitations for claim 1 of the present application:
Limitation of claim 1 of the present application
Limitations of claim 1 of US Patent No. 12,700,123 B2, and claim 2 (depending from claim 1) where indicated
An endoscope system comprising: one or more processors configured to:
An endoscope system comprising: one or more processors configured to:
acquire an endoscope image;
acquire an endoscope image;
detect a detection target and acquire actual position information of the detection target by performing a first detection process on the endoscope image;
detect a detection target and acquire actual position information of the detection target by performing a detection target detection process on the endoscope image;
in a case where the detection target is detected [perform a landmark setting process of detecting a landmark and acquiring position information of the landmark by performing a second detection process on the endoscope image] on which the actual position information of the detection target has been acquired by the first detection process [associating the position information of the landmark with the actual position information of the detection target] [and storing information representing a relative relationship between the actual position information of the detection target and the position information of the landmark], and perform a target detection display process of displaying the actual position information of the detection target [and the position information of the landmark on a display;]
and perform the following: an actual position display control process of displaying the actual position information of the detection target on a display in an actual position display mode, in a case where the detection target is detected
after the landmark setting process has been performed, in a case where, [in a further endoscope image], the detection target is not detected [and the landmark corresponding to the stored relative relationship is detected] calculate estimated position information of the detection target by a position information estimation process based on the [stored relative relationship and the position information of the landmark detected in the further endoscope image]
calculate estimated position information of the detection target by a position information estimation process based on the endoscope image in a case where the detection target is not detected;
and perform a target non-detection display process of displaying the estimated position information of the detection target and [the position information of the landmark] on the display
and an estimated position display control process of displaying the estimated position information of the detection target … in a case where the detection target is not detected
where a display mode of the estimated position information in the target non-detection display process is different from a display mode of the actual position information in the target detection display process so that the estimated position information is distinguishable from the actual position information.
wherein, in the estimated position display control process, the estimated position display mode is changed according to a reliability degree of the estimated position information, … the estimated position information is displayed on the display in different display modes according to the calculated reliability
From claim 2: the position display color information of the actual position display mode is different from the position display color information of the estimated position display mode.
Claim 1 of USPN 12,700,123 does not explicitly recite “perform a landmark setting process of detecting a landmark and acquiring position information of the landmark by performing a second detection process on the endoscope image associating the position information of the landmark with the actual position information of the detection target and storing information representing a relative relationship between the actual position information of the detection target and the position information of the landmark” or “the position information of the landmark on a display,” “and the landmark corresponding to the stored relative relationship is detected,” “based on the stored relative relationship and the position information of the landmark,” or “in a/the further endoscope image.”
However, Mori teaches “perform a landmark setting process of detecting a landmark and acquiring position information of the landmark by performing a second detection process on the endoscope image,” (Mori ¶56, landmark estimation unit determines boundary (position information) of the insertion portion (landmark) based on analysis of the endoscope image) “associating the position information of the landmark with the actual position information of the detection target” (Mori ¶57, boundary line is determined between the insertion point (landmark) and the object (detection target)), and storing information representing a relative relationship between the actual position information of the detection target and the position information of the landmark” (Mori ¶67, information of the marker concerning the shape to go around the outer edge of an insertion portion (landmark) are stored in advance) “the position information of the landmark on a display,” “and the landmark corresponding to the stored relative relationship is detected,” or “based on the stored relative relationship and the position information of the landmark” (Mori ¶¶58, 59, 67, figs. 6A and 6B, position of the landmark is estimated (detected) with high precision, where the landmark location L is with respect to the location on the image displayed via the stored markers).
Soper teaches in a/the further endoscope image (Soper fig. 3, ¶¶40, 69, 80, preoperative and/or interoperative image data (which can be endoscopic images) are obtained to register an anatomic model of landmarks corresponding to a target lesion or other site of interest relative to the anatomic landmarks, and then navigation during an image guided medical procedure (further endoscopic image) is conducted according to the registered anatomic model).
Therefore, taking the limitations of claim 1, and claim 2 which depends from claim 1 of USPN 12,700,123 and the teachings of Mori cited above together as a whole, it would have been obvious to a person having ordinary skill in the art (herein “PHOSITA”) before the effective filing date of the claimed invention to have modified claims 1 and 2 of USPN 12,700,123 with the landmark detection of Mori at least because doing so would provide position estimation with high precision even when a landmark cannot be directly specified as a measurement point. See Mori ¶ 59.
Further, taking the limitations of claim 1, and claim 2 which depends from claim 1 of USPN 12,700,123 and the teachings of Soper cited above together as a whole, it would have been obvious to a PHOSITA before the effective filing date of the claimed invention to have modified claims 1 and 2 of USPN 12,700,123 with the endoscopic images during an image guided medical procedure which references registered anatomic landmark models as disclosed in Soper at least because doing so would improve registration accuracy and efficiency. See Soper ¶27.
Regarding claim 8, the limitations of claim 8 of the present application correspond with the limitations of claim 9 of USPN 12,700,123, in that claim 9 of USPN 12,700,123 recites “wherein the detection target is at least any of a bleeding portion, a lesion part, …, and claim 8 recites in relevant part “wherein the detection target is at least one of a blood vessel, a bleeding portion, a lesion part, or ….”
Regarding claim 15, the correspondence between the limitations of claim 15 of the present application, and claim 13 of USPN 12,700,123 is as given below, with claim interpretation rationale provided additionally in parenthesis (), and deficiencies of claim 13 of USPN 12,700,123 noted in square brackets [] in the listed limitations for claim 15 of the present application. Note that because this is a method claim, and the amendments place claim limitations in a contingent recitation (i.e. “in a case where …”), then per MPEP 2111.04(II), “[t]he broadest reasonable interpretation of a method (or process) claim having contingent limitations requires only those steps that must be performed and does not include steps that are not required to be performed because the condition(s) precedent are not met.” Accordingly, claim 13 of USPN 12,700,123 at least teaches “in a case where the detection target is detected,” and although claim 13 of USPN 12,700,123 does not explicitly teach “in a case where … the detection target is no longer detected,” as such a limitation is contingent, this step is not required to be performed due to the condition precedent not being met.
Limitation of claim 15 of the present application
Limitation of claim 13 of USPN 12,700,123
A method of operating an endoscope system, the method comprising: following steps, executed by one or more processors, of:
A method of operating an endoscope system, the method comprising: following steps, executed by one or more processors, of:
acquiring an endoscope image;
acquiring an endoscope image;
detecting a detection target and acquiring actual position information of the detection target by performing a first detection process on the endoscope image;
detecting a detection target and acquiring actual position information of the detection target by performing a detection target detection process on the endoscope image;
in a case where the detection target is detected, [detecting a landmark and acquiring position information of the landmark by performing a second detection process on the endoscope image] on which the actual position information of the detection target has been acquired by the first detection process [and performing a landmark setting process of associating the position information of the landmark with the actual position information of the detection target] [and storing information representing a relative relationship between the actual position information of the detection target and the position information of the landmark], and performing a target detection display process of displaying the actual position information of the detection target [and the position information of the landmark on a display;]
performing the following: an actual position display control process of displaying the actual position information of the detection target on a display in an actual position display mode in a case where the detection target is detected
Contingent limitation not required: after the landmark setting process has been performed, in a case where, [in a further endoscopic image], the detection target is not detected and the landmark corresponding to the stored relative relationship is detected, calculating estimated position information of the detection target by a position information estimation process based on the stored relative relationship and the position information of the landmark detected [in the further endoscope image], and performing a target non-detection display process of displaying the estimated position information of the detection target and the position information of the landmark on the display
Contingent limitation not required because the target non-detection display process is not performed due to it being part of the contingent limitation above: wherein a display mode of the estimated position information in the target non-detection display process is different from a display mode of the actual position information in the target detection display process
Claim 13 of USPN 12,700,123 does not explicitly recite “detecting a landmark and acquiring position information of the landmark by performing a second detection process on the endoscope image and performing a landmark setting process of associating the position information of the landmark with the actual position information of the detection target and storing information representing a relative relationship between the actual position information of the detection target and the position information of the landmark” “and the position information of the landmark on a display,” “based on the stored relative relationship and the position information of the landmark,” or “in a/the further endoscope image.”
However, Mori teaches “detecting a landmark and acquiring position information of the landmark by,” (Mori ¶56, landmark estimation unit determines boundary (position information) of the insertion portion (landmark) based on analysis of the endoscope image) “performing a second detection process on the endoscope image and performing a landmark setting process of associating the position information of the landmark with the actual position information of the detection target” (Mori ¶57, boundary line is determined between the insertion point (landmark) and the object (detection target)) “the position information of the landmark on a display,” (Mori ¶¶58, 59, figs. 6A and 6B, position of the landmark is estimated (detected) with high precision, where the landmark location L is with respect to the location on the image displayed), “storing information representing a relative relationship between the actual position information of the detection target and the position information of the landmark” (Mori ¶67, information of the marker concerning the shape to go around the outer edge of an insertion portion (landmark) are stored in advance).
Soper teaches in a/the further endoscope image (Soper fig. 3, ¶¶40, 69, 80, preoperative and/or interoperative image data (which can be endoscopic images) are obtained to register an anatomic model of landmarks corresponding to a target lesion or other site of interest relative to the anatomic landmarks, and then navigation during an image guided medical procedure (further endoscopic image) is conducted according to the registered anatomic model).
Therefore, taking the limitations of claim 13 of USPN 12,700,123 and the teachings of Mori cited above together as a whole, it would have been obvious to a PHOSITA before the effective filing date of the claimed invention to have modified claim 13 of USPN 12,700,123 with the landmark detection of Mori at least because doing so would provide position estimation with high precision even when a landmark cannot be directly specified as a measurement point. See Mori ¶ 59.
Further, taking the limitations of claim 13 of USPN 12,700,123 and the teachings of Soper cited above together as a whole, it would have been obvious to a PHOSITA before the effective filing date of the claimed invention to have modified claim 13 of USPN 12,700,123 with the endoscopic images during an image guided medical procedure which references registered anatomic landmark models as disclosed in Soper at least because doing so would improve registration accuracy and efficiency. See Soper ¶27.
Claims 1 and 15 provisionally rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1 and 2, 18 and 19 of co-pending Application No. 18/463,930 (herein ‘930) in view of Kamon, US Patent Application Publication No. US 2020/0008653 A1 (herein “Kamon”). Although the claims at issue are not identical, they are not patentably distinct from each other because the claims of ‘930 recite the same subject matter, even if differently worded, as in the claims of the present application as set forth in detail in the correspondence tables below.
This is a provisional nonstatutory double patenting rejection because the patentably indistinct claims have not in fact been patented.
Regarding claim 1 of the present application, the correspondence between the limitations of claim 1 of the present application, and claims 1 and 2 of ‘930 is as given below, in consideration of the most recent set of claims in ‘930, from a January 5, 2026 amendment, most recent as of the time of drafting this office action (April 27, 2026), and with claim interpretation rationale provided additionally in parenthesis ():
Limitation of claim 1 of the present application
Limitation of claims 1 and 2 (where designated) of ‘930
An endoscope system comprising: one or more processors configured to:
An endoscope system comprising: one or more processors configured to:
acquire an endoscope image;
acquire a first endoscope image
detect a detection target and acquire actual position information of the detection target by performing a first detection process on the endoscope image;
the first endoscope image including a bleeding point (Examiner note – corresponds to claimed detection target) as a detection target and a plurality of landmarks positioned at different portions around the bleeding point, acquire position information of a position of the bleeding point in the first endoscope image by performing a first detection process on the first endoscope image;
in a case where the detection target is detected, perform a landmark setting process of detecting a landmark and acquiring position information of the landmark by performing a second detection process on the endoscope image on which the actual position information of the detection target has been acquired by the first detection process, associating the position information of the landmark with the actual position information of the detection target,
and storing information representing a relative relationship between the actual position information of the detection target and the position information of the landmark,
acquire position information of positions of the plurality of landmarks in the first endoscope image by performing a second detection process on the first endoscope image in response to acquiring the position information of the bleeding point;
store (Examiner note: corresponds to setting) the position information of the bleeding point (thus detection target being detected) in association with (associating) the position information of the plurality of landmarks in a case where the bleeding point is not obscured in the first endoscope image;
and perform a target detection display process of displaying the actual position information of the detection target and the position information of the landmark on a display;
(from claim 2) the one or more processors are configured to display the position information of the bleeding point, … , and the positions of the plurality of landmarks on the display
after the landmark setting process has been performed, in a case where, in a further endoscope image, the detection target is not detected and the landmark corresponding to the stored relative relationship is detected,
calculate estimated position information of the detection target by a position information estimation process based on the stored relative relationship and the position information of the landmark detected in the further endoscope image, and perform a target non-detection display process of displaying the estimated position information of the detection target and the position information of the landmark on the display
acquire a second endoscope image after the position information of the bleeding point is stored in association with the position information of the plurality of landmarks, the second endoscope image including at least a part of the plurality of landmarks and a blood region at a position at which the bleeding point is obscured (detection target not detected);
determine an estimated position of the bleeding point in the second endoscope image (further endoscope image) based on the position information of the plurality of landmarks stored in association with the position information of the bleeding point; and
display, on a display, an estimated position display indicator indicating the estimated position of the bleeding point at the blood region in the second endoscope image.
wherein a display [mode] of the estimated position information in the target non-detection display process is different from a display [mode] of the actual position information in the target detection display process so that the estimated position information is distinguishable from the actual position information.
From claim 2: wherein the one or more processors are configured to display the position information of the bleeding point, the estimated position of the bleeding point, and positions of the plurality of landmarks on the display to be distinguished from each other.
Claims 1 and 2 of ‘930 does not explicitly teach, where Kamon teaches specifically modal changes, therefore Kamon teaches the claimed “display mode” “different from a display mode” (Kamon ¶¶120 and 125–126, the display mode of a marking, such as marks M1 and M2 which are landmarks and estimated position information in a display, is changed is changed using a biological feature value such as a color or a thickness of a marking).
Therefore, taking the limitations of claims 1 and 2 of ‘930 and the teachings of Kamon cited above together as a whole, it would have been obvious to a PHOSITA before the effective filing date of the claimed invention to have modified claims 1 and 2 of ‘930 with the changed display modes of markers of Kamon at least because doing so would allow for a user to recognize positions of a region of interest and shorten diagnosis time for ROIs. See Kamon ¶ 125.
Regarding claim 15 of the present application, the correspondence between claims 15 of the present application and claims 18 and 19 (where 19 depends from claim 18) of ‘930 is as given below, in consideration of the most recent set of claims in ‘930, and with claim interpretation rationale provided additionally in parenthesis ():
Limitation of claim 15 of the present application
Limitation of claims 18 and 19 of ‘930
A method of operating an endoscope system, the method comprising: following steps, executed by one or more processors, of:
A method of operating an endoscope system including one or more processors, the method comprising:
acquiring an endoscope image;
acquiring a first endoscope image;
detecting a detection target and acquiring actual position information of the detection target by performing a first detection process on the endoscope image;
the first endoscope image including a bleeding point as a detection target and a plurality of landmarks positioned at different portions around the bleeding point,
acquiring position information of a position of the bleeding point in the first endoscope image by performing a first detection process on the first endoscope image;
in a case where the detection target is detected, detecting a landmark and acquiring position information of the landmark by performing a second detection process on the endoscope image on which the actual position information of the detection target has been acquired by the first detection process,
performing a landmark setting process of associating the position information of the landmark with the actual position information of the detection target
and storing information representing a relative relationship between the actual position information of the detection target and the position information of the landmark,
acquire position information of positions of the plurality of landmarks in the first endoscope image by performing a second detection process on the first endoscope image in response to acquiring the position information of the bleeding point;
store (Examiner note: corresponds to setting) the position information of the bleeding point (thus detection target being detected) in association with (associating) the position information of the plurality of landmarks in a case where the bleeding point is not obscured in the first endoscope image;
and performing a target detection display process of displaying the actual position information of the detection target and the position information of the landmark on a display;
From claim 19: displaying the position information of the bleeding point (actual position information), … and the positions of the plurality of landmarks on the display
after the landmark setting process has been performed, in a case where, in a further endoscope image, the detection target is not detected and the landmark corresponding to the stored relative relationship is detected,
calculating estimated position information of the detection target by a position information estimation process based on the stored relative relationship and the position information of the landmark detected in the further endoscope image, and
performing a target non-detection display process of displaying the estimated position information of the detection target and the position information of the landmark on the display
acquire a second endoscope image (further endoscope image) after the position information of the bleeding point is stored in association with the position information of the plurality of landmarks, the second endoscope image including at least a part of the plurality of landmarks and a blood region at a position at which the bleeding point is obscured (detection target not detected);
determine an estimated position of the bleeding point in the second endoscope image (further endoscope image) based on the position information of the plurality of landmarks stored in association with the position information of the bleeding point; and
displaying, on a display, an estimated position display indicator indicating the estimated position of the bleeding point at the blood region in the second endoscope image.
From claim 19: displaying … the estimated position of the bleeding point, and the positions of the plurality of landmarks on the display to be distinguishable from each other.
wherein a display [mode] of the estimated position information in the target non-detection display process is different from a display [mode] of the actual position information in the target detection display process so that the estimated position information is distinguishable from the actual position information.
From claim 19:
displaying … the estimated position of the bleeding point, and the positions of the plurality of landmarks on the display to be distinguishable from each other.
Claims 18 and 19 of ‘930 do not explicitly teach, where Kamon teaches specifically modal changes, therefore Kamon teaches the claimed “display mode” “different from a display mode” (Kamon ¶¶120 and 125–126, the display mode of a marking, such as marks M1 and M2 which are landmarks and estimated position information in a display, is changed is changed using a biological feature value such as a color or a thickness of a marking).
Therefore, taking the limitations of claims 18 and 19 of ‘930 and the teachings of Kamon cited above together as a whole, it would have been obvious to a PHOSITA before the effective filing date of the claimed invention to have modified claims 18 and 19 of ‘930 with the changed display modes of markers of Kamon at least because doing so would allow for a user to recognize positions of a region of interest and shorten diagnosis time for ROIs. See Kamon ¶ 125.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1, 6, and 9–10 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Kiyuna et al., US Patent Application Publication No. US 2022/0148182 A1 (herein “Kiyuna”) further in view of Wang et al., US Patent Application Publication No. US 2020/0030038 A1 (herein “Wang”), further in view of Kamon.
Regarding claims 1 and 15, with substantive differences between the claims noted in curly brackets {}, and with claim 1 as exemplary, Kiyuna teaches {A method of operating – claim 15} an endoscope system {the method – claim 15} comprising (Kiyuna ¶ 25, endoscopic system including display device and inspection device):
{following steps, executed by – claim 15} one or more processors {configured to – claim 1 / of – claim 15}: (Kiyuna ¶¶ 29–31, 59, inspection device 1 including a processor executing a program in a memory to execute the process of the inspection device):
acquire an endoscope image (Kiyuna ¶60, acquisition unit acquires a photographed image);
detect a detection target and acquire actual position information of the detection target by performing a first detection process on the endoscope image (Kiyuna ¶¶ 63, 66, in step 106, YES branch for when the attention part has been designated or detected, then attention part information is generated from the position information generation unit, where the position determination unit 44 determines that the position indicated by the photographed image Ic acquired at step S109 corresponds to the attention part (actual position information of the detection target));
in a case where the detection target is detected {perform a landmark setting process of – claim 1} detecting a landmark and acquiring position information of the landmark by performing a second detection process on the endoscope image on which the actual position information of the detection target has been acquired by the first detection process (Kiyuna ¶¶ 62, 63, when step S106 follows the YES path, this is a case where the attention part (detection target) is detected and position information determined from the same image, and step S105, the landmark position information is determined and acquired in a separate step/second detection process, and the landmark position information is stored in the landmark position information storage unit (setting)), and {performing a landmark setting process of – claim 15} associating the position information of the landmark with the actual position information of the detection target (Kiyuna ¶ 67, the output unit displays information on the landmark immediately before or immediately after the attention part when it corresponds to the attention part), [and storing information representing a relative relationship between the actual position information of the detection target and the position information of the landmark, and perform a target detection display process of displaying the actual position information of the detection target and the position information of the landmark on a display]; and
after the landmark setting process has been performed, in a case where, [in a further endoscope image], the detection target is not detected and the landmark [corresponding to the stored relative relationship] is detected [calculate estimated position information of the detection target by a position information estimation process based on the stored relative relationship and the position information of the landmark detected in the further endoscope image] and perform a target non-detection display process of displaying [the estimated position information of the detection target] and the position information of the landmark on the display (Kiyuna fig. 6, ¶67, when the position indicated by the photographed image Ic acquired at step S109 corresponds to the landmark only (step S110; Yes, and step S106 NO because attention part is not detected), the output unit performs an output prompting confirmation of the photographed image Ic displayed on the display device),
[wherein a display mode of the estimated position information in the target non-detection display process is different from a display mode of the actual position information in the target detection display process so that the estimated position information is distinguishable from the actual position information].
Kiyuna does not explicitly teach where Wang teaches and performing a target detection display process of displaying the actual position information of the detection target and the position information of the landmark on a display (Wang ¶107, figs. 12A, 12B, black pointer added (target detection display process) to displayed image for a detection target, with a dashed line as a landmark), calculate estimated position information of the detection target by a position information estimation process based on the position information of the landmark (Wang ¶165, machine-vision algorithm (position information estimation process) using features observed by the camera (landmark) used to calculate the three-dimensional position of the object (estimated position information of the detection target)), displaying the estimated position information of the detection target, and a display mode of the estimated position information in the target non-detection display process (Wang ¶191, screen device displaying reference/fiducial marker (landmark) 1900 and end of guide probe 2210 shown with a circle (detection target)).
Still further, Kiyuna does not explicitly teach where Kamon teaches wherein a display mode in the target non-detection display process is different from a display mode of the actual position information in the target detection display process so that the estimated position information is distinguishable from the actual position information (Kamon ¶¶120 and 125–126, the display mode of a marking, such as marks M1 and M2 which are landmarks and estimated position information in a display, is changed using a biological feature value such as a color or a thickness of a marking in accordance with a discrimination result, where ¶104 teaches different markings M1 and M2 displayed in different manners (mode) based on the discrimination result, and where ¶¶87–88 teaches this difference in discrimination result is for differences between the ROI (actual position information) of tissue to be treated (target detection) versus the ROI of normal tissue not being treated (target non-detection), thus the tissue to be treated being distinguishable from the tissue not being treated).
Yet still further Kiyuna does not explicitly teach where Soper teaches storing information representing a relative relationship between the actual position information of the detection target and the position information of the landmark (Soper ¶¶60–61, and 69-73, a registration being the final product resulting from determination of anatomic landmarks and respective coordinate points as they are associated with model landmarks of a 3D model (information representing a relative relationship), the registration being available for use in performing an image-guided medical procedure (thus stored)).
Soper also teaches in a further endoscope image (Soper ¶¶73, 88 and 101, after registration, the registration is used for performing an image-guided medical procedure where the 3D model is displayed along with the tracked location of the medical instrument for navigating the medical instrument to a target location, where the medical instrument has an endoscope capturing images (thus the further endoscope image being the one when after registration an image-guided medical procedure is performed using the 3D model)), the landmark corresponding to the stored relative relationship being detected and the stored relative relationship used to determine position information estimation from the landmark as it is detected in the further endoscope image (Soper ¶¶107–109, based on the registration (stored relative relationship used), a virtual depiction of patient anatomy from a viewpoint of a virtual camera on the representation of the medical instrument system within the anatomic model (containing landmarks), including the tracked position, shape, pose, orientation, and/or movement of the medical instrument system within the patient (position information from the landmark)).
Therefore, taking the teachings of Kiyuna and Wang together as a whole, it would have been obvious to a PHOSITA before the effective filing date of the claimed invention to have modified the endoscopy system disclosed in Kiyuna to include the lasers, imaging and displaying specifically cited to above from Wang at least because doing so would enhance the performance of image guidance systems.
Further, taking the teachings of Kiyuna as modified by Wang and Kamon cited above together as a whole, it would have been obvious to a PHOSITA before the effective filing date of the claimed invention to have modified the endoscopy system disclosed in Kiyuna with the changed display modes of markers of Kamon at least because doing so would allow for a user to recognize positions of a region of interest and shorten diagnosis time for ROIs. See Kamon ¶ 125.
Still further, taking the teachings of Kiyuna as modified by Wang and Kamon and Soper cited above together as a whole, it would have been obvious to a PHOSITA before the effective filing date of the claimed invention to have modified the endoscopy system disclosed in Kiyuna with the registration of anatomic and model landmarks from one endoscopic image that are used in a further endoscopic image as disclosed in the above cited portions of Soper at least because doing so would improve registration accuracy and efficiency. See Soper ¶27.
Regarding claim 6, Kiyuna teaches the endoscope and endoscope images (Kiyuna ¶ 41, endoscope capturing images in a time series), but does not explicitly teach where Wang teaches wherein the image includes a first image based on first illumination light and a second image based on second illumination light having a spectrum different from a spectrum of the first illumination light (Wang ¶¶ 76, 100, 102, first and second lasers, each with a different spectrum of visible light, used for imaging, thus any image taken being based on the two lasers, with a first image taken in one position, and a second image taken after rotating 90 degrees), and the one or more processors are configured to detect the detection target and the landmark from the second image (Wang ¶105, orthogonal images (including the second) used for trajectory planning and visualization) and to display the actual position information of the detection target and the position information of the landmark on the first image (Wang ¶107, figs. 12A, 12B, black pointer added to displayed image for a detection target, with a dashed line as a landmark).
Therefore, taking the teachings of Kiyuna and Wang together as a whole, it would have been obvious to a PHOSITA before the effective filing date of the claimed invention to have modified the endoscopy system disclosed in Kiyuna to include the lasers, imaging and displaying specifically cited to above from Wang at least because doing so would enhance the performance of image guidance systems.
Regarding claim 9, Kiyuna teaches wherein the one or more processors are configured to:
detect a new landmark at a position different from a position of the landmark that has already been detected in a case where the endoscope image of a new frame is acquired in a state where a position information estimation process based on the position information of the landmark is continued (Kiyuna fig. 6, ¶¶ 64, 60–62, at step 108, the “NO” path leads back to step 102 where a new photographed image is acquired, and at step S105, another “new” landmark position information is generated and stored);
perform a new landmark setting process of associating estimated position information of the detection target with the new landmark; (Kiyuna ¶ 62, step S105, the landmark position information is determined and acquired in a separate step/second detection process, and the landmark position information is stored in the landmark position information storage unit (setting)), after the new landmark setting process, in a case where the endoscope image of a new frame is acquired and the landmark necessary for the position information estimation process is not recognized, perform the position information estimation process using the new landmark setting process (Kiyuna fig. 6, ¶67, when the position indicated by the photographed image Ic acquired at step S109 corresponds to the landmark only (step S110; YES for attention part only not landmark correspondence, but step S105 has already stored a new landmark), the output unit performs an output prompting confirmation of the photographed image Ic displayed on the display device to enable an inspector to recognize the relative position of the attention part with respect to the new landmark from landmark saving step in S105 ); and
and display the estimated position information of the detection target on the display (Kiyuna ¶ 67, the output unit displays information on the landmark immediately before or immediately after the attention part when it corresponds to the attention part).
Kiyuna does not explicitly teach but Wang teaches and calculate the estimated position information of the detection target (Wang ¶165, machine-vision algorithm using features observed by the camera (landmark) used to calculate the three-dimensional position of the object (detection target)).
Therefore, taking the teachings of Kiyuna and Wang together as a whole, it would have been obvious to a PHOSITA before the effective filing date of the claimed invention to have modified the endoscopy system disclosed in Kiyuna to include the calculation specifically cited to above from Wang at least because doing so would enhance the performance of image guidance systems.
Regarding claim 10, Kiyuna teaches wherein the new landmark is position information of at least any of a mucous membrane pattern, a shape of an organ, or marking by a user operation (Kiyuna ¶26, photographed image IC of the lumen of the subject’s organ (shape of an organ) such as a large bowel which is regarded as a landmark), and the position information of the new landmark is displayed on the display in a display mode different from a display mode of the landmark (Kiyuna fig. 6, ¶67, output unit displays information on the landmark immediately before or after (therefore, when before for the new landmark, this is a different mode then displaying after for the first landmark) the attention part).
Claims 3 and 4 are rejected under 35 U.S.C. 103 as being unpatentable over Kiyuna in view of Wang and Kamon and Soper, as set forth above regarding claim 1 from which claims 3 and 4 depend, and further in view of Hussain et al., "Contribution of Augmented Reality to Minimally Invasive Computer-Assisted Cranial Base Surgery," in IEEE Journal of Biomedical and Health Informatics, vol. 24, no. 7, pp. 2093-2106, July 2020, doi: 10.1109/JBHI.2019.2954003 (herein “Hussain”).
Regarding claim 3, Kiyuna does not teach but Wang teaches in the target non-detection display process, the estimated position information of the detection target and the position information of the landmark are displayed on the screen (Wang ¶191, screen device displaying reference/fiducial marker (landmark) 1900 and end of guide probe 2210 shown with a circle (detection target)).
Further, Kiyuna does not teach but Hussain teaches the screen being a sub screen, and the display has a main screen on which the endoscope image is displayed and a sub screen provided at a position different from a position of the main screen (Hussain page 2100, fig. 4, AR display including main screen of endoscopic view on the left, and on the right a sub screen with target points depicted via augmented reality).
Therefore, taking the teachings of Kiyuna and Wang together as a whole, it would have been obvious to a PHOSITA before the effective filing date of the claimed invention to have modified the endoscopy system disclosed in Kiyuna to include the calculation and displaying specifically cited to above from Wang at least because doing so would enhance the performance of image guidance systems.
Further, taking the teachings of Kiyuna modified by Wang and Hussain together as a whole, it would have been obvious to a PHOSITA before the effective filing date of the claimed invention to have modified the endoscopy system disclosed in Kiyuna to include the displaying specifically cited to above from Hussain at least because doing so would increase safety and cost-effectiveness of machine vision assisted surgery. See Hussain Abstract.
Regarding claim 4, Kiyuna does not teach, but Wang teaches in the target non-detection display process, the estimated position information of the detection target and the position information of the landmark are displayed on the main screen (Wang ¶191, screen device displaying reference/fiducial marker (landmark) 1900 and end of guide probe 2210 shown with a circle (detection target)).
Kiyuna does not teach but Hussain teaches the display has a main screen on which the endoscope image is displayed and a sub screen provided at a position different from a position of the main screen (Hussain page 2100, fig. 4, AR display including main screen of endoscopic view on the left, and on the right a sub screen with target points depicted via augmented reality), and position information of the detection target and the position information of the landmark are displayed in a still image of an endoscope image acquired at a timing at which the detection target is detected, on the sub screen (Hussain page 2094, fig. 2, and page 2100, fig. 4, AR display is overlay on the right side of the display (sub screen) showing structures overlaid on the endoscopic view (still image), thus including the position of the detection target and landmarks, and where page 2103, right column teaches that processing speeds for the AR system can be 40 frames per second, providing the perception of continuous and flicker-free visual output (at a timing which the detection target is detected)).
Therefore, taking the teachings of Kiyuna and Wang together as a whole, it would have been obvious to a PHOSITA before the effective filing date of the claimed invention to have modified the endoscopy system disclosed in Kiyuna to include the calculation and displaying specifically cited to above from Wang at least because doing so would enhance the performance of image guidance systems.
Further, taking the teachings of Kiyuna modified by Wang and Hussain together as a whole, it would have been obvious to a PHOSITA before the effective filing date of the claimed invention to have modified the endoscopy system disclosed in Kiyuna to include the displaying specifically cited to above from Hussain at least because doing so would increase safety and cost-effectiveness of machine vision assisted surgery. See Hussain Abstract.
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over Kiyuna in view of Wang in view of Kumon in view of Soper, as set forth above regarding claim 1, and further in view of Kamiyama et al., US Patent Application Publication No. US 2024/0274272 A1 (herein “Kamiyama”).
Regarding claim 5, Kiyuna does not explicitly teach but Kamiyama teaches wherein, in the landmark setting process, the position information of the landmark detected around the detection target among the landmarks is associated with the actual position information of the detection target (Kamiyama ¶77, determination of a bounding box (landmark around the detection target) from among candidate bounding boxes being the one that enables representation of the position and shape (the actual position information of the detection target) in a more detailed manner).
Therefore, taking the teachings of Kiyuna and Kamiyama together as a whole, it would have been obvious to a PHOSITA before the effective filing date of the claimed invention to have modified the endoscopy system disclosed in Kiyuna to include the bounding box selection best suited to the detection target as specifically cited to above from Kamiyama at least because doing so would provide object detection with high-speed processing and maintaining a real-time characteristic. See Kamiyama ¶59.
Claims 7 and 8 are rejected under 35 U.S.C. 103 as being unpatentable over Kiyuna in view of Wang in view of Kumon in view of Soper
, as set forth above regarding claim 1 from which claims 7 and 8 depend, further in view of Sakamoto et al., US Patent Application Publication No. US 2017/0273707 A1 (herein “Sakamoto”).
Regarding claim 7, Kiyuna teaches wherein the landmark includes (Kiyuna ¶¶ 62, 63, step S105, the landmark position information is determined and acquired), but does not where Sakamoto explicitly teaches “a blood vessel emphasized by chemical fluorescence.” (Sakamoto ¶ 47, fluorescent chemical agent ICG is administered to the vein of the patient so that the fluorescent chemical agent ICG is contained in the blood, and the fluorescence in the fluorescent chemical agent ICG is emitted by infrared excitation light to display and observe blood vessels in the joint 100 and bloodstream information on the display through the arthroscope).
Therefore, taking the teachings of Kiyuna and Sakamoto together as a whole, it would have been obvious to a PHOSITA before the effective filing date of the claimed invention to have modified the endoscopy system disclosed in Kiyuna to include the chemical fluorescence as specifically cited to above from Sakamoto at least because doing so would enhance the visibility and thus accuracy of tissue to operate on. See Sakamoto ¶52.
Regarding claim 8, Kiyuna teaches the detection target is at least one of (Kiyuna ¶¶ 63, 66, in step 106, YES branch for when the attention part has been designated or detected), but does not explicitly teach where Sakamoto teaches “a blood vessel, a bleeding portion, a lesion part, or a specific organ emphasized by chemical fluorescence.” (Sakamoto ¶ 47, fluorescent chemical agent ICG is administered to the vein of the patient so that the fluorescent chemical agent ICG is contained in the blood, and the fluorescence in the fluorescent chemical agent ICG is emitted by infrared excitation light to display and observe blood vessels in the joint 100 and bloodstream information on the display through the arthroscope).
Therefore, taking the teachings of Kiyuna and Sakamoto together as a whole, it would have been obvious to a PHOSITA before the effective filing date of the claimed invention to have modified the endoscopy system disclosed in Kiyuna to include the chemical fluorescence as specifically cited to above from Sakamoto at least because doing so would enhance the visibility and thus accuracy of tissue to operate on. See Sakamoto ¶52.
Allowable Subject Matter
Claim 12, and therefore claims 13–14 which depend therefrom are allowed. The following is an examiner’s statement of reasons for allowance: claims 12–14 are directed towards an endoscope system which performs a landmark setting process and a new landmark setting process that includes associating respective landmarks to detection target estimated position information in respective relative relationships. Further, claim 12 claims respective first and second link lines to the relative relationship and the new relative relationship, the first and second link lines being different types and having different line display modes, and respectively connecting a display position between the detection target estimated position information and the display position of the landmark or the new landmark.
The closest cited art of record includes Kiyuna in view of Zhang as applied in the rejection on pages 34–38 of the Final Action issued 4/30/2026. Zhang provides teachings of link lines in a BSplice curve that connects display positions between a detection target and a landmark, but does not teach having a different second link line of a different type and having a different mode, that connects to a display position of a new landmark. Newly cited reference Shelton et al., US 2022/0028078 A1 discloses a variety of link lines as they relate positionally to landmarks and in different “modes” but also does not teach or suggest different types of link lines with different modes for a first landmark and a new landmark, as specifically claimed. Therefore, none of the cited art of record, whether considered alone or in a combination obvious to a PHOSITA teaches or suggests the limitations of claim 12 and as such, claim 12, and claims 13–14 which depend therefrom are allowed.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MICHELLE M KOETH whose telephone number is (571)272-5908. The examiner can normally be reached Monday-Thursday, 09:00-17:00, Friday 09:00-13:00, EDT/EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Vincent Rudolph can be reached at 571-272-8243. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
MICHELLE M. KOETH
Primary Examiner
Art Unit 2671
/MICHELLE M KOETH/Primary Examiner, Art Unit 2671
Prosecution Insights