Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 2, 11, 12, 14, 16-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Yamaguchi US2017/0046842 and further in view of Engel US2015/0223725.
Yamaguchi discloses for claim 1, “A medical observation apparatus comprising:
circuitry configured to:
generate three-dimensional information (fig 3; 0051-0052 describes two imagers for imaging in 3D and 0052 providing depth information) of a surgical field from a surgical field image (surgery region image 0046, 0049) captured by an imaging device (imaging section 21; fig 1; 0045);
set at least one region-of-interest (important area detected by area position detection section 0077 with remarkable point Q; 0077-0080; fig 6, 7) in the surgical field image (surgery region image 0046, 0049);
track the region-of-interest (important area detected by area position detection section 0077 with remarkable point Q; 0163 describes the area position detecting process and the remarkable point estimating process are executed each time the surgery region image is supplied from the imaging sensor 24, i.e. continual detection/tracking based on image capture) in accordance with changes in the surgical field image captured by the imaging device based on the three-dimensional information by estimating an existence position of the region-of-interest (0078 states obtaining an estimated location coordinate of the remarkable point Q at the surgery region 82 in the three-dimensional space for detection) in a subsequent surgical field image (0077 states the detection, i.e. tracking of the RIO/remarkable point Q based on “different frames”, i.e. subsequent surgical field images) based on the three-dimensional information (0078 states the detection of remarkable point Q at the surgery region 82 is based on the three-dimensional space and the estimated location coordinate of the remarkable point Q in the three-dimensional space is supplied to the zoom image generation section 44);
generate a magnified surgical field image (zoom image QB by zoom image generation section 44; fig 7; 0081, 0089) in which the region-of-interest is magnified at a predetermined magnification (0084: “For example, the operation control section 46 supplies zoom magnification information for designating predetermined zoom magnification to the zoom image generation section 44”); and
output the magnified surgical field image to a display (45, 46; fig 1, 7; 0081-0083)”.
Yamaguchi does not disclose:
“estimate a self-position and orientation of the imaging device within the surgical field based on the three-dimensional information;” and by estimating an existence position of the region-of-interest in a subsequent surgical field image based the three-dimensional information “and the estimated self-position and orientation of the imaging device”.
Engle teaches in the same field of endeavor, a tracking system involving a mobile maneuverable device designed for determining and measuring a position in space and/or an orientation in space of bodies, without contact, i.e. SLAM, simultaneous localization and mapping (0008), specifically of an image data capture device, 0011 “The pose (position and/or orientation) of an image data capture device is taken into account in a three-point algorithm”. Further details of the SLAM algorithm described at 0098: “The simultaneously applied SLAM method determines the movement of the camera relative to the environment, based on the image data, and compiles a map 470, which in this case is a 3D map in the form of a series of points, or in the form of a surface model, by means of the images, from different positions and orientations; the latter method, taking into account various different positions and orientations, is also called a 6D method, and particularly a 6D SLAM method. If a map of the application region 301 is already available, the map is either updated or used for navigation on this map 470, 480”. It would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to incorporate the modification of Engle into the invention of Yamaguchi in order to configure the medical observation apparatus e.g. as claimed because it provides “a significant improvement in precision, particularly into the sub-millimeter region” 0048.
Yamaguchi discloses for claim 2, “The medical observation apparatus according to claim 1, wherein the circuitry is configured to output the surgical field image and the magnified surgical field image to the display (fig 7)”.
Yamaguchi discloses for claim 11, “The medical observation apparatus according to claim 1, wherein the circuitry is configured to designate a distance range containing the region-of-interest, and set the region-of-interest within the designated distance range (0256 describes “the area of interest is determined based on a three-dimensional position of a surgical instrument in the image” and also including “an intersection point of the region and an extension line obtained by extending a line segment corresponding to the three-dimensional position of the surgical instrument” 0258-0260)”.
Yamaguchi discloses for claim 12, “The medical observation apparatus according to claim 1, wherein the circuitry is configured to generate the magnified surgical field image at a magnification corresponding to a distance to the region-of-interest (0092 describes the zoom magnification “may be determined such that the tip end portion of at least one forceps 81 and the remarkable point Q estimated from a posture of the forceps 81 are included in the surgery region image 110”, i.e. dependent on the distance of the imager to the region-of-interest in order to keep the required elements in the field of view)”.
Yamaguchi discloses for claim 14, “The medical observation apparatus according to claim 1, wherein the imaging device is mounted on an endoscope (imaging section 21 is shown inside, i.e. mounted on endoscope 11 in fig 1)”.
Yamaguchi discloses for claim 16, “The medical observation apparatus according to claim 1, wherein the circuitry is further configured to set the region-of-interest based on the position and the orientation of the imaging device (0092 describes the zoom magnification “may be determined such that the tip end portion of at least one forceps 81 and the remarkable point Q estimated from a posture of the forceps 81 are included in the surgery region image 110”, i.e. dependent on the distance of the imager to the region-of-interest in order to keep the required elements in the field of view)”.
Modified Yamaguchi (as in claim 1 wherein Yamaguchi discloses below unless otherwise specified by a secondary reference) discloses for claim 17, “A medical observation system, comprising:
an imaging device (imaging section 21; fig 1) that images a surgical field and obtains a surgical field image; and
circuitry configured to:
generate three-dimensional information (fig 3; 0052 describes two imagers for imaging in 3D and 0052 providing depth information) of a surgical field from a surgical field image (surgery region image 0046, 0049) captured by the imaging device;
set at least one region-of-interest (important area detected by area position detection section 0077 with remarkable point Q; 0077-0080; fig 6, 7) in the surgical field image (surgery region image 0046, 0049);
estimate a self-position and orientation of the imaging device within the surgical field based on the three-dimensional information (Engle: SLAM, simultaneous localization and mapping 0008, specifically of an image data capture device, 0011 “The pose (position and/or orientation) of an image data capture device is taken into account in a three-point algorithm” and further details of the SLAM algorithm described at 0098);
track the region-of-interest, independent of optical flow analysis of the surgical field image (important area detected by area position detection section 0077 with remarkable point Q; 0163 describes the area position detecting process and the remarkable point estimating process are executed each time the surgery region image is supplied from the imaging sensor 24, i.e. continual detection/tracking based on image capture) in accordance with changes in the surgical field image captured by the imaging device based on the three-dimensional information by estimating an existence position of the region-of-interest (0078 states obtaining an estimated location coordinate of the remarkable point Q at the surgery region 82 in the three-dimensional space for detection) in a subsequent surgical field image (0077 states the detection, i.e. tracking of the RIO/remarkable point Q based on “different frames”, i.e. subsequent surgical field images) based on the three-dimensional information (0078 states the detection of remarkable point Q at the surgery region 82 is based on the three-dimensional space and the estimated location coordinate of the remarkable point Q in the three-dimensional space is supplied to the zoom image generation section 44) and the estimated self-position and orientation of the imaging device (Engle: SLAM, simultaneous localization and mapping 0008, specifically of an image data capture device, 0011 “The pose (position and/or orientation) of an image data capture device is taken into account in a three-point algorithm” and further details of the SLAM algorithm described at 0098)”;
generate a magnified surgical field image (zoom image QB by zoom image generation section 44; fig 7; 0081, 0089) in which the region-of-interest is magnified at a predetermined magnification (0084: “For example, the operation control section 46 supplies zoom magnification information for designating predetermined zoom magnification to the zoom image generation section 44”); and
output at least the magnified surgical field image to a display (45, 46; fig 1, 7; 0081-0083)”.
Yamaguchi discloses for claim 18, “The medical observation system according to claim 17, wherein the circuitry is further configured to determine a position and an orientation of the imaging device from the three-dimensional information; and set the region-of-interest based on the position and the orientation of the imaging device (0092 describes the zoom magnification “may be determined such that the tip end portion of at least one forceps 81 and the remarkable point Q estimated from a posture of the forceps 81 are included in the surgery region image 110”, i.e. dependent on the distance of the imager to the region-of-interest in order to keep the required elements in the field of view)”.
Modified Yamaguchi (as in claim 1 wherein Yamaguchi discloses below unless otherwise specified) discloses for claim 19, “A medical observation method comprising:
generating three-dimensional information (fig 3; 0052 describes two imagers for imaging in 3D and 0052 providing depth information) of a surgical field from a surgical field image obtained by imaging the surgical field using an imaging device (imaging section 21; fig 1);
setting at least one region-of-interest (important area detected by area position detection section 0077 with remarkable point Q; 0077-0080; fig 6, 7) in the surgical field image (surgery region image 0046, 0049);
estimate a self-position and orientation of the imaging device within the surgical field based on the three-dimensional information (SLAM, simultaneous localization and mapping 0008, specifically of an image data capture device, 0011 “The pose (position and/or orientation) of an image data capture device is taken into account in a three-point algorithm” and further details of the SLAM algorithm described at 0098);
track the region-of-interest (important area detected by area position detection section 0077 with remarkable point Q; 0163 describes the area position detecting process and the remarkable point estimating process are executed each time the surgery region image is supplied from the imaging sensor 24, i.e. continual detection/tracking based on image capture) in accordance with changes in the surgical field image captured by the imaging device by estimating an existence position of the region-of-interest (0078 states obtaining an estimated location coordinate of the remarkable point Q at the surgery region 82 in the three-dimensional space for detection) in a subsequent surgical field image (0077 states the detection, i.e. tracking of the RIO/remarkable point Q based on “different frames”, i.e. subsequent surgical field images) based on the three-dimensional information (0078 states the detection of remarkable point Q at the surgery region 82 is based on the three-dimensional space and the estimated location coordinate of the remarkable point Q in the three-dimensional space is supplied to the zoom image generation section 44) and the estimated self-position and orientation of the imaging device (Engle: SLAM, simultaneous localization and mapping 0008, specifically of an image data capture device, 0011 “The pose (position and/or orientation) of an image data capture device is taken into account in a three-point algorithm” and further details of the SLAM algorithm described at 0098)”;
generating a magnified surgical field image (zoom image QB by zoom image generation section 44; fig 7; 0081, 0089) in which the region-of-interest is magnified at a predetermined magnification (0084: “For example, the operation control section 46 supplies zoom magnification information for designating predetermined zoom magnification to the zoom image generation section 44”); and
outputting at least the magnified surgical field image to a display (45, 46; fig 1, 7; 0081-0083)”.
Yamaguchi discloses for claim 20, “The medical observation method according to claim 19, further comprising: setting the region-of-interest based on the position and the orientation of the imaging device (0092 describes the zoom magnification “may be determined such that the tip end portion of at least one forceps 81 and the remarkable point Q estimated from a posture of the forceps 81 are included in the surgery region image 110”, i.e. dependent on the distance of the imager to the region-of-interest in order to keep the required elements in the field of view)”.
Claim(s) 3 is/are rejected under 35 U.S.C. 103 as being unpatentable over Yamaguchi and Engel as applied to claim 2 above, and further in view of Xie US2006/0173358.
Yamaguchi does not disclose for claim 3, “The medical observation apparatus according to claim 2, wherein the circuitry is configured to control the display to display the surgical field image and the magnified surgical field image adjacent to each other”.
Xie teaches in the same field of endeavor, providing two images (106a and 106b; fig 19) adject to each other. It would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to incorporate the modification of Xie into the invention of Yamaguchi in order to configure the observation apparatus e.g. as claimed because it allows displaying the images without any overlap which covers up the image.
Claim(s) 4 is/are rejected under 35 U.S.C. 103 as being unpatentable over Yamaguchi and Engel as applied to claim 2 above, and further in view of Shioda US2001/0055062.
Yamaguchi does not disclose for claim 4, “The medical observation apparatus according to claim 2, wherein the circuitry is configured to control the display the surgical field image superimposed on a part of the magnified surgical field image”, but to this end, does disclose the superimposing a magnified/narrow field image on a surgical field image (fig 7). Shioda teaches in the same field of endeavor, superimposing a surgical field view image on a narrower field view (fig 36), i.e. the opposite of Yamaguchi’s disclosure and reading on the claimed invention. It would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to incorporate the modification of Shioda into the invention of Yamaguchi in order to configure the observation apparatus e.g. as claimed because it allows greater emphasis on the magnified field of view while still providing an image of the surgical field.
Claim(s) 5 is/are rejected under 35 U.S.C. 103 as being unpatentable over Yamaguchi and Engel as applied to claim 2 above, and further in view of Krupnik et al. US2012/0113239.
Yamaguchi does not disclose for claim 5, “The medical observation apparatus according to claim 2, wherein the display includes two separate display panels, and the circuitry is configured to display the surgical field image on a first display panel and to display the magnified surgical field image individually on a second display panel”. Krupnik teaches in the same field of endeavor, providing multiple monitors which may be used to display image data. It would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to incorporate the modification of Krupnik into the invention of Yamaguchi in order to configure the observation apparatus e.g. as claimed because it allows a dedicated physical monitor to display relevant images.
Claim(s) 6 is/are rejected under 35 U.S.C. 103 as being unpatentable over Yamaguchi and Engel as applied to claim 1 above, and further in view of Elbaum US6,201,880.
Yamaguchi discloses for claim 6, “The medical observation apparatus according to claim 1, wherein the circuitry is configured to designate a specific position (remarkable point Q; fig 7; 0078) of the surgical field image to be displayed on the display, as a region-of-interest (area of interest QA; fig 7; 0078), in a state where the specific position is aligned with a predetermined position (position of zoom image QB; fig 7; 0089)”.
Yamaguchi does not disclose “and on condition that a setting signal instructing the setting of the region-of-interest has occurred”. Elbaum teaches in the same field of endeavor, providing a user controlled instruction for setting a region-of-interest (11:25-30). It would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to incorporate the modification of Elbaum into the invention of Yamaguchi in order to configure the observation apparatus e.g. as claimed because it provides user control for selecting a region-of-interest.
Claim(s) 8 is/are rejected under 35 U.S.C. 103 as being unpatentable over Yamaguchi and Engel as applied to claim 1 above, and further in view of Popovic US2012/0209069.
Yamaguchi does not disclose for claim 8, “The medical observation apparatus according to claim 1, wherein the imaging device includes one imaging element, and the circuitry is configured to generate three-dimensional information of a surgical field based on at least two surgical field images captured by the imaging device at different times”. Popovic teaches in the same field of endeavor, generating a 3D image from a single imager with “two or more images of the same scene taken at different time moments” 0030. It would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to incorporate the modification of Popovic into the invention of Yamaguchi in order to configure the observation apparatus e.g. as claimed because it allows 3D information from a single imager instead of two imagers.
Claim(s) 13 is/are rejected under 35 U.S.C. 103 as being unpatentable over Yamaguchi and Engel as applied to claim 1 above, and further in view of Murakami US2010/0240953.
Yamaguchi discloses for claim 13, “The medical observation apparatus according to claim 1 wherein the circuitry is configured to perform camera shake correction on the surgical field image and the magnified surgical field image”. Murakami teaches in the same field of endeavor, providing camera shake correction for endoscopic imaging (0073). It would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to incorporate the modification of Murakami into the invention of Yamaguchi in order to configure the observation system e.g. as claimed because it provides “for maintaining continuity between the images” 0073.
Claim(s) 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Yamaguchi and Engel as applied to claim 1 above, and further in view of Mick US5,261,404.
Yamaguchi does not disclose for claim 15, “The medical observation apparatus according to claim 1, wherein the imaging device is mounted on a microscope”. Mick teaches in the same field of endeavor, providing 2D and 3D imaging with region of interest extracting in endoscopes, telescopes, and microscopes (3:65-68). It would have been obvious to one having ordinary skill in the art before the effective filing date of the claimed invention to incorporate the modification of Mick into the invention of Yamaguchi in order to configure the observation system e.g. as claimed because it allows the disclosed functionality in multiple different classes of imaging devices.
Response to Arguments
Applicant’s arguments with respect to claim(s) 1 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Applicant’s argument with respect to Iwaki (page 10) is moot since it is not relied upon in the rejection.
Applicant's arguments filed 11/21/2025 have been fully considered but they are not persuasive.
Applicant’s argument that Yamaguchi does not “track” the ROI but re-calculates the ROI at every step by finding a surgical instrument and then mathematically extending a line from the instrument is considered incorrect. As described above, Yamaguchi 0077 describes the detection of an important area/RIO can also be an area position detection section which performs an initial detection based on the images and not dependent on the relative position of an instrument, i.e. “certain tissues, body parts, bleeding or blood vessels, etc.”. Additionally, applicant states (page 10 para 2) that “The claims are directed to tracking that is independent of any instrument and tracks the ROI based on a stable 3D map”, but the claim recites “track the region-of-interest in accordance with changes in the surgical field image … based on the three-dimensional information and the estimated self-position and orientation of the imaging device
Applicant’s argument with respect to Engle relates to the application of secondary reference Iwaki (page 10 para 3, items (1) and (3)) and therefore is considered moot since Iwaki is not relied upon in the rejection. For item (2), i.e. the applicability of Engle’s allegedly unrelated device navigation system, Engle discloses at 0066 that the implementation advantageously leads to a method for the visual navigation of an instrument, but also provides 0067 mapping of the environment for the purpose of compiling a land map, particularly compiling external and internal surfaces of the environment, aspects which are related to and relevant for the detection described by Yamaguchi, e.g. detecting a location and a remarkable point on a surface of the body cavity (Yamaguchi: fig 6).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JAE K WOO whose telephone number is (571)272-0837. The examiner can normally be reached M-F 8:30-2:30p, 6p-9p.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Anhtuan Nguyen can be reached at (571) 272-4963. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Jae Woo/Examiner, Art Unit 3795
/ANH TUAN T NGUYEN/Supervisory Patent Examiner, Art Unit 3795
1/27/26
Prosecution Insights