DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Claims
Claims 1-5 are amended. Claims 6-20 are new.
Claim Objections
Claims 1 and 2 are objected to because of the following informalities: Claims 1 and 2 recite “medica” instead of “medical”. Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 1-20 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention. Claim 1 recites “being mounted relative to an installation surface,”. The monitor being “mounted” is not disclosed in the Specification. Therefore, the monitor being “mounted” is new matter and is rejected under 112a. Claim 1-20 recites “medical robot system”. This claimed subject matter is not disclosed in the Specification. For example, while the “surgical robot” is originally mentioned in the Specification, filed 06/26/2024, the subject “medical robot” is new matter. Besides amending the title of the Specification to “Medical surgical robot system”, the claimed subject matter of “medical robot system” and “medical robot” is not mentioned in the Specification to further disclose the claimed subject matter. Therefore, “medical robot system” is new matter and is rejected under 112a.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-5 and 17-18 are rejected under 35 U.S.C. 103 as being unpatentable over LEE (No. US-20110306986-A1 “Lee”) in view of ITOTANI (No. WO-2020203139-A1 “Itotani”) and in further view of SHEKHAR (No. US-20140303491-A1 “Shekhar”).
Regarding claim 1, Lee teaches, “A medical robot system comprising:” (surgical robot system; 0013);
“a spatial reproduction display including a monitor having a display surface and being mounted relative to an installation surface, the spatial reproduction display being” (a screen display outputted through a monitor unit; 0138); (The monitor unit 6 can be composed of one or more monitors, each of which can individually display information required during surgery; 0169);
Lee discloses a monitor unit that has monitors that can display surgery information like spatial reproduction display. It is known in the art that obvious for a person skilled in the art to know that a monitor
“a medical device connected to the spatial reproduction display; and” (One or more slave robots 2 can be used to perform surgery on a patient, and the laparoscope 5 for displaying the surgical site on the monitor unit 6 as an on-screen image can be implemented on an independent slave robot; 0173);
“a medical robot that operates by an operation of an operator via the medical device.” (a master robot 1, by which the operator remotely controls the slave robot; 0164);
While Lee does not teach “capable of displaying a medica video visually recognized as a stereoscopic image in a stereoscopic space is defined by a plane forming a floor through which a lower side of display surface of the monitor and a normal surface of the installation surface pass and a plane forming a back wall through which an upper side of the display surface and a parallel surface of the installation surface pass;”
Itotani teaches “capable of displaying a medica video visually recognized as a stereoscopic image” (the display unit 60 may be a stereoscopic display capable of displaying a stereoscopic image; Pg.4, Para 4); “in a stereoscopic space is defined by a plane forming a floor through which a lower side of display surface of the monitor and a normal surface of the installation surface pass and a plane forming a back wall through which an upper side of the display surface and a parallel surface of the installation surface pass;” (the object OB is arranged on the front side of the display surface 610 of the display unit 60; Pg.5, Para 4); (the object OB is displayed on the back side of the display surface 610; Pg.5, Para 4); (the display unit 60 is installed so that the display surface 610 of the display unit 60 is perpendicular to the horizontal plane in the real space. ... the display unit 60 is oblique (non-vertical) to the horizontal plane in the real space. ... to intersect the display surface 610 while standing upright with respect to the horizontal plane; Pg.5, Para 5); (it is possible to give a sufficient three-dimensional effect by arranging the floor surface and the wall surface, and it is possible to suppress the burden on the user; Pg.5, Para 7);
Itotani discloses a stereoscopic display that displays an object near the display surface with a front and back side of the display surface. Itotani also discloses the display may be installed perpendicular or oblique or that the object may intersect the display surface while standing upright. While not explicitly disclosing a floor plane or back wall plane, Itotani discloses a floor surface and wall surface that is an obvious geometric implementation.
While Lee and Itotani do not teach “a camera configured to capture medical video;”
Shekhar teaches “a camera configured to capture medical video;” (live stereoscopic video of the surgical field; 0024);
(Utilizing emerging camera technology, a system according to the present disclosure provides a real-time stereoscopic augmented reality (AR) system for laparoscopic surgery by merging live laparoscopic ultrasound image data with stereoscopic video; 0018);
Shekar discloses using camera and having a live medical video of the surgical site. This teaches the claimed subject matter of camera capturing a medical video.
Lee Itotani and Shekhar are analogous art as they are related to image processing for surgical robotic systems.
The motivation for the above is have efficient video capture and display for user friendly operation of robot.
Therefore, it would have been obvious for an ordinary skilled person in the art before the effective filing date of claimed invention to have modified Lee by displaying a medica video visually recognized as a stereoscopic image in a stereoscopic space is defined by a plane forming a floor through which a lower side of display surface of the monitor and a normal surface of the installation surface pass and a plane forming a back wall through which an upper side of the display surface and a parallel surface of the installation surface pass as taught by Itotani and a camera configured to capture medical video as taught by Shekhar.
Regarding claim 2, while Lee and Itotani fails to teach all of claim 2, Shekhar teaches “The medical robot system according to claim 1, wherein
the spatial reproduction display displays a composite image obtained by combining the medical video, reference information indicating preoperative information or intraoperative information, and a base layer that increases stereoscopic effect of the medica video.” (creating composite images by augmenting the endoscopic video with 2D or 3D tomographic images of the surgical anatomy; 0017);
(An existing feature-based registration method is then applied to register pre-operative tomographic images with intra-operative tomographic image volume; 0093);
(Fig. 9 Shows a Registration Module for intra-operative and pre-operative images.)
(An improved multimodality surgical visualization, wherein 3D ultrasound images are combined with laparoscopic video);
Shekhar teaches how to get composite images from a video of a surgical anatomy. Thus, a registration module is applied to the tomographic images for identifying intra-operative and pre-operative images. The images are also combined with laparoscopic video that relates to the increase of stereoscopic effects, to get and improved display of composite images.
The motivation for the above is have a more efficient and accurate display of images in relation to surgery.
Therefore, it would have been obvious for an ordinary skilled person in the art before the effective filing date of claimed invention to have modified Lee and Itotani by combining the surgical video, reference information indicating preoperative information or intraoperative information, and a base unit that increases stereoscopic effect of the surgical video as taught by Shekhar.
Regarding claim 3, while Lee and Itotani fail to teach “the spatial reproduction display displays a composite image obtained by combining the medical video, reference information indicating preoperative information or intraoperative information, a base layer that increases stereoscopic effect of the medical video,”
Shekhar teaches “the spatial reproduction display displays a composite image obtained by combining the medical video, reference information indicating preoperative information or intraoperative information, and a base layer that increases stereoscopic effect of the medical video.” (creating composite images by augmenting the endoscopic video with 2D or 3D tomographic images of the surgical anatomy; 0017);
(An existing feature-based registration method is then applied to register pre-operative tomographic images with intra-operative tomographic image volume; 0093);
(Fig. 9 Shows a Registration Module for intra-operative and pre-operative images.)
(An improved multimodality surgical visualization, wherein 3D ultrasound images are combined with laparoscopic video);
Claim 2 limitation is similar in scope and functions as this limitation, therefore, this claim limitation is also rejected under the same rationale as claim 2.
However, Lee does teach “and an interactive part that is interactively placeable, movable or rotatable on a basis of an operation of a user.”
(The operator's manipulation of the handles 10 can cause the robot arm 3 to perform a position movement, rotation, cutting operation, etc; 0165);
Lee discloses how the operator can manipulate and control the surgical robot with handles, to preform movements, etc. This relates to the interactive part that is placeable, movable, or rotatable by the operation of the user.
The motivation for the above is have a more efficient and accurate display of images in relation to surgery.
Regarding claim 4, while Lee and Itotani fails to teach all of claim 2, Shekhar teaches “The medical robot system according to claim 1, wherein
the spatial reproduction display displays a composite image arranged on a virtual space such that reference information indicating preoperative information or intraoperative information is displayed in front of the medical video.”
(creating composite images by augmenting the endoscopic video with 2D or 3D tomographic images of the surgical anatomy; 0017);
(The availability of volumetric data may also ease the interpretation problem of conventional ultrasound images and allow presenting to operating surgeons 3D rendered views and reformatted cross-sectional views different from the acquisition planes; 0022);
(The pre-operative tomographic image volume is registered with intra-operative tomographic image volume using an intensity-based deformable image registration algorithm; 0092);
Shekhar discloses composite images that allow the surgeons to see rendered viewpoints of the surgical site and have available data that help understand the images. This relates to seeing reference information displayed on the front of the surgical video due to the different viewpoints.
The motivation for the above is an accurate display of information on the surgical video.
Therefore, it would have been obvious for an ordinary skilled person in the art before the effective filing date of claimed invention to have modified Lee and Itotani by arranging on a virtual space such that reference information indicating preoperative information or intraoperative information is displayed in front of the surgical video as taught by Shekhar.
Regarding claim 5, while Lee and Itotani fail to teach all of claim 2, Shekhar teaches “The medical robot system according to claim 1, wherein
the spatial reproduction display displays a composite image arranged on a virtual space such that reference information indicating preoperative information or intraoperative information is displayed on a back side of the medical video.”
(creating composite images by augmenting the endoscopic video with 2D or 3D tomographic images of the surgical anatomy; 0017);
(The availability of volumetric data may also ease the interpretation problem of conventional ultrasound images and allow presenting to operating surgeons 3D rendered views and reformatted cross-sectional views different from the acquisition planes; 0022); ‘
(The pre-operative tomographic image volume is registered with intra-operative tomographic image volume using an intensity-based deformable image registration algorithm; 0092);
Shekhar discloses composite images that allow the surgeons to see rendered viewpoints of the surgical site and have available data that help understand the images. This relates to seeing reference information displayed on the back side of the surgical video due to the different viewpoints.
The motivation for the above is an accurate display of information on the surgical video.
Therefore, it would have been obvious for an ordinary skilled person in the art before the effective filing date of claimed invention to have modified Lee and Itotani by arranging on a virtual space such that reference information indicating preoperative information or intraoperative information is displayed on a back side of the surgical video as taught by Shekhar.
Regarding claim 17, while Lee, Shekhar and Ramirez do not teach the limitations, Itotani teaches “The medical robot system according to claim 1, wherein the monitor of the spatial reproduction display is mounted such that the display surface is inclined non- perpendicularly with respect to a horizontal plane of the installation surface.” (the display unit 60 is installed so that the display surface 610 of the display unit 60 is oblique (non-vertical) to the horizontal plane in the real space; Pg. 5, Para 6);
Itotani discloses a display surface that is oblique to the horizontal plane. An oblique line or plane is defined as one that is neither parallel nor perpendicular to a given line or surface. This teaches the claimed subject matter of the display surface being non- perpendicularly with respect to a horizontal plane.
The motivation for the above is to have an efficient mounted monitor for user friendly viewing of display.
Therefore, it would have been obvious for an ordinary skilled person in the art before the effective filing date of claimed invention to have modified Lee, Shekhar and Ramirez by the monitor of the spatial reproduction display is mounted such that the display surface is inclined non- perpendicularly with respect to a horizontal plane of the installation surface as taught by Itotani.
Regarding claim 18, while Lee, Shekhar and Ramirez do not teach the limitations, Itotani teaches “The medical robot system according to claim 17, wherein the spatial reproduction display is configured so that a stereoscopic object displayed in the stereoscopic space is perceived as standing upright with respect to the horizontal plane while crossing the display surface.” (the object OB is arranged so as to intersect the display surface 610 while standing upright with respect to the horizontal plane; Pg. 5, Para 7); (the stereoscopically displayed object; Pg. 8, Para 4);
Itotani discloses an object in the display surface that is standing upright with respect to the horizontal plane. This teaches the claimed subject matter of an object displayed is perceived as standing upright with respect to the horizontal plane.
The motivation for the above is to have an efficient display of stereoscopic space for user friendly viewing of display.
Therefore, it would have been obvious for an ordinary skilled person in the art before the effective filing date of claimed invention to have modified Lee, Shekhar and Ramirez by the spatial reproduction display is configured so that a stereoscopic object displayed in the stereoscopic space is perceived as standing upright with respect to the horizontal plane while crossing the display surface as taught by Itotani.
Claims 6 is rejected under 35 U.S.C. 103 as being unpatentable over LEE in view of ITOTANI and SHEKHAR and in further view of WANG (No. CN-106773087-A “Wang”).
Regarding claim 6, while Lee, Itotani and Shekhar do not teach the limitation, Wang teaches “The medical robot system according to claim 1, wherein the spatial reproduction display further includes a half-mirror, wherein the monitor is positioned so that a reflected image of the medical video appears to float within the stereoscopic space.” (imaging display system generates a stereoscopic scene 0104, .... then transmitted or reflected by a half mirror; Pg.2, Para 6); (has a display effect in which the stereoscopic picture floats; Pg. 1, Para 4);
Wang discloses a display system that has a half mirror and floats, which teaches the claimed subject matter of a monitor that has a half mirror and floats.
Lee, Itotani, Shekhar and Wang are analogous art as they are related to imaging stereoscopic display device.
The motivation for the above is to have accurate display of stereoscopic image on monitor for user usage.
Therefore, it would have been obvious for an ordinary skilled person in the art before the effective filing date of claimed invention to have modified Lee, Itotani and Shekhar by wherein the spatial reproduction display further includes a half-mirror, wherein the monitor is positioned so that a reflected image of the medical video appears to float within the stereoscopic space as taught by Wang.
Claims 7, 9-10, 13-14 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over LEE in view of in view of ITOTANI and SHEKHAR and in further view of RAMIREZ (No. WO-2019210322-A1 “Ramirez”).
Regarding claim 7, while Lee, Itotani and Shekhar do not teach the limitations, Ramirez teaches “The medical robot system according to claim 1, further comprising control circuitry configured to:” (processing circuitry; Para 0087);
“acquire display surface information related to the display surface of the monitor;” (The Tenderer program l580e formats the first and second combined stereoscopic images such that they are compatible with the type and/or screen size of the display monitors and/or screen; Para 00282);
“acquire user information regarding the operator;” (determine a position of an operator’s eyes with respect to a 3D display, and can adjust the view depending on where in the displayed scene the operator is looking; Para 00430);
“acquire data of content to be displayed on the spatial reproduction display; and control display of the medical video on the spatial reproduction display based on the display surface information, the user information, and the data of the content.” (request (or otherwise receive) display characteristics of the display monitor 512 prior to preparing the stereoscopic image; Para 00284);
Ramirez discloses obtaining display information though the Tenderer program and user information though operator eyes and movement. As well as display characteristics of the monitor that would be obvious to include display information, user information and the content, the stereoscopic image, too. Thus, teaching the claimed subject matter.
Lee, Itotani, Shekhar and Ramirez are analogous art as they are related to control of robot systems.
The motivation for the above is to accurate display and user information for accurate and user-friendly display.
Therefore, it would have been obvious for an ordinary skilled person in the art before the effective filing date of claimed invention to have modified Lee, Itotani and Shekhar by acquire display surface information related to the display surface of the monitor acquire user information regarding the operator acquire data of content to be displayed on the spatial reproduction display and control display of the medical video on the spatial reproduction display based on the display surface information, the user information, and the data of the content as taught by Ramirez.
Regarding claim 9, while Lee, Shekhar and Ramirez do not teach the limitations, Itotani teaches “The medical robot system according to claim 7, wherein the control circuitry is further configured to:
acquire, as the display surface information, an angle formed by a horizontal plane and the display surface of the monitor in a real space; and” (the display unit 60 is installed so that the display surface 610 of the display unit 60 is oblique (non-vertical) to the horizontal plane in the real space; Pg. 5, Para 6);
“set a drawing space based on the angle, the drawing space having an upper end and a lower end of the display surface as diagonal sides.” (the display unit 60 is installed so that the display surface 610 of the display unit 60 is oblique (non-vertical) to the horizontal plane; Pg. 5, Para 6); (the display unit 60 may be a stereoscopic display capable of displaying a stereoscopic image; Pg. 4, Para 4);
Itotani discloses a display unit that is oblique, at an angle, to the horizontal plane which teaches the claimed subject matter of acquiring an angle. The stereoscopic display teaches the drawing space of the display surface. While not explicitly mentioning an upper and lower end, due to the angle of the drawing space, it is obvious to a person skilled in the art that they would be an upper and lower end to an angle as diagonal sides. Thus, teaching the claimed subject matter.
The motivation for the above is to have accurate display of surgical site on monitor for user friendly viewing.
Therefore, it would have been obvious for an ordinary skilled person in the art before the effective filing date of claimed invention to have modified Lee, Shekhar and Ramirez by acquiring, as the display surface information, an angle formed by a horizontal plane and the display surface of the monitor in a real space and set a drawing space based on the angle, the drawing space having an upper end and a lower end of the display surface as diagonal sides as taught by Itotani.
Regarding claim 10, while Lee, Shekhar and Ramirez do not teach the limitations, Itotani teaches “The medical robot system according to claim 9, wherein the control circuitry is further configured to dispose, within the drawing space, a floor surface corresponding to the plane forming the floor, a back wall surface corresponding to the plane forming the back wall, and at least one side wall surface, and to arrange the medical video as a stereoscopic object within the drawing space.” (it is possible to give a sufficient three-dimensional effect by arranging the floor surface and the wall surface, and it is possible to suppress the burden on the user; Pg.5, Para 7); (the CDR is changed (in other words, if the image is rotated by software processing), the stereoscopically displayed object will be distorted; Pg. 8, Para 4);
Itotani discloses arranging the floor surface and wall surface which teaches disposing a floor surface and a wall surface. It would be obvious that the wall surface disclosed can be a back wall and at least one wall surface. Also, Itotani discloses stereoscopically displayed object. While Itotani does not explicitly state a medical video, Lee does teach the medical video, which in combination showcase the arrangement of the medical video as a stereoscopic object within the space.
The motivation for the above is to have accurate display of medical video for user friendly viewing.
Therefore, it would have been obvious for an ordinary skilled person in the art before the effective filing date of claimed invention to have modified Lee, Shekhar and Ramirez by disposing, within the drawing space, a floor surface corresponding to the plane forming the floor, a back wall surface corresponding to the plane forming the back wall, and at least one side wall surface, and to arrange the medical video as a stereoscopic object within the drawing space as taught by Itotani.
Regarding claim 13, while Lee, Shekhar and Ramirez do not teach the limitations, Itotani teaches “The medical robot system according to claim 7, further comprising a display surface sensor configured to detect an orientation of the display surface and provide the orientation to the control circuitry configured to control display of the medical video, wherein the display surface sensor includes at least one of an acceleration sensor, a gyro sensor, and a magnetic sensor.” (a motion sensor for measuring the motion of the joint at a position corresponding to each joint; Pg.3, Para 6); (the control unit 70 receives the information measured by the sensor of the master unit 10 from the master unit 10, and acquires the designated position of the master unit 10 based on the received information. Then, the control unit 70 controls the designated position of the slave unit 20 (for example, the position of the; Pg.4, Para 5);
Itotani discloses a motion sensor as well as receiving information from a sensor of the unit that controls the position the slave unit. While Itotani does not explicitly mention an acceleration sensor, a gyro sensor, and a magnetic sensor, it would be obvious to use a sensor that receives information used to control the position. They do disclose a motion sensor which is the same category as an acceleration sensor, gyro sensor, and magnetic sensor. All four of these falls under the umbrella of motion, inertial, or orientation tracking This teaches the claimed subject matter or a display sensor that provides the orientation for the control circuity.
The motivation for the above is to have efficient detection of orientation of the display for accurate display of medical video.
Therefore, it would have been obvious for an ordinary skilled person in the art before the effective filing date of claimed invention to have modified Lee, Shekhar and Ramirez by a display surface sensor configured to detect an orientation of the display surface and provide the orientation to the control circuitry configured to control display of the medical video, wherein the display surface sensor includes at least one of an acceleration sensor, a gyro sensor, and a magnetic sensor as taught by Itotani.
Regarding claim 14, while Lee, Shekhar and Ramirez do not teach the limitations, Itotani teaches “The medical robot system according to claim 7, further comprising a user detector configured to detect at least one of a position and a posture of the operator observing the stereoscopic image and provide the detected information to the control circuitry configured to control display of the medical video, wherein the user detector includes at least one of a camera, a depth camera, and a human sensor.” (detection unit 30 is configured to detect the eyes of the user (observer); Pg.4, Para 1); (when the detection unit 30 detects the user's eye, the angle calculation unit 710 determines the position coordinates of the user's eye; Pg.9, Para 7); (positional relationship between the imaging unit (for example, a camera) that calculates the first positional relationship; Pg.2, Para 7); (imaging unit 50 can acquire the depth information of the object; Pg.10, Para 4);
Itotani discloses detection unit that detect the observers’ eyes and determines the position information of the eye. As well as having an imaging unit that, while not explicitly mentioning a depth camera, and a human sensor, it is obvious that a user detector would have a camera, that also detect depth and the human in order to track and detect the human user’s eyes. Therefore, this teaches the claimed subject matter.
The motivation for the above is to have efficient detection of operator for user friendly viewing of medical video.
Therefore, it would have been obvious for an ordinary skilled person in the art before the effective filing date of claimed invention to have modified Lee, Shekhar and Ramirez by a user detector configured to detect at least one of a position and a posture of the operator observing the stereoscopic image and provide the detected information to the control circuitry configured to control display of the medical video, wherein the user detector includes at least one of a camera, a depth camera, and a human sensor as taught by Itotani.
Regarding claim 19, Lee teaches “A medical robot system comprising:” (surgical robot system; 0013);
“an medical device connected to the spatial reproduction display;” (One or more slave robots 2 can be used to perform surgery on a patient, and the laparoscope 5 for displaying the surgical site on the monitor unit 6 as an on-screen image can be implemented on an independent slave robot; 0173);
“a medical robot that operates by an operation of an operator via the medical device; and” (a master robot 1, by which the operator remotely controls the slave robot; 0164);
While Lee does not “a spatial reproduction display including a monitor having a display surface, the spatial reproduction display being capable of displaying a medical video visually recognized as a stereoscopic image in a stereoscopic space formed by a plane through which a lower side of the display surface and a normal surface of an installation surface pass and a plane through which an upper side of the display surface and a parallel surface of the installation surface pass;”
Itotani teaches “a spatial reproduction display including a monitor having a display surface, the spatial reproduction display being capable of displaying a medical video visually recognized as a stereoscopic image in a stereoscopic space formed by a plane through which a lower side of the display surface and a normal surface of an installation surface pass and a plane through which an upper side of the display surface and a parallel surface of the installation surface pass;” (the display unit 60 may be a stereoscopic display capable of displaying a stereoscopic image; Pg.4, Para 4); (the object OB is arranged on the front side of the display surface 610 of the display unit 60; Pg.5, Para 4); (the object OB is displayed on the back side of the display surface 610; Pg.5, Para 4); (the display unit 60 is installed so that the display surface 610 of the display unit 60 is perpendicular to the horizontal plane in the real space. ... the display unit 60 is oblique (non-vertical) to the horizontal plane in the real space. ... to intersect the display surface 610 while standing upright with respect to the horizontal plane; Pg.5, Para 5); (it is possible to give a sufficient three-dimensional effect by arranging the floor surface and the wall surface, and it is possible to suppress the burden on the user; Pg.5, Para 7);
Itotani discloses a stereoscopic display that displays an object near the display surface with a front and back side of the display surface. Itotani also discloses the display may be installed perpendicular or oblique or that the object may intersect the display surface while standing upright. While not explicitly disclosing a floor plane or back wall plane, Itotani discloses a floor surface and wall surface that is an obvious geometric implementation.
While Lee and Itotani does not teach “a camera configured to capture medical video;”
Shekhar teaches “a camera configured to capture medical video;” (live stereoscopic video of the surgical field; 0024);
(Utilizing emerging camera technology, a system according to the present disclosure provides a real-time stereoscopic augmented reality (AR) system for laparoscopic surgery by merging live laparoscopic ultrasound image data with stereoscopic video; 0018);
Shekar discloses using camera and having a live medical video of the surgical site. This teaches the claimed subject matter of camera capturing a medical video.
The motivation for the above is have efficient video capture and display for user friendly operation of robot.
While Lee, Itotani and Shekhar do not teach the limitations “further comprising control circuitry configured to: acquire display surface information related to the display surface of the monitor; acquire user information regarding the operator; acquire data of content to be displayed on the spatial reproduction display; and control display of the medical video on the spatial reproduction display based on the display surface information, the user information, and the data of the content”
However, Ramirez teaches “control circuitry configured to:” (processing circuitry; Para 0087);
“acquire display surface information related to the display surface of the monitor;” (The Tenderer program l580e formats the first and second combined stereoscopic images such that they are compatible with the type and/or screen size of the display monitors and/or screen; Para 00282);
“acquire user information regarding the operator;” (determine a position of an operator’s eyes with respect to a 3D display, and can adjust the view depending on where in the displayed scene the operator is looking; Para 00430);
“acquire data of content to be displayed on the spatial reproduction display; and
control display of the medical video on the spatial reproduction display based on the display surface information, the user information, and the data of the content.” (request (or otherwise receive) display characteristics of the display monitor 512 prior to preparing the stereoscopic image; Para 00284);
Ramirez discloses obtaining display information though the Tenderer program and user information though operator eyes and movement. As well as display characteristics of the monitor that would be obvious to include display information, user information and the content, the stereoscopic image, too. Thus, teaching the claimed subject matter.
The motivation for the above is to accurate display and user information for accurate and user-friendly display.
Therefore, it would have been obvious for an ordinary skilled person in the art before the effective filing date of claimed invention to have modified Lee, Itotani and Shekhar by a camera that captures the medical video as taught by Shekhar and by acquire display surface information related to the display surface of the monitor acquire user information regarding the operator acquire data of content to be displayed on the spatial reproduction display and control display of the medical video on the spatial reproduction display based on the display surface information, the user information, and the data of the content as taught by Ramirez.
Claims 8, 11-12, 15-16 and 20 is rejected under 35 U.S.C. 103 as being unpatentable over LEE in view of in view of ITOTANI and SHEKHAR and in further view of RAMIREZ and in further view of CHOI (No. WO-2012060586-A2 “Choi”).
Regarding claim 8, while Lee, Itotani, Shekhar and Ramirez do not teach the limitations, Choi teaches “The medical robot system according to claim 7, wherein the control circuitry is further configured to generate a stereoscopic image by performing orthographic projection based on a position of a left eye and a position of a right eye of the operator as acquired from the user information.” (generating analysis information by analyzing at least one of a position change of a pupil viewed through an eyepiece; Pg.3, Para 10); (generate a stereoscopic image by various other methods; Pg. 26, Para 12);
Choi discloses generating a stereoscopic image based on a position of a pupil viewed though the eyepiece, thus teaching the claimed subject matter of orthographic projection based on a position of a left eye and a position of a right eye of the operator.
Lee, Itotani, Shekhar, Ramirez, and Choi are analogous art as they are related a surgical robot system.
The motivation for the above is to accurate project based on user pupil position, thus for accurate generation of stereoscopic image.
Therefore, it would have been obvious for an ordinary skilled person in the art before the effective filing date of claimed invention to have modified Lee, Itotani, Shekhar and Ramirez by wherein the control circuitry is further configured to generate a stereoscopic image by performing orthographic projection based on a position of a left eye and a position of a right eye of the operator as acquired from the user information as taught by Choi.
Regarding claim 11, while Lee, Itotani, Shekhar and Ramirez do not teach the limitations, Choi teaches “The medical robot system according to claim 7, wherein the control circuitry is further configured to:
acquire, as the user information, a position of a left eye and a position of a right eye of the operator; and” (generating analysis information by analyzing at least one of a position change of a pupil viewed through an eyepiece; Pg.3, Para 10);
“set a direction of a virtual camera based on at least one of a facial orientation of the operator and a line-of-sight direction of the left eye and the right eye of the operator.” (analyzes the direction and size of the facial movement of the operator using the sensing information received from the motion sensing unit 310 and controls the position and the image input angle of the laparoscope 5 according to the interpreted result; Pg.17, Para 2);
Choi discloses gathering information on user based on the pupil viewed through the eyepiece as well as the direction of user based on facial movement and motion sensing. This teaches the claimed subject matter of acquiring user information based on eye position and setting direction of camera based on facial orientation and sight of eyes of the operator.
The motivation for the above is to have accurate gathering of user information based on eye and facial orientation for user friendly usage.
Therefore, it would have been obvious for an ordinary skilled person in the art before the effective filing date of claimed invention to have modified Lee, Itotani, Shekhar and Ramirez by acquire, as the user information, a position of a left eye and a position of a right eye of the operator and set a direction of a virtual camera based on at least one of a facial orientation of the operator and a line-of-sight direction of the left eye and the right eye of the operator as taught by Choi.
Regarding claim 12, while Lee, Itotani, Shekhar and Ramirez do not teach the limitations, Choi teaches “The medical robot system according to claim 11, wherein the control circuitry is further configured to:
virtually install a virtual camera at the position of the left eye or the right eye of the operator as a virtual viewpoint;” (The eye tracker unit includes a camera unit for imaging the inside of the surgical robot toward the eyepiece unit to generate image data; Pg.3, Para 11); (analyzing at least one of a position change of a pupil viewed through an eyepiece, a change of an eye shape, and a viewing direction; Pg.3, Para 10);
“render the drawing space from the virtual viewpoint to acquire a viewpoint image; and” (acquiring a three-dimensional image, and the degree of overlapping of the left / right images for three-dimensional image processing may be adjusted according to the positions of the face and the eyes; Pg.4, Para 4);
“orthogonally project the viewpoint image onto a virtual display surface virtually installed on the display surface to generate a display image.” (projector for projecting an endoscope image on a dome-shaped screen; Pg.7, Para 11); (The projector 192 projects the endoscopic image onto the dome screen 191; Pg.35, Para 4);
Choi discloses the eye tracker unit that analyzes the potion of the eyes of the operator, thus teaching the claimed subject matter of using either left or right eye position as the viewpoint. Also discloses having 3D image from the viewpoint and position of the eye which teach the rendering from the virtual viewpoint to get viewpoint images. In addition, Choi also discloses projection of an image onto a screen, no while they do explicitly state orthogonally, they teach projecting the generate image viewpoint to a display screen.
The motivation for the above is to have an accurate display of viewpoint image for user friendly usage.
Therefore, it would have been obvious for an ordinary skilled person in the art before the effective filing date of claimed invention to have modified Lee, Itotani, Shekhar and Ramirez by virtually install a virtual camera at the position of the left eye or the right eye of the operator as a virtual viewpoint render the drawing space from the virtual viewpoint to acquire a viewpoint image and orthogonally project the viewpoint image onto a virtual display surface virtually installed on the display surface to generate a display image as taught by Choi.
Regarding claim 15, while Lee, Itotani, Shekhar and Ramirez do not teach the limitations, Choi teaches “The medical robot system according to claim 7, wherein the control circuitry is further configured to:
acquire, as the display surface information, a distance from a lower end to an upper end of the display surface; and” (The size of the dome screen 191 may be as large as the user can see, for example, its diameter may be about 1 m to 2 m; Pg.35, Para 6);
“control display of the medical video based on the distance so as to adjust a size of the stereoscopic space.” (projector for projecting an endoscope image on a dome-shaped screen; Pg.7, Para 11); (The projector 192 projects the endoscopic image onto the dome screen 191; Pg.35, Para 4);
Choi discloses a size of the screen that they user can see with example measurements 1m to 2m. This teaches acquiring a distance for the display surface as it showcases the distance within the display. In addition, Choi discloses the projection of an image on a screen which can teaches the control of the display to adjust the size of the space.
The motivation for the above is to have accurate display surface information for accurate distance of the display space for user friendly display.
Therefore, it would have been obvious for an ordinary skilled person in the art before the effective filing date of claimed invention to have modified Lee, Itotani, Shekhar and Ramirez acquire, as the display surface information, a distance from a lower end to an upper end of the display surface and control display of the medical video based on the distance so as to adjust a size of the stereoscopic space by as taught by Choi.
Regarding claim 16, while Lee, Shekhar and Ramirez do not teach the limitations “set a drawing space based on the display surface information, the drawing space having an upper end and a lower end of the display surface as diagonal sides” and “dispose a floor surface, a back wall surface, and at least one side wall surface within the drawing space”.
Itotani teaches “The medical robot system according to claim 15, wherein the control circuitry is further configured to:
set a drawing space based on the display surface information, the drawing space having an upper end and a lower end of the display surface as diagonal sides;” (the display unit 60 is installed so that the display surface 610 of the display unit 60 is oblique (non-vertical) to the horizontal plane; Pg. 5, Para 6); (the display unit 60 may be a stereoscopic display capable of displaying a stereoscopic image; Pg. 4, Para 4);
“dispose a floor surface, a back wall surface, and at least one side wall surface within the drawing space; and” (it is possible to give a sufficient three-dimensional effect by arranging the floor surface and the wall surface, and it is possible to suppress the burden on the user; Pg.5, Para 7); (the CDR is changed (in other words, if the image is rotated by software processing), the stereoscopically displayed object will be distorted; Pg. 8, Para 4);
The motivation for the above is to have accurate display space for user friendly display of image.
However, Lee, Itotani, Shekhar and Ramirez do not teach “generate a display image by orthogonally projecting a viewpoint image acquired from a virtual viewpoint corresponding to a position of an eye of the operator onto a virtual display surface virtually installed on the display surface of the monitor.”
Choi teaches “generate a display image by orthogonally projecting a viewpoint image acquired from a virtual viewpoint corresponding to a position of an eye of the operator onto a virtual display surface virtually installed on the display surface of the monitor.” (projector for projecting an endoscope image on a dome-shaped screen; Pg.7, Para 11); (The projector 192 projects the endoscopic image onto the dome screen 191; Pg.35, Para 4);
Itotani discloses a display unit that is oblique, at an angle, to the horizontal plane which teaches the claimed subject matter of acquiring an angle. The stereoscopic display teaches the drawing space of the display surface. While not explicitly mentioning an upper and lower end, due to the angle of the drawing space, it is obvious to a person skilled in the art that they would be an upper and lower end to an angle as diagonal sides. Itotani discloses arranging the floor surface and wall surface which teaches disposing a floor surface and a wall surface. It would be obvious that the wall surface disclosed can be a back wall and at least one wall surface. In combination, Choi also discloses projection of an image onto a screen, no while they do explicitly state orthogonally, they teach projecting the generate image viewpoint to a display screen.
The motivation for the above is to have accurate generation and projection of image for user friendly display of image.
Therefore, it would have been obvious for an ordinary skilled person in the art before the effective filing date of claimed invention to have modified Lee, Shekhar and Ramirez by set a drawing space based on the display surface information, the drawing space having an upper end and a lower end of the display surface as diagonal sides and dispose a floor surface, a back wall surface, and at least one side wall surface within the drawing space as taught by Itotani and by generate a display image by orthogonally projecting a viewpoint image acquired from a virtual viewpoint corresponding to a position of an eye of the operator onto a virtual display surface virtually installed on the display surface of the monitor as taught by Choi.
Regarding claim 20, while Lee, Itotani, Shekhar and Ramirez do not teach the limitations, Choi teaches “The medical robot system according to claim 19, wherein the control circuitry is further configured to generate a stereoscopic image by performing orthographic projection based on a position of a left eye and a position of a right eye of the operator as acquired from the user information.” (generating analysis information by analyzing at least one of a position change of a pupil viewed through an eyepiece; Pg.3, Para 10); (generate a stereoscopic image by various other methods; Pg. 26, Para 12);
Choi discloses generating a stereoscopic image based on a position of a pupil viewed though the eyepiece, thus teaching the claimed subject matter of orthographic projection based on a position of a left eye and a position of a right eye of the operator.
The motivation for the above is to accurate project based on user pupil position, thus for accurate generation of stereoscopic image.
Therefore, it would have been obvious for an ordinary skilled person in the art before the effective filing date of claimed invention to have modified Lee, Itotani, Shekhar and Ramirez by generating a stereoscopic image by performing orthographic projection based on a position of a left eye and a position of a right eye of the operator as acquired from the user information as taught by Choi.
Response to Arguments
Applicant’s arguments, see Pg. 1-2, filed 06/24/2026, with respect to Specification have been fully considered and are persuasive. The Objection of 01/28/2026 has been withdrawn.
Applicant’s arguments, see Pg. 11, filed 05/19/2026, with respect to the rejection(s) of claim(s) 1 under 102(a)(1) have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of Itotani.
The Applicant argues that Claim 1 now recites a display that creates a stereoscopic space defined by two specific planes that are structurally defined relative to a mounting relationship in a hardware architecture: (1) A plane through which the lower side of the display surface and the normal surface of the installation surface pass; and (2) A plane through which the upper side of the display surface and a parallel surface of the installation surface pass.
The Applicant also argues that Lee does not teach and does not anticipate the amended "spatial reproduction display" limitation. The amended claim requires a display that presents medical video as a stereoscopic image in a stereoscopic space formed by specific geometric planes relative to the monitor's display surface and the installation surface.
The Examiner replies that upon further examination, pertinent art Itotani teaches the amended claim 1 limitation, A plane through which the lower side of the display surface and the normal surface of the installation surface pass; and A plane through which the upper side of the display surface and a parallel surface of the installation surface pass. Itotani discloses a stereoscopic display that displays an object near the display surface with a front and back side of the display surface. Itotani also discloses the display may be installed perpendicular or oblique or that the object may intersect the display surface while standing upright. While not explicitly disclosing a “floor plane” or “back wall plane”, Itotani discloses a floor surface and wall surface that is an obvious geometric implementation.
The Applicant argues that the remaining rejected claims depend from claim 1 and are believed to be allowable for at least the reasons claim 1 is allowable. That Shekhar is directed to a standard stereoscopic display and fails to disclose or suggest a display with the geometric plane definition of claim 1.
The Examiner replies that claims 2-5 are dependent on claim 1. Since claim 1 is rejected, claims 2-5 are rejected under virtue of dependency to claim 1. Therefore, the Examiner maintains the rejections of claims 2-5, at least by virtue of their dependency from claim 1. Shekhar in combination with Lee and Itotani, that teaches a display with the geometric plane, thus teaches the scope of claims 2-5.
The Applicant argues that new claims 6-20 have been added. These claims further define the invention by reciting additional structural and functional features of the medical robot system, including the half- mirror spatial reproduction display (claim 6), control circuitry for adaptive display control responsive to display surface geometry and user position (claims 7-16, 19-20), specific sensor implementations (claims 13-14), non-perpendicular display surface mounting (claims 17-18), and the full rendering pipeline including orthographic projection from user eye positions. None of the cited references, Lee or Shekhar, teaches or suggests these limitations.
The Examiner replies due to new claims 6-20, further examination is required. Basis for rejection are from previously cited pertinent art, Itotani, Ramirez and Choi. An additional updated search was completed necessitated by new added claims 6-20, resulting in prior art Wang. The half- mirror spatial reproduction display (claim 6) was rejected by Wang. The control circuitry for adaptive display control responsive to display surface geometry and user position (claims 7-16, 19-20) were rejected by Ramirez, Choi, and/or Itotani. The specific sensor implementations (claims 13-14) and non-perpendicular display surface mounting (claims 17-18) were rejected by Itotani.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to BRIGITER D PROTAZI whose telephone number is (571)272-7995. The examiner can normally be reached Monday - Friday 7:30-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Said A Broome can be reached at 5712722931. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/B.D.P./Examiner, Art Unit 2612
/Said Broome/Supervisory Patent Examiner, Art Unit 2612
Prosecution Insights