DETAILED ACTION
This communication is responsive to Amendment filed 09/05/2025.
Claims 1, 3, 6, 8-17 and 19-25 are pending in this application. In the Amendment, claims 1, 17, 19-20 and 22-23 are amended, claim 18 is cancelled and claim 25 is new. This action is made Final.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant's arguments filed 09/05/2025 have been fully considered but they are not persuasive.
Applicant argued the cited art does not teach a system in which a display mode can be switched between a stop mode and a traveling mode by operation of a user on a display mode switching unit, and in which a communication target is selected via a selection operation by the user, as required by amended claim 1.
The Examiner respectfully disagrees as Hoffman teaches a display mode switching unit configured to switch the display mode in response to an operation on the operation terminal by a user (Hoffman, para.120, Fig.8A, mode selection of menus 812-816 switches modes from: Drive/Manipulate/Inspect via user selection), and in which a communication target is selected via a selection operation by the user (Hoffman, para.83, object 114 selection).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 3, 6, 8, 10-17, 20 and 22-23 are rejected under 35 U.S.C. 103 as being unpatentable over Hoffman et al. (“Hoffman”, US 2015/0190925) in view of Katayama et al. (“Katayama”, US 2020/0301433) in view of Jordan et al. (“Jordan”, US 2015/0077502) and further in view of Watanuki (US 2019/0389071).
As per claim 1, Hoffman teaches a remote operation system comprising:
a mobile robot (Hoffman, para.56, robot 200/300);
an operation terminal including a terminal processor and configured to communicate with the mobile robot (Hoffman, para.56-57, 77, OCU 100), the mobile robot including a camera that is attached to the mobile robot and that is configured to capture an image of surroundings of the mobile robot (Hoffman, para.64, cameras 218-219), and transmitting, to the operation terminal, monitor image information based on a captured image by the camera (Hoffman, para.77, video feed), the operation terminal including a terminal display unit (Hoffman, Fig.4A, display 110; para.75-76), receiving the monitor image information, and displaying, on the terminal display unit, an operation screen including a monitor image based on the captured image (Hoffman, para.77, video feed); and
a first processor (Hoffman, para.13), wherein
the first processor includes:
a screen display controller configured to display the operation screen on the terminal display unit (Hoffman, para.117, Fig.8A-8C), the operation screen including a first sub screen for displaying the monitor image (Hoffman, Fig.8A, camera view 810; para.119) and a second sub screen displaying an operation icon (Hoffman, Fig.8A, sub-menus 812-816), and configured to change a display content of the second sub screen in accordance with a predetermined display mode (Hoffman, Fig.8A-8C, submenus/display modes 812-816 display different menu options; para.120-123); and
a display mode switching unit configured to switch the display mode in response to an operation on the operation terminal by a user (Hoffman, para.120, Fig.8A, mode selection of menus 812-816);
a mode changeover switch for switching the display mode is displayed on the second sub screen (Hoffman, Fig.8A, sub-menus 812-816),
when switching operation to a stop mode is performed by the mode changeover switch, the display mode switching unit sets the display mode to a stop display mode corresponding to a situation in which the mobile robot is stopped (Hoffman, para.100, 105; robot stops in preview mode),
when the switching operation to a traveling mode is performed by the mode changeover switch, the display mode switching unit sets the display mode to a movement display mode corresponding to a situation in which the mobile robot is moving (Hoffman, Fig.8B, drive display mode 812; para.123),
when the display mode is the movement display mode (Hoffman, Fig.8B, drive display mode 812; para.123), the screen display controller displays, on the second sub screen, operation support information for supporting an operation for moving the mobile robot (Hoffman, para.123, Fig.8B, controller 860),
the first processor further includes:
a surrounding target object recognition unit configured to recognize a plurality of surrounding target objects each of which is either a person or another robot present in the surroundings of the mobile robot (Hoffman, para.74, 83, 125, 129, sensor system 500 detects objects 114); and
a communication target selection reception unit configured to receive a selection operation, by the user, for one of the plurality of surrounding target objects (Hoffman, para.83, object 114 selection), and
the communication target selection reception unit determines whether the selection operation on one of the image parts of the plurality of surrounding target objects displayed on the first sub screen is performed (Hoffman, para.83, object 114 selection).
However, Hoffman does not teach a moving situation recognition unit configured to recognize a moving situation of the mobile robot, the moving situation including a moving speed of the mobile robot; and changes a content of the operation support information in accordance with the moving situation, of the mobile robot, recognized by the moving situation recognition unit. Katayama teaches a system for controlling robots wherein a moving situation recognition unit configured to recognize a moving situation of the mobile robot, the moving situation including a moving speed of the mobile robot and changes a content of the operation support information in accordance with the moving situation, of the mobile robot, recognized by the moving situation recognition unit (Katayama, para.177-180 state recognition unit 41 changes light status based on movement). It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to include Katayama’s teaching with Hoffman’s system in order to automatically perform operations based on movement.
Furthermore, the system of Hoffman and Katayama does not teach when the display mode is the stop display mode, the screen display controller displays, on the second sub screen, a communication window for supporting communication with a communication partner with whom a call is being made via the mobile robot; in the stop display mode, the screen display controller displays, on the second sub screen, at least one of the operation icon and the communication window for supporting communication with the one of the plurality of surrounding target objects, selection of which has been received by the communication target selection reception unit; the screen display controller displays, on either the first sub screen or communication window of the second sub screen, an image obtained by enlarging the image part of the surrounding target object. Jordan teaches a remote presence system wherein the screen displays, on a sub screen, a communication window for supporting communication with a communication partner with whom a call is being made via the mobile robot (Jordan, Fig.48, para.203-204, communication with other remote participants also utilizing robots); in the stop display mode, the screen display controller displays, on the second sub screen, at least one of the operation icon and the communication window for supporting communication with the one of the plurality of surrounding target objects, selection of which has been received by the communication target selection reception unit (Jordan, Fig.48, para.203-204, communication with other remote participants also utilizing robots), the screen display controller displays, on either the first sub screen or communication window of the second sub screen, an image obtained by enlarging the image part of the surrounding target object (Jordan, para.178, Fig.29, click-to-zoom 2925), (Jordan, Fig.48, para.204, select which practitioner to contact). It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to include Jordan’s teaching with the system of Hoffman and Katayama in order to communicate and coordinate with a team.
Additionally, the system of Hoffman, Katayama and Jordan does not teach to set the surrounding target object corresponding to the image part on which the selection operation is performed as a communication target, and the first processor further includes a communication controller configured to enable the communication with only the surrounding target object that has been set as the communication target. Watanuki teaches a robot system wherein a surrounding target object in which a selection operation is performed is set as a communication target, and a communication controller is configured to enable the communication with only the surrounding target object that has been set as the communication target (Watanuki, para.19-20, 42, 46-47, 65, 77, target detected and chosen for communication based on liking). It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to include Watanuki’s teaching with the system of Hoffman, Katayama and Jordan in order to communicate with preferred targets.
As per claim 3, the system of Hoffman, Katayama, Jordan and Watanuki teaches the remote operation system according to claim 1, wherein in the stop display mode, the screen display controller displays, on the second sub screen, a list of identification information of the plurality of surrounding target objects, and the communication target selection reception unit receives the selection of the surrounding target object in response to the selection operation for the list of the identification information that has been displayed (Jordan, Fig.48, list of target IDs; para.204).
As per claim 6, the system of Hoffman, Katayama, Jordan and Watanuki teaches the remote operation system according to claim 1, wherein the surrounding target object recognition unit recognizes either the surrounding target object registered beforehand or the surrounding target object present within a predetermined distance from the mobile robot, from the plurality of surrounding target objects present in the surroundings of the mobile robot (Hoffman, para.74, 83, 125, 129, sensor system 500).
As per claim 8, the system of Hoffman, Katayama, Jordan and Watanuki teaches the remote operation system according to claim 1, wherein in the movement display mode, the screen display controller displays, on the first sub screen, an image part of a first image-capturing range in the captured image by the camera (Hoffman, Fig.8A, camera view 810a; para.119), and also displays, on the second sub screen, an image part of a second image-capturing range different from the first image-capturing range as the operation support information (Hoffman, Fig.8A, camera views 810b-d; para.119).
As per claim 10, the system of Hoffman, Katayama, Jordan and Watanuki teaches the remote operation system according to claim 1, wherein in the movement display mode, the screen display controller displays, on the second sub screen, a map of a movement area of the mobile robot as the operation support information (Hoffman, para.121; Fig.8A, map view 828).
As per claim 11, the system of Hoffman, Katayama, Jordan and Watanuki teaches the remote operation system according to claim 10, wherein in the movement display mode, the screen display controller displays, on the second sub screen, area additional information as the operation support information, the area additional information being information regarding at least one of a sightseeing spot present in the movement area, a facility present in the movement area, and an event performed in the movement area (Johnson, Fig.7, para.119, healthcare facility and rooms).
As per claim 12, the system of Hoffman, Katayama, Jordan and Watanuki teaches the remote operation system according to claim 11, wherein when the selection operation for a specific location displayed on the map is performed, the screen display controller displays, on the second sub screen, the area additional information associated with the specific location as the operation support information (Johnson, Fig.7, para.119, healthcare facility and rooms).
As per claim 13, the system of Hoffman, Katayama, Jordan and Watanuki teaches the remote operation system according to claim 11, wherein the first processor further includes a current location recognition unit configured to recognize a current location of the mobile robot (Hoffman, para.121, Fig.8A, current location icon 834), wherein in the movement display mode, when the current location recognition unit recognizes that the mobile robot is located in a registered area registered as a sightseeing area, the screen display controller displays, on the second sub screen, the area additional information associated with the registered area as the operation support information (Johnson, Fig.7, para.119, healthcare facility and rooms).
As per claim 14, the system of Hoffman, Katayama, Jordan and Watanuki teaches the remote operation system according to claim 11, wherein the first processor further includes a moving speed recognition unit configured to recognize a moving speed of the mobile robot (Jordan, para.167, Fig.27C, length of vector), wherein when the moving speed of the mobile robot is equal to or lower than a predetermined low-speed determination speed, the screen display controller displays, on the second sub screen, the area additional information as the operation support information (Jordan, para.167, Fig.27C, grid 2760 displays descriptive terms like slow, medium, fast).
As per claim 15, the system of Hoffman, Katayama, Jordan and Watanuki teaches the remote operation system according to claim 1, wherein the first processor further includes: a current location recognition unit configured to recognize a current location of the mobile robot (Hoffman, para.121, Fig.8A, current location icon 834); a destination recognition unit configured to recognize a destination of the mobile robot (Hoffman, para.121, Fig.8A, retrotraverse); and a movement route recognition unit configured to recognize a movement route from the current location to the destination (Hoffman, para.121, Fig.8A, waypoints), wherein in the movement display mode, the screen display controller displays, on the second sub screen, navigation information for navigating a movement along the movement route from the current location to the destination as the operation support information (Hoffman, para.121, Fig.8A, waypoints).
As per claim 16, the system of Hoffman, Katayama, Jordan and Watanuki teaches the remote operation system according to claim 15, wherein the first processor further includes a moving speed recognition unit configured to recognize a moving speed of the mobile robot (Jordan, para.167, Fig.27C, length of vector), wherein when the moving speed of the mobile robot is equal to or higher than a predetermined high-speed determination speed, the screen display controller displays the navigation information on the second sub screen (Jordan, para.167, Fig.27C, grid 2760 displays descriptive terms like slow, medium, fast).
As per claim 17, the system of Hoffman, Katayama, Jordan and Watanuki teaches the remote operation system according to claim 1, wherein the mobile robot includes a robot display unit (Hoffman, para.69, Fig.3A, display 370), the first processor further includes a user face image display controller configured to, in the movement display mode, prohibit a face image of the user of the operation terminal from being displayed on the robot display unit, or configured to switch a display on the robot display unit to a display of a predetermined substitute icon (Jordan, para.79, 151, avatars 2091-2094).
As per claim 20, the system of Hoffman, Katayama, Jordan and Watanuki teaches the remote operation system according to claim 1, wherein the first processor further includes a display layout change unit configured to change a display layout of the operation screen in response to an operation on the operation terminal by the user (Hoffman, para.96, settings).
Claims 22-23 are similar in scope to claim 1, and are therefore rejected under similar rationale.
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over Hoffman et al. (“Hoffman”, US 2015/0190925), Katayama et al. (“Katayama”, US 2020/0301433), Jordan et al. (“Jordan”, US 2015/0077502) and Watanuki (US 2019/0389071) in view of Lockwood et al. (“Lockwood”, US 10,564,638).
As per claim 9, the system of Hoffman, Katayama, Jordan and Watanuki teaches the remote operation system according to claim 8, wherein the first processor further includes an obstacle recognition unit configured to recognize an obstacle present in the surroundings of the mobile robot (Hoffman, para.125, ODOA sensors). However, the system of Hoffman, Katayama, Jordan and Watanuki does not teach wherein in the movement display mode, the screen display controller displays, on the second sub screen, the image part of the second image-capturing range including the obstacle by highlighting an image part of the obstacle. Lockwood teaches a remote control system wherein in a movement display mode, the screen display controller displays, on a sub screen, an image part of an image-capturing range including the obstacle by highlighting an image part of the obstacle (Lockwood, Fig.6A, subscreen 628; col.20, lines 7-23; col.29, lines 3-24). It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to include Lockwood’s teaching with the system of Hoffman, Katayama, Jordan and Watanuki in order to zoom in on the obstacle.
Claim 19 is rejected under 35 U.S.C. 103 as being unpatentable over Hoffman et al. (“Hoffman”, US 2015/0190925), Katayama et al. (“Katayama”, US 2020/0301433), Jordan et al. (“Jordan”, US 2015/0077502) and Watanuki (US 2019/0389071) in view of Powers et al. (“Powers”, US 2012/0072052).
As per claim 19, the system of Hoffman, Katayama, Jordan and Watanuki teaches the remote operation system according to claim 1, however does not teach wherein the first processor further includes an indoor and outdoor location recognition unit configured to recognize whether the mobile robot is present indoors or outdoors, wherein the display mode switching unit switches the display mode in accordance with whether the mobile robot is present indoors or outdoors. Powers teaches a remote control interface that includes an indoor and outdoor location recognition unit configured to recognize whether the mobile robot is present indoors or outdoors (Powers, para.49, sensors); and a display mode switching unit configured to switch the display mode in accordance with whether the mobile robot is present indoors or outdoors (Powers, para.51, 53, 65 navigational state switches). It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to include Powers’ teaching with the system of Hoffman, Katayama, Jordan and Watanuki in order to transition between environments.
Claim 21 is rejected under 35 U.S.C. 103 as being unpatentable over Hoffman et al. (“Hoffman”, US 2015/0190925), Katayama et al. (“Katayama”, US 2020/0301433), Jordan et al. (“Jordan”, US 2015/0077502) and Watanuki (US 2019/0389071) in view of Blake et al. (“Blake”, US 2011/0263293).
As per claim 21, the system of Hoffman, Katayama, Jordan and Watanuki teaches the remote operation system according to claim 1, wherein the terminal display unit is a touch panel (Hoffman, para.75, touch screen), however does not teach the screen display controller to display a first operation icon, for which a touch operation is disabled in accordance with the predetermined display mode, in a display mode different from a display mode of a second operation icon for which the touch operation is enabled, out of operation icons displayed on the operation screen. Blake teaches a mobile interface to display a first operation icon, for which a touch operation is disabled in accordance with the predetermined display mode, in a display mode different from a display mode of a second operation icon for which the touch operation is enabled, out of operation icons displayed on the operation screen (Blake, para.40, 44, functions disabled during drive mode). It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to include Blake’s teaching with the system of Hoffman, Katayama, Jordan and Watanuki in order to ensure safety.
Claim 24 is rejected under 35 U.S.C. 103 as being unpatentable over Hoffman et al. (“Hoffman”, US 2015/0190925), Katayama et al. (“Katayama”, US 2020/0301433), Jordan et al. (“Jordan”, US 2015/0077502) and Watanuki (US 2019/0389071) in view of Nimri et al. (“Nimri”, US 2017/0099461) and further in view of Vu et al. (“Vu”, US 2007/0192910).
As per claim 24, the system of Hoffman, Katayama, Jordan and Watanuki teaches the remote operation system according to claim 1, however does not teach wherein the first processor further includes a camera image-capturing range controller configured to move the mobile robot so that all persons and other robots present in surroundings of the mobile robot are included in an image-capturing range of the camera; and an in-motion object recognition unit configured to recognize an object in motion in the surroundings of the mobile robot, in the stop display mode, the camera image-capturing range controller moves the mobile robot to adjust the image-capturing range of the camera so as to include image parts of all persons and another robot, the screen display controller displays, on the first sub screen, a first monitor image where the image parts of a range which is set to include the image parts of all persons and another robot are extracted from the captured image and included, and when the in-motion object recognition unit recognizes the object in motion present in the surroundings of the mobile robot, the screen display controller extracts an image part of a range including an image part of the object in motion, approaching the mobile robot, from the captured image, then, displays, on the first sub screen, a second monitor image where the extracted image part of the range including the image part of the object in motion is enlarged, instead of the first monitor image that does not include the image part of the object in motion.
Nimri teaches a video conferencing system that includes a camera image-capturing range controller configured to move the camera so that all persons present in surroundings of the camera are included in an image-capturing range of the camera (Nimri, para.27, 139, 156-163, 201, panoramic view includes all individuals); and an in-motion object recognition unit configured to recognize an object in motion in the surroundings of the camera (Nimri, para.161-162, motion detection 705), in the stop display mode, the camera image-capturing range controller moves the camera to adjust the image-capturing range of the camera so as to include image parts of all persons, the screen display controller displays a first monitor image where the image parts of a range which is set to include the image parts of all persons are extracted from the captured image and included (Nimri, para.27, 139, 156-163, 201, panoramic view extracts individuals while eliminating empty space) and when the in-motion object recognition unit recognizes the object in motion present in the surroundings of the mobile robot (Nimri, para.145, 161-162, motion detection 705), the screen display controller extracts an image part of a range including an image part of the object in motion from the captured image, then, displays, a second monitor image where the extracted image part of the range including the image part of the object in motion is enlarged, instead of the first monitor image that does not include the image part of the object in motion (Nimri, para.147, 153, 164, Fig.7E, active talker view displayed in zoomed view of detected motion of individual). It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to include Nimri‘s teaching with the system of Hoffman, Katayama, Jordan and Watanuki in order to focus on important regions.
Although Nimri teaches detecting the presence and motion of people (Nimri, para.75, 161), the system of Hoffman, Katayama, Jordan, Watanuki and Nimri does not explicitly teach when the in-motion object recognition unit recognizes the object in motion approaching the mobile robot. Vu teaches a system for controlling robots wherein an object recognition unit recognizes an object in motion approaching the mobile robot (Vu, para.145, detect people approaching too close). It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to include Vu’s teaching with the system of Hoffman, Katayama, Jordan, Watanuki and Nimri in order to avoid collisions.
Claim 25 is rejected under 35 U.S.C. 103 as being unpatentable over Hoffman et al. (“Hoffman”, US 2015/0190925), Katayama et al. (“Katayama”, US 2020/0301433), Jordan et al. (“Jordan”, US 2015/0077502) and Watanuki (US 2019/0389071) in view of Dahlquist et al. (“Dahlquist”, US 2005/0085157).
As per claim 25, the system of Hoffman, Katayama, Jordan and Watanuki teaches the remote operation system according to claim 1, wherein the mobile robot includes a head unit, a neck unit and a body unit (Hoffman, Fig.3A, head 360, neck 350, body 310; para.68), the neck unit is configured to perform a head-shaking motion that causes the head unit to swing vertically and horizontally with respect to the body unit (Hoffman, para.68, neck supports head providing panning/tilting).
However, the system of Hoffman, Katayama, Jordan and Watanuki does not teach when switching operation to a head-shaking mode is performed by the mode changeover switch, the display mode switching unit sets the display mode to the stop display mode corresponding to the situation in which the mobile robot is stopped. Dahlquist teaches a system that includes a switching operation to a head-shaking mode that sets the display mode to the stop display mode corresponding to the situation in which the mobile robot is stopped (Dahlquist, para.74, robot stopped so head can move). It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to include Dahlquist‘s teaching with the system of Hoffman, Katayama, Jordan and Watanuki in order to conserve energy to a single movement.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Inquiries
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SAJEDA MUHEBBULLAH whose telephone number is (571)272-4065. The examiner can normally be reached Mon-Tue/Thur-Fri 10am-8pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, William L Bashore can be reached on 571-272-4088. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/S.M./
Sajeda MuhebbullahExaminer, Art Unit 2174
/WILLIAM L BASHORE/ Supervisory Patent Examiner, Art Unit 2174