DETAILED ACTION
Status of Claims
Claims 38-57 are currently pending and have been examined in this application. This Final Rejection is in response to the amendment submitted on 8/11/2025.
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Response to Arguments and Amendments
Applicant’s arguments, regarding the claim objections with respect to claims 53 and 54 have been fully considered and are persuasive. The objections have been removed.
Applicant’s arguments, filed on 8/11/2025, with respect to the rejection of Claims 38-57 under 35 USC 102 and 35 USC 103 have been fully considered but not persuasive. Applicant’s arguments are directed towards the newly amended claim limitation(s), which are addressed below.
Regarding 102 and 103:
Issue: Applicant asserts that Yanase does not teach the inactive states described in the newly amended claims 38, 50, and 57.
Applicant Remarks:
“Critically, Yanase is silent with respect to either of the robot device or virtual
robot systems being in an inactive state. In this regard, Yanase does not contemplate the possibility of a human interaction system with a coordination system, in which:
an active social robot performs personality functions;
active virtual robot systems present an avatar representation of the social robot;
an inactive social robot does not perform personality functions; and
inactive virtual robot systems do not present the avatar representation. “
Examiner Reply:
Providing Claim 38 as a representative example, claim 38 was amended to include the following limitation: “wherein when the social robot is inactive, the social robot is configured not to undertake functions corresponding to a personality associated with the social robot and when the one or more virtual robot systems are inactive, the one or more virtual robot systems are configured to not present the avatar representation of the social robot.” Further Applicant’s specification recites as an example of inactivity, “an inactive physical social robot 11 can be configured to limit or entirely halt output functionality such as the illumination of lights, emission of sounds, or movement of parts”. Under BRI inactivity could include states ranging from a complete shutdown to a standby mode. Further, the newly applied citation from Yanase paragraph [0117] (see the 35 USC 102 rejection below) includes a state in which, “… the robot device 20 and the virtual robot 40 may be controlled such that only one of the robot device 20 and the virtual robot 40 acts…” The examiner is interpreting the ability to control the social robot and the virtual robot such that only one of the devices acts, as the design intent of the most recently added amendment to claim 38.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 38, 40-44, 47, 50-52 and 54-57 are rejected under 35 U.S.C. 102 (a) (1) as being anticipated by Yanase (US 20200290198 A1)
Claim 38:
Yanase teaches the following limitations:
A human interaction system for interaction by a user comprising social systems comprising at least a social robot and one or more virtual robot systems, and (Yanase - [0020] In an embodiment, an entertainment system in which a user can interact with a robot device in the real world and see how a virtual robot in the virtual world does is provided. ; [0021] FIG. 1 illustrates a schematic configuration of an entertainment system 1 according to the embodiment. The entertainment system 1 includes a robot device 20, a server device 10, and a terminal device 12. The robot device 20 is capable of acting in an autonomous action mode in the real world. The server device 10 causes a virtual robot 40, which is associated with the robot device 20, to act in the virtual world. The terminal device 12 is capable of displaying an image of the virtual world in which the virtual robot 40 acts. The robot device 20 and the terminal device 12 are connected to the server device 10 via an access point (AP) 3 and a network 2 such as the Internet. The server device 10 provides the image of the virtual world to the terminal device 12.) a coordination system, wherein: the social robot is controlled by a robot processing system and (Yanase - [0008] … The robot device according to this aspect includes: a request acquisition section configured to acquire a request from a virtual robot associated with the robot device, the virtual robot being in a virtual world; a mode setting section configured to, when the request acquisition section acquires the request from the virtual robot, set a collaboration action mode in which collaboration is made with the virtual robot; an action determination section configured to, when the collaboration action mode is set, determine an action corresponding to the request from the virtual robot; and a control section configured to control a motion of the robot device such that the robot device takes the action determined by the action determination section. [0026] ... The processing section 30 causes the robot device 20 to autonomously act on the basis of the state information and/or the function information, and also updates the state information and/or the function information according to an action taken by the robot device 20 and/or an external input into the robot device 20 in the real world.) is configured to provide interaction with a user, said interaction including output means and input means, (Yanase - [0044] FIG. 6 illustrates a configuration for controlling the motion of the robot device 20. The robot device 20 includes, as a motion control configuration, an input/output interface 100, a state management section 120, an action management section 130, and the storage section 28. The functions of the input/output interface 100, the state management section 120, and the action management section 130 are realized by the processing section 30. ; [0045] The input/output interface 100 is an interface that transmits and receives various data and messages such as requests and responses to and from the server device 10, and includes an update information transmission section 102, an update information acquisition section 104, an application acquisition section 106, a request acquisition section 108, a response transmission section 110, and a motion data acquisition section 112. The state management section 120 manages the states and functions of the robot device 20 and includes a state determination section 122, an update section 124, and a synchronization section 126. The action management section 130 manages the action of the robot device 20 and includes a mode setting section 132, an action determination section 134, and a control section 136.) the one or more virtual robot systems are configured to controllably present an avatar representation of the social robot, and further configured to receive inputs, (Yanase - [0048] FIG. 7 illustrates a configuration of the server device 10. The server device 10 includes a communication section 200, an input/output interface 210, a state management section 230, a virtual robot control section 240, an image processing section 250, and a storage device 260. The input/output interface 210 is an interface that transmits and receives various data and messages such as requests and responses to and from the robot device 20 and the terminal device 12, and includes an update information transmission section 212, an update information acquisition section 214, a request transmission section 216, a response acquisition section 218, a motion data transmission section 220, an image request acquisition section 222, and an image distribution section 224. ; [0056] The virtual robot control section 240 causes virtual robots of a plurality of users to autonomously act in the virtual space on the basis of the action program stored in the action program storage section 280. At this time, the action of each virtual robot reflects the state information (state values) stored in the state information storage section 264 and the function information (function values) stored in the function information storage section 266.) the coordination system is configured to coordinate operation of the social robot and the one or more virtual robots such that, at any one time, either the social robot or one of the virtual robot systems is active, (Yanase - [0008] … The robot device according to this aspect includes: a request acquisition section configured to acquire a request from a virtual robot associated with the robot device, the virtual robot being in a virtual world; a mode setting section configured to, when the request acquisition section acquires the request from the virtual robot, set a collaboration action mode in which collaboration is made with the virtual robot; an action determination section configured to, when the collaboration action mode is set, determine an action corresponding to the request from the virtual robot; and a control section configured to control a motion of the robot device such that the robot device takes the action determined by the action determination section.; [0080] As described above, the state information of each of the robot device 20 and the virtual robot 40 is managed synchronously in the robot device 20 and the server device 10 through the collaboration between the state management section 120 in the robot device 20 and the state management section 230 in the server device 10. In the robot device 20 in the autonomous action mode, the action determination section 134 determines the action of the robot device 20 according to the state information. Further, in the server device 10, the virtual robot control section 240 causes the virtual robot 40 to act according to the state information. Accordingly, the robot device 20 and the virtual robot 40 autonomously act in their respective worlds under the same emotion, personality, and action characteristics.) wherein when the social robot is inactive, the social robot is configured not to undertake functions corresponding to a personality associated with the social robot and when the one or more virtual robot systems are inactive, the one or more virtual robot systems are configured to not present the avatar representation of the social robot. (Yanase - [0117] Further, while it is assumed in the embodiment that the virtual robot 40 always acts, the robot device 20 and the virtual robot 40 may be controlled such that only one of the robot device 20 and the virtual robot 40 acts. That is, the virtual robot control section 240 may perform controls so as not to allow the virtual robot 40 to act in the virtual world during the time between when the main power of the robot device 20 is turned on and when the main power thereof is turned off, and so as to allow the virtual robot 40 to act in the virtual world only during the time between when the main power of the robot device 20 is turned off and when the main power thereof is turned on.) such that, in operation, a user perceives the robot personality associated with the social robot and the avatar as associated with the active social robot or virtual robot system. (Yanase - [0026] The robot device 20 has state information and function information. The state information represents the personality, emotion, and the like of the robot individual. The function information represents functions of the robot individual. … ; [0037] FIG. 4 illustrates an example of the virtual world. The virtual robot 40 is a CG character associated with the robot device 20 owned by the user and has the appearance similar to that of the robot device 20. The server device 10 creates the virtual world in which a plurality of virtual robots can act and communicate with each other in a meeting with each other. The virtual robot 40 acquires new functions by having various experiences in the virtual world and updates state information (state values) that represents the personality and emotion. ; [0038] FIG. 5 illustrates an example of the state information. Hereinafter, the robot device 20 and the virtual robot 40 may be occasionally collectively referred to as “robot.” The state information according to the embodiment includes information regarding robot internal states of “emotion,” “personality,” and “action characteristics.” “Emotion” includes parameters of “anger,” “sadness,” “surprise,” “fear,” and “happiness.” “Personality” includes parameters of “active,” “emotional,” “strong-mindedness,” “vitality,” “aggressive,” “curiosity,” and “kindness.” … )
Claim 40:
Yanase teaches the following limitations:
The system according to claim 38, wherein the coordination system is configured to: determine a location of the user and to determine a corresponding social system to the location of the user; and communicate a message to the corresponding social system configuring it as active.
(Yanase - [0052] The three-dimensional virtual space may be formed on the basis of a real world map. The virtual robot may be initially positioned at the address in the virtual world that corresponds to the user's address. For example, in a case where the user's address is in Japan, the user's virtual robot is initially positioned at a location corresponding to Japan in the virtual world. Further, in a case where the user's address is in the United States, the user's virtual robot is initially positioned at a location corresponding to the United States in the virtual world. As described above, the “vitality” parameter represents the level of vitality to explore a new place. Therefore, a virtual robot with a high state value of the “vitality” parameter tends to move to another country on a plane. By contrast, a virtual robot with a low state value tends to act only around the location initially positioned. ; [0113] When the request acquisition section 108 acquires the request from the virtual robot 40, the mode setting section 132 sets the motion mode of the robot device 20 to the collaboration action mode. When the collaboration action mode is set, the action determination section 134 determines actions corresponding to the request from the virtual robot 40. Here, the request includes information that indicates making the motions of the virtual robot 40 and the robot device 20 synchronized with each other. In response to the request, the action determination section 134 instructs the control section 136 to perform the actions according to motion data transmitted from the server device 10. )
Claim 41:
Yanase teaches the following limitations:
The system according to claim 40, wherein the coordination system is further configured to in response to determining the corresponding social system to the location of the user, communicate a message to the one or more other social systems configuring each as inactive.
(Yanase - [0052] The three-dimensional virtual space may be formed on the basis of a real world map. The virtual robot may be initially positioned at the address in the virtual world that corresponds to the user's address. For example, in a case where the user's address is in Japan, the user's virtual robot is initially positioned at a location corresponding to Japan in the virtual world. Further, in a case where the user's address is in the United States, the user's virtual robot is initially positioned at a location corresponding to the United States in the virtual world. As described above, the “vitality” parameter represents the level of vitality to explore a new place. Therefore, a virtual robot with a high state value of the “vitality” parameter tends to move to another country on a plane. By contrast, a virtual robot with a low state value tends to act only around the location initially positioned. ; [0117] Further, while it is assumed in the embodiment that the virtual robot 40 always acts, the robot device 20 and the virtual robot 40 may be controlled such that only one of the robot device 20 and the virtual robot 40 acts. That is, the virtual robot control section 240 may perform controls so as not to allow the virtual robot 40 to act in the virtual world during the time between when the main power of the robot device 20 is turned on and when the main power thereof is turned off, and so as to allow the virtual robot 40 to act in the virtual world only during the time between when the main power of the robot device 20 is turned off and when the main power thereof is turned on.)
Claim 42:
Yanase teaches the following limitations:
The system according to claim 38, wherein the one or more virtual robot systems are configured to animate the avatar, and wherein at least one animation is equivalent to a movement of the social robot.
(Yanase - [0028] The server device 10 creates the virtual robot 40 as a virtual object model having the same appearance as the robot device 20 in the real world. This allows the user to see, through the terminal device 12, how the virtual robot 40 having the same appearance as the robot device 20 acts in the virtual world formed by three-dimensional CG (computer graphics). It is noted that the server device 10 preferably matches physical functions of the virtual robot 40 with physical functions of the robot device 20.)
Claim 43:
Yanase teaches the following limitations:
The system according to claim 38, wherein the robot processing system is configured to control, at least in part, the operation of an active virtual robot system.
(Yanase - [0009] Still another aspect of the present invention is a server device that causes a virtual robot associated with a robot device in a real world to act in a virtual world. The server device according to this aspect includes: a request transmission section configured to transmit a request from the virtual robot to the robot device; a response acquisition section configured to acquire a user response from the robot device; and a virtual robot control section configured to reflect the user response in a motion of the virtual robot. ; [0053] Each of elements described as functional blocks that perform various processes in FIG. 7 can be configured by a circuit block, a memory, or other large-scale integration (LSI) in terms of hardware, and is realized by a program or the like loaded into a memory in terms of software. Therefore, it will be understood by those skilled in the art that these functional blocks can be realized in various forms with hardware alone, software alone, or a combination thereof, and the functional blocks are not limited to any one of these forms.)`
Claim 44:
Yanase teaches the following limitations:
The system according to claim 43, wherein the active virtual robot system is configured to interpret commands received from the robot processing system and adapt said commands for display on a display of the virtual robot system.
[0021] FIG. 1 illustrates a schematic configuration of an entertainment system 1 according to the embodiment. The entertainment system 1 includes a robot device 20, a server device 10, and a terminal device 12. The robot device 20 is capable of acting in an autonomous action mode in the real world. The server device 10 causes a virtual robot 40, which is associated with the robot device 20, to act in the virtual world. The terminal device 12 is capable of displaying an image of the virtual world in which the virtual robot 40 acts. The robot device 20 and the terminal device 12 are connected to the server device 10 via an access point (AP) 3 and a network 2 such as the Internet. The server device 10 provides the image of the virtual world to the terminal device 12.
Claim 47:
Yanase teaches the following limitations:
The system according to claim 38, further comprising one or more interaction devices in data communication with the social robot and/or the one or more virtual robot systems, the interaction devices enabling the user to provide inputs and receive outputs.
(Yanase - [0048] FIG. 7 illustrates a configuration of the server device 10. The server device 10 includes a communication section 200, an input/output interface 210, a state management section 230, a virtual robot control section 240, an image processing section 250, and a storage device 260. The input/output interface 210 is an interface that transmits and receives various data and messages such as requests and responses to and from the robot device 20 and the terminal device 12, and includes an update information transmission section 212, an update information acquisition section 214, a request transmission section 216, a response acquisition section 218, a motion data transmission section 220, an image request acquisition section 222, and an image distribution section 224.)
Claim 50:
Yanase teaches the following limitations:
A human interaction system for interaction by a user comprising social systems comprising at least a social robot and one or more virtual robot systems, (Yanase - [0020] In an embodiment, an entertainment system in which a user can interact with a robot device in the real world and see how a virtual robot in the virtual world does is provided. ; [0021] FIG. 1 illustrates a schematic configuration of an entertainment system 1 according to the embodiment. The entertainment system 1 includes a robot device 20, a server device 10, and a terminal device 12. The robot device 20 is capable of acting in an autonomous action mode in the real world. The server device 10 causes a virtual robot 40, which is associated with the robot device 20, to act in the virtual world. The terminal device 12 is capable of displaying an image of the virtual world in which the virtual robot 40 acts. The robot device 20 and the terminal device 12 are connected to the server device 10 via an access point (AP) 3 and a network 2 such as the Internet. The server device 10 provides the image of the virtual world to the terminal device 12.) wherein: the social robot is controlled by a robot processing system and (Yanase - [0026] ... The processing section 30 causes the robot device 20 to autonomously act on the basis of the state information and/or the function information, and also updates the state information and/or the function information according to an action taken by the robot device 20 and/or an external input into the robot device 20 in the real world.) is configured to provide interaction with a control user, said interaction including output means and input means, the social robot being configured to be active or inactive (Yanase - [0044] FIG. 6 illustrates a configuration for controlling the motion of the robot device 20. The robot device 20 includes, as a motion control configuration, an input/output interface 100, a state management section 120, an action management section 130, and the storage section 28. The functions of the input/output interface 100, the state management section 120, and the action management section 130 are realized by the processing section 30. ; [0045] The input/output interface 100 is an interface that transmits and receives various data and messages such as requests and responses to and from the server device 10, and includes an update information transmission section 102, an update information acquisition section 104, an application acquisition section 106, a request acquisition section 108, a response transmission section 110, and a motion data acquisition section 112. The state management section 120 manages the states and functions of the robot device 20 and includes a state determination section 122, an update section 124, and a synchronization section 126. The action management section 130 manages the action of the robot device 20 and includes a mode setting section 132, an action determination section 134, and a control section 136.)
the one or more one or more virtual robot systems are configured to controllably present an avatar representation of the social robot, and further configured to receive inputs, (Yanase - [0048] FIG. 7 illustrates a configuration of the server device 10. The server device 10 includes a communication section 200, an input/output interface 210, a state management section 230, a virtual robot control section 240, an image processing section 250, and a storage device 260. The input/output interface 210 is an interface that transmits and receives various data and messages such as requests and responses to and from the robot device 20 and the terminal device 12, and includes an update information transmission section 212, an update information acquisition section 214, a request transmission section 216, a response acquisition section 218, a motion data transmission section 220, an image request acquisition section 222, and an image distribution section 224. ; [0056] The virtual robot control section 240 causes virtual robots of a plurality of users to autonomously act in the virtual space on the basis of the action program stored in the action program storage section 280. At this time, the action of each virtual robot reflects the state information (state values) stored in the state information storage section 264 and the function information (function values) stored in the function information storage section 266.) wherein the, or each, virtual robot system is associated with a user, each of the one or more virtual robot systems being configured to be active or inactive. (Yanase - [0027] … After the server device 10 registers the robot device 20 in association with the user, the server device 10 creates the virtual robot 40 corresponding to the robot device 20 and causes the virtual robot 40 to act in the virtual world. Through the terminal device 12 such as a smartphone or a tablet, the user can see how the virtual world in which the virtual robot acts looks like.) such that, in operation, the, or each, user perceives a robot personality associated with the social robot and the avatar as associated with the active social robot or virtual robot system (Yanase - [0026] The robot device 20 has state information and function information. The state information represents the personality, emotion, and the like of the robot individual. The function information represents functions of the robot individual. … ; [0037] FIG. 4 illustrates an example of the virtual world. The virtual robot 40 is a CG character associated with the robot device 20 owned by the user and has the appearance similar to that of the robot device 20. The server device 10 creates the virtual world in which a plurality of virtual robots can act and communicate with each other in a meeting with each other. The virtual robot 40 acquires new functions by having various experiences in the virtual world and updates state information (state values) that represents the personality and emotion. ; [0038] FIG. 5 illustrates an example of the state information. Hereinafter, the robot device 20 and the virtual robot 40 may be occasionally collectively referred to as “robot.” The state information according to the embodiment includes information regarding robot internal states of “emotion,” “personality,” and “action characteristics.” “Emotion” includes parameters of “anger,” “sadness,” “surprise,” “fear,” and “happiness.” “Personality” includes parameters of “active,” “emotional,” “strong-mindedness,” “vitality,” “aggressive,” “curiosity,” and “kindness.” … ) wherein when the social robot is inactive, the social robot is configured not to undertake functions corresponding to a personality associated with the social robot and when the one or more virtual robot systems are inactive, the one or more virtual robot systems are configured to not present the avatar representation of the social robot. (Yanase - [0117] Further, while it is assumed in the embodiment that the virtual robot 40 always acts, the robot device 20 and the virtual robot 40 may be controlled such that only one of the robot device 20 and the virtual robot 40 acts. That is, the virtual robot control section 240 may perform controls so as not to allow the virtual robot 40 to act in the virtual world during the time between when the main power of the robot device 20 is turned on and when the main power thereof is turned off, and so as to allow the virtual robot 40 to act in the virtual world only during the time between when the main power of the robot device 20 is turned off and when the main power thereof is turned on.)
Claim 51:
Yanase teaches the following limitations:
The system according to claim 50, wherein the one or more virtual robot systems are configured to animate the avatar, and wherein at least one animation is equivalent to a movement of the social robot and (Yanase - [0028] The server device 10 creates the virtual robot 40 as a virtual object model having the same appearance as the robot device 20 in the real world. This allows the user to see, through the terminal device 12, how the virtual robot 40 having the same appearance as the robot device 20 acts in the virtual world formed by three-dimensional CG (computer graphics). It is noted that the server device 10 preferably matches physical functions of the virtual robot 40 with physical functions of the robot device 20.) at least one animation is not equivalent to a movement of the social robot. (Yanase - [0100] … However, since the virtual robot 40 acts independently from the robot device 20, there is a possibility that when the virtual robot 40 arrives at the gate 1, the main power of the robot device 20 is off. …)
Claim 52:
Yanase teaches the following limitations:
The system according to claim 50, further comprising one or more interaction devices in data communication with the social robot and/or virtual robot system(s), the interaction devices enabling the user to provide inputs and receive outputs.
(Yanase - [0045] The input/output interface 100 is an interface that transmits and receives various data and messages such as requests and responses to and from the server device 10, and includes an update information transmission section 102, an update information acquisition section 104, an application acquisition section 106, a request acquisition section 108, a response transmission section 110, and a motion data acquisition section 112. The state management section 120 manages the states and functions of the robot device 20 and includes a state determination section 122, an update section 124, and a synchronization section 126. The action management section 130 manages the action of the robot device 20 and includes a mode setting section 132, an action determination section 134, and a control section 136.)
Claim 54:
Yanase teaches the following limitations:
The system according to claim 50, wherein the social robot is configured to receive voice commands from the control user, (Yanase - [0025] A microphone 22 collects ambient voice and converts the ambient voice into a voice signal.; [0094] This suggestion is transmitted to the server device 10 when the user speaks to the robot device 20 and the robot device 20 transmits the contents of the user's utterance resulting from language analysis to the server device 10. The server device 10 determines whether or not the virtual robot 40 accepts the user's suggestion according to the state information stored in the state information storage section 264 and/or the function information stored in the function information storage section) wherein at least one voice command corresponds to a request for information from a particular virtual robot system, and wherein the social robot is further configured to communicate said command to said particular virtual robot system. (Yanase - [0097] A mission 1 is set at the gate 1. As the mission 1 for unlocking the key, the user passes a key to the virtual robot 40. The mission 1 is cleared when the user causes the camera 24 of the robot device 20 to capture an image of a key (a car key, a house key, or any other key is acceptable) owned by the user. ; [0098] When the virtual robot 40 arrives at the gate 1, the virtual robot control section 240 generates a request from the virtual robot 40 to unlock the key of the gate 1. This request is to capture an image of the user's key with the camera 24. The virtual robot control section 240 generates voice data of the virtual robot 40 “I want a key, so please show the key to the camera,” and the request transmission section 216 transmits the request from the virtual robot 40 to the robot device 20. Since this request is to request the cooperation of the user, the request may be referred to as “intervention request.” The transmitted request includes mission information indicating capturing of an image of the user's key with the camera 24, and voice data “I want a key, so please show the key to the camera.”)
Claim 55:
Yanase teaches the following limitations:
The system according to claim 54, wherein at least one virtual robot system is further configured to: receive a directed command; undertake an associated action; and
communicate a response to the social robot.
(Yanase - [0097] A mission 1 is set at the gate 1. As the mission 1 for unlocking the key, the user passes a key to the virtual robot 40. The mission 1 is cleared when the user causes the camera 24 of the robot device 20 to capture an image of a key (a car key, a house key, or any other key is acceptable) owned by the user. ; [0098] When the virtual robot 40 arrives at the gate 1, the virtual robot control section 240 generates a request from the virtual robot 40 to unlock the key of the gate 1. This request is to capture an image of the user's key with the camera 24. The virtual robot control section 240 generates voice data of the virtual robot 40 “I want a key, so please show the key to the camera,” and the request transmission section 216 transmits the request from the virtual robot 40 to the robot device 20. Since this request is to request the cooperation of the user, the request may be referred to as “intervention request.” The transmitted request includes mission information indicating capturing of an image of the user's key with the camera 24, and voice data “I want a key, so please show the key to the camera.” ; [0099] In the robot device 20, the request acquisition section 108 acquires the request from the virtual robot 40. At this time, the robot device 20 acts in the autonomous action mode. When the request acquisition section 108 acquires the request from the virtual robot 40, the mode setting section 132 switches the autonomous action mode of the robot device 20 to the collaboration action mode in which collaboration is made with the virtual robot 40. After the switch to the collaboration action mode, the mode setting section 132 sends a notification of the switch to the action determination section 134 and the control section 136. Accordingly, the action determination section 134 temporarily stops any action determination in the autonomous action mode.)
Claim 56:
Yanase teaches the following limitations:
The system according to claim 55, wherein at least one associate action comprises obtaining a result from an associated auxiliary device in communication with the associated virtual robot system.
(Yanase - [0027] The user participates in the entertainment system 1 by registering the robot device 20, which is owned by the user, in the server device 10. After the server device 10 registers the robot device 20 in association with the user, the server device 10 creates the virtual robot 40 corresponding to the robot device 20 and causes the virtual robot 40 to act in the virtual world. Through the terminal device 12 such as a smartphone or a tablet, the user can see how the virtual world in which the virtual robot acts looks like. ; [0028] The server device 10 creates the virtual robot 40 as a virtual object model having the same appearance as the robot device 20 in the real world. This allows the user to see, through the terminal device 12, how the virtual robot 40 having the same appearance as the robot device 20 acts in the virtual world formed by three-dimensional CG (computer graphics). It is noted that the server device 10 preferably matches physical functions of the virtual robot 40 with physical functions of the robot device 20.)
Claim 57:
Yanase teaches the following limitations:
A human interaction method for allowing interaction by a user with a social system comprising at least a social robot and one or more virtual robot systems, comprising: (Yanase - [0020] In an embodiment, an entertainment system in which a user can interact with a robot device in the real world and see how a virtual robot in the virtual world does is provided. ; [0021] FIG. 1 illustrates a schematic configuration of an entertainment system 1 according to the embodiment. The entertainment system 1 includes a robot device 20, a server device 10, and a terminal device 12. The robot device 20 is capable of acting in an autonomous action mode in the real world. The server device 10 causes a virtual robot 40, which is associated with the robot device 20, to act in the virtual world. The terminal device 12 is capable of displaying an image of the virtual world in which the virtual robot 40 acts. The robot device 20 and the terminal device 12 are connected to the server device 10 via an access point (AP) 3 and a network 2 such as the Internet. The server device 10 provides the image of the virtual world to the terminal device 12.) controlling the social robot to provide interaction with a user, said interaction including output means and input means, (Yanase - [0044] FIG. 6 illustrates a configuration for controlling the motion of the robot device 20. The robot device 20 includes, as a motion control configuration, an input/output interface 100, a state management section 120, an action management section 130, and the storage section 28. The functions of the input/output interface 100, the state management section 120, and the action management section 130 are realized by the processing section 30. ; [0045] The input/output interface 100 is an interface that transmits and receives various data and messages such as requests and responses to and from the server device 10, and includes an update information transmission section 102, an update information acquisition section 104, an application acquisition section 106, a request acquisition section 108, a response transmission section 110, and a motion data acquisition section 112. The state management section 120 manages the states and functions of the robot device 20 and includes a state determination section 122, an update section 124, and a synchronization section 126. The action management section 130 manages the action of the robot device 20 and includes a mode setting section 132, an action determination section 134, and a control section 136.) controllably presenting an avatar representation of the social robot (Yanase - [0048] FIG. 7 illustrates a configuration of the server device 10. The server device 10 includes a communication section 200, an input/output interface 210, a state management section 230, a virtual robot control section 240, an image processing section 250, and a storage device 260. The input/output interface 210 is an interface that transmits and receives various data and messages such as requests and responses to and from the robot device 20 and the terminal device 12, and includes an update information transmission section 212, an update information acquisition section 214, a request transmission section 216, a response acquisition section 218, a motion data transmission section 220, an image request acquisition section 222, and an image distribution section 224. ; [0056] The virtual robot control section 240 causes virtual robots of a plurality of users to autonomously act in the virtual space on the basis of the action program stored in the action program storage section 280. At this time, the action of each virtual robot reflects the state information (state values) stored in the state information storage section 264 and the function information (function values) stored in the function information storage section 266.) on one or more displays, such that when an avatar representation is displayed on a display it is active, and (Yanase - [0021] FIG. 1 illustrates a schematic configuration of an entertainment system 1 according to the embodiment. The entertainment system 1 includes a robot device 20, a server device 10, and a terminal device 12. The robot device 20 is capable of acting in an autonomous action mode in the real world. The server device 10 causes a virtual robot 40, which is associated with the robot device 20, to act in the virtual world. The terminal device 12 is capable of displaying an image of the virtual world in which the virtual robot 40 acts. The robot device 20 and the terminal device 12 are connected to the server device 10 via an access point (AP) 3 and a network 2 such as the Internet. The server device 10 provides the image of the virtual world to the terminal device 12.) coordinating operation of the social robot and the one or more virtual robots such that, at any one time, either the social robot or one of the virtual robot systems is active, (Yanase - [0008] … The robot device according to this aspect includes: a request acquisition section configured to acquire a request from a virtual robot associated with the robot device, the virtual robot being in a virtual world; a mode setting section configured to, when the request acquisition section acquires the request from the virtual robot, set a collaboration action mode in which collaboration is made with the virtual robot; an action determination section configured to, when the collaboration action mode is set, determine an action corresponding to the request from the virtual robot; and a control section configured to control a motion of the robot device such that the robot device takes the action determined by the action determination section. [0026] ... The processing section 30 causes the robot device 20 to autonomously act on the basis of the state information and/or the function information, and also updates the state information and/or the function information according to an action taken by the robot device 20 and/or an external input into the robot device 20 in the real world.) wherein when the social robot is inactive, the social robot is configured not to undertake functions corresponding to a personality associated with the social robot and when the one or more virtual robot systems are inactive, the one or more virtual robot systems are configured to not present the avatar representation of the social robot. (Yanase - [0117] Further, while it is assumed in the embodiment that the virtual robot 40 always acts, the robot device 20 and the virtual robot 40 may be controlled such that only one of the robot device 20 and the virtual robot 40 acts. That is, the virtual robot control section 240 may perform controls so as not to allow the virtual robot 40 to act in the virtual world during the time between when the main power of the robot device 20 is turned on and when the main power thereof is turned off, and so as to allow the virtual robot 40 to act in the virtual world only during the time between when the main power of the robot device 20 is turned off and when the main power thereof is turned on.) such that, in operation, a user perceives a robot personality associated with the social robot and the avatar as associated with the active social robot or virtual robot system.(Yanase - [0026] The robot device 20 has state information and function information. The state information represents the personality, emotion, and the like of the robot individual. The function information represents functions of the robot individual. … ; [0037] FIG. 4 illustrates an example of the virtual world. The virtual robot 40 is a CG character associated with the robot device 20 owned by the user and has the appearance similar to that of the robot device 20. The server device 10 creates the virtual world in which a plurality of virtual robots can act and communicate with each other in a meeting with each other. The virtual robot 40 acquires new functions by having various experiences in the virtual world and updates state information (state values) that represents the personality and emotion. ; [0038] FIG. 5 illustrates an example of the state information. Hereinafter, the robot device 20 and the virtual robot 40 may be occasionally collectively referred to as “robot.” The state information according to the embodiment includes information regarding robot internal states of “emotion,” “personality,” and “action characteristics.” “Emotion” includes parameters of “anger,” “sadness,” “surprise,” “fear,” and “happiness.” “Personality” includes parameters of “active,” “emotional,” “strong-mindedness,” “vitality,” “aggressive,” “curiosity,” and “kindness.” … )
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 39, 45, 46, and 49 are rejected under 35 U.S.C. 103 as being unpatentable over Yanase (US 20200290198 A1) as modified by Moon (US 20210046638 A1)
Claim 39:
Yanase teaches the following limitations:
The system according to claim 38, wherein the social robot comprises at least one of: one or more cameras and/or a microphone as input means; a speaker and/or
(Yanase - [0024] FIG. 3 illustrates an input/output system of the robot device 20. A processing section 30 is a main processor that processes and outputs various data such as voice data and sensor data, and instructions. The processing section 30 drives a driving mechanism 34 to move the robot device 20 and causes a speaker 36 to output voice . ... ; [0025] A microphone 22 collects ambient voice and converts the ambient voice into a voice signal. A camera 24 captures an image of the surroundings to acquire a captured image.)
a first portion, such as a head, (Yanase - [0023] FIG. 2 illustrates an example of the appearance of the robot device 20, which is a humanoid robot. Since the entertainment system 1 according to the embodiment aims to provide an environment in which the user interacts with the robot device 20, the robot device 20 is preferably formed in such a way as to make the user feel that the robot device 20 is a living body. Examples of such a form include humanoids and pet types. However, the shape of the robot device 20 is not particularly limited thereto. ; [0076] … For example, when the state determination section 122 detects the motion of stroking the head of the robot device 20 from the sensor data of the touch sensor provided on the head of the robot device 20, the state determination section 122 determines to increase the state value of the “happiness” parameter by one and sends a notification of the determination to the update section 124…) moveable with respect to (Yanase - [0024] ... When the motors are moved, the arms, legs, neck, and the like of the robot device 20 are moved accordingly. It is noted that the processing section 30 may drive the driving mechanism 34 and cause the speaker 36 to output voice on the basis of motion data generated by the server device 10.) a second portion, such as a body. (Yanase - [0022] … The robot device 20 is a humanoid or pet type robot with actuators provided at joint portions such as arms, legs, and a neck, and can take various actions including movement. ... ; [0023] FIG. 2 illustrates an example of the appearance of the robot device 20, which is a humanoid robot. Since the entertainment system 1 according to the embodiment aims to provide an environment in which the user interacts with the robot device 20, the robot device 20 is preferably formed in such a way as to make the user feel that the robot device 20 is a living body. Examples of such a form include humanoids and pet types. However, the shape of the robot device 20 is not particularly limited thereto.)
Yanase does not explicitly teach the following limitations, however Moon teaches:
one or more lights as output means; and
(Moon - [0083] The light output unit 156 may be implemented as a light source such as an LED. The processor 180 may represent the state of the robot 100a through the light output unit 156. In some embodiments, the light output unit 156 is an auxiliary output unit and may provide various types of information to the user along with the display 152 and/or the speaker 154. ; [0097] The robot 100a may output an expression based on the recognized emotion (S120). The processor 180 may control at least one of the display 152, the speaker 154, the light output unit 156 or the rotation unit 160 to output an emotional expression corresponding to the recognized emotion, thereby more emotionally delivering the content which is being output.)
Therefore, prior to the effective filing date of the claimed invention, it would have been
obvious to one of ordinary skill in the art to modify Yanase to include integrated lights as an output device as taught in Moon. The use of lights as an output device can be used to indicate a state of the robot or a particular robot event, which helps to engage and inform the end user of the robot.
Claim 45:
Yanase does not explicitly teach the following limitations, however Moon teaches:
The system according to claim 38, wherein at least one virtual robot system is configured with at least two predefined avatar appearances, and wherein one of said predefined avatar appearances is selected in dependence on an application.
(Moon - [0149] Referring to FIG. 11, the robot 100a may output an emotional expression corresponding to the recognized emotion. For example, if the recognized emotion corresponds to “sadness”, the processor 180 may output, via the display 152, an image indicating the facial expression 1100 corresponding to sadness. In addition, the processor 180 may output, via the speaker 154, speech 1101 corresponding to sadness. In addition, the processor 180 may control the rotation unit 160 to output motion 1102 of the robot corresponding to sadness. One or at least two of the facial expression 1100, the speech 1101 or the motion 1102 may be output.)
Therefore, prior to the effective filing date of the claimed invention, it would have been
obvious to one of ordinary skill in the art to modify Yanase to include multiple virtual appearances for the virtual robot as taught in Moon. Having multiple appearances for the robot can help to indicate a state of the robot, an emotion, personality trait or a particular interaction event which helps to engage and inform the end user of the robot.
Claim 46:
Yanase does not explicitly teach the following limitations, however Moon teaches:
The system according to claim 45, wherein one of the at least two predefined avatar appearances is a neutral appearance.
(Moon - [0109] Alternatively, the emotion recognized and output by the emotion recognizer 800 according to the embodiment of the present disclosure may further include a neutrality emotion class indicating a default emotional state in which six emotions do not occur, in addition to the emotion classes such as surprise, happiness, sadness, displeasure, anger, and fear. ; [0110] In this case, the emotion recognizer 800 may output any one emotion class selected from surprise, happiness, sadness, displeasure, anger, fear and neutrality as an emotion recognition result or output a probability value of each emotion class such as surprise x %, happiness x %, sadness x %, displeasure x %, anger x %, fear x % or neutrality x % as an emotion recognition result. When the emotion of the user is recognized using an artificial intelligence model that performs deep learning with respect to emotion to be recognized, a tagging value of data used during deep learning is output as a result value.)
Therefore, prior to the effective filing date of the claimed invention, it would have been
obvious to one of ordinary skill in the art to modify Yanase to include a neutral or default appearance for a robot as taught in Moon. The neutral appearance provides a foundational appearance from which all other emotions or appearances can be based, which may help the user with recognizing the robot and the appearance variations that are associated with a particular user.
Claim 49:
Yanase does not explicitly teach the following limitations, however Moon teaches:
The system according to claim 38, wherein the avatar appearance is controllable in response to a user command to perform a verbal communication and/or a visual action.
( Moon - [0097] The robot 100a may output an expression based on the recognized emotion (S120). The processor 180 may control at least one of the display 152, the speaker 154, the light output unit 156 or the rotation unit 160 to output an emotional expression corresponding to the recognized emotion, thereby more emotionally delivering the content which is being output.)
Therefore, prior to the effective filing date of the claimed invention, it would have been
obvious to one of ordinary skill in the art to modify Yanase to include a method for the robot to change its appearance based the emotional changes (visual action) of the user as taught in Moon. Providing the ability for the robot to change its appearance, based on the actions of the user, enhance the level of interactivity and connection between the robot and the user.
Claim(s) 48 and 53 are rejected under 35 U.S.C. 103 as being unpatentable over Yanase (US 20200290198 A1) as modified by Guerin (US 20160257000 A1)
Claim 48:
Yanase does not explicitly teach the following limitations, however Guerin teaches:
The system according to claim 47, wherein at least one virtual robot system is configured to present a virtual object corresponding to one of the one or more interaction devices.
(Guerin - [0015] FIG. 10 depicts an example virtual user interface attached to a virtual robot; ; [0068] FIG. 10 depicts an example virtual user interface attached to a virtual robot. In FIG. 10, for example, an attached user interface may follow the position of the end effector, as shown with the motion constraint menu in FIG. 10.)
Therefore, prior to the effective filing date of the claimed invention, it would have been
obvious to one of ordinary skill in the art to modify Yanase to include a system of virtual controls which can displayed of a screen including the virtual robot as taught in Guerin. Providing a system for presenting essential controls on a virtual display, simplifies the control of the robot and increase the level of interactivity between the robot and user.
Claim 53:
Yanase does not explicitly teach the following limitations, however Guerin teaches:
The system according to claim 50, wherein at least one virtual robot system is configured to present a virtual object corresponding to an interaction device.
(Guerin - [0015] FIG. 10 depicts an example virtual user interface attached to a virtual robot; ; [0068] FIG. 10 depicts an example virtual user interface attached to a virtual robot. In FIG. 10, for example, an attached user interface may follow the position of the end effector, as shown with the motion constraint menu in FIG. 10.)
Therefore, prior to the effective filing date of the claimed invention, it would have been
obvious to one of ordinary skill in the art to modify Yanase to include a system of virtual controls which can displayed of a screen including the virtual robot as taught in Guerin. Providing a system for presenting essential controls on a virtual display, simplifies the control of the robot and increase the level of interactivity between the robot and user.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure or directed to the state of the art is listed on the enclosed PTO-892.
The following is a brief description for relevant prior art that was cited but not applied:
Williams (US 20190344449 A1) describes a system for controlling interactions between a plurality of real and virtual robots, includes one or more real robots present in the real environment, one or more virtual robots present in a virtual environment corresponding to the real environment, and a processing device operable to control interactions between one or more of the real robots and one or more of the virtual robots, where the interactions between the real and virtual robots are dependent upon at least the positions of the one or more real robots in the real environment and the positions of the one or more virtual robots in the virtual environment.
Earwood (US 20180117762 A1) describes a data exchange system is disclosed herein for receiving motion and attribute data corresponding to user interactions via a robot animation application. The motion and attribute data can be associated with a virtual animation of a virtual character using the robot animation application on a computing device. The data exchange system can translate the motion and attribute data for implementation on a physical or virtual robotic device to perform the virtual animation.
Gewickey (US 20190366557 A1) describes a method and apparatus for controlling a social robot includes operating an electronic output device based on social interactions between the social robot and a user. The social robot utilizes an algorithm or other logical solution process to infer a user mental state, for example a mood or desire, based on observation of the social interaction. Based on the inferred mental state, the social robot causes an action of the electronic output device to be selected.
Noma (US 20220016757 A1) describes a social robot. Based on the normal condition of a target person as estimated from a learning result regarding the everyday behavior and the everyday condition of the target person, and based on the recognized behavior and the recognized condition of the target person; the control unit controls the provision of information regarding the everyday life of the target person.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ALAN LINDSAY OSTROW whose telephone number is (703)756-1854. The examiner can normally be reached M-F 8 - 5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Adam Mott can be reached on (571) 270 5376. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ALAN LINDSAY OSTROW/
Examiner, Art Unit 3657
/ADAM R MOTT/Supervisory Patent Examiner, Art Unit 3657
Prosecution Insights