Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory obviousness-type double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); and In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on a nonstatutory double patenting ground provided the conflicting application or patent either is shown to be commonly owned with this application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement.
Effective January 1, 1994, a registered attorney or agent of record may sign a terminal disclaimer. A terminal disclaimer signed by the assignee must fully comply with 37 CFR 3.73(b).
Claims 1, 8, and 15 with dependent claims thereof, are rejected on the ground of nonstatutory obviousness-type double patenting as being unpatentable over claims 1, 2, 10, 11, and 19 and any dependent claims thereof of U.S. Patent No. 1218346. Although the conflicting claims are not identical, they are not patentably distinct from each other because said claims of the instant application includes all of the features of said claims of U.S. Patent No. 1218346. It would have been obvious to one of ordinary skill in the art to omit the step of utilizing models as a listener device and subsequent NLU models, In re Karlson 136 USPQ 184 (1963): "Omission of an element and its function is an obvious expedient if the remaining elements perform the same functions as before"
Present invention Conflicting claims - 1218346
1. A hub device for controlling a device based on a voice input, the hub device comprising: a communication interface configured to perform data communication with at least one of a voice assistant server or a plurality of devices comprising a listener device; memory configured to store a program comprising one or more instructions; and at least one processor configured to execute the one or more instructions of the program stored in the memory, wherein the at least one processor is further configured to: receive, through the communication interface, a voice signal from the listener device, convert the received voice signal into text by performing automatic speech recognition (ASR), analyze the text by using a first natural language understanding (NLU) model, determine an operation-performing device which is capable of performing an operation corresponding to the analyzed text by using a device determination model, identify which device stores a function determination model corresponding to the operation-performing device from among the operation-performing device and the listener device, and transmit, through the communication interface, at least part of the text to the identified device that stores the function determination model.
2. The hub device of claim 1, wherein the at least one processor is further configured to determine whether the determined operation-performing device is the same as the listener device.
3. The hub device of claim 2, wherein, in a case that it is determined that the operation-performing device is the same as the listener device, the at least one processor is further configured to: obtain function determination model information about whether the listener device stores the function determination model in an internal memory, and determine the listener device as the device that stores the function determination model based on the obtained function determination model information.
4. The hub device of claim 2, wherein, in a case that it is determined that the operation-performing device is a device different from the listener device, the at least one processor is further configured to: obtain function determination model information about whether the determined operation-performing device stores the function determination model in an internal memory, and determine whether the operation-performing device is the device that stores the function determination model based on the obtained function determination model information.
5. The hub device of claim 1, wherein the at least one processor is further configured to: receive, through the communication interface, update data of the device determination model from the voice assistant server, and update the device determination model, by using the received update data.
6. The hub device of claim 5, wherein the update data comprises data for updating the device determination model to determine an updated function from the text and determine the operation-performing device corresponding to the updated function, based on update information of the function determination model included in at least one of the operation-performing device or the listener device.
7. The hub device of claim 1, wherein the at least one processor is further configured to: receive, through the communication interface, device information of a new device comprising at least one of device identification information, storage information of a device determination model, or storage information of a function determination model of the new device, from the voice assistant server, and update the device determination model by adding the new device to device candidates determinable as the operation-performing device by the device determination model by using the received device information of the new device.
8. A method, performed by a hub device, of controlling a device, the method comprising: receiving a voice signal from a listener device by a hub device; converting the received voice signal into text, by performing automatic speech recognition (ASR); analyzing the text by using a first natural language understanding (NLU) model; determining an operation-performing device corresponding to the analyzed text by using a device determination model; identifying which device stores a function determination model corresponding to the determined operation-performing device from among the determined operation-performing device and the listener device; and transmitting at least part of the text to the identified device.
9. The method of claim 8, further comprising determining whether the determined operation-performing device is the same as the listener device.
10. The method of claim 8, wherein, in a case that it is determined that the operation-performing device is the same as the listener device, the identifying of the device that stores the function determination model comprises: obtaining function determination model information about whether the listener device stores the function determination model in an internal memory; and determining the listener device as the device that stores the function determination model, based on the obtained function determination model information.
11. The method of claim 8, wherein, in a case that it is determined that the operation-performing device is a device different from the listener device, the identifying of the device that stores the function determination model comprises: obtaining function determination model information about whether the determined operation-performing device stores the function determination model in an internal memory; and determining whether the operation-performing device is the device that stores the function determination model, based on the obtained function determination model information.
12. The method of claim 8, further comprising: receiving update data of the device determination model from a voice assistant server; and updating the device determination model, by using the received update data.
13. The method of claim 12, wherein the update data comprises data for updating the device determination model to determine an updated function from the text and determine the operation-performing device corresponding to the updated function, based on update information of the function determination model included in at least one of the operation-performing device or the listener device.
14. The method of claim 8, further comprising: receiving, from a voice assistant server, device information of a new device comprising at least one of device identification information, storage information of a device determination model, or storage information of a function determination model of the new device; and updating the device determination model, by adding the new device to device candidates determinable as the operation-performing device by the device determination model by using the received device information of the new device.
15. A non-transitory computer-readable recording medium having recorded thereon a program for executing a method comprising: receiving a voice signal from a listener device; converting the received voice signal into text by performing automatic speech recognition (ASR); analyzing the text by using a first natural language understanding (NLU) model; determining an operation-performing device corresponding to the analyzed text by using a device determination model; identifying which device stores a function determination model corresponding to the determined operation-performing device from among the determined operation-performing device and the listener device; and transmitting at least part of the text to the identified device.
1. A hub device for controlling a device based on a voice input, the hub device comprising: a communication interface configured to perform data communication with at least one of a plurality of devices, a voice assistant server, or an Internet of things (IoT) server; a microphone configured to receive a voice input of a user; a memory storing one or more instructions; and at least one processor configured to execute the one or more instructions stored in the memory to: convert the voice input received by the microphone into text by performing automatic speech recognition (ASR), determine an operation-performing device from among the plurality of devices based on the text, by using a device determination model, identify a device that stores a function determination model corresponding to the determined operation-performing device, and control the communication interface to provide at least part of the text to the identified device.
2. The hub device of claim 1, wherein the device determination model comprises a first natural language understanding (NLU) model configured to analyze the text and determine the operation-performing device based on a result of the analyzing of the text.
3. The hub device of claim 1, wherein the function determination model comprises a second NLU model configured to analyze at least part of the text and obtain operation information about an operation to be performed by the determined operation-performing device based on a result of the analyzing of the at least part of the text.
4. The hub device of claim 1, wherein the at least one processor is further configured to determine whether the function determination model corresponding to the determined operation-performing device is stored in the memory.
5. The hub device of claim 4, wherein the function determination model corresponding to the determined operation-performing device is stored in the memory of the hub device, and the at least one processor is further configured to control the communication interface to provide the at least part of the text to the function determination model stored in the memory.
6. The hub device of claim 5, wherein the at least one processor is further configured to: determine intent by interpreting the at least part of the text by using a second NLU model of the function determination model stored in the memory, obtain operation information about operation to be performed by the determined operation-performing device based on the determined intent, generate control command based on the operation information, and control the communication interface to transmit the control command to the determined operation-performing device.
7. The hub device of claim 1, wherein the at least one processor is further configured to: determine, based on information about the function determination model, whether the function determination model corresponding to the determined operation-performing device is stored in a memory of the operation-performing device, and based on a result of the determination, control the communication interface to transmit the at least part of the text to the function determination model stored in the memory of the operation-performing device.
8. The hub device of claim 1, wherein the function determination model corresponding to the determined operation-performing device is stored in the voice assistant server, the at least one processor is further configured to control the communication interface to transmit the at least part of the text to the function determination model stored in the voice assistant server.
9. The hub device of claim 1, wherein the plurality of devices are logged in by the same user account as an user account of the hub device and connected to the hub device through a network.
10. A method, performed by a hub device, of controlling a device based on a voice input, the method comprising: receiving a voice input of a user; converting the received voice input into text by performing automatic speech recognition (ASR); determining an operation-performing device based on the text, by using a device determination model; identifying a device that stores a function determination model corresponding to the determined operation-performing device; and providing at least part of the text to the identified device.
11. The method of claim 10, wherein the device determination model comprises a first natural language understanding (NLU) model configured to analyze the text and determine the operation-performing device based on a result of the analyzing of the text.
12. The method of claim 10, wherein the function determination model comprises a second NLU model configured to analyze at least part of the text and obtain operation information about an operation to be performed by the determined operation-performing device based on a result of the analyzing of the at least part of the text.
13. The method of claim 10, wherein the identifying of the device that stores the function determination model comprises determining whether the function determination model corresponding to the determined operation-performing device is stored in the memory.
14. The method of claim 13, wherein the function determination model corresponding to the determined operation-performing device is stored in the memory of the hub device, and the providing of the at least part of the text comprises providing the at least part of the text to the function determination model stored in the memory.
15. The method of claim 14, further comprising: determining intent by interpreting the at least part of the text by using a second NLU model of the function determination model stored in the memory; obtaining operation information about operation to be performed by the determined operation-performing device based on the determined intent; generating control command based on the operation information; and transmitting the control command to the determined operation-performing device.
16. The method of claim 10, wherein the identifying of the device that stores the function determination model comprises determining, based on information about the function determination model, whether the function determination model corresponding to the determined operation-performing device is stored in a memory of the operation-performing device, and the providing of the at least part of the text comprises transmitting the at least part of the text to the function determination model stored in the memory of the operation-performing device based on a result of the determination.
17. The method of claim 10, wherein the function determination model corresponding to the determined operation-performing device is stored in the voice assistant server, the providing of the at least part of the text comprises transmitting the at least part of the text to the function determination model stored in the voice assistant server.
18. The method of claim 10, wherein a plurality of devices are logged in by the same user account as an user account of the hub device and connected to the hub device through a network.
19. A non-transitory computer-readable recording medium having recorded thereon a program for executing a method comprising: receiving a voice input of a user; converting the received voice input into text by performing automatic speech recognition (ASR); determining an operation-performing device based on the text, by using a device determination model; identifying a device that stores a function determination model corresponding to the determined operation-performing device; and providing at least part of the text to the identified device.
Claims 1, 8, and 15 with dependent claims thereof, are rejected on the ground of nonstatutory obviousness-type double patenting as being unpatentable over claims 1, 2, 11, and 12 and any dependent claims thereof of U.S. Patent No. 11721343. Although the conflicting claims are not identical, they are not patentably distinct from each other because said claims of the instant application includes all of the features of said claims of U.S. Patent No. 11721343. It would have been obvious to one of ordinary skill in the art to omit the step of utilizing models as a listener device and subsequent NLU models, In re Karlson 136 USPQ 184 (1963): "Omission of an element and its function is an obvious expedient if the remaining elements perform the same functions as before"
Present invention Conflicting claims - 11721343
1. A hub device for controlling a device based on a voice input, the hub device comprising: a communication interface configured to perform data communication with at least one of a voice assistant server or a plurality of devices comprising a listener device; memory configured to store a program comprising one or more instructions; and at least one processor configured to execute the one or more instructions of the program stored in the memory, wherein the at least one processor is further configured to: receive, through the communication interface, a voice signal from the listener device, convert the received voice signal into text by performing automatic speech recognition (ASR), analyze the text by using a first natural language understanding (NLU) model, determine an operation-performing device which is capable of performing an operation corresponding to the analyzed text by using a device determination model, identify which device stores a function determination model corresponding to the operation-performing device from among the operation-performing device and the listener device, and transmit, through the communication interface, at least part of the text to the identified device that stores the function determination model.
2. The hub device of claim 1, wherein the at least one processor is further configured to determine whether the determined operation-performing device is the same as the listener device.
3. The hub device of claim 2, wherein, in a case that it is determined that the operation-performing device is the same as the listener device, the at least one processor is further configured to: obtain function determination model information about whether the listener device stores the function determination model in an internal memory, and determine the listener device as the device that stores the function determination model based on the obtained function determination model information.
4. The hub device of claim 2, wherein, in a case that it is determined that the operation-performing device is a device different from the listener device, the at least one processor is further configured to: obtain function determination model information about whether the determined operation-performing device stores the function determination model in an internal memory, and determine whether the operation-performing device is the device that stores the function determination model based on the obtained function determination model information.
5. The hub device of claim 1, wherein the at least one processor is further configured to: receive, through the communication interface, update data of the device determination model from the voice assistant server, and update the device determination model, by using the received update data.
6. The hub device of claim 5, wherein the update data comprises data for updating the device determination model to determine an updated function from the text and determine the operation-performing device corresponding to the updated function, based on update information of the function determination model included in at least one of the operation-performing device or the listener device.
7. The hub device of claim 1, wherein the at least one processor is further configured to: receive, through the communication interface, device information of a new device comprising at least one of device identification information, storage information of a device determination model, or storage information of a function determination model of the new device, from the voice assistant server, and update the device determination model by adding the new device to device candidates determinable as the operation-performing device by the device determination model by using the received device information of the new device.
8. A method, performed by a hub device, of controlling a device, the method comprising: receiving a voice signal from a listener device by a hub device; converting the received voice signal into text, by performing automatic speech recognition (ASR); analyzing the text by using a first natural language understanding (NLU) model; determining an operation-performing device corresponding to the analyzed text by using a device determination model; identifying which device stores a function determination model corresponding to the determined operation-performing device from among the determined operation-performing device and the listener device; and transmitting at least part of the text to the identified device.
9. The method of claim 8, further comprising determining whether the determined operation-performing device is the same as the listener device.
10. The method of claim 8, wherein, in a case that it is determined that the operation-performing device is the same as the listener device, the identifying of the device that stores the function determination model comprises: obtaining function determination model information about whether the listener device stores the function determination model in an internal memory; and determining the listener device as the device that stores the function determination model, based on the obtained function determination model information.
11. The method of claim 8, wherein, in a case that it is determined that the operation-performing device is a device different from the listener device, the identifying of the device that stores the function determination model comprises: obtaining function determination model information about whether the determined operation-performing device stores the function determination model in an internal memory; and determining whether the operation-performing device is the device that stores the function determination model, based on the obtained function determination model information.
12. The method of claim 8, further comprising: receiving update data of the device determination model from a voice assistant server; and updating the device determination model, by using the received update data.
13. The method of claim 12, wherein the update data comprises data for updating the device determination model to determine an updated function from the text and determine the operation-performing device corresponding to the updated function, based on update information of the function determination model included in at least one of the operation-performing device or the listener device.
14. The method of claim 8, further comprising: receiving, from a voice assistant server, device information of a new device comprising at least one of device identification information, storage information of a device determination model, or storage information of a function determination model of the new device; and updating the device determination model, by adding the new device to device candidates determinable as the operation-performing device by the device determination model by using the received device information of the new device.
15. A non-transitory computer-readable recording medium having recorded thereon a program for executing a method comprising: receiving a voice signal from a listener device; converting the received voice signal into text by performing automatic speech recognition (ASR); analyzing the text by using a first natural language understanding (NLU) model; determining an operation-performing device corresponding to the analyzed text by using a device determination model; identifying which device stores a function determination model corresponding to the determined operation-performing device from among the determined operation-performing device and the listener device; and transmitting at least part of the text to the identified device.
1. (currently amended): A hub device for controlling a device based on a voice input, the hub device comprising: a communication interface configured to perform data communication with at least one of a plurality of devices, a voice assistant server, or an Internet of Things (IoT) server; a microphone configured to receive a voice input of a user; a memory storing a program comprising one or more instructions; and a processor configured to execute the one or more instructions of the program stored in the memory, wherein the processor is further configured to execute the one or more instructions to: convert the voice input received through the microphone into text by performing automatic speech recognition (ASR), determine an operation-performing device capable of performing an operation corresponding to the text from among the plurality of devices, by using a device determination model, obtain function determination model information from at least one device storing a function determination model used to determine a function related to each of the plurality of devices, identify a device that stores the function determination model corresponding to the determined operation-performing device by using the obtained function determination model information, and control the communication interface to transmit at least part of the text to the identified device.
2. (currently amended): The hub device of claim 1, wherein the device determination model comprises a first natural language understanding (NLU) model configured to analyze the text and determine the operation-performing device based on a result of the analyzing of the text.
3. (currently amended): The hub device of claim 1, wherein the function determination model comprises a second NLU model configured to analyze the at least part of the text and obtain operation information about an operation to be performed by the determined operation-performing device based on a result of the analyzing of the at least part of the text.
4. (currently amended): The hub device of claim 1, wherein the function determination model information comprises information about whether each of the plurality of devices itself stores a function determination model used to obtain operation information for performing an operation.
AMENDMENT UNDER 37 C.F.R. § 1.111 Attorney Docket No.: Q261614
5. (currently amended): The hub device of claim 1, wherein the processor is further configured to obtain, from the plurality of devices, storage information of the function determination model comprising at least one of device identification information, an IP address, or a MAC address of each of the plurality of devices.
6. (currently amended): The hub device of claim 1, wherein the function determination model corresponding to the determined operation-performing device is stored in at least one of a memory the memory of the hub device, a memory an internal memory of the operation-performing device itself, or the voice assistant server.
7. (currently amended): The hub device of claim 1, wherein the plurality of devices are logged in by a same user account as an user account of the hub device and connected to the hub device through a network.
8. (currently amended): The hub device of claim 1, wherein the processor is further configured to: receive update data of the device determination model from the voice assistant server, by using the communication interface, and update the device determination model, by using the received update data.
9. (currently amended): The hub device of claim 8, wherein the update data comprises data for updating the device determination model to determine an updated function from the text and determine the operation-performing device corresponding AMENDMENT UNDER 37 C.F.R. § 1.111 Attorney Docket No.: Q261614 to the updated function, based on update information of the function determination model included in the operation-performing device.
10. (currently amended): The hub device of claim 1, wherein the processor is further configured to: receive, by using the communication interface, device information of a new device comprising at least one of device identification information of the new device, storage information of a device determination model of the new device, or storage information of a function determination model of the new device, from the voice assistant server, and update the device determination model, by adding the new device to device candidates determinable as the operation-performing device by the device determination model by using the received device information of the new device.
11. (currently amended): A method, performed by a hub device, for controlling a device based on a voice input, the method comprising: receiving a voice input of a user by the hub device; converting, by the hub device, the received voice input into text by performing automatic speech recognition (ASR); determining an operation-performing device capable of performing an operation corresponding to the text from among a plurality of devices, by using a device determination model; AMENDMENT UNDER 37 C.F.R. § 1.111 Attorney Docket No.: Q261614 obtaining function determination model information from at least one device storing a function determination model used to determine a function related to each of the plurality of devices; identifying a device that stores the function determination model corresponding to the determined operation-performing device by using the obtained function determination model information; and transmitting at least part of the text to the identified device.
12. (original): The method of claim 11, wherein the device determination model comprises a first natural language understanding (NLU) model configured to analyze the text and determine the operation-performing device based on a result of the analyzing of the text.
13. (original): The method of claim 11, wherein the function determination model comprises a second NLU model configured to analyze the at least part of the text and obtain operation information related to an operation to be performed by the determined operation-performing device based on a result of the analyzing of the at least part of the text.
14. (original): The method of claim 11, wherein the function determination model information comprises information about whether each of the plurality of devices itself stores a function determination model used to obtain operation information for performing an operation.
AMENDMENT UNDER 37 C.F.R. § 1.111 Attorney Docket No.: Q261614
15. (currently amended): The method of claim 11, wherein the obtaining function determination model information comprisesfurther comprising: obtaining, from the plurality of devices, storage information of the function determination model comprising at least one of device identification information, an IP address, or a MAC address of each of the plurality of devices.
16. (currently amended): The method of claim 11, wherein the function determination model corresponding to the determined operation-performing device is stored in at least one of a memory the memory of the hub device, a memory an internal memory of the operation-performing device itself, or a voice assistant server.
17. (currently amended): The method of claim 11, wherein the plurality of devices are logged in by a same user account as an user account of the hub device and connected to the hub device through a network.
18. (original): The method of claim 11, further comprising: receiving update data of the device determination model from a voice assistant server; and updating the device determination model, by using the received update data.
19. (original): The method of claim 18, wherein AMENDMENT UNDER 37 C.F.R. § 1.111 Attorney Docket No.: Q261614 the update data comprises data for updating the device determination model to determine an updated function from the text and determine the operation-performing device corresponding to the updated function, based on update information of the function determination model included in the operation-performing device.
20. (original): The method of claim 11, further comprising: receiving device information of a new device comprising at least one of device identification information of the new device, storage information of a device determination model of the new device, or storage information of a function determination model of the new device, from a voice assistant server; and updating the device determination model, by adding the new device to device candidates determinable as the operation-performing device by the device determination model by using the received device information of the new device.
Claims 1, 8, and 15 with dependent claims thereof, are rejected on the ground of nonstatutory obviousness-type double patenting as being unpatentable over claims 1, 6, 11, and 16 and any dependent claims thereof of U.S. Patent No. 10964327. Although the conflicting claims are not identical, they are not patentably distinct from each other because said claims of the instant application includes all of the features of said claims of U.S. Patent No. 10964327. It would have been obvious to one of ordinary skill in the art to omit the step of utilizing models as a listener device and subsequent NLU models, In re Karlson 136 USPQ 184 (1963): "Omission of an element and its function is an obvious expedient if the remaining elements perform the same functions as before"
Present invention Conflicting claims - 10964327
1. A hub device for controlling a device based on a voice input, the hub device comprising: a communication interface configured to perform data communication with at least one of a voice assistant server or a plurality of devices comprising a listener device; memory configured to store a program comprising one or more instructions; and at least one processor configured to execute the one or more instructions of the program stored in the memory, wherein the at least one processor is further configured to: receive, through the communication interface, a voice signal from the listener device, convert the received voice signal into text by performing automatic speech recognition (ASR), analyze the text by using a first natural language understanding (NLU) model, determine an operation-performing device which is capable of performing an operation corresponding to the analyzed text by using a device determination model, identify which device stores a function determination model corresponding to the operation-performing device from among the operation-performing device and the listener device, and transmit, through the communication interface, at least part of the text to the identified device that stores the function determination model.
2. The hub device of claim 1, wherein the at least one processor is further configured to determine whether the determined operation-performing device is the same as the listener device.
3. The hub device of claim 2, wherein, in a case that it is determined that the operation-performing device is the same as the listener device, the at least one processor is further configured to: obtain function determination model information about whether the listener device stores the function determination model in an internal memory, and determine the listener device as the device that stores the function determination model based on the obtained function determination model information.
4. The hub device of claim 2, wherein, in a case that it is determined that the operation-performing device is a device different from the listener device, the at least one processor is further configured to: obtain function determination model information about whether the determined operation-performing device stores the function determination model in an internal memory, and determine whether the operation-performing device is the device that stores the function determination model based on the obtained function determination model information.
5. The hub device of claim 1, wherein the at least one processor is further configured to: receive, through the communication interface, update data of the device determination model from the voice assistant server, and update the device determination model, by using the received update data.
6. The hub device of claim 5, wherein the update data comprises data for updating the device determination model to determine an updated function from the text and determine the operation-performing device corresponding to the updated function, based on update information of the function determination model included in at least one of the operation-performing device or the listener device.
7. The hub device of claim 1, wherein the at least one processor is further configured to: receive, through the communication interface, device information of a new device comprising at least one of device identification information, storage information of a device determination model, or storage information of a function determination model of the new device, from the voice assistant server, and update the device determination model by adding the new device to device candidates determinable as the operation-performing device by the device determination model by using the received device information of the new device.
8. A method, performed by a hub device, of controlling a device, the method comprising: receiving a voice signal from a listener device by a hub device; converting the received voice signal into text, by performing automatic speech recognition (ASR); analyzing the text by using a first natural language understanding (NLU) model; determining an operation-performing device corresponding to the analyzed text by using a device determination model; identifying which device stores a function determination model corresponding to the determined operation-performing device from among the determined operation-performing device and the listener device; and transmitting at least part of the text to the identified device.
9. The method of claim 8, further comprising determining whether the determined operation-performing device is the same as the listener device.
10. The method of claim 8, wherein, in a case that it is determined that the operation-performing device is the same as the listener device, the identifying of the device that stores the function determination model comprises: obtaining function determination model information about whether the listener device stores the function determination model in an internal memory; and determining the listener device as the device that stores the function determination model, based on the obtained function determination model information.
11. The method of claim 8, wherein, in a case that it is determined that the operation-performing device is a device different from the listener device, the identifying of the device that stores the function determination model comprises: obtaining function determination model information about whether the determined operation-performing device stores the function determination model in an internal memory; and determining whether the operation-performing device is the device that stores the function determination model, based on the obtained function determination model information.
12. The method of claim 8, further comprising: receiving update data of the device determination model from a voice assistant server; and updating the device determination model, by using the received update data.
13. The method of claim 12, wherein the update data comprises data for updating the device determination model to determine an updated function from the text and determine the operation-performing device corresponding to the updated function, based on update information of the function determination model included in at least one of the operation-performing device or the listener device.
14. The method of claim 8, further comprising: receiving, from a voice assistant server, device information of a new device comprising at least one of device identification information, storage information of a device determination model, or storage information of a function determination model of the new device; and updating the device determination model, by adding the new device to device candidates determinable as the operation-performing device by the device determination model by using the received device information of the new device.
15. A non-transitory computer-readable recording medium having recorded thereon a program for executing a method comprising: receiving a voice signal from a listener device; converting the received voice signal into text by performing automatic speech recognition (ASR); analyzing the text by using a first natural language understanding (NLU) model; determining an operation-performing device corresponding to the analyzed text by using a device determination model; identifying which device stores a function determination model corresponding to the determined operation-performing device from among the determined operation-performing device and the listener device; and transmitting at least part of the text to the identified device.
1. A method comprising:
based on receiving a voice input of a user by a hub device from a first external device among a plurality of external devices connected to the hub device:
converting, by the hub device, the received voice input into text by performing automatic speech recognition (ASR);
identifying, by the hub device, a second external device from the plurality of external devices capable of performing an operation corresponding to the text by analyzing, by the hub device, the text by using a first natural language understanding (NLU) model;
identifying an external device, from among the hub device and the plurality of external devices connected to the hub device, as a function determination model storing device, which stores a function determination model corresponding to the second external device capable of performing the operation corresponding to the text;
identifying that the function determination model storing device is different from the hub device and the first external device; and
transmitting at least part of the text to the identified function determination model storing device.
2. (Previously presented) The method of claim 1, wherein the first NLU model is a device determination model.
3. (Currently amended) The method of , further comprising analyzing, using a second natural language understanding (NLU) model included in the device determination model, at least part of the text and obtaining operation information related to the operation corresponding to the text, based on a result of the analyzing of the at least part of the text.
4. (Currently amended) The method of claim 1, further comprising obtaining, from at least one storing the function determination model, information about the function determination model stored in the at least one device.
5. (Currently amended) The method of claim 4, wherein the identifying vi vvmun Ut vil" 6Lvi - "nodel comprises identifying the device storing the function determination model corresponding to the identified device capable of performing the operation, based on the obtained information about the function determination model.
6. (Currently amended) A hub device for controlling a device based on a voice input, the hub device comprising: a communication interface configured to perform data communication with at least one of a plurality of devices, a voice assistant server, or an Internet of things (IoT) server; a memory configured to store a program comprising one or more instructions; and a processor configured to execute the one or more instructions of the program stored in the memory, to: convert the voice input A +1-1, -' ~~ into text by performing automatic speech recognition (ASR), identify a device capable of performing an operation corresponding to the text by analyzing the text by using a first natural language understanding (NLU) model; identify which device stores a function determination model '¾rming the operation corresponding to the text , from among the hub device, and arality of external a plurality of oth devices connected to the hub device AMENDMENT UNDER 37 C.F.R. § 1.111 Attorney Docket No.: Q254574 *he first external r and sed on the identified device that stores the function determination r. cerent from the hub device, control the communication interface to transmit at least part of the text to the identified device
7. (Canceled).
8. (Currently amended) The hub device of claim 6, wherein the function determination model comprises a second NLU model configured to analyze at least part of the text and obtain operation information about an operation to be performed by the device capable of performing the operation corresponding to the text based on a result of the analyzing of the at least part of the text.
9. (Currently amended) The hub device of claim 6, wherein the processor is further configured to execute the one or more instructions to control the communication interface to obtain, from at least one storing the function determination model used to determine a function related to each of the plurality of devices from among the plurality of devices, information about the function determination model stored in the at least one device.
AMENDMENT UNDER 37 C.F.R. § 1.111 Attorney Docket No.: Q254574
10. (Currently amended) The hub device of claim 9, wherein the processor is further configured to execute the one or more instructions to identify 1 mouei device stores the function determination model corresponding to the 1 -* device capable of performing the operation corresponding to the text, based on the obtained information about the function determination model.
11. (Currently amended) A method of operating a system comprising a hub device and a first device that stores a function determination model, the method comprising: based on receiving, by the hub device a voice input of a user converting, by the hub device, the received voice input into text, by performing automatic speech recognition (ASR) by using data about an ASR module stored in a memory of the hub device; identifying, by the hub device, the first device as a device capable of performing an operation corresponding to the text, by analyzing the text by using a first natural language understanding (NLU) model; ,u~uyuin an exteiiai uevlie, nni anioun tue nuu uevice anu tue piuauty 01 devices connected to the hub device, as a function determination model storing ice, which stores a function determination model corresponding to the first external obtaining, by the hub device, from the device, information about the function determination model stored in the device; and AMENDMENT UNDER 37 C.F.R. § 1.111 Attorney Docket No.: Q254574 transmitting, by the hub device, at least part of the text to the device, based on the obtained information about the function determination model.
12. (Currently amended) The method of claim 11, wherein the identifying the first device as the device capable of performing the operation comprises determining the first device from among a plurality of devices as the device capable of performing the operation corresponding to the text based on a result of the analyzing of the text.
13. (Currently amended) The method of claim 11, further comprising analyzing, using the function determination model comprises using a second natural language understanding (NLU) model included in a device determination model, to analyze at least part of the text received from the hub device and obtaining operation information about an operation to be performed by the first device based on a result of the analyzing of the at least part of the text.
14. (Currently amended) The method of claim 13, further comprising analyzing, by the first device, the at least part of the text by using the second NLU model of the function determination model and obtaining the operation information about the operation to be performed by the first device based on a result of the analyzing of the at least part of the text.
AMENDMENT UNDER 37 C.F.R. § 1.111 Attorney Docket No.: Q254574
15. (Currently amended) The method of claim 14, further comprising: generating, by the first device, a control command for controlling an operation of the first device based on the operation information; and executing, by the first device, the operation based on the control command.
16. (Currently amended) A multi-device system comprising a hub device and a first device that stores a function determination model, wherein the hub device comprises: a communication interface configured to perform data communication with the first device that stores the function determination model; a memory configured to store a program comprising one or more instructions; and a processor configured to execute the one or more instructions of the program stored in the memory, to: convert the voice input -A +I ^ _;. into text by performing automatic speech recognition (ASR), identify the first device as a device capable of performing an operation corresponding to the text by analyzing the text by using a device determination model that includes a first natural language understanding (NLU) model, 21uLiy anl UA00U i uuvluc., uum111 am , LuV, 1uu uuvtluc alu LuV, pliwaty v AMENDMENT UNDER 37 C.F.R. § 1.111 Attorney Docket No.: Q254574 control the communication interface to obtain, from the _- . device, information about the function determination model stored in the device, and control the communication interface to transmit at least part of the text to the device, based on the obtained information about the function determination model.
17. (Currently amended) The multi-device system of claim 16, wherein the processor is further configured to determine the first device from among a plurality of devices as the device capable of performing an operation corresponding to the text based on a result of the analyzing of the text.
18. (Currently amended) The multi-device system of claim 16, wherein the first device comprises a communication interface configured to receive the at least part of the text from the hub device, wherein the function determination model comprises a second NLU model configured to analyze the received at least part of the text and obtain operation information about an operation to be performed by the first device based on a result of the analyzing of the at least part of the text.
19. (Currently amended) The multi-device system of claim 18, wherein the first device further comprises a processor configured to analyze the at least part of the text by using the second NLU model and obtain the operation information to be performed by the first device based on a result of the analyzing of the at least part of the text.
20. (Currently amended) The multi-device system of claim 19, wherein the processor of the first device is further configured to control at least one element of the first device to generate a control command for controlling an operation of the first device based on the operation information and execute the operation based on the control command.
21. (New): The method of claim 1, wherein the identified function determination model storing device is the second external device.
Allowable Subject Matter
Claims 1-15 are allowed.
The following is an examiner’s statement of reasons for allowance:
After a full review of the complex claims as a whole. the examiner believes that the prior art taken alone or in combination fails to teach the claims as a whole such as in claims 1, 8, and 15 hub device for controlling a device based on a voice input, the hub device comprising: a communication interface configured to perform data communication with at least one of a voice assistant server or a plurality of devices comprising a listener device; memory configured to store a program comprising one or more instructions; and at least one processor configured to execute the one or more instructions of the program stored in the memory, wherein the at least one processor is further configured to: receive, through the communication interface, a voice signal from the listener device, convert the received voice signal into text by performing automatic speech recognition (ASR), analyze the text by using a first natural language understanding (NLU) model, determine an operation-performing device which is capable of performing an operation corresponding to the analyzed text by using a device determination model, identify which device stores a function determination model corresponding to the operation-performing device from among the operation-performing device and the listener device, and transmit, through the communication interface, at least part of the text to the identified device that stores the function determination model.
The above claims are deemed allowable given the complex nature of delaying transmitting text by the hub until a link between function determination model device and operation-performing device is met, and therefore having a dedicated function determination model per device for identifying all device functions from the plurality and transmitting as precisely claimed. The closest prior art group teaches the transmission of text to multiple devices where the comparison is made, where a one-to-many device selection is possible but after the text has been transmitted for comparison. Additionally art teaches multiple NLU models per device type such as dryer versus thermostat, each having their own dedicated NLU model. Such prior art demonstrates improvements of known model comparison and device group communication in home automation, where a server of the local device makes the comparison. At least one of three devices are missing and under BRI the conditional transmission based on device linking does not appear to be taught. The closest prior art further teaches server-based approaches for models as well as local models, wherein the server models are used when an on-board local model fails e.g. basic English local and complex commands at server e.g. “raise the thermostat 2 degrees”. Such server models are typically tied to any number of devices e.g. music player, gps, overall phone, tv, stove, etc. The core concept of home automation is the same in the prior art and present invention however steps which produce similar results teach away from the prior art, such that the allowable claims utilize a model on each external device e.g. a hub, 1st, and 2nd. Under BRI assuming a hub = server, device1 = phone, and device2 = any device, at best the prior art reads up a hub and device1. There is also the assumption that a server actually performs the operation, which it does not, however one of ordinary skill can argue that ASR is part of the operation itself. At best under BRI the combination of prior art would teach the third device being a separate user device, however the first user must use a second users’ device which must also contain a model to perform the operation of the first user. Prior art in this field typically seek to contain models off-board on a server to avoid the need for each device to have a dedicated model (e.g. Google, SIRI, Alexa), wherein the device can simply execute the commands with a CPU without using a model or any form of post ASR processing (server alone does the heavy ASR model work). The allowed claims appear to demonstrate a deviation from this. Therefore, the prior art fails to teach or suggest the complex claims as a whole.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
US 20170025124 A1 Mixter; Kenneth et al.
Device selection by score comparison for homonymous inputs of speech, text is transmitted to each device regardless
US 20160104480 A1 Sharifi; Matthew
Device command comparison by score general input of speech “Hello computer”, text is transmitted to each device regardless
US 20110060587 A1 Phillips; Michael S. et al.
Various models for various applications on a single device, a server is used to pull models
US 10978046 B2 Huang; Haibin et al.
First and second NLU for each device/appliance which has dedicated functions
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MICHAEL COLUCCI whose telephone number is (571)270-1847. The examiner can normally be reached on M-F 9 AM - 7 PM.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Flanders can be reached at (571)272-7516. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MICHAEL COLUCCI/Primary Examiner, Art Unit 2655 (571)-270-1847
Examiner FAX: (571)-270-2847
Michael.Colucci@uspto.gov
Prosecution Insights