DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Information Disclosure Statement
The information disclosure statements (IDS) submitted on December 18th, 2023, May 20th 2024, December 17th 2024, and March 26th 2025 are in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statements are being considered by the examiner.
Response to Arguments
This is in response to applicant’s amendment/response filed on December 2nd 2025 which have been entered and made of record.
Applicant’s arguments regarding claim rejections under 35 U.S.C. 112(b) have been fully considered and are persuasive. Rejections have been withdrawn from claims 6, 9, and 13.
Applicant’s arguments regarding claim rejections under 35 U.S.C. 103 have been fully considered but they are not persuasive.
Applicant argues the Ross article discloses an augmented reality interface for visualizing and interacting with IoT devices. As noted in the article, an augmented reality interface is "especially relevant to the many devices that lack a direct interface of their own." Page 2. The Ross article relates to an IoT device that does not have display or interface and thus the Ross article is not related to displaying, via a display, of a virtual device. Thus, the Ross article does not disclose or suggest, among other things, in response to identifying that the MR device is worn, transmit, through the communication circuitry, to another electronic device, device information of the virtual device registered using the software application as associated with the virtual device; receive, from the other electronic device, through the communication circuitry, information regarding the second image generated by the other electronic device in accordance with the device information of the virtual device; and transmit, through the communication circuitry to the MR device, a signal for displaying the second image through a display of the virtual device to be displayed on the first image through the MR device.
Examiner respectable disagrees while Ross highlights the benefit of their invention for IoT devices that don’t have displays/interfaces. IoT devices encompass a wide variety of devices that do have displays/interfaces, such as televisions, phones, tablets, etc.… (Ross et al. Page 19, Para. 3). As well as in response to applicant's arguments against the references individually, one cannot show nonobviousness by attacking references individually where the rejections are based on combinations of references. See In re Keller, 642 F.2d 413, 208 USPQ 871 (CCPA 1981); In re Merck & Co., 800 F.2d 1091, 231 USPQ 375 (Fed. Cir. 1986).
Applicant argues in short, Desai et al. simply discloses a camera of the HMND capturing a smartphone image displayed as part of a virtual reality display. Moreover, as noted in Desai et al. at page 223, Desai et al.'s system is limited because of the "jumpiness of the smartphone display within the virtual environment" due to instability of algorithms in the image processing pipeline. This further underscores the difference between Desai et al.'s use of camera images of a display for including in the virtual reality display and claim l's arrangement of the electronic device receiving information regarding the content and transmitting a signal for displaying the content through the display of the virtual device, as claimed.
Examiner respectable disagrees Desai utilizes an electronic device (Smartphone) that transmits content to a virtual device (Virtual Smartphone). ( Desai et al. Fig. 6, Section: C. Smartphone Detection After SDSC, Page 222 Col.2 and Page 223 Col. 1) A display of a virtual smartphone is the displayed content as a screen encompasses all displayed content.
Regarding the remaining arguments applicant argues with respect to the amended claim language, which is fully addressed in the prior art rejections set forth below.
Claim Objections
Claim 9 is objected to because of the following informalities: Claim 9 recites the following limitation “identify that the virtual device from among the virtual device and…”. Which contains a typographical error, the second mention of the virtual device should be plural. Appropriate correction is required.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-7, 9-21 are rejected under 35 U.S.C. 103 as being unpatentable over NPL “Augmented Reality Interface for Visualizing and Interacting with IoT Devices” by Christopher Ross (hereinafter Ross) in view of NPL “A Window to your Smartphone: Exploring Interaction and Communication in Immersive VR with Augmented Virtuality” by Amit P. Desai, Lourdes Pena-Castillo, and Oscar Meruvia-Pastor (hereinafter Desai) in further view of U.S. Patent 9996308 B2 by Park Cheongha (hereinafter Park).
Regarding claim 1, Ross teaches an electronic device (IoT Device or Control Device, Page 8 Para. 2 Lines 3-11 and Page 19 Para. 3 Lines 10-22) comprising:
a microphone; (Page 12 Para. 2 Lines 3-5)
communication circuitry; (Communications Interface, Page 19 Para. 1 Lines 1-3 and Page 16 Para. 3 Lines 11-15)
memory, storing instructions, comprising one or more storage media; (Page 18 Para. 4 Lines 15-22)
and a processor comprising one or more processing circuitry (Processor core, microprocessor, chip, controller) wherein the instructions, when executed by the at least one processor, cause the electronic device to: (Page 18 Para. 4 Lines 15-22),
receive a voice signal (Capture Audio Signals, Page 12 Para. 2 Lines 3-9) through the microphone;
based on a recognition of the voice signal (Page 12 Para. 2-3 Lines 3-12), identify a voice command (Page 12 Para. 2 Lines 3-7) that indicates (Fig.1, Page 21) (Augmented Reality Representation of Physical Device, Page 4 Para. 1 Lines 1-8), having a display (An IoT Physical Device Display, Page 19, Para. 3), registered using a software application (Page 16 Para. 3 Lines 11-15) for controlling another device through the electronic device (IoT Device or Control Device, Page 8 Para. 2 Lines 3-11),
However, Ross fails to explicitly teach:
displaying content through a virtual device, and the virtual device configured to be displayed through a display of a mixed reality(MR) device as superimposed on a first image obtained through a camera of the MR device;
based on the voice command, identify whether the MR device is worn, wherein the identifying of whether the MR device worn is performed before performing obtaining a second image to be displayed through the virtual device;
in response to identifying that the MR device is worn, transmit, through the communication circuitry, to another electronic device, device information of the virtual device registered using the software application as associated with the virtual device;
receive, from the other electronic device, through the communication circuitry, information regarding the second image generated by the other electronic device in accordance with the device information of the virtual device;
and transmit, through the communication circuitry to the MR device, a signal for displaying the second image through a display of the virtual device to be displayed on the first image through the MR device.
Ross and Desai are analogous to the claimed invention because both of them are in the same field of controlling or modifying Head Mounted Displays.
Desai teaches:
displaying content through a virtual device (Fig. 6, Section: C. Smartphone Detection After SDSC, Page 222 Col.2 and Page 223 Col. 1) having a display (Display Content from Smartphone) and the virtual device configured to be displayed through a display of a mixed reality(MR) device (VR Device) as superimposed on a first image (Virtual/Physical Environment) obtained through a camera of the MR device; The content from the smartphone is displayed on top of the virtual/physical environment shown in the MR device. A display of a smartphone is the screen which encompasses all of the content. Thus, the displayed content of the smartphone is the display when recreated virtually.
receive, from the other electronic device (Physical Smartphone), through the communication circuitry, information regarding the second image (Physical Smartphone’s Content) generated by the other electronic device (Physical Smartphone) in accordance with the device information of the virtual device (Virtual Smartphone Aligned with Physical Smartphone); (Section: III Setup, Page 219 Col. 2 and Page 220 Col. 1)
and transmit, through the communication circuitry to the MR device (VR Device), a signal for displaying the second image (Physical Smartphone’s Content) through a display of the virtual device (Virtual Smartphone) to be displayed on the first image (Virtual/Physical Environment) through the MR device (VR Device). (Fig. 6, Section: C. Smartphone Detection After SDSC, Page 222 Col.2 and Page 223 Col. 1)
Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Ross’s IoT Device that can Control a Head Mounted Display (HMD) through Voice Commands to incorporate Desai’s Head Mounted VR Device that can display an image through the VR Device to a Virtual Device. Since doing so would provide the benefit of displaying images from a physical device to a virtual one that mimics it. Which allows a user to interact with the physical device through the VR device without having to remove their VR Device.
However, Ross and Desai fail to explicitly teach:
based on the voice command, identify whether the MR device is worn, wherein the identifying of whether the MR device worn is performed before performing obtaining a second image to be displayed through the virtual device;
in response to identifying that the MR device is worn, transmit, through the communication circuitry, to another electronic device, device information of the virtual device registered using the software application as associated with the virtual device;
Ross, Desai, and Park are analogous to the claimed invention because all of them are in the same field of controlling or modifying Head Mounted Displays.
Park teaches:
based on the voice command, (Col. 5 Lines 32-41) identify whether the MR device (HMD) is worn(Col. 14 Lines 14-39), wherein the identifying of whether the MR device worn is performed before (Before Notification Information is Displayed, Col. 14 Lines 17-31) performing obtaining a second image (Notification Information) to be displayed through the virtual device (Displayed Notification Information Corresponding to Mobile Terminal 200 or Smartphone, Col. 12 Lines 29-38) ; The HMD tracks if the HMD is worn or not by various sensors. To determine when to display a notification to the user.
in response to identifying that the MR device is worn (Col. 14 Lines 14-39), transmit, through the communication circuitry (Mobile Communication Module 212, Col. 7 Lines 8-23), to another electronic device (Another Mobile Terminal 200 or Device Serving as Controller, Col.3 Lines 22-47), device information of the virtual device registered using the software application (Application Program, Col. 9 Lines 28-38) as associated with the virtual device (Displayed Notification Information Corresponding to Mobile Terminal 200 or Smartphone, Col. 12 Lines 29-38);
Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Ross’s system of controlling an HMD altered by Desai’s HMD display through display to incorporate Parks’ HMD that tracks if the device is worn or not. Since doing so would provide the benefit of tracking when a user is actively using the HMD to prevent cases where something is displayed to the HMD, but the user doesn’t see it because they are not wearing the device.
Regarding claim 2, Ross teaches querying the state or context of the various IoT device connected in the system. (Page 4 Para. 3 Lines 14-20)
However, Ross and Desai fail to explicitly teach the electronic device of claim 1, wherein the instructions, when executed by the at least one processor, cause the electronic device to provide, in response to identifying that the MR device is not worn, a notification for informing to wear the MR device.
Park teaches the electronic device (Mobile Terminal 200) of claim 1, wherein the instructions, when executed by the at least one processor, cause the electronic device (Mobile Terminal 200) to provide, in response to identifying that the MR device (HMD) is not worn, a notification for informing to wear the MR device (HMD). (Col. 19, Lines 1-21) Once a user has removed the HMD, notifications are sent to the mobile device connected to the HMD. One of ordinary skill in the art could easily modify the notifications to tell the user to put the device back on once a set time has elapsed, as the HMD does track if the device was taken off permanently or temporally. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Ross’s system that queries the state of the devices connected in the network to corporate Parks’ notification system and system that determines if the HMD is worn or not. Since doing so would provide the benefit of providing feedback to the user when they attempt to interact with a device that may be in a state which can’t be used. Such as, trying to interact with an HMD, but the HMD is not currently on or worn.
Regarding claim 3, Ross teaches the electronic device of claim 1, wherein the instructions, when executed by the at least one processor, cause the electronic device (IoT Device or Control Device, Page 8 Para. 2 Lines 3-11 and Page 19 Para. 3 Lines 10-22) to:
identify a display position(location), based on the recognition of the voice signal; (Page 7 Para. 2 Lines 5-12, Page 15 Para. 3 Lines 8-22 and Page 16 Para. 1 Lines 1-6) The location of each IoT device in the system is tracked based on where the user and the controller device are located.
and identify the voice command (Page 12 Para. 2 Lines 3-7), based on identifying that the display position (Page 7 Para. 2 Lines 5-12, Page 15 Para. 3 Lines 8-22 and Page 16 Para. 1 Lines 1-6) corresponds to a position of the virtual device (Augmented Reality Representation of Physical Device, Page 4 Para. 1 Lines 1-8) registered by using the software application (Page 9, Para. 1, Lines 1-9) as associated with the virtual device. (Augmented Reality Representation of Physical Device) The augmented reality representation would be an interactive 3D object. Such objects in AR spaces have their positions tracked through anchors, and are mapped to the AR space that is based on the physical space. Which means the position of these Augmented Reality Representations are linked to the Control device and the IoT Device the representation is based on. The representations can display the states and other data associated with the physical IoT device virtually.
Regarding claim 4, Ross teaches the electronic device (IoT Device or Control Device, Page 8 Para. 2 Lines 3-11 and Page 19 Para. 3 Lines 10-22) of claim 1, wherein the instructions, when executed by the at least one processor, cause the electronic device (IoT Device or Control Device) to:
identify the voice command (Page 12 Para. 2 Lines 3-7) further indicating that the second image (Context/State of Devices, Page 4 Para. 2-3 and Fig. 3) is obtained from the other electronic device (Other IoT Devices) based on the recognition of the voice signal; (Page 16 Para. 3 Lines 11-15 and Page 16 Para. 4 Lines 16-22) Various IoT devices in the network can download the software application to communicate data and fulfill voice commands. Hence, other IoT devices can share image data to the display.
However, Ross and Desai fail to teach:
and based on the voice command being identified as indicating that the second image is obtained from the other electronic device, transmit, to the other electronic device, through the communication circuitry, the device information of the virtual device.
Park teaches:
and based on the voice command ( being identified as indicating that the second image (Notification Information) is obtained from the other electronic device(Other Device Serving as a Separate Controller Col. 3 Lines 22-31 or other Mobile Terminals Col. 3 Lines 9-17), transmit, to the other electronic device (Other Device Serving as a Separate Controller Col. 3 Lines 22-31 or other Mobile Terminals Col. 3 Lines 9-17) through the communication circuitry (Mobile Communication Module 212, Col. 7 Lines 8-23), the device information of the virtual device(Displayed Notification Information Corresponding to Mobile Terminal 200 or Smartphone, Col. 12 Lines 29-38).
Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Ross’s IoT system that can communicate data among the IoT devices to incorporate altered by Desai’s Display through Display to incorporate Park’s HMD that determines if the device is worn or not. Since doing so would provide the benefit of providing feedback to the user when they attempt to interact with a device that may be in a state which can’t be used. Such as, trying to interact with an HMD, but the HMD is not currently on or worn.
Regarding claim 5, Ross fails to explicitly teach the electronic device of claim 4, wherein the second image corresponds to another image displayed through a display of the other electronic device.
However, Desai teaches the electronic device (Smartphone) of claim 4, wherein the second image (Physical Smartphone’s Content) corresponds to another image (Screenshots and Orientation Data, Section: III Setup Page 219 Col. 2 and Page 220 Col. 1) displayed through a display of the other electronic device (Another Smartphone or VR Device). An app is used to share data from a smartphone to the VR Device. Another smartphone could download the same app to share their data to the VR Device. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Ross’s IoT system that shares data to various devices to incorporate Desai’s app that shares image data to a VR device. Since doing so would provide the benefit of allowing multiple electronic devices to share image data to the VR device, increasing the flexibility of the system.
Regarding claim 6, Ross fails to teach the electronic device of claim 4, wherein the display of the virtual device displaying the second image provides an extended displayed in conjunction with a display of the other electronic device.
However, Desai teaches the electronic device (Smartphone) of claim 4, wherein the display of the virtual device (Fig. 6, Section: C. Smartphone Detection After SDSC, Page 222 Col.2 and Page 223 Col. 1) displaying the second image (Physical Smartphone’s Content) provides an extended display in conjunction with a display of the other electronic device (Another Smartphone or VR Device). Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Ross’s IoT system that shares data to various devices to incorporate Desai’s app that shares image data to a VR device. Since doing so would provide the benefit of allowing multiple electronic devices to share image data to the VR device, increasing the flexibility of the system.
Regarding claim 7, Ross teaches the electronic device (IoT Device or Control Device, Page 8 Para. 2 Lines 3-11 and Page 19 Para. 3 Lines 10-22) of claim 1, wherein the instructions, when executed by the at least one processor, cause the electronic device (IoT Device or Control Device) to:
identify, based on the recognition of the voice signal (Page 12 Para. 2 Lines 3-7), that the voice signal indicates displaying (Fig.1, Page 21) performed through a display of the other electronic device (Other IoT Devices) and (Augmented Reality Representation of Physical Device, Page 4 Para. 1 Lines 1-8); A display can be an IoT device or a screen of an IoT device. It is known in the art that an electrical device can display what its displaying through to another electrical device. For example, casting a YouTube video from a cell phone to a TV.
and identify the voice command (Page 12 Para. 2 Lines 3-7), based on identifying that a position (Location) of the electronic device (IoT Device or Control Device) identified through the communication circuitry corresponds to, among a position (Location) of the virtual device (Augmented Reality Representation of Physical Device, Page 4 Para. 1 Lines 1-8) registered using the software application (Page 9, Para. 1, Lines 1-9) as associated with the virtual device and a position (Location) of the other electronic device (IoT Device or Control Device) registered using the software application as associated with the other electronic device, the position (Location) of the virtual device. (Page 15, Para. 3, Lines 8-22 and Page 16, Para. 1, Lines 1-6) As mentioned above the location of each IoT device in the system is known and the locations of the Augmented Reality Representations of them are known as well. As information is shared between the IoT devices and the Augmented Reality Representations.
However, Ross fails to teach the displaying performed through the display of the virtual device.
Desai teaches the displaying performed through the display of the virtual device (Virtual Smartphone). (Fig. 6, Section: C. Smartphone Detection After SDSC, Page 222 Col.2 and Page 223 Col. 1) Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Ross’s IoT Device that can Control a Head Mounted Display (HMD) through Voice Commands to incorporate Desai’s Head Mounted VR Device that can display an image through the VR Device to a Virtual Device. Since doing so would provide the benefit of displaying images from a physical device to a virtual one that mimics it. Which allows a user to interact with the physical device through the VR device without having to remove their VR Device.
Regarding claim 9, Ross teaches the electronic device (IoT Device or Control Device, Page 8 Para. 2 Lines 3-11 and Page 19 Para. 3 Lines 10-22) of claim 1, wherein the instructions, when executed by the at least one processor, cause the electronic device (IoT Device or Control Device) to:
identify, based on the recognition of the voice signal (Page 12 Para. 2 Lines 3-7), that the voice signal indicates (Augmented Reality Representation of Physical Device, Page 4 Para. 1 Lines 1-8) and displaying (Fig.1, Page 21) performed through a display of the other electronic device (Other IoT Device or Control Device) registered by using the software application; (Page 16, Para. 3, Lines 11-15)
identify (Page 7, Para. 3, Lines 13-21) that a position(location) of the other electronic device (Other IoT Devices) registered by using the software application (Page 16, Para. 3, Lines 11-15 as associated with the other electronic device (IoT Device or Control Device) and a position (Location) of the virtual device (Augmented Reality Representation of Physical Device, Page 4 Para. 1 Lines 1-8) registered using the software application as associated with the virtual device (Augmented Reality Representation of Physical Device) are included in one region; (Device Topology Representation, Page 15, Para. 3, Lines 8-22 and Page 16, Para. 1, Lines 1-6) As stated above all the IoT device locations are known and the Augmented Reality Representations locations are known as well as they are tracked by the Augmented Reality Space. Since, the IoT device and Augmented Reality Representations share data they can easily share image data. As well as specify the IoT device based on the region it is in, which can be the same region like a room. (Page 7, Para. 3, Lines 13-21)
based on identifying (Page 7, Para. 3, Lines 13-21) that the position (Location) of the other electronic device (Other IoT Device) and the position (Location) of the virtual device (Augmented Reality Representation of Physical Device) are included in the one region (Device Topology Representation, Page 15, Para. 3, Lines 8-22 and Page 16, Para. 1, Lines 1-6), identify that the virtual device(Augmented Reality Representation of Physical Device) from among the virtual device (Other Augmented Reality Representations of Other Physical Devices) and the other electronic device (Other IoT Device) is positioned in front (The location of the IoT Devices and Users Current location, Page 5 Para. 1) of the electronic device(IoT Device or Control Device, Page 8 Para. 2 Lines 3-11), in accordance with the position (Location) of the virtual device(Augmented Reality Representation of Physical Device), the position (Location) of the other electronic device (Other IoT Device), and a position (Location) in the one region of the electronic device (IoT Device or Control Device, Page 8 Para. 2 Lines 3-11), wherein the position in the one region of the electronic device(IoT Device or Control Device, Page 8 Para. 2 Lines 3-11 is identified using the communication circuitry(Communications Interface, Page 19 Para. 1 Lines 1-3 and Page 16 Para. 3 Lines 11-15) All IoT Devices/Control Devices and the user’s location are known to each other. (Page 5 Para. 1 and Page 7 Para. 2-3)
and identify (Page 7, Para. 3, Lines 13-21) the voice command (Page 12 Para. 2 Lines 3-7), based on identifying that the virtual device is positioned in front (The location of the IoT Devices and Users Current location, Page 5 Para. 1) of the electronic device (Other IoT devices) A system utilizing voice commands often utilizes phrases and words to determine the voice command. The voice commands can contain location information, such as the room the IoT device is in. (Page 7, Para. 3, Lines 13-21)
However, Ross fails to teach the displaying performed through the display of the virtual device.
Desai teaches the displaying performed through the display of the virtual device (Virtual Smartphone). (Fig. 6, Section: C. Smartphone Detection After SDSC, Page 222 Col.2 and Page 223 Col. 1) Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Ross’s IoT Device that can Control a Head Mounted Display (HMD) through Voice Commands to incorporate Desai’s Head Mounted VR Device that can display an image through the VR Device to a Virtual Device. Since doing so would provide the benefit of displaying images from a physical device to a virtual one that mimics it. Which allows a user to interact with the physical device through the VR device without having to remove their VR Device.
Regarding claim 10, Ross teaches the electronic device (IoT Device or Control Device, Page 8 Para. 2 Lines 3-11 and Page 19 Para. 3 Lines 10-22) of claim 1, wherein the instructions, when executed by the at least one processor, cause the electronic device (IoT Device or Control Device) to:
receive a signal indicating a state (Page 4 Para. 3 Lines 14-20) of the MR device (IoT Device), through the communication circuit from the MR device connected to the electronic device (IoT Device);
However, Ross Desai fails to explicitly teach:
and identify whether the MR device is worn, based on the received signal.
Park teaches:
and identify whether the MR device is worn (Col. 14, Lines 17-31), based on the received signal.
Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Ross’s Control Device that can query states of other IoT Devices altered by Desai’s HMD display through a display to incorporate Park’s sensors on an HMD that track if the HMD is worn or not. Since doing so would provide the benefit of tracking when a user is actively using the HMD to prevent cases where something is displayed to the HMD, but the user doesn’t see it because they are not wearing the device.
Regarding claim 11, Ross teaches the electronic device (IoT Device or Control Device, Page 8 Para. 2 Lines 3-11 and Page 19 Para. 3 Lines 10-22) of claim 10, wherein the signal indicating the state (Page 4 Para. 3 Lines 14-20) of the MR device (IoT Device) includes information, indicating a movement (Location, Page 7, Para.3, Lines 13-21) of the MR device (IoT Device), obtained through a sensor of the MR device (IoT Device). The Control Device knows the location of all IoT devices. Thus, the device would know if an IoT device moved. As most HMD devices have sensors to track movement and could easily send that data to the Control Device or the Control Device itself could be an HMD.
Regarding claim 12, Ross teaches the electronic device (IoT Device or Control Device, Page 8 Para. 2 Lines 3-11 and Page 19 Para. 3 Lines 10-22) of claim 10, wherein the signal indicating the state (Page 4 Para. 3 Lines 14-20) of the MR device (IoT Device) includes information, indicating a distance between the MR device (IoT Device) and an external electronic device (IoT Device), obtained through a sensor of the MR device (IoT Device). However, Ross doesn’t explicitly teach indicating a distance between the MR device and an external electronic device, obtained through a sensor of the MR device
Desai teaches indicating a distance between the MR device (VR Device) and an external electronic device (Smartphone), obtained through a sensor of the MR device (VR Device). (Fig. 6, Section: A. Image-Based Smartphone Detector Page 220 Col. 1) The VR device detects the edges of the smartphone using a sensor. To make sure the virtual smartphone overlays the physical smartphone the VR device has to know the distance between the VR device and smartphone. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Ross’s IoT Device System to incorporate Desai’s VR Device that detects a physical smartphone and determines the different between them. Since doing so would provide the benefit of displaying images from a physical device to a virtual one that mimics it. Which allows a user to interact with the physical device through the VR device without having to remove their VR Device.
Regarding claim 13, Ross and Desai fail to teach the electronic device of claim 1, wherein the instructions, when executed by the at least one processor, cause the electronic device to:
identify, while the MR device is disconnected from the electronic device
and transmit, while the MR device is disconnected from the electronic device, data for displaying the image through the display of the virtual device through the communication circuitry to an external electronic device providing the virtual space.
However, Park teaches the electronic device of claim 1, wherein the instructions, when executed by the at least one processor, cause the electronic device to:
identify, while the MR device (HMD) is disconnected (HMD is Completely Taken Off or Terminated, Col. 14 Lines 53-63) from the electronic device (Mobile Terminal 200 or Device Serving as Controller, Col.3 Lines 22-47), an image (Notifications) to be displayed through the display of the virtual device (Notifications to be displayed in the HMD) positioned in a virtual space that is accessible using the MR device (HMD), using the software application; (Applications, Col. 6 Lines 37-48) When a notification to be displayed is received and the HMD is completely taken off or terminated. The notification can be displayed on the Mobile Terminal 200 or can be bookmarked for future use, Col. 13, Lines 29-27 and Col. 14 Lines 64-67 to Col. 15 Lines 1-19. This bookmark information containing notifications, tasks, other virtual data, etc.… can be stored in the Mobile Terminal 200, Network, Server, External Mobile Terminal, etc.….
and transmit, while the MR device is disconnected (HMD is Completely Taken Off, Col. 14 Lines 53-63) from the electronic device (Mobile Terminal 200 or Device Serving as Controller, Col.3 Lines 22-47) data for displaying the image through the display of the virtual device (Notifications to be displayed in the HMD) through the communication circuitry to an external electronic device (Network/Servers/External Mobile Terminal, Col. 5 Lines 24-31 and Col. 7 Lines 9-23) providing the virtual space.
Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Ross’s Control IoT Device System altered by Desai’s HMD display through a display to incorporate Park’s method that can direct data to other devices when the HMD has been terminated. Since doing so would provide the benefit of saving data that will be displayed in the HMD when it is not in use and has been terminated. Which prevents the data from being lost and allows the virtual space to preload the data the next time the HMD is in use.
Regarding claim 14, Ross teaches the electronic device (IoT Device or Control Device, Page 8 Para. 2 Lines 3-11 and Page 19 Para. 3 Lines 10-22) of claim 1, further comprising:
a display (Screen of IoT Device), wherein the instructions, when executed by the at least one processor, cause the electronic device (IoT Device or Control Device) to:
identify, based on a recognition of another voice command (Page 12 Para. 2 Lines 3-7) obtained through the microphone, another voice command indicating displaying performed in another position distinct from positions (Locations, age 15, Para. 3, Lines 8-22 and Page 16, Para. 1, Lines 1-6) of a plurality of devices (IoT Device) that include the virtual device (Augmented Reality Representation of Physical Device, Page 4 Para. 1 Lines 1-8), the plurality of device (IoT Device) registered through the software application (Page 16 Para. 3 Lines 11-15);
and based on the other voice command (Page 12 Para. 2 Lines 3-7), display, through the display (Screen of IoT Device) of the electronic device (IoT Device or Control Device), a user interface of the software application (Page 16 Para. 3 Lines 11-15) for registering another virtual device (Augmented Reality Representation of Physical Device, Page 4 Para. 1 Lines 1-8) positioned at the other position. (Page 4, Para. 2, Lines 9-13) When a physical device is added to the IoT system an Augmented Reality Representation can be created for the device at various locations. As the user interface can be an augmented one, depending on if the control device is an HMD.
Regarding claim 15, Ross teaches the electronic device of claim 14, wherein the instructions, when executed by the at least one processor, cause the electronic device (IoT Device or Control Device, Page 8 Para. 2 Lines 3-11 and Page 19 Para. 3 Lines 10-22) to:
obtain device information of the other virtual device (Augmented Reality Representation of Physical Device, Page 4 Para. 1 Lines 1-8), through the user interface; (Page 9, Para. 1, Lines 1-9) The user interface is from a software application on the control device. The control device can be any IoT device, hence the user interface can be just for adding physical devices or can be the augmented reality interface when the control device is a HMD, Page. 4, Para. 1, Lines 1-8)
and register, using the software application (Page 16 Para. 3 Lines 11-15), the other virtual device (Augmented Reality Representation of Physical Device) associated with the device information (Page 7 Para. 2 Lines 5-12) and the other position (Locations, age 15, Para. 3, Lines 8-22 and Page 16, Para. 1, Lines 1-6).
Regarding method claim 16, is drawn to the method of using the corresponding device claimed in claim 1. Therefore, it is rejected under the same rationale as claim 1.
Regarding claim 17, has similar limitations as of claim 2, therefore it is rejected under the same rationale as claim 2.
Regarding claim 18, has similar limitations as of claim 3, therefore it is rejected under the same rationale as claim 3.
Regarding claim 19, has similar limitations as of claim 4, therefore it is rejected under the same rationale as claim 4.
Regarding claim 20, has similar limitations as of claim 7, therefore it is rejected under the same rationale as claim 7.
Regarding claim 21, Ross teaches an electronic device (IoT Device or Control Device, Page 8 Para. 2 Lines 3-11 and Page 19 Para. 3 Lines 10-22) comprising:
a microphone; (Page 12 Para. 2 Lines 3-5)
communication circuitry; (Communications Interface, Page 19 Para. 1 Lines 1-3 and Page 16 Para. 3 Lines 11-15)
memory, storing instructions, comprising one or more storage media; (Page 18 Para. 4 Lines 15-22)
and at least one processor comprising processing circuitry (Processor core, microprocessor, chip, controller), wherein the instructions, when executed by the at least one processor, cause the electronic device to: (Page 18 Para. 4 Lines 15-22)
receive a voice signal (Capture Audio Signals, Page 12 Para. 2 Lines 3-9) through the microphone;
based on a recognition of the voice signal (Page 12 Para. 2-3 Lines 3-12), identify a voice command (Page 12 Para. 2 Lines 3-7) that indicates (Fig.1, Page 21) (Augmented Reality Representation of Physical Device, Page 4 Para. 1 Lines 1-8), having a display (An IoT Physical Device Display, Page 19, Para. 3), registered using a software application (Page 16 Para. 3 Lines 11-15) for controlling another device through the electronic device (IoT Device or Control Device, Page 8 Para. 2 Lines 3-11),
However, Ross fails to explicitly teach:
displaying content through a virtual device, and the virtual device configured to be displayed through a display of a mixed reality(MR) device as superimposed on a first image obtained through a camera of the MR device;
based on the voice command, identify whether the MR device is worn, wherein the identifying of whether the MR device is worn is performed before performing obtaining a second image to be displayed through the virtual device;
in response to identifying that the MR device is worn, identify device information of the virtual device registered using the software application as associated with the virtual device;
based on the device information of the virtual device, generate the second image to be displayed via a display of the virtual device;
and transmit, through the communication circuitry to the MR device, a signal for displaying the second image through the display of the virtual device to be displayed on the first image through the MR device.
Desai teaches:
displaying content through a virtual device (Fig. 6, Section: C. Smartphone Detection After SDSC, Page 222 Col.2 and Page 223 Col. 1) having a display (Display Content from Smartphone) and the virtual device configured to be displayed through a display of a mixed reality(MR) device (VR Device) as superimposed on a first image (Virtual/Physical Environment) obtained through a camera of the MR device; The content from the smartphone is displayed on top of the virtual/physical environment shown in the MR device. A display of a smartphone is the screen which encompasses all of the content. Thus, the displayed content of the smartphone is the display when recreated virtually.
based on the device information of the virtual device(Virtual Smartphone), generate the second image (Physical Smartphone’s Content) to be displayed via a display of the virtual device(Virtual Smartphone); (Fig. 6, Section: C. Smartphone Detection After SDSC, Page 222 Col.2 and Page 223 Col. 1)
and transmit, through the communication circuitry to the MR device (VR Device), a signal for displaying the second image (Physical Smartphone’s Content) through a display of the virtual device (Virtual Smartphone) to be displayed on the first image (Virtual/Physical Environment) through the MR device (VR Device). (Fig. 6, Section: C. Smartphone Detection After SDSC, Page 222 Col.2 and Page 223 Col. 1)
Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Ross’s IoT Device that can Control a Head Mounted Display (HMD) through Voice Commands to incorporate Desai’s Head Mounted VR Device that can display an image through the VR Device to a Virtual Device. Since doing so would provide the benefit of displaying images from a physical device to a virtual one that mimics it. Which allows a user to interact with the physical device through the VR device without having to remove their VR Device.
However, Ross and Desai fail to explicitly teach:
based on the voice command, identify whether the MR device is worn, wherein the identifying of whether the MR device is worn is performed before performing obtaining a second image to be displayed through the virtual device;
in response to identifying that the MR device is worn, identify device information of the virtual device registered using the software application as associated with the virtual device;
Park teaches:
based on the voice command, (Col. 5 Lines 32-41) identify whether the MR device (HMD) is worn(Col. 14 Lines 14-39), wherein the identifying of whether the MR device worn is performed before (Before Notification Information is Displayed, Col. 14 Lines 17-31) performing obtaining a second image (Notification Information) to be displayed through the virtual device (Displayed Notification Information Corresponding to Mobile Terminal 200 or Smartphone, Col. 12 Lines 29-38) ; The HMD tracks if the HMD is worn or not by various sensors. To determine when to display a notification to the user.
in response to identifying that the MR device is worn(Col. 14 Lines 14-39), identify device information of the virtual device registered using the software application(Application Program, Col. 9 Lines 28-38) as associated with the virtual device(Displayed Notification Information Corresponding to Mobile Terminal 200 or Smartphone, Col. 12 Lines 29-38);
Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Ross’s system of controlling an HMD altered by Desai’s HMD display through display to incorporate Parks’ HMD that tracks if the device is worn or not. Since doing so would provide the benefit of tracking when a user is actively using the HMD to prevent cases where something is displayed to the HMD, but the user doesn’t see it because they are not wearing the device.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to BRIANNA R COCHRAN whose telephone number is (571)272-4671. The examiner can normally be reached Mon-Fri. 7:30am - 5:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Alicia Harrington can be reached at (571) 272-2330. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/BRIANNA RENAE COCHRAN/Examiner, Art Unit 2615
/ALICIA M HARRINGTON/Supervisory Patent Examiner, Art Unit 2615
Prosecution Insights