DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-11 and 17-18 are rejected under 35 U.S.C. 103 as being unpatentable over Ramalingam et al. (US 20180270542 A1) (Hereinafter referred to as Ramalingam) in view of Takiguchi et al. (JP 2002245490) (Hereinafter referred to as Takiguchi) and in further view of Teraoka et al. (JP 2010091728 A) (Hereinafter referred to as Teraoka).
Regarding Claim 1, Ramalingam discloses A control system comprising: a memory storing instructions; and at least one processor configured to execute the instructions to: (See Abstract, “A display control system to generate a virtual environment in a vehicle . . .” See [0042], “The vehicle 102 may comprise the ECU 120 that may include a microprocessor 202 and a memory 204.”)
acquire data in an imaging device installed in a real space is associated with a frame in a virtual environment; (See [0037], “In various exemplary embodiments, the video-capture unit 134a may be installed at the front of the vehicle 106 to capture a front view outside the vehicle 106. The video-capture units 134b and 134c may be installed at the left side, for example, near the front and side windows, of the vehicle 106.” Here, the video capture unit would correspond to an “imaging device” which is installed in a real space.
See [0059], “FIGS. 3A to 3F illustrate various operations of the disclosed display control system and method to generate a virtual environment in a vehicle . . .” See Fig. 3F showing a the generation a virtual environment in a vehicle, with the windows that displays video 320a – 320c.
See [0019], “In accordance with an embodiment, an augmented-reality view may be generated on one or more windows and/or windshields of the vehicle. The one or more windows and/or windshields may correspond to the one or more display mediums.” See [0079], “For example, the videos 320a to 320c may correspond to the first group of related videos (FIG. 1) previously captured by the video-capture units 134a to 134f, and the selected video 320 may be one of the first group of related videos. The video 320a that includes the front view outside the vehicle 106 captured previously by the video-capture unit 134a, may be displayed on the UI 228c rendered on the front windshield 324c. Similarly, the video 320b that includes the left side view outside the vehicle 106 captured previously by the video-capture unit 134b, may be displayed on the UI 228b rendered on the left front window 324b.”
Here, it is understood that a limitation of a “frame” would be correspond to a window. Note that this this interpretation is supported by dependent claims 9-10. Thus, Ramalingam teaches a window (a frame) associated with a video-capture unit, in which the window is used to generate a virtual environment in a vehicle.)
generate, from a video captured by an imaging device, a video of the frame seen by the user in the virtual environment; and (See [0059], “FIGS. 3A to 3F illustrate various operations of the disclosed display control system and method to generate a virtual environment in a vehicle . . .”
See Fig. 3F showing a the generation a virtual environment in a vehicle, with the windows that displays video 320a – 320c.
See [0079], “For example, the videos 320a to 320c may correspond to the first group of related videos (FIG. 1) previously captured by the video-capture units 134a to 134f, and the selected video 320 may be one of the first group of related videos.”
See [0078], “Each of the videos 320a to 320c may correspond to different FOVs of an external environment of same vehicle at a previous time instance or external environment of another vehicle, such as the vehicle 106 when the vehicle 106 was in motion along the second travel route 138 in the second geographical area 108.”
In this case, the video captured by the video capture unit is displayed in the corresponding window to generate a virtual environment in a vehicle. One can consider the video to be displayed on the window as being generated from a video captured by the video capture unit. This would correspond to limitation of generating from a video captured by an imaging device, a video of the frame seen by a user in the virtual environment.)
display the generated video of the frame in the frame displayed on a display device. (See Fig. 3F showing a the generation a virtual environment in a vehicle, with the windows that displays video 320a – 320c.)
However, Ramalingam fails to explicitly disclose acquire data in which imaging device identification information for identifying an imaging device installed in a real space is associated with frame identification information for identifying a frame in a virtual reality space;
generate, from a video captured by an imaging device identified by the imaging device identification information associated with the frame identification information for identifying the frame based on a positional relationship in the virtual reality space between a user in the virtual reality space and the frame, a video of the frame seen by the user in the virtual reality space; and
Takiguchi teaches acquire data in an imaging device installed in a real space is associated with a frame in a virtual reality space; (See [0003], “The viewer can freely operate the joystick or controller to freely move in the virtual space and see the displayed image to experience as if they were actually moving in that space. You can. As another method, for example, a viewer wears a head-mounted display . . .” Also see Figs. 19A and 19B showing a virtual window D (frame) within a virtual reality space.
In combination with Ramalingam which already teaches the video capture units (imaging device) being associated with display windows (frame), instead of display windows used to generate a virtual environment, one can display the video from the video capture unit to a window in virtual reality space instead.)
generate, from a video captured by an imaging device based on a positional relationship in the virtual reality space between a user in the virtual reality space and the frame, a video of the frame seen by the user in the virtual reality space; and (See [0015], “The movement detecting unit 35 is provided integrally with the head mounted display 34, and detects the position of the viewer (position detecting means, viewpoint position detecting means, walking detecting device) 36, and the orientation of the viewer.” See FIG. 19A and 19B showing a virtual window D with an image of a scenery seen through the window.
See [0043], “Then, as shown by an arrow J2 in FIG. 17(b), when the viewer changes its direction, the movement detecting unit 35 detects this, and the reproduced image display unit 33 displays it as shown in FIG. 19(a). Window D and the image of the scene seen through window D is displayed. Here, when the viewer approaches the window D as indicated by arrow J3 in FIG. 17B, as shown in FIG. 19B, the scenery seen through the window D and the window D at a closer position is displayed.”
Here, Takiguchi teaches that the image of the scenery seen by the viewer in the virtual space would change depending upon the positioning and orientation of the view. This can be seen from Fig. 19B with the image changing as the viewer gets closer to the virtual window D. When combined with Ramalingam teaching a window displaying a captured video of the outside scenery, then with the feature of dynamically changing the scene outside the window based on the positioning and orientation of the viewer taught by Takiguchi, then it would result in the generation of a video of the frame seen by the user “based on a positional relationship in the virtual reality space between a user in the virtual reality space and the frame”.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Ramalingam with Takiguchi to include having the window be in virtual reality space and have the scenery depicted in the window be based on the positional relationship between the user and the window.
The motivation to combine Ramalingam with Takiguchi would have been obvious as both arts are related to generating a virtual environment and specifically having windows which depict the outside scenery (See Takiguchi [0043], and Figs. 19A-19B). The benefit of specifically using a virtual reality window and using positional relationship between the viewer and the window when displaying the outside scenery is that it would provide a more immersive and realistic experience. This is showcased by Takiguchi Figs. 19A-19B in which the outside scene changes when the viewer moves closer to the window. Similar to the real world, one can see more of the outside scenery if they are closer to the window, and thus mimicking this in virtual reality would be obvious.
However, Ramalingam in view of Takiguchi still fails to explicitly disclose acquire data in which imaging device identification information for identifying an imaging device installed in a real space is associated with frame identification information for identifying a frame in a virtual reality space;
generate, from a video captured by an imaging device identified by the imaging device identification information associated with the frame identification information for identifying the frame based on a positional relationship in the virtual reality space between a user in the virtual reality space and the frame, a video of the frame seen by the user in the virtual reality space; and
Teraoka teaches acquire data in which imaging device identification information for identifying an imaging device installed in a real space is associated with frame identification information for identifying a frame in a virtual reality space; (See Page 7 Paragraph 2, “FIG. 8 is a diagram illustrating window information. In this example, the window information includes an identifier (window ID) that identifies the window and the coordinates of the vertex of the window frame.”
See page 7 Paragraph 4, “FIG. 9 is a diagram illustrating correspondence information. In this example, the correspondence information includes a window ID and an identifier (image forming apparatus ID) of the image forming apparatus 40 that forms an image in the window. That is, the correspondence information includes information that makes the plurality of image forming apparatuses 40 and the plurality of windows correspond one-to-one.”
Here, Teraoka teaches that imaging forming apparatuses would usually have an identifier (imaging device identification information for identifying an imaging device) and that it can have a correspondence with a window which can also have identifier (associated with frame identification information for identifying a frame). It should be noted that Ramalingam already teaches a one to one correspondence between the windows and video capture devices (See Ramalingam [0079]), Teraoka simply shows that this correspondence can be done through IDs for the video capturing units and windows.)
generate, from a video captured by an imaging device identified by the imaging device identification information associated with the frame identification information for identifying the frame based on a positional relationship in the virtual reality space between a user in the virtual reality space and the frame, a video of the frame seen by the user in the virtual reality space; and (See Page 7 Paragraph 2 and Page 7 Paragraph 4 teaching window IDs and image forming apparatuses that correspond one-to-one to the plurality of windows.
In combination with Ramalingam already teaching a one to one correspondence between the windows and video capture devices, and the generation of a video to display on the window and Takiguchi teaching to use a positional relationship between a user and window in virtual reality space, then the above limitations are taught.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Ramalingam in view of Takiguchi with Teraoka to include having identification information for the imaging apparatus associated with window identification information.
The motivation to combine Ramalingam in view of Takiguchi with Teraoka would have been obvious as both Ramalingam and Teraoka are related to imaging devices and windows (See Teraoka Page 7 Paragraph 4). Ramalingam already teaches having a correspondence between specific video capture units and windows (See Ramalingam [0079]), so Teraoka also teaching to have a correspondence between imaging devices and windows using IDs would have been an obvious implementation to a person of ordinary skill of the art. The benefit of having and associating IDs for the imaging devices and windows is that it would make it easy to map specific imaging devices to specific windows.
Regarding Claim 2, Ramalingam in view of Takiguchi and Teraoka discloses The control system according to claim 1, wherein the at least one processor is further configured to execute the instructions to: cutout the video of the frame seen by the user in the virtual reality space from the video captured by the imaging device based on the positional relationship in the virtual reality space between the user in the virtual reality space and the frame. (See Ramalingam [0078], [0059], and [0079] teaching to generate videos that correspond to the FOVs of an external environment captured by video-capture units and displaying on windows to generate a virtual environment.
See Takiguchi Figs. 19A and 19B showing a virtual window D in virtual reality space with an image of a scenery seen through the window. Specifically, Fig. 19A has the left and right side that have the trees in the scenery cutout. This would be based on the position of the user in the virtual reality space and the frame. The combination would result in the video of Ramalingam having a cutout based on the position of the user, and this would be for a window in virtual reality space. The motivation to combine would have been similar to that of Claim 1 rejection motivation.)
Regarding Claim 3, Ramalingam in view of Takiguchi and Teraoka discloses The control system according to claim 1, wherein the at least one processor is further configured to execute the instructions to: generate a new video of the frame seen by the user in the virtual reality space from the video captured by the imaging device based on the positional relationship in the virtual reality space between the user in the virtual reality space and the frame. (See Ramalingam [0078], [0059], and [0079] teaching to generate videos that correspond to the FOVs of an external environment captured by video-capture units and displaying on windows to generate a virtual environment.
See Takiguchi Figs. 19A and 19B showing a virtual window D in virtual reality space with an image of a scenery seen through the window, noting that the proportion of the outside scenery displayed would depend on the user’s positioning relative to the window. The combination would result in the video of Ramalingam changing based on the position of the user and this would be for a window in virtual reality space. This dynamically changing video view can be considered as the generation of a “new video”. The motivation to combine would have been similar to that of Claim 1 rejection motivation.)
Regarding Claim 4, Ramalingam in view of Takiguchi and Teraoka discloses The control system according to claim 1,wherein the at least one processor is further configured to execute the instructions to: acquire data in which imaging device identification information for identifying each of a plurality of imaging devices installed at different positions is associated with the frame identification information for identifying the frame, and (See Ramalingam [0037], “In various exemplary embodiments, the video-capture unit 134a may be installed at the front of the vehicle 106 to capture a front view outside the vehicle 106. The video-capture units 134b and 134c may be installed at the left side, for example, near the front and side windows, of the vehicle 106.”
Also see Ramalingam [0079], “For example, the videos 320a to 320c may correspond to the first group of related videos (FIG. 1) previously captured by the video-capture units 134a to 134f. . .” Here, Ramalingam teaches video-capture units (a plurality of imaging devices) which are installed at different positions and are associated with different windows (frames).
See Teraoka Page 7 Paragraph 2 and Page 7 Paragraph 4 teaching window IDs and image forming apparatuses that correspond one-to-one to the plurality of windows.)
generate the video of the frame seen by the user in the virtual reality space from a plurality of videos captured by the plurality of imaging devices identified by the plurality of imaging device identification information associated with the frame identification information for identifying the frame based on the positional relationship in the virtual reality space between the user in the virtual reality space and the frame. (See Ramalingam Fig. 3F showing a the generation a virtual environment in a vehicle, with the windows that displays video 320a – 320c. Further see Ramalingam [0079], “For example, the videos 320a to 320c may correspond to the first group of related videos (FIG. 1) previously captured by the video-capture units 134a to 134f, and the selected video 320 may be one of the first group of related videos. The video 320a that includes the front view outside the vehicle 106 captured previously by the video-capture unit 134a, may be displayed on the UI 228c rendered on the front windshield 324c. Similarly, the video 320b that includes the left side view outside the vehicle 106 captured previously by the video-capture unit 134b, may be displayed on the UI 228b rendered on the left front window 324b.”
See Takiguchi Figs. 19A and 19B showing a virtual window D in virtual reality space with an image of a scenery seen through the window, noting that how much of the outside scenery displayed would depend on the user’s positioning relative to the window. The combination would result in the video of Ramalingam changing how much is shown based on the position of the user and this would be for a window in virtual reality space. The motivation to combine would have been similar to that of Claim 1 rejection motivation.)
Regarding Claim 5, Ramalingam in view of Takiguchi and Teraoka discloses The control system according to claim 1,wherein the at least one processor is further configured to execute the instructions to: generate, for each of a plurality of users, a video of the frame seen by the each user in the virtual reality space from the video captured by the imaging device based on a positional relationship in the virtual reality space between a position of the each user in the virtual reality space and the frame. (See Ramalingam [0021], “There is further shown a second user 136 associated with the vehicle 106 . . .” See Ramalingam [0078], [0059], and [0079] teaching to generate videos that correspond to the FOVs of an external environment captured by video-capture units and displaying on windows to generate a virtual environment.
See Takiguchi [0003], “The viewer can freely operate the joystick or controller to freely move in the virtual space and see the displayed image to experience as if they were actually moving in that space. You can. As another method, for example, a viewer wears a head-mounted display . . .” Further see Takiguchi Figs. 19A and 19B showing a virtual window D in virtual reality space with an image of a scenery seen through the window, noting that how much of the outside scenery display would depend on the user’s positioning relative to the window. The combination would result in the video of Ramalingam changing how much is shown based on the position of the user and this would be for a window in virtual reality space. Since Ramalingam teaches that there can be second user, then when combined with Takiguchi, the second user would be a viewer that wears their own head-mounted display. If that second user is in a different position than a first user to the window, then the video outside of the window would be showing a different view depending on how close they are to the window. The motivation to combine would have been similar to that of Claim 1 rejection motivation.)
Regarding Claim 6, Ramalingam in view of Takiguchi and Teraoka discloses The control system according to claim 1,wherein the at least one processor is further configured to execute the instructions to: generate a new video based on a positional relationship in the virtual reality space between the user and the frame in the virtual reality space and an orientation of the user in the virtual reality space. (See Ramalingam [0078], [0059], and [0079] teaching to generate videos that correspond to the FOVs of an external environment captured by video-capture units and displaying on windows to generate a virtual environment.
See Takiguchi [0015] teaching detecting the position and orientation of the viewer. See Takiguchi Figs. 19A and 19B showing a virtual window D in virtual reality space with an image of a scenery seen through the window, noting that how much of the outside scenery display would depend on the user’s positioning and orientation relative to the window. The combination would result in the video of Ramalingam changing how much is shown based on the position of the user and this would be for a window in virtual reality space. This can be considered as generating a new video based on a positional relationship. The motivation to combine would have been similar to that of Claim 1 rejection motivation.)
Regarding Claim 7, Ramalingam in view of Takiguchi and Teraoka discloses The control system according to claim 1, the at least one processor is further configured to execute the instructions to: associate a position and an orientation of the user in the real space with a position and an orientation of the user in the virtual reality space; (See Takiguchi [0015], “The movement detecting unit 35 is provided integrally with the head mounted display 34, and detects the position of the viewer (position detecting means, viewpoint position detecting means, walking detecting device) 36, and the orientation of the viewer.” Here, Takiguchi teaches detecting the position and orientation of the viewer which is done through the head mounted display (HMD), and thus would implicitly mean that the HMD is measuring and associating the position and orientation of the user in real space with the position and orientation of the user in virtual reality space.)
detect movement of the user in the real space; (See Takiguchi [0015], “. . . the walking of the viewer is detected and the number of steps of the walking is detected. The position is detected by detecting the amount of movement of the viewer.”)
identify a position and an orientation of the user in the virtual reality space based on the movement of the user in the real space, wherein generate a new video based on a positional relationship in the virtual reality space between the user in the virtual reality space at the identified position and the frame and the identified orientation. (See Takiguchi [0005], “On the other hand, when the viewer actually moves using the head-mounted display, the viewer's position in the virtual space must be determined by moving the viewer, more specifically, the viewer's position and orientation.”
See Takiguchi [0043], “Then, as shown by an arrow J2 in FIG. 17(b), when the viewer changes its direction, the movement detecting unit 35 detects this, and the reproduced image display unit 33 displays it as shown in FIG. 19(a). Window D and the image of the scene seen through window D is displayed. Here, when the viewer approaches the window D as indicated by arrow J3 in FIG. 17B, as shown in FIG. 19B, the scenery seen through the window D and the window D at a closer position is displayed.”
See Takiguchi Figs. 19A and 19B showing a virtual window D in virtual reality space with an image of a scenery seen through the window, noting that how much of the outside scenery display would depend on the user’s positioning relative to the window. The combination would result in the video of Ramalingam changing how much is shown based on the position of the user and this would be for a window in virtual reality space. This can be considered as generating a new video based on a positional relationship. The motivation to combine would have been similar to that of Claim 1 rejection motivation.)
Regarding Claim 8, Ramalingam in view of Takiguchi and Teraoka discloses The control system according to claim 1, the at least one processor is further configured to execute the instructions to: receive movement of the user in the virtual reality space by an operation by the user in the real space; and (See Takiguchi [0005], “On the other hand, when the viewer actually moves using the head-mounted display, the viewer's position in the virtual space must be determined by moving the viewer, more specifically, the viewer's position and orientation.”
See Takiguchi [0015], “Here, the position detecting device 36 may use an acceleration sensor or a geomagnetic sensor as in the compound-eye image capturing system shown in FIG. 1, but in the present embodiment, the walking of the viewer is detected and the number of steps of the walking is detected. The position is detected by detecting the amount of movement of the viewer.”)
identify a position and an orientation of the user in the virtual reality space based on the received movement, wherein generate a new video based on a positional relationship in the virtual reality space between the user in the virtual reality space at the identified position and the frame, and the identified orientation. (See Takiguchi [0005] and [0015] teaching to identify a position and an orientation of the user in the virtual reality space based on the received movement.
See Takiguchi Figs. 19A and 19B showing a virtual window D in virtual reality space with an image of a scenery seen through the window, noting that how much of the outside scenery display would depend on the user’s positioning relative to the window. The combination would result in the video of Ramalingam changing how much is shown based on the position of the user and this would be for a window in virtual reality space. This can be considered as generating a new video based on a positional relationship. The motivation to combine would have been similar to that of Claim 1 rejection motivation.)
Regarding Claim 9, Ramalingam in view of Takiguchi and Teraoka discloses The control system according to claim 1, wherein the frame is a window or a door. (See Ramalingam [0019], “In accordance with an embodiment, an augmented-reality view may be generated on one or more windows and/or windshields of the vehicle. The one or more windows and/or windshields may correspond to the one or more display mediums.”)
Regarding Claim 10, Ramalingam in view of Takiguchi and Teraoka discloses The control system according to claim 9, wherein the window is a vehicle window or a building window, and the door is a vehicle door or a building door. (See Ramalingam [0019], “In accordance with an embodiment, an augmented-reality view may be generated on one or more windows and/or windshields of the vehicle. The one or more windows and/or windshields may correspond to the one or more display mediums.”)
Regarding Claim 11, Ramalingam in view of Takiguchi and Teraoka discloses The control system according to claim 10, wherein the imaging device is installed at a mobile object in the real space. (See Ramalingam [0037], “In various exemplary embodiments, the video-capture unit 134a may be installed at the front of the vehicle 106 to capture a front view outside the vehicle 106. The video-capture units 134b and 134c may be installed at the left side, for example, near the front and side windows, of the vehicle 106.”)
Regarding Claim 17, Ramalingam in view of Takiguchi and Teraoka discloses A control method comprising: (See Ramalingam [0002], “Various embodiments of the disclosure relate to a display control system and method for a vehicle.”)
acquiring data in which imaging device identification information for identifying an imaging device installed in a real space is associated with frame identification information for identifying a frame in a virtual reality space; generating, based on a positional relationship in the virtual reality space between a user in the virtual reality space and the frame, a video of the frame seen by the user in the virtual reality space from a video captured by an imaging device identified by the imaging device identification information associated with the frame identification information for identifying the frame; and displaying the generated video of the frame in the frame displayed on a display device. (The above limitations are similar to those of Claim 1 and thus are rejected under a similar rationale as that of Claim 1.)
Regarding Claim 18, Ramalingam in view of Takiguchi and Teraoka discloses A non-transitory computer-readable recording medium that records a program, the program causing a computer to execute the steps of: (See Ramalingam [0121], “Various embodiments of the disclosure may provide a non-transitory computer readable medium and/or storage medium having stored thereon, a set of computer-executable instructions to cause a machine and/or a computer to generate a virtual environment in a vehicle (such as the vehicle 102).”)
acquiring data in which imaging device identification information for identifying an imaging device installed in a real space is associated with frame identification information for identifying a frame in a virtual reality space; generating, based on a positional relationship in the virtual reality space between a user in the virtual reality space and the frame, a video of the frame seen by the user in the virtual reality space from a video captured by an imaging device identified by the imaging device identification information associated with the frame identification information for identifying the frame; and displaying the generated video of the frame in the frame displayed on a display device. (The above limitations are similar to those of Claim 1 and thus are rejected under a similar rationale as that of Claim 1.)
Claims 12-14 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Ramalingam in view of Takiguchi and Teraoka and in further view of Vandyke et al. (US 11812194 B1) (Hereinafter referred to as Vandyke).
Regarding Claim 12, Ramalingam in view of Takiguchi and Teraoka fails to explicitly disclose The control system according to claim 1,wherein the at least one processor is further configured to execute the instructions to: display a graphic representing a voice of another user other than a user having a conversation with the user in the virtual reality space.
Vandyke teaches display a graphic representing a voice of another user other than a user having a conversation with the user in the virtual reality space. (See Col 1 Lines 35 – 48, “The private conversation may take place while both users are participating in a computer-generated reality (CGR) setting (or environment). . . In addition, speech of each user may be picked up by microphones of the user's HMD and be projected into the CGR setting, as if said by the user's avatar.”
See Col 5 Lines 55 – 59, “Stage 100 illustrates user 1 120 talking to user 2 125 and another user who is participating in the virtual meeting. In particular, the avatar 135 is projecting speech 155 into the CGR setting 115, which is being heard by avatars 140 and 145.”
See Col 6 Lines 35 – 41, “In response to determining that user 1 120 wishes to engage in the private conversation, the electronic device 130 is configured to cause a privacy cloak 160 to activate in the CGR setting 115. The privacy cloak 160 is a visual indication presented in the CGR setting 115 to represent that users associated with avatars within the confines of the cloak 160 are having a private conversation.” Lastly, see Fig. 1 showing multiple users in CGR having conversations with each other as well as the visualization of a privacy cloak and conversations between users.
In summary, Vandyke teaches computer-generated reality (CGR), the equivalent of VR, and multiple users being able to have conversations using avatars within CGR. When one of the users want to engage in a private conversation, a privacy cloak is activated with a visual indication in the CGR in which other users are able to see that a private conversation is happening. In this case, when considering perspective of the user of avatar 150 from Fig. 1, that user sees a visual indication of a privacy cloak for users 1 and 2. Thus, one can consider the privacy cloak visual indication to correspond to “a graphic representing a voice of another user other than a user having a conversation with the user in the virtual reality space”.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Ramalingam in view of Takiguchi and Teraoka with Vandyke to include users being able to have conversations and a graphic representing the voice of another user other than a user having a conversation with the user.
The motivation to combine Ramalingam in view of Takiguchi and Teraoka with Vandyke would have been obvious as both Takiguchi and Vandyke are related to VR/CGR (See Vandyke Col 1 Lines 35 – 48). In this case, audio is a common component for a VR experience, and in the case where there are more than one user, including the ability to have conversations between them would provide a better experience. Specifically including a graphic representing the voice of another user like a privacy cloak visual indication, allows users to be aware of that other conversations are happening, which in turns brings the benefit of allowing the user to better engage with others.
Regarding Claim 13, Ramalingam in view of Takiguchi, Teraoka, and Vandyke discloses The control system according to claim 12, wherein the at least one processor is further configured to execute the instructions to: receive, from among a plurality of the graphics, selection of a graphic by the user in the real space, wherein output by sound a sound represented by the selected graphic. (See Vandyke Col 21 Lines 38 – 41, “FIG. 5 is a flowchart of one aspect of a process 500 to allow a (e.g., third) user who is not participating in a private conversation between two other users (e.g., a first user and a second user) to join the private conversation.”
Further see Vandyke Col 21 Lines 47 – 65, “The process 500 detects that the third user (e.g., the user of avatar 145) who is not participating in the private conversation is requesting to join the private conversation (at block 510). Specifically, the electronic device 130 may base this determination on an implicit request from the electronic device of the third user or an explicit request. An implicit request may be based on actions performed by the third user. For example, when a privacy cloak is activated, such as the cloak 160 illustrated in FIG. 1, the determination may be based on whether avatar 145 of the third user is within a threshold distance of the privacy cloak 160 and/or within a threshold distance of avatars 135 and/or 140 of users who are already within the cloak 160. This may be similar to a person in a cocktail party walking up to a group of people to interject in a conversation. An explicit request may be an action performed by the third user (and/or an avatar of the user) to join the conversation (and the cloak). For example, the user may select a virtual menu item within the CGR setting through a hand gesture.”
Once again, consider the privacy cloak visualization as the graphic representing a voice of another user. In this case, a third user requesting to join the private conversation represented by the privacy cloak can be considered as the third user making a selection of this specific graphic to join in. Then, Vandyke teaches that the joining can be done through the selection of a menu item using a hand gesture (selection of a graphic by the user in the real space). Once allowed in, then third user would be able to hear the conversation (wherein output by sound a sound represented by the selected graphic). The motivation to combine would have been similar to that of Claim 12 rejection motivation.)
Regarding Claim 14, Ramalingam in view of Takiguchi, Teraoka, and Vandyke discloses The control system according to claim 12, wherein the at least one processor is further configured to execute the instructions to: receive, from among a plurality of the graphics, selection of a graphic by the user in the real space, wherein do not display the selected graphic.(See Vandyke Col 21 Lines 38 – 41 and Col 21 Lines 47 – 65 teaching a third user requesting to join the private conversation, noting that this request can be considered selection of a graphic by the user, and that this can be done using a hand gesture (in the real space).
See Vandyke Col 18 Lines 55 – 60, “In one aspect, users associated with avatars that are within a privacy cloak may not visualize the privacy cloak 160 at all. Specifically, although those outside the privacy cloak may see a barrier, users who are engaging in the private conversation may new the CGR setting normally (e.g., without seeing a barrier surrounding their avatars).”
Once again, the privacy cloak would correspond to the graphic. Once the avatars are within the privacy cloak, it is not visualized anymore and thus correspond to “wherein do not display the selected graphic.” The motivation to combine would have been similar to that of Claim 12 rejection motivation.)
Regarding Claim 16, Ramalingam in view of Takiguchi, Teraoka, and Vandyke disclose The control system according to claim 1, the at least one processor is further configured to execute the instructions to: receive an output format of a voice of the another user, wherein output the voice of the another user in the received output format. (See Vandyke Col 1 Lines 35 – 48, “The private conversation may take place while both users are participating in a computer-generated reality (CGR) setting (or environment). . . In addition, speech of each user may be picked up by microphones of the user's HMD and be projected into the CGR setting, as if said by the user's avatar.” See Vandyke Col 8 Lines 14 – 19, “In another aspect, the server 215 may render audio data of the CGR settings and transmit it (e.g., as data packets) to each of the electronic devices participating within the CGR setting. For example, the electronic devices may transmit microphone signals captured by microphones of the electronic devices.” The motivation to combine would have been similar to that of Claim 12 rejection motivation.)
Claim 15 is rejected under 35 U.S.C. 103 as being unpatentable over Ramalingam in view of Takiguchi, Teraoka, and Vandyke and in further view of Nashida et al. (US 20190121515 A1) (Hereinafter referred to Nashida).
Regarding Claim 15, Ramalingam in view of Takiguchi, Teraoka, and Vandyke The control system according to claim 1, wherein the at least one processor is further configured to execute the instructions to: output by voice a conversation behind the user in the virtual reality space and displays a conversation in front of the user in the virtual reality space graphically. (See Vandyke Col 1 Lines 35 – 48, “The private conversation may take place while both users are participating in a computer-generated reality (CGR) setting (or environment). . . In addition, speech of each user may be picked up by microphones of the user's HMD and be projected into the CGR setting, as if said by the user's avatar.” Also see Fig. 1 showing multiple users in CGR having conversations with each other as well as the visualization of a privacy cloak and conversations between users.)
However, Ramalingam in view of Takiguchi, Teraoka, and Vandyke fails to explicitly disclose The control system according to claim 1,wherein the at least one processor is further configured to execute the instructions to: in a case where the display device is a hemispherical dome-shaped display,
Nashida teaches in a case where the display device is a hemispherical dome-shaped display, (See [0001], “The technology disclosed in this specification relates to an information processing device and an information processing method that mediates interactions between a user and information, and for example, relates to an information processing device and an information processing method that mediates interactions between information in a virtual reality (VR) space in which a first-person view or the like is viewed, and a user.”
See [0099], “Basically, it is sufficient for the image display device 102 to be provided with a configuration enabling the viewing of a first-person view photographed by the image provision device 101, and if a device by which a wide-angle field of view is obtained, such as a dome-type (hemispherical or aspherical) display or an immersive head-mounted display, is applied to the display unit 514, an image having a higher sense of immediacy can be viewed, and the viewer 112 is able to experience the same sight as the observer 111 more realistically.”)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Ramalingam in view of Takiguchi, Teraoka, and Vandyke with Nashida to include a hemispherical dome-shaped display.
The motivation to combine Ramalingam in view of Takiguchi, Teraoka, and Vandyke with Nashida would have been obvious as Takiguchi, Vandyke, and Nashida are all arts related to the field of virtual reality (See Nashida [0001]). It should be noted that having a hemispherical dome-shaped display is a mainly design choice, in which a person of ordinary skill in the art would easily be able to choose to include. The benefit of this type of display is that it gives a wider field of view, and could create a more immersive experience for the user.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to THANG G HUYNH whose telephone number is (571)272-5432. The examiner can normally be reached Mon-Thu 7:30am-4:30pm EST | Fri 7:30am-11:30am EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kee Tung can be reached at (571)272-7794. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/T.G.H./Examiner, Art Unit 2611
/KEE M TUNG/Supervisory Patent Examiner, Art Unit 2611
Prosecution Insights