DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-5, 7, 10-14 and 16-25 is/are rejected under 35 U.S.C. 103 as being unpatentable over SARKIS et al. (US 20240062467 A1), referred herein as SARKIS in view of SINGH et al. (US 20190005717 A1), referred herein as SINGH.
Regarding Claims 1, 24 and 25, SARKIS in view of SINGH teaches a method, a device comprising: a non-transitory computer-readable storage medium; and one or more processors coupled to the non-transitory computer-readable storage medium, wherein the non-transitory computer-readable storage medium comprises program instructions that, when executed on the one or more processors, cause the one or more processors to perform operations comprising (SARKIS Abst: Systems and techniques are described for establishing one or more virtual sessions between users; [0007] In another example, an apparatus for generating virtual content at a first device in a distributed system is provided that includes at least one memory and at least one processor (e.g., configured in circuitry) coupled to the at least one memory; [0008] In another example, a non-transitory computer-readable medium is provided that has stored thereon instructions that, when executed by one or more processors; [0010] As another example, a method is provided for establishing one or more virtual sessions between users):
at a first electronic device having a processor (SARKIS [0104] FIG. 3 is a diagram illustrating an example of an extended reality system 300 in which two client devices (client device 302 and client device 304); [0116] FIG. 5 is a diagram illustrating an example of a device 500. The device 500 can be implemented as a client device (e.g., device 405 of FIG. 4) or as an animation and scene rendering system (e.g., the animation and scene rendering system 410). As shown, the device 500 includes a central processing unit (CPU) 510):
presenting a view of a first three-dimensional (3D) environment (SARKIS [0004] The metaverse is essentially a virtual universe that includes one or more three-dimensional (3D) virtual worlds; [0099] The device 105 may overlay virtual objects with real-world objects in a view 130. For example, the view 130 may generally refer to visual input to a user 110 via the device 105);
obtaining data representing a second 3D environment (SARKIS [0007] an apparatus for generating virtual content at a first device in a distributed system is provided that includes at least one memory and at least one processor (e.g., configured in circuitry) coupled to the at least one memory. The at least one processor is configured to: receive, from a second device associated with a virtual session, input information associated with at least one of the second device or a user of the second device; [0080] a virtual session provided by an XR system may include a 3D collaborative virtual environment for a group of users),
the data representing the second 3D environment based at least in part on sensor data captured in a physical environment of a second electronic device (SARKIS [0100] the device 105 may project virtual objects onto the real-world environment. In some cases, the device 105 may include a camera and may display both real-world objects (e.g., as frames or images captured by the camera) and virtual objects overlaid on displayed real-world objects; [0102] the extended reality system 100 may provide various types of virtual experiences, such as a three-dimensional (3D) collaborative virtual environment for a group of users (e.g., including the user 110). FIG. 2 is a diagram illustrating an example of a 3D collaborative virtual environment 200 in which various users interact with one another in a virtual session via virtual representations (or avatars) of the users in the virtual environment 200. The virtual representations include including a virtual representation 202 of a first user, a virtual representation 204 of a second user, a virtual representation 206 of a third user, a virtual representation 208 of a fourth user, and a virtual representation 210 of a fifth user); Any two of client devices 602-606 are interpreted as the first and second electronic device.
determining SARKIS [0102] Other background information of the virtual environment 200 is also shown, including a virtual calendar 212, a virtual web page 214, and a virtual video conference interface 216. The users may visually, audibly, haptically, or otherwise experience the virtual environment from each user's perspective while interacting with the virtual representations of the other users. For example, the virtual environment 200 is shown from the perspective of the first user (represented by the virtual representation 202); [0134] As shown, device 500 may include an extended reality manager 550. The extended reality manager 550 may implement aspects of extended reality, augmented reality, virtual reality, etc. In some cases, such as when the device 500 is implemented as a client device (e.g., device 405 of FIG. 4), the extended reality manager 550 may determine information associated with a user of the device and/or a physical environment in which the device 500 is located, such as facial information, body information, hand information, device pose information, audio information, etc); and
displaying, in the view of the first 3D environment, a portal with the portal content, wherein the portal content depicts a portion of the second 3D environment viewed SARKIS [0103] in the example of the 3D collaborative virtual environment 200 of FIG. 2, an XR system of the first user (e.g., the XR system 100) displays the virtual representations 204-210 of the other users participating in the virtual session. The virtual representations 204-210 of the users and the background of the virtual environment 200 should be displayed in a realistic manner (e.g., as if the users were meeting in a real-world environment), such as by animating the heads, bodies, arms, and hands of the other users' virtual representations 204-210 as the users move in the real world; [0170] The pre-processing can be performed because, in some cases, the animation and scene rendering system 1010 is configured to modify a neutral face expression of the user of the client device 1002 (e.g., obtained as an enrolled neutral face offline by capturing an image or scan of the user's face) to animate the virtual representation (or avatar) of the user of the client device 1002 with different expressions under different viewpoints). Background information anticipates portal content;
SARKIS disclosed background information of the virtual environment including a virtual calendar 212, a virtual web page 214, and a virtual video conference interface 216, but does not explicitly teach portal content. However, SINGH teaches
determining portal content (SINGH [0008] generating the portal comprises defining a linking position in the second VR immersive space for the portal, and wherein virtually placing the user in the second VR immersive space comprises virtually placing the user at the linking position; [0011] providing a preview of the second VR immersive space on a surface of the portal located in the first VR immersive space), and
3D environment viewed through the portal (SINGH [0183] Spaces can be entered through one or more portals. An entrance portal can be defined by position and direction in a 3D space, by specifying components for X, Y and Z).
SINGH discloses a method and system for linking a first virtual reality (VR) immersive space with a second VR immersive space, which is analogous to the present patent application.
It would have been obvious for a person of ordinary skill in the art before the effective filing date of the claimed invention to have modified SARKIS to incorporate the teachings of SINGH, and apply the 3D spaces interconnected by portals into the systems and techniques for providing distributed generation of virtual content.
Doing so would be able to provide a comprehensive solution for presenting web content in an immersive, social and collaborative VR or AR setting by linking a first virtual reality (VR) immersive space with a second VR immersive space.
Regarding Claim 2, SARKIS in view of SINGH teaches the method of claim 1, and further teaches wherein the portal content is based on synthesizing data representing a portion of the second 3D environment not represented in sensor data captured in the physical environment of the second electronic device (The face-body combiner engine 844 can then combine the body model (SARKIS [0173] the view-dependent texture synthesis engine 1021 receives virtual representation of the user's face (the facial representation) from the face decoder 1013 and the pre-processed 3D model information from the pre-processing engine 1019. The view-dependent texture synthesis engine 1021 can combine information associated with the 3D model of the user's head (output from the pre-processing engine 1019) with the facial representation of the user (output from the face decoder 1013) to generate a full head model with facial features for the user of the client device 1002).
Regarding Claim 3, SARKIS in view of SINGH teaches the method of claim 1, and further teaches wherein the portal content is updated based on detecting a change in the second 3D environment or in the physical environment of the second electronic device (SARKIS [0183] to generate the virtual scene, the first device (or component thereof) may obtain a background representation of the virtual scene and adjust, based on the background representation of the virtual scene, lighting of the virtual representation of the user of the second device to generate a modified virtual representation of the user).
Regarding Claim 4, SARKIS in view of SINGH teaches the method of claim 1, and further teaches wherein obtaining the data representing the second 3D environment comprises obtaining a parameter associated with the data representing the second 3D environment (SARKIS [0011] receiving, at the first device from the second device, second mesh animation parameters for the second virtual representation of the second user of the second device; and generating, at the first device based on the second mesh information and the second mesh animation parameters, the second virtual representation of the second user of the second device).
Regarding Claim 5, SARKIS in view of SINGH teaches the method of claim 1, and further teaches wherein the parameter identifies a field of view or an orientation of the second 3D environment, and wherein determining portal content is further based on the parameter (SARKIS [0144] the input frame(s) received by the body engine 714 may include frames (or images) captured by one or more cameras with a field of view of body (e.g., a portion other than the face, such as a neck, shoulders, torso, lower body, feet, of the user etc.) of the user of the client device 702).
Regarding Claim 7, SARKIS in view of SINGH teaches the method of claim 1, and further teaches further comprising obtaining data representing a user of the second electronic device, wherein determining the portal content is further based on the data representing the user of the second electronic device, and wherein the portal content depicts the representation of the user of the second electronic device in front of the portion of the second 3D environment (SARKIS [0010] at the first device from the second device, second mesh information for a second virtual representation of a second user of the second device; receiving, at the first device from the second device, second mesh animation parameters for the second virtual representation of the second user of the second device; and generating, at the first device based on the second mesh information and the second mesh animation parameters, the second virtual representation of the second user of the second device; [0086] the animation and scene rendering system may compose the virtual scene (including virtual representations of the users and background information) and generate a frame representing a view of the virtual scene from the perspective of a second user (representing a view of the second user in the real or physical world) of a second device of the one or more devices associated with the virtual session).
Regarding Claim 10, SARKIS in view of SINGH teaches the method of claim 1, and further teaches wherein the data representing the second 3D environment depicts a 360-degree view of the second 3D environment (a high dynamic range (HDR) map (e.g., a high dynamic range image (SINGH [0158] HDRI) map including a 360-degrees image) of a scene may be captured by rotating a camera around the scene in a plurality of directions (e.g., in all directions), in some cases from a single viewpoint).
Regarding Claim 11, SARKIS in view of SINGH teaches the method of claim 1, and further teaches further comprising: determining a position at which to display the portal within the view of the first 3D environment based on the viewpoint (SARKIS [0099] positional differences in real-world objects (e.g., and thus overlaid virtual objects) may arise from view 130-A shifting to view 130-B at 135 due to head motion 115).
Regarding Claim 12, SARKIS in view of SINGH teaches the method of claim 1, and further teaches further comprising: changing the portal content based on changes to the viewpoint within the first 3D environment (SARKIS [0101] head motion 115 may include user 110 head rotations, translational head movement, etc. The device 105 may update the view 130 of the user 110 according to the head motion 115. For example, the device 105 may display view 130-A for the user 110 before the head motion 115. In some cases, after the head motion 115, the device 105 may display view 130-B to the user 110. The extended reality system (e.g., device 105) may render or update the virtual objects and/or other portions of the virtual environment for display as the view 130-A shifts to view 130-B).
Regarding Claim 13, SARKIS in view of SINGH teaches the method of claim 1, and further teaches wherein displaying, in the view of the first 3D environment, the portal with the portal content is based on determining a positional relationship of the viewpoint relative to the portal (SARKIS [0099] positional differences in real-world objects (e.g., and thus overlaid virtual objects) may arise from view 130-A shifting to view 130-B at 135 due to head motion 115).
Regarding Claim 14, SARKIS in view of SINGH teaches the method of claim 1, and further teaches wherein a position of the portal within the first 3D environment is constant as the viewpoint changes within the first 3D environment (SINGH [0183] Spaces can be entered through one or more portals. An entrance portal can be defined by position and direction in a 3D space, by specifying components for X, Y and Z. For example, the position where X=2, Y=3, and Z=4 can be expressed concisely as “2 3 4”).
Regarding Claim 16, SARKIS in view of SINGH teaches the method of claim 1, and further teaches wherein the data representing the second 3D environment comprises a stereoscopic image pair comprising left eye content corresponding to a left eye viewpoint and right eye content corresponding to a right eye viewpoint (SINGH [0261] An Image3D tag represents a 3D image that is geometrically the same as an Image, but its texture uses two different images, where each is shown to either the left or right eye to produce a 3D effect).
Regarding Claim 17, SARKIS in view of SINGH teaches the method of claim 1, and further teaches wherein the data representing the second 3D environment comprises a 180-degree stereo image (SINGH [0032] To provide a more flexible presentation space for emerging web content such as panoramic and stereo images and video).
Regarding Claim 18, SARKIS in view of SINGH teaches the method of claim 1, and further teaches wherein the data representing the second 3D environment comprises two-dimensional (2D) image data and depth data (SARKIS [0171] the UV face position map can be a 2D image that records and/or maps the 3D positions of points… the UV position map can record, model, identify, represent, and/or calculate a 3D shape, structure, contour, depth and/or other details of the face).
Regarding Claim 19, SARKIS in view of SINGH teaches the method of claim 1, and further teaches wherein determining portal content comprises rendering at least a portion of the data representing the second 3D environment on at least a portion of a sphere (SINGH [0249] the portal can have the shape of a three-dimensional sphere representing a globe, with the display surface encompassing the exterior of the sphere).
Regarding Claim 20, SARKIS in view of SINGH teaches the method of claim 1, and further teaches wherein the data representing the second 3D environment comprises a three-dimensional (3D) model (SARKIS [0168] A geometry encoder engine 1011 of the client device 1002 may generate a 3D model (e.g., a 3D morphable model or 3DMM) of the user's head or face based on the one or more frames 715).
Regarding Claim 21, SARKIS in view of SINGH teaches the method of claim 1, and further teaches wherein the data representing the second 3D environment is obtained during a communication session between the first electronic device and a second electronic device (SARKIS [0006] receiving, from a second device associated with a virtual session, input information associated with at least one of the second device or a user of the second device).
Regarding Claim 22, SARKIS in view of SINGH teaches the method of claim 1, and further teaches wherein the sensor data captured in the physical environment of the second electronic device is obtained by one or more sensors of the second electronic device (SARKIS [0100] the device 105 may include a camera and may display both real-world objects (e.g., as frames or images captured by the camera)).
Regarding Claim 23, SARKIS in view of SINGH teaches the method of claim 1, and further teaches wherein the first 3D environment is an extended reality (XR) environment (SARKIS [0102] In some cases, the extended reality system 100 may provide various types of virtual experiences, such as a three-dimensional (3D) collaborative virtual environment for a group of users (e.g., including the user 110)).
Claim(s) 6 is/are rejected under 35 U.S.C. 103 as being unpatentable over SARKIS et al. (US 20240062467 A1), referred herein as SARKIS in view of SINGH et al. (US 20190005717 A1), referred herein as SINGH and YOO et al. (US 20240265603 A1), referred herein as YOO.
Regarding Claim 6, SARKIS in view of SINGH teaches the method of claim 5, but does not teach the claimed limitation herein. However, YOO teaches wherein determining portal content comprises blurring some of the portion of the second 3D environment based on the identified field of view or the orientation of the second 3D environment (YOO [0103] the blur effect may be provided at the peripheries 1010 for harmony in between the multimedia content superimposed on the background layer 1000 and an external object viewed through an area around the multimedia content. For example, the blur effect may be provided at the peripheries 1010 to minimize or prevent the background layer 1000 from being viewed when the multimedia content is superimposed on the background layer 1000).
YOO discloses a wearable device, a method, and a non-transitory computer readable storage medium for a displaying multimedia content, which is analogous to the present patent application.
It would have been obvious for a person of ordinary skill in the art before the effective filing date of the claimed invention to have modified SARKIS to incorporate the teachings of YOO, and apply the blurring effect of peripheries background into the systems and techniques for providing distributed generation of virtual content.
Doing so would be able to minimize or prevent the background layer from being viewed when the multimedia content is superimposed on the background layer.
Claim(s) 8 and 9 is/are rejected under 35 U.S.C. 103 as being unpatentable over SARKIS et al. (US 20240062467 A1), referred herein as SARKIS in view of SINGH et al. (US 20190005717 A1), referred herein as SINGH and Barbosa et al. (US 20230300292 A1), referred herein as Barbosa.
Regarding Claim 8, SARKIS in view of SINGH teaches the method of claim 7, but does not teach the claimed limitation herein. However, Barbosa teaches wherein determining portal content comprises blurring the portion of the second 3D environment behind the representation of the user of the second electronic device. (Barbosa [0102] the blur effect may be provided at the peripheries 1010 for harmony in between the generate a segmented user portion 410. Indeed, as shown in FIG. 4, the shared AR video call system 104 generates the segmented user portion 410 from the video frame 406 to segment a foreground depicting a participant user from a background of the video).
Barbosa discloses a video call systems allow users to electronically communicate via computing devices, which is analogous to the present patent application.
It would have been obvious for a person of ordinary skill in the art before the effective filing date of the claimed invention to have modified SARKIS to incorporate the teachings of Barbosa, and apply the user portion segmentation technique into the systems and techniques for providing distributed generation of virtual content.
Doing so would enable the client devices to render the AR environments as shared AR environments that render synchronized AR spaces and/or AR effects across the client devices participating in the video call.
Regarding Claim 9, SARKIS in view of SINGH teaches the method of claim 1, but does not teach the claimed limitation herein. However, Barbosa teaches wherein the data representing the second 3D environment depicts less than a 360-degree view of the second 3D environment (Barbosa [0029] The AR background environment includes the captured video of the one or more participants within a shared AR background environment that is viewable from multiple viewing angles (e.g., 360 degrees, 270 degrees) utilizing movement of a participant and/or movement of the client device).
The same motivation as claim 8 applies here.
Claim(s) 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over SARKIS et al. (US 20240062467 A1), referred herein as SARKIS in view of SINGH et al. (US 20190005717 A1), referred herein as SINGH and Toksvig et al. (US 20190037137 A1), referred herein as Toksvig.
Regarding Claim 15, SARKIS in view of SINGH teaches the method of claim 1, but does not teach the claimed limitation herein. However, Toksvig teaches wherein a position of the portal within the first 3D environment changes based on changes to the viewpoint within the first 3D environment (Toksvig [0034] a parallax viewer system using the portal implementation would shift the virtual viewpoint but not the virtual screen position in response to tracked user movement; [0035] The portal implementation can accurately represent virtual objects on or near to the plane of the screen 120, allowing the user to simulate looking at near objects from different angles. However, as with a real window viewed from a short viewing distance, the portal implementation can produce a screen image which rapidly shifts based on relatively small shifts in the user's viewpoint).
Toksvig discloses a parallax viewer system allows 3D content, such as 360 degree 3D panoramas or other 3D environments, to be viewed by a user through a traditional 2D screen, which is analogous to the present patent application.
It would have been obvious for a person of ordinary skill in the art before the effective filing date of the claimed invention to have modified SARKIS to incorporate the teachings of Toksvig, and apply the portal implementation in a 3D environment into the systems and techniques for providing distributed generation of virtual content.
Doing so would provide an improved methods of viewing immersive VR or 360 degree panoramic content on a 2D screen are needed.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Samantha (Yuehan) Wang whose telephone number is (571)270-5011. The examiner can normally be reached Monday-Friday, 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, King Poon can be reached at (571)272-7440. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Samantha (YUEHAN) WANG/
Primary Examiner
Art Unit 2617