DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Election/Restrictions
Applicant’s election without traverse of Group I, claims 1-7 in the reply filed on 04/28/2026 is acknowledged.
Claim Objections
Claims 1 and 3 objected to because of the following informalities:
Claim 1, Page 2, line 12, recites “the VR images”. There is no antecedent basis for VR images. For the purpose of examination, it is interpreted as “the virtual images”.
Claim 1, Page 2, line 13, recites “when the cloud server side transmits the VR data stream to each live streaming terminal, virtual image of the live-streamed person is capable of being absent”. It is unclear of the meaning of “capable of being absent”. For the purpose of examination, it is interpreted as “when the cloud server side transmits the VR data stream to each live streaming terminal, virtual image of the live-streamed person can be changed”.
Claim 3, line 11, recites “the multiple groups”. There is no antecedent basis for the multiple groups. For the purpose of examination, it reads as “multiple groups”;
Claim 3, line 12, recites “based on the time frames and numbers”, there is no definition for the “numbers”. For the purpose of examination, it reads as “based on the time frames”;
Appropriate correction is required.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1 is/are rejected under 35 U.S.C. 103 as being unpatentable over Yerli et al. (US 20220070235 A1), hereinafter as Yerli, in view of Pintaric et al. (US 20190371074 A1), hereinafter as Pintaric, further in view of Zhu et al. (CN 113709515 A), hereinafter as Zhu. The original and a machine translation of Zhu are provided by the examiner. The paragraph of Zhu starts with both number and letter ‘n’.
Regarding claim 1, Yerli teaches A method for real social interaction using a virtual scene , involving a cloud server side (Yerli paragraph [0164] “A system 100 of the current disclosure enabling interactions in virtual environments comprises one or more cloud server computers 102”) and a live streaming terminal (Yerli paragraph [0164] “the system 100 further comprises at least one camera 112 obtaining live data feed 114 from a user 116 of a client device 118. The one or more client devices 118 communicatively connect to the one or more cloud server computers 102 and at least one camera 112 via a network.”), and comprising the following steps:
step one: acquiring, by the cloud server side, the number N of live streaming rooms in a social virtual scene based on the number of live streaming terminals, wherein the live streaming room is a live streaming scene in reality, and N>=2 (Yerli teaches two client devices A and B in Figure 1, and three client devices A, B and C in Figure 12, paragraph [0164] “The system 100 enables real-time multi-user collaboration and interactions in the virtual environment 110 by accessing a graphical user interface through the client device 118.”);
step two: establishing N+1 identical three-dimensional coordinate systems based on the number N of the live streaming rooms, wherein a three-dimensional coordinate system is established for each live streaming room, 1 to N three-dimensional coordinate systems are formed for the N live streaming rooms, and the cloud server side establishes an (N+1)th three-dimensional coordinate system in the virtual scene (Yerli teaches the 3D coordinate of virtual environment on the cloud server, further teaches inserting user graphical representation in the virtual environment, implies the 3D coordinate systems for the users and the cloud server are the same, paragraph [0164] “inserting a user graphical representation into a virtual environment involves graphically combining the user graphical representation in the virtual environment such that the user graphical representation appears in the virtual environment (e.g., at a specified 3D coordinate position).”, paragraph [0200] “an orchestra and/or singer may be holding a musical concert through a live recording of their performance via a camera and through their user graphical representations, e.g., through their user live 3D virtual cutout. The user graphical representation of each performer may be inserted into a corresponding three-dimensional coordinate of the stage from where they may perform.”);
step three: ……
there is no obstacle in the Ki-KN, and a real human figure in the Ki-KN is capable of performing three-dimensional live streaming; and the obstacle refers to the degree of occlusion that prevents the completion of live streaming, and obstacles capable of being filtered and compensated for in the live streaming process mean that there is no obstacle (Yerli teaches latency permitted and the designed quality of service as obstacle, further teaches the amount of data to be processed can be adjusted to reduce the obstacle, paragraph [0239-0242] “FIG. 6A depicts the client-server side 304 comprising a client device A, a cloud server 102 and a client device B. In FIG. 6A, the client device A is the sender of data to be processed, and the client device B is the receiver of the data. A plurality of image processing tasks are depicted and classified based on whether they are performed by the client device A, cloud server 102 and/or client device B,……The image processing combination and corresponding level of usage of the client device A processing 602, server image processing 604, and client device B processing 606 depend on the amount of data to be processed, the latency permitted to sustain a smooth user experience, the desired quality of service (QOS), the services required, or the like.”);
step four: defining each live streaming room with at least one live-streamed person as a live streaming room with persons, and defining the number of live streaming rooms with persons as M, wherein N>=M>=2 (Yerli Figure 12 teaches three live streaming room with users); and collecting human body position information and human figure information of each live-streamed person and sound information and sound position information of the live-streamed person from each live streaming room in the M live streaming rooms separately, transmitting the information to the cloud service side, and processing, by the cloud server side, the figure information of each live-streamed person in each live streaming room into three-dimensional portrait information (Yerli teaches the 3D live avatar/3D virtual cutouts as the three-dimensional portrait information, the cloud server is used to generate the 3D portrait information, further teaches combining user position and audio information with the 3D portrait information, paragraph [0171] “updating and insertion of the user graphical representation 120 and combination with the virtual environment is performed by the at least one processor 104 of the one or more cloud server computers 102 upon the client device 118 sending the unprocessed live data feed 114 of the user 116. The one or more cloud server computers 102 thus receive the unprocessed live data feed 114 of the user 116 from the client device 118 and then generate, process and render from the unprocessed live data feed, a user graphical representation 120 that is positioned within a three-dimensional coordinate of the virtual environment 110 before transmitting the cloud-rendered user graphical representation within the virtual environment to other client devices 118.”, paragraph [0203] “an electronic music concert may be played by a disk jockey (DJ) performing behind a virtual table on a stage, where the DJ may be represented by a 3D live virtual avatar or user real-time 3D virtual cutout. If the DJ is represented by the user real-time 3D virtual cutout, the real-time movements of the DJ playing the audio mixing console may be projected onto the real-time 3D virtual cutout from the live data feed obtained by a camera capturing images from the DJ at the DJ location (e.g., from the DJ's house or a recording studio). Furthermore, each member of the audience may also be represented by their own user graphical representations, where some of the users may be represented by 3D live virtual avatars and others by user real-time 3D virtual cutouts depending on the user preferences.”);
step five: instead of step four, defining each live streaming room with at least one live-streamed person as a live streaming room with persons, and defining the number of live streaming rooms with persons as M, wherein N>=M>=2 (Yerli Figure 12 teaches three live streaming room with users); and collecting human body position information and human figure information of each live-streamed person and sound information and sound position information of the live-streamed person from each live streaming room in the M live streaming rooms separately, processing the information into three-dimensional portrait information, and transmitting the portrait information to the cloud server side (Yerli teaches generating 3D portrait information on the client server, paragraph [0170] “The one or more cloud server computers 102 may receive the client-rendered user graphical representation 120, insert the client-rendered user graphical representation 120 into a three-dimensional coordinate of the virtual environment 110, combine the inserted user graphical representation 120 with a the virtual environment 110 and then proceed to transmit the client-rendered user graphical representation 120 to receiving client devices.”);
and step six: importing, by the cloud server side, the human body position information and the three- dimensional portrait information of each live-streamed person and the sound information and the sound position information of the live-streamed person in each live streaming room into the virtual scene in real time to form a VR data stream, and transmitting, by the cloud server side, the VR data stream to each live streaming terminal (Yerli paragraph [0170] “For example, as viewed in FIG. 1, client device A may receive the live data feed 114 from the respective camera 112, may process and render the data from the live data feed 114, generating the user graphical representation A, and may then transmit the client-rendered user graphical representation A to the at least one cloud server computer 102, which may position the user graphical representation A in a three-dimensional coordinate of the virtual environment 118 before transmitting the user graphical representation A to client device B. A similar process applies to the client device B and the user graphical representation B from user B. Both user graphical representations A and B may thus view each other in the virtual environment A and interact.”, paragraph [0203] “an electronic music concert may be played by a disk jockey (DJ) performing behind a virtual table on a stage, where the DJ may be represented by a 3D live virtual avatar or user real-time 3D virtual cutout. If the DJ is represented by the user real-time 3D virtual cutout, the real-time movements of the DJ playing the audio mixing console may be projected onto the real-time 3D virtual cutout from the live data feed obtained by a camera capturing images from the DJ at the DJ location (e.g., from the DJ's house or a recording studio). Furthermore, each member of the audience may also be represented by their own user graphical representations, where some of the users may be represented by 3D live virtual avatars and others by user real-time 3D virtual cutouts depending on the user preferences.”),
wherein each live-streamed person wears a display component, namely AR glasses for the live streaming terminal in the corresponding live streaming room (Yerli paragraph [0162] “Client or peer devices of the current disclosure may comprise, for example, computers, headsets, mobile phones, glasses, transparent screens, tablets and generally input devices with cameras built-in or which may connect to cameras and receive data feed from said cameras.”); at this time, virtual images of all live-streamed persons who wear AR glasses are gathered in the virtual scene (Yerli paragraph [0298] “In the example illustrated in FIG. 12, users A-C access the virtual environment 1212 through their corresponding client devices, wherein each user A-C has a camera 1216 sending multimedia streams corresponding to each user A-C, which may be used in the generation of the user graphical representations A-C and insertion into and combination with the virtual environment 1212”), a physical image of the live-streamed person in each live streaming room overlaps with the virtual image (Yerli paragraph [0065] “updating of the user 3D virtual cutout constructed from a user-uploaded or third-party-source photo comprises applying machine vision algorithms by the client device or the at least one cloud server computer on the generated user 3D virtual cutout for recognizing facial expressions of the user and graphically simulating the facial expressions on the user 3D virtual cutout.”), …… and certainly, when the cloud server side transmits the VR data stream to each live streaming terminal, the virtual image of the live-streamed person (Yerli paragraph [0299] “as the user graphical representation A is closer in the virtual environment 1212 to the virtual multimedia streams source 1224 , the forwarding of the outbound media streams may be optimized to comprise, for example, images with higher resolution for the user graphical representation A than those provided to user graphical representations B and C.”)……
Yerli is not relied on for the below claim language …… setting the N+1 three-dimensional coordinate systems, wherein each three- dimensional coordinate system is composed of an x axis, a y axis, and a z axis with the same length unit; and defining the ground of the virtual scene as a plane formed by the x axis and the y axis of each three-dimensional coordinate system, and also defining the ground of the live streaming room as the plane formed by the x axis and the y axis of the three-dimensional coordinate system…… AR glass …… AR glass…… and the live-streamed person is only capable of seeing the VR images of other live-streamed persons through the AR glasses …… and the live-streamed person is also only capable of seeing the virtual images of other live-streamed persons through the AR glasses.
Pintaric teaches setting the N+1 three-dimensional coordinate systems, wherein each three- dimensional coordinate system is composed of an x axis, a y axis, and a z axis with the same length unit; and defining the ground of the virtual scene as a plane formed by the x axis and the y axis of each three-dimensional coordinate system, and also defining the ground of the live streaming room as the plane formed by the x axis and the y axis of the three-dimensional coordinate system wherein a space occupied by the virtual scene in the (N+1)th three-dimensional coordinate system is represented as KN+1, a corresponding space in the three-dimensional coordinate system of each live streaming room is represented as Ki-KN (Pintaric teaches align X-Z plane to a planar floor, it would have been obvious for a person of ordinary skill in the art before the effective filing date of the claimed invention to substitute the X-Z plane with the X-Y plane, Pintaric paragraph [0052] “FIG. 2B depicts an example 220 of a coordinate system for the AR device 202. The coordinate system may be oriented according to a vertical Y-axis (yaw-axis) and a horizontal X (pitch) and Z (roll) axis such that the Z-axis is pointing in front of a user of the device with the X-axis perpendicular to the Z-axis, and such that an X-Z plane is parallel to a planar floor, regardless of the original orientation of the AR device 202. The angular rotation is measured according to the determined coordinate system relative to the respective X, Y, and Z axes.”)
Yerli and Pintaric are in the same field of endeavor, namely multiple user interactions. Pintaric teaches using a coordinate system to improve performance (Pintaric paragraph [0016] “The described system increases performance by reducing the workload of the host peer while maintaining a higher levels of detail for the local AR experience at each remote peer. Because a host AR device, such as the HoloLens™, does not need to generate the same level of detail as may be required by a remote peer, the workload of the host device is dramatically reduced and thus improves overall performance.”). Therefore, it would have been obvious for a person of ordinary skill in the art before the effective filing date of the claimed invention to combine the teaching of Pintaric with the method of Yerli to improve performance.
Yerli in view of Pintaric are not relied on for the below claim language ……AR glass …… AR glass…… and the live-streamed person is only capable of seeing the VR images of other live-streamed persons through the AR glasses …… and the live-streamed person is also only capable of seeing the virtual images of other live-streamed persons through the AR glasses. Zhu teaches ……AR glass …… AR glass…… and the live-streamed person is only capable of seeing the VR images of other live-streamed persons through the AR glasses …… and the live-streamed person is also only capable of seeing the virtual images of other live-streamed persons through the AR glasses (Zhu teaches using AR glass in live streaming event, and further teaches collecting 3D information from user end and live broadcast end to synthesis a virtual video, this implies the virtual image can only be seen through AR glass, paragraph [n0030] “Merge the 3D human body model into the 3D group photo scene to dynamically synthesize a 3D group photo video. This video can be watched online in real time through AR glasses, allowing viewers to see the live streamer and participating users interacting online”, paragraph [n0049] “The first dynamic information of the participating users is collected through the three dimensional data acquisition module on the user end, and the second dynamic information of the users on the live broadcast end is collected through the three-dimensional data acquisition module on the live broadcast end. The first dynamic information and the second dynamic information are combined to generate a two-dimensional image and a three dimensional human body model. The three-dimensional data acquisition module can collect three-dimensional human body motion video, and the real-time motion of the anchor and participating users is captured in three dimensions. The captured three-dimensional motion video is synthesized through the computer server.”).
Yerli, Pintaric and Zhu are in the same field of endeavor, namely multiple user interactions. Zhu teaches using AR glass to improve user experience (Zhu paragraph [n0029] “It can be watched online in real time through AR glasses, and the scene of the anchor and participating users interacting online can be viewed in real time, improving the sense of realism and experience.”). Therefore, it would have been obvious for a person of ordinary skill in the art before the effective filing date of the claimed invention to combine the teaching of Zhu with the method of Yerli and Pintaric to improve user experience.
Claim(s) 2 is/are rejected under 35 U.S.C. 103 as being unpatentable over Yerli et al. (US 20220070235 A1), hereinafter as Yerli, in view of Pintaric et al. (US 20190371074 A1), hereinafter as Pintaric, further in view of Zhu et al. (CN 113709515 A), hereinafter as Zhu, and Moore et al. (US 20110249090 A1), hereinafter as Moore.
Regarding claim 2, Yerli in view of Pintaric and Zhu teach The method for real social interaction using a virtual scene according to claim 1, wherein the generation of three-dimensional portrait information comprises the following steps: and further teach …… and S3): transmitting, by the live streaming terminal, the stereoscopic human body image to the cloud server side, recognizing, by the cloud server side, the stereoscopic human body image frame by frame, recording a video frame with the AR glasses in the stereoscopic human body image, and fusing a face image with the same time frame with the stereoscopic human body image to form the three-dimensional portrait information (Yerli paragraph [0065] “the one or more cloud server computers are further configured to animate the matching user graphical representation or the new user graphical representation. Animating the matching user graphical representation comprises applying machine vision algorithms by the client device or the at least one cloud server computer on the respective user graphical representation for recognizing facial expressions of the user and graphically simulating the facial expressions on the user graphical representation. In further embodiments, updating of the user 3D virtual cutout constructed from a user-uploaded or third-party-source photo comprises applying machine vision algorithms by the client device or the at least one cloud server computer on the generated user 3D virtual cutout for recognizing facial expressions of the user and graphically simulating the facial expressions on the user 3D virtual cutout.”).
Yerli in view of Pintaric and Zhu are not relied on for the below claim language S1): arranging at least three video cameras for each live-streamed person in each live streaming room, and performing, by the at least three video cameras, synchronous tracking and shooting on the live-streamed person in the live streaming room, wherein the synchronous tracking and shooting means that all frames shot by different video cameras have the same time; S2): performing, by the live streaming terminal, human body image matting at different angles frame by frame from videos shot by the different video cameras in the same live streaming room, and synthesizing images into a stereoscopic human body image……
Moore teaches S1): arranging at least three video cameras for each live-streamed person in each live streaming room, and performing, by the at least three video cameras, synchronous tracking and shooting on the live-streamed person in the live streaming room, wherein the synchronous tracking and shooting means that all frames shot by different video cameras have the same time (Moore paragraph [0009] “The three-dimensional location of a live video object or presenter in the three-dimensional virtual space of a multi-layer video presentation to be generated may be derived using three cameras positioned in a triangle pointing to the center of a chroma-key stage, to capture the contour of the live video object or presenter from three different directions.”),
S2): performing, by the live streaming terminal, human body image matting at different angles frame by frame from videos shot by the different video cameras in the same live streaming room, and synthesizing images into a stereoscopic human body image (Moore teaches the three cameras positioned based on a triangle, implies each camera has a different angle towards the center of the streaming terminal, further teaches generating a virtual three-dimensional presentation based on camera data, paragraph [0012] “provides a virtual three-dimensional presentation in accordance with the present invention includes three stereoscopic layers, a stereoscopic background layer, a stereoscopic video insert layer, and a stereoscopic three-dimensional foreground layer. A surface, the Z-sphere, is defined in the virtual three-dimensional space of the three-dimensional scene with reference to the aim point, the distance between the cameras, and the distance from the camera of the video scene, of a virtual stereoscopic camera looking at the three-dimensional scene being created.”)……
Yerli, Pintaric, Zhu and Moore are in the same field of endeavor, namely multiple user interaction. Moore teaches using three cameras positioned on a triangle pointing towards the center to generate multi-layer video presentation to improve quality (Moore paragraph [0027] “may be employed in any application where a high-quality virtual three-dimensional video presentation is desired.”). Therefore, it would have been obvious for a person of ordinary skill in the art before the effective filing date of the claimed invention to combine the teaching of Moore with the method of Yerli and Pintaric and Zhu to improve quality.
Claim(s) 5 is/are rejected under 35 U.S.C. 103 as being unpatentable over Yerli et al. (US 20220070235 A1), hereinafter as Yerli, in view of Pintaric et al. (US 20190371074 A1), hereinafter as Pintaric, further in view of Zhu et al. (CN 113709515 A), hereinafter as Zhu, and Wang et al. (CN 113822970 A), hereinafter as Wang. The original and a machine translation of Wang are provided by the examiner. The paragraph number starts with either a number or the letter ‘n’.
Regarding claim 5, Yerli in view of Pintaric and Zhu teach The method for real social interaction using a virtual scene according to claim 1, but are not relied on for the below claim language, wherein the human body position information comprises position coordinates and a posture, the live streaming terminal collects the position coordinates and the posture of each live-streamed person in each live streaming room and transmits the collected position coordinates and posture to the cloud server side, and the live streaming terminal synchronously collects the coordinates and the posture of each person in the live streaming room in real time based on a time sequence of frames shot by the video camera; and the position coordinates are collected by a positioning device, posture data is collected by a gyroscope, and the positioning device and the gyroscope are fixed to the chest of the live- streamed person.
Wang teaches wherein the human body position information comprises position coordinates and a posture, the live streaming terminal collects the position coordinates and the posture of each live-streamed person in each live streaming room and transmits the collected position coordinates and posture to the cloud server side (Wang paragraph [n0042] “At the same time, the performance data of the motion capture actor captured by the camera is transmitted to the virtual live broadcast engine, so that the virtual live broadcast engine can synthesize the live broadcast picture according to the performance data of the motion capture actor, generate a live video stream containing the virtual avatar, and send it to the live broadcast client. Viewers can watch the live video through the live broadcast client.”, paragraph [n0081] “motion data collected by inertial sensors, such as the posture and orientation of the motion capture actor's body parts, can be used to determine whether the position of the motion capture actor has changed. When it is determined that the position of the motion capture actor has changed, the movement distance and direction of the motion capture actor in the real scene can be calculated based on kinematic principles”),
and the live streaming terminal synchronously collects the coordinates and the posture of each person in the live streaming room in real time based on a time sequence of frames shot by the video camera (Wang paragraph [n0087] “multiple infrared emitting cameras are installed in the space where the motion capture actor is located to cover and film the entire space. Infrared reflective dots can be installed on the motion capture actor's body. By capturing the light emitted by the camera and the light reflected by the reflective dots, and performing program calculations, the position of the motion capture actor at the current moment can be obtained.”);
and the position coordinates are collected by a positioning device, posture data is collected by a gyroscope, and the positioning device and the gyroscope are fixed to the chest of the live-streamed person (Wang Paragraph [n0062] “Inertial sensors can be placed on the moving parts of motion capture actors to measure their motion data …… When it is necessary to collect motion data of the actor's whole body, inertial sensors can also be configured on specific parts of the actor's body, such as ankles, knees, thighs, arms, hands, chest, waist and shoulders, head, etc. Specifically, inertial sensors can include accelerometers, gyroscopes, and magnetometers”).
Yerli, Pintaric, Zhu and Wang are in the same field of endeavor, namely multiple user interaction. Wang teaches using sensors to generate the positioning data of the user in order to improve user interaction (Wang paragraph [n0105] “This allows the motion capture actors and the real streamers to cooperate with each other in performance activities based on the positions, actions, and language of the virtual
avatars and the real streamers, thereby improving the interactivity between the virtual
avatars and the real streamers.”). Therefore, it would have been obvious for a person of ordinary skill in the art before the effective filing date of the claimed invention to combine the teaching of Wang with the method of Yerli, Pintaric and Zhu to improve user interaction.
Claim(s) 6 is/are rejected under 35 U.S.C. 103 as being unpatentable over Yerli et al. (US 20220070235 A1), hereinafter as Yerli, in view of Pintaric et al. (US 20190371074 A1), hereinafter as Pintaric, further in view of Zhu et al. (CN 113709515 A), hereinafter as Zhu, and French et al. (US 20180063205 A1), hereinafter as French.
Regarding claim 6, Yerli in view of Pintaric and Zhu teach The method for real social interaction using a virtual scene according to claim 1, but are not relied on for the below claim language wherein the sound information is collected by a sound recording device of the live streaming terminal, and the sound position information is collected by a sound source positioning device of the live streaming terminal. French teaches wherein the sound information is collected by a sound recording device of the live streaming terminal, and the sound position information is collected by a sound source positioning device of the live streaming terminal (French paragraph [0106] “the device can going to track where a person is and what is around them and re-create the scene to be transmitted to the platform to be broadcast to other users or people who are viewing this event. The location tagging can include where a person is when they are recording the video; and a sound recording can include any device recording that is possible (such as sound, video, and geographic location. The recordings may depend on the capabilities of the device. The device can record the data and send it to the platform to be given to the users so that event information can be displayed.”).
Yerli, Pintaric, Zhu and French are in the same field of endeavor, namely multiple user interaction. French teaches a sound device to collect the location and contents of the sound to improve user interaction. Therefore, it would have been obvious for a person of ordinary skill in the art before the effective filing date of the claimed invention to combine the teaching of French with the method of Yerli, Pintaric and Zhu to improve user interaction.
Claim(s) 7 is/are rejected under 35 U.S.C. 103 as being unpatentable over Yerli et al. (US 20220070235 A1), hereinafter as Yerli, in view of Pintaric et al. (US 20190371074 A1), hereinafter as Pintaric, further in view of Zhu et al. (CN 113709515 A), hereinafter as Zhu, Moore et al. (US 20110249090 A1), hereinafter as Moore and Wang et al. (CN 113822970 A), hereinafter as Wang. The original and a machine translation of Zhu and Wang are provided by the examiner. The paragraph number starts with either a number or the letter ‘n’.
Regarding claim 7, Yerli in view of Pintaric and Zhu teach A system for real social interaction using a virtual scene according to claim 1, and further teach comprising a cloud server side and a direct streaming terminal, the cloud server side being equipped with a cloud service processor, the live streaming terminal being equipped with a terminal processor (Yerli Figure 1 with cloud servers 102 with processor 104, and client device A and B), the live streaming terminal further comprising at least …… AR glasses (Zhu paragraph [n0030] “Merge the 3D human body model into the 3D group photo scene to dynamically synthesize a 3D group photo video. This video can be watched online in real time through AR glasses, allowing viewers to see the live streamer and participating users interacting online”), wireless in-ear headphones (Yerli teaches using various devices, it would have been obvious for a person of ordinary skill in the art before the effective filing date of the claimed invention to combine it with a wireless headphone, paragraph [0167] “the client devices 118 may be one or more of mobile devices, personal computers, game consoles, media centers, and head-mounted displays, amongst others.”), …… , wherein the cloud service processor is communicatively connected to the terminal processor through a TCP/IP, the video cameras, a VR headset, the wireless in-ear headphones, …… (Yerli paragraph [0162] “Client or peer devices of the current disclosure may comprise, for example, computers, headsets, mobile phones, glasses, transparent screens, tablets and generally input devices with cameras built-in or which may connect to cameras and receive data feed from said cameras.”, paragraph [0236-0237] “In one embodiment of FIGS. 6A-6C, the at least one cloud server 102 may be configured as a Traversal Using Relay Network Address Translation (NAT) (sometimes referred to as TURN) server, which may be suitable in situations where the server cannot establish a connection between the client devices 118 . TURN is an extension of Session Traversal Utilities for NAT (STUN). NAT is a method of remapping an Internet Protocol (IP) address space into another by modifying network address information in the IP header of packets while they are in transit across a traffic routing device.”)
Yerli in view of Pintaric and Zhu are not relied on for the below claim language …… three groups of video cameras, …… a positioning device, and a gyroscope …… the positioning device, and the gyroscope are all electrically connected to the terminal processor, the positioning device and the gyroscope are fixedly integrated, and the positioning device and the gyroscope are fixed to the chest of a live-streamed person.
Moore teaches …… three groups of video cameras (Moore paragraph [0009] “The three-dimensional location of a live video object or presenter in the three-dimensional virtual space of a multi-layer video presentation to be generated may be derived using three cameras positioned in a triangle pointing to the center of a chroma-key stage, to capture the contour of the live video object or presenter from three different directions.”),
Yerli, Pintaric, Zhu and Moore are in the same field of endeavor, namely multiple user interaction. Moore teaches using three cameras positioned on a triangle pointing towards the center to generate multi-layer video presentation to improve quality (Moore paragraph [0027] “may be employed in any application where a high-quality virtual three-dimensional video presentation is desired.”). Therefore, it would have been obvious for a person of ordinary skill in the art before the effective filing date of the claimed invention to combine the teaching of Moore with the method of Yerli and Pintaric and Zhu to improve quality.
Yerli in view of Pintaric, Zhu and Moore are not relied on for the below claim language …… a positioning device, and a gyroscope …… the positioning device, and the gyroscope are all electrically connected to the terminal processor, the positioning device and the gyroscope are fixedly integrated, and the positioning device and the gyroscope are fixed to the chest of a live-streamed person. Wang teaches …… a positioning device, and a gyroscope …… the positioning device, and the gyroscope are all electrically connected to the terminal processor, the positioning device and the gyroscope are fixedly integrated, and the positioning device and the gyroscope are fixed to the chest of a live-streamed person (Wang Paragraph [n0062] “Inertial sensors can be placed on the moving parts of motion capture actors to measure their motion data …… When it is necessary to collect motion data of the actor's whole body, inertial sensors can also be configured on specific parts of the actor's body, such as ankles, knees, thighs, arms, hands, chest, waist and shoulders, head, etc. Specifically, inertial sensors can include accelerometers, gyroscopes, and magnetometers”, paragraph [n0081] “motion data collected by inertial sensors, such as the posture and orientation of the motion capture actor's body parts, can be used to determine whether the position of the motion capture actor has changed. When it is determined that the position of the motion capture actor has changed, the movement distance and direction of the motion capture actor in the real scene can be calculated based on kinematic principles”).
Yerli, Pintaric, Zhu, Moore and Wang are in the same field of endeavor, namely multiple user interaction. Wang teaches using sensors to generate the positioning data of the user in order to improve user interaction (Wang paragraph [n0105] “This allows the motion capture actors and the real streamers to cooperate with each other in performance activities based on the positions, actions, and language of the virtual
avatars and the real streamers, thereby improving the interactivity between the virtual
avatars and the real streamers.”). Therefore, it would have been obvious for a person of ordinary skill in the art before the effective filing date of the claimed invention to combine the teaching of Wang with the method of Yerli, Pintaric, Zhu and Moore to improve user interaction.
Allowable Subject Matter
Claims 3-4 objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter:
Regarding claim 3, the closest prior art of Moore teaches the three cameras positioned based on a triangle, implying each camera has a different angle towards the center of the streaming terminal, further teaches generating a virtual three-dimensional presentation based on camera data, paragraph [0012] “provides a virtual three-dimensional presentation in accordance with the present invention includes three stereoscopic layers, a stereoscopic background layer, a stereoscopic video insert layer, and a stereoscopic three-dimensional foreground layer. A surface, the Z-sphere, is defined in the virtual three-dimensional space of the three-dimensional scene with reference to the aim point, the distance between the cameras, and the distance from the camera of the video scene, of a virtual stereoscopic camera looking at the three-dimensional scene being created.”). However, Moore fails to teach the combined limitation below as a whole, “and the live- streamed person wears the AR glasses in the live streaming room, the AR glasses are equipped with cameras toward the face, time frames of the cameras are synchronized with time frames of the video cameras, and the cameras shoot the face of the live-streamed person, mainly a part of the face of the person that is obscured by the AR glasses; S2): sequentially fusing, by the live streaming terminal, videos shot by the multiple groups of cameras based on the time frames and numbers to form a face image”. Furthermore, no prior art of record either alone or in combination teaches the above limitation as a whole. Therefore, claim 3 is considered to allowable.
Claim 4 contain allowable subject matter because they depend on claim 3 that contains allowable subject matter.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Mack et al. (US 20190358547 A1) teaches a method to combine live action and virtual image in real time.
Amayeh et al. (US 20220357582 A1) teaches using two eye cameras to capture a face image (paragraph [0109] “As shown in FIG. 11, the eye camera 1160 a and the eye camera 1160 b can have an overlapping FOV 1130 . Because of this overlapping FOV, the two eye cameras may be treated as a single stereoscopic system for imaging the user's face when the user's face is within the region 1130”, paragraph [0110] “For example, a pair of images may include an image taken by the eye camera 1160 a and an image taken by the camera 1160 b at the same time.”).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to XIAOMING WEI whose telephone number is (571)272-3831. The examiner can normally be reached M-F 8:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kee Tung can be reached at (571)272-7794. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KEE M TUNG/Supervisory Patent Examiner, Art Unit 2611
/XIAOMING WEI/Examiner, Art Unit 2611
Prosecution Insights