Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 15 is/are rejected under 35 U.S.C. 102(a) (2) as being anticipated by Zhou et al. (US 12,149,864).
Consider claim 15. ZHOU discloses "A computer-implemented method comprising: establishing a real-time communication session between a sender computing device and a recipient computing device" (Claim 1).
ZHOU discloses the limitation of "receiving, via a single camera, a video stream of a face of a user;". ZHOU teaches a camera 120 on a computing device capturing digital video frames for a real-time communication session (col. 2, lines 56-63). ZHOU teaches "In one or more implementations, the sender computing device 114a and the recipient computing device 114b can also include the camera 120. The camera 120 can capture digital photographs and/or digital videos. The camera 120 can also provide a live stream of digital video frames to the social networking system application 116 for transmission during a real-time communication session." (col. 2, lines 56-63).
ZHOU discloses the limitation of "generating a depth map based on the video stream, a location of the face of the user, and a neural network;". ZHOU teaches a landmark map manager 402 that generates a landmark map utilizing a machine learning model trained to identify body and/or facial features (col. 8, lines 16-25). ZHOU's landmark map manager 402 is disclosing the claimed limitation of "generating a depth map based on the video stream, a location of the face of the user, and a neural network;" because the machine learning model (neural network) processes digital video frames to identify facial features and generate a landmark map based on the coordinate locations of identified features. ZHOU teaches "In one or more implementations, the landmark map manager 402 can generate the landmark map utilizing a machine learning model that is trained to identify body and/or facial features. To demonstrate, the landmark map manager 402 can input a digital video frame from the digital video stream of the sender computing device 114a into the machine learning model. In at least one implementation, the machine learning model can output one or more coordinates representing a location of the features identified in the digital video frame." (col. 8, lines 16-25).
ZHOU discloses the limitation of "generating an avatar representing the user based on the depth map and the video stream, the avatar being based on less data than the video stream.". ZHOU teaches rendering a sender's avatar according to the landmark map (col. 5, lines 35-39) and transmitting less data than a standard video call (col. 9, lines 39-43). ZHOU's rendered avatar is disclosing the claimed limitation of "generating an avatar representing the user based on the depth map and the video stream, the avatar being based on less data than the video stream." because the avatar is generated on the recipient side based on the transmitted landmark map and requires a reduced volume of data compared to a standard video stream. ZHOU teaches "At the recipient computing device 114b, the live avatar system 102 can render the sender’s avatar according to the landmark map so that the recipient computing device 114b displays the avatar with positioning that mirrors the sender’s physical positioning." (col. 5, lines 35-39) and "For example, by only transmitting the sender’s avatar and landmark map to the recipient computing device 114b, the live avatar system 102 transmits less data to the recipient computing device 114b than would be transmitted in a standard video call." (col. 9, lines 39-43).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 17, 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Valentin (EP3593323) in view of Zhou et al. (US 12,149,864).
Consider claims 1, 17, 19. VALENTIN discloses the limitation of "receiving, via a single camera, a first video stream of a face of a user;". VALENTIN teaches a depth camera 105 capturing depth images 115 of a face 120 (para. [0008]). VALENTIN teaches "The face tracking system 100 includes a face tracking module 110 for estimating a current pose 140 of a face 120 based on a depth image 115 captured by a depth camera 105" (para. [0008]).
VALENTIN discloses the limitation of "determining a location of the face of the user based on the first video stream and a first model;". VALENTIN teaches fitting a face model mesh to the pixels of the depth map that correspond to the face 120 to identify a pose of the face (para. [0010]). VALENTIN teaches "The face tracking module 110 identifies a pose of the face 120 by fitting a face model mesh to the pixels of the depth map that correspond to the face 120" (para. [0010]).
VALENTIN discloses the limitation of "receiving, via the single camera, a second video stream of the face of the user;". VALENTIN teaches receiving a depth image 115 from the depth camera 105 to generate a depth map (para. [0010]). VALENTIN's continuous reception of depth images from the same depth camera is disclosing the claimed limitation of "receiving, via the single camera, a second video stream of the face of the user;" because receiving subsequent depth images from the same single camera constitutes receiving a subsequent or distinct sequence of digital video frames captured over time. VALENTIN teaches "The face tracking module 110 receives a depth image 115 from the depth camera 105 and generates a depth map based on the depth image 115" (para. [0010]).
VALENTIN discloses the limitation of "generating a depth map based on the second video stream, the location of the face of the user, and a second model; and". VALENTIN teaches generating a depth map from the depth image 115 and fitting a face model mesh to the depth map (para. [0010]). VALENTIN's depth map generation and mesh fitting are disclosing the claimed limitation of "generating a depth map based on the second video stream, the location of the face of the user, and a second model; and" because the depth map is generated from the camera input and utilizes the face's location/pose along with a face model mesh (representing a second model) to process the spatial depth data. VALENTIN teaches "The face tracking module 110 receives a depth image 115 from the depth camera 105 and generates a depth map based on the depth image 115. The face tracking module 110 identifies a pose of the face 120 by fitting a face model mesh to the pixels of the depth map that correspond to the face 120" (para. [0010]).
VALENTIN does NOT explicitly disclose the limitation of "generating a three-dimensional avatar representing the user based on the depth map and the second video stream, the three-dimensional avatar being based on less data than the second video stream.". However, ZHOU teaches rendering a sender's avatar according to a landmark map (col. 1, lines 66-69). ZHOU teaches "the recipient computing device can render the sender’s avatar according to the landmark map. In this way, the sender can appear as their avatar within the real-time communication session" (col. 1, lines 66-69) and that "the implementations described herein can make the video call more efficient and less data-heavy because the avatar rendering happens on the recipient computing device" (col. 2, lines 6-9).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine VALENTIN's face tracking and depth map generation with ZHOU's avatar replacement and data reduction techniques. One of ordinary skill in the art, recognizing the latency and bandwidth problems identified in both systems, would have been motivated to apply ZHOU's avatar rendering technique to VALENTIN's face tracking system to reduce data transmission overhead, thereby yielding a predictable reduction in bandwidth.
Claims 2-6, 9, 13, 18 are rejected under 35 U.S.C. 103 as being unpatentable over Valentin (EP3593323) in view of Zhou et al. (US 12,149,864) as applied to claims 1, 17, 19 above, and further in view of Kuster et al (US 9684953).
Consider claim 2. Valentin in view of Zhou does not clearly teach the second model was trained based on comparing depth data based on images captured by a depth camera to color data based on images captured by a color camera.
Kuster further teaches the second model was trained based on comparing depth data based on images captured by a depth camera to color data based on images captured by a color camera (“In the remainder of the text, the gaze correction system and method shall be explained in terms of a system using only one real video camera (i.e., a color or black and white image camera) in addition to the depth map.” Kuster (12)).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to utilize the teachings of Kuster into the teachings of Valentin in view of Zhou, this would improve the accuracy of facial-feature localization, depth estimation, avatar construction, gaze or pose correction, and alignment between the color and depth images.
Consider claim 3. Kuster, further teaches the first video stream includes color data (“A combination of such elements and in particular of a camera and a depth scanner is embodied, for example, in a Microsoft Kinect device capable of acquiring a hybrid depth/color image, wherein the color image, in this context the “original image” 10 is obtained using a regular camera” Kuster (34)); and the second video stream includes color data (“The gaze correction system is targeted at a peer-to-peer video conferencing model that runs in real-time on average consumer hardware and, in one embodiment, requires only one hybrid depth/color sensor such as the Kinect.” Kuster (11). The hybrid system allows the second video stream to include color data along with the depth data).
Consider claim 4. Kuster, further teaches the second video stream does not include depth data (“or with a polygon determined for a previous image of a sequence of images” Kuster (17). According to the applicant’s specification, “In some examples, the local computing device 102 captures only a single video stream (which can be analyzed at different starting and ending points for a first video stream, a second video stream, and/or a third video stream) with only a single color camera” [0033]. As a result, a plurality of frames could each be treated as a different video stream.).
Consider claim 5. Kuster further teaches at least one frame included in the first video stream is included in the second video stream (“A combination of such elements and in particular of a camera and a depth scanner is embodied, for example, in a Microsoft Kinect device capable of acquiring a hybrid depth/color image, wherein the color image, in this context the “original image” 10 is obtained using a regular camera and a depth map 13 is obtained from reflection patterns of a structured infrared image.” Kuster (34). A hybrid depth/color image can have frames from both video streams including the color and depth video streams.).
Consider claim 6. Kuster further teaches the generating the depth map includes cropping the second video stream based on the location of the face of the user (“In an embodiment, the method comprises the step of acquiring, for each original image, typically at the same time, an associated depth map comprising the face of the interlocutor, and wherein the step of synthesizing the corrected view of the interlocutor's face comprises mapping the original image onto a 3D model of the interlocutor's face based on the depth map” Kuster (13)).
Zhou also teaches the three-dimensional avatar based on body shape such as chin, ears, mouth, eyes, nose of the face of the user (col. 4, lines 15-36).
Consider claim 9. Kuster further teaches the method is performed by a local computing device (“all typically connected to a general-purpose computer 7 or dedicated video conferencing device comprising a data processing unit and programmed to perform the method as described herein.” Kuster (34)); and the camera is included in the local computing device (“They can be realized by using a real camera 1 combined with a depth scanner 2, as shown in a schematic fashion in FIG. 1.” Kuster (34)).
Consider claim 13. Kuster further teaches sending the presentation to a remote computing device (“at least one of displaying the final image and transmitting the final image, typically over a data communication network for display at another user's computer.” Kuster (34)).
Zhou also teaches sending three-dimensional avatar being based on less data than the video stream (col. 1, line 59 to col. 2, line 9; col. 99, lines 29-46).
Consider claim 18. Kuster discloses the non-transitory computer-readable storage medium of claim 17, wherein the second model (“For example, in various implementations, the depth prediction system enforces different supervisions across multi-data sources to generate a multi-branch architecture depth prediction model (e.g., a depth prediction machine-learning model).” Yin [0018]) was trained based on comparing depth data based on images captured by a depth camera to color data based on images captured by a color camera (“They can be realized by using a real camera 1 combined with a depth scanner 2, as shown in a schematic fashion in FIG. 1.” Kuster (34)).
Claims 8 and 10 are rejected under 35 U.S.C. 103 as being unpatentable over Valentin (EP3593323) in view of Zhou et al. (US 12,149,864) as applied to claim 1 above, and further in view of Gronau et al (US 20220345665).
Consider claim 8. Valentin in view Zhou does not expressively teach “wherein the generating the representation includes generating a representation of the user and an object held by the user.”
However, Gronau does teach wherein the generating the representation includes generating a representation of the user and an object held by the user (“As an example, a person speaking in a conference may be holding an object that is significant to the specific conference call or may not be significant at all. The speaker may be holding a pen which has no significance to the meeting or a diagram which is very significant to the meeting. To transmit these objects to the other viewers, they may be recognized and modelled as 3D objects.” Gronau [0300]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Valentin in view Zhou, with Gronau to be able “to enhance the virtual interaction between participants” (Gronau [0013]). In some instances, an object that a participant is holding may be critical to a virtual meeting i.e. a participant holding a product for an investor demonstration. In these cases, it is crucial that the 3D representation of the participant include the desired object of interest to best enhance the virtual conference between the participants.
Consider claim 10. Valentin in view Zhou do not expressively teach “wherein: the method is performed by a server that is remote from a local computing device.”
However, Gronau does teach wherein: the method is performed by a server that is remote from a local computing device (“The processing of this system may be performed on the user's device such as a computer, a phone or a tablet or on a remote computer such as a server on the cloud.” Gronau [0328]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Valentin in view Zhou, with Gronau to be able “to enhance the virtual interaction between participants” (Gronau [0013]). In some instances, an object that a participant is holding may be critical to a virtual meeting i.e. a participant holding a product for an investor demonstration. In these cases, it is crucial that the 3D representation of the participant include the desired object of interest to best enhance the virtual conference between the participants.
Claim 11 is rejected under 35 U.S.C. 103 as being unpatentable over Valentin (EP3593323) in view of Zhou et al. (US 12,149,864) as applied to claim 1 above, and further in view of Shan et al. (US 20180025248).
Consider claim 11. Valentin in view of Zhou does not expressively teach “determining whether to adjust the camera based on the location of the face of the user.”
However, Shan does teach further comprising determining whether to adjust the camera based on the location of the face of the user (“According to some embodiments, during shooting of the local writing area, the intelligent device may determine a current orientation of the camera and its position relative to the writing surface by detecting a changing angle of the head posture of the user.” Shan [0128]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Valentin in view of Zhou, with Shan to be able improve the communication between participants. If the face of a user is out of the scope of the camera, it could provide a disconnect between participants. Seeing the face of a participant can be indicative of their mood, receptivity to your words, and other important metrics which can be essential in ensuring a positive interaction between virtual participants.
Claim 12 is rejected under 35 U.S.C. 103 as being unpatentable over Valentin (EP3593323) in view of Zhou et al. (US 12,149,864) as applied to claim 1 above, and further in view of Yang et al (US 20230090916).
Consider claim 12. Valentin in view of Zhou does not expressively teach “adjusting the camera based on the location of the face of the user.”
However, Yang does teach further comprising adjusting the camera based on the location of the face of the user (“The controller adjusts a photographing angle of the camera according to the determined angle that the camera needs to rotate, so that a photographing area of the camera directly faces the position of the user when the user is speaking, and the photographing angle of the camera is adjusted according to the position of the user to photograph an image including the user.” Yang [0089]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Valentin in view of Zhou with Yang to be able improve the communication between participants. If the face of a user is out of the scope of the camera, it could provide a disconnect between participants. Seeing the face of a participant can be indicative of their mood, receptivity to your words, and other important metrics which can be essential in ensuring a positive interaction between virtual participants.
Claim 14 is rejected under 35 U.S.C. 103 as being unpatentable over Valentin (EP3593323) in view of Zhou et al. (US 12,149,864) and Kuster et al (US 9684953) as applied to claims 1, 13 and further in view of Ryu et. al (US 20170048481).
Consider claim 14. Valentin in view of Zhou and Kuster does not expressively teach “before sending the representation (i.e., avatar) of the user to the remote computing device, reducing a data size of the representation (i.e., the avatar) of the user.”
However, Ryu does teach further comprising, before sending the representation of the user to the remote computing device, reducing a data size of the representation of the user (“According to an embodiment, the processor 220 may resize an image to reduce a data size of the image, based on system resource information of the electronic device 200 (e.g., performance of the processor 220 or information indicating whether the electronic device 200 includes the resizer 223), feature information (e.g., resolution or a region of interest (ROI)) of the image, or specific capture quality information.” Ryu [0064]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Valentin in view of Zhou and Kuster, with Ryu to lower storage consumption and improve transmission speeds between devices.
Claim 16 is rejected under 35 U.S.C. 103 as being unpatentable over Zhou et al (12,149,864) in view of Nistico et. al (US 20150288944).
Consider claim 16. Zhou does not expressively teach “wherein the depth map includes distances of portions of the user from the camera.”
However, Nistico does teach wherein the depth map includes distances of portions of the user from the camera (“It has turned out to be advantageous when the processing unit is designed to calculate the 3D position of a point of regard, in particular the 3D position of the point of regard and the distances to each of the eyes, that the user is fixating using the 3D eye position of the left and the right eye and the orientation vectors of the left and the right eye.” Nistico [0014]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Zhou, with Nistico to be able to accurately create a 3D representation of the user. The depth prediction model must be able to provide depth information for different portions of the user’s face (i.e. eyes, nose, mouth, etc.) to be able to render the user properly in a three-dimensional environment.
Claim 20 is rejected under 35 U.S.C. 103 as being unpatentable over Valentin (EP3593323) in view of Zhou et al. (US 12,149,864) as applied to claim 19, and further in view of Ryu et. al (US 20170048481).
Consider claim 20. Valentin in view of Zhou does not expressively teach “reduce a data size of the representation of the user.”
However, Ryu does teach reduce a data size of the representation of the user (“According to an embodiment, the processor 220 may resize an image to reduce a data size of the image, based on system resource information of the electronic device 200 (e.g., performance of the processor 220 or information indicating whether the electronic device 200 includes the resizer 223), feature information (e.g., resolution or a region of interest (ROI)) of the image, or specific capture quality information.” Ryu [0064]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Valentin in view of Zhou, with Ryu to lower storage consumption and improve transmission speeds between devices.
Claim(s) 21 is/are rejected under 35 U.S.C. 103 as being unpatentable over Valentin (EP3593323) in view of Zhou et al. (US 12,149,864) as applied to claim 1, and further in view Molyneaux et al (US 9001118).
Consider claim 21. Valentin in view of Zhou does not teach generating the three-dimensional avatar includes: generating a depth mesh based on the depth map and a template mesh, the depth mesh following movements of the face of the user; and projecting colors from frames of the second video stream onto triangles included in the depth map.
Molyneaux teaches generating the three-dimensional avatar includes: generating a depth mesh based on the depth map and a template mesh, the depth mesh following movements of the face of the user; and projecting colors from frames of the second video stream onto triangles included in the depth map (abstract; col. 3, lines 18-44; col. 5, lines 27-38; col. 7, lines 3-12).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to utilize the teachings of Molyneaux into the teachings of Valentin in view of Zhou, the modification would improve the geometric accuracy, facial-motion tracking, visual realism, and texture fidelity of the generated avatar.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DUC NGUYEN whose telephone number is (571)272-7503. The examiner can normally be reached 6:30AM-3:45PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Duc Nguyen can be reached at 571-272-7503. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
DUC NGUYEN
Supervisory Patent Examiner
Art Unit 2691
/DUC NGUYEN/Supervisory Patent Examiner, Art Unit 2691
