Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-7, 9, 13, and 17-19 are rejected under 35 U.S.C. 103 as being unpatentable over Kuster et al (US 9684953) in view of Yu et al (US 20170091535), Yin et. al (US 20220284613), and Zhou et al (12,149,864).
Regarding claim 1, Kuster discloses a method comprising: receiving, via a camera (“real camera 1” Kuster (34); Fig. 1, 1), a first video stream of a face of a user (Fig. 3, 10; Kuster. This is the original photo taken by the color camera (Fig. 1, 1; Kuster).); receiving, via the camera (“In the remainder of the text, the gaze correction system and method shall be explained in terms of a system using only one real video camera (i.e., a color or black and white image camera) in addition to the depth map” Kuster (12); Fig. 1, 1), a second video stream of the face of the user (“The second one is a semi-automatic technique where two snapshots are taken from the Kinect's camera 1: one while the user is looking straight at the Kinect's camera 1 and one while the user is looking straight at the video conference window on the display 5.” Kuster (44)); generating a depth map based on the second video stream, the location of the face of the user (“In an embodiment, the method comprises the step of temporal smoothing of the 3D position of face tracking vertices over a sequence of depth maps by estimating the 3D position of the interlocutor's head in each depth map and combining the 3D position of the vertices as observed in a current depth map with a prediction of their position computed from their position in at least one preceding depth map and from the change in the head's 3D position and orientation, in particular: from the preceding depth map to the current depth map.” Kuster (19)); and generating a representation of the user based on the depth map and the second video stream (Fig. 2, 12; Kuster. The output is based on 2b, in which “the subject 9 makes eye contact using the geometry from the depth camera (FIG. 2b)” Kuster (35)).
Kuster does not expressively teach “determining a location of the face of the user based on the first video stream and a first model (i.e., a facial landmark detection)” and “a second model (i.e., depth prediction).”
However, Yu does teach determining a location of the face of the user based on the first video stream and a first model (“In one embodiment, the first images and the second images are processed at least to detect landmark locations associated with the user's eye and the user's eyebrow from the first images, and detect landmark locations associated with a lower portion of the user's face from the second images.” Yu [0009]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Kuster with Yu to provide a “more enriched interaction between users.” (Yu [0006]). Through this, “more information about a user's state of mind can be effectively communicated to other users in VR or AR environment” (Yu [0006]).
Additionally, Yin does teach and a second model (“For example, in various implementations, the depth prediction system enforces different supervisions across multi-data sources to generate a multi-branch architecture depth prediction model (e.g., a depth prediction machine-learning model).” Yin [0018]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Kuster, in view of Yu, with Yin to be able to accurately create a 3D representation of the user. The depth prediction model provides essential information in measuring real-time depth measurements to apply these measurements in accurately rendering the 3D representation of the user.
Kuster teaches generating a representation of three-dimensional representing the user based on the depth map and the second video stream (Fig. 2, 12; Kuster. The output is based on 2b, in which “the subject 9 makes eye contact using the geometry from the depth camera (FIG. 2b)” Kuster (35). However, Kuster in view of Yu and Yin does not teach the three-dimensional presentation (i.e., avatar) being based on less data than the second video stream.
Zhou teaches three-dimensional avatar being based on less data than the video stream (col. 1, line 59 to col. 2, line 9; col. 99, lines 29-46).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to utilize the teachings of Zhou into the teachings of Kuster in view of Yu and Yin, since that can make the video call more efficient and less data-heavy because the avatar rendering happens on the recipient computing device.
Regarding claim 2, Kuster further teaches the second model was trained based on comparing depth data based on images captured by a depth camera to color data based on images captured by a color camera (“In the remainder of the text, the gaze correction system and method shall be explained in terms of a system using only one real video camera (i.e., a color or black and white image camera) in addition to the depth map.” Kuster (12)).
Regarding claim 3, Kuster, further teaches the first video stream includes color data (“A combination of such elements and in particular of a camera and a depth scanner is embodied, for example, in a Microsoft Kinect device capable of acquiring a hybrid depth/color image, wherein the color image, in this context the “original image” 10 is obtained using a regular camera” Kuster (34)); and the second video stream includes color data (“The gaze correction system is targeted at a peer-to-peer video conferencing model that runs in real-time on average consumer hardware and, in one embodiment, requires only one hybrid depth/color sensor such as the Kinect.” Kuster (11). The hybrid system allows the second video stream to include color data along with the depth data).
Regarding claim 4, Kuster, further teaches the second video stream does not include depth data (“or with a polygon determined for a previous image of a sequence of images” Kuster (17). According to the applicant’s specification, “In some examples, the local computing device 102 captures only a single video stream (which can be analyzed at different starting and ending points for a first video stream, a second video stream, and/or a third video stream) with only a single color camera” [0033]. As a result, a plurality of frames could each be treated as a different video stream.).
Regarding claim 5. Kuster further teaches at least one frame included in the first video stream is included in the second video stream (“A combination of such elements and in particular of a camera and a depth scanner is embodied, for example, in a Microsoft Kinect device capable of acquiring a hybrid depth/color image, wherein the color image, in this context the “original image” 10 is obtained using a regular camera and a depth map 13 is obtained from reflection patterns of a structured infrared image.” Kuster (34). A hybrid depth/color image can have frames from both video streams including the color and depth video streams.).
Regarding claim 6. Kuster further teaches the generating the depth map includes cropping the second video stream based on the location of the face of the user (“In an embodiment, the method comprises the step of acquiring, for each original image, typically at the same time, an associated depth map comprising the face of the interlocutor, and wherein the step of synthesizing the corrected view of the interlocutor's face comprises mapping the original image onto a 3D model of the interlocutor's face based on the depth map” Kuster (13)).
Regarding claim 7. Kuster further teaches the generating the presentation includes cropping the second video stream based on the location of the face of the user (Fig 2, 12; Kuster).
Zhou teaches the three-dimensional avatar based on body shape such as chin, ears, mouth, eyes, nose of the face of the user (col. 4, lines 15-36).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to utilize the teachings of Zhou into the teachings of Kuster in view of Yu and Yin, since it can make the video call more efficient and less data-heavy because the avatar rendering happens on the recipient computing device.
Regarding claim 9. Kuster further teaches the method is performed by a local computing device (“all typically connected to a general-purpose computer 7 or dedicated video conferencing device comprising a data processing unit and programmed to perform the method as described herein.” Kuster (34)); and the camera is included in the local computing device (“They can be realized by using a real camera 1 combined with a depth scanner 2, as shown in a schematic fashion in FIG. 1.” Kuster (34)).
Regarding claim 13. Kuster in view of Yu and Yin, discloses the method of claim 1, further comprising sending the presentation to a remote computing device (“at least one of displaying the final image and transmitting the final image, typically over a data communication network for display at another user's computer.” Kuster (34)).
Zhou teaches sending three-dimensional avatar being based on less data than the video stream (col. 1, line 59 to col. 2, line 9; col. 99, lines 29-46).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to utilize the teachings of Zhou into the teachings of Kuster in view of Yu and Yin, since that can make the video call more efficient and less data-heavy because the avatar rendering happens on the recipient computing device.
Regarding claim 17. Kuster discloses a non-transitory computer-readable storage medium comprising instructions stored thereon that, when executed by at least one processor, are configured to cause a computing device to (“A method of manufacturing a non-transitory computer readable medium comprises the step of storing, on the computer readable medium, computer-executable instructions which, when executed by a processor of a computing system, cause the computing system to perform the method steps for image processing in video conferencing as described in the present document.” Kuster (32)): receive, via a camera (“real camera 1” Kuster (34); Fig. 1, 1), a first video stream of a face of a user (Fig. 3, 10; Kuster. This is the original photo taken by the color camera (Fig. 1, 1; Kuster).); receive, via the camera (“In the remainder of the text, the gaze correction system and method shall be explained in terms of a system using only one real video camera (i.e., a color or black and white image camera) in addition to the depth map” Kuster (12); Fig. 1, 1), a second video stream of the face of the user (“The second one is a semi-automatic technique where two snapshots are taken from the Kinect's camera 1: one while the user is looking straight at the Kinect's camera 1 and one while the user is looking straight at the video conference window on the display 5.” Kuster (44)); generate a depth map based on the second video stream, the location of the face of the user (“In an embodiment, the method comprises the step of temporal smoothing of the 3D position of face tracking vertices over a sequence of depth maps by estimating the 3D position of the interlocutor's head in each depth map and combining the 3D position of the vertices as observed in a current depth map with a prediction of their position computed from their position in at least one preceding depth map and from the change in the head's 3D position and orientation, in particular: from the preceding depth map to the current depth map.” Kuster (19)).
Kuster does not expressively teach determine a location of the face of the user based on the first video stream and a first model (i.e., facial landmark detection model)” and “and a second model (i.e., a depth prediction model.).
However, Yu teaches determine a location of the face of the user based on the first video stream and a first model (“In one embodiment, the first images and the second images are processed at least to detect landmark locations associated with the user's eye and the user's eyebrow from the first images, and detect landmark locations associated with a lower portion of the user's face from the second images.” Yu [0009]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Kuster with Yu to provide a “more enriched interaction between users.” (Yu [0006]). Through this, “more information about a user's state of mind can be effectively communicated to other users in VR or AR environment” (Yu [0006]).
Additionally, Yin does teach and a second model (“For example, in various implementations, the depth prediction system enforces different supervisions across multi-data sources to generate a multi-branch architecture depth prediction model (e.g., a depth prediction machine-learning model).” Yin [0018]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Kuster, in view of Yu, with Yin to be able to accurately create a 3D representation of the user. The depth prediction model provides essential information in measuring real-time depth measurements to apply these measurements in accurately rendering the 3D representation of the user.
However, Kuster in view of Yu and Yin does not teach the three-dimensional presentation (i.e., avatar) being based on less data than the second video stream.
Zhou teaches three-dimensional avatar being based on less data than the video stream (col. 1, line 59 to col. 2, line 9; col. 99, lines 29-46).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to utilize the teachings of Zhou into the teachings of Kuster in view of Yu and Yin, since that can make the video call more efficient and less data-heavy because the avatar rendering happens on the recipient computing device.
Regarding claim 18. Kuster, in view of Yu and Yin, discloses the non-transitory computer-readable storage medium of claim 17, wherein the second model (“For example, in various implementations, the depth prediction system enforces different supervisions across multi-data sources to generate a multi-branch architecture depth prediction model (e.g., a depth prediction machine-learning model).” Yin [0018]) was trained based on comparing depth data based on images captured by a depth camera to color data based on images captured by a color camera (“They can be realized by using a real camera 1 combined with a depth scanner 2, as shown in a schematic fashion in FIG. 1.” Kuster (34)).
Regarding claim 19. Kuster discloses a computing device comprising: at least one processor (“In an embodiment, a computer program or a computer program product for image processing in video conferencing is loadable into an internal memory of a digital computer or a computer system, and comprises computer-executable instructions to cause one or more processors of the computer or computer system execute the method for image processing in video conferencing.” Kuster (31)); and a non-transitory computer-readable storage medium comprising instructions stored thereon that, when executed by the at least one processor, are configured to cause the computing device to (“A method of manufacturing a non-transitory computer readable medium comprises the step of storing, on the computer readable medium, computer-executable instructions which, when executed by a processor of a computing system, cause the computing system to perform the method steps for image processing in video conferencing as described in the present document.” Kuster (32)): receive, via a camera (“real camera 1” Kuster (34); Fig. 1, 1), a first video stream of a face of a user (Fig. 3, 10; Kuster. This is the original photo taken by the color camera (Fig. 1, 1; Kuster).); receive, via the camera (“In the remainder of the text, the gaze correction system and method shall be explained in terms of a system using only one real video camera (i.e., a color or black and white image camera) in addition to the depth map” Kuster (12); Fig. 1, 1), a second video stream of the face of the user (“The second one is a semi-automatic technique where two snapshots are taken from the Kinect's camera 1: one while the user is looking straight at the Kinect's camera 1 and one while the user is looking straight at the video conference window on the display 5.” Kuster (44)); generate a depth map based on the second video stream, the location of the face of the user (“In an embodiment, the method comprises the step of temporal smoothing of the 3D position of face tracking vertices over a sequence of depth maps by estimating the 3D position of the interlocutor's head in each depth map and combining the 3D position of the vertices as observed in a current depth map with a prediction of their position computed from their position in at least one preceding depth map and from the change in the head's 3D position and orientation, in particular: from the preceding depth map to the current depth map.” Kuster (19)).
Kuster does not expressively teach determine a location of the face of the user based on the first video stream and a first model (i.e., facial landmark detection model) and “and a second model (i.e., depth prediction model.).
However, Yu teaches determine a location of the face of the user based on the first video stream and a first model (“In one embodiment, the first images and the second images are processed at least to detect landmark locations associated with the user's eye and the user's eyebrow from the first images, and detect landmark locations associated with a lower portion of the user's face from the second images.” Yu [0009]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Kuster with Yu to provide a “more enriched interaction between users.” (Yu [0006]). Through this, “more information about a user's state of mind can be effectively communicated to other users in VR or AR environment” (Yu [0006]).
Additionally, Yin does teach and a second model (“For example, in various implementations, the depth prediction system enforces different supervisions across multi-data sources to generate a multi-branch architecture depth prediction model (e.g., a depth prediction machine-learning model).” Yin [0018]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Kuster, in view of Yu, with Yin to be able to accurately create a 3D representation of the user. The depth prediction model provides essential information in measuring real-time depth measurements to apply these measurements in accurately rendering the 3D representation of the user.
However, Kuster in view of Yu and Yin does not teach the three-dimensional presentation (i.e., avatar) being based on less data than the second video stream.
Zhou teaches three-dimensional avatar being based on less data than the video stream (col. 1, line 59 to col. 2, line 9; col. 99, lines 29-46).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to utilize the teachings of Zhou into the teachings of Kuster in view of Yu and Yin, since that can make the video call more efficient and less data-heavy because the avatar rendering happens on the recipient computing device.
Claims 8 and 10 are rejected under 35 U.S.C. 103 as being unpatentable over Kuster et al (US 9684953) in view of Yu et al (US 20170091535), Yin et al (US 20220284613), and Zhou et al (12,149,864) as applied to claim 1 above, and further in view of Gronau et al (US 20220345665).
Regarding claim 8. Kuster in view of Yu, Yin and Zhou, discloses the method of claim 1, based on the depth map and the second video stream (“In an embodiment, the method comprises the step of acquiring, for each original image, typically at the same time, an associated depth map comprising the face of the interlocutor, and wherein the step of synthesizing the corrected view of the interlocutor's face comprises mapping the original image onto a 3D model of the interlocutor's face based on the depth map” Kuster (13)).
Kuster, in view of Yu, Yin and Zhou, does not expressively teach “wherein the generating the representation includes generating a representation of the user and an object held by the user.”
However, Gronau does teach wherein the generating the representation includes generating a representation of the user and an object held by the user (“As an example, a person speaking in a conference may be holding an object that is significant to the specific conference call or may not be significant at all. The speaker may be holding a pen which has no significance to the meeting or a diagram which is very significant to the meeting. To transmit these objects to the other viewers, they may be recognized and modelled as 3D objects.” Gronau [0300]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Kuster in view of Yu, Yin and Zhou, with Gronau to be able “to enhance the virtual interaction between participants” (Gronau [0013]). In some instances, an object that a participant is holding may be critical to a virtual meeting i.e. a participant holding a product for an investor demonstration. In these cases, it is crucial that the 3D representation of the participant include the desired object of interest to best enhance the virtual conference between the participants.
Regarding claim 10. Kuster, in view of Yu, Yin and Zhou, discloses the method of claim 1, and the camera is included in the local computing device (“They can be realized by using a real camera 1 combined with a depth scanner 2, as shown in a schematic fashion in FIG. 1.” Kuster (34)).
Kuster, in view of Yu, Yin and Zhou, do not expressively teach “wherein: the method is performed by a server that is remote from a local computing device.”
However, Gronau does teach wherein: the method is performed by a server that is remote from a local computing device (“The processing of this system may be performed on the user's device such as a computer, a phone or a tablet or on a remote computer such as a server on the cloud.” Gronau [0328]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Kuster in view of Yu, Yin and Zhou, with Gronau to be able “to enhance the virtual interaction between participants” (Gronau [0013]). In some instances, an object that a participant is holding may be critical to a virtual meeting i.e. a participant holding a product for an investor demonstration. In these cases, it is crucial that the 3D representation of the participant include the desired object of interest to best enhance the virtual conference between the participants.
Claim 11 is rejected under 35 U.S.C. 103 as being unpatentable over Kuster et al (US 9684953) in view of Yu et al (US 20170091535), Yin et. al (US 20220284613), and Zhou et al (12,149,864) as applied to claim 1 above, and further in view of Shan et. al (US 20180025248).
Regarding claim 11. Kuster in view of Yu, Yin and Zhou, discloses the method of claim 1.
Kuster in view of Yu, Yin and Zhou, does not expressively teach “determining whether to adjust the camera based on the location of the face of the user.”
However, Shan does teach further comprising determining whether to adjust the camera based on the location of the face of the user (“According to some embodiments, during shooting of the local writing area, the intelligent device may determine a current orientation of the camera and its position relative to the writing surface by detecting a changing angle of the head posture of the user.” Shan [0128]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Kuster in view of Yu, Yin and Zhou, with Shan to be able improve the communication between participants. If the face of a user is out of the scope of the camera, it could provide a disconnect between participants. Seeing the face of a participant can be indicative of their mood, receptivity to your words, and other important metrics which can be essential in ensuring a positive interaction between virtual participants.
Claim 12 is rejected under 35 U.S.C. 103 as being unpatentable over Kuster et al (US 9684953) in view of Yu et al (US 20170091535), Yin et. al (US 20220284613), and Zhou et al (12,149,864) as applied to claim 1 above, and further in view of Yang et. al (US 20230090916 A1, hereinafter Yang).
Regarding claim 12. Kuster, in view of Yu and Yin and Zhou, discloses the method of claim 1.
Kuster in view of Yu, Yin and Zhou, does not expressively teach “adjusting the camera based on the location of the face of the user.”
However, Yang does teach further comprising adjusting the camera based on the location of the face of the user (“The controller adjusts a photographing angle of the camera according to the determined angle that the camera needs to rotate, so that a photographing area of the camera directly faces the position of the user when the user is speaking, and the photographing angle of the camera is adjusted according to the position of the user to photograph an image including the user.” Yang [0089]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Kuster in view of Yu, Yin and Zhou, with Yang to be able improve the communication between participants. If the face of a user is out of the scope of the camera, it could provide a disconnect between participants. Seeing the face of a participant can be indicative of their mood, receptivity to your words, and other important metrics which can be essential in ensuring a positive interaction between virtual participants.
Claims 14 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Kuster et al (US 9684953) in view of Yu et al (US 20170091535), Yin et. al (US 20220284613), and Zhou et al (12,149,864) as applied to claims 1, 13, 19, and further in view of Ryu et. al (US 20170048481).
Regarding claim 14. Kuster in view of Yu, Yin and Zhou, discloses the method of claim 13.
Kuster in view of Yu, Yin and Zhou, does not expressively teach “before sending the representation (i.e., avatar) of the user to the remote computing device, reducing a data size of the representation (i.e., the avatar) of the user.”
However, Ryu does teach further comprising, before sending the representation of the user to the remote computing device, reducing a data size of the representation of the user (“According to an embodiment, the processor 220 may resize an image to reduce a data size of the image, based on system resource information of the electronic device 200 (e.g., performance of the processor 220 or information indicating whether the electronic device 200 includes the resizer 223), feature information (e.g., resolution or a region of interest (ROI)) of the image, or specific capture quality information.” Ryu [0064]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Kuster in view of Yu, Yin and Zhou, with Ryu to lower storage consumption and improve transmission speeds between devices.
Regarding claim 20. Kuster, in view of Yu, Yin and Zhou, discloses the computing device of claim 19, wherein the instructions are further configured to cause the computing device to: and send the representation of the user to a remote computing device (“at least one of displaying the final image and transmitting the final image, typically over a data communication network for display at another user's computer.” Kuster (34)).
Kuster in view of Yu, Yin and Zhou, does not expressively teach “reduce a data size of the representation of the user.”
However, Ryu does teach reduce a data size of the representation of the user (“According to an embodiment, the processor 220 may resize an image to reduce a data size of the image, based on system resource information of the electronic device 200 (e.g., performance of the processor 220 or information indicating whether the electronic device 200 includes the resizer 223), feature information (e.g., resolution or a region of interest (ROI)) of the image, or specific capture quality information.” Ryu [0064]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Kuster in view of Yu, Yin and Zhou, with Ryu to lower storage consumption and improve transmission speeds between devices.
Claim 15 is rejected under 35 U.S.C. 103 as being unpatentable over Kuster et. al (US 9684953) in view of Yin et. al (US 20220284613) and Zhou et al (12,149,864).
Regarding claim 15, Kuster discloses a method comprising: receiving, via a camera (“real camera 1” Kuster (34); Fig. 1, 1), a video stream of a face of a user (Fig. 3, 10; Kuster. This is the original photo taken by the color camera (Fig. 1, 1; Kuster).); generating a depth map based on the video stream, a location of the face of the user (“In an embodiment, the method comprises the step of temporal smoothing of the 3D position of face tracking vertices over a sequence of depth maps by estimating the 3D position of the interlocutor's head in each depth map and combining the 3D position of the vertices as observed in a current depth map with a prediction of their position computed from their position in at least one preceding depth map and from the change in the head's 3D position and orientation, in particular: from the preceding depth map to the current depth map.” Kuster (19)); and generating a representation of the user based on the depth map and the video stream (Fig. 2, 12; Kuster. The output is based on 2b, in which “the subject 9 makes eye contact using the geometry from the depth camera (FIG. 2b)” Kuster (35).).
Kuster does not expressively teach “and a neural network.”
However, Yin does teach and a neural network (“For example, in various implementations, the depth prediction system enforces different supervisions across multi-data sources to generate a multi-branch architecture depth prediction model (e.g., a depth prediction machine-learning model).” Yin [0018]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Kuster with Yin to be able to accurately create a 3D representation of the user. The depth prediction model provides essential information in measuring real-time depth measurements to apply these measurements in accurately rendering the 3D representation of the user.
However, Kuster in view of Yin does not teach the three-dimensional presentation (i.e., avatar) being based on less data than the second video stream.
Zhou teaches three-dimensional avatar being based on less data than the video stream (col. 1, line 59 to col. 2, line 9; col. 99, lines 29-46).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to utilize the teachings of Zhou into the teachings of Kuster in view of Yin, since that can make the video call more efficient and less data-heavy because the avatar rendering happens on the recipient computing device.
Claim 16 is rejected under 35 U.S.C. 103 as being unpatentable over Kuster et. al (US 9684953) in view of Yin et. al (US 20220284613) and Zhou et al (12,149,864) as applied to claim 15 above, and further in view of Nistico et. al (US 20150288944).
Regarding claim 16, Kuster in view of Yin and Zhou, does not expressively teach “wherein the depth map includes distances of portions of the user from the camera.”
However, Nistico does teach wherein the depth map includes distances of portions of the user from the camera (“It has turned out to be advantageous when the processing unit is designed to calculate the 3D position of a point of regard, in particular the 3D position of the point of regard and the distances to each of the eyes, that the user is fixating using the 3D eye position of the left and the right eye and the orientation vectors of the left and the right eye.” Nistico [0014]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Kuster, in view of Yin and Zhou, with Nistico to be able to accurately create a 3D representation of the user. The depth prediction model must be able to provide depth information for different portions of the user’s face (i.e. eyes, nose, mouth, etc.) to be able to render the user properly in a three-dimensional environment.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DUC M NGUYEN whose telephone number is (571)272-7503. The examiner can normally be reached 6:30AM-3:45PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is
encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Duc M. Nguyen can be reached at 571-272-7503. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
DUC M. NGUYEN
Supervisory Patent Examiner
Art Unit 2691
/DUC NGUYEN/Supervisory Patent Examiner, Art Unit 2691
Prosecution Insights