DETAILED ACTION
Notice of Pre-AIA or AIA Status
1. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
2. Applicant’s amendments filed on 08/06/2026 have been entered. Claims 1-4, 6-7, 9-14 and 16-20 have been amended. Claims 1-20 are pending in this application, with claims 1, 11 and 20 being independent.
Response to Arguments
3. Applicant's arguments filed on 08/06/2026, with respect to the 103 rejection have been fully considered but are moot in view of the new grounds of rejection.
In light of the current Office Action, the Examiner respectfully submits that independent claims 1, 11 and 20 are rejected in view of newly discovered reference(s) to Buddemeier et al. (US-2003/0043153-A1).
Examiner notes that independent claims 1, 11 and 20 have been amended to include new limitation. Examiner finds these limitations to be unpatentable as can be found in below detail action.
On pages 10-11 of Applicant's Remarks, the Applicant argues that the dependent claims are not taught by the prior art, insomuch as they depend from claims that are not taught by the prior art. Examiner respectfully disagrees with these arguments, for the reasons discussed below.
Claim Rejections - 35 USC § 103
4. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
5. Claims 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over Raghoebardajal et al., (“Raghoebardajal”) [US-2016/0217621-A1] in view of Vorskus, (“Vorskus”) [WO-2023/278081-A1], further in view of Buddemeier et al., (“Buddemeier”) [US-2003/0043153-A1]
Regarding claim 1, Raghoebardajal discloses a method of generating and rendering an image, on a first device having a display (Raghoebardajal- ¶0037, at least discloses the display is not a head-mountable display […] the display (whether head mountable or not) may be referred to as an immersive display, in that in normal use it fills at least a threshold angular range (for example, at least 40°) of the field of view of the user) the image being representative of a user of a communications system (Raghoebardajal- ¶0065, at least discloses There is a desire to implement the use of HMD devices in this context, as an HMD device could be used to provide an even more immersive experience, making a user feel as if they are present in the room with the other person or persons in the call; ¶0067-0068, at least discloses A solution to this problem is to use image processing means to process images for display, for example to the other party or parties in a communications situation, and in doing so either to render the HMD device as a partly see-through object, or to remove it from the image recorded by the camera altogether, rendering the previously-occluded area of the face (or another feature) in its place […] The HMD may be identified and tracked (relative to the camera capturing the images of the HMD-wearing user) in a series of images [image being representative of a user] using any known means, such as the use of a fiducial marker attached to the device or the use of hardware motion detectors mounted on the HMD), the method comprising:
receiving, by the first device from a second device, a current facial expression represented in sensor data associated with the second device, the second device having a three-dimensional display (Raghoebardajal- ¶0061, at least discloses if motion or other sensors are provided at the HMD 20, signals from those sensors may be at least partially processed at the HMD 20 and/or may be at least partially processed at the games console 2000; ¶0064, at least discloses where an interaction between two devices is basically symmetrical, for example where a camera or sensor on one device detects a signal or feature of the other device, it will be understood that unless the context prohibits this, the two devices could be interchanged without any loss of functionality; ¶0082, at least discloses This feature could be implemented by applying the image processing upon receiving an image from the other user's camera as opposed to the other user transmitting an already processed image”, disclosing the processing at the recipient corresponding to the first device of the claim; Fig. 18 and ¶0101, at least discloses FIG. 18 schematically illustrates the mapping of facial motion to the rendering of a head model. As discussed above, the 3D facial model may comprise a mapping of the interactions of facial muscles. In such a situation, and referring to FIG. 18, at a step 660 the facial motion of the user wearing the HMD is detected (for example, by the cameras of FIG. 15 or by one or more other cameras directed towards the user's face, and/or by electrical or other sensors such as one or more EMG sensors of FIG. 1.”, the facial motion and interactions of facial muscles correspond to expression data. They are extracted from the images from the cameras. According to ¶0082, these images including expression data are received by the first device; see also ¶0102, “The step of detecting facial muscle motion may comprise estimating the facial muscle motion from captured images.”);
inputting, by the first device, the expression to a user-specific generative model trained on enrollment image data comprising an enrollment image indicative of
baseline facial image content associated with the user (Raghoebardajal- ¶0101, at least discloses mapping of facial motion to the rendering of a head model; Fig. 17 and ¶0100, at least disclose FIG. 17 schematically illustrates a method for generating a head model, for example using the arrangement discussed with reference to FIG. 12, comprising: at a step 640, capturing one or more images of a user's face (without the HMD in place) and, at a step 650, deriving the 3D facial model from captured images of the user's face.”, disclosing that the model is trained on an enrollment image);
receiving, by the first device from the user-specific generative model, facial image information usable to render an video frame on the first device (Raghoebardajal- ¶0042, at least discloses the arrangement of FIG. 1 provides an example of a head-mountable display system comprising a frame to be mounted onto an observer's head, the frame defining one or two eye display positions which, in use, are positioned in front of a respective eye of the observer and a display element mounted with respect to each of the eye display positions, the display element providing a virtual image of a video display of a video signal from a video signal source to that eye of the observer; ¶0101, at least discloses At a step 670 the facial model used in the rendering of the user's head is modified according to the detected facial movements; ¶0102, at least discloses This therefore provides an example of detecting facial muscle motion; and changing the generation of the modified image in accordance with the detected facial muscle motion. The step of detecting facial muscle motion may comprise estimating the facial muscle motion from captured images), the video frame including facial image content corresponding to the current facial expression excluding visor image content (Raghoebardajal- ¶0069, at least discloses Image processing means (such as comparison to a reference image) can then be used to identify the region of the user's face that is obscured by the HMD, as although it is mounted in a position that does not change, the orientation of the HMD with respect to the camera will cause different areas to be occluded as illustrated in FIGS. 11a and 11 b; ¶0074-0075, at least disclose A model of the HMD-wearing user's head and face (or at least a part of the head and face) is generated and is used to render occluded areas of the HMD-wearing user's head and face in images displayed at another display (such as another caller's HMD) […] the user's actual head and face (or parts of their head and face) could be modelled. In other examples, a model of another head and face could be used, either for the occluded portions or in place of the user's head and face altogether […] The camera (or another camera) is used prior to the HMD being put on by the user to take pictures of the user's head from a number of angles; ¶0082, at least discloses This feature could be implemented by applying the image processing upon receiving an image from the other user's camera as opposed to the other user transmitting an already processed image”, disclosing the processing at the recipient corresponding to the first device of the claim; Fig. 18 and ¶0101, at least discloses FIG. 18 schematically illustrates the mapping of facial motion to the rendering of a head model. As discussed above, the 3D facial model may comprise a mapping of the interactions of facial muscles. In such a situation, and referring to FIG. 18, at a step 660 the facial motion of the user wearing the HMD is detected (for example, by the cameras of FIG. 15 or by one or more other cameras directed towards the user's face, and/or by electrical or other sensors such as one or more EMG sensors of FIG. 1.”, the facial motion and interactions of facial muscles correspond to expression data. They are extracted from the images from the cameras. According to ¶0082, these images including expression data are received by the first device); and
rendering, by the first device, the video frame on the display in context of an on-going session of the communications system (Raghoebardajal- ¶0074, at least discloses A model of the HMD-wearing user's head and face (or at least a part of the head and face) is generated and is used to render occluded areas of the HMD-wearing user's head and face in images displayed at another display (such as another caller's HMD). As mentioned above, various options are available. For example, the user's actual head and face (or parts of their head and face) could be modelled. In other examples, a model of another head and face could be used, either for the occluded portions or in place of the user's head and face altogether. In examples, the rendering process is such that the HMD is completely removed from the images. In other embodiments, the parts of the image relating to the HMD are mixed or combined with the head/face model so as to render the HMD as though it was transparent).
Raghoebardajal does not explicitly disclose a first device having a two-dimensional display; an expression vector that encodes a current facial expression represented in sensor data associated with the second device, the expression vector comprising a compact vector representation of facial-expression information; facial image information comprising two-dimensional pixel information usable to render an two-dimensional video frame, the two-dimensional video frame including facial image content corresponding to the current facial expression represented by the expression vector; rendering, by the first device, the two-dimensional video frame on the two-dimensional display.
However, Virskus disloses
a first device having a two-dimensional display (Virskus- ¶0048, at least discloses the holographic calling system can provide a holographic calls between any combination of artificial reality (XR) devices, mobile devices (e.g., smartphones), traditional computing systems (e.g., a laptop or desktop with a webcam), etc. The experience of each user may depend on the capture and processing capabilities of the sending system and the display and processing capabilities of the receiving system. For example, where one user is using a mobile phone and another is using an XR system, the user of the XR system may see a hologram of the other user in their space while the mobile phone user may see a representation of the other user on their phone screen);
two-dimensional pixel information usable to render an two-dimensional video frame (Virskus- ¶0048, at least discloses the holographic calling system can provide a holographic calls between any combination of artificial reality (XR) devices, mobile devices (e.g., smartphones), traditional computing systems (e.g., a laptop or desktop with a webcam), etc. The experience of each user may depend on the capture and processing capabilities of the sending system and the display and processing capabilities of the receiving system. For example, where one user is using a mobile phone and another is using an XR system, the user of the XR system may see a hologram of the other user in their space while the mobile phone user may see a representation of the other user on their phone screen; ¶0087, at least discloses the creation of such synthetic images can include labeling each pixel of the image with depth data from a virtual camera at which the perspective of the image is created, labeling portions as foreground/background, body parts, and XR device, and with assigned points and point connections for a body model; ¶0094, at least discloses rendering the 3D mesh on a display or projecting the 3D mesh into the eyes of the user […] this projection can be accomplished by flattening the 3D mesh into multiple 2D planes (one for each eye with different perspectives to preserve the 3D appearance to the user) and painting these 2D planes into an image, e.g., using MEMS mirrors and microLEDs, lasers, or other color projection systems, this light can then pass through a series of waveguides and filters to project them into a user's eyes);
facial image information comprising two-dimensional pixel information usable to render an two-dimensional video frame, the two-dimensional video frame including facial image content (Virskus- ¶0051, at least discloses the densification can occur in systems that use structured light or where the captured depth data otherwise does not include a depth value for each pixel in a corresponding color image; ¶0057, at least discloses during the holographic call, the holographic calling system can analyze an input image of the user wearing the XR device to determine a head pose and facial expression of the user. The holographic calling system can then select one or more of the expression meshes corresponding to the determined facial expressions and blend them into an output mesh corresponding to the user's current expression; ¶0087, at least discloses the creation of such synthetic images can include labeling each pixel of the image with depth data from a virtual camera at which the perspective of the image is created, labeling portions as foreground/background, body parts, and XR device, and with assigned points and point connections for a body model; ¶00206, at least discloses the depth image and color image are used at the sender device as input to a machine learning model at 1206 that is trained to density the depth image (creating a depth value for each pixel of the depth image), segment the images to generate segmenting masks, and generate a body model.);
rendering, by the first device, the two-dimensional video frame on the two-dimensional display (Virskus- ¶0048, at least discloses the holographic calling system can provide a holographic calls between any combination of artificial reality (XR) devices, mobile devices (e.g., smartphones), traditional computing systems (e.g., a laptop or desktop with a webcam), etc. The experience of each user may depend on the capture and processing capabilities of the sending system and the display and processing capabilities of the receiving system. For example, where one user is using a mobile phone and another is using an XR system, the user of the XR system may see a hologram of the other user in their space while the mobile phone user may see a representation of the other user on their phone screen; ¶0094, at least discloses rendering the 3D mesh on a display or projecting the 3D mesh into the eyes of the user […] this projection can be accomplished by flattening the 3D mesh into multiple 2D planes (one for each eye with different perspectives to preserve the 3D appearance to the user) and painting these 2D planes into an image, e.g., using MEMS mirrors and microLEDs, lasers, or other color projection systems, this light can then pass through a series of waveguides and filters to project them into a user's eyes; ¶00184, at least discloses the color and depth data can be obtained from one or more previous captured frames during the holographic call (e.g., frames where process 1000 determines that the sending user's torso is not occluded). In some cases, process 1000 can accumulate portions of multiple such frames, knitting them together to fill in portions of the sending user that are occluded in the frames; ¶00192, at least discloses the training items are in the form of a sequence from a video, in which case model output from one or more previous frames can be stored and then also used as additional input for the next frame).
It would have been obvious to one of ordinary in the art before the effective filing date of the claimed invention to have modified Raghoebardajal to incorporate the teachings of Virskus, and apply the two-dimensional display into the Raghoebardajal’s teachings for receiving, by the first device from the user-specific generative model, facial image information usable to render a two-dimensional video frame on the first device; and rendering, by the first device, the two-dimensional video frame on the two-dimensional display in context of an on-going session of the communications system.
Doing so would provide users an interactive calling experience where the users experience greater presence with the other conversation participants, as compared to existing communication system such as voice or video calling.
The prior art does not explicitly disclose, but Buddemeier discloses
an expression vector that encodes a current facial expression represented in sensor data associated with the second device (Buddemeier- ¶0006, at least discloses the animation vector may be determined by sensing facial features, associated with independent targets, of a human subject. Alternatively, the animation vector is obtained by applying facial feature position sensing to annotated vertices on a model; ¶0012, at least discloses the animation vector is an abstracted sensing result and is a most compact representation of the facial expression as determined by audio-visual sensing), the expression vector comprising a compact vector representation of facial-expression information (Buddemeier- ¶0012, at least discloses the animation vector is an abstracted sensing result and is a most compact representation of the facial expression as determined by audio-visual sensing);
the current facial expression represented by the expression vector (Buddemeier- ¶0006, at least discloses the animation vector may be determined by sensing facial features, associated with independent targets, of a human subject. Alternatively, the animation vector is obtained by applying facial feature position sensing to annotated vertices on a model; ¶0012, at least discloses the animation vector is an abstracted sensing result and is a most compact representation of the facial expression as determined by audio-visual sensing);
It would have been obvious to one of ordinary in the art before the effective filing date of the claimed invention to have modified Raghoebardajal/Virskus to incorporate the teachings of Buddemeier, and apply the animation vector is a most compact representation of the facial expression into the Raghoebardajal/Virskus’s teachings for receiving, by the first device from a second device, an expression vector that encodes a current facial expression represented in sensor data associated with the second device, the expression vector comprising a compact vector representation of facial-expression information, the second device having a three-dimensional display; receiving, by the first device from the user-specific generative model, facial image information comprising two-dimensional pixel information usable to render a two-dimensional video frame on the first device, the two-dimensional video frame including facial image content corresponding to the current facial expression represented by the expression vector and excluding visor image content.
Doing so would provide a technique for translating facial animation values to head mesh positions for rendering facial features of an animated avatar.
Regarding claim 2, Raghoebardajal in view of Virskus and Buddemeier, discloses the method of claim 1, and further discloses wherein the expression vector is received from one of a webcam or a VR headset (Raghoebardajal- ¶0010, at least discloses By using such a webcam or a comparable image capturing device in conjunction with such a service, users may be able to transmit real-time footage of themselves to their contact in addition to their voice; ¶0087, at least discloses information about the user's eye movements (including blinking) may be incorporated in order to render motion of the virtual eyes for improved realism. This could be motion that is determined by a computer or actual detected motion of the user's eyes, for example by eye tracking cameras mounted inside the HMD"; Virskus- ¶0048, at least discloses webcam; ¶0100, at least discloses cameras or sensors on the XR headset can also capture color and/or depth data, e.g., tracking the position of the user's eyes, facial appearance behind the XR headset, etc.; Buddemeier- ¶0006, at least discloses the animation vector may be determined by sensing facial features, associated with independent targets, of a human subject. Alternatively, the animation vector is obtained by applying facial feature position sensing to annotated vertices on a model; ¶0012, at least discloses the animation vector is an abstracted sensing result and is a most compact representation of the facial expression as determined by audio-visual sensing).
It would have been obvious to one of ordinary in the art before the effective filing date of the claimed invention to have modified Raghoebardajal to incorporate the teachings of Virskus and Buddemeier, and apply the webcam or a VR headset and the expression vector into the Raghoebardajal’s teachings in order the expression vector data is received from one of a webcam or a VR headset.
The same motivation that was utilized in the rejection of claim 1 applies equally to this claim.
Regarding claim 3, Raghoebardajal in view of Virskus and Buddemeier, discloses the method of claim 1, and further discloses wherein the facial image information excludes visor image content corresponding to a visor worn by the user (Raghoebardajal- ¶0069, at least discloses Image processing means (such as comparison to a reference image) can then be used to identify the region of the user's face that is obscured by the HMD, as although it is mounted in a position that does not change, the orientation of the HMD with respect to the camera will cause different areas to be occluded as illustrated in FIGS. 11a and 11 b; ¶0074-0075, at least disclose A model of the HMD-wearing user's head and face (or at least a part of the head and face) is generated and is used to render occluded areas of the HMD-wearing user's head and face in images displayed at another display (such as another caller's HMD) […] the user's actual head and face (or parts of their head and face) could be modelled. In other examples, a model of another head and face could be used, either for the occluded portions or in place of the user's head and face altogether […] The camera (or another camera) is used prior to the HMD being put on by the user to take pictures of the user's head from a number of angles).
Regarding claim 4, Raghoebardajal in view of Virskus and Buddemeier, discloses the method of claim 1, and discloses the method further comprising:
receiving, by the first device, pose data indicative of a current hand and body pose represented in sensor data associated with the second device (Raghoebardajal- Raghoebardajal- ¶0061, at least discloses if motion or other sensors are provided at the HMD 20, signals from those sensors may be at least partially processed at the HMD 20 and/or may be at least partially processed at the games console 2000; ¶0064, at least discloses where an interaction between two devices is basically symmetrical, for example where a camera or sensor on one device detects a signal or feature of the other device, it will be understood that unless the context prohibits this, the two devices could be interchanged without any loss of functionality; ¶0065, at least discloses There is a desire to implement the use of HMD devices in this context, as an HMD device could be used to provide an even more immersive experience, making a user feel as if they are present in the room with the other person or persons in the call; ¶0067, at least discloses A solution to this problem is to use image processing means to process images for display, for example to the other party or parties in a communications situation, and in doing so either to render the HMD device as a partly see-through object, or to remove it from the image recorded by the camera altogether, rendering the previously-occluded area of the face (or another feature) in its place; Virskus- ¶0109, at least discloses At block 554, process 550 can receive from the sending system compressed color images, depth images, masks, and a body model (e.g., from a sending device executing process 500, sent at block 514) and compressed audio data; ¶0111, at least discloses Gaps resulting from occlusions (such as by the user's hands or arms) are filled in with the torso disocclusion process discussed below in relation to block 562. Other gaps can be filled with a generic model of a body. For example, the back of the sending user can be filled in with a generic model of a body (which may be selected from a set of generic models bodies based on characteristics of the sending user such as height, weight, gender, ethnicity, etc.) and which may be customized based on the sending user's current status (e.g., applying similar clothing, hair, accessories, etc.) Based on the body model, areas on the 3D mesh (e.g., eyes, mouth, shoulders, hands, etc.) can be labeled for texture applications at block 560; ¶0114, at least discloses);
inputting, by the first device, the pose data to the user-specific generative model, wherein the user-specific generative model is further trained on additional enrollment image data indicative of baseline hand and body image content associated with the user (Virskus- ¶0109, at least discloses At block 554, process 550 can receive from the sending system compressed color images, depth images, masks, and a body model (e.g., from a sending device executing process 500, sent at block 514) and compressed audio data; ¶0111, at least discloses Gaps resulting from occlusions (such as by the user's hands or arms) are filled in with the torso disocclusion process discussed below in relation to block 562. Other gaps can be filled with a generic model of a body. For example, the back of the sending user can be filled in with a generic model of a body (which may be selected from a set of generic models bodies based on characteristics of the sending user such as height, weight, gender, ethnicity, etc.) and which may be customized based on the sending user's current status (e.g., applying similar clothing, hair, accessories, etc.) Based on the body model, areas on the 3D mesh (e.g., eyes, mouth, shoulders, hands, etc.) can be labeled for texture applications at block 560);
receiving, by the first device from the user-specific generative model, hand and body image information usable to render hand and body image content corresponding to the current hand and body pose represented by the pose data in the two-dimensional video frame (Virskus- ¶0109, at least discloses ¶0109, at least discloses At block 554, process 550 can receive from the sending system compressed color images, depth images, masks, and a body model (e.g., from a sending device executing process 500, sent at block 514) and compressed audio data; ¶0111, at least discloses Gaps resulting from occlusions (such as by the user's hands or arms) are filled in with the torso disocclusion process discussed below in relation to block 562. Other gaps can be filled with a generic model of a body. For example, the back of the sending user can be filled in with a generic model of a body (which may be selected from a set of generic models bodies based on characteristics of the sending user such as height, weight, gender, ethnicity, etc.) and which may be customized based on the sending user's current status (e.g., applying similar clothing, hair, accessories, etc.) Based on the body model, areas on the 3D mesh (e.g., eyes, mouth, shoulders, hands, etc.) can be labeled for texture applications at block 560); and
rendering, by the first device, the hand and body image content in the two-dimensional video frame (Virskus- ¶0114, at least discloses Due to the placement of the user's hands, a portion of the user's torso may not be visible to the depth sensor. Thus, when the 3D mesh is created, there may be a hole in the representation of the sending user behind the user's hands. This can be filled in with that portion of the existing model of the sending user, created from a snapshot in the where the user's hands did not occlude the user's torso).
It would have been obvious to one of ordinary in the art before the effective filing date of the claimed invention to have modified Raghoebardajal/Buddemeier to incorporate the teachings of Virskus, and apply the body model into the Raghoebardajal/Buddemeier’s teachings for receiving pose data indicative of a current hand and body pose of the user of the communication system; inputting the pose data to the generative model, wherein the generative model is further trained on an additional enrollment image indicative of a baseline image of the user's hand and body; receiving, from the generative model, hand and body image information usable to render a two-dimensional image of the current hand and body pose; and sending the hand and body image information to the first device for rendering of the two-dimensional image of the current hand and body pose.
The same motivation that was utilized in the rejection of claim 1 applies equally to this claim.
Regarding claim 5, Raghoebardajal in view of Virskus and Buddemeier, discloses the method of claim 1, and further discloses wherein the enrollment image comprises an image of the user not wearing a visor (Raghoebardajal- ¶0069, at least disclose Image processing means (such as comparison to a reference image) can then be used to identify the region of the user's face that is obscured by the HMD, as although it is mounted in a position that does not change, the orientation of the HMD with respect to the camera will cause different areas to be occluded as illustrated in FIGS. 11a and 11 b; ¶0074-0075, at least disclose A model of the HMD-wearing user's head and face (or at least a part of the head and face) is generated and is used to render occluded areas of the HMD-wearing user's head and face in images displayed at another display (such as another caller's HMD) […] In examples, the rendering process is such that the HMD is completely removed from the images).
Regarding claim 6, Raghoebardajal in view of Virskus and Buddemeier, discloses the method of claim 3, and further discloses wherein the visor image content is excluded by implementing a mask comprising pixels indicating which portions to exclude (Raghoebardajal- ¶0069, at least discloses Image processing means (such as comparison to a reference image) can then be used to identify the region of the user's face that is obscured by the HMD, as although it is mounted in a position that does not change, the orientation of the HMD with respect to the camera will cause different areas to be occluded as illustrated in FIGS. 11a and 11 b; ¶0074-0075, at least disclose A model of the HMD-wearing user's head and face (or at least a part of the head and face) is generated and is used to render occluded areas of the HMD-wearing user's head and face in images displayed at another display (such as another caller's HMD) […] the user's actual head and face (or parts of their head and face) could be modelled. In other examples, a model of another head and face could be used, either for the occluded portions or in place of the user's head and face altogether. In examples, the rendering process is such that the HMD is completely removed from the images. In other embodiments, the parts of the image relating to the HMD are mixed or combined with the head/face model so as to render the HMD as though it was transparent […] The camera (or another camera) is used prior to the HMD being put on by the user to take pictures of the user's head from a number of angles; Virskus- ¶0049, at least discloses densify the depth images to have a depth value for each pixel while generating parts masks and a body model; use the masks to segment the images into parts needed for hologram generation; ¶0087, at least discloses Densification and segmentation module 436 can receive depth images from image creator 434 (and in some cases also color images) and can perform densification, segmentation, and body modeling. The densification can include defining a depth measurement for each pixel of a depth image. The segmentation can identify portions of an image, such as the foreground (the portion showing a user), portions of a body such as a torso, arms, hands, and head, and a worn XR device. The body modeling can include identifying particular points, in 3D space, matching particular body parts and links between some of these points. In some implementations, densification and segmentation module 436 can perform the densification, segmentation, and body modeling simultaneously be applying a machine learning model trained to perform all three functions. Such a machine learning model can be trained on synthetic images of people in various environments, where the creation of such synthetic images can include labeling each pixel of the image with depth data from a virtual camera at which the perspective of the image is created, labeling portions as foreground/background, body parts, and XR device, and with assigned points and point connections for a body model).
Regarding claim 7, Raghoebardajal in view of Virskus and Buddemeier, discloses the method of claim 6, and further discloses wherein an inverse of the mask is applied to image data before masked rendered-output data is added to the image data (Virskus- ¶0087, at least discloses The densification can include defining a depth measurement for each pixel of a depth image. The segmentation can identify portions of an image, such as the foreground (the portion showing a user), portions of a body such as a torso, arms, hands, and head, and a worn XR device […] densification and segmentation module 436 can perform the densification, segmentation, and body modeling simultaneously be applying a machine learning model trained to perform all three functions; ¶0103, at least discloses At block 510, process 500 can apply one or more of the masks generated at block 508 to the color and densified depth images. For example, the background of the user in the captured images is not needed to generate the holographic representation of the sending user).
It would have been obvious to one of ordinary in the art before the effective filing date of the claimed invention to have modified Raghoebardajal/Buddemeier to incorporate the teachings of Virskus, and apply the segmentation can identify portions of an image, such as the foreground into the Raghoebardajal’s teachings in order an inverse of the mask is applied to image data before masked rendered-output data is added to the image data.
The same motivation that was utilized in the rejection of claim 1 applies equally to this claim.
Regarding claim 8, Raghoebardajal in view of Virskus and Buddemeier, discloses the method of claim 1, and discloses the method further comprising using a trained rendering module to generate a composited output (Virskus- ¶0056, at least discloses the model can be trained using training items that are generated by taking, as ground truth, either an image of a user without an XR device or that image augmented with an XR device with the lens portion transparent, and pairing that ground truth image with a generated image showing the XR device added to the user image over the user's eyes, as it would be captured during a call […] The encoding of the image can be provided to both the geometry projection branch and the texture projection branch. The geometry projection branch can predict a geometry (3D mesh) of the user's face excluding the XR device and the texture projection branch can predict a texture of the user's face. In some implementations, the holographic calling system can compare a geometry generated for the ground truth in the training item with the predicted geometry and can apply a first set of loss functions to update parameters of the geometry projection branch, training it. Similarly in these implementations, the holographic calling system can compare a texture generated for the ground truth in the training item with the predicted texture and can apply a second set of loss functions to update parameters of the texture projection branch, training it also).
It would have been obvious to one of ordinary in the art before the effective filing date of the claimed invention to have modified Raghoebardajal/Buddemeier to incorporate the teachings of Virskus, and apply the model can be trained using training items into the Raghoebardajal/Buddemeier’s teachings for using a trained rendering module to generate a composited output.
The same motivation that was utilized in the rejection of claim 1 applies equally to this claim.
Regarding claim 9, Raghoebardajal in view of Virskus and Buddemeier, discloses the method of claim 3, and discloses the method further comprising running a segmentation model to generate a mask corresponding to visor image content in each frame (Raghoebardajal- ¶0074, at least discloses a model of the HMD-wearing user's head and face (or at least a part of the head and face) is generated and is used to render occluded areas of the HMD-wearing user's head and face in images displayed at another display (such as another caller's HMD); Virskus- ¶0049, at least discloses densify the depth images to have a depth value for each pixel while generating parts masks and a body model; use the masks to segment the images into parts needed for hologram generation; ¶0087, at least discloses Densification and segmentation module 436 can receive depth images from image creator 434 (and in some cases also color images) and can perform densification, segmentation, and body modeling. The densification can include defining a depth measurement for each pixel of a depth image. The segmentation can identify portions of an image, such as the foreground (the portion showing a user), portions of a body such as a torso, arms, hands, and head, and a worn XR device. The body modeling can include identifying particular points, in 3D space, matching particular body parts and links between some of these points. In some implementations, densification and segmentation module 436 can perform the densification, segmentation, and body modeling simultaneously be applying a machine learning model trained to perform all three functions. Such a machine learning model can be trained on synthetic images of people in various environments, where the creation of such synthetic images can include labeling each pixel of the image with depth data from a virtual camera at which the perspective of the image is created, labeling portions as foreground/background, body parts, and XR device, and with assigned points and point connections for a body model).
Regarding claim 10, Raghoebardajal in view of Virskus and Buddemeier, discloses the method of claim 1, and discloses the method further comprising adding audio data to the expression vector, wherein the rendering of the two-dimensional video frame includes generating facial expressions based on the audio data (Virskus- ¶0048, at least discloses the holographic calling system can provide a holographic calls between any combination of artificial reality (XR) devices, mobile devices (e.g., smartphones), traditional computing systems (e.g., a laptop or desktop with a webcam), etc. The experience of each user may depend on the capture and processing capabilities of the sending system and the display and processing capabilities of the receiving system. For example, where one user is using a mobile phone and another is using an XR system, the user of the XR system may see a hologram of the other user in their space while the mobile phone user may see a representation of the other user on their phone screen; ¶0094, at least discloses rendering the 3D mesh on a display or projecting the 3D mesh into the eyes of the user […] this projection can be accomplished by flattening the 3D mesh into multiple 2D planes (one for each eye with different perspectives to preserve the 3D appearance to the user) and painting these 2D planes into an image, e.g., using MEMS mirrors and microLEDs, lasers, or other color projection systems, this light can then pass through a series of waveguides and filters to project them into a user's eyes; ¶00118, at least discloses At block 566, process 550 can synchronize the decompressed audio data, from block 556 with the 3D mesh by matching timestamps added to the audio based on when it was captured with timestamps on the color and depth images used to create the 3D mesh; ¶00184, at least discloses the color and depth data can be obtained from one or more previous captured frames during the holographic call (e.g., frames where process 1000 determines that the sending user's torso is not occluded). In some cases, process 1000 can accumulate portions of multiple such frames, knitting them together to fill in portions of the sending user that are occluded in the frames; ¶00192, at least discloses the training items are in the form of a sequence from a video, in which case model output from one or more previous frames can be stored and then also used as additional input for the next frame; Buddemeier- ¶0006, at least discloses the animation vector may be determined by sensing facial features, associated with independent targets, of a human subject. Alternatively, the animation vector is obtained by applying facial feature position sensing to annotated vertices on a model; ¶0012, at least discloses the animation vector is an abstracted sensing result and is a most compact representation of the facial expression as determined by audio-visual sensing)
It would have been obvious to one of ordinary in the art before the effective filing date of the claimed invention to have modified Raghoebardajal to incorporate the teachings of Virskus and Buddemeier, and apply synchronizing the decompressed audio data, video frame and expression vector into the Raghoebardajal’s teachings for adding audio data to the expression vector, wherein the rendering of the two-dimensional video frame includes generating facial expressions based on the audio data.
The same motivation that was utilized in the rejection of claim 1 applies equally to this claim.
Regarding claim 11, Raghoebardajal in view of Virskus and Buddemeier discloses a computing system for generating and rendering an image representative of a user of a communications system on a two-dimensional display device (Raghoebardajal- Fig. 1 shows an example of a head-mountable display system; ¶0037, at least discloses the display is not a head-mountable display […] the display (whether head mountable or not) may be referred to as an immersive display, in that in normal use it fills at least a threshold angular range (for example, at least 40°) of the field of view of the user; Fig. 1 and ¶0038-0042, at least disclose an example of a head-mountable display system comprising a frame to be mounted onto an observer's head, the frame defining one or two eye display positions which, in use, are positioned in front of a respective eye of the observer and a display element mounted with respect to each of the eye display positions, the display element providing a virtual image of a video display of a video signal from a video signal source to that eye of the observer; ¶0065, at least discloses There is a desire to implement the use of HMD devices in this context, as an HMD device could be used to provide an even more immersive experience, making a user feel as if they are present in the room with the other person or persons in the call; ¶0067, at least discloses A solution to this problem is to use image processing means to process images for display, for example to the other party or parties in a communications situation, and in doing so either to render the HMD device as a partly see-through object, or to remove it from the image recorded by the camera altogether, rendering the previously-occluded area of the face (or another feature) in its place), the computing system comprising:
one or more processors (Raghoebardajal- Fig. 22 and ¶0105, at least disclose an image processing apparatus 820 configured to replace, at least partially, the portion of a captured image corresponding to the first head mountable display device 800 with a corresponding portion of a 3D facial model); and
a computer-readable storage medium having computer-executable instructions stored thereupon which, when executed by the processor, cause the computing system to perform operations (Raghoebardajal- ¶0108, at least disclose at least in part, by software-controlled data processing apparatus, it will be appreciated that a non-transitory machine-readable medium carrying such software, such as an optical disk, a magnetic disk, semiconductor memory or the like; ¶0116, at least discloses “A machine-readable non-transitory storage medium which stores computer software”; Claim 15 cites “A machine-readable non-transitory storage medium which stores computer readable instructions thereon that, when executed by a computer, cause the computer to perform an image processing method”; Fig. 21 and ¶0104, at least disclose a processor (renderer) 750 operable to identify a region of the user's face that is occluded by the head mountable display device, the processor being operable to replace, at least partially, the portion of the captured image corresponding to the head mountable display device with a corresponding portion of a 3D facial model (for example, stored in a memory 740), to provide a modified image.) comprising:
receiving, from a second device, an expression vector that encodes a current facial expression represented in sensor data associated with the second device (see Claim 1 rejection for detailed analysis), the expression vector comprising a compact vector representation of facial-expression information (see Claim 1 rejection for detailed analysis), the second device having a three-dimensional display (see Claim 1 rejection for detailed analysis);
inputting the expression vector to a user-specific generative model trained on enrollment image data comprising an enrollment image indicative of baseline facial image content associated with the user (see Claim 1 rejection for detailed analysis);
receiving, from the user-specific generative model, facial image information comprising two-dimensional pixel information usable to render a two-dimensional video frame on the two-dimensional display device (see Claim 1 rejection for detailed analysis), the two-dimensional video frame including facial image content corresponding to the current facial expression represented by the expression vector and excluding visor image content (see Claim 1 rejection for detailed analysis); and
rendering the two-dimensional video frame in context of an on-going session of the communications system (see Claim 1 rejection for detailed analysis).
Regarding claims 12-19, all claim limitations are set forth as claims 2-9 in computing system form and rejected as per discussion for claims 2-9.
Regarding claim 20, Raghoebardajal in view of Virskus and Buddemeier, discloses a computer-readable storage medium having computer-executable instructions stored thereupon which, when executed by a processor of a computing system, cause the computing system to perform operations (Raghoebardajal- ¶0108, at least disclose at least in part, by software-controlled data processing apparatus, it will be appreciated that a non-transitory machine-readable medium carrying such software, such as an optical disk, a magnetic disk, semiconductor memory or the like; ¶0116, at least discloses “A machine-readable non-transitory storage medium which stores computer software”; Claim 15 cites “A machine-readable non-transitory storage medium which stores computer readable instructions thereon that, when executed by a computer, cause the computer to perform an image processing method”; Fig. 21 and ¶0104, at least disclose a processor (renderer) 750 operable to identify a region of the user's face that is occluded by the head mountable display device, the processor being operable to replace, at least partially, the portion of the captured image corresponding to the head mountable display device with a corresponding portion of a 3D facial model (for example, stored in a memory 740), to provide a modified image.) comprising the tasks of computing system of claim 11.
Conclusion
6. Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action. 7. Any inquiry concerning this communication or earlier communications from the examiner should be directed to MICHAEL LE whose telephone number is (571)272-5330. The examiner can normally be reached 9am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kent Chang can be reached at (571) 272-7667. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MICHAEL LE/Primary Examiner, Art Unit 2614
