DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP §§ 706.02(l)(1) - 706.02(l)(3) for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/process/file/efs/guidance/eTD-info-I.jsp.
Claim 1 is rejected on the ground of nonstatutory double patenting as being unpatentable over claim 1 of Krol et al. US Patent No. 11,776,227. Although the claims at issue are not identical, they are not patentably distinct from each other because all of the claim limitations set forth in the Claim 1 of the instant application are only a subset of the claim limitations set forth in the patented claim 1. Therefore, it would have been obvious to one of ordinary skill in the art to make the claim made in this application, because it is only a subset of what has been claimed before.
Claim 8 is rejected on the ground of nonstatutory double patenting as being unpatentable over claim 8 of the US Patent No. 11,776,227. Although the claims at issue are not identical, they are not patentably distinct from each other because all of the claim limitations set forth in the Claim 8 of the instant application are only a subset of the claim limitations set forth in the patented claim 8. Therefore, it would have been obvious to one of ordinary skill in the art to make the claim made in this application, because it is only a subset of what has been claimed before.
Claim 15 is rejected on the ground of nonstatutory double patenting as being unpatentable over claim 15 of the US Patent No. 11,776,227. Although the claims at issue are not identical, they are not patentably distinct from each other because all of the claim limitations set forth in the Claim 15 of the instant application are only a subset of the claim limitations set forth in the patented claim 15. Therefore, it would have been obvious to one of ordinary skill in the art to make the claim made in this application, because it is only a subset of what has been claimed before.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over Mittal US Patent No. 11,450,104 (hereinafter Mittal) in view of Han et al. US-PGPUB No. 2022/0353438 (hereinafter Han);
Rabinovich et al. US-PGPUB No. 2021/0392296 (hereinafter Rabinovich);
Oz et al. US-PGPUB No. 2022/0286657 (hereinafter Oz ‘657);
Oz et al. US-PGPUB No. 2022/0191431 (hereinafter Oz ‘431).
Re Claim 1:
Mittal teaches a computer implemented method comprising: (
Mittal teaches at column 6, lines 45-52 that the user may select three-dimensional image data of image data for classification as objectionable and/or non-objectionable), comprising:
Altering, by one or more processors, portions of a video stream of a first user, the portions determined to represent physical surroundings of the first user, to generate an altered background (Mittal teaches at column 6, lines 45-63 that the user may select three-dimensional image data of image data for classification as objectionable and/or non-objectionable…the labeled objectionable portions 202 and non-objectionable portions 204 may be indicator data 206…Indicator data 206 may be used by whichever computing device performs the segmentation task.
Mittal teaches at column 2, lines 12-20 that often users engaging in video conferencing may face the camera and are depicted in front of a relatively static background and at column 5, lines 24-60 that computing device 102 may capture image data 170….the person is sitting in front of a background that includes some objectionable content 180…recipient device 130 may generate the segmentation maps and/or perform the blurring operations.
Mittal at FG. 2 and column 6, lines 9-25 that the initial segmentation map 140 may denote pixels depicting a user as foreground pixels and pixels denoting the user’s background as background pixels. Mittal teaches at column 7, lines 10-50 that a segmentation map 302 has been generated using CNN 306 trained to perform the segmentation task…the first segmentation map 302 may generated using the indicator data 206 and the frame of image data labeled with the indicator data 206….after generating an initial segmentation map 302, the video chat session may be initiated. The initial segmentation map 302 may be used to obfuscate the objectionable content on a frame-by-frame basis as the video chat is conducted.
Mittal teaches at column 5, lines 54-60 that background-end computing devices 120 may generate the segmentation map 140 and/or may perform the obfuscation/removal operation….the recipient device 130 may generate the segmentation maps and/or perform the blurring obfuscation operations).
Mittal at least suggests the claim limitation:
Sending, by the one or more processors, to a second user, an avatar of the first user, the video stream with the altered background, and settings specifying how to present the altered background of the avatar of the first user to the second user in a virtual environment;
Based on the settings, causing, by the one or more processors, a device associated with the second user to render the avatar of the first user and the altered background of the physical surroundings of the first user from a perspective of a virtual camera corresponding to the second user (Mittal teaches at column 6, lines 45-67 and column 7, lines 1-40 that the user may select three-dimensional image data of image data for classification as objectionable and/or non-objectionable…the labeled objectionable portions 202 and non-objectionable portions 204 may be indicator data 206…Indicator data 206 may be used by whichever computing device performs the segmentation task…indicator data 206 is sent to back-end computing devices 120 so that the back-end computing devices 120 can use the indicator data 206 to perform the segmentation task…although back-end computing devices 120 are depicted as deploying CNN 306… and sending segmentation maps 302, these operations can instead be performed by sender computing device 102…., recipient devices 130).
Oz ‘657 teaches at Paragraph 0293 that the objects (pen and paper) held by the user are not segmented together with the person but rather as part of the background that has to be replaced by the virtual background and at Paragraph 0295 objects such as earphones, cables connected to the earphones, may be removed in the modelling and rendering processes…The option to show or eliminate such objects may be selected by users and at Paragraph 0298 that the system will learn the preferences of the user and use the feedback regarding which objects, textures or pixels may be part of the foreground and which may be part of the background in order to improve the segmentation process. Accordingly, the user can specify whether to segment pen and paper as part of the foreground together with the avatar and therefore the pen and paper or cable with the earphone can be rendered as the avatar’s surroundings/backgrounds with the avatar.
Oz ‘431 teaches at Paragraph 0191 that that these objects are not segmented together with the person but rather as part of the background that has to be replaced by the virtual background and at Paragraph 0193 the option to show or eliminate such objects may be selected by users and at Paragraph 0196 that the system will learn the preferences of the user and use the feedback regarding which objects, textures may be part of the foreground and which may be part of the background…machine learning method may learn what objects may be typically chosen by users as parts of the foreground and what objects may be typically chosen by users as part of the background.
It is understood that the definition of the foreground/background for the segmentation process is different from the definition of foreground/background in applicant’s claim invention. For example, when the pen and paper are chosen by the user(s) in the segmentation process as foreground together with the user, the pen and paper will show together with the avatar in the 3D video conferencing environment. However, the pen and paper constitute backgrounds of the avatar in the 3D video conferencing environment.
Oz ‘431 also teaches at Paragraph 0329 that the creation of avatars within virtual environments required a good blending of the avatar image with the virtual background and at Paragraph 0340 that each image may include the person and an arbitrary background. The arbitrary background may include any compositions of objects and/or colors and/or shapes and at Paragraph 0341 that converting each input image of the person by portrait matting predictor to (a) a first colored background image that includes the person and a background of a first color and (b) a second colored background image that includes the person and a background of a second color.
Oz ‘657/Oz ‘431’s teaching when applied Han/Rabinovich explicitly teaches the claim limitation:
Altering, by one or more processors, portions of a video stream of a first user, the portions determined to represent physical surroundings of the first user, to generate an altered background (
The meta-data (configuration setting) includes the avatar’s surroundings (backgrounds) including hair appearance, earring, piercing and/or clothes, wherein Rabinovich clearly shows at Paragraph 0141 that each participant receives other participants’ avatars’ backgrounds.
Rabinovich teaches that the second participant’s avatar represents the second participant at a location of a second virtual camera (second viewpoint of the HMD’s camera), and the second participant’s avatar and second virtual camera (second viewpoint) are controlled by the second participant due to second participant change of viewpoint or change of the head/gaze direction.
Rabinovich teaches at FIG. 5 and Paragraph 0147-0149 that tracking the user’s eyes and gaze direction may also be used to determine the direction in which the user is looking…This information can be used to rotate the avatar’s head and eyes so that in the virtual space it also appears as if the user is looking at the same person or object as in the real world….if the user looks at the right side of the screen, the point of view of the virtual camera may move to the right, so that the person at which the user is looking is located at the center of the user’s screen and at Paragraph 0156 that a new view is created based on a real-time image obtained from a video camera and the position of the new point view of the virtual camera.
Rabinovich teaches at Paragraph 0356 that this real-time 3D textured model can then be used to render the view of the user from various angles and camera positions and specifically may be used to correct the viewing position of the virtual camera as if it were virtually located inside the screen of the user.
Rabinovich teaches at Paragraph 0489 that a viewpoint dependent texture map may be produced as a function of the required pose and expression and as function of the viewpoint and at Paragraph 0069 receiving direction of gaze regarding a direction of gaze of each participant with a representation of a virtual 3D video conference environment and at Paragraph 0247 the up-sampled model and texture map are then used to render an image of the textured model from a viewpoint that is similar to the camera and at Paragraph 0297-0298 if the viewer uses a 3D display such as a virtual headset, then two images corresponding to the viewpoint of the right and left eyes may be generated to create a stereoscopic image.
Rabinovich teaches at Paragraph 0425-0427 that Step 1654 may include receiving, by a user device of a first participant and during the 3D video conference call, second participant circumstances metadata indicative of one or more current circumstances regarding a second participant (first user) and Step 1656 may include updating by the user device of the first participant (second user) a 3D participant representation of the second participant (first user) within a first representation of virtual 3D video conference environment. The different circumstances may include at least one out of different image acquisition conditions, different directions of gaze, different viewpoints of a viewer, different expressions. Rabinovich teaches at Paragraph 0005 that each participant is typically shown with the background of their own office or with a virtual background of their selection and at Paragraph 0141 that each participant may choose or be otherwise associated with a virtual or actual background and/or may select or otherwise receive any virtual or actual background in which avatars are related to at least some of the participants are displayed);
Sending, by the one or more processors, to a second user, an avatar of the first user, the video stream with the altered background, and settings specifying how to present the altered background of the avatar of the first user to the second user in a virtual environment;
Based on the settings, causing, by the one or more processors, a device associated with the second user to render the avatar of the first user and the altered background of the physical surroundings of the first user from a perspective of a virtual camera corresponding to the second user (
The meta-data (configuration setting) includes the avatar’s surroundings (backgrounds) including hair appearance, earring, piercing and/or clothes, wherein Rabinovich clearly shows at Paragraph 0141 that each participant receives other participants’ avatars’ backgrounds.
Rabinovich teaches that the second participant is able to specify how the altered background is used to render the second participant’s avatar when presented to the first participant, wherein the first virtual camera (first viewpoint) of the first participant corresponds to a location of the first avatar, and wherein the first participant’s virtual camera (viewpoint) and first avatar are controlled by the first participant.
Rabinovich teaches that the second participant’s avatar represents the second participant at a location of a second virtual camera (second viewpoint of the HMD’s camera), and the second participant’s avatar and second virtual camera (second viewpoint) are controlled by the second participant due to second participant change of viewpoint or change of the head/gaze direction.
Rabinovich teaches at FIG. 5 and Paragraph 0147-0149 that tracking the user’s eyes and gaze direction may also be used to determine the direction in which the user is looking…This information can be used to rotate the avatar’s head and eyes so that in the virtual space it also appears as if the user is looking at the same person or object as in the real world….if the user looks at the right side of the screen, the point of view of the virtual camera may move to the right, so that the person at which the user is looking is located at the center of the user’s screen and at Paragraph 0156 that a new view is created based on a real-time image obtained from a video camera and the position of the new point view of the virtual camera.
Rabinovich teaches at Paragraph 0356 that this real-time 3D textured model can then be used to render the view of the user from various angles and camera positions and specifically may be used to correct the viewing position of the virtual camera as if it were virtually located inside the screen of the user.
Rabinovich teaches at Paragraph 0425-0427 that Step 1654 may include receiving, by a user device of a first participant and during the 3D video conference call, second participant circumstances metadata indicative of one or more current circumstances regarding a second participant (first user) and Step 1656 may include updating by the user device of the first participant (second user) a 3D participant representation of the second participant (first user) within a first representation of virtual 3D video conference environment. The different circumstances may include at least one out of different image acquisition conditions, different directions of gaze, different viewpoints of a viewer, different expressions. Rabinovich teaches at Paragraph 0005 that each participant is typically shown with the background of their own office or with a virtual background of their selection and at Paragraph 0141 that each participant may choose or be otherwise associated with a virtual or actual background and/or may select or otherwise receive any virtual or actual background in which avatars are related to at least some of the participants are displayed and at Paragraph 0818 that receiving, by the user device of the first participant (second user) and during the 3D video conference call, second participant (first user) constraints metadata indicative of one or more current constraints regarding a second participant. The second constraint metadata may refer to any aspect of the appearance of the avatar and its surroundings---for example, makeup, hair appearance, and/or inclusion or removal of accessories, e.g., glasses, sunglasses, earring, neckless, bracelet, piercing and/or clothes, inclusion or removal of tattoo and/or a scar and at Paragraph 0520 receiving second participant meta-data and first viewpoint metadata by a first unit that may be associated with a first participant, wherein the second participant metadata may be indicative a pose of a second participant and an expression of the second participant, wherein the first viewpoint metadata ma be indicative of a virtual position from which the first participant requests to view an avatar of the second participant.
Rabinovich teaches at Paragraph 0141 that each participant may be associated with a virtual or actual background and/or may receive any virtual or actual background in which avatars related to at least one of participants are displayed and at Paragraph 0310 that some parts of the video image may not be modelled as 3D objects and may be transmitted to the other users as pixel values…Examples include flat objects placed in the background such as white board or a picture on a wall and at Paragraph 0312 that in these cases the system may decide to transmit to viewers a video stream that would include at least some of the non-modelled parts and then the 3D models that exist would be rendered on top of the transmitted video in their respective locations.
Rabinovich further teaches at Paragraph 0520 receiving second participant meta-data and first viewpoint metadata by a first unit that may be associated with a first participant wherein the first viewpoint meta-data may be indicative of a virtual position from which the first participant request to view an avatar of the second participant. It is understood that the second participant representation information includes the configuration settings specifying how a foreground of the second participant’s avatar to be presented to a first participant in the 3D environment which is equivalent to specifying a perimeter of a background of a second participant’s avatar to be presented to a first participant in the 3D virtual environment because the background representation of the virtual 3D video conference environment is generated based on the 3D entity representation information of each of the least one relevant person/participant (Paragraph 0649).
Rabinovich teaches at Paragraph 0296 that a 3D environment is rendered from a virtual camera according to the location of the viewer’s eyes and at FIG. 17 Step 74 and Paragraph 0403-406 obtaining a virtual 3D environment and obtaining information regarding avatars related to participants….receiving information regarding spatial relationships between the locations of the 3D models of the participants and the directions of gaze of the participants and updating at least the orientation of the 3D models related to the participants within the virtual 3D environment and at Paragraph 0425 receiving by a user device of a first participant and during the 3D video conference call, second participant circumstances meta-data indicative of one or more current circumstances regarding a second participant and at Paragraph 0449 that the method may include determining a size of the avatar wherein participants are displayed from different angles and in different sizes and at Paragraph 0311 all the areas that may be not modelled accurately enough may be transmitted separately.
Rabinovich teaches at Paragraph 0794 that the person itself forms the foreground 3491 and the ventilator 3494 is a part of the background 3492 and at Paragraph 0818 receiving, by the user device of the first participant, a second participant constraints meta-data indicative of one or more current constraints regarding a second participant. The second constraint meta-data may refer to any aspect of the appearance of the avatar and its surroundings.
Han teaches at Paragraph 0015-0017 that the participants may receive an image file of the immersive background….each participant receives the dynamic immersive background….each participant also receives metadata that describes how video streams are to be displayed in the dynamic immersive background. For example, a participant may receive an XML file that includes properties indicating the locations within the dynamic immersive background at which video streams may be displayed….the example system removes the background from the video stream…The area of the video stream outside the outline is then made removable…so that the area of the video stream outside the area does not obscure the dynamic immersive background.
Han discloses at Paragraph 0041-0041 that a dynamic immersive background 242 is distributed to each video conference participant….the participants also receive data indicating how the various video streams from the different users are to be displayed in the immersive scene.
Han teaches at Paragraph 0042-0043 that the hatched background displayed with User A on client device 210 is replaced by a removable background that allows the recipient’s client device to display the immersive scene in place of the background….a client device 140 receives a plurality of video streams…each of the plurality of video streams is recorded by a corresponding client device and has had its background identified made removable.
Han shows at Paragraph 0048-0050 that the XML for providing instruction includes an instruction for display a solid colored background “bkcolor” and at Paragraph 0044 that the user may select their own background as a dynamic immersive background….the client software may create an image file such as a JPG or TIFF file that contains the background.
Han teaches at Paragraph 0046-0048 that the client software may transmit the dynamic immersive background to the video conference provider 110, which then forwards the dynamic immersive background to each participant of the video conference and at Paragraph 0054 that the client device 140 receives a collection of meta-data associated with an immersive scene.
Han teaches at Paragraph 0049-0052 that the client device receives the above XML file, parses the XML file in order to identify properties associated with each of the various locations within the immersive scene with respect to the dynamic immersive background…The example XML also includes information identifying an order in which to assign video feeds to the different locations. Thus, when video feeds from different client devices are received, the client’s device can assign them to a corresponding seat id….the client device 140 displays immersive scene, including the immersive dynamic background and one or more of the plurality of video streams associated with the video conference….the client device 140 changes perspective with respect to the immersive scene…such as change in perspective might cause a video stream that is displayed at a first size to then be displayed at a second size…it may also zoom in on the background image).
It would have been obvious to one of the ordinary skill in the art before the filing date of the instant application to have arranged the sender participant according to the perspective of the recipient participant and according to the metadata relating to the background and properties or seat id of the sender participant to have rendered the sender participant according to the perspective of the recipient participant in a video conferencing. One of the ordinary skill in the art would have been motivated to have rendered the video conferencing according to the meta-data defining the background and the metadata defining the video streams.
Re Claim 2:
The claim 2 encompasses the same scope of invention as that of the claim 1 except additional claim limitation that the settings are selectable by the first user, the settings comprising a coloring property of an avatar background color, whether the avatar has a volumetric appearance, a framerate for applying background removal, a shape of the avatar, and a smoothness to edges of the video stream of the first user.
Oz ‘431 when applied to Han/Rabinovich explicitly teaches the claim limitation that the settings are selectable by the first user, the settings comprising a coloring property of an avatar background color, whether the avatar has a volumetric appearance, a framerate for applying background removal, a shape of the avatar, and a smoothness to edges of the video stream of the first user (
Oz ‘431 also teaches at Paragraph 0329 that the creation of avatars within virtual environments required a good blending of the avatar image with the virtual background and at Paragraph 0340 that each image may include the person and an arbitrary background. The arbitrary background may include any compositions of objects and/or colors and/or shapes and at Paragraph 0341 that converting each input image of the person by portrait matting predictor to (a) a first colored background image that includes the person and a background of a first color and (b) a second colored background image that includes the person and a background of a second color.
The meta-data (configuration setting) includes the avatar’s surroundings (backgrounds) including hair appearance, earring, piercing and/or clothes, wherein Rabinovich clearly shows at Paragraph 0141 that each participant receives other participants’ avatars’ backgrounds.
Rabinovich teaches removal of accessories and/or clothes and thereby cutting off earing, piercing, clothes and/or hairs.
Rabinovich teaches at Paragraph 0818 that receiving, by the user device of the first participant (second user) and during the 3D video conference call, second participant (first user) constraints metadata indicative of one or more current constraints regarding a second participant. The second constraint metadata may refer to any aspect of the appearance of the avatar and its surroundings---for example, makeup, hair appearance, and/or inclusion or removal of accessories, e.g., glasses, sunglasses, earring, neckless, bracelet, piercing and/or clothes, inclusion or removal of tattoo and/or a scar.
Rabinovich teaches at Paragraph 0141 that each participant may be associated with a virtual or actual background and/or may receive any virtual or actual background in which avatars related to at least one of participants are displayed and at Paragraph 0310 that some parts of the video image may not be modelled as 3D objects and may be transmitted to the other users as pixel values…Examples include flat objects placed in the background such as white board or a picture on a wall and at Paragraph 0312 that in these cases the system may decide to transmit to viewers a video stream that would include at least some of the non-modelled parts and then the 3D models that exist would be rendered on top of the transmitted video in their respective locations.
Han teaches at Paragraph 0015-0017 that the participants may receive an image file of the immersive background….each participant receives the dynamic immersive background….each participant also receives metadata that describes how video streams are to be displayed in the dynamic immersive background. For example, a participant may receive an XML file that includes properties indicating the locations within the dynamic immersive background at which video streams may be displayed….the example system removes the background from the video stream…The area of the video stream outside the outline is then made removable…so that the area of the video stream outside the area does not obscure the dynamic immersive background.
Han teaches at Paragraph 0042-0043 that the hatched background displayed with User A on client device 210 is replaced by a removable background that allows the recipient’s client device to display the immersive scene in place of the background….a client device 140 receives a plurality of video streams…each of the plurality of video streams is recorded by a corresponding client device and has had its background identified made removable).
Re Claim 3:
The claim 3 encompasses the same scope of invention as that of the claim 1 except additional claim limitation that altering the portions of the video stream further comprises: applying a neural network to the video stream to detect first pixels representing the first user and second pixels represent the physical surroundings of the first user; and altering the second pixels representing the physical surroundings of the first user with a solid colored background.
Mittal further teaches the claim limitation that altering the portions of the video stream further comprises: applying a neural network to the video stream to detect first pixels representing the first user and second pixels represent the physical surroundings of the first user; and altering the second pixels representing the physical surroundings of the first user with a solid colored background (Mittal at FG. 2 and column 6, lines 9-25 that the initial segmentation map 140 may denote pixels depicting a user as foreground pixels and pixels denoting the user’s background as background pixels. Mittal teaches at column 7, lines 10-50 that a segmentation map 302 has been generated using CNN 306 trained to perform the segmentation task…the first segmentation map 302 may generated using the indicator data 206 and the frame of image data labeled with the indicator data 206….after generating an initial segmentation map 302, the video chat session may be initiated. The initial segmentation map 302 may be used to obfuscate the objectionable content on a frame-by-frame basis as the video chat is conducted).
Re Claim 4:
The claim 4 encompasses the same scope of invention as that of the claim 3 except additional claim limitation that replacing the solid colored background using an HTML element; and translating the solid colored background into the video stream using an Application Programming Interface.
Han further teaches the claim limitation that replacing the solid colored background using an HTML element; and translating the solid colored background into the video stream using an Application Programming Interface (
Han shows at Paragraph 0048-0050 that the XML for providing instruction includes an instruction for display a solid colored background “bkcolor” and at Paragraph 0044 that the user may select their own background as a dynamic immersive background….the client software may create an image file such as a JPG or TIFF file that contains the background.
Han teaches at Paragraph 0046-0048 that the client software may transmit the dynamic immersive background to the video conference provider 110, which then forwards the dynamic immersive background to each participant of the video conference and at Paragraph 0054 that the client device 140 receives a collection of meta-data associated with an immersive scene.
Han teaches at Paragraph 0049-0052 that the client device receives the above XML file, parses the XML file in order to identify properties associated with each of the various locations within the immersive scene with respect to the dynamic immersive background…The example XML also includes information identifying an order in which to assign video feeds to the different locations. Thus, when video feeds from different client devices are received, the client’s device can assign them to a corresponding seat id….the client device 140 displays immersive scene, including the immersive dynamic background and one or more of the plurality of video streams associated with the video conference….the client device 140 changes perspective with respect to the immersive scene…such as change in perspective might cause a video stream that is displayed at a first size to then be displayed at a second size…it may also zoom in on the background image).
Re Claim 5:
The claim 5 encompasses the same scope of invention as that of the claim 1 except additional claim limitation that receiving, from the second user, an avatar of the second user, a video stream of the second user with a background, and second settings specifying how to present the avatar of the second user to the first user in a virtual environment of the video stream of the second user; and based on the second settings, rendering, from a perspective of a virtual camera corresponding to the first user, the avatar of the second user with the video stream of the second user and the background.
Han explicitly teaches the claim limitation that receiving, from the second user, an avatar of the second user, a video stream of the second user with a background, and second settings specifying how to present the avatar of the second user to the first user in a virtual environment of the video stream of the second user; and based on the second settings, rendering, from a perspective of a virtual camera corresponding to the first user, the avatar of the second user with the video stream of the second user and the background (Han teaches at Paragraph 0043-0044 removing the background can include flagging pixels as being outside of the outline as being removable…removing the background may involve making the background transparent, such as by applying or modifying an alpha value for each pixel in the background to set the pixel to be transparent. Han shows at Paragraph 0048-0050 that the XML for providing instruction includes an instruction for display a solid colored background “bkcolor” and at Paragraph 0044 that the user may select their own background as a dynamic immersive background….the client software may create an image file such as a JPG or TIFF file that contains the background.
Han teaches at Paragraph 0015-0017 that the participants may receive an image file of the immersive background….each participant receives the dynamic immersive background….each participant also receives metadata that describes how video streams are to be displayed in the dynamic immersive background. For example, a participant may receive an XML file that includes properties indicating the locations within the dynamic immersive background at which video streams may be displayed….the example system removes the background from the video stream…The area of the video stream outside the outline is then made removable…so that the area of the video stream outside the area does not obscure the dynamic immersive background.
Han discloses at Paragraph 0041-0041 that a dynamic immersive background 242 is distributed to each video conference participant….the participants also receive data indicating how the various video streams from the different users are to be displayed in the immersive scene.
Han teaches at Paragraph 0042-0043 that the hatched background displayed with User A on client device 210 is replaced by a removable background that allows the recipient’s client device to display the immersive scene in place of the background….a client device 140 receives a plurality of video streams…each of the plurality of video streams is recorded by a corresponding client device and has had its background identified made removable.
Han teaches at Paragraph 0046-0048 that the client software may transmit the dynamic immersive background to the video conference provider 110, which then forwards the dynamic immersive background to each participant of the video conference and at Paragraph 0054 that the client device 140 receives a collection of meta-data associated with an immersive scene.
Han teaches at Paragraph 0049-0052 that the client device receives the above XML file, parses the XML file in order to identify properties associated with each of the various locations within the immersive scene with respect to the dynamic immersive background…The example XML also includes information identifying an order in which to assign video feeds to the different locations. Thus, when video feeds from different client devices are received, the client’s device can assign them to a corresponding seat id….the client device 140 displays immersive scene, including the immersive dynamic background and one or more of the plurality of video streams associated with the video conference….the client device 140 changes perspective with respect to the immersive scene…such as change in perspective might cause a video stream that is displayed at a first size to then be displayed at a second size…it may also zoom in on the background image).
Re Claim 6:
The claim 6 encompasses the same scope of invention as that of the claim 5 except additional claim limitation that the second settings are selectable by the second user, comprising a coloring property of an avatar background color, a volumetric appearance of the avatar of the second user, and a shape of the avatar of the second user.
Oz ‘431 when applied to Han/Rabinovich explicitly teaches the claim limitation that the second settings are selectable by the second user, comprising a coloring property of an avatar background color, a volumetric appearance of the avatar of the second user, and a shape of the avatar of the second user (
Oz ‘431 also teaches at Paragraph 0329 that the creation of avatars within virtual environments required a good blending of the avatar image with the virtual background and at Paragraph 0340 that each image may include the person and an arbitrary background. The arbitrary background may include any compositions of objects and/or colors and/or shapes and at Paragraph 0341 that converting each input image of the person by portrait matting predictor to (a) a first colored background image that includes the person and a background of a first color and (b) a second colored background image that includes the person and a background of a second color.
The meta-data (configuration setting) includes the avatar’s surroundings (backgrounds) including hair appearance, earring, piercing and/or clothes, wherein Rabinovich clearly shows at Paragraph 0141 that each participant receives other participants’ avatars’ backgrounds.
Rabinovich teaches removal of accessories and/or clothes and thereby cutting off earing, piercing, clothes and/or hairs.
Rabinovich teaches at Paragraph 0818 that receiving, by the user device of the first participant (second user) and during the 3D video conference call, second participant (first user) constraints metadata indicative of one or more current constraints regarding a second participant. The second constraint metadata may refer to any aspect of the appearance of the avatar and its surroundings---for example, makeup, hair appearance, and/or inclusion or removal of accessories, e.g., glasses, sunglasses, earring, neckless, bracelet, piercing and/or clothes, inclusion or removal of tattoo and/or a scar.
Rabinovich teaches at Paragraph 0141 that each participant may be associated with a virtual or actual background and/or may receive any virtual or actual background in which avatars related to at least one of participants are displayed and at Paragraph 0310 that some parts of the video image may not be modelled as 3D objects and may be transmitted to the other users as pixel values…Examples include flat objects placed in the background such as white board or a picture on a wall and at Paragraph 0312 that in these cases the system may decide to transmit to viewers a video stream that would include at least some of the non-modelled parts and then the 3D models that exist would be rendered on top of the transmitted video in their respective locations.
Han teaches at Paragraph 0015-0017 that the participants may receive an image file of the immersive background….each participant receives the dynamic immersive background….each participant also receives metadata that describes how video streams are to be displayed in the dynamic immersive background. For example, a participant may receive an XML file that includes properties indicating the locations within the dynamic immersive background at which video streams may be displayed….the example system removes the background from the video stream…The area of the video stream outside the outline is then made removable…so that the area of the video stream outside the area does not obscure the dynamic immersive background.
Han teaches at Paragraph 0042-0043 that the hatched background displayed with User A on client device 210 is replaced by a removable background that allows the recipient’s client device to display the immersive scene in place of the background….a client device 140 receives a plurality of video streams…each of the plurality of video streams is recorded by a corresponding client device and has had its background identified made removable).
Re Claim 7:
The claim 7 encompasses the same scope of invention as that of the claim 5 except additional claim limitation that the rendering comprising: rendering a front surface of the avatar containing the video stream of the second user; rendering a back surface of the avatar containing a silhouette corresponding to the video stream of the second user; and rendering a side surface of the avatar of the second user.
Rabinovich further teaches the claim limitation that the rendering comprising: rendering a front surface of the avatar containing the video stream of the second user; rendering a back surface of the avatar containing a silhouette corresponding to the video stream of the second user; and rendering a side surface of the avatar of the second user (Rabinovich teaches at Paragraph 0356 that this real-time 3D textured model can then be used to render the view of the user from various angles and camera positions and specifically may be used to correct the viewing position of the virtual camera as if it were virtually located inside the screen of the user.
Rabinovich teaches at FIG. 5 that the front surface the avatar 55 is rendered as well as the side surface of the avatar 55 is rendered).
Re Claim 8:
The claim 8 recites a system, comprising: a memory;
at least one processor coupled to the memory and configured to:
alter portions of a video stream of a first user, the portions determined to represent physical surroundings of the first user, to generate an altered background;
send, to a second user, an avatar of the first user, the video stream with the altered background, and settings specifying how to present the altered background of the avatar of the first user to the second user in a virtual environment; and
based on the settings, cause a device associated with the second user to render the avatar of the first user and the altered background of the physical surroundings of the first user from a perspective of a virtual camera corresponding to the second user.
The claim 8 is in parallel with the method of the claim 1 in an apparatus form. The claim 9 is subject to the same rationale of rejection as the claim 1.
Moreover, Mittal further teaches a system, comprising: a memory;
at least one processor coupled to the memory to [perform the method steps of the claim 1] (Mittal teaches at column 8, lines 34-60 that the architecture 400 may include one or more processing elements 404 may be effective to perform object segmentation techniques for image data. The storage element 402 can include one or more different types of memory, data storage or computer-readable storage media devoted to different purposes within the architecture 400…Different portions of the storage element 402 may be used for program instructions for execution by the processing element 404….The storage element 402 may also store software for execution by the processing element 404).
Re Claim 9:
The claim 9 encompasses the same scope of invention as that of the claim 8 except additional claim limitation that the settings are selectable by the first user, the settings comprising a coloring property of an avatar background color, whether the avatar has a volumetric appearance, a framerate for applying background removal, a shape of the avatar, and a smoothness to edges of the video stream of the first user.
The claim 9 is in parallel with the method of the claim 2 in an apparatus form. The claim 9 is subject to the same rationale of rejection as the claim 2.
Re Claim 10:
The claim 10 encompasses the same scope of invention as that of the claim 8 except additional claim limitation that to alter the portions of the video stream contains further instructions stored thereon that when executed by the processor cause the computing device to:
apply a neural network to the video stream to detect first pixels representing the first user and second pixels represent the physical surroundings of the first user; and alter the second pixels representing the physical surroundings of the first user with a solid colored background.
The claim 10 is in parallel with the method of the claim 3 in an apparatus form. The claim 10 is subject to the same rationale of rejection as the claim 3.
Re Claim 11:
The claim 11 encompasses the same scope of invention as that of the claim 10 except additional claim limitation that the memory contains further instructions stored thereon that when executed by the processor cause the computing device to: replace the solid colored background using an HTML element; and translate the solid colored background into the video stream using an Application Programming Interface.
The claim 11 is in parallel with the method of the claim 4 in an apparatus form. The claim 11 is subject to the same rationale of rejection as the claim 4.
Re Claim 12:
The claim 12 encompasses the same scope of invention as that of the claim 8 except additional claim limitation that the memory contains further instructions stored thereon that when executed by the processor cause the computing device to:
receive an avatar of the second user, a video stream of the second user with a background, and second settings specifying how to present the avatar of the second user to the first user in a virtual environment of the video stream of the second user; and
based on the second settings, render, from a perspective of a virtual camera corresponding to the first user, the avatar of the second user with the video stream of the second user and the background.
The claim 12 is in parallel with the method of the claim 5 in an apparatus form. The claim 12 is subject to the same rationale of rejection as the claim 5.
Re Claim 13:
The claim 13 encompasses the same scope of invention as that of the claim 12 except additional claim limitation that the second settings are selectable by the second user, comprising a coloring property of an avatar background color, a volumetric appearance of the avatar of the second user, and a shape of the avatar of the second user.
The claim 13 is in parallel with the method of the claim 6 in an apparatus form. The claim 13 is subject to the same rationale of rejection as the claim 6.
Re Claim 14:
The claim 14 encompasses the same scope of invention as that of the claim 12 except additional claim limitation that to render contains further instructions stored thereon that when executed by the processor cause the computing device to:
render a front surface of the avatar containing the video stream of the second user; render a back surface of the avatar containing a silhouette corresponding to the video stream of the second user; and
render a side surface of the avatar of the second user.
The claim 14 is in parallel with the method of the claim 7 in an apparatus form. The claim 14 is subject to the same rationale of rejection as the claim 7.
Re Claim 15:
The claim 15 recites a non-transitory computer-readable medium having instructions stored thereon that, when executed by at least one computing device, causes the at least one computing device to perform operations comprising:
altering portions of a video stream of a first user, the portions determined to represent physical surroundings of the first user, to generate an altered background;
sending to a second user, an avatar of the first user, the video stream with the altered background, and settings specifying how to present the altered background of the avatar of the first user to the second user in a virtual environment; and
based on the settings, causing a device associated with the second user to render the avatar of the first user and the altered background of the physical surroundings of the first user from a perspective of a virtual camera corresponding to the second user.
Moreover, Mittal further teaches a non-transitory computer-readable medium having instructions stored thereon that, when executed by at least one computing device, causes the at least one computing device to perform operations [of the method steps of the claim 1] (Mittal teaches at column 8, lines 34-60 that the architecture 400 may include one or more processing elements 404 may be effective to perform object segmentation techniques for image data. The storage element 402 can include one or more different types of memory, data storage or computer-readable storage media devoted to different purposes within the architecture 400…Different portions of the storage element 402 may be used for program instructions for execution by the processing element 404….The storage element 402 may also store software for execution by the processing element 404).
Re Claim 16:
The claim 16 encompasses the same scope of invention as that of the claim 15 except additional claim limitation that the settings are selectable by the first user, the settings comprising a coloring property of an avatar background color, whether the avatar has a volumetric appearance, a framerate for applying background removal, a shape of the avatar, and a smoothness to edges of the video stream of the first user.
The claim 16 is in parallel with the method of the claim 2 in an apparatus form. The claim 16 is subject to the same rationale of rejection as the claim 2.
Re Claim 17:
The claim 17 encompasses the same scope of invention as that of the claim 15 except additional claim limitation that altering the portions of the video stream further comprises:
applying a neural network to the video stream to detect first pixels representing the first user and second pixels represent the physical surroundings of the first user; and
altering the second pixels representing the physical surroundings of the first user with a solid colored background.
The claim 17 is in parallel with the method of the claim 3 in an apparatus form. The claim 17 is subject to the same rationale of rejection as the claim 3.
Re Claim 18:
The claim 18 encompasses the same scope of invention as that of the claim 15 except additional claim limitation that receiving, from the second user, an avatar of the second user, a video stream of the second user with a background, and second settings specifying how to present the avatar of the second user to the first user in a virtual environment of the video stream of the second user; and
based on the second settings, rendering, from a perspective of a virtual camera corresponding to the first user, the avatar of the second user with the video stream of the second user and the background.
The claim 18 is in parallel with the method of the claim 5 in an apparatus form. The claim 18 is subject to the same rationale of rejection as the claim 5.
Re Claim 19:
The claim 19 encompasses the same scope of invention as that of the claim 18 except additional claim limitation that the second settings are selectable by the second user, comprising a coloring property of an avatar background color, a volumetric appearance of the avatar of the second user, and a shape of the avatar of the second user.
The claim 13 is in parallel with the method of the claim 6 in an apparatus form. The claim 13 is subject to the same rationale of rejection as the claim 6.
Re Claim 20:
The claim 20 encompasses the same scope of invention as that of the claim 18 except additional claim limitation that the rendering comprising:
rendering a front surface of the avatar containing the video stream of the second user; rendering a back surface of the avatar containing a silhouette corresponding to the video stream of the second user; and
rendering a side surface of the avatar of the second user..
The claim 20 is in parallel with the method of the claim 7 in an apparatus form. The claim 20 is subject to the same rationale of rejection as the claim 7.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JIN CHENG WANG whose telephone number is (571)272-7665. The examiner can normally be reached Mon-Fri 8:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, King Poon can be reached at 571-270-0728. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JIN CHENG WANG/Primary Examiner, Art Unit 2617
Prosecution Insights