DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP §§ 706.02(l)(1) - 706.02(l)(3) for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/process/file/efs/guidance/eTD-info-I.jsp.
Claim 1 is rejected on the ground of nonstatutory double patenting as being unpatentable over claim 1 of Hellge et al. US Patent No. 12,112,426. Although the claims at issue are not identical, they are not patentably distinct from each other because all of the claim limitations set forth in the Claim 1 of the instant application are only a subset of the claim limitations set forth in the patented claim 1. Therefore, it would have been obvious to one of ordinary skill in the art to make the claim made in this application, because it is only a subset of what has been claimed before.
Claim 2 is rejected on the ground of nonstatutory double patenting as being unpatentable over claim 1 of the US Patent No. 12,112,426. Although the claims at issue are not identical, they are not patentably distinct from each other because all of the claim limitations set forth in the Claim 2 of the instant application are only a subset of the claim limitations set forth in the patented claim 1. Therefore, it would have been obvious to one of ordinary skill in the art to make the claim made in this application, because it is only a subset of what has been claimed before.
Claim 3 is rejected on the ground of nonstatutory double patenting as being unpatentable over claim 2 of the US Patent No. 12,112,426. Although the claims at issue are not identical, they are not patentably distinct from each other because all of the claim limitations set forth in the Claim 3 of the instant application are only a subset of the claim limitations set forth in the patented claim 2. Therefore, it would have been obvious to one of ordinary skill in the art to make the claim made in this application, because it is only a subset of what has been claimed before.
Claim 4 is rejected on the ground of nonstatutory double patenting as being unpatentable over claim 3 of the US Patent No. 12,112,426. Although the claims at issue are not identical, they are not patentably distinct from each other because all of the claim limitations set forth in the Claim 4 of the instant application are only a subset of the claim limitations set forth in the patented claim 3. Therefore, it would have been obvious to one of ordinary skill in the art to make the claim made in this application, because it is only a subset of what has been claimed before.
Claim 5 is rejected on the ground of nonstatutory double patenting as being unpatentable over claim 4 of the US Patent No. 12,112,426. Although the claims at issue are not identical, they are not patentably distinct from each other because all of the claim limitations set forth in the Claim 5 of the instant application are only a subset of the claim limitations set forth in the patented claim 4. Therefore, it would have been obvious to one of ordinary skill in the art to make the claim made in this application, because it is only a subset of what has been claimed before.
Claim 6 is rejected on the ground of nonstatutory double patenting as being unpatentable over claim 5 of the US Patent No. 12,112,426. Although the claims at issue are not identical, they are not patentably distinct from each other because all of the claim limitations set forth in the Claim 6 of the instant application are only a subset of the claim limitations set forth in the patented claim 5. Therefore, it would have been obvious to one of ordinary skill in the art to make the claim made in this application, because it is only a subset of what has been claimed before.
Claim 7 is rejected on the ground of nonstatutory double patenting as being unpatentable over claim 6 of the US Patent No. 12,112,426. Although the claims at issue are not identical, they are not patentably distinct from each other because all of the claim limitations set forth in the Claim 7 of the instant application are only a subset of the claim limitations set forth in the patented claim 6. Therefore, it would have been obvious to one of ordinary skill in the art to make the claim made in this application, because it is only a subset of what has been claimed before.
Claim 8 is rejected on the ground of nonstatutory double patenting as being unpatentable over claim 7 of the US Patent No. 10,861,205. Although the claims at issue are not identical, they are not patentably distinct from each other because all of the claim limitations set forth in the Claim 8 of the instant application are only a subset of the claim limitations set forth in the patented claim 7. Therefore, it would have been obvious to one of ordinary skill in the art to make the claim made in this application, because it is only a subset of what has been claimed before.
Claim 9 is rejected on the ground of nonstatutory double patenting as being unpatentable over claim 8 of the US Patent No. 12,112,426. Although the claims at issue are not identical, they are not patentably distinct from each other because all of the claim limitations set forth in the Claim 9 of the instant application are only a subset of the claim limitations set forth in the patented claim 8. Therefore, it would have been obvious to one of ordinary skill in the art to make the claim made in this application, because it is only a subset of what has been claimed before.
Claim 10 is rejected on the ground of nonstatutory double patenting as being unpatentable over claim 10 of the US Patent No. 12,112,426. Although the claims at issue are not identical, they are not patentably distinct from each other because all of the claim limitations set forth in the Claim 10 of the instant application are only a subset of the claim limitations set forth in the patented claim 10. Therefore, it would have been obvious to one of ordinary skill in the art to make the claim made in this application, because it is only a subset of what has been claimed before.
Claim 11 is rejected on the ground of nonstatutory double patenting as being unpatentable over claim 9 of the US Patent No. 12,112,426. Although the claims at issue are not identical, they are not patentably distinct from each other because all of the claim limitations set forth in the Claim 11 of the instant application are only a subset of the claim limitations set forth in the patented claim 9. Therefore, it would have been obvious to one of ordinary skill in the art to make the claim made in this application, because it is only a subset of what has been claimed before.
Claim 12 is rejected on the ground of nonstatutory double patenting as being unpatentable over claim 11 of the US Patent No. 12,112,426. Although the claims at issue are not identical, they are not patentably distinct from each other because all of the claim limitations set forth in the Claim 12 of the instant application are only a subset of the claim limitations set forth in the patented claim 11. Therefore, it would have been obvious to one of ordinary skill in the art to make the claim made in this application, because it is only a subset of what has been claimed before.
Claim 13 is rejected on the ground of nonstatutory double patenting as being unpatentable over claim 12 of the US Patent No. 12,112,426. Although the claims at issue are not identical, they are not patentably distinct from each other because all of the claim limitations set forth in the Claim 13 of the instant application are only a subset of the claim limitations set forth in the patented claim 12. Therefore, it would have been obvious to one of ordinary skill in the art to make the claim made in this application, because it is only a subset of what has been claimed before.
Claim 14 is rejected on the ground of nonstatutory double patenting as being unpatentable over claim 13 of the US Patent No. 12,112,426. Although the claims at issue are not identical, they are not patentably distinct from each other because all of the claim limitations set forth in the Claim 14 of the instant application are only a subset of the claim limitations set forth in the patented claim 13. Therefore, it would have been obvious to one of ordinary skill in the art to make the claim made in this application, because it is only a subset of what has been claimed before.
Claim 15 is rejected on the ground of nonstatutory double patenting as being unpatentable over claim 14 of the US Patent No. 12,112,426. Although the claims at issue are not identical, they are not patentably distinct from each other because all of the claim limitations set forth in the Claim 15 of the instant application are only a subset of the claim limitations set forth in the patented claim 14. Therefore, it would have been obvious to one of ordinary skill in the art to make the claim made in this application, because it is only a subset of what has been claimed before.
Claim 16 is rejected on the ground of nonstatutory double patenting as being unpatentable over claim 15 of the US Patent No. 12,112,426. Although the claims at issue are not identical, they are not patentably distinct from each other because all of the claim limitations set forth in the Claim 16 of the instant application are only a subset of the claim limitations set forth in the patented claim 15. Therefore, it would have been obvious to one of ordinary skill in the art to make the claim made in this application, because it is only a subset of what has been claimed before.
Claim 17 is rejected on the ground of nonstatutory double patenting as being unpatentable over claim 16 of the US Patent No. 12,112,426. Although the claims at issue are not identical, they are not patentably distinct from each other because all of the claim limitations set forth in the Claim 17 of the instant application are only a subset of the claim limitations set forth in the patented claim 16. Therefore, it would have been obvious to one of ordinary skill in the art to make the claim made in this application, because it is only a subset of what has been claimed before.
Claim 18 is rejected on the ground of nonstatutory double patenting as being unpatentable over claim 16 of the US Patent No. 12,112,426. Although the claims at issue are not identical, they are not patentably distinct from each other because all of the claim limitations set forth in the Claim 18 of the instant application are only a subset of the claim limitations set forth in the patented claim 16. Therefore, it would have been obvious to one of ordinary skill in the art to make the claim made in this application, because it is only a subset of what has been claimed before.
Claim 19 is rejected on the ground of nonstatutory double patenting as being unpatentable over claim 16 of the US Patent No. 12,112,426. Although the claims at issue are not identical, they are not patentably distinct from each other because all of the claim limitations set forth in the Claim 19 of the instant application are only a subset of the claim limitations set forth in the patented claim 16. Therefore, it would have been obvious to one of ordinary skill in the art to make the claim made in this application, because it is only a subset of what has been claimed before.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-6 and 15-19 are rejected under 35 U.S.C. 103 as being unpatentable over Babu et al. US-PGPUB No. 2019/0199993 (hereinafter Babu) in view of Bathiche US-PGPUB No. 2012/0320169 (hereinafter Bathiche).
Re Claim 1:
Babu teaches a method of augmenting a view of a real-world environment with a view of a volumetric video object on a user device, the method comprising the steps of:
determining a current pose information (CPI) indicating a current pose of the view of the real-world environment and a desired pose of the volumetric video object in the real-world environment (Babu teaches at Paragraph 0077 that the mixed reality media player 164 may select and retrieve data having a mixed reality video format 162 from a 3D video (server) database 160….to provide the final composition of the video based at least in part on the user’s head pose to be displayed to the user 50…The mixed reality engine 166 may render the video and models that may be used for the final composition of the scene based on the user’s head pose and at Paragraph 0084 that the 3D video 115 may include 3D animation objects that may be rendered and displayed along with the 3D video to provide the user 50 using the display system 104 with a realistic view of a 3D effect…..if the user moves to a viewing perspective to a side view, the user may actually see a 3D object and/or 3D animated object come out from the virtual television display screen and into the user’s physical environment 105).
Babu at least suggests the claim limitation:
sending the current pose information to a remote server (
Babu teaches at Paragraph 0077 that the mixed reality media player 164 may select and retrieve data having a mixed reality video format 162 from a 3D video (server) database 160….to provide the final composition of the video based at least in part on the user’s head pose to be displayed to the user 50…The mixed reality engine 166 may render the video and models that may be used for the final composition of the scene based on the user’s head pose and at Paragraph 0081 that the remote processing module may include on or more relatively powerful processors configured to analyze and process data and/or image information);
receiving a rendered view of the volumetric video object that has been rendered in accordance with the current pose information from the remote server (Babu teaches at Paragraph 0140 that displaying the pre-rendered 3D video. Babu teaches at Paragraph 0077 that the mixed reality media player 164 may select and retrieve data having a mixed reality video format 162 from a 3D video (server) database 160….to provide the final composition of the video based at least in part on the user’s head pose to be displayed to the user 50…The mixed reality engine 166 may render the video and models that may be used for the final composition of the scene based on the user’s head pose); and
augmenting the view of the real-world environment by at least mapping the rendered view of the volumetric video object onto a planar mapping surface arranged according to the desired position of the volumetric video object (
Babu teaches at Paragraph 0089 that the occlusion mask may be placed relative to the real-world mesh to give the appearance of a virtual opening in a real-world wall and at Paragraph 0128 that the 3D video content may be rendered onto the objects identified form the user’s environment.
Babu teaches at FIG. 1 mapping the rendered view of the volumetric view 115 within the virtual television 120 onto a planar mapping surface (e.g., a real picture frame of the vertical wall 125 corresponding to the virtual television 120) arranged according to the desired position of the volumetric video object 115, Babu teaches at Paragraph 0083 that the user’s physical environment 105 may include a virtual television 120 displayed on a vertical wall 125 and the virtual television 120 may be anchored and/or fixed to either a blank vertical wall 125 or displayed over a picture frame (not shown) hanging on a vertical wall at Paragraph 0110 that display screen/planar surface 320 may be one or more of display screen of a television, a computer monitor for displaying a 3D video.
Babu teaches at Paragraph 0084 that the 3D video 115 may include 3D animation objects that may be rendered and displayed along with the 3D video to provide the user 50 using the display system 104 with a realistic view of a 3D effect…..if the user moves to a viewing perspective to a side view, the user may actually see a 3D object and/or 3D animated object come out from the virtual television display screen and into the user’s physical environment 105).
Bathiche teaches the claim limitation:
sending the current pose information to a remote server (
Bathiche teaches at Paragraph 0024 that video presentation device 1120 is configured to receive viewer tracking image data from various sensors on head-mounted display system 110 to allow a position and an orientation of the eyes of viewer 100 in presentation space 102 to be tracked);
receiving a rendered view of the volumetric video object that has been rendered in accordance with the current pose information from the remote server (
Bathiche teaches at Paragraph 0035-0036 presenting volumetric video data via a head-mounted display…a video presentation device may calculate image sensors located in the presentation space to the volume of the presentation space….receiving volumetric image data and mapping the volumetric image data to the presentation space and at Paragraph 0039-0042 performing various processes to determine an image to send to the head-mounted display for presentation to the viewer….based on the viewer tracking data, orientation data and/or eyeball tracking data, method 400 comprises determining a portion and orientation of the current frame of volumetric video data that is within the viewer’s field of view); and
augmenting the view of the real-world environment by at least mapping the rendered view of the volumetric video object onto a planar mapping surface arranged according to the desired position of the volumetric video object (
Bathiche teaches at Paragraph 0020 mapping volumetric video data to the presentation space and at Paragraph 0037 that the location of images of walls in the scene may correspond to the location of actual walls in the presentation space.
Bathiche teaches at Paragraph 0042 that determining a portion and orientation of the current frame of volumetric video data that is within the viewer’s field of view in the presentation space based upon the mapping of the presentation space to the video data…the video presentation computing device determines a presentation image to send to the head-mounted display system for presentation and sends the image…for each frame of volumetric video data, a volumetric presentation may be presented to the viewer).
It would have been obvious to one of the ordinary skill in the art before the filing date of the instant application to have combined Bathiche’s teaching of presenting volumetric video object on the presentation space of the head-mounted display system into Babu’s system of presenting a volumetric video object on the physical space of the head-mounted display system for augmented reality viewing. One of the ordinary skill in the art would have been motivated to have sent a pose/posture of the head-mounted display system to the video presentation device for retrieving the volumetric video object according to the pose/posture of the head-mounted display system and to have mapped the volumetric video object onto surfaces of the presentation space.
Re Claim 2:
The claim 2 encompasses the same scope of invention as that of the claim 1 except additional claim limitation that the planar mapping surface is oriented such that the rendered view of the volumetric video object is projected into the view of the real-world environment substantially free of perspective distortion.
Babu further teaches the claim limitation that the planar mapping surface is oriented such that the rendered view of the volumetric video object is projected into the view of the real-world environment substantially free of perspective distortion (Babu teaches at Paragraph 0087 that a user 50 may view through the opaque planar surface via the opening of the planar surface….if a user is to view an object from a first perspective having a direct frontal view position, the object may appear to be a flat 2D frontal view of the object with the portal framing a border around the object. The portable may appear to be any shape such as a circle, a rectangle, a square, a polygon from this first perspective and at Paragraph 0088 that if the user is to view the object from a second perspective having a side view position, portions of the object may be visible to the user and other portions of the object may be blocked or not visible, depending on the side view angle of the second perspective.
Babu teaches at Paragraph 0068 that the AR system 100 may be operated in conjunction with an AR projection subsystem 110 which may provide 3D videos in a field of view of an end user 50 and at FIG. 1 mapping the rendered view of the volumetric view 115 within the virtual television 120 onto a planar mapping surface (e.g., a real picture frame of the vertical wall 125 corresponding to the virtual television 120) arranged according to the desired position of the volumetric video object 115, Babu teaches at Paragraph 0083 that the user’s physical environment 105 may include a virtual television 120 displayed on a vertical wall 125 and the virtual television 120 may be anchored and/or fixed to either a blank vertical wall 125 or displayed over a picture frame (not shown) hanging on a vertical wall at Paragraph 0110 that display screen/planar surface 320 may be one or more of display screen of a television, a computer monitor for displaying a 3D video.
Babu teaches at Paragraph 0084 that the 3D video 115 may include 3D animation objects that may be rendered and displayed along with the 3D video to provide the user 50 using the display system 104 with a realistic view of a 3D effect…..if the user moves to a viewing perspective to a side view, the user may actually see a 3D object and/or 3D animated object come out from the virtual television display screen and into the user’s physical environment 105).
Re Claim 3:
The claim 3 encompasses the same scope of invention as that of the claim 1 except additional claim limitation that the rendered view of the volumetric video object comprises at least one background region that does not show the volumetric video object, and wherein the augmenting step comprises setting the at least one background region to be transparent.
Babu further teaches the claim limitation that the rendered view of the volumetric video object comprises at least one background region that does not show the volumetric video object, and wherein the augmenting step comprises setting the at least one background region to be transparent (Babu teaches at FIG. 1 along with FIG. 3A that the rendered view of the volumetric video object 330b comprises at least one background region of the wall surface 125 and wherein the augmenting step comprises setting the at least one background region corresponding to the masked portion of the video object 330b to be transparent.
Babu teaches at FIG. 1 mapping the rendered view of the volumetric view 115 within the virtual television 120 onto a planar mapping surface (e.g., a real picture frame of the vertical wall 125 corresponding to the virtual television 120) arranged according to the desired position of the volumetric video object 115, Babu teaches at Paragraph 0083 that the user’s physical environment 105 may include a virtual television 120 displayed on a vertical wall 125 and the virtual television 120 may be anchored and/or fixed to either a blank vertical wall 125 or displayed over a picture frame (not shown) hanging on a vertical wall at Paragraph 0110 that display screen/planar surface 320 may be one or more of display screen of a television, a computer monitor for displaying a 3D video.
Babu teaches at Paragraph 0084 that the 3D video 115 may include 3D animation objects that may be rendered and displayed along with the 3D video to provide the user 50 using the display system 104 with a realistic view of a 3D effect…..if the user moves to a viewing perspective to a side view, the user may actually see a 3D object and/or 3D animated object come out from the virtual television display screen and into the user’s physical environment 105).
Re Claim 4:
The claim 4 encompasses the same scope of invention as that of the claim 3 except additional claim limitation that the rendered view of the volumetric video object comprises a transparency map that indicates the at least one background region is indicated in the rendered view of the volumetric video object by a predefined color, and wherein the augmenting step further comprises determining the at least one background region based on the transparency map or the predefined color.
Babu further teaches the claim limitation that the rendered view of the volumetric video object comprises a transparency map that indicates the at least one background region is indicated in the rendered view of the volumetric video object by a predefined color, and wherein the augmenting step further comprises determining the at least one background region based on the transparency map or the predefined color (
Babu teaches at Paragraph 0115 that the 3D models may correspond to the 3D video such that if the 3D video scene is a certain blue color and the 3D model of the 3D object is of the same or substantially similar blue color, then the 3D model may not be visible to a user. Therefore, the 3D model may be slightly adjusted in color, texture, contrast or other characteristics in order for the user to detect the 3D model as it is being displayed with the 3D video.
Babu teaches at FIG. 1 along with FIG. 3A that the rendered view of the volumetric video object 330b comprises at least one background region of the wall surface 125 and wherein the augmenting step comprises setting the at least one background region corresponding to the masked portion of the video object 330b to be transparent.
Babu teaches at FIG. 1 mapping the rendered view of the volumetric view 115 within the virtual television 120 onto a planar mapping surface (e.g., a real picture frame of the vertical wall 125 corresponding to the virtual television 120) arranged according to the desired position of the volumetric video object 115, Babu teaches at Paragraph 0083 that the user’s physical environment 105 may include a virtual television 120 displayed on a vertical wall 125 and the virtual television 120 may be anchored and/or fixed to either a blank vertical wall 125 or displayed over a picture frame (not shown) hanging on a vertical wall at Paragraph 0110 that display screen/planar surface 320 may be one or more of display screen of a television, a computer monitor for displaying a 3D video.
Babu teaches at Paragraph 0084 that the 3D video 115 may include 3D animation objects that may be rendered and displayed along with the 3D video to provide the user 50 using the display system 104 with a realistic view of a 3D effect…..if the user moves to a viewing perspective to a side view, the user may actually see a 3D object and/or 3D animated object come out from the virtual television display screen and into the user’s physical environment 105).
Re Claim 5:
The claim 5 encompasses the same scope of invention as that of the claim 3 except additional claim limitation that the augmenting step further comprises segmenting the volumetric video object as shown in the rendered view of the volumetric video object and determining the at least one background region based on the segmentation of the volumetric video object.
Babu further teaches the claim limitation that the augmenting step further comprises segmenting the volumetric video object as shown in the rendered view of the volumetric video object and determining the at least one background region based on the segmentation of the volumetric video object (
Babu teaches at FIG. 1 along with FIG. 3A that the rendered view of the volumetric video object 330b comprises at least one background region of the wall surface 125 and wherein the augmenting step comprises setting the at least one background region corresponding to the masked portion of the video object 330b to be transparent.
Babu teaches at FIG. 1 mapping the rendered view of the volumetric view 115 within the virtual television 120 onto a planar mapping surface (e.g., a real picture frame of the vertical wall 125 corresponding to the virtual television 120) arranged according to the desired position of the volumetric video object 115, Babu teaches at Paragraph 0083 that the user’s physical environment 105 may include a virtual television 120 displayed on a vertical wall 125 and the virtual television 120 may be anchored and/or fixed to either a blank vertical wall 125 or displayed over a picture frame (not shown) hanging on a vertical wall at Paragraph 0110 that display screen/planar surface 320 may be one or more of display screen of a television, a computer monitor for displaying a 3D video.
Babu teaches at Paragraph 0084 that the 3D video 115 may include 3D animation objects that may be rendered and displayed along with the 3D video to provide the user 50 using the display system 104 with a realistic view of a 3D effect…..if the user moves to a viewing perspective to a side view, the user may actually see a 3D object and/or 3D animated object come out from the virtual television display screen and into the user’s physical environment 105).
Re Claim 6:
The claim 6 encompasses the same scope of invention as that of the claim 1 except additional claim limitation that the size and/or shape of the planar mapping surface is dynamically set based on the size and/or shape of the volumetric video object as shown in the rendered view of the volumetric video object.
Babu further teaches the claim limitation that the size and/or shape of the planar mapping surface is dynamically set based on the size and/or shape of the volumetric video object as shown in the rendered view of the volumetric video object (Babu teaches at Paragraph 0017 that the 3D video is rendered within the planar viewing display of the virtual television).
Re Claim 15:
The claim 15 encompasses the same scope of invention as that of the claim 1 except additional claim limitation that the view of the real-world environment and the rendered view of the volumetric video object are stereoscopic views comprising two different views for the two eyes of a user, and wherein the augmenting step comprises at least mapping each of the two different views of the rendered view of the volumetric video object onto a planar mapping surface.
Babu further teaches the claim limitation that the view of the real-world environment and the rendered view of the volumetric video object are stereoscopic views comprising two different views for the two eyes of a user, and wherein the augmenting step comprises at least mapping each of the two different views of the rendered view of the volumetric video object onto a planar mapping surface (Babu teaches at Paragraph 0014 that the first 2D stereoscopic image is for a right eye and the second 2D stereoscopic image is for a left eye….the first 2D stereoscopic image and the second 2D stereoscopic image are sourced from a traditional 3D video, e.g., a 3D movie and at Paragraph 0017 that the volume space is a portal view of a 3D object, wherein the 3D object is a virtual television having a planar viewing display. The 3D video is rendered within the planar viewing display of the virtual television).
Re Claim 16:
The claim 16 encompasses the same scope of invention as that of the claim 1 except additional claim limitation that the view of the real-world environment is a stereoscopic view comprising two different views for the two eyes of a user, wherein the receiving step comprises receiving stereoscopic view generation information for rendering a stereoscopic view of the volumetric video object from the rendered view of the volumetric video object, and wherein the augmenting step further comprises generating the stereoscopic view of the volumetric video object from the rendered view of the volumetric video object in accordance with the stereoscopic view generation information and mapping each of the two different views of the generated stereoscopic view of the volumetric video object onto a planar mapping surface.
Babu further teaches the claim limitation that the view of the real-world environment is a stereoscopic view comprising two different views for the two eyes of a user, wherein the receiving step comprises receiving stereoscopic view generation information for rendering a stereoscopic view of the volumetric video object from the rendered view of the volumetric video object, and wherein the augmenting step further comprises generating the stereoscopic view of the volumetric video object from the rendered view of the volumetric video object in accordance with the stereoscopic view generation information and mapping each of the two different views of the generated stereoscopic view of the volumetric video object onto a planar mapping surface (Babu teaches at Paragraph 0014 that the first 2D stereoscopic image is for a right eye and the second 2D stereoscopic image is for a left eye….the first 2D stereoscopic image and the second 2D stereoscopic image are sourced from a traditional 3D video, e.g., a 3D movie and at Paragraph 0017 that the volume space is a portal view of a 3D object, wherein the 3D object is a virtual television having a planar viewing display. The 3D video is rendered within the planar viewing display of the virtual television).
Re Claim 17:
The claim 17 recites a user device for augmenting a view of a real-world environment with a view of a volumetric video object, comprising:
a determining unit for determining a current pose information (CPI) indicating a current pose of the view of the real-world environment and a desired pose of the volumetric video object in the real-world environment;
a sending unit for sending the CPI to a remote server;
a receiving unit for receiving a rendered view of the volumetric video object that has been rendered in accordance with the CPI from the remote server; and
an augmenting unit for augmenting the real-world environment by at least mapping the rendered view of the volumetric video object onto a planar mapping surface arranged according to the desired position of the volumetric video object.
The claim 17 is in parallel with the claim 1 in the form of an apparatus claim. The claim 38 is subject to the same rationale of rejection as the claim 1.
Re Claim 18:
The claim 18 recites a method of rendering a view of a volumetric video object for augmenting a view of a real-world environment on a remote server, comprising:
receiving a current pose information (CPI) indicating a current pose of the view of the real-world environment and a desired pose of the volumetric video object in the real-world environment from a user device;
rendering a view of the volumetric video object in accordance with the CPI; and
sending the rendered view of the volumetric video object to the user device.
The claim 18 is broader than the claim 1 and thus is subject to the same rationale of rejection as the claim 1.
Re Claim 19:
The claim 19 recites a remote server for rendering a view of a volumetric video object for augmenting a view of a real-world environment, comprising:
a receiving unit for receiving a current pose information (CPI) indicating a current pose of the view of the real-world environment and a desired pose of the volumetric video object in the real-world environment from a user device;
a rendering unit for rendering a view of the volumetric video object in accordance with the CPI; and
a sending unit for sending the rendered view of the volumetric video object to the user device.
The claim 19 is in parallel with the claim 18 in the form of an apparatus claim. The claim 19 is subject to the same rationale of rejection as the claim 18.
Claims 7-9 and 11-14 are rejected under 35 U.S.C. 103 as being unpatentable over Babu et al. US-PGPUB No. 2019/0199993 (hereinafter Babu) in view of Bathiche US-PGPUB No. 2012/0320169 (hereinafter Bathiche) and Olsen et al. US-Patent No. 11,513,658 (hereinafter Olsen).
Re Claim 7:
The Claim 7 encompasses the same scope of invention as that of the claim 1 except additional claim limitation that the receiving step further comprises receiving an approximate volume information indicating the approximate volume of the volumetric video object, and wherein the augmenting step further comprises using the approximate volume information for one or more of collision avoidance, correcting a lighting of the rendered view of the volumetric video object, generating at least one shadow of the volumetric video object in the real-world environment, and generating at least one reflection of the volumetric video object in the real-world environment.
Babu at least suggests the claim limitation that the receiving step further comprises receiving an approximate volume information indicating the approximate volume of the volumetric video object, and wherein the augmenting step further comprises using the approximate volume information for one or more of collision avoidance, correcting a lighting of the rendered view of the volumetric video object (Babu teaches at FIG. 3A that the augmenting step further comprises the approximate truncated volume for one or more of collision avoidance with the surface 320 and at Paragraph 0115 correcting a lighting of the rendered view of the volumetric video object by adjusting the 3D model in color, texture, contrast in order for the user to detect the 3D model is being displayed with the 3D video).
However, Olsen et al. US-Patent No. 11,513,658 (hereinafter Olsen) further teaches generating at least one shadow of the volumetric video object in the real-world environment, and generating at least one reflection of the volumetric video object in the real-world environment (Olsen teaches at column 10, lines 40-65 that the graphics data may be used in 3D rendering of video content…that is used to generate models of objects for the scene…general scene information such as surfaces, textures, colors, lighting sources and so on 3D video may be rendered according to one or more 3D video formats…information for global operations or effects in the scenes such as illumination, reflection, shadows, and simulated effects such as rain, fire, smoke, dust and fog…may be used in generating a modeled world for the scene and in rendering 2D representations of the world as video output…instead of generating 2D video output, 3D video may be rendered according to one or more 3D video format).
It would have been obvious to one of the ordinary skill in the art before the filing date of the instant application to have incorporated Olsen’s teaching of adjusting the reflection, shadow of the volumetric video object into Babu to have further enhanced Babu’s realistic rendering of the 3D video object in the physical environment. One of the ordinary skill in the art would have been motivated to have provided lighting sources, textures, illuminations, colors, reflections and shadows for rendering the 3D video object.
Re Claim 8:
The claim 8 encompasses the same scope of invention as that of the claim 7 except additional claim limitation that the approximate volume information comprises first approximate volume information for visible parts of the volumetric video object as shown in the rendered view of the volumetric video object and/or second approximate volume information for non-visible parts of the volumetric video object as shown in the rendered view of the volumetric video object.
Babu further teaches the claim limitation that that the approximate volume information comprises first approximate volume information for visible parts of the volumetric video object as shown in the rendered view of the volumetric video object and/or second approximate volume information for non-visible parts of the volumetric video object as shown in the rendered view of the volumetric video object (Babu teaches at FIG. 1 along with FIG. 3A that the rendered view of the volumetric video object 330b comprises at least one background region of the wall surface 125 and wherein the augmenting step comprises setting the at least one background region corresponding to the masked portion of the video object 330b to be transparent.
Babu teaches at FIG. 3A that the volumetric video object 330a is segmented into visible parts 330b and non-visible parts---the truncated parts).
Re Claim 9:
The claim 9 encompasses the same scope of invention as that of the claim 8 except additional claim limitation that the first approximate volume information and/or the second approximate volume information comprises at least one of a depth map, mesh data, and one or more geometric primitives that approximate the volume of the volumetric video object.
Babu further teaches the claim limitation that the first approximate volume information and/or the second approximate volume information comprises at least one of a depth map, mesh data, and one or more geometric primitives that approximate the volume of the volumetric video object (Babu teaches at Paragraph 0017 that a first depth information from the 3D video is added to a second depth information from a first location of the portion of the virtual and/or augmented environment volume space to a second location of a user viewing the 3D video and at Paragraph 0109 that one solution to this problem may include a video file format and player that includes mesh, e.g., 3D models and/or 3D animations and at Paragraph 0137 that some assets may include 3D models, mesh, animations, textures, shaders, and lights and at Paragraph 0138 that the data store 710 may include information 712 about the video, models 718, e.g., meshes of 3D models, material 720, texture 724 and control data 726).
Re Claim 11:
The claim 11 encompasses the same scope of invention as that of the claim 7 except additional claim limitation that the determining step further comprises determining a reflection information indicating at least one reflection of the volumetric video object in the real-world environment, wherein the sending step further comprises sending the reflection information to the remote server, wherein the receiving step further comprises receiving at least one rendered view of the at least one reflection that has been rendered taking into account the reflection information from the remote server, and wherein the augmenting step further comprises generating the at least one reflection of the volumetric video object in the real-world environment based on the at least one rendered view of the at least one reflection.
However, Olsen et al. US-Patent No. 11,513,658 (hereinafter Olsen) further teaches the determining step further comprises determining a reflection information indicating at least one reflection of the volumetric video object in the real-world environment, wherein the sending step further comprises sending the reflection information to the remote server, wherein the receiving step further comprises receiving at least one rendered view of the at least one reflection that has been rendered taking into account the reflection information from the remote server, and wherein the augmenting step further comprises generating the at least one reflection of the volumetric video object in the real-world environment based on the at least one rendered view of the at least one reflection (Olsen teaches at column 10, lines 40-65 that the graphics data may be used in 3D rendering of video content…that is used to generate models of objects for the scene…general scene information such as surfaces, textures, colors, lighting sources and so on 3D video may be rendered according to one or more 3D video formats…information for global operations or effects in the scenes such as illumination, reflection, shadows, and simulated effects such as rain, fire, smoke, dust and fog…may be used in generating a modeled world for the scene and in rendering 2D representations of the world as video output…instead of generating 2D video output, 3D video may be rendered according to one or more 3D video format).
It would have been obvious to one of the ordinary skill in the art before the filing date of the instant application to have incorporated Olsen’s teaching of adjusting the reflection, shadow of the volumetric video object into Babu to have further enhanced Babu’s realistic rendering of the 3D video object in the physical environment. One of the ordinary skill in the art would have been motivated to have provided lighting sources, textures, illuminations, colors, reflections and shadows for rendering the 3D video object.
Re Claim 12:
The claim 12 encompasses the same scope of invention as that of the claim 1 except additional claim limitation that the sending step further comprises sending the view of the real-world environment to the remote server, wherein the rendered view of the volumetric video object has been rendered taking into account a lighting of the real- world environment, and/or wherein the receiving step further comprises receiving at least one rendered view of at least one shadow of the volumetric video object in the real-world environment, and/or wherein the receiving step further comprises receiving at least one rendered view of at least one reflection that has been rendered taking into account at least one reflection of the volumetric video object in the real-world environment, and/or wherein the augmenting step comprises generating the at least one shadow of the volumetric video object in the real-world environment based on the at least one rendered view of the at least one shadow, and/or wherein the augmenting step comprises generating the at least one reflection of the volumetric video object in the real-world environment based on the at least one rendered view of the at least one reflection, wherein the lighting and/or the at least one shadow and/or the at least one reflection has been determined based on the view of the real- world environment.
However, Olsen et al. US-Patent No. 11,513,658 (hereinafter Olsen) further teaches the sending step further comprises sending the view of the real-world environment to the remote server, wherein the rendered view of the volumetric video object has been rendered taking into account a lighting of the real- world environment, and/or wherein the receiving step further comprises receiving at least one rendered view of at least one shadow of the volumetric video object in the real-world environment, and/or wherein the receiving step further comprises receiving at least one rendered view of at least one reflection that has been rendered taking into account at least one reflection of the volumetric video object in the real-world environment, and/or wherein the augmenting step comprises generating the at least one shadow of the volumetric video object in the real-world environment based on the at least one rendered view of the at least one shadow, and/or wherein the augmenting step comprises generating the at least one reflection of the volumetric video object in the real-world environment based on the at least one rendered view of the at least one reflection, wherein the lighting and/or the at least one shadow and/or the at least one reflection has been determined based on the view of the real- world environment (Olsen teaches at column 10, lines 40-65 that the graphics data may be used in 3D rendering of video content…that is used to generate models of objects for the scene…general scene information such as surfaces, textures, colors, lighting sources and so on 3D video may be rendered according to one or more 3D video formats…information for global operations or effects in the scenes such as illumination, reflection, shadows, and simulated effects such as rain, fire, smoke, dust and fog…may be used in generating a modeled world for the scene and in rendering 2D representations of the world as video output…instead of generating 2D video output, 3D video may be rendered according to one or more 3D video format).
It would have been obvious to one of the ordinary skill in the art before the filing date of the instant application to have incorporated Olsen’s teaching of adjusting the reflection, shadow of the volumetric video object into Babu to have further enhanced Babu’s realistic rendering of the 3D video object in the physical environment. One of the ordinary skill in the art would have been motivated to have provided lighting sources, textures, illuminations, colors, reflections and shadows for rendering the 3D video object.
Re Claim 13:
The claim 13 encompasses the same scope of invention as that of the claim 12 except additional claim limitation that the rendered view of the volumetric video object is received in a packed texture in which it is packed together with a rendered view of a further volumetric video object and/or the at least one rendered view of the at least one shadow and/or the at least one rendered view of the at least one reflection.
However, Olsen et al. US-Patent No. 11,513,658 (hereinafter Olsen) further teaches the rendered view of the volumetric video object is received in a packed texture in which it is packed together with a rendered view of a further volumetric video object and/or the at least one rendered view of the at least one shadow and/or the at least one rendered view of the at least one reflection (Olsen teaches at column 10, lines 40-65 that the graphics data may be used in 3D rendering of video content…that is used to generate models of objects for the scene…general scene information such as surfaces, textures, colors, lighting sources and so on 3D video may be rendered according to one or more 3D video formats…information for global operations or effects in the scenes such as illumination, reflection, shadows, and simulated effects such as rain, fire, smoke, dust and fog…may be used in generating a modeled world for the scene and in rendering 2D representations of the world as video output…instead of generating 2D video output, 3D video may be rendered according to one or more 3D video format).
It would have been obvious to one of the ordinary skill in the art before the filing date of the instant application to have incorporated Olsen’s teaching of adjusting the reflection, shadow of the volumetric video object into Babu to have further enhanced Babu’s realistic rendering of the 3D video object in the physical environment. One of the ordinary skill in the art would have been motivated to have provided lighting sources, textures, illuminations, colors, reflections and shadows for rendering the 3D video object.
Re Claim 14:
The claim 14 encompasses the same scope of invention as that of the claim 13 except additional claim limitation that the receiving further comprises receiving metadata comprising unpack information indicating how to unpack the packed texture into the different rendered views.
Babu further teaches the claim limitation that the receiving further comprises receiving metadata comprising unpack information indicating how to unpack the packed texture into the different rendered views (Babu teaches at Paragraph 0030 that the data store may include one or more control data, one or more 3D models, one or more textures and one or more materials…a control stream may be preloaded into memory from one or more control data from the data store and at Paragraph 0031 that each animation stream of the one or more animation streams may correspond to at least one of a 3D model, a texture, or a material of the data store)
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Babu et al. US-PGPUB No. 2019/0199993 (hereinafter Babu) in view of Bathiche US-PGPUB No. 2012/0320169 (hereinafter Bathiche) and Liu et al. US-PGPUB No. 2020/0169715 (hereinafter Liu).
Re Claim 10:
The claim 31 encompasses the same scope of invention as that of the claim 1 except additional claim limitation that the determining step further comprises determining a lighting information indicating a lighting of the real-world environment, wherein the sending step further comprises sending the lighting information to the remote server, and wherein the rendered view of the volumetric video object has been rendered taking into account the lighting information.
Babu at least suggests the claim limitation that the determining step further comprises determining a lighting information indicating a lighting of the real-world environment, wherein the sending step further comprises sending the lighting information to the remote server, and wherein the rendered view of the volumetric video object has been rendered taking into account the lighting information (Babu teaches at Paragraph 0017 that a first depth information from the 3D video is added to a second depth information from a first location of the portion of the virtual and/or augmented environment volume space to a second location of a user viewing the 3D video and at Paragraph 0109 that one solution to this problem may include a video file format and player that includes mesh, e.g., 3D models and/or 3D animations and at Paragraph 0137 that some assets may include 3D models, mesh, animations, textures, shaders, and lights and at Paragraph 0138 that the data store 710 may include information 712 about the video, models 718, e.g., meshes of 3D models, material 720, texture 724 and control data 726).
Liu et al. US-PGPUB No. 2020/0169715 (hereinafter Liu) teaches the claim limitation that the determining step further comprises determining a lighting information indicating a lighting of the real-world environment, wherein the sending step further comprises sending the lighting information to the remote server, and wherein the rendered view of the volumetric video object has been rendered taking into account the lighting information (Liu teaches at Paragraph 0027 that the set of information values for respective voxels may change from one frame of the volumetric video to the next as objects move through the physical environment 140, as lighting or other environment conditions change and at Paragraph 0014 that the processing system may implement one or more manipulations of the presentation of the volumetric video to mitigate the obstruction of the view….may automatically make an occluding object transparent/hidden to keep the object of focus in view….some of the types of manipulations of the volumetric video may include color/’transparency modification such as hiding or making partially transparent an occluding object based on movement of the object…adjusting lighting conditions if another object were to shade/shadow an object of focus).
It would have been obvious to one of the ordinary skill in the art before the filing date of the instant application to have incorporated Liu’s teaching of modifying the information values of the volumetric video object based on the lighting information of the real-world environment to have sent the lighting information together with the pose of the user to the server for retrieving a rendered view of the volumetric video object taking into the lighting information. One of the ordinary skill in the art would have been motivated to have adjusted the property of the volumetric video object according to the lighting of the physical environment.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JIN CHENG WANG whose telephone number is (571)272-7665. The examiner can normally be reached Mon-Fri 8:00-5:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, King Poon can be reached at 571-270-0728. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JIN CHENG WANG/Primary Examiner, Art Unit 2617
Prosecution Insights