DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Claim 14 is objected to because of the following informalities:
Claim 14 recites the limitation(s): “the first UV map usable to apply material” on PG(s). 6, Line(s) 3-4; examiner suggests amending this to “the first UV map is usable to apply material”;
Claim 14 recites the limitation(s): “the second UV map usable to apply material” on PG(s). 6, Line(s) 6-7; examiner suggests amending this to “the second UV map is usable to apply material”; and
Claim 14 recites the limitation(s): “the third UV map usable to displace vertices” on PG(s). 6, Line(s) 10-11; examiner suggests amending this to “the third UV map is usable to displace vertices”.Appropriate correction is required.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-6, 8, 12, 15-16, 18-19, and 26 are rejected under 35 U.S.C. 103 as being unpatentable over Castaneda et al. (US 20200221114 A1), hereinafter referenced as Castaneda, in view of Ollila (US 11463674 B1).
Regarding Claim 1, Castaneda discloses a computer-implemented method for providing stereoscopic volumetric video of a scene (Castaneda, [0112]: teaches a video encoding method for color and depth data representative of a virtual reality scene <read on stereoscopic volumetric video>), the method comprising:
receiving, by a processor, captured video data of the scene from a first imaging device and a second imaging device (Castaneda, FIG. 3 teaches capture devices 302 capturing a real-world scene 306 controlled by capture controller 304 <read on receiving captured video data of scene>; [0043]: teaches the image capture system 202 including "a plurality of capture devices (e.g., video cameras <read on first and second imaging devices>, depth imaging devices, etc.) configured to capture images for processing and distribution by image capture system 202"),
PNG
media_image1.png
406
561
media_image1.png
Greyscale
the first imaging device and the second imaging device being synchronized to capture data at the same time [[and being positioned to provide a first overlapping field of view that includes the scene]] (Castaneda, [0046]: teaches capture controller 304 receiving "images <read on capture data> captured by each of capture devices 302 <read on first and second imaging devices> and may manage (e.g., buffer, aggregate, synchronize <read on first and second imaging devices being synchronized to capture data simultaneously>, etc.) the images to prepare image sequences that may be provided to downstream systems in the pipeline," such as to a scene reconstruction system 204);
receiving, by the processor, captured depth data of the scene from a third imaging device and a fourth imaging device (Castaneda, [0043]: teaches the image capture system 202 including "a plurality of capture devices (e.g., video cameras, depth imaging devices <read on third and fourth imaging devices>, etc.) configured to capture images for processing and distribution by image capture system 202"; [0045]: teaches each capture device 302 being configured "to capture both color data and depth data <read on receiving captured depth data>, or may include separate devices for capturing these different types of data"; [0046]: teaches the capture controller 304 receiving images captured by each of capture devices 302; Note: it should be noted that it is being interpreted that each capture device captures both color and depth data),
the third imaging device and the fourth imaging device being synchronized to capture data at the same time [[and being positioned to provide a second overlapping field of view that includes the scene]] (Castaneda, [0046]: teaches capture controller 304 receiving "images <read on capture data> captured by each of capture devices 302 <read on third and fourth imaging devices> and may manage (e.g., buffer, aggregate, synchronize <read on third and fourth imaging devices being synchronized to capture data simultaneously>, etc.) the images to prepare image sequences that may be provided to downstream systems in the pipeline," such as to a scene reconstruction system 204);
combining, by the processor, the captured video data and the captured depth data to generate a first atlas frame sequence comprising multiple atlas frames (Castaneda, [0070]: teaches the generation of a full atlas frame sequence 220 <read on first atlas frame sequence>, which includes a plurality of full atlas frames 602 <read on multiple atlas frames>, where "each full atlas frame 602 in full atlas frame sequence 220 is shown to include a plurality of images 604 (e.g., images 604-1 through 604-20)" as shown in FIG. 6; [0070]: further teaches "images 604 may include various color data images depicted from various vantage points 502 and/or 504, various depth data images depicted from the same or additional vantage points 502 and/or 504, and/or a combination of both color data images and depth data images <read on combining captured video and depth data>"), wherein
PNG
media_image2.png
405
596
media_image2.png
Greyscale
each atlas frame of the first atlas frame sequence includes the captured video data and the captured depth data for a given synchronized capture time (Castaneda, [0075]: teaches image sequences 700-C and 700-D being displayed along a common timeline 704, where each color data image and depth data image in the image sequences are synchronized <read on given synchronized capture time> with an image in the opposite image sequence as shown in FIG. 7);
PNG
media_image3.png
408
499
media_image3.png
Greyscale
processing, by the processor, each atlas frame of the first atlas frame sequence to generate a reconstructed scene in a virtual environment (Castaneda, [0049]: teaches a reconstruction system being configured "to provide the image sequences to video encoders 210 by way of atlas selectors 206 <read on processing each atlas frame of first atlas frame sequence>," which generates a volumetric model <read on generating reconstructed scene in virtual environment>);
capturing, by the processor, each frame of the reconstructed scene using a virtual imaging device to generate a second atlas frame sequence comprising multiple atlas frames (Castaneda, [0062]: teaches perspective vantage points 502-9 through 502-11 <read on virtual imaging device> not corresponding to any physical capture devices used to capture real-world scene 306, where the perspective vantage points are used to render unique views of a virtual object; [0078]: teaches atlas selectors 206-1 receiving a "full atlas frame sequence 220 <read on capturing each frame of reconstructed scene> and to select particular combination subsets of image sequences from full atlas frame sequence 220 that may be desirable to send different media player devices 214, which each may be providing virtual reality experiences in different parts of virtual reality scene 506"; [0079]: teaches a partial atlas frame sequence <read on generate second atlas frame sequence> including a subset of partial atlas frames <read on multiple atlas frames> selected from full atlas frame sequence 220 as shown in FIG. 8; Note: it should be noted that paragraph [0119] of the specification defines the virtual imaging device as "one or more synthetic/virtual cameras located in the virtual environment"), wherein
PNG
media_image4.png
298
456
media_image4.png
Greyscale
each atlas frame of the second atlas frame sequence includes virtual video data and virtual depth data of the reconstructed scene (Castaneda, [0079]: teaches a partial atlas frame sequence 222 including a plurality of partial atlas frames 802 <read on atlas frame of second atlas frame sequence>, color data images <read on virtual video data>, and depth data images <read on virtual depth data>); and
providing, by the processor, the stereoscopic volumetric video of the scene based on the virtual video data and the virtual depth data (Castaneda, [0109]: teaches a virtual reality experience 1200 <read on stereoscopic volumetric video> being presented to user 216 "from a dynamically selectable arbitrary experience location <read on virtual video and depth data> within virtual reality scene 506").
However, Castaneda does not expressly disclose
the first imaging device and the second imaging device being synchronized to capture data at the same time and being positioned to provide a first overlapping field of view that includes the scene; and
the third imaging device and the fourth imaging device being synchronized to capture data at the same time and being positioned to provide a second overlapping field of view that includes the scene.
Ollila discloses
the first imaging device and the second imaging device being synchronized to capture data at the same time and being positioned to provide a first overlapping field of view that includes the scene (Ollila, [Col. 10, Line 67 - Col. 11, Lines 1-5]: teaches a first camera having a wide field of view <read on first overlapping field of view> of a real-world environment); and
the third imaging device and the fourth imaging device being synchronized to capture data at the same time and being positioned to provide a second overlapping field of view that includes the scene (Ollila, [Col. 10, Line 67 - Col.11, Lines 1-9]: teaches the first camera having a wide field of view and the second cameras having narrow field of views <read on second overlapping field of view> of the real-world environment>, which can eliminate auto focus).
Ollila is analogous art with respect to Castaneda because they are from the same field of endeavor, namely capturing real-world environments with a plurality of cameras. Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to have cameras that have a wider field of view than other cameras as taught by Ollila into the teaching of Castaneda. The suggestion for doing so would allow the system to determine objects of interest without requiring autofocusing, thereby streamlining the overall video-capture process. Therefore, it would have been obvious to combine Ollila with Castaneda.
Regarding Claim 15, it recites the limitations that are similar in scope to Claim 1, but in a system. As shown in the rejection, the combination of Castaneda and Ollila discloses the limitations of Claim 1. Additionally, Castaneda discloses a system for providing stereoscopic volumetric video of a scene (Castaneda, [0049]: teaches an image capture system 202 providing "image data 218 to scene reconstruction system 204" to generate a virtual reality experience <read on stereoscopic volumetric video>), the system comprising:
a processor in communication with a first imaging device (Castaneda, FIG. 3 teaches a capture controller 304 <read on processor> that controls cameras 302 <read on first imaging device>),
a second imaging device (Castaneda, FIG. 3 teaches a capture controller 304 <read on processor> that controls cameras 302 <read on second imaging device>),
a third imaging device (Castaneda, FIG. 3 teaches a capture controller 304 <read on processor> that controls cameras 302 <read on third imaging device>) and
a fourth imaging device (Castaneda, FIG. 3 teaches a capture controller 304 <read on processor> that controls cameras 302 <read on fourth imaging device>), wherein:…
Thus, Claim 15 is met by Castaneda according to the mapping presented in the rejection of Claim 1, given the computer-implemented method corresponds to a system.
Regarding Claims 2 and 16, the combination of Castaneda and Ollila discloses the method and the system of Claims 1 and 15 respectively. Additionally, Castaneda further discloses wherein
an imaging plane of the virtual imaging device is shifted with respect to the reconstructed scene to capture a portion of the reconstructed scene at a higher image resolution compared with the other portions of the reconstructed scene (Castaneda, [0060]: teaches vantage points 502 being movable in various ways (e.g., rotating, sliding, panning, instantaneously hopping, etc.) with respect to the virtual reality scene, such as adjusting a resolution <read on higher resolution compared to other portions of reconstructed scene> when zooming in <read on shifting with respect to reconstructed scene>; [0061]: teaches vantage points being disposed along two dimensions associated with the virtual reality scene 506 (e.g., along a plane <read on imaging plane of virtual imaging device> such as the ground); [0062]: teaches perspective vantage points 502-9 through 502-11 being configured to provide better view of certain parts of the virtual reality scene 506 (e.g., the immediate vicinity of virtual object 508) <read on capture portion of reconstructed scene>).
Regarding Claim 3, the combination of Castaneda and Ollila discloses the method and the system of Claims 1 and 15 respectively. Additionally, Castaneda further discloses wherein the method is performed to
provide the stereoscopic volumetric video of the scene to a user device in real-time (Castaneda, [0029]: teaches real-time encoding of video data <read on provide stereoscopic volumetric video> to allow virtual reality users <read on user device> to experience virtual events; [0028]: teaches the video encoding system 100 used for color and depth data that is representative of a virtual reality scene).
Regarding Claims 4 and 18, the combination of Castaneda and Ollila discloses the method and the system of Claims 1 and 15 respectively. Additionally, Castaneda further discloses wherein
the first imaging device and the second imaging device are high-resolution color video cameras (Castaneda, [0043]: teaches the image capture system using a plurality of capture devices, such as video cameras imaging devices <read on high-resolution color cameras>, which captures color data; Note: it should be noted that although not expressly stated, it is common in the art to use high-resolution cameras for an optimal viewing experience).
Regarding Claims 5 and 19, the combination of Castaneda and Ollila discloses the method and the system of Claims 1 and 15 respectively. Additionally, Castaneda further discloses wherein
the third imaging device and the fourth imaging device are infrared depth-sensing cameras (Castaneda, [0043]: teaches the image capture system using a plurality of capture devices, such as depth imaging devices <read on infrared depth-sensing cameras>, which captures depth data; Note: it should be noted that although not expressly stated, it is common in the art for depth-sensing cameras to use infrared light, such as near-infrared (NIR) light).
Regarding Claim 6, the combination of Castaneda and Ollila discloses the method and the system of Claims 1 and 15 respectively. Castaneda does not expressly disclose the limitations of Claim 6; however, Ollila discloses wherein
the second overlapping field of view is smaller than the first overlapping field of view (Ollila, [Col. 10, Line 67 - Col.11, Lines 1-9]: teaches the first camera having a wide field of view and the second cameras having narrow field of views <read on second overlapping field of view being smaller than first overlapping field of view> due to using different focal length lenses).
Ollila is analogous art with respect to Castaneda because they are from the same field of endeavor, namely capturing real-world environments with a plurality of cameras. Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to have cameras that have a wider field of view than other cameras as taught by Ollila into the teaching of Castaneda. The suggestion for doing so would allow the system to determine objects of interest without requiring autofocusing, thereby streamlining the overall video-capture process. Therefore, it would have been obvious to combine Ollila with Castaneda.
Regarding Claim 8, the combination of Castaneda and Ollila discloses the method of Claim 1. Additionally, Castaneda further discloses wherein, before processing each atlas frame of the first atlas frame sequence to generate the reconstructed scene, the method further comprises
editing the first atlas frame sequence to select only a portion of the first atlas frame sequence for processing (Castaneda, [0078]: teaches atlas selectors 206-1 being employed within image generation system 208 "to receive full atlas frame sequence 220 and to select particular combination subsets of image sequences from full atlas frame sequence 220 <read on select only a portion of first atlas frame sequence for processing> that may be desirable <read on editing first atlas frame sequence> to send different media player devices 214, which each may be providing virtual reality experiences in different parts of virtual reality scene 506").
Regarding Claims 12 and 26, the combination of Castaneda and Ollila discloses the method and the system of Claims 1 and 15 respectively. Additionally, Castaneda further discloses wherein capturing each frame of the reconstructed scene using a virtual imaging device to generate a second atlas frame sequence comprises:
capturing first virtual video data in a first capture pass corresponding to a left eye perspective of a viewer of the stereoscopic volumetric video (Castaneda, [0062]: teaches perspective vantage points 502-9 through 502-11 <read on virtual imaging device> not corresponding to any physical capture devices used to capture real-world scene 306, where the perspective vantage points are used to render unique views of a virtual object; [0078]: teaches atlas selectors 206-1 receiving a full atlas frame sequence 220 and to select particular combination subsets of image sequences from full atlas frame sequence 220 that may be desirable to send different media player devices 214, where the different partial atlas frame sequences 222 are supplied to each video encoder <read on capturing first virtual video data in first capture pass>; [0079]: teaches a partial atlas frame sequence <read on generate second atlas frame sequence> including a subset of partial atlas frames selected from full atlas frame sequence 220 as shown in FIG. 8; FIG. 12A teaches a media player device, where images of the scene are displayed for each eye <read on left eye perspective of viewer>);
PNG
media_image5.png
363
201
media_image5.png
Greyscale
capturing second virtual video data in a second capture pass corresponding to a right eye perspective of a viewer of the stereoscopic volumetric video (Castaneda, [0062]: teaches perspective vantage points 502-9 through 502-11 <read on virtual imaging device> not corresponding to any physical capture devices used to capture real-world scene 306, where the perspective vantage points are used to render unique views of a virtual object; [0078]: teaches atlas selectors 206-1 receiving a full atlas frame sequence 220 and to select particular combination subsets of image sequences from full atlas frame sequence 220 that may be desirable to send different media player devices 214, where the different partial atlas frame sequences 222 are supplied to each video encoder <read on capturing second virtual video data in second capture pass>; FIG. 12A teaches a media player device, where images of the scene are displayed for each eye <read on right eye perspective of viewer>); and
capturing the virtual depth data of the reconstructed scene in relation to a virtual location of the virtual imaging device in the virtual environment (Castaneda, FIG. 5 teaches perspective vantage points 502-9 to 502-11 <read on virtual location of virtual imaging device> capturing virtual object 508, which includes both color and depth data of said virtual object <read on capturing virtual depth data>).
Claim 11 is rejected under 35 U.S.C. 103 as being unpatentable over Castaneda et al. (US 20200221114 A1), hereinafter referenced as Castaneda, in view of Ollila (US 11463674 B1) as applied to Claim 2 above respectively, and further in view of Wen et al. (US 20190253638 A1), hereinafter referenced as Wen.
Regarding Claim 11, the combination of Castaneda and Ollila discloses the method of Claim 1. The combination of Castaneda and Ollila does not expressly disclose the limitations of Claim 11; however, Wen discloses wherein the portion of the reconstructed scene captured at the higher image resolution includes
a face portion of a subject (Wen, [0065]: teaches the camera recognizing a human face <read on face portion of subject>).
Wen is analogous art with respect to Castaneda, in view of Ollila because they are from the same field of endeavor, namely capturing volumetric geometry. Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to implement an infrared pattern projector to capture real-world objects and/or shapes as taught by Wen into the teaching of Castaneda, in view of Ollila. The suggestion for doing so would the system to determine and recognize objects, thereby allowing the system to focus on it, allowing for a high resolution representation of the said object. Therefore, it would have been obvious to combine Wen with Castaneda, in view of Ollila.
Claims 7, 10, 13, and 24 are rejected under 35 U.S.C. 103 as being unpatentable over Castaneda et al. (US 20200221114 A1), hereinafter referenced as Castaneda, in view of Ollila (US 11463674 B1) as applied to Claims 1 and 15 above respectively, and further in view of George et al. (US 20220215628 A1), hereinafter referenced as George.
Regarding Claim 7, the combination of Castaneda and Ollila discloses the method of Claim 1. The combination of Castaneda and Ollila does not expressly disclose the limitations of Claim 7; however, George discloses wherein
the captured depth data is encoded using a hue saturation luminance (HSL) scale (George, [0115]: teaches a process of calculating contributions of each perspective, where a Hue, Saturation, Value (HSV) conversion <read on using HSL scale> is used to improve the range of a depth sample of a given frame <read on captured depth data>), wherein
a hue value of the HSL scale includes depth information of the scene (George, [0142]: teaches a hue-encoded depth map of an interactive 3D scene) and
a luminance value of the HSL scale includes mask information of the scene (George, [0072]: teaches color information containing intensity values <read on luminance value> for each pixel; [0074]: teaches a segmentation stream being used as a refinement mask <read on mask information>, "where each pixel in a respective frame indicates whether the pixel depicts a background (e.g., pixel value==0) or a foreground object (e.g., pixel value==1)").
George is analogous art with respect to Castaneda, in view of Ollila because they are from the same field of endeavor, namely capturing volumetric video data. Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to obtain segmentation streams from an input video stream as taught by George into the teaching of Castaneda, in view of Ollila. The suggestion for doing so would allow the system to obtain additional information, such as color and depth information about objects within a scene, thereby providing multiple accurate different perspectives of virtual objects. Therefore, it would have been obvious to combine George with Castaneda, in view of Ollila.
Regarding Claims 10 and 24, the combination of Castaneda and Ollila discloses the method and the system of Claims 1 and 15 respectively. Additionally, Castaneda further discloses
[[receiving, by the processor, intrinsics data and extrinsics data of each imaging device; and wherein]]
[[processing each atlas frame of the first atlas frame sequence to generate the reconstructed scene comprises:]]reconstructing a frame geometry based on the captured depth data, [[the intrinsics data of the third imaging device and the fourth imaging device, and the extrinsics data of the third imaging device and the fourth imaging device]] (Castaneda, [0049]: teaches a reconstruction system, which includes a volumetric modeling system, being configured "to provide the image sequences to video encoders 210 by way of atlas selectors 206 <read on processing each atlas frame of first atlas frame sequence>," which generates a volumetric model <read on generating reconstructed scene in virtual environment>; [0054]: teaches a plurality of vantage points of the volumetric model, where each vantage point captures a color data image sequence and a depth data image sequence <read on captured depth data> for the volumetric modeling system to render <read on reconstructing frame geometry>); and
projecting the captured video data onto the frame geometry using a first pass corresponding to video data captured by the first imaging device [[and based on the intrinsics data and the extrinsics data of the first imaging device]], and a second pass corresponding to video data captured by the second imaging device [[and based on the intrinsics data and the extrinsics data of the second imaging device]] (Castaneda, [0063]: teaches an orthographic projection of color data for surfaces within a particular sub-volume <read on projecting captured video data onto frame geometry> from an orthographic vantage point <read on first imaging device>, where the color data is captured from processed image sequences captured by physical capture devices 302; [0088]: teaches the video encoder performing a first pass encoding and a second pass encoding of color data images <read on video data captured by first and second imaging devices>).
However, the combination of Castaneda and Ollila does not expressly disclose
receiving, by the processor, intrinsics data and extrinsics data of each imaging device; and wherein
processing each atlas frame of the first atlas frame sequence to generate the reconstructed scene comprises: reconstructing a frame geometry based on the captured depth data, the intrinsics data of the third imaging device and the fourth imaging device, and the extrinsics data of the third imaging device and the fourth imaging device; and
projecting the captured video data onto the frame geometry using a first pass corresponding to video data captured by the first imaging device and based on the intrinsics data and the extrinsics data of the first imaging device, and a second pass corresponding to video data captured by the second imaging device and based on the intrinsics data and the extrinsics data of the second imaging device.
George discloses
receiving, by the processor, intrinsics data and extrinsics data of each imaging device (George, [0130]: teaches a refinement system 430 receiving "a depth stream and a corresponding color stream captured by a camera system (e.g., a video camera and depth sensor), and calibration information corresponding to the camera system, including for example, intrinsic calibration information <read on intrinsics data> relating to the depth sensor lens, intrinsic calibration information relating to the video camera lens, and extrinsic calibration information <read on extrinsics data> relating to a depth to color pose"); and wherein
processing each atlas frame of the first atlas frame sequence to generate the reconstructed scene comprises: reconstructing a frame geometry based on the captured depth data, the intrinsics data of the third imaging device and the fourth imaging device, and the extrinsics data of the third imaging device and the fourth imaging device (George, [0153]: teaches the refinement system receiving a set of parameters, where it includes calibration information, which further includes "intrinsic calibration information relating to the depth sensor lens <read on intrinsics data of third and fourth imaging devices>, intrinsic calibration information relating to the video camera lens, and extrinsic calibration information relating to a depth to color pose <read on extrinsics data of third and fourth imaging devices>"); and
projecting the captured video data onto the frame geometry using a first pass corresponding to video data captured by the first imaging device and based on the intrinsics data and the extrinsics data of the first imaging device, and a second pass corresponding to video data captured by the second imaging device and based on the intrinsics data and the extrinsics data of the second imaging device (George, [0153]: teaches the refinement system receiving a set of parameters, where it includes calibration information, which further includes "intrinsic calibration information relating to the depth sensor lens, intrinsic calibration information relating to the video camera lens <read on intrinsics data of first and second imaging devices>, and extrinsic calibration information relating to a depth to color pose <read on extrinsics data of first and second imaging devices>").
George is analogous art with respect to Castaneda, in view of Ollila because they are from the same field of endeavor, namely capturing volumetric video data. Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to obtain segmentation streams from an input video stream as taught by George into the teaching of Castaneda, in view of Ollila. The suggestion for doing so would allow the system to obtain additional information, such as color and depth information about objects within a scene, thereby providing multiple accurate different perspectives of virtual objects. Therefore, it would have been obvious to combine George with Castaneda, in view of Ollila.
Regarding Claim 13, the combination of Castaneda and Ollila discloses the method of Claim 1. The combination of Castaneda and Ollila does not expressly disclose the limitations of Claim 13; however, George discloses wherein
the virtual depth data is encoded using a hue saturation luminance (HSL) scale (George, [0115]: teaches a process of calculating contributions of each perspective, where a Hue, Saturation, Value (HSV) conversion <read on using HSL scale> is used to improve the range of a depth sample of a given frame; [0122]: teaches a deferred surface reconstruction engine receiving a viewing position that indicates a virtual camera position of a viewer in relation to the virtual content object being rendered as input <read on virtual depth data>), wherein
a hue value of the HSL scale includes virtual depth information of the reconstructed scene in relation to the virtual location (George, [0142]: teaches a hue-encoded depth map of an interactive 3D scene from customizable inspection views <read on virtual depth information in relation to virtual location>) and
a luminance value of the HSL scale includes mask information of the reconstructed scene (George, [0072]: teaches color information containing intensity values <read on luminance value> for each pixel; [0074]: teaches a segmentation stream being used as a refinement mask <read on mask information of reconstructed scene>, "where each pixel in a respective frame indicates whether the pixel depicts a background (e.g., pixel value==0) or a foreground object (e.g., pixel value==1)").
George is analogous art with respect to Castaneda, in view of Ollila because they are from the same field of endeavor, namely capturing volumetric video data. Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to obtain segmentation streams from an input video stream as taught by George into the teaching of Castaneda, in view of Ollila. The suggestion for doing so would allow the system to obtain additional information, such as color and depth information about objects within a scene, thereby providing multiple accurate different perspectives of virtual objects. Therefore, it would have been obvious to combine George with Castaneda, in view of Ollila.
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over Castaneda et al. (US 20200221114 A1), hereinafter referenced as Castaneda, in view of Ollila (US 11463674 B1) as applied to Claim 8 above respectively, and further in view of Pesonen (US 20190116352 A1).
Regarding Claim 9, the combination of Castaneda and Ollila discloses the method of Claim 8. Additionally, Castaneda further discloses wherein editing the first atlas frame sequence to select only a portion of the first atlas frame sequence for processing comprises:
[[transcoding the first atlas frame sequence into a proxy sequence, wherein]]
[[the proxy sequence corresponds to a smaller file size compared with the first atlas frame sequence;]]
[[using the proxy sequence to make one or more selections; and]]
editing the first atlas frame sequence to correspond to the one or more selections (Castaneda, [0078]: teaches atlas selectors 206-1 being employed within image generation system 208 "to receive full atlas frame sequence 220 and to select particular combination subsets of image sequences from full atlas frame sequence 220 <read on select only a portion of first atlas frame sequence for processing> that may be desirable <read on editing first atlas frame sequence> to send different media player devices 214, which each may be providing virtual reality experiences in different parts of virtual reality scene 506").
However, the combination of Castaneda and Ollila does not expressly disclose
transcoding the first atlas frame sequence into a proxy sequence, wherein
the proxy sequence corresponds to a smaller file size compared with the first atlas frame sequence; and
using the proxy sequence to make one or more selections.
Pesonen discloses
transcoding the first atlas frame sequence into a proxy sequence (Pesonen, [0100]: teaches packing a texture atlas <read on transcoding first atlas frame sequence> more compactly into blocks <read on proxy sequence>; [0099]: teaches a modifiable video compression level that is dependent on if the block is empty, full, or partially full), wherein
the proxy sequence corresponds to a smaller file size compared with the first atlas frame sequence (Pesonen, [0100]: teaches packing a texture atlas <read on transcoding first atlas frame sequence> more compactly into blocks <read on proxy sequence>; Note: it should be noted that although not expressly stated, it is common knowledge in the art that compression leads to smaller file sizes compared to the input file size); and
using the proxy sequence to make one or more selections (Pesonen, [0100]: teaches using empty blocks for packing a texture atlas; [0105]: teaches the rendering process using <read on making selections> two rendering lists: "one for rendering partially full blocks and one for rendering full blocks," where "these rendering lists are per view as each view can have different rendering parameters (projection type, depth min/max values etc.)").
Pesonen is analogous art with respect to Castaneda, in view of Ollila because they are from the same field of endeavor, namely processing volumetric 3D data. Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to compress texture atlas data into blocks as taught by Pesonen into the teaching of Castaneda, in view of Ollila. The suggestion for doing so would allow the system to dynamically modify video compression detail based on the block being empty, full, or partially full, thereby resulting in a more efficient video processing pipeline. Therefore, it would have been obvious to combine Pesonen with Castaneda, in view of Ollila.
Claim 14 is rejected under 35 U.S.C. 103 as being unpatentable over Castaneda et al. (US 20200221114 A1), hereinafter referenced as Castaneda, in view of Ollila (US 11463674 B1) as applied to Claim 12 above respectively, and further in view of Lange (US 20050117215 A1).
Regarding Claim 14, the combination of Castaneda and Ollila discloses the method of Claim 12. Additionally, Castaneda further discloses wherein providing the stereoscopic volumetric video of the scene based on the virtual video data and the virtual depth data comprises:
[[generating a first UV map corresponding to the first virtual video data,]]
[[the first UV map usable to apply material to an output mesh to generate rendering corresponding to the left eye perspective of the viewer of the stereoscopic volumetric video;]]
[[generating a second UV map corresponding to the second virtual video data,]]
[[the second UV map usable to apply material to the output mesh to generate rendering corresponding to the right eye perspective of the viewer of the stereoscopic volumetric video; and]]
generating a third UV map corresponding to the virtual depth data (Castaneda, [0071]: teaches atlas frames being referred to as texture atlases <read on third UV map>, which combines a plurality of images (e.g., also referred to as atlas tiles, patches, sprites, etc.) that have certain attributes in common; [0072]: teaches image sequences included in full atlas frame sequence 220 being a particular color data image sequence and a particular depth data image sequence <read on virtual depth data>),
the third UV map usable to displace vertices of the output mesh to recreate geometry of the scene (Castaneda, [0100]: teaches using motion vectors as image blocks that represent an object <read on output mesh> that is moving from one image to the next in the image sequence; [0103]: teaches "content represented by each video stream in a particular virtual reality dataset may be rendered only in part along with content represented by other video streams, or the content may be combined and/or otherwise processed so as to recreate a 3D virtual reality scene <read on recreate geometry>").
However, the combination of Castaneda and Ollila does not expressly disclose
generating a first UV map corresponding to the first virtual video data,
the first UV map usable to apply material to an output mesh to generate rendering corresponding to the left eye perspective of the viewer of the stereoscopic volumetric video; and
generating a second UV map corresponding to the second virtual video data,
the second UV map usable to apply material to the output mesh to generate rendering corresponding to the right eye perspective of the viewer of the stereoscopic volumetric video.
Lange discloses
generating a first UV map corresponding to the first virtual video data (Lange, [0156]: teaches left and right stereo imagery being composed and processed, where left and right images are decomposed into a pair of left and right texture maps <read on generating first UV map> of a virtual scene <read on first virtual video data>; the left texture map is being interpreted as the first UV map),
the first UV map usable to apply material to an output mesh to generate rendering corresponding to the left eye perspective of the viewer of the stereoscopic volumetric video (Lange, [0156]: teaches left and right stereo imagery being composed and processed, where left and right images are decomposed into a pair of left and right texture maps mapped to the surface of a stereo-recorded object <read on apply material to output mesh> in a virtual scene; [0157]: teaches stereo rendering <read on generate rendering corresponding to left eye>; Note: it should be noted that although not expressly stated, it is common in the art to apply a material when texture mapping geometry, such as bump mapping); and
generating a second UV map corresponding to the second virtual video data (Lange, [0156]: teaches left and right stereo imagery being composed and processed, where left and right images are decomposed into a pair of left and right texture maps <read on generating second UV map> of a virtual scene <read on second virtual video data>; the right texture map is being interpreted as the second UV map),
the second UV map usable to apply material to the output mesh to generate rendering corresponding to the right eye perspective of the viewer of the stereoscopic volumetric video (Lange, [0156]: teaches left and right stereo imagery being composed and processed, where left and right images are decomposed into a pair of left and right texture maps <read on apply material to output mesh> of a virtual scene; [0157]: teaches stereo rendering <read on generate rendering corresponding to right eye>).
Lange is analogous art with respect to Castaneda, in view of Ollila because they are from the same field of endeavor, namely processing volumetric 3D data. Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to implement stereo rendering for pairs of images that correspond to each eye of a user as taught by Lange into the teaching of Castaneda, in view of Ollila. The suggestion for doing so would allow the system to render stereoscopic virtual objects using obtained color and depth information, thereby yielding predictable results. Therefore, it would have been obvious to combine Lange with Castaneda, in view of Ollila.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Lev (US 20230222761 A1) discloses a plurality of sensors with mostly overlapping field of views that simultaneously acquire temporary images;
Aflaki Beni et al. (US 20210144404 A1) discloses a method for volumetric video encoding and decoding;
Mekuria (US 20200202608 A1) discloses receiving volumetric video including a first video track that carries geometry information and a second video track carrying occupancy information;
Juang et al. (US 20170310945 A1) discloses compressing geometric data and video for volumetric video;
Gotsman et al. (US 20090109280 A1) discloses remotely viewing a video from selected viewpoints; and
Pang et al. (US 20170244948 A1) discloses displaying a viewpoint from a viewpoint.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KARL TRUONG whose telephone number is (703)756-5915. The examiner can normally be reached 7:30 AM - 5:00 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kent Chang can be reached at (571) 272-7667. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/K.D.T./Examiner, Art Unit 2614
/KENT W CHANG/Supervisory Patent Examiner, Art Unit 2614