DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Allowable Subject Matter
Claims 7-10, and 17-20 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1 and 11 are rejected under 35 U.S.C. 103 as being unpatentable over Hog (Pub No. US 20210248754 A1) in view of Mobasher (Pub No. US 20180091634 A1).
As per claim 1, Hog teaches the claimed:
A computer system for rendering three-dimensional video, comprising ([0090] “Light field segmentation has a high computational complexity and using super-rays is a good strategy for decreasing it, similarly to temporal super-pixels for video segmentation”):
one or more processors; and one or more computer-storage media having stored thereon executable instructions that when executed by the one or more processors configure the computer system to perform at least the following ([0030] “The disclosure also concerns an information medium readable by a data processor and comprising instructions of a program as mentioned here above”):
receive virtual reality (VR) scene data for a first eye viewpoint ([009] “Note that equation (6) requires to interpolate the virtual views, which can be quite memory and time consuming. However, in our strategy we do not explicitly compute them but we use the super-rays depth d.sub.c to approximate ray colors in virtual views”
Claim 1 “A method for processing a light field comprising a plurality of views of a 3D scene, the method comprising: selecting a first pixel in a first view of the plurality of views”);
reproject at least a portion of the VR scene data from the first eye viewpoint to a second eye viewpoint ( Claim 1 “selecting a first pixel in a first view of the plurality of views; generating a first super-ray for the first pixel comprising: de-projecting the first pixel to identify a corresponding first point in the 3D scene; re-projecting the corresponding first point in the 3D scene on a second view of the plurality of views to identify a corresponding first pixel in the second view, wherein the second view is different from the first view”);
identify a set of individual pixels that are missing in the second eye viewpoint ( Claim 7 “selecting a second pixel in the first view of the plurality of views; generating a second super-ray for the second pixel comprising: de-projecting the second pixel to identify a corresponding second point in the 3D scene; re-projecting the corresponding second point in the 3D scene on a second view of the plurality of views to identify a corresponding second pixel in the second view, wherein the second view is different from the first view”
In this passage, Hog teaches that there are various points (first and second) in the second view where the second view is different from the first view. The system identifies that the second pixel is missing from the first view, thus generates the second pixel and reprojecting it onto the corresponding second point in the second view); and
Hog alone does not explicitly teach the remaining claim limitations.
However, Hog in combination with Mobasher teaches the claimed:
patch the set of individual pixels by sampling pixel colors adjacent to the set of individual pixels (Mobasher [0112] “For example, if pixel data of any of the pixels (or any color in the pixel) is missing, the receiver runs an averaging method to replace the missing pixel or color by averaging a group of 4 or 8 surrounding pixels (number of averaging pixel can be dynamic).”
In this passage, Mobasher teaches that if any pixel data is considered missing, the system patches the set of individual missing pixels by conducting a color averaging process. This process samples surrounding adjacent pixels and replaces the missing individua pixels with the average of these adjacent pixels.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use the color averaging method as taught by Mobasher with the system of Hog in order to replace any individual set of pixels. This allows the system to optimize the rendering process while achieving computational efficiency and visual accuracy.
As per claim 11, this claim is similar in scope to limitations recited in claim 1, and thus is rejected under the same rationale.
Claims 2-3, and 12-13 are rejected under 35 U.S.C. 103 as being unpatentable over Hog in view of Mobasher, in further view of Neill (Pub No. US 20140125650 A1).
As per claim 2, Hog and Mobasher alone do not explicitly teach the remaining claim limitations.
However, Hog in combination with Mobasher and Neill teaches the claimed:
The computer system of claim 1, wherein the VR scene data comprises a color data and depth data (Neill [0032] “In one embodiment, image frames and depth frames may include multiple samples for each pixel location corresponding to the screen surface of the display device 110 […] Each sample may be associated with various parameters, including, without limitation, pixel color, depth value, transparency, and the amount of the pixel area covered by the sample.” ).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to consider both color data and depth data as taught by Neill with the system of Hog as modified by Mobasher in order to produce a more accurately rendered VR scene.
As per claim 3, Hog and Mobasher alone do not explicitly teach the remaining claim limitations.
However, Hog in combination with Mobasher and Neill teaches the claimed:
The computer system of claim 1, wherein the executable instructions to identify the set of individual pixels that are missing in the second eye viewpoint include instructions that are executable to configure the computer system to (Hog Claim 12: “The electronic device of claim 10, wherein the instructions are further operative to cause the electronic device to: select a second pixel in the first view of the plurality of views; generate a second super-ray for the second pixel comprising: de-projecting the second pixel to identify a corresponding second point in the 3D scene; re-projecting the corresponding second point in the 3D scene on a second view of the plurality of views to identify a corresponding second pixel in the second view, wherein the second view is different from the first view;” ):
Hog and Mobasher alone do not explicitly teach the remaining claim limitations.
However, Hog in combination with Mobasher and Neill teaches the claimed:
track a disocclusion region during the reprojection of the VR scene data from the first eye viewpoint to the second eye viewpoint (Please see Neill figure 3, picture 330. Neill shows the right eye image frame with disocclusion. The disocclusion region is identified by the dark line.
This is further explained in Neill paragraph [0036] “The right eye image frame with disocclusion 330 illustrates a right eye image frame after reprojection and before filling the resulting disocclusion gaps. The disocclusion gaps appear as dark lines or bands in the right eye image frame with disocclusion 330. As described above, the disocclusion gaps result from portions of the right eye image frame that are visible to the right eye, but are not visible to the left eye as depicted in the left eye image frame 310”
In this passage and in figure 3, the left eye image frame corresponds to the first eye viewpoint, and the right eye image frame corresponds to the second eye viewpoint.); and
store disocclusion information within intermediate buffers, wherein the disocclusion information comprises a location of a nearest non-disocclusion pixel and a width of the disocclusion (Neill [0040] “Typically, this re-projection process results in disocclusion, because the right eye sees some portions of the scene that the left eye does not see. The pixels at these portions of the image frame for the right eye do not map to pixels in the image frame for the left eye, resulting in one or more holes or gaps in the image frame for the right eye. At step 408, the processing unit fills these disocclusion gaps using pixel information from a prior image frame for the right eye. This pixel information may be stored in a memory buffer. At step 410, the processing unit stores the left eye image frame received in step 402 in a memory buffer”
In this passage, Neill teaches that the pixel information of the left eye frame (first eye viewpoint) is stored in a memory buffer.
This is further explained in Neill [0043] “Holes or gaps in the reprojected image, caused by disocclusion, are filled by using pixel information from a prior image frame for the second eye, where the prior image frame is stored in memory buffer.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to identify and store the disocclusion region as taught by Neill with the system of Hog as modified by Mobasher in order to have a more accurate and smooth rendering of the second eye viewpoint based on the first eye viewpoint.
As per claims 12 and 13, these claims are similar in scope to limitations recited in claims 2 and 3 respectively, and thus are rejected under the same rationale.
Claims 4 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over Hog in view of Mobasher and Neill, in further view of Taylor (Pub No. US 20210142575 A1)
As per claim 4, Hog, Mobasher, and Neill alone do not explicitly teach the remaining claim limitations. However, Hog, Mobasher, Neill in combination with Taylor teaches the claimed:
The computer system of claim 3, wherein the executable instructions to patch the set of individual pixels by sampling pixel colors adjacent to the set of individual pixels include instructions that are executable to configure the computer system to (Taylor [0009] “Another aspect of the present disclosure includes a system comprising one or more processors and a non-transitory computer-readable media that includes instructions that, when executed by the one or more processors, cause the one or more processors to perform methods described herein”):
determine, using the disocclusion information within the intermediate buffers, that a target pixel within the second eye viewpoint comprises a disocclusion (Please see the explanation in claim 3 above); accumulate values of pixels within a kernel; and apply weights to the pixels (Taylor [0102] “At block 832, the AR device may apply a convolution filter to the two-dimensional coordinates of the display to fill in the gaps introduced from disocclusion (e.g., where occluded objects or portions thereof become visible), changes in pixel density, or the like. The convolution filter may use a convolution kernel to modify each pixel based on characteristics (e.g., depth, color, etc.) of neighboring pixels”
In this passage, Taylor teaches that a kernel is used in the disocclusion process. Instead of treating all surrounding pixels equally, the kernel changes shape or weight based on the directional gradient of the image edges. By comparing pixels and applying weights to each pixel, the kernal is able to predict hidden intensity values directly in the space.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use kernels as taught by Taylor with the system of Hog as modified by Mobasher and Neill in order to increase rending accuracy by identifying and focusing on a selective number of local neighboring pixels. Doing so allows the system to conduct a selective feature extraction. This also allows the system to segment disocclusions accurately and separate complex foreground-background boundaries from uniform background-background areas.
As per claim 14, this claim is similar in scope to limitations recited in claim 4, and thus is rejected under the same rationale.
Claims 5-6 and 15-16 are rejected under 35 U.S.C. 103 as being unpatentable over Hog in view of Mobasher and Neill, in further view of Jampani (Pub No. US 20240249422 A1)
As per claim 5, Hog, Mobasher, Neill, and Taylor alone do not explicitly teach the remaining claim limitations. However, Hog, Mobasher, Neill in combination with Taylor and Jampani teaches the claimed:
The computer system of claim 4, wherein the executable instructions to accumulate values of pixels within a kernel include instructions that are executable to configure the computer system to (Jampani [0046] “Processor 206 may be capable of executing program instructions 218 (e.g., compiled or non-compiled program logic and/or machine code) stored in data storage 208 to carry out the various functions described herein”):
determine, using depth pixel information within the VR scene data, that a pixel comprises a foreground pixel (Jampani [0055] “Foreground visibility map 310 may thus have n pixels, each of which may be associated with a corresponding visibility value (which may be referred to as a foreground visibility value)”
In this passage, Jampani teaches that the foreground pixels are identified and mapped, thus separating the foreground object from the background pixels.); and
exclude the foreground pixel from the accumulated values of pixels within the kernel (Jampani [0031] “The soft background disocclusion mask may be used by an inpainting model to inpaint parts of the monocular image and the depth image that could be disoccluded, thereby generating the background layer. In particular, the same inpainting model may be trained to inpaint the monocular image and the corresponding depth image, such that inpainted pixel values are based on both the color/intensity values of the monocular image and the depth values of the depth image. […] Accordingly, the inpainting model may be configured to inpaint disoccluded background regions by borrowing information from other background regions, rather than by borrowing information from foreground regions, resulting in a background layer that is visually consistent.”).
In this passage, Jampani discloses the usage of a background disocclusion mask, which excludes the unnecessary foreground pixels that are previously identified.
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to identify and exclude the foreground pixel as taught by Jampani with the system of Hog as modified by Mobasher and Neill in order to segment disocclusions accurately and separate complex foreground-background boundaries from uniform background-background areas. Doing so allows the system to accurately replace the missing pixels for edge preservation and produce the anti-blurring effect.
As per claim 6, Hog, Mobasher, Neill, and Taylor alone do not explicitly teach the remaining claim limitations. However, Hog, Mobasher, Neill in combination with Taylor and Jampani teaches the claimed:
The computer system of claim 4, wherein the executable instructions to accumulate values of pixels within a kernel include instructions that are executable to configure the computer system to (Taylor [0009] “Another aspect of the present disclosure includes a system comprising one or more processors and a non-transitory computer-readable media that includes instructions that, when executed by the one or more processors, cause the one or more processors to perform methods described herein):
determine, using depth pixel information within the VR scene data, that a pixel comprises a background pixel (Jampani [0031] “The soft background disocclusion mask may be used by an inpainting model to inpaint parts of the monocular image and the depth image that could be disoccluded, thereby generating the background layer. In particular, the same inpainting model may be trained to inpaint the monocular image and the corresponding depth image, such that inpainted pixel values are based on both the color/intensity values of the monocular image and the depth values of the depth image”
In this passage, Jampani further teaches that the background pixels are identified, which are separate from the foreground pixels as taught in the explanation in claim 5 above.); and
accumulate values of background pixels within the kernel (Taylor [0102] “At block 832, the AR device may apply a convolution filter to the two-dimensional coordinates of the display to fill in the gaps introduced from disocclusion (e.g., where occluded objects or portions thereof become visible), changes in pixel density, or the like. The convolution filter may use a convolution kernel to modify each pixel based on characteristics (e.g., depth, color, etc.) of neighboring pixels.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to determine and accumulate the background pixel values as taught by Jampani with the system of Hog as modified by Mobasher, Neill and Taylor in order to only pull the necessary neighboring pixels that will be used for filling in the missing pixels of the second eye viewpoint. Doing so ensures a more accurate edge preservation.
As per claims 15 and 16, these claims are similar in scope to limitations recited in claims 5 and 6 respectively, and thus is rejected under the same rationale.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SARAH Y. LEE whose telephone number is (571)272-8374. The examiner can normally be reached 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Daniel F. Hajnik can be reached at (571) 272-7642. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
SARAH Y. LEE
Examiner
Art Unit 2616
/DANIEL F HAJNIK/Supervisory Patent Examiner, Art Unit 2616
