DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 1-8 and 18-20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph for lack of antecedent basis.
In regards to independent claim 1, the limitation recites “the basis of the classification result” in lines 3-4, in which no previous instance of “a basis of a classification result” has been provided, and thus there is insufficient antecedent basis for this limitation in the claim.
In regards to dependent claim 8, the limitation recites “the type” in line 2, in which no previous instance of “a type” has been provided, and thus there is insufficient antecedent basis for this limitation in the claim.
In regards to dependent claim 18, the limitation recites “on the basis of” in line 2, in which no previous instance of “a basis of” has been provided, and thus there is insufficient antecedent basis for this limitation in the claim.
In regards to independent claim 20, the limitation recites “the basis of the classification result” in lines 4-5, in which no previous instance of “a basis of a classification result” has been provided, and thus there is insufficient antecedent basis for this limitation in the claim.
In regards to dependent claims 2-7 and 19, these claims depend from a rejected base claims, and thus there is insufficient antecedent basis for the limitations in this claim.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claim 20 is rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter.
Claim 20 describes a “computer-readable recording medium”.
Further, Applicant's specification, fails to explicitly define the scope of “computer-readable recording medium”. Thus, in giving the term its plain meaning (see MPEP 2111.01), the claimed “computer-readable recording medium” is considered to include data signals per se. Data signals per se are not statutory as they fail to fall into one of the four statutory categories of invention.
As an additional note, a non-transitory computer-readable medium having executable programming instructions stored thereon is considered statutory as non-transitory computer readable media excludes transitory data signals.
Allowable Subject Matter
Claims 6, 14, 18, and 19 are objected to as being dependent upon a rejected base claim, but would be allowable if the 35 U.S.C. 112(b) rejections are resolved and the claims are rewritten in independent form including all of the limitations of the base claim and any intervening claims. The following is a statement of reasons for the indication of allowable subject matter:
In regards to dependent claim 6, none of the cited prior art alone or in combination provides motivation to teach “wherein the information on the spherical harmonic function further includes information on the coefficients for the spherical harmonic function if the information on the spherical harmonic function used in the previous frame is not reused” as the references only teach encoding and decoding of multi-view images where spherical harmonic functions are used including determined coefficients, however the references fail to explicitly disclose identifying data that delineates and unused previous frame, in conjunction with the features of claim 5 with which it depends regarding the function having information that indicates whether to reuse information on a spherical harmonic function used in a previous frame.
In addition, there is no teaching, suggestion, or motivation found in the current references and none that can be inferred from the examiner’s own knowledge with respect to the current limitation.
In regards to dependent claim 14, this claim recites limitations similar in scope to claim 6, and thus is objected to based on the same rationale as provided above.
In regards to dependent claim 18, none of the cited prior art alone or in combination provides motivation to teach “wherein the relighting effect is applied on the basis of an irradiance component obtained by a spherical harmonic function and diffused light obtained based on an albedo of the surface of the heterogeneous object” as the references only teach as the references only teach encoding and decoding of multi-view images where spherical harmonic functions are used including determined coefficients as well as the ability to apply relighting effects to image, however the references fail to explicitly disclose the relighting effect in consideration of irradiance and albedo for a heterogenous object through application of a spherical harmonic function, in conjunction with the features of claim 10 with which it depends regarding information indicating order or the coefficients of the spherical harmonic function.
In addition, there is no teaching, suggestion, or motivation found in the current references and none that can be inferred from the examiner’s own knowledge with respect to the current limitation.
In regards to dependent claims 19, this claims depends from an objected to base claim 18, and thus is objected to based on the same rationale as provided above.
As allowable subject matter has been indicated, applicant's reply must either comply with all formal requirements or specifically traverse each requirement not complied with. See 37 CFR 1.111(b) and MPEP § 707.07(a).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1, 8, 9, 16, 17, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Lee (US 2020/0413094 A1, hereinafter referenced “Lee”) in view of Shin (US 2021/0099687 A1, hereinafter referenced “Shin”).
In regards to claim 1. Lee discloses a video encoding method (Lee, Abstract) comprising:
-classifying a plurality of view images into basic images and additional images (Lee, para [0059]; Reference discloses the view optimizer 210 may select at least one basic-view image and at least one additional view image from at least one source view image taken by a plurality of cameras);
-performing pruning on at least one of the plurality of view images on the basis of the classification result (Lee, para [0061]; Reference discloses the atlas constructor 220 may remove an overlapping region between view images (i.e. performing pruning on at least one of the plurality of view images) on the basis of at least one of the depth information of at least one basic-view image, the depth information of at least one additional view image, and a camera parameter (i.e. on the basis of the classification result));
-generating an atlas based on the pruning results (Lee, para [0065]; Reference discloses A patch packer 223 may generate a patch of each view position image by extracting a texture component (non-overlapping image region) corresponding to a pruning mask of each view position image from view position images and generate a packed view by packing patches of view position images into a small number of images. Herein, a packed view may be referred to as atlas);
Lee does not explicitly disclose but Shin teaches
-and encoding the atlas and metadata for the atlas, wherein the metadata includes information on a spherical harmonic function for approximating an irradiance environment map at an arbitrary position in a three-dimensional space (Shin, para [0252]; Reference discloses metadata may include information indicating whether or not an immersive and heterogeneous media are jointly encoded (i.e. encoding the atlas and metadata for the atlas)…. the metadata may include the metadata may include intensity and/or direction of major light, and information on ambient light. Herein, the information on ambient light may be information on light irradiating around the heterogeneous media object like spherical harmonic. When the information on the ambient light is information on irradiating light, the information may be transmitted in a form of light map (i.e. the metadata includes information on a spherical harmonic function for approximating an irradiance environment map at an arbitrary position in 3D space regarding the immersive video)).
Lee and Shin are combinable because they are in the same field of endeavor regarding encoding/decoding of multi-view images. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention for the encoding/decoding image system of Lee to include the immersive video features of Shin in order to provide the user with an image decoding/encoding system for storing a bitstream to provide omnidirectional video for corresponding motion parallax to a viewer' s left-and-right/up-and-down rotation and left-and-right/up-and-down translations as taught by Lee while incorporating the immersive video features of Shin to allow for determining a priority order of pruning for input videos, extracting patches based on the priority, generating an atlas based on the patches, and encoding metadata to improve processing for supporting motion parallax for rotational and translation motions in immersive video, applicable to the omnidirectional video processing as taught in Lee.
In regards to claim 8. Lee in view of Shin teach the video encoding method of claim 1.
Lee does not explicitly disclose but Shin teaches
-wherein the metadata further includes at least one of information indicating the type of a function used to apply a relighting effect to a heterogeneous object or information indicating coefficients of the function (Shin, para [0252]; Reference discloses when an immersive video and heterogeneous media are jointly encoded, the metadata may include at least one of information on transparency and/or reflectance of heterogeneous media object, information on a space of immersive video in which heterogeneous media are to be jointly reproduced (e.g., at least one of spatial position, size, viewing space and user's manipulation range), intensity and/or direction of major light, and information on ambient light… When a heterogeneous media object that is produced in a different from an immersive video is jointly rendered with the immersive video, a realistic video may be reproduced by rendering the reflectance and shadow of the heterogeneous media object based on at least one of the intensity/direction of light of the immersive video and information on ambient light (i.e. at least one of information indicating the type of a function used to apply a relighting effect to a heterogeneous object or information indicating coefficients of the function regarding the spherical harmonics)).
Lee and Shin are combinable because they are in the same field of endeavor regarding encoding/decoding of multi-view images. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention for the encoding/decoding image system of Lee to include the immersive video features of Shin in order to provide the user with an image decoding/encoding system for storing a bitstream to provide omnidirectional video for corresponding motion parallax to a viewer' s left-and-right/up-and-down rotation and left-and-right/up-and-down translations as taught by Lee while incorporating the immersive video features of Shin to allow for determining a priority order of pruning for input videos, extracting patches based on the priority, generating an atlas based on the patches, and encoding metadata to improve processing for supporting motion parallax for rotational and translation motions in immersive video, applicable to the omnidirectional video processing as taught in Lee.
In regards to claim 9. Lee discloses a video decoding method (Lee, Abstract) comprising:
-decoding an atlas and metadata for the atlas (Lee, para [0115]; Reference discloses referring to FIG. 10, a multi-view image decoder may obtain a bitstream including basic-view image encoding information on a basic-view image and residual additional view image encoding information on a plurality of residual additional view images (S1001). Here, the residual additional view image encoding information may include the packing information of a patch (i.e. decoding atlas and metadata or packing information of a patch);
-and generating a viewport image using the atlas and the metadata (Lee, para [0122]; Reference discloses in addition, a multi-view image decoder may reconstruct at least one additional view image (i.e. viewport image) from a plurality of residual additional view images on the basis of basic-view image encoding information, residual additional view image encoding information, and a basic-view image (S1003) (i.e. using the atlas and the metadata)),
Lee does not explicitly disclose but Shin teaches
-wherein the metadata includes information on a spherical harmonic function for approximating an irradiance environment map at an arbitrary position in a three-dimensional space (Shin, para [0252]; Reference discloses metadata may include information indicating whether or not an immersive and heterogeneous media are jointly encoded …. the metadata may include the metadata may include intensity and/or direction of major light, and information on ambient light. Herein, the information on ambient light may be information on light irradiating around the heterogeneous media object like spherical harmonic. When the information on the ambient light is information on irradiating light, the information may be transmitted in a form of light map (i.e. the metadata includes information on a spherical harmonic function for approximating an irradiance environment map at an arbitrary position in 3D space regarding the immersive video)).
Lee and Shin are combinable because they are in the same field of endeavor regarding encoding/decoding of multi-view images. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention for the encoding/decoding image system of Lee to include the immersive video features of Shin in order to provide the user with an image decoding/encoding system for storing a bitstream to provide omnidirectional video for corresponding motion parallax to a viewer' s left-and-right/up-and-down rotation and left-and-right/up-and-down translations as taught by Lee while incorporating the immersive video features of Shin to allow for determining a priority order of pruning for input videos, extracting patches based on the priority, generating an atlas based on the patches, and encoding metadata to improve processing for supporting motion parallax for rotational and translation motions in immersive video, applicable to the omnidirectional video processing as taught in Lee.
In regards to claim 16. Lee in view of Shin teach the video decoding method of claim 9.
Lee does not explicitly disclose but Shin teaches
-wherein the metadata further includes at least one of information indicating the type of a function used to apply a relighting effect to a heterogeneous object or information indicating coefficients of the function (Shin, para [0252]; Reference discloses when an immersive video and heterogeneous media are jointly encoded, the metadata may include at least one of information on transparency and/or reflectance of heterogeneous media object, information on a space of immersive video in which heterogeneous media are to be jointly reproduced (e.g., at least one of spatial position, size, viewing space and user's manipulation range), intensity and/or direction of major light, and information on ambient light… When a heterogeneous media object that is produced in a different from an immersive video is jointly rendered with the immersive video, a realistic video may be reproduced by rendering the reflectance and shadow of the heterogeneous media object based on at least one of the intensity/direction of light of the immersive video and information on ambient light (i.e. at least one of information indicating the type of a function used to apply a relighting effect to a heterogeneous object or information indicating coefficients of the function regarding the spherical harmonics)).
Lee and Shin are combinable because they are in the same field of endeavor regarding encoding/decoding of multi-view images. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention for the encoding/decoding image system of Lee to include the immersive video features of Shin in order to provide the user with an image decoding/encoding system for storing a bitstream to provide omnidirectional video for corresponding motion parallax to a viewer' s left-and-right/up-and-down rotation and left-and-right/up-and-down translations as taught by Lee while incorporating the immersive video features of Shin to allow for determining a priority order of pruning for input videos, extracting patches based on the priority, generating an atlas based on the patches, and encoding metadata to improve processing for supporting motion parallax for rotational and translation motions in immersive video, applicable to the omnidirectional video processing as taught in Lee.
In regards to claim 17. Lee in view of Shin teach the video decoding method of claim 9.
Lee does not explicitly disclose but Shin teaches
-wherein the generating of the viewport image comprises applying a relighting effect to a heterogeneous object included in the viewport image (Shin, para [0252]; Reference discloses when an immersive video and heterogeneous media are jointly encoded, the metadata may include at least one of information on transparency and/or reflectance of heterogeneous media object, information on a space of immersive video in which heterogeneous media are to be jointly reproduced (e.g., at least one of spatial position, size, viewing space and user's manipulation range), intensity and/or direction of major light, and information on ambient light… When a heterogeneous media object that is produced in a different from an immersive video is jointly rendered with the immersive video, a realistic video may be reproduced by rendering the reflectance and shadow of the heterogeneous media object based on at least one of the intensity/direction of light of the immersive video and information on ambient light (i.e. generating of the viewport image comprises applying a relighting effect to a heterogeneous object included in the viewport image)).
Lee and Shin are combinable because they are in the same field of endeavor regarding encoding/decoding of multi-view images. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention for the encoding/decoding image system of Lee to include the immersive video features of Shin in order to provide the user with an image decoding/encoding system for storing a bitstream to provide omnidirectional video for corresponding motion parallax to a viewer' s left-and-right/up-and-down rotation and left-and-right/up-and-down translations as taught by Lee while incorporating the immersive video features of Shin to allow for determining a priority order of pruning for input videos, extracting patches based on the priority, generating an atlas based on the patches, and encoding metadata to improve processing for supporting motion parallax for rotational and translation motions in immersive video, applicable to the omnidirectional video processing as taught in Lee.
In regards to claim 20. Lee discloses a computer-readable recording medium storing a video encoding method (Lee, Abstract) comprising:
-classifying a plurality of view images into basic images and additional images (Lee, para [0059]; Reference discloses the view optimizer 210 may select at least one basic-view image and at least one additional view image from at least one source view image taken by a plurality of cameras);
-performing pruning on at least one of the plurality of view images on the basis of the classification result (Lee, para [0061]; Reference discloses the atlas constructor 220 may remove an overlapping region between view images (i.e. performing pruning on at least one of the plurality of view images) on the basis of at least one of the depth information of at least one basic-view image, the depth information of at least one additional view image, and a camera parameter (i.e. on the basis of the classification result));
-generating an atlas based on the pruning results (Lee, para [0065]; Reference discloses A patch packer 223 may generate a patch of each view position image by extracting a texture component (non-overlapping image region) corresponding to a pruning mask of each view position image from view position images and generate a packed view by packing patches of view position images into a small number of images. Herein, a packed view may be referred to as atlas);
Lee does not explicitly disclose but Shin teaches
-and encoding the atlas and metadata for the atlas, wherein the metadata includes information on a spherical harmonic function for approximating an irradiance environment map at an arbitrary position in a three-dimensional space (Shin, para [0252]; Reference discloses metadata may include information indicating whether or not an immersive and heterogeneous media are jointly encoded (i.e. encoding the atlas and metadata for the atlas)…. the metadata may include the metadata may include intensity and/or direction of major light, and information on ambient light. Herein, the information on ambient light may be information on light irradiating around the heterogeneous media object like spherical harmonic. When the information on the ambient light is information on irradiating light, the information may be transmitted in a form of light map (i.e. the metadata includes information on a spherical harmonic function for approximating an irradiance environment map at an arbitrary position in 3D space regarding the immersive video)).
Lee and Shin are combinable because they are in the same field of endeavor regarding encoding/decoding of multi-view images. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention for the encoding/decoding image system of Lee to include the immersive video features of Shin in order to provide the user with an image decoding/encoding system for storing a bitstream to provide omnidirectional video for corresponding motion parallax to a viewer' s left-and-right/up-and-down rotation and left-and-right/up-and-down translations as taught by Lee while incorporating the immersive video features of Shin to allow for determining a priority order of pruning for input videos, extracting patches based on the priority, generating an atlas based on the patches, and encoding metadata to improve processing for supporting motion parallax for rotational and translation motions in immersive video, applicable to the omnidirectional video processing as taught in Lee.
Claims 2-5, 7, 10-13, and 15 are rejected under pre-AIA 35 U.S.C. 103(a) as being unpatentable over Lee (US 2020/0413094 A1) in view of Shin (US 2021/0099687 A1) as applied to claims 1 and 9 above, and further in view of Kaplanyan (US 2011/0012901 A1, hereinafter referenced “Kap”).
In regards to claim 2. Lee in view of Shin teach the video encoding method of claim 1.
Lee and Shin does not explicitly disclose but Kap teaches
-wherein the information on the spherical harmonic function includes at least one of information indicating an order of the spherical harmonic function or information on coefficients for the spherical harmonic function (Kap, para [0049] and [0051]; Reference at [0049] discloses we use spherical harmonics (SH) to represent the directional distribution of intensity. Using n bands of SH yields n2 coefficients, cl,m for the basis functions, yl,m(ω), both indexed by the band l and degree m with −l≦m≦l, we can easily derive analytical expressions for the spherical harmonics (SH) coefficients. Para [0051] discloses after determining the grid cell we simply accumulate the spherical coefficients of the respective grid cell (interpreted as the information on the spherical harmonic function includes information on coefficients for the spherical harmonic function)).
Lee and Shin are combinable because they are in the same field of endeavor regarding encoding/decoding of multi-view images. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention for the encoding/decoding image system of Lee to include the immersive video features of Shin in order to provide the user with an image decoding/encoding system for storing a bitstream to provide omnidirectional video for corresponding motion parallax to a viewer' s left-and-right/up-and-down rotation and left-and-right/up-and-down translations as taught by Lee while incorporating the immersive video features of Shin to allow for determining a priority order of pruning for input videos, extracting patches based on the priority, generating an atlas based on the patches, and encoding metadata to improve processing for supporting motion parallax for rotational and translation motions in immersive video, applicable to the omnidirectional video processing as taught in Lee.
Lee and Kap are also combinable because they are in the same field of endeavor regarding image processing. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention for the encoding/decoding image system of Lee, in view of the immersive video features of Shin, to include the indirect illumination image rendering features of Kap in order to provide the user with an image decoding/encoding system for storing a bitstream to provide omnidirectional video for corresponding motion parallax to a viewer' s left-and-right/up-and-down rotation and left-and-right/up-and-down translations as taught by Lee while incorporating the immersive video features of Shin to allow for determining a priority order of pruning for input videos, extracting patches based on the priority, generating an atlas based on the patches, and encoding metadata to improve processing for supporting motion parallax for rotational and translation motions in immersive video. Furthermore, incorporating the indirect illumination image rendering features of Kap allows for computing real-time indirect illumination for an image in a scene through use of 3D grid cells iteratively for more computational efficiency in dynamic scene rendering, applicable to the omnidirectional image processing systems as taught in Lee and Shin.
In regards to claim 3. Lee in view of Shin in further view of Kap teach the video encoding method of claim 2.
Lee and Shin does not explicitly disclose but Kap teaches
-wherein the information on the coefficients is encoded separately for each channel of the atlas (Kap, para [0051] and [0067]; Reference at [0051] discloses after determining the grid cell we simply accumulate the spherical coefficients of the respective grid cell. Para [0067] discloses subsequent iterations of intensity propagation take the LPV from the previous iteration as input. Each cell stores the intensity as a spherical harmonics vector and the light is then propagated to its six neighbors along the axial directions as shown in FIG. 10 b, which shows the 2D case with 4 axial directions (i.e. accumulation of coefficients iteratively for each cell of the grid interpreted as analogous to encoding coefficients separately for each channel of an atlas)).
Lee and Kap are also combinable because they are in the same field of endeavor regarding image processing. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention for the encoding/decoding image system of Lee, in view of the immersive video features of Shin, to include the indirect illumination image rendering features of Kap in order to provide the user with an image decoding/encoding system for storing a bitstream to provide omnidirectional video for corresponding motion parallax to a viewer' s left-and-right/up-and-down rotation and left-and-right/up-and-down translations as taught by Lee while incorporating the immersive video features of Shin to allow for determining a priority order of pruning for input videos, extracting patches based on the priority, generating an atlas based on the patches, and encoding metadata to improve processing for supporting motion parallax for rotational and translation motions in immersive video. Furthermore, incorporating the indirect illumination image rendering features of Kap allows for computing real-time indirect illumination for an image in a scene through use of 3D grid cells iteratively for more computational efficiency in dynamic scene rendering, applicable to the omnidirectional image processing systems as taught in Lee and Shin.
In regards to claim 4. Lee in view of Shin in further view of Kap teach the video encoding method of claim 2.
Lee and Shin does not explicitly disclose but Kap teaches
-wherein the information on the spherical harmonic function further includes position information indicating a position corresponding to the irradiance environment map in the three-dimensional space (Kap, para [0120]; Reference discloses the step of light injection (i.e. which includes the spherical harmonics calculations) according to FIG. 2 comprises a loop designated by reference numeral S201 which extends over all secondary light source with light intensity I in the respective reflective shadow map of the scene. In step S202 this light intensity I is position into the corresponding grid cell of the 3D grid of the light propagating volume (i.e. position information indicating a position corresponding to the irradiance environment map in the three-dimensional space). In step S203 the colored intensity and the orientation of the light intensity is transformed into a lighting distribution. Performing the steps S202-2203 overall secondary light sources in the reflective shadow map finally yields a distribution of secondary light sources over the scene to be rendered. These secondary light sources are mapped (projected) onto the grid cells of the light propagation volume).
Lee and Kap are also combinable because they are in the same field of endeavor regarding image processing. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention for the encoding/decoding image system of Lee, in view of the immersive video features of Shin, to include the indirect illumination image rendering features of Kap in order to provide the user with an image decoding/encoding system for storing a bitstream to provide omnidirectional video for corresponding motion parallax to a viewer' s left-and-right/up-and-down rotation and left-and-right/up-and-down translations as taught by Lee while incorporating the immersive video features of Shin to allow for determining a priority order of pruning for input videos, extracting patches based on the priority, generating an atlas based on the patches, and encoding metadata to improve processing for supporting motion parallax for rotational and translation motions in immersive video. Furthermore, incorporating the indirect illumination image rendering features of Kap allows for computing real-time indirect illumination for an image in a scene through use of 3D grid cells iteratively for more computational efficiency in dynamic scene rendering, applicable to the omnidirectional image processing systems as taught in Lee and Shin.
In regards to claim 5. Lee in view of Shin teach the video encoding method of claim 1.
Lee and Shin does not explicitly disclose but Kap teaches
-wherein the information on the spherical harmonic function includes information indicating whether to reuse information on a spherical harmonic function used in a previous frame (Kap, Claim 11; Reference discloses occluding surface elements and secondary light sources are re-used from a previously computed image frame).
Lee and Kap are also combinable because they are in the same field of endeavor regarding image processing. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention for the encoding/decoding image system of Lee, in view of the immersive video features of Shin, to include the indirect illumination image rendering features of Kap in order to provide the user with an image decoding/encoding system for storing a bitstream to provide omnidirectional video for corresponding motion parallax to a viewer' s left-and-right/up-and-down rotation and left-and-right/up-and-down translations as taught by Lee while incorporating the immersive video features of Shin to allow for determining a priority order of pruning for input videos, extracting patches based on the priority, generating an atlas based on the patches, and encoding metadata to improve processing for supporting motion parallax for rotational and translation motions in immersive video. Furthermore, incorporating the indirect illumination image rendering features of Kap allows for computing real-time indirect illumination for an image in a scene through use of 3D grid cells iteratively for more computational efficiency in dynamic scene rendering, applicable to the omnidirectional image processing systems as taught in Lee and Shin.
In regards to claim 7. Lee in view of Shin teach the video encoding method of claim 1.
Lee and Shin does not explicitly disclose but Kap teaches
-wherein the metadata further includes resolution information of an environment map image used to obtain the irradiance environment map (Kap, para [0035]; Reference at [0035] discloses for this purpose it is assumed in the following that the scene to be computed is embedded into a 3D grid of a fixed resolution (this limitation may be removed by introducing cascaded grids as outlined in the following) (i.e. includes resolution information of an environment map image). According to the present invention, basically two grids used, namely a first grid storing the intensity that is initialized from the surfaces causing indirect lighting or low-frequency direct lighting; and a second grid that stores a volumetric approximation of the scene geometry and is used for fuzzy blocking as the light travels through the scene. Both grids store a spherical function represented as low-frequency second harmonics (SH) approximation (i.e. used to obtain the irradiance environment map)).
Lee and Kap are also combinable because they are in the same field of endeavor regarding image processing. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention for the encoding/decoding image system of Lee, in view of the immersive video features of Shin, to include the indirect illumination image rendering features of Kap in order to provide the user with an image decoding/encoding system for storing a bitstream to provide omnidirectional video for corresponding motion parallax to a viewer' s left-and-right/up-and-down rotation and left-and-right/up-and-down translations as taught by Lee while incorporating the immersive video features of Shin to allow for determining a priority order of pruning for input videos, extracting patches based on the priority, generating an atlas based on the patches, and encoding metadata to improve processing for supporting motion parallax for rotational and translation motions in immersive video. Furthermore, incorporating the indirect illumination image rendering features of Kap allows for computing real-time indirect illumination for an image in a scene through use of 3D grid cells iteratively for more computational efficiency in dynamic scene rendering, applicable to the omnidirectional image processing systems as taught in Lee and Shin.
In regards to claim 10. Lee in view of Shin teach the video decoding method of claim 9.
Lee and Shin does not explicitly disclose but Kap teaches
-wherein the information on the spherical harmonic function includes at least one of information indicating an order of the spherical harmonic function or information on coefficients for the spherical harmonic function (Kap, para [0049] and [0051]; Reference at [0049] discloses we use spherical harmonics (SH) to represent the directional distribution of intensity. Using n bands of SH yields n2 coefficients, cl,m for the basis functions, yl,m(ω), both indexed by the band l and degree m with −l≦m≦l, we can easily derive analytical expressions for the spherical harmonics (SH) coefficients. Para [0051] discloses after determining the grid cell we simply accumulate the spherical coefficients of the respective grid cell (interpreted as the information on the spherical harmonic function includes information on coefficients for the spherical harmonic function)).
Lee and Kap are also combinable because they are in the same field of endeavor regarding image processing. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention for the encoding/decoding image system of Lee, in view of the immersive video features of Shin, to include the indirect illumination image rendering features of Kap in order to provide the user with an image decoding/encoding system for storing a bitstream to provide omnidirectional video for corresponding motion parallax to a viewer' s left-and-right/up-and-down rotation and left-and-right/up-and-down translations as taught by Lee while incorporating the immersive video features of Shin to allow for determining a priority order of pruning for input videos, extracting patches based on the priority, generating an atlas based on the patches, and encoding metadata to improve processing for supporting motion parallax for rotational and translation motions in immersive video. Furthermore, incorporating the indirect illumination image rendering features of Kap allows for computing real-time indirect illumination for an image in a scene through use of 3D grid cells iteratively for more computational efficiency in dynamic scene rendering, applicable to the omnidirectional image processing systems as taught in Lee and Shin.
In regards to claim 11. Lee in view of Shin in further view of Kap teach the video decoding method of claim 10.
Lee and Shin does not explicitly disclose but Kap teaches
-wherein the information on the coefficients is decoded separately for each channel of the atlas (Kap, para [0051] and [0067]; Reference at [0051] discloses after determining the grid cell we simply accumulate the spherical coefficients of the respective grid cell. Para [0067] discloses subsequent iterations of intensity propagation take the LPV from the previous iteration as input. Each cell stores the intensity as a spherical harmonics vector and the light is then propagated to its six neighbors along the axial directions as shown in FIG. 10 b, which shows the 2D case with 4 axial directions (i.e. accumulation of coefficients iteratively for each cell of the grid interpreted as analogous to decoding coefficients separately for each channel of an atlas)).
Lee and Kap are also combinable because they are in the same field of endeavor regarding image processing. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention for the encoding/decoding image system of Lee, in view of the immersive video features of Shin, to include the indirect illumination image rendering features of Kap in order to provide the user with an image decoding/encoding system for storing a bitstream to provide omnidirectional video for corresponding motion parallax to a viewer' s left-and-right/up-and-down rotation and left-and-right/up-and-down translations as taught by Lee while incorporating the immersive video features of Shin to allow for determining a priority order of pruning for input videos, extracting patches based on the priority, generating an atlas based on the patches, and encoding metadata to improve processing for supporting motion parallax for rotational and translation motions in immersive video. Furthermore, incorporating the indirect illumination image rendering features of Kap allows for computing real-time indirect illumination for an image in a scene through use of 3D grid cells iteratively for more computational efficiency in dynamic scene rendering, applicable to the omnidirectional image processing systems as taught in Lee and Shin.
In regards to claim 12. Lee in view of Shin in further view of Kap teach the video decoding method of claim 10.
Lee and Shin does not explicitly disclose but Kap teaches
-wherein the information on the spherical harmonic function further includes position information indicating a position corresponding to the irradiance environment map in the three-dimensional space (Kap, para [0120]; Reference discloses the step of light injection (i.e. which includes the spherical harmonics calculations) according to FIG. 2 comprises a loop designated by reference numeral S201 which extends over all secondary light source with light intensity I in the respective reflective shadow map of the scene. In step S202 this light intensity I is position into the corresponding grid cell of the 3D grid of the light propagating volume (i.e. position information indicating a position corresponding to the irradiance environment map in the three-dimensional space). In step S203 the colored intensity and the orientation of the light intensity is transformed into a lighting distribution. Performing the steps S202-2203 overall secondary light sources in the reflective shadow map finally yields a distribution of secondary light sources over the scene to be rendered. These secondary light sources are mapped (projected) onto the grid cells of the light propagation volume).
Lee and Kap are also combinable because they are in the same field of endeavor regarding image processing. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention for the encoding/decoding image system of Lee, in view of the immersive video features of Shin, to include the indirect illumination image rendering features of Kap in order to provide the user with an image decoding/encoding system for storing a bitstream to provide omnidirectional video for corresponding motion parallax to a viewer' s left-and-right/up-and-down rotation and left-and-right/up-and-down translations as taught by Lee while incorporating the immersive video features of Shin to allow for determining a priority order of pruning for input videos, extracting patches based on the priority, generating an atlas based on the patches, and encoding metadata to improve processing for supporting motion parallax for rotational and translation motions in immersive video. Furthermore, incorporating the indirect illumination image rendering features of Kap allows for computing real-time indirect illumination for an image in a scene through use of 3D grid cells iteratively for more computational efficiency in dynamic scene rendering, applicable to the omnidirectional image processing systems as taught in Lee and Shin.
In regards to claim 13. Lee in view of Shin teach the video decoding method of claim 9.
Lee and Shin does not explicitly disclose but Kap teaches
-wherein the information on the spherical harmonic function includes information indicating whether to reuse information on a spherical harmonic function used in a previous frame (Kap, Claim 11; Reference discloses occluding surface elements and secondary light sources are re-used from a previously computed image frame).
Lee and Kap are also combinable because they are in the same field of endeavor regarding image processing. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention for the encoding/decoding image system of Lee, in view of the immersive video features of Shin, to include the indirect illumination image rendering features of Kap in order to provide the user with an image decoding/encoding system for storing a bitstream to provide omnidirectional video for corresponding motion parallax to a viewer' s left-and-right/up-and-down rotation and left-and-right/up-and-down translations as taught by Lee while incorporating the immersive video features of Shin to allow for determining a priority order of pruning for input videos, extracting patches based on the priority, generating an atlas based on the patches, and encoding metadata to improve processing for supporting motion parallax for rotational and translation motions in immersive video. Furthermore, incorporating the indirect illumination image rendering features of Kap allows for computing real-time indirect illumination for an image in a scene through use of 3D grid cells iteratively for more computational efficiency in dynamic scene rendering, applicable to the omnidirectional image processing systems as taught in Lee and Shin.
In regards to claim 15. Lee in view of Shin teach the video decoding method of claim 9.
Lee and Shin does not explicitly disclose but Kap teaches
-wherein the metadata further includes resolution information of an environment map image used to obtain the irradiance environment map (Kap, para [0035]; Reference at [0035] discloses for this purpose it is assumed in the following that the scene to be computed is embedded into a 3D grid of a fixed resolution (this limitation may be removed by introducing cascaded grids as outlined in the following) (i.e. includes resolution information of an environment map image). According to the present invention, basically two grids used, namely a first grid storing the intensity that is initialized from the surfaces causing indirect lighting or low-frequency direct lighting; and a second grid that stores a volumetric approximation of the scene geometry and is used for fuzzy blocking as the light travels through the scene. Both grids store a spherical function represented as low-frequency second harmonics (SH) approximation (i.e. used to obtain the irradiance environment map)).
Lee and Kap are also combinable because they are in the same field of endeavor regarding image processing. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention for the encoding/decoding image system of Lee, in view of the immersive video features of Shin, to include the indirect illumination image rendering features of Kap in order to provide the user with an image decoding/encoding system for storing a bitstream to provide omnidirectional video for corresponding motion parallax to a viewer' s left-and-right/up-and-down rotation and left-and-right/up-and-down translations as taught by Lee while incorporating the immersive video features of Shin to allow for determining a priority order of pruning for input videos, extracting patches based on the priority, generating an atlas based on the patches, and encoding metadata to improve processing for supporting motion parallax for rotational and translation motions in immersive video. Furthermore, incorporating the indirect illumination image rendering features of Kap allows for computing real-time indirect illumination for an image in a scene through use of 3D grid cells iteratively for more computational efficiency in dynamic scene rendering, applicable to the omnidirectional image processing systems as taught in Lee and Shin.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure: See the Notice of References Cited (PTO-892)
Any inquiry concerning this communication or earlier communications fro