DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Acknowledgment is made of applicant's claim for foreign priority based on an application filed in the Republic of Korea on 1/16/2024. It is noted, however, that applicant has not filed a certified copy of the KR10-2024-0006938 application as required by 37 CFR 1.55. While a standalone copy of the foreign disclosure is noted amongst the filings, it does not satisfy the requirements of certification under 37 CFR 1.55.
Claim Interpretation
The claims will be read under the broadest reasonable interpretation standard outlined in
MPEP § 2111.01.
Claim Objections
Claim 7 is objected to for the following informalities: “the transparency images” is recited without antecedent basis, as claim 4 only introduces one transparency image.
Claim 9 is objected to for the following informalities: “other than the first coefficient” is recited without antecedent basis, as the claim only previously introduced “a first coefficient value”
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention..
Claims 2-7 are rejected under 35 U.S.C. 112(b) as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor regards as the invention.
With respect to claim 2, the claim recites: “the plurality of layer images correspond to one of the plurality of references planes”. It is unclear whether this claim recites a 1:1 relationship between a layer image and reference plane, or a correspondence of all the layer images to a single reference plane. For the purposes of compact prosecution, both interpretations will read upon the prior art.
Claims 3-7 are rejected by virtue of their dependency on claim 2.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-12 and 14-15 are rejected under 35 U.S.C. 103 as being unpatentable over Morgenstern et. al Compact 3D Scene Representation via Self-Organizing Gaussian Grids (Hereinafter, “Morgenstern”) in view of Yu et. al Plenoxels: Radiance Fields without Neural Networks (Hereinafter, “Yu”)
With respect to claim 1, Morgenstern teaches:
An image encoding method ([Abstract]; [1]; Fig. 1), comprising:
deriving coefficients of a spherical harmonic function [1]; [2]; “The parameters (3D coordinates, color, size, density, orientation, and spherical harmonics) of these Gaussians are optimized using an efficient differentiable rasterization algorithm…”; [3] “The idea of our method is to map originally unstructured 3D Gaussians to a 2D grid in such a way that local spatial relationships are preserved. By mapping all Gaussian parameters (position, color, etc.) to the same 2D position, we effectively create multiple data layers within the same layout”; [4] “Here, the training with activated spherical harmonics shows the best rendering results, especially when inspecting transparent materials such as the windshield of the truck”; [9]; Fig. 2; Fig. 3)
generating a plurality of layer images based on the coefficients ([1] “We exploit the fact that numerous permutations of Gaussian parameters can equivalently represent it with the same visual quality by arranging the millions of multidimensional list entries of 3DGS into a structured 2D grid with increased smoothness that can efficiently be encoded…”; [2]; [3] “By mapping all Gaussian parameters (position, color, etc.) to the same 2D position, we effectively create multiple data layers within the same layout”; Fig. 1, “…we are able to significantly reduce the memory footprint using state-of-the-art image compression methods”; Fig. 2, “…which help to compress the 2D grids into small files using off-the-shelf compression methods”; Fig. 3; [4])
encoding the plurality of layer images (Fig. 2; Fig. 6; [3]; [4.4]; [5] “This is achieved by a novel, highly parallelized sorting strategy that smoothly arranges the high dimensional, initially unordered Gaussian parameter sets in a 2D grid, which can be encoded by standard image coding techniques like jpeg-xl”)
wherein each of the plurality of layer images includes a coefficient value in an order corresponding to the spherical harmonic function (Fig. 1; [3] “In our application, we normalize all attributes of the Gaussians (coordinates, scale, etc.) independently to the range [0,1] and then scale them with individual weights before calculating the distance of the possible re-assignments. We use the activated values for all attributes (e.g. exponentially activated scale), as they would be seen by the renderer”; [9] “We clip the RGB values (DC values of the spherical harmonics) to the range [-2,4] and normalize it to the range [0,1]…”; The examiner notes that DC values of the spherical harmonics are a 0th order spherical harmonic function value, and thus meet the plain language of “coefficient value in an order corresponding to the spherical harmonic function”; Fig. 2
Regardless, the examiner maintains that any further implementation of spherical harmonics would have been obvious. While Morgenstern does not explicitly generate a layer/attribute image including higher order spherical harmonic information (instead using the information in a training phase), the authors evidence the feasibility of including such information in the main method and possibly even prior experimentation with the same; [9] “We have not used spherical harmonics (SH), as they have a relatively small effect on scene quality, while taking three quarters of all space in vanilla 3DGS (45 of 59 attributes in 3DGS are spherical harmonics, with the default third degree SH). We believe that a better space-for quality trade-off is required for efficient representation of view-dependent effects”). Yu outputs 3D spherical harmonic voxel information, which can be mapped to 2D pixel representation under the methods of Morgenstern, in a similar manner to which Morgenstern maps other non-color image attribute information (such as opacity) amongst depth slices)
Morgenstern does not explicitly teach:
for a plurality of vertices
However, Yu, in the same field of endeavor of 3D image processing, teaches:
An image encoding method ([1]), comprising:
deriving coefficients of a spherical harmonic function for a plurality of vertices on a three- dimensional space (Fig. 2; [2]; [3])
PNG
media_image1.png
367
833
media_image1.png
Greyscale
[2]; [3] “We use spherical harmonics of degree 2, which requires 9 coefficients per color channel for a total of 27 harmonic coefficients per voxel. We use degree 2 harmonics because PlenOctress found that higher order harmonics confer only minimal benefit”; Fig. 2)
It would have been obvious to a person of ordinary skill in the art as of the effective filing date of the claimed invention to modify Morgenstern to include the limitations of vertices processing as well as higher spherical harmonic ordering, as taught by Yu. The derived spherical harmonic voxel information of Yu would have been obvious to additionally map as another 2D numerical pixel representation under the conversion methods of Morgenstern. Doing so allows Morgenstern’s 2D grid framework to retain increased visual detail while taking advantage of spatial coefficient alignment across grid layers to optimize 2D image compression. The systems readily integrate, as higher order spherical harmonic information represents an additional class of layer information to be included amongst the sets of attribute-layer groupings.
While Morgenstern states a preference for computational efficiency, it is insufficient to establish that the reference “teaches away” from a combination which has advantages for other stated objectives. Computational efficiency is a design incentive, but not the only design incentive. See MPEP § 2145(X)(D)(1), “UCB, Inc. v. Actavis Labs, UT, Inc., 65 F.4th 679, 692, 2023 USPQ2d 448 (Fed. Cir. 2023) ("a reference does not teach away if it merely expresses a general preference for an alternative invention but does not criticize, discredit or otherwise discourage investigation into the invention claimed.")”. Morgenstern is not inconsistent with a person of ordinary skill in the art’s attempts to optimize visual quality (especially for analysis of transparent materials, which Morgenstern teaches towards [4.2]) to the maximum under another objective. Yu provides a specific means to do so through derivation of coefficients for a given 3D set of vertices. Accordingly, an artisan seeking to provide even more visual quality than Morgenstern would predictably consider additional coefficient information in layer images carrying explicit higher order spherical harmonic information.
With respect to claim 2, Morgenstern/Yu teaches:
The method of claim 1, wherein:
the vertices on the three-dimensional space form a plurality of reference planes (Yu, Fig. 2a; noting author’s depiction of connecting grid lines that define flat, two-dimensional planes for visual reference (relative to incident light) across the 3D lattice)
the plurality of layer images correspond to one of the plurality of reference planes (Morgenstern, Fig. 2; Morgenstern, Fig. 6; in combination with Yu, Fig. 2a, a layer image would include information from the reference planes of Yu)
With respect to claim 3, Morgenstern/Yu teaches:
The method of claim 2, wherein:
a resolution of each of the plurality of layer images is equal to a size of an array of vertices included in a corresponding reference plane (Morgenstern, Fig. 6; Yu, Fig. 2; The method of Yu divides an input volume into a grid of vertices. Under the combination discussed in claim 1, the method of Morgenstern takes the same input volume in generating layer images, such that a layer image would predictably be of the same size as “an array” of vertices included in a “corresponding reference plane”. See MPEP § 2144.04(V)(B) – Making Integral (discussing obvious engineering implementation within the bounds of ordinary skill in the art).
With respect to claim 4, Morgenstern/Yu teaches:
The method of claim 2, wherein:
a transparency image including a transparency value of vertices included in a reference plane is additionally generated (Morgenstern, [4.4] “To ensure that we do not throw away any Gaussians that might bring meaningful contributions to the scene, we sort all Gaussians by opacity and only discard Gaussians with the lowest opacity. In figure 7, we show that the PSNR of a rendered scene is not affected when removing up to 30% of Gaussians that have a low opacity. In contrast, removing Gaussians with small scaling, affects the quality of the scene considerably more”; Yu, [1] “We call our model Plenoxel for plenoptic volume elements, as it consists of a sparse voxel grid in which each voxel stores opacity and spherical harmonic coefficients”; as Morgenstern stores opacity information in its images, any of its given images can be called a “transparency image” under BRI. In accordance with obvious engineering implementation, it would necessarily follow that this opacity value would correspond to opacity values of the vertices)
With respect to claim 5, Morgenstern/Yu teaches:
The method of claim 4, wherein:
a number of the layer images for the reference plane corresponds to a number of the coefficients (Morgenstern, Fig. 6; Yu, Fig. 2; under the language “comprising”, it is sufficient that a certain number of layer images among the layer images (e.g., 1) and a certain number of the coefficients among the coefficients (e.g., 1) is equal in value. As this is always true so long as a single layer image/coefficient is generated, the claim is satisfied)
a number of transparency images for the reference plane is 1 (Morgenstern, Fig. 6; under the same reasoning outlined above)
With respect to claim 6, Morgenstern/Yu teaches:
The method of claim 4, wherein:
as many transparency images as a number of the layer images are generated (Morgenstern, Fig. 6; necessarily so under the same reasoning outlined above in claim 5)
With respect to claim 7, Morgenstern/Yu teaches:
The method of claim 6, wherein:
at least one of a position or a size of a valid region is different between the transparency images (Morgenstern, Fig. 6)
With respect to claim 8, Morgenstern/Yu teaches:
The method of claim 1, wherein:
an attribute of the each of the plurality images is designated as a texture (Morgenstern, [3]; Morgenstern, [4.2] “For some datasets…we achieve a slightly higher PSNR and slightly lower L-PIPS, but at the same time allow for real time rendering and fast training, given the efficient rasterization algorithm of 3DGS”; Morgenstern, [9] “We compress the RGB grid with JPEG XL, as an 8-Bit image with a quality level of 90, and store all other attributes as 32-Bit OpenEXR images with zip compression”; noting that rasterization of 2D color images meets the standard of a texture under BRI; Morgenstern, Fig. 6)
With respect to claim 9, Morgenstern/Yu teaches:
The method of claim 1, wherein:
an attribute of a first layer image including a first coefficient value among the plurality of layer images is different from an attribute of a second layer image including a coefficient value other than the first coefficient (Morgenstern, Fig. 3; Morgenstern, [3] “By mapping all Gaussian parameters (position, color, etc.) to the same 2D position, we effectively create multiple data layers with the same layout. In this 2-dimensional layout, neighboring Gaussians will have similar attribute values, facilitating compression and enabling the storage of the same scene using reduced memory”)
With respect to claim 10, Morgenstern/Yu teaches:
The method of claim 1, wherein:
metadata including directional image configuration information is additionally encoded (Morgenstern, [3]; Morgenstern, [4] “For the compression of our 2D attribute grids, we use a combination of EXR and JPEG-XL”; Morgenstern, [9] “We quantize the different attributes by rounding them to the closest value of a linear range of q total values, with qcoords=213, qscale=qrotation=26 and qopacity=25.”; Yu, [3.2]; Yu, Fig. 2. As the layer images of Morgenstern/Yu are encoded as JPEG images and subsequently rasterized, they must necessarily include metadata/header information which signals to the GPU the identity of the information, including “directional image configuration information” (i.e labeling of spherical harmonic information) to the extent a person of ordinary skill in the art chooses to integrate higher order spherical harmonics as discussed in claim 1; See MPEP § 2144.04(V)(B))
With respect to claim 11, Morgenstern/Yu teaches:
The method of claim 10, wherein:
the directional image information includes at least one of number information of the layer images or number information of the coefficients (Morgenstern, [3]; Morgenstern, [4]; Morgenstern, [9]; Yu, [3.2] “We use spherical harmonics of degree 2, which requires 9 coefficients per color channel for a total of 27 harmonic coefficients per voxel…”; Yu, Fig. 2; necessary further implementation under the combination outlined above, to organize the per-channel information)
With respect to claim 12, Morgenstern/Yu teaches:
The method of claim 10, wherein:
the directional image configuration information includes basis function information for the spherical harmonic function (Morgenstern, [3]; Morgenstern, [4]; Morgenstern, [9]; Yu, [3.2] “We use spherical harmonics of degree 2, which requires 9 coefficients per color channel for a total of 27 harmonic coefficients per voxel…”; Yu, Fig. 2; as a similar recitation to “number information of the coefficients”; necessarily further implementation under the combination outlined above)
With respect to claim 14, Morgenstern teaches:
An image decoding method ([Abstract]), comprising:
decoding a plurality of layer images (Fig. 1; Fig. 2; Fig. 5)
reconstructing a target scene based on the layer images (Fig. 1; Fig. 2; Fig. 5)
wherein based on the decoded layer images, coefficients of a spherical harmonic function for a [4.2] “For some datasets, compared to Mip-NeRF360 and VQ-TensoRF, we achieve a slightly higher PSNR and slightly lower L-PIPS, but at the same time allow for real time rendering and fast training ,given the efficient rasterization algorithm of 3DGS”; [9] “We clip the RGB values (DC values of the spherical harmonics) to the range [-2,4] and normalize it to the range [0,1]…”; The examiner notes that DC values of the spherical harmonics are a 0th order spherical harmonic function value, and thus meet the plain language of “coefficients of a spherical harmonic function”; Fig. 1; Fig. 2; Fig. 5)
wherein each of the plurality of layer images includes a coefficient value in an order corresponding to the spherical harmonic function ([3]; [4]; [9]; Fig. 1; Fig. 2; Fig. 5)
Morgernstern does not explicitly teach:
plurality of vertices
However, Yu, in the same field of endeavor of 3D image processing, teaches:
An image decoding method, comprising:
decoding ([2]; [3]; Fig. 2)
reconstructing a target scene ([2]; [3]; Fig. 2)
[2]; [3]; Fig. 2)
[2]; [3] “We use spherical harmonics of degree 2, which requires 9 coefficients per color channel for a total of 27 harmonic coefficients per voxel. We use degree 2 harmonics because PlenOctress found that higher order harmonics confer only minimal benefit”; Fig. 2)
It would have been obvious to a person of ordinary skill in the art as of the effective filing date of the claimed invention to modify Morgenstern to include the limitations of vertices processing as well as higher spherical harmonic ordering, as taught by Yu. The derived spherical harmonic voxel information of Yu would have been obvious to additionally map as another 2D numerical pixel representation under the conversion methods of Morgenstern. Doing so allows Morgenstern’s 2D grid framework to retain increased visual detail while taking advantage of spatial coefficient alignment across grid layers to optimize 2D image compression. The systems readily integrate, as higher order spherical harmonic information represents an additional class of layer information to be included amongst the sets of attribute-layer groupings.
While Morgenstern states a preference for computational efficiency, it is insufficient to establish that the reference “teaches away” from a combination which has advantages for other stated objectives. Computational efficiency is a design incentive, but not the only design incentive. See MPEP § 2145(X)(D)(1), “UCB, Inc. v. Actavis Labs, UT, Inc., 65 F.4th 679, 692, 2023 USPQ2d 448 (Fed. Cir. 2023) ("a reference does not teach away if it merely expresses a general preference for an alternative invention but does not criticize, discredit or otherwise discourage investigation into the invention claimed.")”. Morgenstern is not inconsistent with a person of ordinary skill in the art’s attempts to optimize visual quality (especially for analysis of transparent materials, which Morgenstern teaches towards [4.2]) to the maximum under another objective. Yu provides a specific means to do so through derivation of coefficients for a given 3D set of vertices. Accordingly, an artisan seeking to provide even more visual quality than Morgenstern would predictably consider additional coefficient information in layer images carrying explicit higher order spherical harmonic information.
With respect to claim 15, the claim recites a parallel software implementation of the method of claim 1. Morgenstern teaches the hardware and software necessary for the same ([4.1]). Accordingly, the claim is rejected in line with the analysis above.
Claim 13 is rejected under 35 U.S.C. 103 as being unpatentable over Morgenstern/Yu in view of Tucker et. al Single-View View Synthesis with Multiplane Images (Hereinafter, “Tucker”)
With respect to claim 13, Morgenstern/Yu teaches the method of claim 10.
Morgenstern/Yu does not teach the further limitations of claim 13.
However, Tucker, in the same field of endeavor of 3D processing, teaches:
the directional image configuration information includes interval information between reference planes (Fig. 1; Fig. 2; [3.2] “As illustrated in Fig. 2, an MPI consists of a set of D fronto-parallel planes in the frustum of a reference camera, arranged at fixed depths d1,...,dD, from d1 = dfar to dD = dnear, and equally spaced in disparity (inverse depth))
It would have been obvious to a person of ordinary skill in the art as of the effective filing date of the claimed invention to modify Morgenstern/Yu to include the limitations of interval information between reference planes, as taught by Tucker. Doing so would allow the renderer of Morgenstern/Yu to calculate exact spatial warping and blending between layer planes during view integration. The systems readily integrate, as detailing multi-view information remains within the central object of Morgenstern/Yu.
Additional References
Additionally cited references (see attached PTO-892) otherwise not relied upon above have been made of record in view of the manner in which they evidence the general state of the art.
The examiner in particular notes a revised version of Morganstern, made available a few months after the release of the version cited in the rejections above. While only applicable due to the issues of foreign priority claiming, the reference still is worthy of mention, as it evidences the authors shifting towards the inclusion of more spherical harmonic information via layer images.
PNG
media_image2.png
413
1118
media_image2.png
Greyscale
Inquiry
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NOAH WILLIAM BOYAR whose telephone number is 571-272-8392. The examiner can normally be reached 10:00 AM – 6:00 PM EST, Monday – Friday.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at https://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chan Park can be reached at 571-272-7409. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/NOAH W BOYAR/Examiner, Art Unit 2669
/CHAN S PARK/Supervisory Patent Examiner, Art Unit 2669
Prosecution Insights