DETAILED ACTION
This action is in response to the application filed on April 29th, 2024. Claims 1-20 are pending and have been examined.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statements (IDS) submitted on April 29th, 2024, May 10th, 2024, October 17th, 2024, and July 8th, 2025 are being considered by the examiner.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1, 6-10, 13-15, and 19 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by “Random-Access Neural Compression of Material Textures” (herein after referred to by its primary author, Vaidyanathan).
In regards to claim 1, Vaidyanathan teaches an apparatus, comprising: one or more memories; and one or more processors, coupled to the one or more memories (Vaidyanathan Section 1 “A highly optimized implementation of our compressor, with fused backpropogation, enabling practical per-material optimization with resolutions up to 8192 × 8192 (8k). Our compressor can process a 9-channel, 4k material texture set in 1-15 minutes on an NVIDIA RTX 4090 GPU, depending on the desired quality level.”), configured to: receive a plurality of sets of features corresponding to a texture, wherein the plurality of sets of features comprises a respective set of features for each respective grid point of a grid, wherein each respective grid point of the grid is associated with a respective portion of the texture, wherein the grid has a first resolution (Vaidyanathan Figure 4 “Feature pyramid”; Section 4.1 “As shown in Figure 4 (a), our compressed representation is a pyramid of multiple feature levels 𝐹𝑗 , with each level, 𝑗, comprising a pair of 2D grids, 𝐺𝑗 0 and 𝐺𝑗 1 . The grids’ cells store feature vectors of quantized latent values, which are utilized to predict multiple mip levels. This sharing of features across two or more mip levels lowers the storage cost of a traditional mipmap chain from 33% to ∼ 6.7% or less. Furthermore, within a feature level, grid 𝐺0 is at a higher resolution, which helps preserve high-frequency details, while 𝐺1 is at a lower resolution, improving the reconstruction of low- frequency content, such as smooth gradients.”); receive coordinate information corresponding to a texel of the texture (Vaidyanathan Figure 4 Description “Overview of our method. a) Our compressed representation comprises multiple feature levels, each having two feature grids; a high resolution grid 𝐺0 and a low resolution grid 𝐺1 (Section 4.1). The solid circles represent the grid cells accessed for a target texel (in red)… c) During inference and training, we sample the four neighboring feature vectors (orange circles) from the grid 𝐺0 (Section 4.3.1) and bilinearly interpolate features from 𝐺1 (hollow gray circle), concatenating them with local positional encoding (Section 4.3.2) and a normalized level-of-detail (LOD) value for the target mip level”); receive level of detail information indicating a second resolution at which to reconstruct the texture, wherein the second resolution is lower than the first resolution (Vaidyanathan Section 4.3 “In this stage, we first select a feature level based on the desired level of detail (LOD) (Table1)”); select a subset of grid points of the grid based on the second resolution being lower than the first resolution; sample one or more grid points from among the subset of grid points based on the coordinate information to obtain sampled features associated with the one or more grid points (Vaidyanathan Figure 4 “Sampling and concatenation” Section 4.3 “In this stage, we first select a feature level based on the desired level of detail (LOD) (Table1),and then resample both the grids in the feature level to the target resolution. In the next section, we describe how grids are resampled by interpolating the features at the target texel location.”); input, to a machine-learning model, the sampled features; and receive, from the machine-learning model, based on the sampled features, a reconstruction of the texel of the texture at the second resolution (Vaidyanathan Figure 4 “Network” and “Predicted mip level”).
In regards to claim 6, Vaidyanathan teaches the apparatus of claim 1, wherein the plurality of sets of features are quantized to discrete levels (Vaidyanathan Section 4.2 “Since we do not use entropy coding, we enforce a fixed quantization rate for all latent values in a feature grid and only optimize for image distortion.”).
In regards to claim 7, Vaidyanathan teaches the apparatus of claim 1, wherein to sample the one or more grid points comprises to perform one or more of four nearest neighbor sampling or bilinear sampling (Vaidyanathan Section 4.3.1 “We use a learned interpolation approach for the higher resolution grid 𝐺0 and bilinear interpolation for the lower resolution grid 𝐺1”).
In regards to claim 8, Vaidyanathan teaches the apparatus of claim 7, wherein to sample the one or more grid points comprises to perform nearest-neighbor interpolation of the four nearest neighbor sampling (Vaidyanathan Figure 4 Description “During inference and training, we sample the four neighboring feature vectors (orange circles) from the grid 𝐺0 (Section 4.3.1) and bilinearly interpolate features from 𝐺1 (hollow gray circle), concatenating them with local positional encoding (Section 4.3.2) and a normalized level-of-detail (LOD) value for the target mip level.”).
In regards to claim 9, Vaidyanathan teaches the apparatus of claim 1, wherein the one or more processors are configured to: receive a second plurality of sets of features corresponding to the texture, wherein the second plurality of sets of features comprises a respective set of features for each respective grid point of a second grid, wherein each respective grid point of the second grid is associated with a respective portion of the texture, wherein the second grid has the first resolution; sample the second grid at one or more second grid points to obtain second features associated with the one or more second grid points; and input, to the machine-learning model, the second features, wherein to receive, from the machine-learning model the reconstruction of the texel of the texture is further based on the second features (Vaidyanathan Figure 4; Section 4 “We represent the texture set as a tensor with dimensions𝑤×ℎ×𝑐 and our model compresses the tensor without making any assumptions about the channel count or the specific semantics of each channel. For example, the normals or diffuse albedo could be mapped to any channels without affecting compression. This is possible because we learn the compressed representation for each material individually, effectively specializing it for its unique semantics.” Examiner note: Vaidyanathan describes that their method can be applied to a plurality of textures, and that any input would be represented as a feature pyramid and passed to the network for decompression. Furthermore, Vaidyanathan describes that their method can be used to output a plurality of different MIP levels, as shown in table 1. Therefore, Vaidyanathan teaches that a second set of features could be decompressed and output as a reconstruction of the texture).
In regards to claim 10, Vaidyanathan teaches the apparatus of claim 1, wherein each set of features of the plurality of sets of features comprises a multi-channel feature vector (Vaidyanathan Section 4 “We represent the texture set as a tensor with dimensions𝑤×ℎ×𝑐 and our model compresses the tensor without making any assumptions about the channel count or the specific semantics of each channel.”).
In regards to claim 13, Vaidyanathan teaches the apparatus of claim 1, wherein the level of detail information indicates a mipmap level of texture (Vaidyanathan Table 1 “Predicted mip levels”).
In regards to claim 14, Vaidyanathan teaches the apparatus of claim 1, wherein the reconstruction of the texel comprises texture attributes corresponding to material properties (Vaidyanathan Figure 4 “MIP 0”; Section 4 “For example, the normals or diffuse albedo could be mapped to any channels without affecting compression. This is possible because we learn the compressed representation for each material individually, effectively specializing it for its unique semantics.”).
In regards to claim 15, Vaidyanathan teaches the apparatus of claim 1, wherein the coordinate information is encoded as a position-encoding vector based on values of a pair of coordinate variables (Vaidyanathan Section 2.3 “Coordinate networks frequently employ positional encoding, a concept originating from language modeling literature [79]. Instead of passing the input coordinates p directly to the MLP, this method encodes it as a vector of sin(2ℎ𝜋p) and cos(2ℎ𝜋p) terms, where ℎ represents an octave.”; Section 4.3.1 “We use a learned interpolation approach for the higher resolution grid𝐺0 and bilinear interpolation for the lower resolution grid 𝐺1. In the case of learned interpolation, we concatenate four neighboring feature vectors and rely on phase information from the positional encoding (Section 4.3.2) to reconstruct high-frequency details.”).
In regards to claim 19, Vaidyanathan anticipates the claim language as in the consideration of claim 1.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 2-5, 18, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Vaidyanathan in view of US20240169652 (herein after referred to by its primary author, Fu).
In regards to claim 2, Vaidyanathan teaches the apparatus of claim 1, wherein the one or more processors are configured to train the machine-learning model using a loss function to adjust weights of the machine-learning model (Vaidyanathan Section 4.5 “We jointly optimize the feature pyramid and the decoder, using gradient descent with the ADAM [34] optimizer. Unless stated otherwise, our model is trained for 250k iterations. Our method can use and minimize an arbitrary image loss function.”).
Vaidyanathan fails to teach wherein the one or more processors are configured to train an encoder using a loss function to adjust weights of the encoder; input the texture into the encoder; and receive as output from the encoder the plurality of sets of features.
However, Fu teaches wherein the one or more processors are configured to train an encoder using a loss function to adjust weights of the encoder (Fu Paragraph [0061] “To complete the iteration, the geometry training engine 132 updates the values of any number of the learnable parameters of the geometry encoder 142 and the geometry decoder 144 and the texture training engine 136 updates the values of any number of the learnable parameters of the texture encoder 146 and the texture decoder 148 based on a goal of reducing the RGBD reconstruction loss.”); input the texture into the encoder; and receive as output from the encoder the plurality of sets of features (Fu Paragraph [0054] “The texture training engine 136 uses the texture encoder 146 to generate a texture surface representation of the RGB image 124(x) based on the RGB image 124(x), the camera metadata 128(x), and the key points included in the geometric surface representation. The texture surface representation includes the key points and a different texture feature vector for each of the key points.”).
Fu is considered to be analogous to the claimed invention because they both are in the same field of encoder/decoder structured neural networks involving texture. Therefore it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the system of Vaidyanathan to include the teachings of Fu, to provide the advantage of a system which only requires a single set of learning parameters, which saves on computational resources (Fu Paragraph [0008] “Because only a single neural network is trained and only a single set of values for the learnable parameters is stored, the amount of processing resources and the amount of memory required to generate 3D representations for multiple scenes can be reduced relative to what can be achieved using prior art scene reconstruction techniques.”)
In regards to claim 3, Vaidyanathan in view of Fu teaches the apparatus of claim 2, wherein the loss function is based on a difference between an output of the machine-learning model and an input to the encoder (Vaidyanathan Section 4.5 “To optimize our compressed representation, we explored several different loss functions, including SSIM [81], a version of VGG loss that supports texture sets [12], adversarial as well as L1 and L2 losses, and combinations thereof.” Examiner note: This reference teaches using L2 loss, which is defined as the difference between a predicted value and a ground truth value).
In regards to claim 4, Vaidyanathan in view of Fu teaches the apparatus of claim 2, wherein the encoder comprises a convolutional layer (Fu Paragraph [0133] “ some embodiments, the texture encoder 146 is a 2D convolution neural network that performs any number and/or types of image processing operations using any technically-feasible approach.”).
In regards to claim 5, Vaidyanathan in view of Fu teaches the apparatus of claim 2, wherein to train the encoder and the machine-learning model comprises to: generate, by the encoder, a first candidate plurality of sets of features; reconstruct, by the machine-learning model, one or more texels, at one or more resolutions, based on the first candidate plurality of sets of features; and adjust weights of the encoder and the machine-learning model based on the loss function (Fu Paragraph [0061] “To complete the iteration, the geometry training engine 132 updates the values of any number of the learnable parameters of the geometry encoder 142 and the geometry decoder 144 and the texture training engine 136 updates the values of any number of the learnable parameters of the texture encoder 146 and the texture decoder 148 based on a goal of reducing the RGBD reconstruction loss.”) (Vaidyanathan Section 4.5 “We jointly optimize the feature pyramid and the decoder, using gradient descent with the ADAM [34] optimizer. Unless stated otherwise, our model is trained for 250k iterations. Our method can use and minimize an arbitrary image loss function.” Examiner note: Vaidyanathan teaches using gradient descent, which is an iterative optimization technique that updates the models parameters based on the error (or loss function) after using the model to predict an output).
In regards to claim 18, Vaidyanathan in view of Fu teaches the apparatus of claim 1, wherein the one or more processors are configured to: input the texture into an encoder; and receive, as output from the encoder, the plurality of sets of features corresponding to the texture (Fu Paragraph [0054] “The texture training engine 136 uses the texture encoder 146 to generate a texture surface representation of the RGB image 124(x) based on the RGB image 124(x), the camera metadata 128(x), and the key points included in the geometric surface representation. The texture surface representation includes the key points and a different texture feature vector for each of the key points.”).
In regards to claim 20, Vaidyanathan in view of Fu renders obvious the claim language as in the consideration of claim 2.
Claim 11 is rejected under 35 U.S.C. 103 as being unpatentable over Vaidyanathan in view of US20240169478 (herein after referred to by its primary author, Ransom).
In regards to claim 11, Vaidyanathan teaches the apparatus of claim 1, wherein the one or more processors are configured to wherein to select the subset of grid points of the grid based on the second resolution being lower than the first resolution comprises to select the subset of grid points of the grid (Vaidyanathan Section 4.3 “In this section, we describe the first stage of decompression, which samples the grids of a feature level and prepares the input to the MLP, as shown in Figure 4 (c). In this stage, we first select a feature level based on the desired level of detail (LOD) (Table 1), and then resample both the grids in the feature level to the target resolution.”)
Vaidyanathan fails to teach select a striding level based on a ratio between the first resolution and the second resolution; select the subset of grid points of the grid based on the striding level.
However, Ransom teaches select a striding level based on a ratio between the first resolution and the second resolution; select the subset of grid points of the grid based on the striding level. (Ransom Figure 4; Paragraph [0050] “In the example of FIG. 4, the image tile 410 is a 10×10 array of pixel values (labeled 0 to 99). The 1st iteration of the downscaling operation converts the image tile 410 to an intermediate tile 420 with a downscaling factor equal to 3 (S.sub.1=3). More specifically, the 1st iteration of the downscaling operation preserves every 3rd pixel value of the image tile 410 to produce the intermediate tile 420.”; [0057] “For example, the image tile size 504 may be calculated as a function (y) of the base tile size (x), the number (K) of iterations, and the scaling factor (S.sub.j) associated with each iteration of the downscaling operation, where y represents the length of one side (such as a row or column) of an image tile 505 and x represents the length of one side of a downscaled tile 506:” Examiner note: Vaidyanathan describes sampling the grids in the feature pyramid to the target resolution. For example, if the input resolution is 1024x1024, and the target resolution is 512x512, then the grid would be resampled down to a size of 512x512. Ransom describes a down sampling operation where a distance between pixels (analogous to stride) is use which is based upon the size of the image tiles (analogous to a grid with grid points, as seen in figure 4). When the method of Ransom is applied to the disclosure of Vaidyanathan, the texture would be downsampled by sampling pixels that are a fixed distance apart based on the desired size of the downsampled image and the size of the input image)
Ransom is considered to be analogous to the claimed invention because they both are in the same field of encoder/decoder structured neural networks which perform sampling. Therefore it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the system of Vaidyanathan to include the teachings of Ransom, to provide the advantage of preserving image detail when downsampling (Ransom Paragraph [0054] “Aspects of the present disclosure recognize that, for any given set of image scaling parameters (such as base tile size, number of iterations, and scaling factors), an optimal image tile size can be selected such that all of the reconstructed pixel values in an upscaled tile can be interpolated based on two or more preserved pixel values. ”)
Claim 12 is rejected under 35 U.S.C. 103 as being unpatentable over Vaidyanathan in view of “Edge-Based Video Compression Texture Synthesis Using Generative Adversarial Network” (herein after referred to by its primary author, Zhu).
In regards to claim 12, Vaidyanathan teaches the apparatus of claim 1, wherein the machine-learning model comprises a multilayer perceptron architecture (Vaidyanathan Figure 4 “Network”).
Vaidyanathan fails to teach skip connections.
However, Zhu teaches skip connections (Zhu Figure 6(a) “Resblock” Examiner note: This figure shows the architecture of a resblock within a texture based generative adversarial network. This resblock includes a skip connection, where the input feature map is provided to the output, as can be seen by the top bolded arrow line).
Zhu is considered to be analogous to the claimed invention because they both are in the same field of encoder/decoder structured neural networks involving texture. Therefore it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the system of Vaidyanathan to include the teachings of Zhu, to provide the advantage of learning network parameters based o prior conditions (Zhu Section III F “Besides, motivated by the super-resolution work [47], we add the spatial feature transform (SFT) layer before each original convolutional layer in Resblock. All SFT layers receive the same edge conditions acquired by convolutional operations of Erec. The reason why we introduce SFT is that it can adaptively train network parameters based on prior conditions, where a pair of modulation parameters (α, β) are learned from conditions by independent convolutional layers in SFT, corresponding to intermediate feature maps.”)
Claims 16-17 are rejected under 35 U.S.C. 103 as being unpatentable over Vaidyanathan in view of US20170227765 (herein after referred to by its primary author, Mammou).
In regards to claim 16, Vaidyanathan teaches the apparatus of claim 1, but fails to teach a modem, coupled to one or more antennas, and coupled to the one or more processors, wherein the modem and the one or more antennas are configured to receive the texture.
However, Mammou teaches a modem, coupled to one or more antennas, and coupled to the one or more processors, wherein the modem and the one or more antennas are configured to receive the texture (Mammou Paragraph [0037] “The wireless VR system 300 is in a client-server configuration and includes a 3D display device, for example, a HMD 305 that is in wireless communication with a server 310, such as a desktop machine, using a wireless communication protocol 307. The wireless communication protocol 307 may use any number of wireless communication standards or wireless network configurations such as WiFi and the like. In another embodiment, the communications in the client-server configuration may be wired communications. The server 310 may include a game engine 315 that sends commands to a processing system 320 to perform texture-space rendering operations and feeds/transmits updated object motion data to a video decoder 335. The server 310 and processing system 320 may include central processing units (CPUs), graphics processing units (GPUs), advanced processing units (APUs) and the like which are configured to store and transmit texture-space rendering information in accordance with the embodiments disclosed herein.”).
Mammou is considered to be analogous to the claimed invention because they both are in the same field of texture rendering. Therefore it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the system of Vaidyanathan to include the teachings of Mammou, to provide the advantage of server side rendering of graphics/textures (Mammou Paragraph [0004] “The advancement in networking and increased bandwidth have allowed for the possibility of offloading rendering computations from client devices to remote servers, which can stream rendered graphics to the client. Under such a rendering scheme, a client may transmit input commands over a network and a server can perform rendering of a scene based on the input and transmit the rendered scene back to the client.”)
In regards to claim 17, Vaidyanathan in view of Mammou teaches the apparatus of claim 16, wherein the modem and the one or more antennas are integrated into one of a vehicle, an extra-reality device, or a mobile device (Mammou Paragraph [0037] “The wireless VR system 300 is in a client-server configuration and includes a 3D display device, for example, a HMD 305 that is in wireless communication with a server 310, such as a desktop machine, using a wireless communication protocol 307. The wireless communication protocol 307 may use any number of wireless communication standards or wireless network configurations such as WiFi and the like. In another embodiment, the communications in the client-server configuration may be wired communications. The server 310 may include a game engine 315 that sends commands to a processing system 320 to perform texture-space rendering operations and feeds/transmits updated object motion data to a video decoder 335. The server 310 and processing system 320 may include central processing units (CPUs), graphics processing units (GPUs), advanced processing units (APUs) and the like which are configured to store and transmit texture-space rendering information in accordance with the embodiments disclosed herein.”).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
“Compression of Synthesized Textures” teaches a method of compression texture and further describes real time decompression.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CALEB LOGAN ESQUINO whose telephone number is (703)756-1462. The examiner can normally be reached M-Fr 8:00AM-4:00PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Bee can be reached at (571) 270-5183. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CALEB L ESQUINO/ Examiner, Art Unit 2677
/ANDREW W BEE/ Supervisory Patent Examiner, Art Unit 2677
Prosecution Insights