DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 03/26/2025 is being considered by the examiner.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-4, 8-11 and 15-18 are rejected under 35 U.S.C. 103 as being unpatentable over HÖLLEIN (Höllein, L., Johnson, J., & Nießner, M. (2022, June). Stylemesh: Style transfer for indoor 3d scene reconstructions. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 6188-6198). IEEE. (Year: 2022) “Hollein”) in view of NICOLET (Nicolet, B., Jacobson, A., & Jakob, W. (2021). Large steps in inverse rendering of geometry. ACM Transactions on Graphics (TOG), 40(6), 1-13. (Year: 2021) “Nicolet”).
Regarding claim 1, Hollein teaches “A system comprising:
a hardware processor; and
a system memory storing a software code and a style transfer machine learning (ML) model;
the hardware processor configured to execute the software code to:” (We perform style transfer on reconstructed 3D meshes by synthesizing stylized textures. We compute style transfer losses on views of the scene and backpropagate gradients to the texture; Pg.6198, Col Fig.1 Desc); (We propose an optimization-based NST method, that converges in roughly 3 hours on a single RTX 3090 GPU; Pg.6205, Col 4.4 Para 1); (We encode the render pyramid ˆP with a pretrained VGG network [48] into the feature pyramid ˆ F; Pg.6201, Col 3.4 Para 1); (We optimize a stylized RGB texture T ∗ from all RGB images {Ik}N k=1 and a separate style image Is; Pg.6200, Col 3.1 Para 1);
receive an image and a style sample of a selected stylization for an original surface mesh depicted by the image;” (we want to create a texture that is a mixture of original RGB colors and a style image; Pg.6199, Col 3);
stylize, using the style transfer ML model, the style sample and the plurality of perspective images of the 3-D representation of the reparametrized surface mesh, the original surface mesh, to provide a stylized version of the original surface mesh having the selected stylization.” (We compute style transfer losses on views of the scene and backpropagate gradients to the texture; Pg.6198, Col Fig.1 Desc);
Hollein discloses a neural style transfer operating on reconstructed mesh. It discloses the technique NST that is optimization based and operates by CNN features. The NST technique requires a reconstructed scene mesh and RGB images captured from different poses. Hollein also discloses images captured a different camera poses and computes style transfer losses for the views. The losses compare the rendered mesh views with selected style image and the result update the mesh texture. While Hollein does not explicitly disclose a system, hardware processor or system memory and all that comprises, Hollein discloses a technique of neural style transfer, it would be obvious for a person skilled in the art to recognize that Hollein’s method of neural style transfer apply to a system with a hardware processor, and memory.
While, Hollein fails to teach “perform a view-independent reparametrization of the original surface mesh to provide a reparametrized surface mesh”, “render a three-dimensional (3-D) representation of the reparametrized surface mesh” and “generate, using a plurality of virtual cameras, a plurality of perspective images of the 3-D representation of the reparametrized surface mesh; and”.
Nicolet teaches “perform a view-independent reparametrization of the original surface mesh to provide a reparametrized surface mesh;” (interpreted as casting differentiable rendering into the framework of Sobolev preconditioned gradient descent or as a re-parameterization of the input geometry resembling differential coordinates; Pg.2 Col 1 Intro);
“render a three-dimensional (3-D) representation of the reparametrized surface mesh;” (Fig.4 showcases the rendering of 3D representations of parametrized mesh; Pg.7);
“generate, using a plurality of virtual cameras, a plurality of perspective images of the 3-D representation of the reparametrized surface mesh; and” (Fig.8 showcases a plurality of viewpoints; Pg.10); (that 𝑅 renders images with known camera poses; Pg.4, Col 3 Method);
Nicolet discloses a mesh representation based on Laplacian. The reparametrization is defined from the mesh vertex positions and connectivity rather than camera view, this constitutes a view independent reparametrization of the original mesh. Nicolet’s also discloses inverse rendering process that converts the differential representation back to the mesh vertex potions and renders the resulting mesh. Retrieving the vertex positions from the latent variable and calling the render on the updated shape teaches the rendering to an updated mesh.
A person skilled in the art would incorporate Nicolet’s reparametrization technique into Hollein’s multiview stylization process because differentiable rendering generates sparse or localized mesh gradients. Nicolet provides a differential mesh representation that propagates the gradients across the surface and permits more stable mesh updates.
Hollein and Nicolet are analogous art as both of them are related to style transfer of mesh and image processing.
The motivation for the above is to have a more accurate stylization process to generate localized mesh gradients.
Therefore, it would have been obvious for an ordinary skilled person in the art before the effective filing date of claimed invention to have modified Hollein by perform a view-independent reparametrization of the original surface mesh to provide a reparametrized surface mesh, render a three-dimensional (3-D) representation of the reparametrized surface mesh and generate, using a plurality of virtual cameras, a plurality of perspective images of the 3-D representation of the reparametrized surface mesh as taught by Nicolet.
Regarding claim 2, Hollein further teaches “The system of claim 1, wherein the hardware processor is further configured to execute the software code to:
output an image depicting the stylized version of the surface mesh.” (See Figure 1, showcases the output image of a mesh styled texture image.);
Regarding claim 3, Hollein further teaches “The system of claim 1, wherein the style transfer ML model comprises a neural network (NN).” (Neural Style Transfer (NST) shows great results for stylization of images);
Hollein discloses a Neural Style Transfer, this teaches the neural network from a styler transfer learning model.
Regarding claim 4, while Hollein fails to teach the limitation of claim 4, Nicolet further teaches “The system of claim 1, wherein the view-independent reparametrization of the original surface mesh is performed using a Laplace Beltrami operator.” (derived by integrating the Laplace-Beltrami operator; Pg.3, Col 2.4);
Nicolet discloses Laplace-Beltrami operator which teaches the claimed subject matter. Nicolet’s reparametrization technique is defined from the mesh vertex positions and connectivity rather than camera view, this constitutes a view independent reparametrization of the original mesh.
The motivation for the above is to have a more accurate stylization process to generate localized mesh gradients.
Therefore, it would have been obvious for an ordinary skilled person in the art before the effective filing date of claimed invention to have modified Hollein by wherein the view-independent reparametrization of the original surface mesh is performed using a Laplace Beltrami operator as taught by Nicolet.
Claim 8 is directed to a method and its limitations are similar in scope and functions performed by the system of claim 1. Therefore, claim 8 limitations are also rejected with the same rationale as regarding claim 1.
Claim 9 is directed to a method and its limitations are similar in scope and functions performed by the system of claim 2. Therefore, claim 9 limitations are also rejected with the same rationale as regarding claim 2.
Claim 10 is directed to a method and its limitations are similar in scope and functions performed by the system of claim 3. Therefore, claim 10 limitations are also rejected with the same rationale as regarding claim 3.
Claim 11 is directed to a method and its limitations are similar in scope and functions performed by the system of claim 4. Therefore, claim 11 limitations are also rejected with the same rationale as regarding claim 4.
Claim 15 is directed to a computer-readable non-transitory storage medium and its limitations are similar in scope and functions performed by the system of claim 1. Therefore, claim 15 limitations are also rejected with the same rationale as regarding claim 1.
Claim 16 is directed to a computer-readable non-transitory storage medium and its limitations are similar in scope and functions performed by the system of claim 2. Therefore, claim 16 limitations are also rejected with the same rationale as regarding claim 2.
Claim 17 is directed to a computer-readable non-transitory storage medium and its limitations are similar in scope and functions performed by the system of claim 3. Therefore, claim 17 limitations are also rejected with the same rationale as regarding claim 3.
Claim 18 is directed to a computer-readable non-transitory storage medium and its limitations are similar in scope and functions performed by the system of claim 4. Therefore, claim 18 limitations are also rejected with the same rationale as regarding claim 4.
Claim(s) 5, 12 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over HÖLLEIN in view of NICOLET in further view HUANG (Huang, J., Shi, X., Liu, X., Zhou, K., Wei, L. Y., Teng, S. H., ... & Shum, H. Y. (2006). Subspace gradient domain mesh deformation. In ACM SIGGRAPH 2006 Papers (pp. 1126-1134). (Year: 2006) “Huang”).
Regarding claim 5, while Hollein and Nicolet fails to teach the limitation of claim 5, Huang teaches “The system of claim 1, wherein stylizing the original surface mesh to provide the stylized version of the original surface mesh having the selected stylization is performed subject to a volumetric constraint.” (The constraints we introduce include the volume constraint for volume preservation; Pg.1126, Col 1 Intro Para 2);
Huang discloses volume constraint for volume preservation which teaches the claimed subject matter. Huang’s teaches of adding volume preservation constraint to an objective used to deform a mesh, thus applying that constraint to the geometry optimization taught by Hollein and Nicolet combination would cause the selected stylization to deform the mesh while restricting the undesired changes to its enclosed volume. It would be obvious to a skilled person in the art would combine to prevent large neural style gradients from collapsing portions of the surface mesh.
Hollein, Nicolet and Huang are analogous art as they are related to mesh deformation and rendering.
The motivation for the above is to have a more accurate calculation of constraints and prevent large neural style gradients from collapsing portions of the surface mesh.
Therefore, it would have been obvious for an ordinary skilled person in the art before the effective filing date of claimed invention to have modified Hollein and Nicolet by stylizing the original surface mesh to provide the stylized version of the original surface mesh having the selected stylization is performed subject to a volumetric constraint as taught by Huang.
Claim 12 is directed to a method and its limitations are similar in scope and functions performed by the system of claim 5. Therefore, claim 12 limitations are also rejected with the same rationale as regarding claim 5.
Claim 19 is directed to a computer-readable non-transitory storage medium and its limitations are similar in scope and functions performed by the system of claim 5. Therefore, claim 19 limitations are also rejected with the same rationale as regarding claim 5.
Claim(s) 6 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over HÖLLEIN in view of NICOLET in further view REIMANN (Reimann, M., Buchheim, B., Semmo, A., Döllner, J., & Trapp, M. (2022). Controlling strokes in fast neural style transfer using content transforms: M. Reimann et al. The Visual Computer, 38(12), 4019-4033. (Year: 2022) “Reimann”) in further view of ZHANG (Zhang, E., Mischaikow, K., & Turk, G. (2006). Vector field design on surfaces. ACM Transactions on Graphics (ToG), 25(4), 1294-1326. (Year: 2006) “Zhang”).
Regarding claim 6, while Hollein and Nicolet fails to teach the limitations of claim 6, Reimann teaches “The system of claim 1, wherein the hardware processor is further configured to execute the software code to:
receive, from a system user, flow field data specifying a plurality of different planar orientations of the style sample;” (supports users to adjust both the size and orientation of style elements, such as brushstrokes and texture patches... For additional level-of-control, we propose a network agnostic method for stroke-orientation adjustment by utilizing the rotation-variance of Convolutional Neural Networks (CNNs).; Pg.1, Col Abstract);
Reimann discloses an interface that users can control style elements and enables the user to adjust the orientation of style elements. By the rotation variance of the neural network, it performs orientation adjustments thus applying differently rotated orientations of the style information.
While Reimann fails to teach “wherein stylizing the original surface mesh to provide the stylized version of the original surface mesh having the selected stylization further uses the flow field data”.
Zhang teaches “wherein stylizing the original surface mesh to provide the stylized version of the original surface mesh having the selected stylization further uses the flow field data.” (example-based texture synthesis makes use of a vector field to define local texture orientation and scale. Pg.1295, Para 1);
Zhang discloses the surface flow filed structure which is a vector field define over different surface locations and used to guide texture synthesis. Combing Reimann and Zhang would result in a user defined orientation field associated with the mesh, where the direction of each location determines which planner orientation of the style samples is used during the stylization processes.
A person skilled in the art to use Zhang’s surface vector field to organize Reimann’s local orientation controls. Using a flow field would algin the synthesized strokes with different parts of the surface and provide finer artistic control.
Hollein, Nicolet Reiman and Zhang are analogous art as they are related to mesh processes and stylization.
The motivation for the above is to have a more efficient control of stylization and orientation process to provide finer artistic control.
Therefore, it would have been obvious for an ordinary skilled person in the art before the effective filing date of claimed invention to have modified Hollein and Nicolet by receive, from a system user, flow field data specifying a plurality of different planar orientations of the style sample as taught by Reimann and by stylizing the original surface mesh to provide the stylized version of the original surface mesh having the selected stylization further uses the flow field data as taught by Zhang.
Claim 13 is directed to a method and its limitations are similar in scope and functions performed by the system of claim 6. Therefore, claim 13 limitations are also rejected with the same rationale as regarding claim 6.
Claim(s) 7 and 14 are rejected under 35 U.S.C. 103 as being unpatentable over HÖLLEIN in view of NICOLET in further view of MAX REIMANN (Reimann, M., Klingbeil, M., Pasewaldt, S., Semmo, A., Trapp, M., & Döllner, J. (2019). Locally controllable neural style transfer on mobile devices: M. Reimann et al. The Visual Computer, 35(11), 1531-1547. (Year: 2019) “Max Reimann”).
Regarding claim 7, Hollein further teaches “wherein stylizing the original surface mesh to provide the stylized version of the original surface mesh omits the selected stylization from the one or more masked regions of the surface mesh.” (We compute style transfer losses on views of the scene and backpropagate gradients to the texture; Pg.6198, Col Fig.1 Desc);
Hollein discloses 3D mesh stylization pipeline, that renders pose dependent views of the mesh, evaluates neural style losses on those views and backpropagates the image space gradients for the corresponding mesh texture.
However, while Hollein and Nicolet fails to teach the limitation “receive, from a system user, masking data identifying one or more masked regions of the surface mesh from which the selected stylization is to be omitted”
Max Reimann teaches “The system of claim 1, wherein the hardware processor is further configured to execute the software code to:
receive, from a system user, masking data identifying one or more masked regions of the surface mesh from which the selected stylization is to be omitted;” (neural style transfer techniques that can be locally controlled by on-screen painting using image masking; Pg.1532, Col 1 Para 3); (spatial control over the style transfer, the training or configuration of the network can be limited to user-defined regions of the style and content image; Pg.1534, Col 4.1 Para 1); (Location-based control over NSTs can be achieved by segmenting the style and content image into different local control masks.... the style loss term is adjusted to include masks; Pg. 1535, Col 4.2.1 Para 1);
Max Reimann discloses user-controlled masking functionality that permits a user to paint local masks, limit style processing to user defined regions and modify the neural style loss based on the masks. The mask determines a particular style is permit ed to appear and regions outside the mask do not receive the selected style.
It would be obvious to person skilled in the art to incorporate Max Reimann’s controllable masking into Hollein because Hollein applies style, losses across the rendered scene and Max Reimann expressly addresses the need for artistic control allowing suer to define masks and user defined regions.
Hollein, Nicolet and Max Reimann are analogous art as they are related to mesh processes and stylization.
The motivation for the above is to have a preservation of user defined regions for user artistic control.
Therefore, it would have been obvious for an ordinary skilled person in the art before the effective filing date of claimed invention to have modified Hollein and Nicolet by receive, from a system user, masking data identifying one or more masked regions of the surface mesh from which the selected stylization is to be omitted as taught by Max Reimann.
Claim 14 is directed to a method and its limitations are similar in scope and functions performed by the system of claim 7. Therefore, claim 14 limitations are also rejected with the same rationale as regarding claim 7.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Mishra, S., & Granskog, J. (2022). Clip-based neural neighbor style transfer for 3d assets. arXiv preprint arXiv:2208.04370. (Year: 2022) – Discloses a method for transferring the style from a set of images to a 3D object. The texture appearance of an asset is optimized with a differentiable renderer in a pipeline based on losses using pretrained deep neural networks.
US-10872399-B2 (Li) – Discloses photorealistic image stylization concerns transferring style of a reference photo to a content photo with the constraint that the stylized photo should remain photorealistic.
US-9710965-B2 (Beeler) – Discloses receiving a plurality of images of a hairstyle in an n dimensional space at a plurality of different angles. Further, the process, computer program product, and apparatus generate a mesh surface in an n−1 dimensional space. In addition, the process, computer program product, and apparatus combine color data from the plurality of images at the plurality of different angles with mesh geometry data of the mesh surface.
US-20230215062-A1 (Gudkov) – Discloses an image stylization system accesses a set of images corresponding to a target domain style, generates a set of paired images using a first machine learning model, analyze the generated set of paired images using a second machine learning model trained to analyze the generated set of paired images based on a plurality of protected feature criteria, determines a set of image transformations for the generated set of pairs, generates a transformed set of paired images by performing the set of image transformations on the set of paired images, and generates stylized images corresponding to the target domain style using a supervised image translation model trained on the transformed set of paired images.
US-20230326157-A1 (Shayani) – Discloses a technique for performing style transfer. The technique includes generating an input shape representation that includes a plurality of points near a surface of an input three-dimensional (3D) shape, where the input 3D shape includes content-based attributes associated with an object.
US-20230326158-A1 (Shayani) – Discloses technique for training a machine learning model to perform style transfer. The technique includes applying one or more augmentations to a first input three-dimensional (3D) shape to generate a second input 3D shape. The technique also includes generating, via a first set of neural network layers, a style code based on a first latent representation of the first input 3D shape and a second latent representation of the second input 3D shape.
US-20230376656-A1 (Da Costa De Azevedo) – Discloses a software code and a machine learning (ML) model trained to apply a stylization to an image. To provide a stylized content having the desired stylization, wherein stylizing includes applying an exponential moving average (EMA) temporal smoothing algorithm to sequential image pairs of the first sequence of images to generate a second sequence of images providing a depiction of the content having the desired stylization, and output the stylized content having the desired stylization.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to BRIGITER D PROTAZI whose telephone number is (571)272-7995. The examiner can normally be reached Monday - Friday 7:30-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Said A Broome can be reached at 5712722931. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/B.D.P./Examiner, Art Unit 2612
/Said Broome/Supervisory Patent Examiner, Art Unit 2612
Prosecution Insights