DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This action is in response to the Amendment filed on 9/8/2026.
Claims 1-20 are pending. Claims 1, 8, 16, 18, 19, 20 have been amended.
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claims 1, 3, 4, 5, 6, 16 of Application 18/905,841 (Instant Application) are rejected on the ground of nonstatutory double patenting as being unpatentable over claims (7, 6, 12), (7, 10), 7, 7, 12, (1, 5, 6) of Application 17/467,792 (Now is US 12112445 B2) respectively. Although the claims at issue are not identical, they are not patentably distinct from each other because they both claim substantially the same subject matter and limitations as explained below.
. Claim 1 of the Instant Application is determined to be obvious in light of Claim 7 (with Claims 6 / 12) of the 17/467,792 (Now is US 12112445 B2), based on the similar limitations identified in the table below:
Instant Application — Claim 1
17/467,792 — Claim 7 (in view of Claims 6, 12)
(Claim 1) A processor, comprising: one or more circuits to:
(Claim 7) A processor, comprising: one or more circuits to:
(Claim 1) determine one or more source geometric segments and one or more source textures corresponding to a source three-dimensional (3D) object model;
(Claim 7) process, using a trained first neural network, a source object to determine one or more source geometric features of the source object; determine … a source texture corresponding to the source object
(Claim 1) determine one or more target geometric segments and one or more target textures corresponding to a semantically different target 3D object model;
(Claim 7) process, using the trained first neural network, a target object to determine one or more target geometric features of the target object; … a target texture corresponding to the target object (Claim 12) wherein a number of segments for the segmenting is based, at least in part, on an object type for at least one of the source object or the target object
(Claim 1) generate, using a neural network trained and jointly optimized over geometry and texture to output data corresponding to geometric style transforms between semantically different objects, one or more modifications to at least one of the one or more source geometric segments and at least one of the one or more source textures based, at least in part, on at least one of one or more corresponding target geometric segments of the one or more target geometric segments for the one or more source geometric segments and one or more corresponding target textures of the one or more target textures for the one or more source textures;
(Claim 7) adjust a selected source geometric feature based, at least in part, on the one or more target geometric features; determine an output texture by adjusting a source texture corresponding to the source object based, at least in part, on a target texture corresponding to the target object;
(Claim 1) generate a stylized 3D object model based, at least in part, on the one or more modifications.
(Claim 7) generate, based at least in part on the adjusted selected source geometric feature and the output texture, an output object.
Although the claims at issue are not identical, they are not patentably distinct. Claim 7 of the Now is US 12112445 B2 discloses "adjusting a selected source geometric feature". While Claim 1 of the Instant Application specifies "generating... one or more modifications" and Claim 4 specifies "applying a 3D affine transformation" , these are merely specific types of adjustments. A "geometric segment" as claimed in the Instant Application is a species of the "geometric feature" claimed in the 17/467,792 (Now is US 12112445 B2). It would have been obvious to one of ordinary skill in the art to utilize specific geometric transformations (like affine transforms) to perform the adjustment step taught in the 17/467,792 (Now is US 12112445 B2). With respect to the additional limitation “and jointly optimized over geometry and texture to output data corresponding to geometric style transforms between semantically different objects” in the instant application., although they are not in the 17/467.792, however, prior art Segu teaches in Page 5, § 3.3 teaches those limitations. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the 17/467.792 combination in view of Segu such that the geometric-style-transfer neural network is trained jointly with texture CNN under a unified multi-term objective in the manner for its own multi-loss joint training, such that the resulting single learned model is trained and jointly optimized over both geometry and texture to output data corresponding to geometric style transforms between semantically different objects. The motivation is to enable style transfer that generalizes across semantic categories rather than only within a single category.
Claim 3 of the Instant Application is determined to be obvious in light of Claim (9, 10) of the 17/467,792 (Now is US 12112445 B2) based on the similar limitations identified in the table below:
Instant Application — Claim 3
17/467,792 — Claims 9, 10
(Claim 3) generate a plurality of images depicting a plurality of views of the stylized 3D object model;
(Claim 9) generate a plurality of views of the output object, each view of the plurality of views being at a different angle;
(Claim 3) apply a mask to at least one image of the plurality of images to identify one or more background pixels;
(Claim 9) determine, using the plurality of views, one or more features of the output object associated with a background.
(Claim 3) remove, from the stylized 3D object model, one or more features computed from the one or more background pixels.
(Claim 10) adjust the output object, based at least in part on the one or more features, to remove the one or more features associated with the background.
The limitations are substantially identical. Claim 3 of the Instant Application identifies "background pixels" via a "mask" , which is the functional equivalent of determining "features... associated with a background" as taught in Claim 9 of the 17/467,792 (Now is US 12112445 B2). Removing features based on background pixels is not patentably distinct from removing features associated with the background.
Claim 4 of the Instant Application is determined to be obvious in light of Claim 7 of the 17/467,792 based on the similar limitations identified in the table below:
Instant Application — Claim 4
17/467,792 — Claim 7
(Claim 4) the one or more circuits are to generate the one or more modifications by applying a 3D affine transformation to the one or more source geometric segments.
(Claim 7) adjust a selected source geometric feature based, at least in part, on the one or more target geometric features.
Although the claims at issue are not identical, they are not patentably distinct. Claim 7 of the 17/467,792 recites the step of adjusting a selected source geometric feature. While Instant Claim 4 further specifies that the modification is performed by "applying a 3D affine transformation," a 3D affine transformation is merely a specific type of adjustment. It would have been obvious to one of ordinary skill in the art to utilize a 3D affine transformation to perform the adjustment step taught in Claim 7 of the 17/467,792 (Now is US 12112445 B2).
Claim 5 of the Instant Application is determined to be obvious in light of Claim 7 of the 17/467,792 based on the similar limitations identified in the table below:
Instant Application — Claim 5
17/467,792 — Claim 7
(Claim 5) determine, for the one or more source geometric segments, one or more ellipsoid approximations.
(Claim 7) adjust a selected source geometric feature based, at least in part, on the one or more target geometric features.
Although the claims at issue are not identical, they are not patentably distinct. Determining an ellipsoid approximation for a source geometric segment is a well-known intermediate representation used in adjusting a selected source geometric feature. It would have been obvious to one of ordinary skill in the art to determine ellipsoid approximations for the source geometric features of the 17/467,792 as an implementation detail of the adjustment step.
Claim 6 of the Instant Application is determined to be obvious in light of Claim 12 of the 17/467,792 (Now is US 12112445 B2) based on the similar limitations identified in the table below:
Instant Application Claim 6
17/467,792 Claim 12
(Claim 6) determine an object type associated with the source 3D object model and the target 3D object model;
(Claim 12) wherein a number of segments for the segmenting is based... on an object type...
(Claim 6) determine, based at least in part on the object type, a number of segments...
(Claim 12) [The step of determining a number of segments based on object type].
Claim 16 of the Instant Application is determined to be obvious in light of Claim (1, 5, 6) of the 17/467,792 (Now is US 12112445 B2) based on the similar limitations identified in the table below:
Instant Application — Claim 16
17/467,792 — Claim 1 (in view of Claims 5, 6)
(Claim 16) A method, comprising:
(Claim 1) A method, comprising:
(Claim 16) determining one or more output geometric features for an output object corresponding to one or more source geometric features identified from a source object modified based on one or more target geometric features identified from a semantically different target object;
(Claim 1) processing, using a trained first neural network, a source object to determine one or more source geometric features of the source object; processing, using the trained first neural network, a target object to determine one or more target geometric features of the target object; adjusting a selected source geometric feature based, at least in part, on the one or more target geometric features; (Claim 6) wherein a number of segments for the segmenting is based, at least in part, on an object type for at least one of the source object or the target object.
(Claim 16) determining an output texture for the output object corresponding to a source texture of the source object modified based on a target texture of the target object; and
(Claim 1) determining an output texture by adjusting a source texture corresponding to the source object based, at least in part, on a target texture corresponding to the target object; and
(Claim 16) generating the output object, based on the output one or more geometric features and the output texture, using one or more trained models jointly optimized over both geometry and texture.
(Claim 1) generating, based at least in part on the adjusted selected source geometric feature and the output texture, an output object.
Although the claims at issue are not identical, they are not patentably distinct from each other and has are substantially identical limitations. discloses almost every single limitations the same, but the instant (child) applicant has broader scope, although the parent case has additional limitations by neural network, but those can be fully anticipated by the instant application based on the similarity in between the Claim 1 from both applications. With respect to the additional limitation “and jointly optimized over geometry and texture to output data corresponding to geometric style transforms between semantically different objects” in the instant application., although they are not in the 17/467.792, however, prior art Segu teaches in Page 5, § 3.3 teaches those limitations. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the 17/467.792 combination in view of Segu such that the geometric-style-transfer neural network is trained jointly with texture CNN under a unified multi-term objective in the manner for its own multi-loss joint training, such that the resulting single learned model is trained and jointly optimized over both geometry and texture to output data corresponding to geometric style transforms between semantically different objects. The motivation is to enable style transfer that generalizes across semantic categories rather than only within a single category.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 2, 4, 8, 9, 12, 16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Fu et al. (US 20210407200 A1, hereinafter Fu) in view of Ranade et al. (US 20220122311 A1, hereinafter Ranade), further in view of Segu et al - 3DSNet Unsupervised Shape-to-Shape 3D Style Transfer - 202105 – arXiv, hereinafter Segu).
Regarding Claim 1, Fu teaches a processor, comprising: one or more circuits to: (Fu, Paragraph [0082], "The model is trained for 200 epochs with batch size equaling to 8 on a single GTX 1080Ti GPU" <read on one or more circuits of a processor implementing the model>), determine one or more source geometric segments [[and one or more source textures]] corresponding to a source three-dimensional (3D) object model (Fu, Paragraph [0068], " Given two meshes Mid=M(α1; β1; θ1) and Mpose=M(α2; β2; θ2), the goal of the model is to transfer the pose to the identity mesh" <read on source 3D object model corresponding to Mid>; Paragraph [0066], "These per-vertex features are then concatenated with the vertex locations in the target mesh providing identity" <read on determining source geometric segments corresponding to the vertex coordinates of Mid>), determine one or more target geometric segments [[and one or more target textures corresponding to a semantically different target 3D object model]] (Fu, Paragraph [0068], " Given two meshes Mid=M(α1; β1; θ1) and Mpose=M(α2; β2; θ2), the goal of the model is to transfer the pose to the identity mesh" <read on target 3D object model corresponding to Mpose>; Paragraph [0070], "The encoder E takes Mpose vertices coordinates through pose feature extractor" <read on determining target geometric segments as the vertex coordinates of Mpose>), generate, using a neural network trained [[and jointly optimized over geometry and texture]] to output data corresponding to geometric style transforms [[between semantically different objects,]] one or more modifications to at least one of the one or more source geometric segments [[and at least one of the one or more source textures]] based, at least in part, on at least one of one or more corresponding target geometric segments of the one or more target geometric segments for the one or more source geometric segments [[and one or more corresponding target textures of the one or more target textures for the one or more source textures]] (Fu, Paragraph [0011], "In this invention it is proposed an efficient deep learning based architecture to efficiently transfer the pose from source meshes to target meshes. The whole network is designed as generalizing the style transfer in the image domain to deal with points" <read on neural network trained to output data corresponding to geometric style transforms>; Paragraph [0066], "These per-vertex features are then concatenated with the vertex locations in the target mesh providing identity, which is fed into the style transfer decoder consists of SPAdaIn ResBlocks" <read on modifications to source geometric segments based on target geometric segments>; Fu, FIG. 4(b) Paragraph [0071], " in SPAdaIN the external data Mid is fed into 2 different 1×1 convolution layers to produce the modulation parameters γ and β. The parameters are multiplied and added to the normalized feature." <read on geometric style transform parameters γ and β applied to the source geometric segments>), generate a stylized 3D object model based, at least in part, on the one or more modifications (Fu, Paragraph [0066], "The final output mesh inherits the pose from the source mesh and the identity from the target" <read on stylized 3D object model>; Paragraph [0068], "the goal of the model is to transfer the pose to the identity mesh by producing output mesh Moutput={circumflex over (M)}(a1; β2; θ1)" <read on stylized 3D object model>).
But Fu does not explicitly disclose one or more source textures corresponding to a source three-dimensional (3D) object model; one or more target textures corresponding to a … target 3D object model; at least one of the one or more source textures; and one or more corresponding target textures of the one or more target textures for the one or more source textures.
However, Ranade teaches one or more source textures corresponding to a source three-dimensional (3D) object model (Ranade, Paragraph [0048], " At 303, the low resolution images ILR are applied to a mesh M of the 3D model to obtain the low resolution texture atlas TLR" <read on source textures corresponding to the source 3D object model>; Paragraph [0041], "The forward 3D texture formation model outputs a low resolution texture atlas TLR, which enables rendering of a 3D textured model at various views"), one or more target textures corresponding to a … target 3D object model (Ranade, Paragraph [0043], "a ground truth texture atlas T and a geometric mesh are provided as input" <read on target textures corresponding to the target 3D object model>; Paragraph [0043], " exture maps from the ground truth texture atlas T are applied to various camera locations i, with known camera intrinsic and extrinsic properties, to form a 3D ground truth model that is made up of a collection of ground truth images Ii GT "), and modifications to … at least one of the one or more source textures based, at least in part, on … one or more corresponding target textures of the one or more target textures for the one or more source textures (Ranade, Paragraph [0063], "The low resolution texture atlas TLR may first be passed through the CNN f(θ, . . .) to obtain the network output fHR" <read on modifying source texture TLR to produce output fHR>; Paragraph [0059], "a ground truth texture atlas T may be input to a differentiable renderer 403 to render ground truth images I0 GT, I1 GT, I2 GT, and IN−1 GT at N different camera locations. Figure US20220122311A1-20220421-P00002 rendering may then be calculated between the rendered pairs of images based on Equation (4), above, and back-propagated through the network via differentiable renderers 402 and 403." <read on modifying source texture based on target texture T via the rendering loss>).
Fu and Ranade are analogous since both deal with neural-network-based transformation of 3D object models — Fu transferring pose/style between meshes using a neural network with a style transfer decoder (SPAdaIN), and Ranade determining and manipulating texture atlases for 3D models using a CNN trained against a target/ground-truth texture via a differentiable rendering loss. Fu provided a way of transferring geometric pose/style from a source mesh to a target mesh through a style transfer network producing a stylized output mesh. Ranade provided a way of processing a source texture atlas (TLR) against a reference/target texture (T) via a CNN with differentiable rendering to produce an output texture applied to the 3D model. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate Ranade's source-texture and target-texture determination and rendering-loss-based texture modification into Fu's geometric style transfer system such that the source and target 3D objects also have associated source and target textures determined from each object, and such that Fu's system further modifies at least one source texture based on the corresponding target texture. The motivation is to produce higher-quality stylized 3D models that exhibit both the geometric and the visual (textural) characteristics of the target, as Ranade discusses in its FIG. 1 Description and the purpose of rendering-loss-based texture manipulation in Paragraph [0059].
But the combination of Fu, Ranade and Segu does not explicitly disclose corresponding to a semantically different target 3D object model; a neural network trained and jointly optimized over geometry and texture; and geometric style transforms between semantically different objects.
However, Segu teaches corresponding to a semantically different target 3D object model (Segu, Page 1, Abstract, " we propose the first learning-based approach for style transfer between 3D objects based on disentangled content and style representations. The proposed method can synthesize new 3D shapes both in the form of point clouds and meshes, combining the content and style of a source and target 3D model to generate a novel shape that resembles in style the target while retaining the source content" <read on source 3D model and target 3D model>; Segu, Page 3, § 3.1, "Let X1 and X2 be two domains of 3D shapes (e.g. straight chair and armchair)" <read on source and target 3D object models from semantically different domains>; Segu, Page 7, § 4.3, "we validate hence our approach on two of the most populated object categories in ShapeNet and identify for each two subcategories as corresponding stylistic variations: Chairs (armchair↔ straight chair), Planes (jet↔ fighter aircraft). We further test our method on the categories hippos and horses in SMAL" <read on target 3D object model from a semantically different category than the source, e.g., target = armchair while source = straight chair, or target = hippo while source = horse>), a neural network trained and jointly optimized over geometry and [[texture]] to output data corresponding to geometric style transforms (Segu, Page 5, § 3.3, "We jointly train the encoders Ec, Es1, Es2, the decoder G, the mapping functions M1, M2 and the discriminators D1 and D2 to optimize the total loss" <read on trained and jointly optimized over geometry>; Segu, § 3.3, "Ltot = λrec(Lx1rec+Lx2rec) + λadv(Lx1adv+Lx2adv) + λcycle(Lx1cycle+Lx2cycle)" <read on jointly optimized over geometry >; Segu, page 4, § 3.2, "The overall architecture is thus an autoencoder through which we estimate both reconstruction and style translation functions" <read on a single jointly optimized neural network outputting data corresponding to geometric style transforms>), and geometric style transforms between semantically different objects (Segu, Page 1, "3DSNet learns to combine the content of one shape with the style of another. We illustrate style transfer results for the pairs armchair↔ straight chair and jet↔ fighter aircraft" <read on geometric style transforms between semantically different objects>; Segu, Page 1, "transferring the style between 3D objects can be interpreted as transferring the peculiar traits of a target shape to a source shape while preserving the underlying pose and structure of the latter" <read on geometric style transforms between semantically different objects>; Segu, Page 7, § 4.3, "hippos and horses in SMAL" and "The SMAL dataset is created by fitting the SMPL body model to different animal categories" <read on style transforms between semantically different objects, i.e., different animal categories>).
Segu is analogous art to Fu and Ranade because all three references address neural-network-based style transfer applied to 3D object models — Fu transferring geometric style between source and target meshes with a style-transfer decoder, Ranade transferring texture between source and target 3D texture atlases via a rendering-loss-trained CNN, and Segu transferring style between source and target 3D shape domains via a jointly trained encoder-decoder architecture with disentangled content and style representations. Segu provided a way of applying source-to-target 3D style transfer between different semantic domains (e.g., armchair vs straight chair, hippo vs horse) using a single neural network trained under a unified joint objective combining reconstruction, adversarial and cycle-consistency losses. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the Fu-Ranade combination in view of Segu such that (i) the source and target 3D object models are drawn from semantically different categories/domains, in the manner Segu expressly teaches for armchair vs straight chair, jet vs fighter aircraft, and hippo vs horse pairs, and (ii) the geometric-style-transfer neural network of Fu is trained jointly with Ranade's texture CNN under a unified multi-term objective in the manner Segu teaches for its own multi-loss joint training, such that the resulting single learned model is trained and jointly optimized over both geometry (from Fu) and texture (from Ranade) to output data corresponding to geometric style transforms between semantically different objects. The motivation is to enable style transfer that generalizes across semantic categories rather than only within a single category, and to avoid the loss of consistency that arises when the geometry and texture pipelines are trained in isolation — an approach expressly endorsed by Segu at § 3.2, "The overall architecture is thus an autoencoder through which we estimate both reconstruction and style translation functions," and § 3.3, "We jointly train … to optimize the total loss."
Regarding Claim 2, the combination of Fu, Ranade and Segu teaches the invention in Claim 1.
The combination further teaches one or more circuits are further to: generate a stylized texture (Ranade, Paragraph [0063], "The low resolution texture atlas TLR may first be passed through the CNN f(θ, . . . ) to obtain the network output T̂HR <read on generating a stylized texture T̂HR>”; it is noted that the CNN being trained to style the source texture TLR toward the target texture T, such that T̂HR constitutes the stylized texture output); and apply the stylized texture to the stylized 3D object model (Ranade, Paragraph [0038], "The texture atlas is input to the texture atlas super-resolution unit 104 to generate a 3D model").
Fu and Ranade are analogous since both are directed to neural-network-based processing and transformation of 3D object models which geometrically deforms the source 3D mesh to match the target's geometric style, producing a stylized 3D mesh, while transforms the source texture atlas (TLR) toward the target's visual appearance using a CNN with differentiable rendering, producing a stylized texture for a 3D model.
Fu provided a way of generating a stylized 3D object model (Moutput) with modified geometry reflecting the target's pose/shape, but without any associated texture. Ranade provided a way of generating a stylized texture (T̂HR) by passing a source texture (TLR) through a CNN trained to minimize a rendering loss relative to the target reference texture (T), and then applying the resulting stylized texture to the 3D model's geometric mesh. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate Ranade's stylized texture generation and texture application steps into Fu's geometric style transfer framework, such that the stylized 3D object model produced by Fu further provided with a stylized texture (T̂HR) generated by Ranade and applied to the output mesh. The motivation is that a stylized 3D object model with only modified geometry but no texture would be visually incomplete and of limited practical use; adding a stylized texture as taught by Ranade directly addresses this limitation by giving the output model a visual appearance consistent with the target's style, as Ranade discusses in its FIG. 1 and FIG. 4B descriptions.
Regarding Claim 4, the combination of Fu, Ranade and Segu teaches the invention in Claim 1.
The combination further teaches one or more circuits are to generate the one or more modifications by applying a 3D affine transformation to the one or more source geometric segments (Fu, Paragraph [0007], "Alternatively, some works leverage affine transformations to generate target shapes"; FIG. 4(b); Paragraph [0071], "in SPAdaIN the external data Mid is fed into 2 different 1×1 convolution layers to produce the modulation parameters γ and β. The parameters are multiplied and added to the normalized feature." <read on affine transformation applied to source geometric segments Mid>), (Fu, FIG. 4(b) Description equation, "SPAdaIN(h, M) = γvi(M)·((hn,c,vi − μn,ci)/σn,ci) + βvi(M)" <read on affine transformation with γ and β applied to normalized features of source geometric segments>).
Regarding Claim 8, Fu teaches a processor, comprising: one or more circuits to: (Fu, Paragraph [0082], "The model is trained for 200 epochs with batch size equaling to 8 on a single GTX 1080Ti GPU" <read on one or more circuits of a processor implementing the model>), segment a source three-dimensional (3D) object into one or more source components (Fu, Paragraph [0065], "Given Mid and Mpose as input, the model produces mesh transferred to new posture" <read on source 3D object Mid processed as one source component>; Paragraph [0074] TABLE 1, "Input Identity Mesh N × 3 × V" <read on the source 3D object Mid processed as one source component comprising V vertices>), segment a [[semantically different]] target 3D object into one or more target components (Fu, Paragraph [0065], "Given Mid and Mpose as input, the model produces mesh transferred to new posture" <read on target 3D object Mpose processed as one target component>; Paragraph [0074] TABLE 1, "Input Pose Mesh N × 3 × V" <read on the target 3D object Mpose processed as one target component comprising V vertices>), [[determine a source texture from the 3D object and a target texture from the target 3D object;]] determine a difference between at least one of the one or more source components and a respective at least one of the one or more target components (Fu, Paragraph [0070], "The encoder E takes Mpose vertices coordinates through pose feature extractor"; "The encoder aims to extract the feature Fpose for the orderless input mesh vertices" <read on determining geometric features of the target component Mpose>; Paragraph [0071], "in SPAdaIN the external data Mid is fed into 2 different 1x1 convolution layers to produce the modulation parameters γ and β. The parameters are multiplied and added to the normalized feature." <read on determining a difference between source component Mid and target component Mpose, encoded in the computed transform parameters γ and β>), [[determine an output texture based at least on the source texture and the target texture;]] warp, using a neural network trained [[and jointly optimized over geometry and texture]] to output data corresponding to geometric style transforms [[between semantically different objects,]] the at least one of the one or more source components based, at least in part, on the difference (Fu, Paragraph [0011], "The whole network is designed as generalizing the style transfer in the image domain to deal with points" <read on neural network trained to output data corresponding to geometric style transforms>; Paragraph [0066], "These per-vertex features are then concatenated with the vertex locations in the target mesh providing identity, which is fed into the style transfer decoder consists of SPAdaIn ResBlocks. Throughout the decoder, each feature produces one vertex in the output mesh under the guide of a vertex from the target mesh." <read on warping source component Mid based on the computed difference with target component Mpose>), generate one or more stylized components based, at least in part, on the warped at least one of the one or more source components [[and the output texture]] (Fu, Paragraph [0068], "the goal of the model is to transfer the pose to the identity mesh by producing output mesh Moutput={circumflex over (M)}(a1; β2; θ1)" <read on stylized components based on warped source component>; Paragraph [0066], "The final output mesh inherits the pose from the source mesh and the identity from the target").
But Fu does not explicitly disclose determine a source texture from the 3D object and a target texture from the target 3D object; determine an output texture based at least on the source texture and the target texture; and generating stylized components based on the output texture.
However, Ranade teaches determine a source texture from the 3D object (Ranade, Paragraph [0048], "At 303, the low resolution images IiLR are applied to a mesh M of the 3D model to obtain the low resolution texture atlas TLR" <read on source texture TLR determined from the source 3D object>; Paragraph [0041], "The forward 3D texture formation model outputs a low resolution texture atlas TLR, which enables rendering of a 3D textured model at various views"), a target texture from the target 3D object (Ranade, Paragraph [0043], "a ground truth texture atlas T and a geometric mesh are provided as input" <read on target texture T from the target 3D object>; Paragraph [0043], "texture maps from the ground truth texture atlas T are applied to various camera locations i, with known camera intrinsic and extrinsic properties, to form a 3D ground truth model that is made up of a collection of ground truth images IiGT"), determine an output texture based at least on the source texture and the target texture (Ranade, Paragraph [0059], "a low resolution texture atlas TLR may be input to the CNN 401 to obtain the network output fLR" <read on output texture fLR derived from source texture TLR via the CNN>; Paragraph [0059], "a ground truth texture atlas T may be input to a differentiable renderer 403 to render ground truth images I0GT, I1GT, I2GT, and IN-1GT at N different camera locations. Lrendering may then be calculated between the rendered pairs of images based on Equation (4), above, and back-propagated through the network via differentiable renderers 402 and 403." <read on output texture determined based on both source texture TLR and target texture T via rendering loss>), and generating stylized components based on the output texture (Ranade, Paragraph [0063], "the low resolution texture atlas TLR may first be passed through the CNN f(θ, . . .) to obtain the network output fHR. The ground truth atlas T and fHR may then be rendered using differentiable renderers 402 and 403 to render images INGT and INHR at N different camera locations." <read on stylized components based on output texture fHR>).
Fu and Ranade are analogous art since both are directed to neural-network-based processing of 3D objects — Fu geometrically warps a source 3D mesh component using a style transfer decoder based on the difference relative to a target component, and Ranade uses a CNN with differentiable multi-view rendering to produce an output texture for a 3D model based on comparing source texture (TLR) with a reference target texture (T). Fu provided a way of segmenting source and target 3D objects into components, computing the component-wise difference, and warping source components via a geometric style transfer network to produce stylized output components. Ranade provided a way of determining source and target textures for 3D objects and computing an output texture by passing the source texture through a CNN trained with a rendering loss relative to the target texture. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate Ranade's source-texture determination, target-texture determination, and output-texture computation into Fu's 3D-component warping system such that the resulting stylized components are generated based on both the warped source component (from Fu) and the output texture (from Ranade). The motivation is to generate stylized 3D object components that reflect both the geometric and the visual appearance characteristics of the target object, thereby improving realism and completeness of the stylized output, as Ranade discusses in its FIG. 1 Description and FIG. 4A/4B rendering-loss training framework.
But the combination of Fu, Ranade and Segu does not explicitly disclose the target 3D object being semantically different from the source 3D object; the neural network being trained and jointly optimized over geometry and texture; and geometric style transforms between semantically different objects.
However, Segu teaches segment a semantically different target 3D object into one or more target components (Segu, § 3.1, "Let X1 and X2 be two domains of 3D shapes (e.g. straight chair and armchair)" <read on source 3D object from domain X1 and semantically different target 3D object from a different domain X2>; Segu, § 4.3, "we validate hence our approach on two of the most populated object categories in ShapeNet and identify for each two subcategories as corresponding stylistic variations: Chairs (armchair↔ straight chair), Planes (jet↔ fighter aircraft). We further test our method on the categories hippos and horses in SMAL" <read on the target 3D object (e.g., armchair) being semantically different from the source 3D object (e.g., straight chair)>; Segu, § 3.2, "we leverage a shared content encoder Ec and two domain-specific style encoders Es1 and Es2" <read on segmenting the semantically different target 3D object into components via the domain-specific style encoder>), a neural network trained and jointly optimized over geometry and [[texture]] to output data corresponding to geometric style transforms (Segu, § 3.3, "We jointly train the encoders Ec, Es1, Es2, the decoder G, the mapping functions M1, M2 and the discriminators D1 and D2 to optimize the total loss" <read on neural network trained and jointly optimized over the geometry side>; Segu, page 2, "By leveraging an autoencoder architecture, we simultaneously learn reconstruction and style-driven transformations of the input 3D objects" <read on jointly optimizing a single network over the geometric style transformation>; Segu, § 3.3, "Ltot = λrec(Lx1rec+Lx2rec) + λadv(Lx1adv+Lx2adv) + λcycle(Lx1cycle+Lx2cycle)" <read on unified joint loss>), and geometric style transforms between semantically different objects (Segu, page 1, "3DSNet learns to combine the content of one shape with the style of another. We illustrate style transfer results for the pairs armchair↔ straight chair and jet↔ fighter aircraft" <read on style transforms between semantically different objects>; Segu, page 1, "transferring the style between 3D objects can be interpreted as transferring the peculiar traits of a target shape to a source shape while preserving the underlying pose and structure of the latter" <read on geometric style transforms between semantically different objects>).
Segu is analogous to Fu and Ranade because all three references address neural-network-based style transfer applied to 3D object models — Fu transferring geometric style between meshes with a style-transfer decoder, Ranade transferring texture between 3D texture atlases via a rendering-loss-trained CNN, and Segu transferring style between source and target 3D shape domains via a jointly-trained encoder-decoder architecture that supports transfer across semantically different domains. Segu provided a way of applying source-to-target 3D style transfer between semantically different categories (e.g., armchair↔straight chair, jet↔fighter aircraft, hippo↔horse) using a single neural network trained under a unified joint objective. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the Fu-Ranade combination in view of Segu such that (i) the target 3D object is drawn from a category/domain semantically different from that of the source 3D object, as Segu expressly teaches, and (ii) the geometric warping neural network of Fu is trained together with Ranade's texture CNN under a unified multi-term loss, in the same joint-training manner Segu teaches for its own multi-loss end-to-end training, so that the resulting network is trained and jointly optimized over both geometry (from Fu) and texture (from Ranade) to output data corresponding to geometric style transforms between semantically different objects. The motivation is to enable style transfer that generalizes across semantic domains rather than only within a single domain, and to avoid the artifacts and inconsistencies that arise when geometry and texture pipelines are trained separately — a motivation Segu articulates at § 3.2 ("The overall architecture is thus an autoencoder through which we estimate both reconstruction and style translation functions") and § 3.3 ("We jointly train … to optimize the total loss").
Regarding Claim 9, the combination of Fu, Ranade and Segu teaches the invention in Claim 8.
The combination further teaches one or more circuits are further to: generate a stylized 3D output object based, at least in part, on the one or more stylized components (Fu, FIG. 3; Paragraph [0068], "the goal of the model is to transfer the pose to the identity mesh by producing output mesh Moutput={circumflex over (M)}(α1; β2; θ1)"; [0067], "The final output mesh inherits the pose from the source mesh and the identity from the target"; <read on generating stylized 3D output object based on stylized components>).
Regarding Claim 12, the combination of Fu, Ranade and Segu teaches the invention in Claim 8.
The combination further teaches warping the at least one of the one or more source components comprises applying a 3D affine transformation to the at least one of the one or more source components (Fu, FIG. 4(b); Paragraph [0071], "in SPAdaIN the external data Mid is fed into 2 different 1×1 convolution layers to produce the modulation parameters γ and β. The parameters are multiplied and added to the normalized feature." <read on affine transformation applied to source components>), (Fu, FIG. 4(b); Paragraph [0028], "SPAdaIN(h, M) = γvi(M)·((hn,c,vi − μn,ci)/σn,ci) + βvi(M)" <read on 3D affine transformation γ, β applied to source component Mid>).
Regarding Claim 16, Fu teaches a method, comprising: determining one or more output geometric features for an output object corresponding to one or more source geometric features identified from a source object modified based on one or more target geometric features identified from a [[semantically different]] target object (Fu, Paragraph [0066], "The final output mesh inherits the pose from the source mesh and the identity from the target" <read on determining one or more output geometric features for an output object>; Paragraph [0068], "Given two meshes Mid=M(a1; β1; θ1) and Mpose=M(a2; β2; θ2), the goal of the model is to transfer the pose to the identity mesh by producing output mesh Moutput={circumflex over (M)}(a1; β2; θ1)" <read on output geometric features corresponding to source features identified from source object Mid, modified based on target features identified from target object Mpose>; Paragraph [0070], "The encoder E takes Mpose vertices coordinates through pose feature extractor" <read on target geometric features identified from the target object>). [[determining an output texture for the output object corresponding to a source texture of the source object modified based on a target texture of the target object;]] generating the output object, based on the output one or more geometric features [[and the output texture, using one or more trained models jointly optimized over both geometry and texture]] (Fu, Paragraph [0068], "the goal of the model is to transfer the pose to the identity mesh by producing output mesh Moutput={circumflex over (M)}(a1; β2; θ1)" <read on generating the output object based on the output one or more geometric features>; Paragraph [0082], "The model is trained for 200 epochs with batch size equaling to 8 on a single GTX 1080Ti GPU" <read on generating using a trained model>).
But Fu does not explicitly disclose determining an output texture for the output object corresponding to a source texture of the source object modified based on a target texture of the target object; and generating the output object based on the output texture.
However, Ranade teaches determining an output texture for the output object corresponding to a source texture of the source object modified based on a target texture of the target object (Ranade, Paragraph [0048], "At 303, the low resolution images IiLR are applied to a mesh M of the 3D model to obtain the low resolution texture atlas TLR" <read on source texture of the source object>; Paragraph [0043], "a ground truth texture atlas T and a geometric mesh are provided as input" <read on target texture of the target object>; Paragraph [0059], "a low resolution texture atlas TLR may be input to the CNN 401 to obtain the network output fLR" <read on output texture corresponding to source texture modified by the CNN>; Paragraph [0059], "Lrendering may then be calculated between the rendered pairs of images based on Equation (4), above, and back-propagated through the network" <read on the output texture being modified based on the target texture via the rendering loss>), and generating the output object, based on the output one or more geometric features and the output texture (Ranade, Paragraph [0063], "The low resolution texture atlas TLR may first be passed through the CNN f(θ, . . .) to obtain the network output fHR. The ground truth atlas T and fHR may then be rendered using differentiable renderers 402 and 403 to render images INGT and INHR at N different camera locations." <read on generating the output object based on both the output geometric mesh and the output texture fHR>; Paragraph [0038], "The texture atlas is input to the texture atlas super-resolution unit 104 to generate a 3D model" <read on generating the output object based on the output texture>).
Fu and Ranade are analogous since both are directed to neural-network-based transformation of 3D object models — Fu geometrically warping a source mesh under the guidance of target mesh vertices to produce an output mesh, and Ranade producing an output texture by processing a source texture atlas against a reference target texture atlas using a CNN with differentiable rendering. Fu provided a way of generating an output object from source and target geometric features using a trained neural network. Ranade provided a way of generating an output texture for an output object corresponding to a source texture modified based on a target texture using a trained CNN. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to incorporate Ranade's output-texture determination and texture-based object generation into Fu's method such that the resulting output object is generated based on both the output geometric features (from Fu) and the output texture (from Ranade). The motivation is to produce a stylized 3D object that reflects both the geometric characteristics inherited from the target's geometry and the visual (textural) characteristics inherited from the target's texture, as Ranade discusses in its FIG. 1 Description and rendering-loss-based training framework.
But the combination of Fu, Ranade and Segu does not explicitly disclose the target object being semantically different from the source object; and generating the output object using one or more trained models jointly optimized over both geometry and texture.
However, Segu teaches one or more target geometric features identified from a semantically different target object (Segu, § 3.1, "Let X1 and X2 be two domains of 3D shapes (e.g. straight chair and armchair)" <read on the target object being from a semantically different domain than the source object>; Segu, § 4.3, "Chairs (armchair↔ straight chair), Planes (jet↔ fighter aircraft). We further test our method on the categories hippos and horses in SMAL" <read on semantically different target object, e.g., the target being armchair when the source is straight chair, or the target being hippo when the source is horse>; Segu, page 1, "transferring the style between 3D objects can be interpreted as transferring the peculiar traits of a target shape to a source shape while preserving the underlying pose and structure of the latter" <read on identifying target geometric features from the semantically different target object>), and using one or more trained models jointly optimized over both geometry and [[texture]] (Segu, § 3.3, "We jointly train the encoders Ec, Es1, Es2, the decoder G, the mapping functions M1, M2 and the discriminators D1 and D2 to optimize the total loss" <read on one or more trained models jointly optimized over the geometry side>; Segu, page 2, "By leveraging an autoencoder architecture, we simultaneously learn reconstruction and style-driven transformations of the input 3D objects" <read on the one or more trained models being jointly optimized>; Segu, § 3.3, "Ltot = λrec(Lx1rec+Lx2rec) + λadv(Lx1adv+Lx2adv) + λcycle(Lx1cycle+Lx2cycle)" <read on unified joint loss optimizing the model over geometry-side reconstruction, adversarial and cycle-consistency terms simultaneously>).
Segu is analogous to Fu and Ranade because all three references address neural-network-based style transfer applied to 3D object models — Fu on the geometry side, Ranade on the texture side, and Segu on the joint end-to-end training of a single style-transfer model spanning source and target 3D objects from semantically different domains. Segu provided a way of performing source-to-target 3D style transfer across semantically different categories using a jointly trained end-to-end model with a unified multi-term objective. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the Fu-Ranade combination in view of Segu such that (i) the target object is drawn from a semantically different category/domain than the source object, as Segu expressly teaches for armchair↔straight chair, jet↔fighter aircraft, and hippo↔horse pairs, and (ii) the geometry-side network of Fu and the texture-side CNN of Ranade are trained together as one or more trained models under a unified joint objective, in the joint-training manner Segu teaches, so that the resulting one or more trained models are jointly optimized over both geometry (from Fu) and texture (from Ranade). The motivation is to enable style transfer across semantically different object categories rather than only within one category, and to obtain the consistency benefits of joint end-to-end optimization over separately trained geometry and texture pipelines, as Segu discusses at § 3.2 ("The overall architecture is thus an autoencoder through which we estimate both reconstruction and style translation functions") and § 3.3 ("We jointly train … to optimize the total loss").
Claim(s) 5, 13, 17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Fu et al. (US 20210407200 A1, hereinafter Fu) in view of Ranade et al. (US 20220122311 A1, hereinafter Ranade) further in view of Segu et al - 3DSNet Unsupervised Shape-to-Shape 3D Style Transfer - 202105 – arXiv, hereinafter Segu) as applied to Claims 1, 8, 16 above respectively and further in view of Fu et al. (US 20210027536 A1, hereinafter Fu536).
Regarding Claim 5 the combination of Fu, Ranade and Segu teaches the invention in Claim 1.
The combination does not explicitly disclose but Fu536 teaches one or more circuits are further to: determine, for the one or more source geometric segments, one or more ellipsoid approximations (Fu536, Paragraph [0017], "In this application, a deep learning model is proposed to generate the object shape from multiple color images...A new network architecture is designed in this application, named Multi-View Deformation Network (MDN), which works in conjunction with the Graph Convolutional Network (GCN)"); [0100], "a coarse shape generation network (e.g. a Pixel2Mesh network), and Multi-View Deformation Network (MDN) for shape refinement. The semantic features and the initial ellipsoid mesh are input to the Pixel2Mesh Network" <read on using an ellipsoid mesh as the approximation representation for the source 3D object's geometric segment — the ellipsoid mesh being the ellipsoid approximation determined for the source geometric segment>),; [0022], "The GCN includes a coarse shape generation network for generating a coarse shape...the coarse shape generation network is so configured as to output a coarse mesh based on the semantic features extracted from the perceptual network and an initial ellipsoid mesh" <read on determining an ellipsoid approximation for the source geometric segment as a coarse shape>),; [0057], "generating a coarse mesh from an initial ellipsoid mesh based on the extracted semantic features" <read on determining an ellipsoid approximation for the source geometric segment prior to mesh deformation>).
Fu536 and Fu are analogous since both of them are directed to 3D mesh deformation and generation using neural networks for style-transfer-based mesh warping; Fu provided a way of transferring geometric pose/style from one 3D mesh to another using a neural network with a style transfer decoder (SPAdaIN), producing a stylized output mesh. Fu536 provided a way of representing 3D object geometry as an initial ellipsoid mesh approximation, from which the final refined mesh is generated by deformation — effectively using the ellipsoid as a geometric approximation of the source object's segments. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to incorporate Fu536's ellipsoid mesh approximation into the combined Fu system such that the source geometric segments are represented by ellipsoid approximations prior to the geometric style transform. The motivation is that ellipsoid approximations provide a compact, differentiable parameterization of 3D segment geometry that simplifies the shape matching between source and target segments during the geometric style transform, as Fu536 recognizes in its coarse-to-fine 3D generation framework.
Regarding Claim 13, it recites limitations similar in scope to the limitations of Claim 5 and therefore is rejected under the same rationale.
Regarding Claim 17, it recites limitations similar in scope to the limitations of Claim 5 and therefore is rejected under the same rationale.
Claim(s) 6, 7, 14, 18, 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Fu et al. (US 20210407200 A1, hereinafter Fu) in view of Ranade et al. (US 20220122311 A1, hereinafter Ranade) further in view of Segu et al - 3DSNet Unsupervised Shape-to-Shape 3D Style Transfer - 202105 – arXiv, hereinafter Segu) as applied to Claims 1, 8, 16 above respectively and further in view of Wang US 20210279971 A1).
Regarding Claim 6 the combination of Fu, Ranade and Segu teaches the invention in Claim 1.
The combination does not explicitly disclose but Wang teaches determine an object type associated with the source 3D object model and the target 3D object model
(Wang, Paragraph [0139], "an object category and outline identifying module, configured to identify the category of each object contained in each picture in the 2D picture set and outline information of each object by using a deep learning algorithm" <read on determining an object type/category associated with each 3D object model>; Abstract, "The method includes: identifying the category of each object contained in each picture in the 2D picture set and outline information of each object by using a deep learning algorithm" <read on determining the object type/category for both the source 3D object model and the target 3D object model>); determine, based at least in part on the object type, a number of segments for the one or more source geometric segments (Wang, Paragraph [0139], "an object category and outline identifying module, configured to identify the category of each object contained in each picture in the 2D picture set and outline information of each object"; it is noted that using the identified object category to determine the outline/segmentation of the object — with the outline information defining the number and extent of the geometric segments, where the category determines the expected segmentation topology; [0153]-[0154], "a classification and regression module, configured to map the pooled P1...Pn to a fully-connected feature, identify the object category of each anchor box on the fully-connected features and reduce the size of the anchor box; and a full convolutional network, configured to identify outline information of objects in region of each reduced anchor box by using the full convolutional network." <read on using the identified object category to determine how many segments/parts the object outline delineates>).
Wang and Fu are analogous since both of them are directed to neural-network-based 3D object processing for geometric style transfer and Wang for 2D-to-3D model conversion that determines both the object category and its outline/segmentation boundaries using deep learning. Wang provided a way of first determining the category/type of each 3D object and then using that category to identify the object's outline information — effectively using the object type to guide the number and shape of segments. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to incorporate Wang's object-category-driven segmentation into the invention of Fu such that the object type of the source and target 3D object models is first determined, and that object type is then used to determine the appropriate number of geometric segments, as different object types (e.g., human figures, chairs, vehicles) have characteristically different part counts that a PHOSITA would recognize as informing the segmentation granularity.
Regarding Claim 7, the combination of Fu, Ranade and Wang teaches the invention in Claim 6.
The combination further teaches segment the source 3D object model and target 3D object model into the number of segments (Wang, Paragraph [0139], "an object category and outline identifying module, configured to identify the category of each object contained in each picture in the 2D picture set and outline information of each object" <read on segmenting each 3D object into segments determined by the object category, where the outline information defines segment boundaries>; [0005]"identifying the category of each object...and outline information of each object by using a deep learning algorithm; extracting detail characteristic information of each object in each picture" <read on segmenting both source and target 3D object models using the category-guided outline identification>).
As explained in rejection of claim 1, the obviousness for combining of Wang into Fuis provided above.
Regarding Claim 14, it recites limitations similar in scope to the limitations of Claim 6 and therefore is rejected under the same rationale.
Regarding Claim 18 the combination of Fu, Ranade and Segu teaches the invention in Claim 16.
The combination does not explicitly disclose but Wang teaches determining an object type associated with the source object and the target object (Wang, Abstract, "The method includes: identifying the category of each object contained in each picture in the 2D picture set and outline information of each object by using a deep learning algorithm" <read on determining the object type/category for each object, including both the source object and the target object, using a deep learning classification step>; [0139], "an object category and outline identifying module, configured to identify the category of each object contained in each picture in the 2D picture set and outline information of each object by using a deep learning algorithm" <read on a dedicated module for determining the object type (category) associated with the source and target objects>); determining, based at least in part on the object type, a number of segments for the one or more source geometric features (Wang, Paragraph [0139], "an object category and outline identifying module, configured to identify the category of each object contained in each picture in the 2D picture set and outline information of each object" <read on using the identified object category/type to determine the outline segmentation topology — i.e., how many geometric segments the source object's features are divided into — where the category informs which segmentation schema applies>; [0153]-[0154], "a classification and regression module, configured to map the pooled P1...Pn to a fully-connected feature, identify the object category of each anchor box on the fully-connected features and reduce the size of the anchor box; and a full convolutional network, configured to identify outline information of objects in region of each reduced anchor box" <read on the object category driving the downstream outline/segment identification, such that the number of segments derived from the outline is determined based on the object type>).
Wang and Fu are analogous since both of them are directed to neural-network-based 3D object processing. Fu provided for geometric style transfer. Wang provided a way of first classifying the object type/category of each input object using a CNN-based classification module, and then using that category to guide the determination of the object's outline segmentation — effectively using the object type to set the number and scope of geometric segments. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to incorporate Wang's object-type classification into the invention of Fu such that the type/category of the source and target objects is first determined, and that object type is then used to determine the appropriate number of geometric segments for the source object's features, before performing the geometric style transform. The motivation is well-recognized in the art: different object types (e.g., human figures, chairs, vehicles) have characteristically different part structures and segment counts, and it would recognize that the object type directly informs the segmentation granularity most appropriate for that object class, as Wang reflects in its category-guided outline identification framework.
Regarding Claim 19, it recites limitations similar in scope to the limitations of Claim 7 and therefore is rejected under the same rationale.
Claim(s) 3, 10, 11, 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Fu et al. (US 20210407200 A1, hereinafter Fu) in view of Ranade et al. (US 20220122311 A1, hereinafter Ranade) further in view of Segu et al - 3DSNet Unsupervised Shape-to-Shape 3D Style Transfer - 202105 – arXiv, hereinafter Segu) as applied to Claims 1, 8, 16 above respectively and further in view of Deng et al. (US 11055910 B1, hereinafter Deng).
Regarding Claim 3, the combination of Fu, Ranade and Segu teaches the invention in Claim 1.
The combination further teaches generate a plurality of images depicting a plurality of views of the stylized 3D object model (Ranade, Paragraph [0059], "a low resolution texture atlas TLR may be input to the CNN 401 to obtain the network output T̂LR...may be input to a differentiable renderer 402 to render low resolution images I0LR, I1LR, I2LR, and IN−1LR at N different camera locations" <read on generating a plurality of images depicting a plurality of views of the stylized 3D object model>).
As explained in rejection of claim 1, the obviousness for combining Ranade into Fu is provided above.
But the combination of Fu, Ranade and Segu do not explicitly disclose apply a mask to at least one image of the plurality of images to identify one or more background pixels or remove, from the stylized 3D object model, one or more features computed from the one or more background pixels.
However, Deng teaches apply a mask to at least one image of the plurality of images to identify one or more background pixels (Deng, DETAILED DESCRIPTION, Column 4, Line 1-7, "segmentation masks 308-1, 308-2, . . . 308-36 that illustrate the contour of the objects represented in images 306-1, 306-2, . . . 306-36 can be determined. Pixels inside the contour can be a first color and pixels outside the contour can be a second color." <read on applying a segmentation mask to images depicting multiple views of the 3D object to identify background pixels as those located outside the object contour>;Column 7, Line 45-50, "a segmentation mask can have value 0 at pixels outside an object's contour and value 1 at pixels within the object's contour. The difference in values can be used to determine the contour of the object." <read on the mask identifying background pixels as those with value 0, i.e., pixels outside the object contour in each view image>); emove, from the stylized 3D object model, one or more features computed from the one or more background pixels (Deng, Column 3, Line 60-67, "after object images are captured, segmentation masks can be created based on the object images, and an object mesh structure can be created based on the segmentation masks." <read on the 3D object mesh being constructed using only the foreground pixels identified by the masks, thereby removing from the 3D object model any features computed from background pixels — i.e., pixels with value 0 outside the object contour>),
(Deng, Column 5, Line 60-65, "for each segmentation mask, the camera parameters can be used to project a voxel or pixel from the region in the reference coordinate system onto a point in a segmentation mask in a camera coordinate system. A decision can be made about the probability of a voxel being within a surface of the object based on the projection." <read on removing from the 3D model any voxels/features projected from background pixels, retaining only features within the foreground object surface>).
Deng and Fu are analogous since both of them are directed to neural-network-based processing of 3D object models using multi-view imagery. Fu provided a way of geometric style transfer between 3D meshes. Deng provided a way of capturing a plurality of images of a 3D object from multiple viewpoints, applying segmentation masks to each image to identify foreground object pixels (value 1) versus background pixels (value 0), and constructing the 3D object mesh exclusively from the foreground pixels, thereby removing from the 3D model all features derived from background pixels.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to incorporate Deng's multi-view segmentation masking into the combined Fu stylized 3D object generation pipeline, such that after generating the plurality of rendered views of the stylized 3D object model, a segmentation mask is applied to each view image to identify background pixels (value 0), and features computed from those background pixels are removed from the stylized 3D object model. The motivation is that background pixels introduce spurious geometric and texture features into the 3D model that do not correspond to the actual object surface — a problem Deng expressly recognizes and solves through its segmentation-mask-based 3D reconstruction approach.
Regarding Claim 20, it recites limitations similar in scope to the limitations of Claim 3 and therefore is rejected under the same rationale.
Regarding Claim 10, the combination of Fu, Ranade and Segu teaches the invention in Claim 1.
The combination further teaches generate a plurality of images depicting a plurality of views of the stylized 3D output object (Ranade, Paragraph [0059], "The network output T̂LR...may be input to a differentiable renderer 402 to render low resolution images I0LR, I1LR, I2LR, and IN−1LR at N different camera locations." <read on generating a plurality of images depicting a plurality of views of the stylized 3D output object, where the stylized 3D output object is rendered from N different viewpoints>).
But the combination of Fu, Ranade and Segu does not explicitly disclose identify one or more background pixels in at least one image of the plurality of images or remove, from the stylized 3D output object, one or more features computed from the one or more background pixels. Ranade's differentiable renderer processes the full rendered image without a distinct step to identify or exclude background pixels.
However, Deng teaches identify one or more background pixels in at least one image of the plurality of images (Deng, Column 7, Line 45-50, "a segmentation mask can have value 0 at pixels outside an object's contour and value 1 at pixels within the object's contour. The difference in values can be used to determine the contour of the object <read on identifying background pixels as those with value 0 in at least one image of the plurality of view images — pixels outside the object's contour constituting the background pixels>; Column 4, Line 1-7, "segmentation masks 308-1, 308-2, . . . 308-36 that illustrate the contour of the objects represented in images 306-1, 306-2, . . . 306-36 can be determined. Pixels inside the contour can be a first color and pixels outside the contour can be a second color." <read on identifying background pixels as those pixels outside the object contour in each of the plurality of view images>); remove, from the stylized 3D output object, one or more features computed from the one or more background pixels (Deng, Column 3, Line 60-67, "after object images are captured, segmentation masks can be created based on the object images, and an object mesh structure can be created based on the segmentation masks." <read on the 3D object mesh being constructed using only the foreground pixels identified by the masks, thereby removing from the 3D object model any features computed from background pixels — i.e., pixels with value 0 outside the object contour>; Column 5, Line 60-65, "for each segmentation mask, the camera parameters can be used to project a voxel or pixel from the region in the reference coordinate system onto a point in a segmentation mask in a camera coordinate system. A decision can be made about the probability of a voxel being within a surface of the object based on the projection." <read on removing from the 3D model any voxels/features projected from background pixels, retaining only features within the foreground object surface>).
Deng and Fu are analogous since both of them are directed to neural-network-based processing of 3D object models using multi-view imagery. Fu provided a way of geometric style transfer between 3D meshes. Deng provided a way of capturing a plurality of images of a 3D object from multiple viewpoints, applying segmentation masks to each image to identify foreground object pixels (value 1) versus background pixels (value 0), and constructing the 3D object mesh exclusively from the foreground pixels, thereby removing from the 3D model all features derived from background pixels.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to incorporate Deng's background pixel identification into the invention of Fu such that after generating the plurality of rendered view images, background pixels are identified in each image and features computed from those background pixels are removed from the stylized 3D output object. The motivation is that background pixels introduce spurious geometric features into the 3D output object that do not correspond to the actual object surface — a problem Deng expressly recognizes and addresses through its segmentation-based 3D mesh construction approach.
Regarding Claim 11 the combination of Fu, Ranade and Deng teaches the invention in Claim 10.
The combination further teaches apply a mask to at least one image of the plurality of images (Deng, , Column 4, Line 1-7, "segmentation masks 308-1, 308-2, . . . 308-36 that illustrate the contour of the objects represented in images 306-1, 306-2, . . . 306-36 can be determined." <read on applying a segmentation mask to each image of the plurality of view images — mask 308-i applied to image 306-i — in order to identify foreground vs. background pixels at each viewpoint>),
(Deng, Column 7, Line 45-50, "a segmentation mask can have value 0 at pixels outside an object's contour and value 1 at pixels within the object's contour." <read on the mask being applied to the image to assign a binary value to each pixel, thereby implementing the mask application step>; Column 5, Line 60-65, "For each segmentation mask, the camera parameters can be used to project a voxel or pixel from the region in the reference coordinate system onto a point in a segmentation mask in a camera coordinate system." <read on the mask being applied per image to map each image pixel to a mask value used in the downstream 3D object reconstruction>).
Deng and Fu are analogous since both of them are directed to neural-network-based processing of 3D object models using multi-view imagery. Fu provided a way of geometric style transfer between 3D meshes. Deng provided a way of capturing a plurality of images of a 3D object from multiple viewpoints, applying segmentation masks to each image to identify foreground object pixels (value 1) versus background pixels (value 0), and constructing the 3D object mesh exclusively from the foreground pixels, thereby removing from the 3D model all features derived from background pixels.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to incorporate Deng's multi-view segmentation masking into the modified invention of Fu such that after generating the plurality of rendered views of the stylized 3D object model, a segmentation mask is applied to each view image to identify background pixels (value 0), and features computed from those background pixels are removed from the stylized 3D object model. The motivation is that background pixels introduce spurious geometric and texture features into the 3D model that do not correspond to the actual object surface — a problem Deng expressly recognizes and solves through its segmentation-mask-based 3D reconstruction approach.
Claim(s) 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Fu et al. (US 20210407200 A1, hereinafter Fu) in view of Ranade et al. (US 20220122311 A1, hereinafter Ranade) further in view of Segu et al - 3DSNet Unsupervised Shape-to-Shape 3D Style Transfer - 202105 – arXiv, hereinafter Segu) as applied to Claim 8 above and further in view of Panneer et al. (US 20220101597 A1, hereinafter Panneer).
Regarding Claim 15, the combination of Fu, Ranade and Segu teaches the invention in Claim 1.
The combination does not explicitly disclose but Panneer teaches at least a portion of the segmenting of the one or more source components or the one or more target components is semi-supervised. at least a portion of the segmenting is semi-supervised (Panneer, Paragraph [0209], " Variations on supervised and unsupervised training may also be employed. Semi-supervised learning is a technique in which in the training dataset 1702 includes a mix of labeled and unlabeled data of the same distribution. Incremental learning is a variant of supervised learning in which input data is continuously used to further train the model").
Panneer and Fu are analogous since both of them are directed to neural-network-based learning systems for 3D object processing. Fu provided for geometric style transfer (supervised). Panneer provided a way of training neural networks with semi-supervised learning, where only a portion of the training data carries labels and the network learns from both labeled and unlabeled examples that directly applicable to the segmentation network responsible for segmenting source/target 3D object components. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date to apply Panneer's semi-supervised training technique to the segmentation component of Fu’s 3D object processing system, particularly when fully annotated 3D segment labels are scarce or expensive to obtain. The motivation is well-recognized in the art: semi-supervised learning reduces the annotation burden while maintaining acceptable segmentation performance, as Panneer expressly teaches in its discussion of training paradigm variations for neural networks.
Response to Arguments
The rejection of Claims 1, 3, 4, 5, 6, 16 under Nonstatutory Double Patenting are maintained. Suggestion: applicant to file eTerminal/Terminal Disclaimer in order to overcome the rejection.
Applicant’s arguments with respect to claim 1, 8, 16, filed on 9/8/20126, with respect to rejection under 35 USC § 103 have been considered but are moot in view of the new ground(s) of rejection. They have been taught by the combination of Fu, Ranade and Segu.
In regard to Claims 2-7, 9-15, 17-20, they directly/indirectly depends on independent Claim 1, 8, 16 respectively. Applicant does not argue anything other than the independent claim 1, 8, 16. The limitations in those claims in conjunction with combination previously established as explained.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
US 20220122311 A1 3D TEXTURING VIA A RENDERING LOSS
US 20220014723 A1 ENHANCING PERFORMANCE CAPTURE WITH REAL-TIME NEURAL RENDERING
US 20210407200 A1 Network and System for Neural Pose Transfer
US 20200218948 A1 THUNDERNET: A TURBO UNIFIED NETWORK FOR REAL-TIME SEMANTIC SEGMENTATION
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to YUJANG TSWEI whose telephone number is (571)272-6669. The examiner can normally be reached 8:30am-5:30pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kent Chang can be reached at (571)272-7667. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/YuJang Tswei/Primary Examiner, Art Unit 2614
