CTNF 18/958,699 CTNF 80339 DETAILED ACTION Notice of Pre-AIA or AIA Status 07-03-aia AIA 15-10-aia The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA. Claim Rejections - 35 USC § 103 07-06 AIA 15-10-15 In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status. 07-20-aia AIA The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action: A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made. 07-23-aia AIA The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows: 1. Determining the scope and contents of the prior art. 2. Ascertaining the differences between the prior art and the claims at issue. 3. Resolving the level of ordinary skill in the pertinent art. 4. Considering objective evidence present in the application indicating obviousness or nonobviousness. 07-21-aia AIA Claim (s) 1, 2, 11, and 12 is/are rejected under 35 U.S.C. 103 as being unpatentable over Mann (US 20240193835A1) and Han (see citation below) . As per Claim 1, Mann teaches a method for a neural network driven vertex animation ( generate, using the neural network, animated representations of the object in which a geometry of the object is controllable by the set of adjustable parameters , [0007], blendshape may correspond to deformations over the entire set of vertices, or may correspond to deformations over a specific subset of vertices , [0062]), the method comprising: receiving, by an encoder component of a trained neural network ( neural networks may be implemented as encoder-decoder networks , [0085], audio data may be used as driving data for visual dubbing, in which the neural renderer(s) trained at 812 are used to generate animated representations of the actors’ faces , [0119]), an input driving signal including audio data (302); processing, by the encoder component, the input driving signal to generate blendshape coefficient information (blendshape weightings) based on the input driving signal ( primary dialogue audio 302 are processed using the facial animation model 312 to determine a sequence of primary blendshapes 314 encoding estimated facial deformations corresponding to the primary dialogue , [0093], deformation parameters may control weightings for a linear combination of a predetermined set of principal components referred to as blendshapes, by specifying particular weightings, the linear combination of blendshapes may be capable of expressing a wide range of deformations , [0062]). Mann teaches the encoder 602 maps a space-time volume to a latent variable [0106], and the decoder 604 maps the latent variable to a low-resolution image frame [0107]. Thus, Mann teaches transmitting, by the encoder component, information to a decoder component of the trained neural network; receiving, by the decoder component, the information from the encoder component [0106, 0107, 0085, 0119]; and generating vertex position information based on the blendshape coefficient information (blendshape weightings) for animating ( deformation parameters may control weightings for a linear combination of a predetermined set of principal components referred to as blendshapes, blendshape may correspond to deformations over the entire set of vertices, or may correspond to deformations over a specific subset of vertices, delta blendshape may control vertices associated with a mouth shape, but may have no effect on vertices associated with eyes, by specifying particular weightings, the linear combination of blendshapes may be capable of expressing a wide range of deformations to the base geometry , [0062]) a three-dimensional (3D) model ( neural network is trained to generate animated representations of the object based at least in part on a view of the three-dimensional deformable model , [0018]). However, Mann does not expressly teach transmitting, by the encoder component, the blendshape coefficient information to the decoder component; receiving, by the decoder component, the blendshape coefficient information from the encoder component; and generating, by the decoder component, the vertex position information. However, Han teaches that the encoder component generates blendshape coefficient information ( autoencoder was used to create the blendshape , p. 118, 1 st sentence). Han teaches the decoder performs as blendshapes, so the face geometry can be gained from the weights, blendshape coefficients (p. 115, right column, 4 th paragraph). Thus, this means that it transmits, by the encoder component, the blendshape coefficient information to the decoder component; receives, by the decoder component, the blendshape coefficient information from the encoder component (p. 118, 1 st sentence; p. 115, right column, 4 th paragraph). Since Mann teaches generating vertex position information based on the blendshape coefficient information for animating [0062] a 3D model [0018], this teaching from Han of receiving, by the decoder component, the blendshape coefficient information from the encoder component can be implemented into the decoder component of Mann so that it generates, by the decoder component, vertex position information based on the blendshape coefficient information for animating a 3D model. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Mann to include transmitting, by the encoder component, the blendshape coefficient information to the decoder component; receiving, by the decoder component, the blendshape coefficient information from the encoder component; and generating, by the decoder component, the vertex position information because Han suggests that this automatically generates individually optimized blendshapes from real-time captured facial expressions (p. 114, Abstract). As per Claim 2, Mann teaches further comprising: displaying the 3D model with animated movements based on the vertex position information ( rendering may be performed in which the combined blendshapes are provided to the object representation model, thereby to generate a video layer comprising an animated representation of the actor’s face, or a portion of the actor’s face, compositing may be performed in which the video layer is composited with the input video to generate composite video data, which may then be displayed , [0097], [0062, 0018]). As per Claims 11-12, these claims are similar in scope to Claims 1-2 respectively, and therefore are rejected under the same rationale . 07-21-aia AIA Claim (s) 4 and 14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Mann (US 20240193835A1) and Han (see citation below) in view of Wang (US 20230021460A1) . As per Claim 4, Mann and Han are relied upon for the teachings as discussed above relative to Claim 1. However, Mann and Han do not teach wherein the encoder component is located on a server, and wherein the decoder component is located on an edge device that is separate from the server. However, Wang teaches wherein the encoder component is located on a server, and wherein the decoder component is located on an edge device that is separate from the server ( encoder 310 may be disposed in an AIoT device at the front end of the system, while the decoder 320 may be disposed in an edge device , [0036]). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Man and Han so that the encoder component is located on a server, and wherein the decoder component is located on an edge device that is separate from the server because Wang suggests that it is well-known in the art for the front-end devices to perform preliminary computing functions, while in-depth computation is performed in edge devices [0004]. As per Claim 14, Claim 14 is similar in scope to Claim 4, and therefore is rejected under the same rationale . 07-21-aia AIA Claim (s) 5 and 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Mann (US 20240193835A1) and Han (see citation below) in view of Kwatra (US 20230343010A1) . As per Claim 5, Mann and Han are relied upon for the teachings as discussed above relative to Claim 1. However, Mann and Han do not teach wherein the decoder component includes a fully connected layer. However, Kwatra teaches wherein the decoder component includes a fully connected layer ( decoder can include two fully connected layers, these can be thought of as mapping of speech to a linear blendshape facial representation with 468 vertices , [0053]). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Mann and Han so that the decoder component includes a fully connected layer as suggested by Kwatra. It is well-known in the art that fully connected layers are widely used in neural network because this makes it possible to combine information from different parts of the input in a flexible way. As per Claim 15, Claim 15 is similar in scope to Claim 5, and therefore is rejected under the same rationale . 07-21-aia AIA Claim (s) 6 and 16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Mann (US 20240193835A1), Han (see citation below), and Kwatra (US 20230343010A1) in view of Feigl (see citation below) . As per Claim 6, Mann, Han, and Kwatra are relied upon for the teachings as discussed above relative to Claim 5. However, Mann, Han, and Kwatra do not teach wherein the fully connected layer is represented as WX + b, where W is a weight matrix, b is a bias vector or a base mesh, and X is the blendshape coefficient information output form the encoder component. However, Feigl teaches fully connected layer multiplies the input by a weight matrix and then adds a bias vector (p. 86, 3.6). Thus, fully connected layer is represented as WX + b, where W is a weight matrix, b is a bias vector or a base mesh. Feigl teaches morphing target animations (blendshape) (p. 84, 1 st paragraph of Introduction). Since Han teaches that the blendshape coefficient information is output from the encoder component and input into the decoder component, as discussed in the rejection for Claim 1, and Kwatra teaches that the decoder component includes a fully connected layer [0053], this teaching from Feigl can be implemented into the combination of Han and Kwatra so that X is the blendshape coefficient information output from the encoder component. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Mann, Han, and Kwatra so that the fully connected layer is represented as WX + b, where W is a weight matrix, b is a bias vector or a base mesh, and X is the blendshape coefficient information output form the encoder component as suggested by Feigl. It is well-known in the art that a fully connected layer multiples the input by a weight matrix and then adds a bias vector. As per Claim 16, Claim 16 is similar in scope to Claim 6, and therefore is rejected under the same rationale . 07-21-aia AIA Claim (s) 7 is/are rejected under 35 U.S.C. 103 as being unpatentable over Mann (US 20240193835A1), Han (see citation below), and Kwatra (US 20230343010A1) in view of Ma (see citation below) and Borovikov (US 20240331293A1) . Mann, Han, and Kwatra are relied upon for the teachings as discussed above relative to Claim 5. Mann teaches each blendshape defines positions of vertices for a specific expression or animation [0062]. However, Mann, Han, and Kwatra do not teach transforming information into a set of blendshapes based on converting the weight matrix into distinct blendshapes, wherein each blendshape in the set of blendshapes includes a matrix corresponding to a single blendshape defining the positions of the vertices for the specific expression or animation. However, Ma teaches that the blendshape B* is calculated using the weight matrix W, and thus it transforms information into a set of blendshapes based on converting the weight matrix into distinct blendshapes, wherein each blendshape in the set of blendshapes includes a matrix corresponding to a single blendshape (p. 3, right column). Since Mann teaches each blendshape defines positions of vertices for a specific expression or animation [0062], this teaching from Ma can be implemented into the device of Mann so that the single blendshape defines positions of vertices for a specific expression or animation. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Mann, Han, and Kwatra to include transforming information into a set of blendshapes based on converting the weight matrix into distinct blendshapes, wherein each blendshape in the set of blendshapes includes a matrix corresponding to a single blendshape defining the positions of the vertices for the specific expression or animation because Ma suggests that this way, the blend-shape basis becomes significantly more distinctive (p. 3, right column). However, Mann, Han, Kwatra, and Ma do not teach that the information is transformed based on the decoder component. However, Borovikov teaches transforming information based on the decoder component into a set of blendshapes ( authoring parameters generated by the decoder engine 120 can identify the blendshape weights that can be used to generate a face model , [0059]). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Mann, Han, Kwatra, and Ma so that the information is transformed based on the decoder component because Borovikov suggests that this automatically generates synthetic face shapes that are representative of realistic human faces [0037] . 07-21-aia AIA Claim (s) 8 and 17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Mann (US 20240193835A1) and Han (see citation below) in view of Lee (US 20220067532A1) . As per Claim 8, Mann and Han are relied upon for the teachings as discussed above relative to Claim 1. However, Mann and Han do not teach generating the trained neural network by inputting pairs of input data and ground-truth tensors including vertex position information to a neural network, outputting predicted tensors by the neural network, and optimizing the neural network to minimize a loss function based on a difference between the predicted tensors and ground-truth tensors for producing the trained neural network. However, Lee teaches further comprising: generating the trained neural network by inputting pairs of input data and ground-truth tensors including vertex position information to a neural network, outputting predicted tensors by the neural network, and optimizing the neural network to minimize a loss function based on a difference between the predicted tensors and ground-truth tensors for producing the trained neural network ( method to train the model, the computing an accuracy of a prediction box by comparing the prediction box output from the neural network model and the ground truth box may include determining a vertex of the prediction box as a second reference point by comparing the vertex with an edge of the ground truth box, when the vertex is located inside the ground truth box, and computing the accuracy of the prediction box based on the first reference point and the second reference point , [0012], loss function for learning the neural network , [0083]). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Mann and Han to include generating the trained neural network by inputting pairs of input data and ground-truth tensors including vertex position information to a neural network, outputting predicted tensors by the neural network, and optimizing the neural network to minimize a loss function based on a difference between the predicted tensors and ground-truth tensors for producing the trained neural network because Lee suggests that this guides model improvement by directing the algorithm to adjust parameters iteratively to reduce loss and improve predictions [0012, 0083]. As per Claim 17, Claim 17 is similar in scope to Claim 8, and therefore is rejected under the same rationale . 07-21-aia AIA Claim (s) 9 and 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Mann (US 20240193835A1) and Mitchell (US 20200334886A1) . As per Claim 9, Mann teaches a method for controlling an artificial intelligence (AI) device [0007], the method comprising: receiving an input driving signal including audio data (302); processing the input driving signal by an encoder component of a trained neural network to generate blendshape coefficient information (blendshape weightings) based on the input driving signal [0085, 0119, 0093, 0062]; and using the blendshape coefficient information (blendshape weightings) for animating [0062] a three-dimensional (3D) model [0018]. However, Mann does not teach transmitting the blendshape coefficient information over a network to a target device for animating the 3D model. However, Mitchell teaches transmitting the blendshape coefficient information over a network to a target device (110a) for animating the 3D model ( generating device 110b replicates the skeletal rig (composed of blendshapes), then, each frame of the pose of the entity is retrieved as a list of skeleton bone transformations, plus blendshape weight activations, these data are transmitted in a network packet 200 to the target device 110a and these shape values are injected into the target rig’s frame pose , [0051]). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Mann to include transmitting the blendshape coefficient information over a network to a target device for animating the 3D model because Mitchell suggests that this way, the blendshape weights can be generated in a server that provides additional processing power for providing an AR experience, and then the blendshape weights are transmitted to an AR system for display [0015, 0051]. As per Claim 18, Mann teaches an artificial intelligence (AI) device [0007], comprising: a display configured to display an image ( device configured to display the output video data , [0041]); a memory (102) configured to store three-dimensional (3D) model information ( memory 102 may store model data 120 encoding object representation models , [0058], three-dimensional object model , [0018]); and a controller configured to: receive blendshape coefficient information (blendshape weightings), generate vertex position information based on a trained neural network for animating [0062, 0007, 0119] a 3D model [0018], and display the 3D model with animated movements based on the vertex position information [0097, 0062, 0018]. However, Mann does not teach receiving the blendshape coefficient information from an external device. However, Mitchell teaches receiving blendshape coefficient information from an external device (110b) ( generating device 110b replicates the skeletal rig (composed of blendshapes), then, each frame of the pose of the entity is retrieved as a list of skeleton bone transformations, plus blendshape weight activations, these data are transmitted in a network packet 200 to the target device 110a and these shape values are injected into the target rig’s frame pose , [0051]). This would be obvious for the reasons given in the rejection for Claim 9 . 07-21-aia AIA Claim (s) 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Mann (US 20240193835A1) and Mitchell (US 20200334886A1) in view of Feigl (see citation below) . Claim 20 is similar in scope to Claim 6, and therefore is rejected under the same rationale . Allowable Subject Matter 12-151-08 AIA 07-43 12-51-08 Claim s 3, 10, 13, and 19 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims. 13-03-01 AIA The following is a statement of reasons for the indication of allowable subject matter: The prior art taken singly or in combination do not teach or suggest the combination of all the limitations of Claim 3 and base Claim 1, and in particular, do not teach wherein the blendshape coefficient information includes an F x B matrix of blendshape coefficients, where F corresponds to a number of animation frames corresponding to an audio length of the input driving signal, and B corresponds to a number of blendshapes, and wherein the vertex position information includes an F’ x V x 3 tensor, where F’ corresponds to a number of animation frames for animating the 3D model, V corresponds to a number of vertices in the 3D model, and 3 corresponds to x, y, and z coordinates of the vertices. Claims 10, 13, and 19 are each similar in scope to Claim 3, and therefore also contain allowable subject matter. The closest prior art (Lewis (see citation below)) teaches the m x t performance matrix is split into m x n blendshape basis and the n x t animation weights matrix, where t is the number of frames (p. 6, 4.2). However, Lewis does not teach that the blendshape coefficient information includes an F x B matrix of blendshape coefficients, where F corresponds to a number of animation frames corresponding to an audio length of the input driving signal, and B corresponds to a number of blendshapes, and wherein the vertex position information includes an F’ x V x 3 tensor, where F’ corresponds to a number of animation frames for animating the 3D model, V corresponds to a number of vertices in the 3D model, and 3 corresponds to x, y, and z coordinates of the vertices . 12-57 AIA Prior Art of Record 1. Han, Ju Hee; Generate Individually Optimized Blendshapes; March 2021; 2021 IEEE International Conference on Big Data and Smart Computing; p. 114-120; https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9373223 2. Feigl, Tobias; Real-Time Gait Reconstruction For Virtual Reality Using a Single Sensor; December 2020; 2020 IEEE International Symposium on Mixed and Augmented Reality Adjunct; p. 84-89; https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=9288370 3. Ma, Wan-Duo Kurt; Blind Facial Basis Discovery Using the Hilbert-Schmidt Independence Criterion; February 2019; 2018 International Conference on Image and Vision Computing New Zealand; p. 1-5; https://ieeexplore.ieee.org/stamp/stamp.jsp?tp=&arnumber=8634682 07-96 AIA 4. The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Lewis, J.P.; Practice and Theory of Blendshape Facial Models; April 2014; Eurographics 2014 State of the Art Report; p. 1-20; https://www.scribblethink.org/Work/Pdfs/blendshapes_MAIN.pdf Conclusion Any inquiry concerning this communication or earlier communications from the examiner should be directed to JONI HSU whose telephone number is (571)272-7785. The examiner can normally be reached M-F 10am-6:30pm. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kee Tung can be reached at (571)272-7794. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. JH /JONI HSU/Primary Examiner, Art Unit 2611 Application/Control Number: 18/958,699 Page 2 Art Unit: 2611 Application/Control Number: 18/958,699 Page 3 Art Unit: 2611 Application/Control Number: 18/958,699 Page 4 Art Unit: 2611 Application/Control Number: 18/958,699 Page 5 Art Unit: 2611 Application/Control Number: 18/958,699 Page 6 Art Unit: 2611 Application/Control Number: 18/958,699 Page 7 Art Unit: 2611 Application/Control Number: 18/958,699 Page 8 Art Unit: 2611 Application/Control Number: 18/958,699 Page 9 Art Unit: 2611 Application/Control Number: 18/958,699 Page 10 Art Unit: 2611 Application/Control Number: 18/958,699 Page 11 Art Unit: 2611 Application/Control Number: 18/958,699 Page 12 Art Unit: 2611 Application/Control Number: 18/958,699 Page 13 Art Unit: 2611 Application/Control Number: 18/958,699 Page 14 Art Unit: 2611 Application/Control Number: 18/958,699 Page 15 Art Unit: 2611
Prosecution Insights