DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
The Amendment filed March 19th, 2026 have been entered. Claims 1-10 remain pending in the application. Claims 1 and 3-10 are amended. Claim 2 has been canceled. The Examiner acknowledges the corrections to Figures 2 and 5 in the attached sheets of drawings and to the specification in Paragraphs [0035], [0038], and [0055]; the objection to the Drawings with regards to a “stabilizer” in Figures 2 and 5 are withdrawn. The Examiner acknowledges the amendments to Claims 1, 2, 4, and 6-10 that result in the aforementioned Claims no longer being interpreted under U.S.C. 112(f); Hence, Claims 1, 2, 4, and 6-10 are no longer interpreted under 35 U.S.C. 112(f). The amendments to the Claims have overcome each and every objection and 112(b) rejection previously set forth in the Non-Final Office Action mailed December 19th, 2025, with the exception of an objection to a minor informality regarding “the encoding unit 210 estimator unit (230) configured on the control unit (135)” in Claim 1.
Claim Objections
Claim 1 is objected to because of the following informalities: “the encoding unit 210 estimator unit 230 being configured on the control unit 135 for penalizing inaccurate generation for each resolution” is interpreted as being “the encoding unit 210 and estimator unit 230 being configured on the control unit 135 for penalizing inaccurate generation for each resolution”. Furthermore, “a processing unit 105, the processing unit 105 being configured to receive input data from an input unit 110 and to send the output data an output unit 115” is interpreted as being “a processing unit 105, the processing unit 105 being configured to receive input data from an input unit 110 and to send the output data to an output unit 115”. Appropriate correction is required.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1, 3 and 6-10 are rejected under 35 U.S.C. 103 as being unpatentable over Sinha (US Pub. No. US2021/0366173 A1) in view of Bednarik (Learning to Reconstruct Texture-Less Deformable Surfaces from a Single View, 2018).
Regarding Claim 1, Sinha teaches a device for synchronization of features of digital objects with audio contents comprising:
A processing unit (105) (Paragraph [0004]: “in one aspect, there is provided a processor implemented method for identity preserving realistic talking face generation using audio speech of a user. The method comprises: obtaining, via one or more hardware processors”);
configured to receive input data from an input unit (110) and to send the output data to an output unit (115), the processing unit (105) including a communication unit (130), a control unit (135), and a storage unit (140) for processing the input data and generating the output; a processing module (205), the processing module (205) being configured on the control unit (135) for extracting the audio segments from the input data; an encoding unit (210), the encoding unit (210) being configured on the control unit (135) for separating audio segments and inputs into various framesets and embedding into various features sets (Paragraph [0046]: “FIG. 3 depicts a system 100 for identity preserving realistic talking face generation using audio speech of a target individual, in accordance with an embodiment of the present disclosure. In an embodiment, the system 100 includes one or more hardware processors 104, communication interface device(s) or input/output (I/O) interface(s) 106 (also referred as interface(s)), and one or more data storage devices or memory 102 operatively coupled to the one or more hardware processors 104. The one or more processors 104 may be one or more software processing components and/or hardware processors. In an embodiment, the hardware processors can be implemented as one or more microprocessors, microcomputers, microcontrollers, digital signal processors, central processing units, state machines, logic circuitries, and/or any devices that manipulate signals based on operational instructions. Among other capabilities, the processor(s) is/are configured to fetch and execute computer-readable instructions stored in the memory. In an embodiment, the system 100 can be implemented in a variety of computing systems, such as laptop computers, notebooks, hand-held devices, workstations, mainframe computers, servers, a network cloud and the like”; Paragraph [0004]: “obtaining, via one or more hardware processors, an audio speech and an identity image of a target individual; extracting, via the one or more hardware processors, one or more DeepSpeech features of the target individual from the audio speech”; Figure 8 demonstrates an encoder decoder architecture of the texture generator, where the latent vectors learned from the encoding units are the landmarks from the landmark encoder; Figure 6 demonstrates an encoder in reference to the system of Figure 3, which produces landmark displacement features represented as a latent representation. Notes: A latent representation is derived as a process of encoding. Feature extraction in its broadest reasonable interpretation is performed via encoding. Hence, extracting DeepSpeech features is deemed to separate audio segments and inputs into various framesets and embed them into feature sets. An encoding unit, in its broadest reasonable interpretation, is an entity that is essentially distinguished by an encoder or multiple encoders, for the general purpose of encoding. Since Sinha teaches a device with multiple encoders, Sinha teaches an encoding unit),
wherein the encoding unit (210) includes a first encoder (320), a second encoder (325) and a third encoder (330);
the first encoder (320) is a three-dimensional pose encoder (Paragraph [0071]: “For a given face image, OpenFace predicted 68 facial landmarks and used frame-wise tracking to obtain temporally stable landmarks” Notes: encoders are implicit in feature extraction. Furthermore, facial landmarks are known in the art to represent variations in features such as pose),
the second pose encoder (325) is an expression encoder that estimates expression parameters from an upper half region of the object (Paragraph [0061]: “The encoder-decoder architecture of the generator network uses facial action units to generate attention for facial expression generation”; Paragraph [0075]: “[0075]: The generative adversarial network comprised in the memory 102 of the system 100 when executed focuses on generating unique texture for image regions that are responsible for facial expressions” Notes: parameters are inherently estimated in the encoder decoder architecture of a generator network. Expression parameters, in their broadest reasonable interpretation, are parameters related to expression. Since the encoder in the generator network is utilized for expression generation for facial expressions (which include the upper half region of the face as an object), the parameters modified within the generator network are expression parameters), and
the third encoder (330) is a lip movement encoder that estimates and maps movements from a lower half of the object (Paragraph [0071]: “for the lip region, it often gives erroneous prediction especially for the frames with faster lip movements. To capture an exact lip movement corresponding to input audio, a more accurate method is needed for the ground truth landmark extraction. Hence, face segmentation as known in the art technique was implemented by the system and method of the present disclosure, wherein the entire face was segmented in different regions like hair, eyes, nose, upper lip, lower lip, and rest of the face. Upper and lower lip landmarks are selected from the boundary of lip segments with the help of OpenFace predicted landmark points, which gives quite accurate estimations of lip landmarks”. Notes: segmentation techniques as known in the art utilize encoders);
a feature master (215), the feature master configured on the control unit (135) for concatenating difference feature vectors and feature sets (Paragraph [0075]: “The architecture of the texture generator is shown in FIG. 8. More specifically, FIG. 8, with reference to FIGS. 3 through 7, depicts an attention-based texture generation network implemented by the system 100 of FIG. 3 for identity preserving realistic talking face generation using audio speech of the target individual, in accordance with an embodiment of the present disclosure. The current landmark images Lt and the identity landmark image Lid images were each encoded using a landmark encoder. The difference in encoded landmark features was concatenated with the input identity image Iid and fed to an encoder-decoder architecture which generated attention map attt and color map Ct.” Notes: Feature master, in its broadest reasonable interpretation, is an entity configured for concatenating difference feature vectors and feature sets, which is achieved within the architecture);
a generator (220) configured on the control unit (135) for decoding the latent vectors learned from the encoding unit (210) (Figure 8 demonstrates an encoder decoder architecture of the texture generator, where the latent vectors learned from the encoding units are the landmarks from the landmark encoder) and
generating objects that are synced with the audio (Paragraph [0033]: “FIG. 12 depicts animation of different identities generated by the system of FIG. 3 and which are synchronized with the same speech input, containing spontaneous generation of eye blinks, in accordance with an embodiment of the present disclosure”, where Figure 12 demonstrates instances of objects synced with the audio);
a transformation module (225), the transformation module (225) configured on the control unit (135) for aligning target frames with the predefined shape of features as generated in synced predicted frames (240) (Paragraph [0072]: “To prepare ground-truth landmark displacements for training audio-to-landmark prediction network (or speech-to-landmark generation network) lip movements were imposed on the mean neutral face by assigning the displacement of lips … from a neutral face with closed lips… in person-specific landmarks. For this, the person-specific landmark … were aligned with the mean face landmark … using rigid Procrustes alignment. Per frame lip displacements from the person-specific neutral face, was added with the mean neutral face … transfer the motion from person specific landmarks to mean face landmarks… Displacements were scaled with the ratio of person-specific face height-width to mean face height”);
an estimator unit (230), the encoding unit (210) and estimator unit (230) configured on the control unit (135) for computing displacement between the input objects and the generated objects (Paragraph [0072]: “To prepare ground-truth landmark displacements for training audio-to-landmark prediction network (or speech-to-landmark generation network) lip movements were imposed on the mean neutral face by assigning the displacement of lips … from a neutral face with closed lips… in person-specific landmarks. For this, the person-specific landmark … were aligned with the mean face landmark … using rigid Procrustes alignment. Per frame lip displacements from the person-specific neutral face, was added with the mean neutral face … transfer the motion from person specific landmarks to mean face landmarks… Displacements were scaled with the ratio of person-specific face height-width to mean face height”; Figure 6 demonstrates an encoder in reference to the system of Figure 3, which produces landmark displacement features represented as a latent representation. Figure 8 further demonstrates the displacement between generated and inputted objects. Notes: an estimator in its broadest reasonable interpretation is an entity that learns a relationship from input data and outputs an estimated value based on the learned relationship. Therefore, encoders and other trainable components of predictive models fall under the definition of an estimator);
a discriminator unit (235) configured on the control unit (135) for penalizing inaccurate generation for each resolution (Paragraph [0067]: “A discriminator network has been implemented by the system 100 and method of the present disclosure to make the generated texture sharper and more distinct especially in regions of motion”; Paragraph [0067]: “LSGAN, as known in the art, has been implemented for adversarial training of the texture generation network, because of its better training stability as well as its ability to generate higher quality images than the regular GAN. Regular GANs use the sigmoid cross entropy loss function, which is prone to the problem of vanishing gradients. The LSGAN helps overcome this problem by using the least squares loss function which penalizes samples which are correctly classified yet far from the decision boundary, unlike regular GANs. Due to this property of LSGANs, generation of samples is closer to real data”); and
a stabilizer (260), the stabilizer (260) configured on the control unit (135) for stabilizing the frames and sends transformed frameset to the output unit (115) (Paragraph [0067]: “LSGAN, as known in the art, has been implemented for adversarial training of the texture generation network, because of its better training stability as well as its ability to generate higher quality images than the regular GAN. Regular GANs use the sigmoid cross entropy loss function, which is prone to the problem of vanishing gradients. The LSGAN helps overcome this problem by using the least squares loss function which penalizes samples which are correctly classified yet far from the decision boundary, unlike regular GANs. Due to this property of LSGANs, generation of samples is closer to real data”; Paragraph [0029]: “FIG. 8 depicts an attention-based texture generation network implemented by the system of FIG. 3 for identity preserving realistic talking face generation using audio speech of the target individual, in accordance with an embodiment of the present disclosure”; Figure 8 demonstrates that an output is obtained from the texture generation network, in which LSGAN is used for adversarial training).
Sinha does not teach that the first encoder implements pixel wise depth mapping and 3D mesh modelling to estimate and map the shape of features and appearance of an object.
However, Bednarik teaches an encoder that implements pixel wise depth mapping and 3D mesh modelling to estimate and map the shape of features and appearance of an object (Figure 2 clearly illustrates an encoder taking in an image input, and subsequently forming a depth map and a 3D mesh model, where the mesh is described as being 3D in Section 3.1: “By contrast, obtaining training data for 3D meshes for real images is harder and requires much more processing, since depth sensors do not provide correspondences between points on the surface. However, unlike the other two representations, 3D meshes can represent self-occluded parts of the surface, albeit at the cost of constraining the topology much more. We explain the process of obtaining the GT mesh coordinates in the supplementary materia.”; Conclusion: “We have introduced a framework for reconstructing the 3D shape of a texture-less, deformable surface from a single image”; The encoder encodes the shape of the features, and estimates and maps from the encoded features to an appearance of an object, as is evident in Figure 2. Notes: an encoder, in its broadest reasonable interpretation, includes models such as encoder-decoders. Depth mapping is well known in the art to be pixel wise, corresponding with mapping depth values per pixel of an image).
Sinha and Bednarik are considered analogous in the art with respect to encoding features related to the shape of features. A common motivation is to utilize depth maps and 3D mesh modelling to more accurately estimate and map features to the shape of an object, as is evident in Bednarik.
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to combine the device and first encoder utilized for encoding 3D pose information of an object of Sinha with the encoder that utilizes depth mapping and 3D mesh modeling of Bednarik; Doing so would yield the predictable result of a 3D pose encoder that produces more accurate estimations regarding object shape.
Regarding Claim 3, the device for synchronization of features of digital objects with audio contents (100) as claimed in Claim 1 is rejected over Sinha as modified.
Sinha as modified teaches the encoding unit (210) further includes a fourth encoder (335) is an audio encoder (Sinha, Paragraph [0014]: “extracting, via the one or more hardware processors, one or more DeepSpeech features of the target individual from the audio speech”. Notes: Feature extraction is performed via encoders in the art), and
A fifth encoder (340), the fifth encoder is an identity encoder (Sinha, Paragraph [0075]: “The current landmark images L.sub.t and the identity landmark image L.sub.id images were each encoded using a landmark encoder”, and is demonstrated in Figure 8).
Regarding Claim 6, the device for synchronization of features of digital objects with audio contents (100) as claimed in Claim 1 is rejected over Sinha as modified.
Sinha teaches an estimator unit (230) being configured with:
a first estimator (505) (Sinha, Paragraph [0041]: “To address the problem of accurate attention and color map generation, the present disclosure provides system and methods that implement an architecture for texture generation which uses LSGAN (e.g., refer "X. Mao, Q. Li, H. Xie, R. Y. Lau, Z. Wang, and S. Paul Smolley. Least squares generative adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision, pages 2794-2802, 2017." herein after referred as Mao and may be interchangeably used) for learning sharp image texture and plausible mouth shapes”),
a second estimator (510) (Sinha, Paragraph [0061]: “A blink generation network comprised in the system 100 is trained to learn/generate a realistic eye blink, duration of eye blinks and permissible intervals between two blinks from the training datasets”),
a third estimator (515) (Sinha, Paragraph [0071]: “Hence, face segmentation as known in the art technique was implemented by the system and method of the present disclosure, wherein the entire face was segmented in different regions like hair, eyes, nose, upper lip, lower lip, and rest of the face. Upper and lower lip landmarks are selected from the boundary of lip segments with the help of OpenFace predicted landmark points, which gives quite accurate estimations of lip landmark”), and
a fourth estimator (520) (Sinha, Paragraph [0071]: “LSGAN, as known in the art, has been implemented for adversarial training of the texture generation network, because of its better training stability as well as its ability to generate higher quality images than the regular GAN. Regular GANs use the sigmoid cross entropy loss function, which is prone to the problem of vanishing gradients. The LSGAN helps overcome this problem by using the least squares loss function which penalizes samples which are correctly classified yet far from the decision boundary, unlike regular GANs. Due to this property of LSGANs, generation of samples is closer to real data”).
Regarding Claim 7, the device for synchronization of features of digital objects with audio contents (100) as claimed in Claim 6 is rejected over Sinha as modified.
Sinha as modified teaches the first estimator (505) being configured to predict the shape of the features for each of the synced predicted frames (240) (Sinha, Paragraph [0041]: “To address the problem of accurate attention and color map generation, the present disclosure provides system and methods that implement an architecture for texture generation which uses LSGAN (e.g., refer "X. Mao, Q. Li, H. Xie, R. Y. Lau, Z. Wang, and S. Paul Smolley. Least squares generative adversarial networks. In Proceedings of the IEEE International Conference on Computer Vision, pages 2794-2802, 2017." herein after referred as Mao and may be interchangeably used) for learning sharp image texture and plausible mouth shapes”).
Regarding Claim 8, the device for synchronization of features of digital objects with audio contents (100) as claimed in Claim 6 is rejected over Sinha as modified.
Sinha as modified teaches the second estimator (510) being configured to predict the naturalness of movement and change in shape of the features for each of the predicted frames (240) (Sinha, Paragraph [0061]: “A blink generation network comprised in the system 100 is trained to learn/generate a realistic eye blink, duration of eye blinks and permissible intervals between two blinks from the training datasets”).
Regarding Claim 9, the device for synchronization of features of digital objects with audio contents (100) as claimed in Claim 6 is rejected over Sinha as modified.
Sinha as modified teaches the third estimator (515) that predicts the segments of the features for each of the synced predicted frames (240) (Sinha, Paragraph [0071]: “Hence, face segmentation as known in the art technique was implemented by the system and method of the present disclosure, wherein the entire face was segmented in different regions like hair, eyes, nose, upper lip, lower lip, and rest of the face. Upper and lower lip landmarks are selected from the boundary of lip segments with the help of OpenFace predicted landmark points, which gives quite accurate estimations of lip landmark”).
Regarding Claim 10, the device for synchronization of features of digital objects with audio contents (100) as claimed in Claim 6 is rejected over Sinha as modified.
Sinha as modified teaches the fourth estimator (520) being connected to the stabilizer for stabilizing the video frames (240) (Sinha, Paragraph [0067]: “LSGAN, as known in the art, has been implemented for adversarial training of the texture generation network, because of its better training stability as well as its ability to generate higher quality images than the regular GAN. Regular GANs use the sigmoid cross entropy loss function, which is prone to the problem of vanishing gradients. The LSGAN helps overcome this problem by using the least squares loss function which penalizes samples which are correctly classified yet far from the decision boundary, unlike regular GANs. Due to this property of LSGANs, generation of samples is closer to real data”).
Claim 4 is rejected under 35 U.S.C. 103 as being unpatentable over Sinha (US Pub. No. US2021/0366173 A1) in view of Bednarik (Learning to Reconstruct Texture-Less Deformable Surfaces from a Single View, 2018), in further view of Kim (Pub. No. US 2021/0357728 A1).
Regarding Claim 4, the device for synchronization of features of digital objects with audio contents (100) as claimed in Claim 1 is rejected over Sinha as modified.
Sinha as modified teaches a discriminator unit (235) configured with a first discriminator (405) (Sinha, Paragraph [0067]: “A discriminator network has been implemented by the system 100 and method of the present disclosure to make the generated texture sharper and more distinct especially in regions of motion”).
Sinha as modified does not teach a second and third discriminator within the discriminator unit.
However, Kim teaches a second discriminator (410) and a third discriminator (415) (Paragraph [0006]: “a first discriminator learned to distinguish between actual data and the synthetic data, a second discriminator learned to distinguish between the actual data and the synthetic data while satisfying differential privacy, and a third discriminator learned to distinguish between first synthetic data which is output from the generator learned by the first discriminator and second synthetic data which is output from the generator learned by the second discriminator”).
Sinha as modified and Kim are considered analogous in the art with regards to their use of discriminators to train models. A well-known motivation within the art for using multiple discriminators is for further improving the performance of a model.
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to modify the existing discriminator unit of Sinha as modifiied to encompass a second and third discriminator as taught by Kim; Doing so would yield the predictable result of improving the performance of trainable components within the device for synchronization of features of digital objects with audio contents.
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over Sinha (US Pub. No. US2021/0366173 A1) in view of Bednarik (Learning to Reconstruct Texture-Less Deformable Surfaces from a Single View, 2018) and Kim (Pub. No. US 2021/0357728 A1), in further view of Tang (MsCGAN: Multi-scale Conditional Generative Adversarial Networks for Person Image Generation, 2020) and Prajwal (A Lip Sync Expert Is All You Need for Speech to Lip Generation In The Wild, 2020).
Regarding Claim 5, the device for synchronization of features of digital objects with audio contents (100) as claimed in Claim 4 is rejected over Sinha as modified.
Sinha as modified teaches a discriminator unit with a first discriminator wherein the first discriminator is a landmark based discriminator (Sinha, Paragraph [0067]: “A discriminator network has been implemented by the system 100 and method of the present disclosure to make the generated texture sharper and more distinct especially in regions of motion”; Sinha, Paragraph [0075]: “The architecture of the texture generator is shown in FIG. 8. More specifically, FIG. 8, with reference to FIGS. 3 through 7, depicts an attention-based texture generation network implemented by the system 100 of FIG. 3 for identity preserving realistic talking face generation using audio speech of the target individual, in accordance with an embodiment of the present disclosure. The current landmark images Lt and the identity landmark image Lid images were each encoded using a landmark encoder. The difference in encoded landmark features was concatenated with the input identity image Iid and fed to an encoder-decoder architecture which generated attention map attt and color map Ct. The generated image It was then passed to a discriminator network which determines if the generated image is real or fake”, with support presented in Sinha, Figure 8. Notes: Given that the landmarks are encoded and the difference of the landmark encodings are used to generate an attention and color map (which are used to generate an output image), where said output image is fed to a discriminator to determine whether the image is real or fake, the discriminator is considered to be landmark based because the images it discriminates are derived from landmarks).
Sinha as modified does not teach a second discriminator that is a multiscale perceptual discriminator, and does not teach a third discriminator that is an audio-visual alignment discriminator.
Furthermore, Tang teaches a discriminator that is a multiscale perceptual discriminator (Section 1, Introduction: “MsCGAN contains two strategies to ensure the visual quality of the generated images to be consistent with the conditional image. One of them is using the global-tolocal generators, which generate a coarse image of the specific pose globally, and refine the coarse image locally. The other is that multi-scale discriminators are adopted to discriminate the generated image and its downsampled images respectively, which aim to handle the visual features on multiple levels… Compared with existing methods, our proposed model has several advantages:1) Joint the global generation and the local refinement can model both the accuracy and quality of the synthetic image simultaneously. 2) Images with different resolutions contain different levels of visual features, so the proposed multi-scale discriminators can increase the receptive field of the discriminator. 3) The combination of the global to-local generators and multi-scale discriminators ensure the synthetic person images have the target pose and more detailed appearance features than existing methods”).
Sinha as modified and Tang are considered analogous in the art with regards to generating images via GANs, of which competing generators and discriminators is obvious. One would be motivated to modify the discriminator unit of Sinha by integrating the multiscale perceptual discriminator of Tang to improve the discrimination process, considering multiscale perceptual discriminators are well known for handling visual features at different levels. The motivation for using multiscale perceptual discriminators as opposed to normal discriminators is also taught by Tang (Section 1, Introduction: “Images with different resolutions contain different levels of visual features, so the proposed multi-scale discriminators can increase the receptive field of the discriminator”). The motivation for using multi-scale discriminators for images can be applied to the device for synchronization of features of digital objects with audio contents as image generation is also performed within the device.
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to modify the existing discriminator unit of Sinha as modified to encompass the multiscale perceptual discriminator of Tang; Doing so would yield the predictable result of improving the performance of trainable components within the device for synchronization of features of digital objects with audio contents, especially with regards to trainable components that deal with images with different resolutions with different levels of visual features.
Lastly, Prajwal teaches a discriminator that is an audio-visual alignment discriminator (Section 3.3.2: “Our expert lip-sync discriminator. We make the following changes to SyncNet [9] to train an expert lip-sync discriminator that suits our lip generation task. Firstly, instead of feeding grayscale images concatenated channel-wise as in the original model, we feed color images. Secondly, our model is significantly deeper, with residual skip connections [15]. Thirdly, inspired by this public implementation2, we use a different loss function: cosine-similarity with binary cross-entropy loss. That is, we compute a dot product between the ReLU-activated video and speech embeddings v,s to yield a single value between [0, 1] for each sample that indicates the probability that the input audio-video pair is in sync: Psync = v · s max(∥v∥2 · ∥s ∥2, ϵ) (1) We train our expert lip-sync discriminator on the LRS2 train split (≈ 29 hours) with a batch size of 64, with Tv = 5 frames using the Adam optimizer [12] with an initial learning rate of 1e −3”).
Sinha as modified and Prajwal are considered analogous in the art with regards to synchronization of audio and video. One would be motivated to modify the discriminator unit of Sinha by encompassing the audio-visual alignment discriminator of Prajwal, considering that utilizing a discriminator for syncing audio and video would significantly improve the model of Sinha as modified.
Therefore, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to modify the discriminator unit of Sinha as modified to encompass the audio-visual alignment discriminator of Prajwal; Doing so would yield the predictable result of a discriminator unit that improves the performance of the device for synchronization of features of digital objects with audio contents, especially with regards to the performance of the audio-visual synchronization components.
Response to Arguments
Applicant’s arguments, see page 9 of Applicant Remarks, filed March 19th, 2026, with respect to the rejection of Claim 1, and necessarily dependent Claims 3 and 6-10 under U.S.C. 102 have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of the amended limitation of Claim 1, from which Claims 3 and 6-10 are dependent on, that reads “wherein the encoding unit (210) includes a first encoder (320), a second encoder (325) and a third encoder (330): the first encoder (320) is a three dimensional (3D) pose encoder that implements pixel wise depth mapping and 3D mesh modelling to estimate and map the shape of features and appearance of an object, the second encoder (325) is an expression encoder that estimates expression parameters from an upper half region of the object, and the third encoder (330) is a lip movement encoder that estimates and maps movements from a lower half of the object”.
As noted in the Rejections, Claim 1, 3 and 6-10 are rejected over Sinha (US Pub. No. US2021/0366173 A1) in view of Bednarik (Learning to Reconstruct Texture-Less Deformable Surfaces from a Single View, 2018) under U.S.C. 103.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to RAYMOND CHUN LAM LI whose telephone number is (571)272-5124. The examiner can normally be reached M-F 8:30-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kent Chang can be reached at 571-272-7667. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/RAYMOND CHUN LAM LI/Examiner, Art Unit 2614
/KENT W CHANG/Supervisory Patent Examiner, Art Unit 2614
Prosecution Insights