Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
Priority
No foreign or domestic priority is claimed. The effective filing date of U.S. Application No. 19/015,288 is 01/09/2025.
Status of Claims
Claims 1–20 are pending in the application. Claims 1-5, 7-15, 17-20 are rejected.
Claims 6, 16 are objected to.
Allowable Subject Matter
Claims 6, 16 are objected to as being dependent upon a rejected base claim(s), but would be allowable if rewritten in independent form including all of the limitations of the base claim(s) and any intervening claim(s).
Overview of Grounds of Rejection
Ground of Rejection
Claim(s)
Statute(s) (e.g., § 102, § 103)
Reference(s)
Ground 1
1, 2, 7, 11, 12, 19
§ 103
Saunders et al. in view of Ikeuchi et al.
Ground 2
3, 4, 13, 14, 20
§ 103
Saunders et al. in view of Ikeuchi et al., and further in view of Zhang et al.
Ground 3
5, 15
§ 103
Saunders et al. in view of Ikeuchi et al., further in view of Zhang et al., and still further in view of Tandon et al.
Ground 4
8, 9, 10, 17, 18
§ 103
Saunders et al. in view of Ikeuchi et al., and further in view of Zhang et al.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. § 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
(Please see the cited paragraphs, sections, pages, or surrounding text in the references for the paraphrased content.)
Ground of Rejection 1
Claims 1, 2, 7, 11, 12, 19 are rejected under 35 U.S.C. § 103 as being unpatentable over Saunders et al. (NPL) in view of Ikeuchi et al. (NPL).
As per Claim 1, Saunders et al. (NPL) teaches the following portion of claim which recites: “A computer implemented method comprising: accessing text;”
Saunders teaches, “Given a spoken language sequence, X, we first translate to a sign language grammar and order, represented by a gloss sequence.” Saunders et al. (NPL), § 3.1, Fig. 2, p. 5143. Saunders further states that its models are “implemented using PyTorch.” Saunders et al. (NPL), § 4.1, p. 5146. Thus, Saunders teaches a computer-implemented method that accesses input text for sign-language production.
Saunders et al. (NPL) teaches the following portion of claim which recites: “generating sets of sign language keypoints for the text;”
Saunders states that it produces “novel poses...from a given spoken language sentence” and further “produce[s] a continuous signing pose sequence...from the translated gloss sequence.” Saunders et al. (NPL), §§ 2, 3.2, pp. 5143-5144. The generated signing pose sequence provides the body pose/keypoint information corresponding to the input text.
Saunders alone does not explicitly teach all the limitation(s) of the claim. However, when combined with Ikeuchi et al. (NPL), they collectively teach all of the limitation(s).
Ikeuchi et al. (NPL) teach the following portion of claim, which recites: “generating skeletons for the sets of sign language keypoints;”
Ikeuchi et al. (NPL) teaches that its observation module “records human motions using a depth sensor and then coverts those measurements into skeleton data.” Figure 9 shows the resulting “Skeleton data” as connected body joints used for subsequent pose processing. Ikeuchi et al. (NPL), § 5.1, Fig. 9, p. 1424. Thus, Ikeuchi teaches generating a skeleton representation from body/joint position information.
Saunders et al. (NPL) teaches the following portion of claim which recites: “applying a texturing model to the skeletons, the texturing model have been trained on training data comprising training skeletons in interpretive poses and images of people in corresponding interpretive poses;”
Saunders teaches that “skeleton pose sequences are subsequently used to condition a video-to-video synthesis model” called SIGNGAN. Saunders et al. (NPL), § 1, Fig. 2, p. 5142. For training, Saunders uses a skeletal pose condition, where “each skeletal limb [is] plotted on a separate feature channel,” together with corresponding target/ground-truth signer images. Saunders et al. (NPL), §§ 3.3, 4.1, pp. 5145-5146. Thus, SIGNGAN corresponds to the claimed texturing model trained using skeletal poses and corresponding images of people.
Saunders et al. (NPL) teaches the following portion of claim which recites: “and obtaining images of an avatar in interpretative poses from the texturing model.”
Saunders teaches that SIGNGAN “synthesises images of a signer...given a human pose” and generates a “photo-realistic sign language video...from the continuous skeleton pose.” Saunders et al. (NPL), § 3.3, Fig. 2, pp. 5143, 5145. Thus, Saunders teaches obtaining generated images of a signer/avatar in the corresponding sign-language poses.
Before the effective filing date of the claimed invention, a person of ordinary skill in the art would have been motivated to apply the skeleton-generation technique of Ikeuchi to the sign-language pose/keypoint information generated by Saunders because Saunders already uses skeleton poses as input to SIGNGAN, while Ikeuchi teaches generating structured skeleton data from body/joint information. The combination would provide a suitable skeletal representation for Saunders's pose-conditioned image generation and would predictably result in generation of signer/avatar images corresponding to the sign-language poses.
PNG
media_image1.png
9
307
media_image1.png
Greyscale
As per Claim 2, Saunders et al. (NPL) teaches Claim 2 which recites: “The method of claim 1 and further comprising interpolating between the sets of sign language keypoints to generate sets of intermediate keypoints representing motion between the sign language keypoints.”
Saunders teaches that each sign is represented as a “sequence of skeleton pose,” and then states:
“We first convert the stack of dictionary signs into a continuous sequence by linearly interpolating between neighbouring signs for a predefined fixed NLI frames.”
Saunders further identifies the result as the “final interpolated dictionary sequence.” Saunders et al. (NPL), § 3.2, p. 5144.
Thus, Saunders teaches interpolating between neighboring sign-language skeletal/keypoint poses to generate intermediate pose frames representing motion between the signs.
PNG
media_image1.png
9
307
media_image1.png
Greyscale
As per Claim 7, Saunders et al. (NPL) teaches Claim 7 which recites: “The method of claim 1 wherein the sets of keypoints are normalized to adjust locations and dimensions within a maximum reference for which the model was trained.”
Saunders teaches using a “pre-trained 2D hand pose estimator” to extract hand keypoints from “cropped hand regions (i.e. a 60x60 patch centered around the middle knuckle)”. Saunders et al. (NPL), § 3.3, p. 5145. Thus, Saunders adjusts the location and dimensions of the hand/keypoint region to a fixed centered 60×60 reference before processing by the trained pose model.
PNG
media_image1.png
9
307
media_image1.png
Greyscale
Claim 11 does not include any additional limitations that would significantly distinguish it from claim 1. Therefore, it is likewise rejected under 35 U.S.C. § 103 in view of the same references and for the same reasons set forth above.
PNG
media_image1.png
9
307
media_image1.png
Greyscale
Claim 12 does not include any additional limitations that would significantly distinguish it from claim 2. Therefore, it is likewise rejected under 35 U.S.C. § 103 in view of the same references and for the same reasons set forth above.
PNG
media_image1.png
9
307
media_image1.png
Greyscale
Claim 19 does not include any additional limitations that would significantly distinguish it from claim 1. Therefore, it is likewise rejected under 35 U.S.C. § 103 in view of the same references and for the same reasons set forth above.
PNG
media_image1.png
9
307
media_image1.png
Greyscale
Ground of Rejection 2
Claims 3, 4, 13, 20 are rejected under 35 U.S.C. § 103 as being unpatentable over Saunders et al. (NPL) in view of Ikeuchi et al. (NPL), and further in view of Zhang et al. (US20210390945A1).
As per Claim 3, Saunders et al. (NPL) teaches the following porition of Claim 3 which recites: “The method of claim 2 and further comprising: generating intermediate skeletons for the sets of intermediate keypoints;”
Saunders et al. (NPL) teaches representing signs as a “sequence of skeleton pose” and “linearly interpolating between neighbouring signs,” with the resulting sequence comprising the skeleton pose and interpolation. Saunders et al. (NPL), § 3.2, p. 5144. Thus, Saunders teaches generating intermediate skeleton poses corresponding to the intermediate sign pose/keypoint frames.
Saunders alone does not explicitly teach all the limitation(s) of the claim. However, when combined with Ikeuchi et al. (NPL), they collectively teach all of the limitation(s).
Zhang and Saunders teach the following portion of claim, which recites: “applying the texturing model to the intermediate skeletons, the texturing model have been further trained on the training data that includes intermediate skeletons and training data video of one or more persons signing training text; and”
Zhang et al. teaches that “a generative adversarial network (GAN) is trained to generate video from interpolated phoneme poses” and that training image sequences and their “corresponding poses” from training videos are input to the GAN. Zhang et al., ¶¶ [0037], [0078]-[0082]. Zhang further teaches interpolation producing new pose frames between key poses. Zhang et al., ¶¶ [0100]-[0101].
Saunders supplies the sign-language context, including sign-language videos, skeleton-pose-conditioned SIGNGAN, and generation of photo-realistic signer video. Saunders et al. (NPL), §§ 3.3, 4.1, pp. 5145-5146.
Saunders et al. (NPL) teaches the following porition of Claim 3 which recites: “obtaining intermediate images of an avatar in intermediate poses from the texturing model.”
Saunders et al. (NPL) teaches that SIGNGAN “synthesises images of a signer ... given a human pose,” and Figure 4 shows successive skeleton poses with corresponding photo-realistic signer frames generated by SIGNGAN. Saunders et al. (NPL), § 3.3, Fig. 4, pp. 5145-5146.
Before the effective filing date of the claimed invention, a POSITA would have been motivated to modify Saunders et al. (NPL) with Zhang et al. (US20210390945A1) by training and applying the generative video model to interpolated poses, because Saunders already generates interpolated skeleton poses for sign-language production, while Zhang teaches training a GAN to generate photorealistic video from interpolated poses. Such combination would improve continuity and realism of the generated signing motion and predictably produce intermediate signer images corresponding to the intermediate poses.
PNG
media_image1.png
9
307
media_image1.png
Greyscale
As per Claim 4, Saunders et al. (NPL) teaches Claim 4 which recites: “The method of claim 3 and further comprising concatenating the obtained images of an avatar to generate a video of the avatar interpreting the text in sign language.”
Saunders defines the output as a continuous photo-realistic sign-language video, “ with T frame[s]”, and teaches that SIGNGAN “synthesises images of a signer” from the respective poses. Saunders further states that its pipeline translates a spoken-language sequence into a “photo-realistic sign language video.” Saunders et al. (NPL), §§ 3, 3.3, Fig. 2, pp. 5143, 5145.
Thus, Saunders teaches combining the successive generated signer images/frames into a video representing the input text in sign language.
PNG
media_image1.png
9
307
media_image1.png
Greyscale
Claim 13 does not include any additional limitations that would significantly distinguish it from claim 3. Therefore, it is likewise rejected under 35 U.S.C. § 103 in view of the same references and for the same reasons set forth above.
PNG
media_image1.png
9
307
media_image1.png
Greyscale
Claim 14 does not include any additional limitations that would significantly distinguish it from claim 4. Therefore, it is likewise rejected under 35 U.S.C. § 103 in view of the same references and for the same reasons set forth above.
PNG
media_image1.png
9
307
media_image1.png
Greyscale
Claim 20 does not include any additional limitations that would significantly distinguish it from claim 3. Therefore, it is likewise rejected under 35 U.S.C. § 103 in view of the same references and for the same reasons set forth above.
PNG
media_image1.png
9
307
media_image1.png
Greyscale
Ground of Rejection 3
Claims 5, 15 are rejected under 35 U.S.C. § 103 as being unpatentable over Saunders et al. (NPL) in view of Ikeuchi et al. (NPL), further in view of Zhang et al. (US20210390945A1), and still further in view of Tandon et al. (US20190130585A1).
As per Claim 5, Saunders alone does not explicitly teach all the limitation(s) of the claim. However, when combined with Tandon, they collectively teach all of the limitation(s).
Saunders and Tandon teach Claim 5 which recites: “The method of claim 3 and further comprising cleaning the training data by removing video frames having movement caused blur.”
Saunders recognizes “motion blur in sign language datasets from fast moving hands.” Saunders et al. (NPL), § 1, p. 5142.
Tandon teaches “excluding the partially motion-blurred video frames from selection” so that selected video frames are relatively free of motion blur. Tandon et al. (US20190130585A1), ¶¶ [0010], [0013]. Thus, Tandon teaches removing movement-caused blurred frames, which can be used to clean Saunders's sign-language training data.
Before the effective filing date of the claimed invention, a POSITA would have been motivated to apply Tandon's motion-blur frame removal technique to Saunders's sign-language training videos to reduce blurred training samples caused by fast hand movement and improve the quality of the training data, with the predictable result of cleaner input data for training the video-generation model.
PNG
media_image1.png
9
307
media_image1.png
Greyscale
Claim 15 does not include any additional limitations that would significantly distinguish it from claim 5. Therefore, it is likewise rejected under 35 U.S.C. § 103 in view of the same references and for the same reasons set forth above.
PNG
media_image1.png
9
307
media_image1.png
Greyscale
Ground of Rejection 4
Claims 8, 9, 10, 17, 18 are rejected under 35 U.S.C. § 103 as being unpatentable over Saunders et al. (NPL) in view of Ikeuchi et al. (NPL), and further in view of Zhang et al. (US20210390945A1).
As per Claim 8, Saunders alone does not explicitly teach all the limitation(s) of the claim. However, when combined with Zhang, they collectively teach all of the limitation(s).
Saunders and Zhang teach Claim 8 which recites: “The method of claim 1 wherein generating skeletons comprises using different colors for a face, body, and arms.”
Saunders teaches a skeleton heat-map representation in which “each skeletal limb [is] plotted on a separate feature channel.” Saunders et al. (NPL), § 4.1, p. 5146.
Zhang further teaches an input skeleton image in which “a color circle is drawn on the face” and the face region is identified by its “special colors.” Zhang et al. (US20210390945A1), ¶ [0080].
Before the effective filing date of the claimed invention, a POSITA would have been motivated to apply Zhang’s color-based distinction of skeletal regions to Saunders’s separate skeletal feature channels so that the face, body, and arms are more clearly distinguished, with the predictable result of an easier-to-process color-coded skeleton representation.
PNG
media_image1.png
9
307
media_image1.png
Greyscale
As per Claim 9, Saunders alone does not explicitly teach all the limitation(s) of the claim. However, when combined with Zhang, they collectively teach all of the limitation(s).
Zhang teaches Claim 9 which recites: “The method of claim 1 wherein generating skeletons comprises generating eyes, a nose, and a mouth.”
Zhang teaches that OpenPose “jointly detect[s] human body, hand, facial, and foot keypoints” and further uses “all 68 face keypoints”, separately identifying mouth keypoints as points 48-67. Zhang et al. (US20210390945A1), ¶¶ [0056], [0113]-[0114]. Thus, Zhang teaches generating a facial keypoint/skeleton representation containing facial landmarks corresponding to the eyes, nose, and mouth.
Before the effective filing date of the claimed invention, a POSITA would have been motivated to apply Zhang’s facial keypoint representation to the skeleton generation of Saunders so that facial features, including the eyes, nose, and mouth, are represented along with the body pose, with the predictable result of a more complete skeleton for sign-language image generation.
PNG
media_image1.png
9
307
media_image1.png
Greyscale
As per Claim 10, Saunders alone does not explicitly teach all the limitation(s) of the claim. However, when combined with Zhang, they collectively teach all of the limitation(s).
Zhang teaches Claim 10 which recites: “The method of claim 1 wherein the skeletons and training skeletons are generated using a same skeleton generator.”
Zhang teaches using OpenPose as the key pose extractor “for extracting key poses from training videos.” Zhang et al., ¶ [0071]. The corresponding training poses are then used to train the generative model. Zhang et al., ¶ [0078]. At inference, the same OpenPose-derived phoneme-pose dictionary is used, with interpolation, to generate the pose sequence corresponding to the input text. Zhang et al., ¶¶ [0092]-[0093]. Thus, the training poses and generated poses originate from the same OpenPose-based pose/skeleton generation framework.
Before the effective filing date of the claimed invention, a POSITA would have been motivated to use Zhang’s same OpenPose-based pose/skeleton generator for both training and generated poses to maintain consistent pose representation and model compatibility, with predictable results.
PNG
media_image1.png
9
307
media_image1.png
Greyscale
Claim 17 does not include any additional limitations that would significantly distinguish it from claims 7, 8, and 9. Therefore, it is likewise rejected under 35 U.S.C. § 103 in view of the same references and for the same reasons set forth above.
PNG
media_image1.png
9
307
media_image1.png
Greyscale
Claim 18 does not include any additional limitations that would significantly distinguish it from claim 10. Therefore, it is likewise rejected under 35 U.S.C. § 103 in view of the same references and for the same reasons set forth above.
PNG
media_image1.png
9
307
media_image1.png
Greyscale
Conclusion
The prior art made of record and relied upon in this action is as follows:
Patent Literature:
Tandon et al. (US20190130585A1) — “Detection of partially motion-blurred video frames.”
Zhang et al. (US20210390945A1) — “Text-driven video synthesis with phonetic dictionary.”
Non-Patent Literature (NPL):
Ikeuchi et al. — “Describing Upper-Body Motions Based on Labanotation for Learning-from-Observation Robots,” 2018-10-05. Available at: [https://link.springer.com/article/10.1007/s11263-018-1123-1]
Saunders et al. — “Signing at Scale: Learning to Co-Articulate Signs for Large-Scale Photo-Realistic Sign Language Production,” June 2022. Available at: [https://openaccess.thecvf.com/content/CVPR2022/papers/Saunders_Signing_at_Scale_Learning_to_Co-Articulate_Signs_for_Large-Scale_Photo-Realistic_CVPR_2022_paper.pdf]. For date: [https://openaccess.thecvf.com/content/CVPR2022/html/Saunders_Signing_at_Scale_Learning_to_Co-Articulate_Signs_for_Large-Scale_Photo-Realistic_CVPR_2022_paper.html]
Note: A PDF copy of each NPL reference is attached with this Office Action. URLs are included for applicant convenience. If a link becomes unavailable in the future, the citation information may be used to locate the reference or access archived versions via the Wayback Machine.
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure and is listed as follows:
Patent Literature:
Baran et al. (US20170359549A1) — “Video capture with frame rate based on estimate of motion periodicity.”
Non-Patent Literature (NPL):
(none)
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ADEEL BASHIR whose telephone number is (571) 270-0440. The examiner can normally be reached Monday-Thursday.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Daniel Hajnik can be reached on (571) 276-7642. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ADEEL BASHIR/
Examiner, Art Unit 2616
/DANIEL F HAJNIK/Supervisory Patent Examiner, Art Unit 2616
Prosecution Insights