DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 3-5, 10, 19-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Schnan (US12303255B1) in view of Lee (US 20240296585 A1).
Regarding Claim 1, Schnan teaches computer-implemented method comprising:
obtaining an input video including a plurality of video frames in a sequence, wherein the video frames include pixels that depict movement of a person based on a plurality of poses of the person in the input video. Para [35] discloses “FIG. 4 is a flowchart illustrating methods 400 for video-based gait characterization (interpreted as the input video) and analysis (interpreted as depict movement) in accordance with various embodiments (interpreted as person). The methods begin with the acquisition of a monocular video frame sequence of a subject walking 402”;
detecting, by at least one processor, keypoints of the person in the video frames of the input video. Para [35] discloses “The video frames (interpreted as video frames for the input video) are processed, e.g., in stages using the machine-learned 2D keypoint model 104 and 3D keypoint model 108 of the processing pipeline 100, to compute 3D keypoint coordinates for a set of anatomical keypoints associated with various joints and body segments (interpreted as detecting joints and body segments of a person) 404”;
and determining, by the at least one processor, a sequence of 3D body poses that correspond to the plurality of poses of the person in the video frames of the input video. Para [35] discloses “In one example, thirty-five 3D keypoints are predicted by the models (interpreted as persons). These 3D keypoints are then further processed, e.g., using the machine-learned gait models 130, 132, 160, to predict joint angles and/or body-segment rotations (406) (interpreted as body poses of a person), and/or to detect gait events in the video frames (interpreted as sequence) and classify the frames accordingly (408)”, wherein the spatial-temporal transformer separately encodes inputs in a temporal dimension across the video frames. “Based on the gait contact event classification (interpreted as spatial-temporal transformer) 162 for each frame, gait contact events of a given type can be associated with specific frames and/or their associated time stamps (interpreted as temporal dimension)” (Schnan, [25]).
Schnan does not explicitly discloses, but Lee teaches spatial dimensions within each video frame. “For example, if the first coordinates of an object for one point in a 3D full point cloud are (x1, y1, z1) and the second coordinate are (x2, y2, z2), the one point may be expressed by the 6D coordinates of (x1, x2, y1, y2, z1, z2) by concatenating the first and second coordinates (3D full point cloud coordinates are interpreted as joint angles of the keypoints)” (Lee, [0079]).
Regarding Claim 3, Lee teaches wherein determining the sequence of 3D body poses includes determining, by the at least one processor, a global translation in 3D world coordinates for the 3D body poses. Para [0020] discloses “ The 3D full point cloud may include a 3D camera-based first coordinates and an object-based second coordinates matched to the 3D camera-based first coordinates, and the processor may be configured to estimate the 6D pose of the object using a transformation matrix to minimize an error between the first coordinates (interpreted as 3D world coordinates) and the second coordinates (interpreted as 3D body poses)”.
Schnan and Lee are combinable because they are in the same field of endeavor regarding subject video extraction. It would have been obvious to a person having ordinary skill in the art before the effective filing date of the application to combine the input video of 3D body poses of Schnan and spatial-temporal transformer of Lee in order to provide the advantage of real-time 6D pose estimation (Lee, [0004]).
Regarding Claim 4, Schnan teaches wherein determining the global translation in 3D world coordinates includes predicting translation velocity of the global translation. Para [2] discloses “a pressure-sensitive walkway (such as, e.g., GAITRite® available from CIR Systems, Inc., Franklin, NJ) can be used to determine the time and location of various gait events occurring during a gait cycle, such as “heel strike” (i.e., the heel hitting the ground), “heel off” (i.e., the heel lifting off the ground), and “toe off” (i.e., the toe lifting off the ground), from which gait parameters such as step size, gait speed, etc. can be computed”. Para [30] discloses” Spatiotemporal parameters include, for example and without limitation, averages or variabilities (e.g., standard deviations or variances) over generally multiple gait cycles of: walking speed (e.g., measured in cm/s…”. Gait cycle or gate speed are interpreted as translation velocity.
Regarding Claim 5, Lee teaches wherein determining the global translation in 3D world coordinates includes using a second spatial-temporal transformer that separately encodes inputs in the spatial dimensions within each video frame and in the temporal dimension across the video frames. Para [0101] discloses “the modeling converter 160 may include the first autoencoder 210, the second autoencoder 220”. The second spatial-temporal transformer is interpreted as the second autoencoder.
Schnan and Lee are combinable because they are in the same field of endeavor regarding subject video extraction. It would have been obvious to a person having ordinary skill in the art before the effective filing date of the application to combine the input video of 3D body poses of Schnan and spatial-temporal transformer of Lee in order to provide the advantage of real-time 6D pose estimation (Lee, [0004]).
Regarding Claim 19, Schnan teaches a non-transitory computer-readable medium with instructions stored thereon that, when executed by a processor, cause the processor to perform operations comprising:
obtaining an input video including a plurality of video frames in a sequence, wherein the video frames include pixels that depict movement of a person based on a plurality of poses of the person in the input video. Para [35] discloses “FIG. 4 is a flowchart illustrating methods 400 for video-based gait characterization (interpreted as the input video) and analysis (interpreted as depict movement) in accordance with various embodiments (interpreted as person). The methods begin with the acquisition of a monocular video frame sequence of a subject walking 402”;
detecting keypoints of the person in the video frames of the input video. Para [35] discloses “The video frames (interpreted as video frames for the input video) are processed, e.g., in stages using the machine-learned 2D keypoint model 104 and 3D keypoint model 108 of the processing pipeline 100, to compute 3D keypoint coordinates for a set of anatomical keypoints associated with various joints and body segments (interpreted as detecting joints and body segments of a person) 404”;
and determining a plurality of joint angles for the detected keypoints that provide a sequence of 3D body poses corresponding to poses of the person in the video frames of the input video. Para [35] discloses “In one example, thirty-five 3D keypoints are predicted by the models (interpreted as persons). These 3D keypoints are then further processed, e.g., using the machine-learned gait models 130, 132, 160, to predict joint angles and/or body-segment rotations (406) (interpreted as body poses of a person), and/or to detect gait events in the video frames (interpreted as sequence) and classify the frames accordingly (408)”;
and determining a global translation in 3D world coordinates for the 3D body poses using a transformer that predicts translation velocity of the global translation. Para [2] discloses “a pressure-sensitive walkway (such as, e.g., GAITRite® available from CIR Systems, Inc., Franklin, NJ) can be used to determine the time and location of various gait events occurring during a gait cycle, such as “heel strike” (i.e., the heel hitting the ground), “heel off” (i.e., the heel lifting off the ground), and “toe off” (i.e., the toe lifting off the ground), from which gait parameters such as step size, gait speed, etc. can be computed”. Para [30] discloses” Spatiotemporal parameters include, for example and without limitation, averages or variabilities (e.g., standard deviations or variances) over generally multiple gait cycles of: walking speed (e.g., measured in cm/s…”. Gait cycle or gate speed are interpreted as translation velocity.
Regarding Claim 20, wherein the transformer is a spatial-temporal transformer that separately encodes inputs in spatial dimensions within each video frame and in a temporal dimension across the video frames. “Based on the gait contact event classification (interpreted as spatial-temporal transformer) 162 for each frame, gait contact events of a given type can be associated with specific frames and/or their associated time stamps (interpreted as temporal dimension)” (Schnan, [25]).
Schnan does not explicitly discloses, but Lee teaches spatial dimensions within each video frame. “For example, if the first coordinates of an object for one point in a 3D full point cloud are (x1, y1, z1) and the second coordinate are (x2, y2, z2), the one point may be expressed by the 6D coordinates of (x1, x2, y1, y2, z1, z2) by concatenating the first and second coordinates (3D full point cloud coordinates are interpreted as joint angles of the keypoints)” (Lee, [0079]).
Claim(s) 2 is/are rejected under 35 U.S.C. 103 as being unpatentable over Schnan (US12303255B1) in view of Lee (US 20240296585 A1, further in view of Cetin (Learning in-the-wild).
Regarding Claim 2, Lee teaches wherein the spatial-temporal transformer outputs 6-dimensional (6D) circular representations of the joint angles of the keypoints of the 3D body poses, "the pose estimator (interpreted as the spatial-temporal transformer) 170 may estimate the 6D pose of the object from the 3D full point cloud of the object output (interpreted as 3D body poses) from the modeling converter 160 and output the estimated 6D pose S” [0106].
Lee does not explicitly teach, but Cetin teaches the method further comprises converting the 6D circular representations of the joint angles into 3-dimensional (3D) joint angles of the keypoints. “The also present a method to continuously represent rotations in fewer dimension and obtain continuous representations for 3D rotations in 5D and 6D… Similarly, Equation 3 illustrate how these 6D rotations can be converted back to rotation matrices, where N(v)=v/||v|| denote vector normalization” (Pg. 9, fig. 3).
Schnan, Lee, and Cetin are combinable because they are in the same field of endeavor regarding 3D and 6D poses. It would have been obvious to a person having ordinary skill in the art before the effective filing date of the application to combine 6D keypoints of 3D body poses of Schnan, spatial-temporal transformer of Lee, and converting 6D into 3D keypoints of Cetin in order to result framework employs a simple model, trained on poses without paired images, that can achieve competitive performance on common evaluation scenarios (Cetin, abstract).
8. Regarding Claim 11, Schnan teaches a system comprising:
at least one processor; and a memory coupled to the at least one processor, with software instructions stored thereon that, when executed by the at least one processor, cause the at least one processor to perform operations comprising:, para [14] discloses “The processing facility may be, for example, a computer or computer cluster including one or more general-purpose hardware processors (interpreted as a processor) that execute instructions (interpreted as software instructions) stored in computer-readable memory”.
obtaining an input video including a plurality of video frames in a sequence, wherein the video frames include pixels that depict movement of a person based on a plurality of poses of the person in the input video; Para [35] discloses “FIG. 4 is a flowchart illustrating methods 400 for video-based gait characterization (interpreted as the input video) and analysis (interpreted as depict movement) in accordance with various embodiments (interpreted as person). The methods begin with the acquisition of a monocular video frame sequence of a subject walking 402”
detecting keypoints of the person in the video frames of the input video; determining, using a transformer, Para [35] discloses “In one example, thirty-five 3D keypoints are predicted by the models (interpreted as persons). These 3D keypoints are then further processed, e.g., using the machine-learned gait models 130, 132, 160, to predict joint angles and/or body-segment rotations (406) (interpreted as body poses of a person), and/or to detect gait events in the video frames (interpreted as sequence) and classify the frames accordingly (408)”, 6-dimensional (6D) circular representations of joint angles of the keypoints, wherein the transformer outputs the 6D circular representations of the joint angles of the keypoints. "The pose estimator (interpreted as the 6D circular representations) 170 may estimate the 6D pose of the object from the 3D full point cloud of the object output (interpreted as 3D body poses) from the modeling converter 160 and output the estimated 6D pose S” [0106];
converting the 6D circular representations of the joint angles into 3-dimensional (3D) joint angles of the keypoints. Cetin teaches the method further comprises converting the 6D circular representations of the joint angles into 3-dimensional (3D) joint angles of the keypoints. “The also present a method to continuously represent rotations in fewer dimension and obtain continuous representations for 3D rotations in 5D and 6D… Similarly, Equation 3 illustrate how these 6D rotations can be converted back to rotation matrices, where N(v)=v/||v|| denote vector normalization” (Pg. 9, fig. 3);
outputting a sequence of 3D body poses that correspond to the plurality of poses of the person in the video frames of the input video, wherein the sequence of 3D body poses includes the 3D joint angles of the keypoints for the video frames of the input video. “The video may be streamed over a mobile connection to a separate computer or computer cluster, e.g., … and outputs of the processing pipeline, such as the 3D keypoint coordinates, joint angles and body-segment rotations, gait event classifications and gait-phase labels (interpreted as 3D body poses)” (Schnan, [55]).
Claim(s) 6-7 is/are rejected under 35 U.S.C. 103 as being unpatentable over Schnan (US12303255B1) in view of Lee (US 20240296585 A1) and further in view of Liu (Fast Tracking Algorithm).
Regarding 6 and 7, Liu teaches further comprising smoothing, by the at least one processor, jitters in the sequence of 3D body poses using a smoothing filter that includes an optimization solver; wherein the optimization solver includes an alternating direction method of multipliers (ADMM) solver. “The proposed fast-tracking algorithm, based on a spatially regularized correlation filter (interpreted as a smoothing filter), aims to improve the speed and accuracy of the SRDCF algorithm [13]. First, this algorithm replaces the filter solver method in the SRDCF tracker from the Gauss–Seidel solver to the ADMM [14] solver (interpreted as ADMM solver), improving the computational speed and tracking efficiency” (Pg. 5, Section 3.2).
Schnan, Lee, and Liu are combinable because they are in the same field of endeavor regarding spatial characterization. It would have been obvious to a person having ordinary skill in the art before the effective filing date of the application to combine an input video of Schnan, spatial dimension of Lee, and ADMM solver with filters of Liu in order to exhibits a more stable and accurate tracking performance in the presence of occlusion and background clutter during tracking (Liu, Abstract).
Claim(s) 8-9 is/are rejected under 35 U.S.C. 103 as being unpatentable over Schnan (US12303255B1) in view of Lee (US 20240296585 A1) and further in view of Logothetis (US20220051471A1).
Regarding Claim 8 and 9, Logothetis teaches wherein the smoothing filter minimizes an acceleration of one or more keypoints in the sequence of 3D body poses; wherein the smoothing filter minimizes an L1 recovery error for the acceleration of the one or more keypoints in the sequence of 3D body poses. Para [0130] discloses “A normal map is obtained from the BRDF samples by the CNN, and through numerical integration of the Normal map, a new estimate of the object geometry (depth map) is obtained. The numerical integration is an iterative ADMM process with a l.sub.1 loss function (interpreted as L1 recovery error). The variation optimisation includes Tikhonov regulariser z=z.sub.0 (weight λ=10.sup.−6)”.
Schnan, Lee, and Logothetis are combinable because they are in the same field of endeavor regarding image data processing. It would have been obvious to a person having ordinary skill in the art before the effective filing date of the application to combine an input video of Schnan, spatial dimension of Lee, and L1 recovery error in order to allow for using a fast-to-obtain training data while still allowing network to learn global illumination effects and real-world imperfections (Logothetis, [0003]).
Claim(s) 10 is/are rejected under 35 U.S.C. 103 as being unpatentable over Schnan (US12303255B1) in view of Lee (US 20240296585 A1, further in view of Mafia Game Videos (Behind the Scenes).
Regarding Claim 10, Mafia Game Videos teaches further comprising applying the sequence of 3D body poses to an avatar in a virtual environment to cause an animation of the avatar based on the sequence of 3D body poses that corresponds to the movement of the person in the input video. The video shows sequence of poses and virtual avatar uses sequence of poses.
Schnan, Lee, and Mafia Game Videos are combinable because they are in the same field of endeavor regarding image data processing. It would have been obvious to a person having ordinary skill in the art before the effective filing date of the application to combine an input video of Schnan, spatial dimension of Lee, and applying sequence of 3D body poses to an avatar in order to make the animation smoother.
Claim(s) 12-18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Schnan (US12303255B1) in view of Lee (US20240296585A1), further in view of Cetin
Regarding Claim 12, Schnan teaches wherein the spatial-temporal transformer separately encodes inputs in a temporal dimension across the video frames. “Based on the gait contact event classification (interpreted as spatial-temporal transformer) 162 for each frame, gait contact events of a given type can be associated with specific frames and/or their associated time stamps (interpreted as temporal dimension)” (Schnan, [25]).
Schnan does not explicitly discloses, but Lee teaches spatial dimensions within each video frame. “For example, if the first coordinates of an object for one point in a 3D full point cloud are (x1, y1, z1) and the second coordinate are (x2, y2, z2), the one point may be expressed by the 6D coordinates of (x1, x2, y1, y2, z1, z2) by concatenating the first and second coordinates (3D full point cloud coordinates are interpreted as joint angles of the keypoints)” (Lee, [0079]).
Regarding Claim 13, wherein the operations further include determining a global translation in 3D world coordinates for the 3D body poses using a transformer that predicts translation velocity of the global translation, the claim is directed to a system claim with similar limitations as Claim 4. As such Claim 13 is rejected on the same grounds.
Regarding Claim 14, wherein the operation of determining the global translation in 3D world coordinates includes using a second spatial-temporal transformer that separately encodes inputs in spatial dimensions within each video frame and in a temporal dimension across the video frames, the claim is directed to a system claim with the similar limitations as Claim 5. As such Claim 14 is rejected on the same grounds.
Regarding Claim 15, wherein the operations further comprise smoothing jitters in the sequence of 3D body poses using a smoothing filter that includes an optimization solver, the claim is directed to a system claim with the similar limitations as Claim 6. As such Claim 15 is rejected on the same grounds.
Regarding Claim 16, wherein the smoothing filter minimizes an acceleration of one or more keypoints in the sequence of 3D body poses, the claim is directed to a system claim with the similar limitations as Claim 8. As such Claim 16 is rejected on the same grounds.
Regarding Claim 17, wherein the optimization solver includes an alternating direction method of multipliers (ADMM) solver, the claim is directed to a system claim with the similar limitations as Claim 7. As such Claim 17 is rejected on the same grounds.
Regarding Claim 18, wherein the operations further comprise applying the sequence of 3D body poses to an avatar in a virtual environment to cause an animation of the avatar based on the sequence of the 3D body poses that corresponds to the movement of the person in the input video, the claim is directed to a system claim with the similar limitations as Claim 10. As such Claim 18 is rejected on the same grounds.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to BRIANA SZE whose telephone number is (571)272-9916. The examiner can normally be reached Monday-Thursday 6am-4pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kent Chang can be reached at (571) 272-7667. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/B.S./Examiner, Art Unit 2614
/TERRELL M ROBINSON/Primary Examiner, Art Unit 2614
Prosecution Insights