DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Claims 6, 28, and 31-32 are objected to because of the following informalities:
Claim 6 recites the limitation "said generating a query" in line 1. There is insufficient antecedent basis for this limitation in the claim. For the sake of examination, claim 6 is interpreted to be dependent on claim 3.
Claim 28 recites the limitation "each of the N primary keypoints" in line 2. There is insufficient antecedent basis for this limitation in the claim. For the sake of examination, claim 28 is interpreted to be dependent on claim 27.
Claim 31 recites the limitation "actuating the autonomous device" in line 5. There is insufficient antecedent basis for this limitation in the claim. “the autonomous device” in line 5 should read “an autonomous device”.
Claim 32 recites the limitation " actuating the autonomous device" in line 5. There is insufficient antecedent basis for this limitation in the claim. “the autonomous device” should read “an autonomous device”.
In claim 33, it is unclear whether the “image capturing device” in line 2 is the same or different from the “image capturing device” in claim 23.
Appropriate correction is required.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Goel et al. (Humans in 4D: Reconstructing and Tracking Humans with Transformers), hereinafter Goel.
Regarding claim 1, Goel teaches a computer-implemented method for recovering a three-dimensional (3D) mesh of N humans in a 3D scene (Paragraph 1-2 in 2nd Col. of Page 1, Paragraph 1 in 1st Col. of Page 2 – “In this paper, we present a fully transformer-based approach for recovering 3D meshes of human bodies from single images, and tracking them over time in video…We take that as a starting point and we develop a new version of HMR, which we call HMR 2.0 to acknowledge its antecedent. We use HMR 2.0 to build a system that can simultaneously reconstruct and track humans from videos… This system can reconstruct Humans in 4D, which gives the name to our method, 4DHumans. 4DHumans can be deployed on any video and can jointly track and reconstruct people in video”; Note: 4DHumans is a method that recovers 3D meshes of humans. It is implied to be computer-implemented since it would not be able to be executed otherwise), the method comprising:
receiving a two-dimensional (2D) image of the scene from an image capturing device, the 2D image including a plurality of regions (Paragraph 4 in 1st Col. of Page 3, Paragraph 1 and 4 in 2nd Col. of Page 3 – “Each camera π = (R,t) consists of a global orientation R ∈ R3×3 and translation t ∈ R3. Given these parameters, points in the SMPL space (e.g., joints X) can be projected to the image as x =π(X)=Π(K(RX+t)), where Π is a perspective projection with camera intrinsics K…The input image is first patchified into input tokens...We use a ViT-H/16, the ‘Huge’ variant with 16×16 input patch size”; Note: the input image is the 2D image of the scene. It is captured by a camera, and it includes a plurality of patches/regions);
by one or more processors, encoding the received image to extract embedded features for each of the plurality of regions (Paragraph 4 in 2nd Col. of Page 3 – “The Vision Transformer, or ViT [15] is a transformer [74] that has been modified to operate on an image. The input image is first patchified into input tokens and passed through the transformer to get output tokens. The output tokens are then passed to the transformer decoder. We use a ViT-H/16, the “Huge” variant with 16×16 input patch size”; Note: the vision transformer encodes the input image into tokens, which are equivalent to the embedded features for each of the regions/patches. It is implied that the encoding and the steps below are performed by a processor since they would not be able to be executed otherwise);
by one or more processors, detecting N humans in N respective regions among the plurality of regions (Paragraph 3 in 1st Col. of Page 4, Paragraph 1 in 2nd Col. of Page 4, Paragraph 4 in 2nd Col. of Page 13, Paragraph 1 in 1st Col. of Page 14 – “The basic idea is to detect people in individual frames, and “lift” them to 3D, extracting their 3D pose, location in 3D space (derived from the estimated camera), and 3D appearance (derived from the texture map). A tracklet representation is incrementally built up for each individual person over time… Each output token regresses the 3D pose and 3D location of the person at the specified time-step”; Note: humans are detected in the image, and the output tokens from the vision transformer represent each person in a corresponding region of the image, making it obvious that there are N people and N regions where those people are located);
by one or more processors, processing the embedded features in the N respective regions and the embedded features for each of the plurality of regions to predict body model and depth parameters for each of the N detected humans (Paragraph 2 and 5 in 2nd Col. of Page 3, Paragraph 4 in 2nd Col. of Page 13, Paragraph 1 in 1st Col. of Page 14 – “we model f to predict Θ = [θ,β,π] = f(I) where θ and β are the SMPL pose and shape parameters and π is the camera translation…We use a standard transformer decoder [74] with multi-head self-attention. It processes a single (zero) input token by cross-attending to the output image tokens and ends with a linear readout of Θ…Each output token regresses the 3D pose and 3D location of the person at the specified time-step”; Note: the tokens (embedded features) are processed in a decoder to predict pose and shape parameters (body model parameters) and camera translation (depth parameter) for each person), wherein said processing uses a decoder comprising a cross-attention module (Paragraph 2 in 1st Col. of Page 13 – “Our transformer decoder is a standard transformer decoder architecture [23] with 6 layers, each containing multi-head self-attention, multi-head cross-attention, and feed-forward blocks, with layer normalization”);
providing the predicted body model parameters for each of the N detected humans to a 3D parametric model for generating N 3D meshes (Paragraph 3 in 1st Col. of Page 3, Paragraph 2 and 5 in 2nd Col. of Page 3 – “Given input parameters for pose (θ ∈ R24×3×3) and shape (β ∈ R10), it outputs a mesh M ∈ R3×N with N = 6890 vertices…we model f to predict Θ = [θ,β,π] = f(I) where θ and β are the SMPL pose and shape parameters and π is the camera translation…We use a standard transformer decoder [74] with multi-head self-attention. It processes a single (zero) input token by cross-attending to the output image tokens and ends with a linear readout of Θ”; Note: the predicted parameters are input into SMPL model, which is a 3D parametric model, to generate 3D meshes);
and placing each of the N generated meshes at a respective 3D spatial location in the 3D scene based on the predicted depth parameters (Fig. 1, Paragraph 4 in 1st Col. of Page 3, Paragraph 1-2 in 2nd Col. of Page 3 – “Each camera π = (R,t) consists of a global orientation R ∈ R3×3 and translation t ∈ R3. Given these parameters, points in the SMPL space (e.g., joints X) can be projected to the image as x =π(X)=Π(K(RX+t)), where Π is a perspective projection with camera intrinsics K… The goal of the human mesh reconstruction (HMR) task is to learn a predictor f(I) that given a single image I, reconstructs the person in the image by predicting their 3D pose and shape parameters”; Note: the human mesh is projected/placed in the image based on a 3D translation, which represents the depth. Fig. 1 shows how N generated meshes are placed in the 3D scene; see screenshot of Fig. 1 below).
PNG
media_image1.png
464
1154
media_image1.png
Greyscale
Screenshot of Fig. 1 (taken from Goel)
Regarding claim 2, Goel teaches the method of claim 1. Goel further teaches wherein N is greater than one (Fig. 2 – The figure shows that there is more than one human; see screenshot below), the body model parameters comprise pose and shape parameters (Paragraph 2 in Col. 2 of Page 3 – “we model f to predict Θ = [θ,β,π] = f(I) where θ and β are the SMPL pose and shape parameters and π is the camera translation”), and each generated 3D mesh is a whole-body mesh (Fig. 2 – The figure shows that the generated 3D meshes represent the whole body; see screenshot below).
PNG
media_image2.png
359
238
media_image2.png
Greyscale
Screenshot of Fig. 2 (taken from Goel)
Regarding claim 20, Goel teaches the method of claim 1. Goel further teaches wherein for each generated 3D mesh, the output parameters for generating each 3D mesh have a dimension that is lower than a dimension of each generated 3D mesh (Paragraph 3 in 1st Col. of Page 3 – “parameters for pose (θ ∈ R24×3×3) and shape (β ∈ R10), it outputs a mesh M ∈ R3×N with N = 6890 vertices”; Note: the parameters for generating the mesh have a dimension less than the dimension of the mesh; the dimension is indicated by the exponent of the real number symbol R).
Claims 3-5 are rejected under 35 U.S.C. 103 as being unpatentable over Goel in view of Meinhardt et al. (TrackFormer: Multi-Object Tracking with Transformers), hereinafter Meinhardt.
Regarding claim 3, Goel teaches the method of claim 2. Goel does not teach wherein said processing the embedded features for each of the detected N humans comprises: generating a query from the embedded features for each of the N respective regions to provide N generated cross-attention queries; and inputting the N generated cross-attention queries and the embedded features for each of the plurality of regions into the cross-attention module, wherein the embedded features for each of the plurality of regions provide cross-attention keys and values for the cross-attention module. However, Meinhardt teaches generating a query from the embedded features for each of the N respective regions to provide N generated cross-attention queries (Paragraph 4-5 in 2nd Col. of Page 3, Paragraph 7 in 1st Col. of Page 6 – “we introduce the concept of track queries to the decoder. Track queries follow objects through a video sequence carrying over their identity information while adapting to their changing position in an autoregressive manner. For this purpose, each new object detection initializes a track query with the corresponding output embedding of the previous frame…for the decoder we stack the feature maps of the previous and current frame and compute cross-attention with queries over both frames”; Note: the queries are generated/initialized with embeddings. There are N cross-attention queries because the cross-attention is computed with those queries and there are N queries (one per detected object and each object corresponds to a region in the frame)); and inputting the N generated cross-attention queries and the embedded features for each of the plurality of regions into the cross-attention module (Paragraph 5 in 2nd Col. of Page 3, Paragraph 7 in 1st Col. of Page 6 – “The Transformer encoder-decoder performs attention on frame features and decoder queries continuously updating the instance-specific representation of an object‘s identity and location in each track query embedding… for the decoder we stack the feature maps of the previous and current frame and compute cross-attention with queries over both frames”), wherein the embedded features for each of the plurality of regions provide cross-attention keys and values for the cross-attention module (Paragraph 3 and 5 in 2nd Col. of Page 3, Paragraph 1 in 1st Col. of Page 4, Paragraph 7 in 1st Col. of Page 6 – “each object query learns to predict objects with certain spatial proper ties, such as bounding box size and position…The Transformer encoder-decoder performs attention on frame features and decoder queries continuously updating the instance-specific representation of an object‘s identity and location in each track query embedding… queries (white) are decoded to output embeddings for potential track initializations. Each valid object detection {b00, b10, . . . } with a classification score above σobject, i.e., output embedding not predicting the background class (crossed), initializes a new track query embedding… for the decoder we stack the feature maps of the previous and current frame and compute cross-attention with queries over both frames”; Note: the embeddings of the object position/location and actual object identity are the keys and values respectively for cross-attention). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Goel to incorporate the teachings of Meinhardt to generate cross-attention queries for each region for the benefit of finding features for each human in the image so that parameters can be predicted for each one. Additionally, cross-attention helps connect the object and position data from the image, which improves parameter predictions. It also would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Goel to incorporate the teachings of Meinhardt to have the embedded features be the keys and values for the cross-attention module because logically, the features are what the model is trying to find and process.
Regarding claim 4, Goel in view Meinhardt teaches the method of claim 3. Goel further teaches wherein the decoder comprises a transformer model (Paragraph 2 in 1st Col. of Page 13 – “Our transformer decoder is a standard transformer decoder architecture [23] with 6 layers, each containing multi-head self-attention, multi-head cross-attention, and feed-forward blocks, with layer normalization”). Goel does not teach wherein the cross-attention module generates updated queries. However, Meinhardt teaches wherein the cross-attention module generates updated queries (Paragraph 5 in 2nd Col. of Page 3, Paragraph 7 in 1st Col. of Page 6 – “The Transformer encoder-decoder performs attention on frame features and decoder queries continuously updating the instance-specific representation of an object‘s identity and location in each track query embedding…For track queries, the deformable reference points for the current frame are dynamically adjusted to the previous frame bounding box centers. Furthermore, for the decoder we stack the feature maps of the previous and current frame and compute cross-attention with queries over both frames”; Note: encoder-decoder attention is the same as cross-attention, and it updates the queries). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Goel to incorporate the teachings of Meinhardt to update the queries with cross-attention because it “gives queries global access to the visual information of the encoded features. The output embeddings accumulate bounding box and class information over multiple decoding layers” (Meinhardt: Paragraph 5 in 1st Col. of Page 3). In other words, cross-attention helps connect the object and position data from the image, which improves parameter predictions.
Regarding claim 5, Goel in view Meinhardt teaches the method of claim 4. Goel further teaches wherein said processing the embedded features for each of the detected N humans further comprises: regressing the body model and depth parameters from the updated or further updated queries (Paragraph 1 in 2nd Col. of Page 4 – “A tracklet representation is incrementally built up for each individual person over time. The recursion step is to predict for each tracklet, the pose, location and appearance of the person in the next frame, all in 3D, and then find best matches between these top-down predictions and the bottom-up detections of people in that frame after lifting them to 3D. The state represented by each tracklet is then updated by the incoming observation, and the process is iterated. It is possible to track through occlusions because the 3D representation of a tracklet continues to be updated based on past history”; Note: the 3D pose, location, and appearance, which are equivalent to the body model and depth parameters, are regressed/predicted based on past history that gets updated, which is the updated queries in this case); wherein said regressing the body model and depth parameters from the updated queries uses respective multi-layer perceptrons (MLPs) (Paragraph 2 in 1st Col. of Page 13 – “Our transformer decoder is a standard transformer decoder architecture [23] with 6 layers, each containing multi-head self-attention, multi-head cross-attention, and feed-forward blocks, with layer normalization [2]. It has a 2048 hidden dimension, 8 (64-dim) heads for self- and cross-attention, and a hidden dimension of 1024 in the feed-forward MLP block. It operates on a single learnable 2048-dimensional SMPL query token as input and cross-attends to the 16 × 12 image tokens. Finally, a linear readout on the output token from the transformer decoder gives pose θ, shape β, and camera π”; Note: the output token is regressed from an MLP based on a query, and it contains the body model parameters (pose and shape) and depth parameters (camera)). Goel does not teach further updating the updated queries using a self-attention module. However, Meinhardt teaches further updating the updated queries using a self-attention module (Paragraph 5 in 2nd Col. of Page 3 – “The Transformer encoder-decoder performs attention on frame features and decoder queries continuously updating the instance-specific representation of an object‘s identity and location in each track query embedding. Self-attention over the joint set of both query types allows for the detection of new objects while simultaneously avoiding re-detection of already tracked objects”; Note: self-attention further updates the queries). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Goel to incorporate the teachings of Meinhardt to update the queries with self-attention because self-attention “allows for joint reasoning about the objects in a scene… Self-attention over the joint set of both query types allows for the detection of new objects while simultaneously avoiding re-detection of already tracked objects” (Meinhardt: Paragraph 5 in 1st Col. of Page 3, Paragraph 5 in 2nd Col. of Page 3). In other words, it helps with identifying and tracking the humans in the image.
Claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over Goel in view of Meinhardt and Kanazawa et al. (End-to-end Recovery of Human Shape and Pose), hereinafter Kanazawa.
Regarding claim 6, Goel in view Meinhardt teaches the method of claim 3. Goel does not teach wherein said generating a query further comprises one of (i) combining the embedded features for each of the N respective regions with a learned query initialization based on a 2D position, and (ii) combining the embedded features for each of the N respective regions with mean body model parameters. However, Meinhardt teaches combining the embedded features for each of the N respective regions with a learned query initialization based on a 2D position (Paragraph 3 in 2nd Col. of Page 3 – “Track initialization. New objects appearing in the scene are detected by a fixed number of N object output embeddings each initialized with a static and learned object encoding referred to as object queries [7]. Intuitively, each object query learns to predict objects with certain spatial proper ties, such as bounding box size and position”; Note: the object embeddings are equivalent to the embedded features, and the learned object encoding is equivalent to the learned query initialization. Fig. 1 below shows that the positions, represented by bounding boxes, are in 2D). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Goel to incorporate the teachings of Meinhardt to combine the features with a learned query initialization based on a 2D position for the benefit of properly defining the queries so that the model knows what to look for, which improves the accuracy of the results.
Goel modified by Meinhardt still does not teach combining the embedded features for each of the N respective regions with mean body model parameters. However, Kanazawa teaches combining the embedded features for each of the N respective regions with mean body model parameters (Paragraph 4 in 2nd Col. of Page 4, Paragraph 1 and 3 in 1st Col. of Page 5 – “the set of parameters that represent the 3D reconstruction of a human body is expressed as a 85 dimensional vector Θ={θ,β,R,t,s}…the 3D regression module takes the image features φ and the current parameters Θt as an input and outputs the residual ∆Θt. The parameter is updated by adding this residual to the current estimate Θt+1 = Θt + ∆Θt. The initial estimate Θ0 is set as the mean ¯ Θ”; Note: ¯ Θ represents the mean body parameters, which are combined with the embedded image features φ). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Goel to incorporate the teachings of Kanazawa to combine the embedded features with mean body model parameters because the mean body model parameters provide a baseline for what data corresponds to a mesh. Since the goal of Goel is to generate meshes, when initializing the queries, both the mean body model parameters and the image features help to guide the model to produce more accurate results.
PNG
media_image3.png
408
561
media_image3.png
Greyscale
Screenshot of Fig. 1 (taken from Meinhardt)
Claims 7-8, 23, 29, and 34 are rejected under 35 U.S.C. 103 as being unpatentable over Goel in view of Tan et al. (US 12524954 B2), hereinafter Tan.
Regarding claim 7, Goel teaches the method of claim 1. Goel further teaches wherein said receiving a two-dimensional (2D) image of the scene from the image capturing device further comprises receiving intrinsic parameters of the image capturing device, the 2D image including a plurality of regions (Paragraph 4 in 1st Col. of Page 3, Paragraph 1 and 4 in 2nd Col. of Page 3 – “Each camera π = (R,t) consists of a global orientation R ∈ R3×3 and translation t ∈ R3. Given these parameters, points in the SMPL space (e.g., joints X) can be projected to the image as x =π(X)=Π(K(RX+t)), where Π is a perspective projection with camera intrinsics K…The input image is first patchified into input tokens...We use a ViT-H/16, the ‘Huge’ variant with 16×16 input patch size”; Note: camera intrinsics of the camera are received. The input image is captured by the camera, and it includes a plurality of patches/regions). Goel does not teach wherein said encoding the received image to extract embedded features for each of the plurality of regions further comprises encoding the intrinsic parameters of the image capturing device. However, Tan teaches wherein said encoding the received image to extract embedded features for each of the plurality of regions further comprises encoding the intrinsic parameters of the image capturing device (Col. 6 lines 56-58, Col.7 lines 6-8, Col. 9 lines 4-10 – “image encoder 210 encodes the input image to obtain 2D features including a set of 2D tokens corresponding to patches of the input image…the machine learning model combines the 2D features with camera view information in the form of a camera features tensor…The camera features 335 represent a camera condition c∈R20. In some examples, the camera condition c is constructed by flattening out a 4×4 camera extrinsic matrix P and concatenating it with the camera focal length and principal point”; Note: camera features are encoded into a tensor. The camera features include intrinsic parameters like focal length). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Goel to incorporate the teachings of Tan to encode the intrinsic camera parameters because the camera parameters assist the model with understanding the spatial layout of the scene and projecting the mesh onto the image.
Regarding claim 8, Goel in view of Tan teaches the method of claim 7. Goel does not teach before said processing the embedded features for each of the detected N humans, concatenating embedded features for each of the plurality of regions with intrinsic parameters of the image capturing device. However, Tan teaches before said processing the embedded features for each of the detected N humans, concatenating embedded features for each of the plurality of regions with intrinsic parameters of the image capturing device (Col. 6 lines 56-58, Col.7 lines 6-8 and 11-14, Col. 9 lines 4-10 – “image encoder 210 encodes the input image to obtain 2D features including a set of 2D tokens corresponding to patches of the input image…the machine learning model combines the 2D features with camera view information in the form of a camera features tensor… feature decoder 215 decodes the 2D features based on the camera view information to obtain 3D features including a set of 3D tokens corresponding to regions of a 3D representation… Embodiments then combine 2D features 330 with camera features 335. The camera features 335 represent a camera condition c∈R20. In some examples, the camera condition c is constructed by flattening out a 4×4 camera extrinsic matrix P and concatenating it with the camera focal length and principal point”; Note: encoded image features are combined with encoded camera features. This occurs before further processing/decoding of the features). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Goel to incorporate the teachings of Tan to concatenate the embedded features with the intrinsic parameters because the features and intrinsic parameters when combined together, rather than used separately, would help the model create a more accurate projection of the 3D meshes onto the 2D image plane.
Regarding claim 23, Goel teaches a system for recovering a three-dimensional (3D) mesh of N humans in a 3D scene (Paragraph 1-2 in 2nd Col. of Page 1, Paragraph 1 in 1st Col. of Page 2 – “In this paper, we present a fully transformer-based approach for recovering 3D meshes of human bodies from single images, and tracking them over time in video…We take that as a starting point and we develop a new version of HMR, which we call HMR 2.0 to acknowledge its antecedent. We use HMR 2.0 to build a system that can simultaneously reconstruct and track humans from videos… This system can reconstruct Humans in 4D, which gives the name to our method, 4DHumans. 4DHumans can be deployed on any video and can jointly track and reconstruct people in video”; Note: 4DHumans is a system that recovers 3D meshes of humans), comprising:
an image encoder configured to receive a two-dimensional (2D) image of the scene including a plurality of regions from an image capturing device and encoding the received image to extract embedded features for each of the plurality of regions (Paragraph 4 in 1st Col. of Page 3, Paragraph 4 in 2nd Col. of Page 3 – “Each camera π = (R,t) consists of a global orientation R ∈ R3×3 and translation t ∈ R3. Given these parameters, points in the SMPL space (e.g., joints X) can be projected to the image as x =π(X)=Π(K(RX+t)), where Π is a perspective projection with camera intrinsics K…The Vision Transformer, or ViT [15] is a transformer [74] that has been modified to operate on an image. The input image is first patchified into input tokens and passed through the transformer to get output tokens. The output tokens are then passed to the transformer decoder. We use a ViT-H/16, the “Huge” variant with 16×16 input patch size”; Note: The input image is captured by a camera, and it includes a plurality of patches/regions. The vision transformer encodes the input image into tokens, which are equivalent to the embedded features for each of the regions/patches. The vision transformer is equivalent to the encoder);
a detector configured to detect N humans at 2D locations in N respective regions among the plurality of regions in the encoded image (Paragraph 3 in 1st Col. of Page 4, Paragraph 1 in 2nd Col. of Page 4, Paragraph 4 in 2nd Col. of Page 13, Paragraph 1 in 1st Col. of Page 14 – “The basic idea is to detect people in individual frames, and “lift” them to 3D, extracting their 3D pose, location in 3D space (derived from the estimated camera), and 3D appearance (derived from the texture map). A tracklet representation is incrementally built up for each individual person over time… Each output token regresses the 3D pose and 3D location of the person at the specified time-step”; Note: humans are detected in the image, and the output tokens from the vision transformer represent each person in a corresponding region of the image, making it obvious that there are N people and N regions where those people are located. The detection part of the architecture is equivalent to the detector);
a decoder configured to process the embedded features in the N respective regions and the embedded features for each of the plurality of regions to predict body model and depth parameters for each of the N detected humans (Paragraph 2 and 5 in 2nd Col. of Page 3, Paragraph 4 in 2nd Col. of Page 13, Paragraph 1 in 1st Col. of Page 14 – “we model f to predict Θ = [θ,β,π] = f(I) where θ and β are the SMPL pose and shape parameters and π is the camera translation…We use a standard transformer decoder [74] with multi-head self-attention. It processes a single (zero) input token by cross-attending to the output image tokens and ends with a linear readout of Θ…Each output token regresses the 3D pose and 3D location of the person at the specified time-step”; Note: the tokens (embedded features) are processed in a decoder to predict pose and shape parameters (body model parameters) and camera translation (depth parameter) for each person), said decoder comprising a cross-attention module (Paragraph 2 in 1st Col. of Page 13 – “Our transformer decoder is a standard transformer decoder architecture [23] with 6 layers, each containing multi-head self-attention, multi-head cross-attention, and feed-forward blocks, with layer normalization”);
a 3D parametric model configured to receive the predicted body model parameters for each of N detected humans and generate N 3D meshes (Paragraph 3 in 1st Col. of Page 3, Paragraph 2 and 5 in 2nd Col. of Page 3 – “Given input parameters for pose (θ ∈ R24×3×3) and shape (β ∈ R10), it outputs a mesh M ∈ R3×N with N = 6890 vertices…we model f to predict Θ = [θ,β,π] = f(I) where θ and β are the SMPL pose and shape parameters and π is the camera translation…We use a standard transformer decoder [74] with multi-head self-attention. It processes a single (zero) input token by cross-attending to the output image tokens and ends with a linear readout of Θ”; Note: the predicted parameters are input into SMPL model, which is a 3D parametric model, to generate 3D meshes);
and a mesh positioning module configured to place each of the generated N 3D meshes at a respective 3D spatial location in the 3D scene based on the predicted depth parameters and the 2D locations (Fig. 1, Paragraph 3 in 1st Col. of Page 3, Paragraph 4 in 1st Col. of Page 3, Paragraph 1-2 in 2nd Col. of Page 3 – “The body joints X ∈ R3×k are defined as a linear combination of the vertices…Each camera π = (R,t) consists of a global orientation R ∈ R3×3 and translation t ∈ R3. Given these parameters, points in the SMPL space (e.g., joints X) can be projected to the image as x =π(X)=Π(K(RX+t)), where Π is a perspective projection with camera intrinsics K… The goal of the human mesh reconstruction (HMR) task is to learn a predictor f(I) that given a single image I, reconstructs the person in the image by predicting their 3D pose and shape parameters”; Note: the human mesh is projected/placed in the image based on a 3D translation, which represents the depth. Fig. 1 shows how N generated meshes are placed in the 3D scene; see screenshot of Fig. 1 above. The projection part of the architecture is equivalent to the mesh position module).
Goel does not teach a processor and memory coupled to the processor, the memory including instructions executable by the processor. However, Tan teaches a processor and memory coupled to the processor, the memory including instructions executable by the processor implementing (Col. 16 lines 4-5 and 28-31 – “computing device 1000 includes one or more processors 1005…memory is used to store computer-readable, computer-executable software including instructions that, when executed, cause a processor to perform various functions described herein”; Note: it is implied that the processor and memory are coupled since the processor accesses instructions from the memory). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Goel to incorporate the teachings of Tan to have a processor and memory that are coupled for the benefit of being able to execute the process and store it so that it could be used again in the future.
Regarding claim 29, Goel in view of Tan teaches the system of claim 23. Goel further teaches a plurality of feature tokens, each feature token respectively corresponding to each of the plurality of regions and having a feature dimension (Paragraph 4 in 2nd Col. of Page 3, Paragraph 2 in 1st Col. of Page 13 – “The input image is first patchified into input tokens and passed through the transformer to get output tokens. The output tokens are then passed to the transformer decoder. We use a ViT-H/16, the “Huge” variant with 16×16 input patch size…We use a ViT-H/16 (“huge”) pre-trained on the task of 2D key point localization [25]. It has 50 transformer layers, takes a 256×192 sized image as input, and outputs 16×12 image tokens, each of dimension 1280”; Note: the output image tokens are feature tokens corresponding to patches/regions and having dimension 1280). Goel does not teach wherein the image encoder is configured to generate a feature tensor comprising a plurality of feature tokens. However, Tan teaches wherein the imager encoder is further configured to generate a feature tensor comprising a plurality of feature tokens (Col. 6 lines 52-58 – “Image encoder 210 is configured to encode a 2D image to generate 2D image features. In some cases, image encoder 210 processes a tensor including 2D image data directly (height, width, and color information). According to some aspects, image encoder 210 encodes the input image to obtain 2D features including a set of 2D tokens corresponding to patches of the input image”; Note: the generated encoded features include a feature tensor containing feature tokens). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Goel to incorporate the teachings of Tan to have a feature tensor because the tensor would make it easier to process the feature data and keep track of it while it is updated within a transformer.
Regarding claim 34, Goel teaches an information processing apparatus that recovers a three-dimensional (3D) whole-body mesh of N humans in a 3D scene, where N is at least one (Paragraph 1-2 in 2nd Col. of Page 1, Paragraph 1 in 1st Col. of Page 2 – “In this paper, we present a fully transformer-based approach for recovering 3D meshes of human bodies from single images, and tracking them over time in video…We take that as a starting point and we develop a new version of HMR, which we call HMR 2.0 to acknowledge its antecedent. We use HMR 2.0 to build a system that can simultaneously reconstruct and track humans from videos… This system can reconstruct Humans in 4D, which gives the name to our method, 4DHumans. 4DHumans can be deployed on any video and can jointly track and reconstruct people in video”; Note: 4DHumans is a system that recovers 3D meshes of humans. It is implied to be computer-implemented (information processing apparatus) since it would not be able to be executed otherwise. N is at least one, as shown by Fig. 1), by:
receiving a two-dimensional (2D) image of the scene from an image capturing device, the 2D image including a plurality of regions (Paragraph 4 in 1st Col. of Page 3, Paragraph 1 and 4 in 2nd Col. of Page 3 – “Each camera π = (R,t) consists of a global orientation R ∈ R3×3 and translation t ∈ R3. Given these parameters, points in the SMPL space (e.g., joints X) can be projected to the image as x =π(X)=Π(K(RX+t)), where Π is a perspective projection with camera intrinsics K…The input image is first patchified into input tokens...We use a ViT-H/16, the ‘Huge’ variant with 16×16 input patch size”; Note: the input image is the 2D image of the scene. It is captured by a camera, and it includes a plurality of patches/regions);
by one or more processors, encoding the received image to extract embedded features for each of the plurality of regions (Paragraph 4 in 2nd Col. of Page 3 – “The Vision Transformer, or ViT [15] is a transformer [74] that has been modified to operate on an image. The input image is first patchified into input tokens and passed through the transformer to get output tokens. The output tokens are then passed to the transformer decoder. We use a ViT-H/16, the “Huge” variant with 16×16 input patch size”; Note: the vision transformer encodes the input image into tokens, which are equivalent to the embedded features for each of the regions/patches. It is implied that the encoding and the steps below are performed by a processor since they would not be able to be executed otherwise);
by one or more processors, detecting N humans in N respective regions among the plurality of regions (Paragraph 3 in 1st Col. of Page 4, Paragraph 1 in 2nd Col. of Page 4, Paragraph 4 in 2nd Col. of Page 13, Paragraph 1 in 1st Col. of Page 14 – “The basic idea is to detect people in individual frames, and “lift” them to 3D, extracting their 3D pose, location in 3D space (derived from the estimated camera), and 3D appearance (derived from the texture map). A tracklet representation is incrementally built up for each individual person over time… Each output token regresses the 3D pose and 3D location of the person at the specified time-step”; Note: humans are detected in the image, and the output tokens from the vision transformer represent each person in a corresponding region of the image, making it obvious that there are N people and N regions where those people are located);
by one or more processors, processing the embedded features in the N respective regions and the embedded features for each of the plurality of regions to predict body model and depth parameters for each of the N detected humans (Paragraph 2 and 5 in 2nd Col. of Page 3, Paragraph 4 in 2nd Col. of Page 13, Paragraph 1 in 1st Col. of Page 14 – “we model f to predict Θ = [θ,β,π] = f(I) where θ and β are the SMPL pose and shape parameters and π is the camera translation…We use a standard transformer decoder [74] with multi-head self-attention. It processes a single (zero) input token by cross-attending to the output image tokens and ends with a linear readout of Θ…Each output token regresses the 3D pose and 3D location of the person at the specified time-step”; Note: the tokens (embedded features) are processed in a decoder to predict pose and shape parameters (body model parameters) and camera translation (depth parameter) for each person), wherein said processing uses a decoder comprising a cross-attention module (Paragraph 2 in 1st Col. of Page 13 – “Our transformer decoder is a standard transformer decoder architecture [23] with 6 layers, each containing multi-head self-attention, multi-head cross-attention, and feed-forward blocks, with layer normalization”);
providing the predicted body model parameters for each of the N detected humans to a 3D parametric model for generating N whole-body 3D meshes (Paragraph 3 in 1st Col. of Page 3, Paragraph 2 and 5 in 2nd Col. of Page 3 – “Given input parameters for pose (θ ∈ R24×3×3) and shape (β ∈ R10), it outputs a mesh M ∈ R3×N with N = 6890 vertices…we model f to predict Θ = [θ,β,π] = f(I) where θ and β are the SMPL pose and shape parameters and π is the camera translation…We use a standard transformer decoder [74] with multi-head self-attention. It processes a single (zero) input token by cross-attending to the output image tokens and ends with a linear readout of Θ”; Note: the predicted parameters are input into SMPL model, which is a 3D parametric model, to generate 3D meshes. The meshes are whole-body as shown in Fig. 1);
and placing each of the N generated meshes at a respective 3D spatial location in the 3D scene based on the predicted depth parameters (Fig. 1, Paragraph 4 in 1st Col. of Page 3, Paragraph 1-2 in 2nd Col. of Page 3 – “Each camera π = (R,t) consists of a global orientation R ∈ R3×3 and translation t ∈ R3. Given these parameters, points in the SMPL space (e.g., joints X) can be projected to the image as x =π(X)=Π(K(RX+t)), where Π is a perspective projection with camera intrinsics K… The goal of the human mesh reconstruction (HMR) task is to learn a predictor f(I) that given a single image I, reconstructs the person in the image by predicting their 3D pose and shape parameters”; Note: the human mesh is projected/placed in the image based on a 3D translation, which represents the depth. Fig. 1 shows how N generated meshes are placed in the 3D scene; see screenshot of Fig. 1 above).
Goel does not teach a non-transitory computer-readable medium storing a program including instructions that, when executed by a processor, causes an information processing apparatus to perform operations. However, Tan teaches a non-transitory computer-readable medium storing a program including instructions that, when executed by a processor, causes an information processing apparatus to perform operations (Col. 16 lines 4-5 and 28-31, Col. 17 lines 47-51 – “computing device 1000 includes one or more processors 1005…memory is used to store computer-readable, computer-executable software including instructions that, when executed, cause a processor to perform various functions described herein…Computer-readable media includes both non-transitory computer storage media and communication media including any medium that facilitates transfer of code or data. A non-transitory storage medium may be any available medium that can be accessed by a computer”: Note: the computing device is equivalent to the information processing apparatus). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Goel to incorporate the teachings of Tan to have a non-transitory computer-readable medium storing a program executable by a processor for the benefit of having a persistent storage so that the process can be used and executed again in the future.
Claims 9 and 26 are rejected under 35 U.S.C. 103 as being unpatentable over Goel in view of Tan and Martins et al. (Ray-Patch: An Efficient Decoder for Light Field Transformers), hereinafter Martins.
Regarding claim 9, Goel in view of Tan teaches method of claim 8. Goel further teaches wherein each region comprises a 2D patch, and wherein the plurality of regions defines a grid of patches forming the 2D image (Paragraph 4 in 2nd Col. of Page 3 – “The Vision Transformer, or ViT [15] is a transformer [74] that has been modified to operate on an image. The input image is first patchified into input tokens and passed through the transformer to get output tokens. The output tokens are then passed to the transformer decoder. We use a ViT-H/16, the “Huge” variant with 16×16 input patch size”; Note: the image is divided into patches. By “patchifying” the image, it is implied to be defined into a grid of patches). Goel does not teach wherein the intrinsic parameters comprises embedded values of ray directions from a center of each of the plurality of regions, wherein the embedded values are generated using Fourier encoding. However, Martins teaches wherein the intrinsic parameters comprises embedded values of ray directions from a center of each of the plurality of regions (Paragraph 3-4 in 2nd Col. of Page 4 – “the view is split into hw/k2 square patches of size [k,k], being the split image now defined as {Itp ∈ Rh/k × w/k × 3}. Each patch p is parametrized by the location of the camera ot, and the ray rtp that passes both by the camera position and the center of the patch. Given the camera intrinsic Kt and extrinsic parameters WTCt = [Rt|ot] ∈ SE(3), the ray rtp is computed as the unprojection of the center of patch p in the 2D camera plane… Using Fourier positional encoding [29], the parametrization of each patch is mapped to a higher frequency, to generate a set of queries for the decoder
PNG
media_image4.png
43
264
media_image4.png
Greyscale
”; Note: intrinsic parameters inherently include focal length and principal points, which form a ray direction rtp that becomes encoded. See screenshot of Fig. 1 below, which shows the ray directions), wherein the embedded values are generated using Fourier encoding (Paragraph 3-4 in 2nd Col. of Page 4 – “Each patch p is parametrized by the location of the camera ot, and the ray rtp that passes both by the cam era position and the center of the patch. Given the camera intrinsic Kt and extrinsic parameters WTCt = [Rt|ot] ∈ SE(3), the ray rtp is computed as the unprojection of the center of patch p in the 2D camera plane… Using Fourier positional encoding [29], the parametrization of each patch is mapped to a higher frequency, to generate a set of queries for the decoder
PNG
media_image4.png
43
264
media_image4.png
Greyscale
”; Note: Fourier encoding is used on the ray, which is computed by intrinsic parameters). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Goel to incorporate the teachings of Martins to embed ray directions from a center of each of the plurality of regions using Fourier encoding because using only the rays from a center of the patches reduces “the querying cost by a factor of k2 without losing accuracy” (Martins: Fig. 1 Caption on Page 1), and Fourier encoding is beneficial for encapsulating the ray distances to allow for efficient learning. Overall, it would help the model with spatial awareness with reduced computational overhead.
PNG
media_image5.png
202
278
media_image5.png
Greyscale
Screenshot of Fig. 1 (taken from Martins)
Regarding claim 26, Goel in view of Tan teaches the system of claim 23. Goel does not teach a camera intrinsics encoder configured to embed intrinsic parameters of the image capturing device; wherein the embedded intrinsic parameters are concatenated with the embedded features for each of the plurality of regions upstream of said decoder; wherein the intrinsic parameters comprise embedded values of ray directions from a center of each of the plurality of regions; wherein said camera intrinsics encoder comprises a Fourier encoder. However, Tan teaches a camera intrinsics encoder configured to embed intrinsic parameters of the image capturing device (Col. 6 lines 56-58, Col.7 lines 6-8, Col. 9 lines 4-10 – “image encoder 210 encodes the input image to obtain 2D features including a set of 2D tokens corresponding to patches of the input image…the machine learning model combines the 2D features with camera view information in the form of a camera features tensor…The camera features 335 represent a camera condition c∈R20. In some examples, the camera condition c is constructed by flattening out a 4×4 camera extrinsic matrix P and concatenating it with the camera focal length and principal point”; Note: camera parameters are encoded into a tensor. The camera features include intrinsic parameters like focal length. The model that encodes the camera parameters is equivalent to the camera intrinsics encoder); wherein the embedded intrinsic parameters are concatenated with the embedded features for each of the plurality of regions upstream of said decoder (Col. 6 lines 56-58, Col.7 lines 6-8 and 11-14, Col. 9 lines 4-10 – “image encoder 210 encodes the input image to obtain 2D features including a set of 2D tokens corresponding to patches of the input image…the machine learning model combines the 2D features with camera view information in the form of a camera features tensor… feature decoder 215 decodes the 2D features based on the camera view information to obtain 3D features including a set of 3D tokens corresponding to regions of a 3D representation… Embodiments then combine 2D features 330 with camera features 335. The camera features 335 represent a camera condition c∈R20. In some examples, the camera condition c is constructed by flattening out a 4×4 camera extrinsic matrix P and concatenating it with the camera focal length and principal point”; Note: encoded image features are combined with encoded camera features. This occurs before (upstream) further processing/decoding of the features). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Goel to incorporate the teachings of Tan to encode the intrinsic camera parameters because the camera parameters assist the model with understanding the spatial layout of the scene and projecting the mesh onto the image. It also would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Goel to incorporate the teachings of Tan to concatenate the embedded features with the intrinsic parameters because the features and intrinsic parameters when combined together, rather than used separately, would help the model create a more accurate projection of the 3D meshes onto the 2D image plane. Goel modified by Tan still does not teach wherein the intrinsic parameters comprise embedded values of ray directions from a center of each of the plurality of regions; wherein said camera intrinsics encoder comprises a Fourier encoder. However, Martins teaches wherein the intrinsic parameters comprises embedded values of ray directions from a center of each of the plurality of regions (Paragraph 3-4 in 2nd Col. of Page 4 – “the view is split into hw/k2 square patches of size [k,k], being the split image now defined as {Itp ∈ Rh/k × w/k × 3}. Each patch p is parametrized by the location of the camera ot, and the ray rtp that passes both by the camera position and the center of the patch. Given the camera intrinsic Kt and extrinsic parameters WTCt = [Rt|ot] ∈ SE(3), the ray rtp is computed as the unprojection of the center of patch p in the 2D camera plane… Using Fourier positional encoding [29], the parametrization of each patch is mapped to a higher frequency, to generate a set of queries for the decoder
PNG
media_image4.png
43
264
media_image4.png
Greyscale
”; Note: intrinsic parameters inherently include focal length and principal points, which form a ray direction rtp that becomes encoded. See screenshot of Fig. 1 above, which shows the ray directions), wherein said camera intrinsics encoder comprises a Fourier encoder (Paragraph 3-4 in 2nd Col. of Page 4 – “Each patch p is parametrized by the location of the camera ot, and the ray rtp that passes both by the camera position and the center of the patch. Given the camera intrinsic Kt and extrinsic parameters WTCt = [Rt|ot] ∈ SE(3), the ray rtp is computed as the unprojection of the center of patch p in the 2D camera plane… Using Fourier positional encoding [29], the parametrization of each patch is mapped to a higher frequency, to generate a set of queries for the decoder
PNG
media_image4.png
43
264
media_image4.png
Greyscale
”; Note: Fourier encoding is used on the ray, which is computed by intrinsic parameters). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Goel to incorporate the teachings of Martins to embed ray directions from a center of each of the plurality of regions using Fourier encoding because using only the rays from a center of the patches reduces “the querying cost by a factor of k2 without losing accuracy” (Martins: Fig. 1 Caption on Page 1), and Fourier encoding is beneficial for encapsulating the ray distances to allow for efficient learning. Overall, it would help the model with spatial awareness with reduced computational overhead.
Claims 10-14 are rejected under 35 U.S.C. 103 as being unpatentable over Goel in view of Sun et al. (Monocular, One-stage, Regression of Multiple 3D People), hereinafter Sun.
Regarding claim 10, Goel teaches the method of claim 1. Goel does not teach wherein said detecting N humans comprises detecting a primary keypoint in each of the N respective regions; wherein each generated 3D mesh is centered around the primary keypoint; and wherein each of the primary keypoints comprises a human head, torso, midsection, spine, or pelvis. However, Sun teaches detecting a primary keypoint in each of the N respective regions (Fig. 2, Paragraph 3 in 2nd Col. of Page 4 – “For stable parameter sampling, we need an explicit body center. Therefore, we calculate each body center from the ground truth 2D pose”; Note: the body center is equivalent to the primary keypoint. Fig. 2, shown below, demonstrates how a body center is calculated for each region with a human in it); wherein each generated 3D mesh is centered around the primary keypoint (Paragraph 3 in 2nd Col. of Page 3, Paragraph 1 in 1st Col. of Page 4, Paragraph 3 in 2nd Col. of Page 4 – “In the Body Center heatmap, we predict the probability of each position being a human body center… we sample the 3D body mesh parameter results from the Mesh Parameter map at the 2D body center locations parsed from the Body Center heatmap. Finally, we put the sampled parameters into the SMPL model to generate the 3D body meshes…For stable parameter sampling, we need an explicit body center. Therefore, we calculate each body center from the ground truth 2D pose”; Note: the body center is the primary keypoint so it is implied that the mesh of the body is centered at the keypoint); and wherein each of the primary keypoints comprises a human head, torso, midsection, spine, or pelvis (Paragraph 3 in 2nd Col. of Page 4 – “we calculate each body center from the ground truth 2D pose. Considering that any body joint may be occluded in general cases, we define the body center as the center of visible torso joints (neck, left/right shoulders, pelvis, and left/right hips)”). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Goel to incorporate the teachings of Sun to detect a primary keypoint that is the center of each mesh because it creates a uniform and consistent way to identify a person and their location in the image and place the mesh in the correct location. It also would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Goel to incorporate the teachings of Sun to have the primary keypoint be a torso, midsection, or pelvis because those areas represent the center of the human body, and thus would represent the center of the mesh.
PNG
media_image6.png
285
656
media_image6.png
Greyscale
Screenshot of Fig. 2 (taken from Sun)
Regarding claim 11, Goel in view of Sun teaches the method of claim 10. Goel does not teach wherein said detecting a primary keypoint comprises: generating, for each of the plurality of regions, a probability that the primary keypoint is present within the region; and determining that the primary keypoint is present by comparing the generated probability to a threshold; wherein the generated probabilities for each of the plurality of regions defines a 2D heatmap. However, Sun teaches generating, for each of the plurality of regions, a probability that the primary keypoint is present within the region (Fig. 2 Caption on Page 3 – “the Body Center heatmap predicts the probability of each position being a body center”; Note: the body center is the primary keypoint); and determining that the primary keypoint is present by comparing the generated probability to a threshold (Paragraph 4-5 in 1st Col. of Page 5, Paragraph 2 in 1st Col. of Page 6, Paragraph 3 in 2nd Col. of Page 12 – “To parse the 3D body meshes from the estimated maps, we need to first parse the 2D body center coordinates c ∈ RK×2 from Cm, where K is the number of the detected people, and then use them to sample Pm for the SMPL parameters… Cm is a probability map whose local maxima are regarded as the body centers …Let c be the 2D coordinates of a local maximum with confidence score larger than a threshold tc. We rank the confidence score at each c and take the top N as the final centers…The threshold tc of the Body Center heatmap is 0.2…The confidence threshold tc is used to filter out the detected people with the confidence value lower than tc”; Note: only the body centers above a threshold are considered present); wherein the generated probabilities for each of the plurality of regions defines a 2D heatmap (Paragraph 3 in 1st Col. of Page 4, Paragraph 5 in 1st Col. of Page 5 – “Cm ∈ R1× H× W is a heatmap representing the 2D human body center in the image. Each body center is represented as a Gaussian distribution in the Body Center heatmap…Cm is a probability map whose local maxima are regarded as the body centers”). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Goel to incorporate the teachings of Sun to determine the primary keypoint based on probabilities of a heatmap because heatmaps are effective for detecting multiple objects at one time, and the probabilities would help with identifying the most accurate point for the primary keypoint.
Regarding claim 12, Goel in view of Sun teaches the method of claim 11. Goel does not teach for each of the N primary keypoints, determining a 2D location of the primary keypoint within the respective region. However, Sun teaches for each of the N primary keypoints, determining a 2D location of the primary keypoint within the respective region (Paragraph 4-5 in 1st Col. of Page 5 – “To parse the 3D body meshes from the estimated maps, we need to first parse the 2D body center coordinates c ∈ RK×2 from Cm, where K is the number of the detected people, and then use them to sample Pm for the SMPL parameters… Cm is a probability map whose local maxima are regarded as the body centers …Let c be the 2D coordinates of a local maximum with confidence score larger than a threshold tc. We rank the confidence score at each c and take the top N as the final centers”; Note: the 2D body center coordinates are the 2D location of the primary keypoint). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Goel to incorporate the teachings of Sun to determine the 2D location of the primary keypoints because doing so helps with efficiently determining the location of each human and later placing the mesh for each human.
Regarding claim 13, Goel in view of Sun teaches the method of claim 12. Goel does not teach wherein, for each of the N primary keypoints, said determining a location of the primary keypoint within the respective region comprises regressing an offset from a center of the respective region. However, Sun teaches wherein, for each of the N primary keypoints, said determining a location of the primary keypoint within the respective region comprises regressing an offset from a center of the respective region (Paragraph 4-5 in 1st Col. of Page 5, Paragraph 1 in 2nd Col. of Page 5 – “To parse the 3D body meshes from the estimated maps, we need to first parse the 2D body center coordinates c ∈ RK×2 from Cm, where K is the number of the detected people, and then use them to sample Pm for the SMPL parameters… Cm is a probability map whose local maxima are regarded as the body centers …Let c be the 2D coordinates of a local maximum with confidence score larger than a threshold tc. We rank the confidence score at each c and take the top N as the final centers. During inference, we directly sample the parameters from Pm at c. During training, the estimated c are matched with the nearest ground truth body center according to the L2 distance”; Note: L2 regression loss is used to compute a distance/offset from a ground truth body center). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Goel to incorporate the teachings of Sun to regress an offset from a center of the respective region for the benefit of improving the model to learn to identify the correct location of the humans.
Regarding claim 14, Goel in view of Sun teaches the method of claim 13. Goel does not teach wherein said generating a 3D mesh at a 3D spatial location for each of the N detected humans in the 3D scene comprises: for each generated 3D mesh generated by the 3D parametric model, determining the 3D spatial location based on the determined 2D location of the primary keypoint and the predicted depth; and placing the generated 3D mesh at the determined 3D spatial location in the scene; wherein the 3D spatial location is in camera space. However, Sun teaches for each generated 3D mesh generated by the 3D parametric model, determining the 3D spatial location based on the determined 2D location of the primary keypoint and the predicted depth (Paragraph 4-5 in 1st Col. of Page 4, Paragraph 4 in 1st Col. of Page 5, Paragraph 1 in 2nd Col. of Page 5 – “Pm ∈ R145× H× W consists of two parts, the Camera map and SMPL map… we employ a weak-perspective camera model to project K 3D body joints J = (xk,yk,zk),k = 1···K of the estimated 3D mesh back to the 2D joints J = (xk, yk) on the image plane…Camera map: Am ∈ R3× H× W contains the 3-dim camera parameters (s,tx,ty) that describe the 2D scale s and translation t = (tx,ty) of the person in the image. The scale s reflects the body size and the depth…To parse the 3D body meshes from the estimated maps, we need to first parse the 2D body center coordinates c ∈ RK×2 from Cm, where K is the number of the detected people, and then use them to sample Pm for the SMPL parameters… we approximate the depth order between multiple people by using the center confidence from Cm and the 2D body scale s of the camera parameters from Am”; Note: the 2D coordinate of the primary keypoint (body center) and depth from the body scale s are used to determine SMPL parameters, which includes the 3D location of the mesh); and placing the generated 3D mesh at the determined 3D spatial location in the scene (Paragraph 4 in 1st Col. of Page 4, Paragraph 1 in 1st Col. of Page 13 – “we employ a weak-perspective camera model to project K 3D body joints J = (xk,yk,zk),k = 1···K of the estimated 3D mesh back to the 2D joints J = (xk, yk) on the image plane…For better visualization, we attempt to approximate the depth ordering between the estimated multi-person body meshes to render the meshes onto the original 2D images. In detail, we construct a depth ordering map to determine the visible meshes in front, using 2D body scale and center confidence as the cue…In this way, we bring to the front, the body mesh that occupies a larger area on the image plane”; Note: the meshes are placed in the scene based on the determined 3D position); wherein the 3D spatial location is in camera space (Paragraph 4-5 in 1st Col. of Page 4 – “Pm ∈ R145× H× W consists of two parts, the Camera map and SMPL map…we employ a weak-perspective camera model to project K 3D body joints J = (xk,yk,zk),k = 1···K of the estimated 3D mesh back to the 2D joints J = (xk, yk) on the image plane… Am ∈ R3× H× W contains the 3-dim camera parameters (s,tx,ty) that describe the 2D scale s and translation t = (tx,ty) of the person in the image. The scale s reflects the body size and the depth to some extent. tx and ty, ranging in (−1,1), reflect the normalized translation of the human body relative to the image center on the x and y axis, respectively. The 2D projection J of 3D body joints J can be derived as xk = sxk + tx, yk = syk + ty”; Note: it is implied that the 3D location is in camera space because the 3D location includes a translation of the human body to be in front of the camera and thus in the camera coordinate system). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Goel to incorporate the teachings of Sun to determine the 3D spatial location in camera space and place the mesh at the location for the benefit of accurately projecting the mesh onto the image so that it can be in alignment with image.
Claims 15 and 27-28 are rejected under 35 U.S.C. 103 as being unpatentable over Goel in view of Tan and Sun.
Regarding claim 15, Goel in view of Sun teaches the method of claim 14. Goel further teaches a plurality of feature tokens, each feature token respectively corresponding to each of the plurality of regions and having a feature dimension (Paragraph 4 in 2nd Col. of Page 3, Paragraph 2 in 1st Col. of Page 13 – “The input image is first patchified into input tokens and passed through the transformer to get output tokens. The output tokens are then passed to the transformer decoder. We use a ViT-H/16, the “Huge” variant with 16×16 input patch size…We use a ViT-H/16 (“huge”) pre-trained on the task of 2D key point localization [25]. It has 50 transformer layers, takes a 256×192 sized image as input, and outputs 16×12 image tokens, each of dimension 1280”; Note: the output image tokens are feature tokens corresponding to patches/regions and having dimension 1280). Goel does not teach wherein the extracted embedded features comprise a feature tensor comprising a plurality of feature tokens. However, Tan teaches wherein the extracted embedded features comprise a feature tensor comprising a plurality of feature tokens (Col. 6 lines 52-58 – “Image encoder 210 is configured to encode a 2D image to generate 2D image features. In some cases, image encoder 210 processes a tensor including 2D image data directly (height, width, and color information). According to some aspects, image encoder 210 encodes the input image to obtain 2D features including a set of 2D tokens corresponding to patches of the input image”; Note: the generated encoded features include a feature tensor containing feature tokens). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Goel to incorporate the teachings of Tan to have a feature tensor because the tensor would make it easier to process the feature data and keep track of it while it is updated within a transformer.
Regarding claim 27, Goel in view of Tan teaches the system of claim 23. Goel further teaches wherein each region comprises a 2D patch, and wherein the plurality of regions defines a grid of patches forming the 2D image (Paragraph 4 in 2nd Col. of Page 3 – “The Vision Transformer, or ViT [15] is a transformer [74] that has been modified to operate on an image. The input image is first patchified into input tokens and passed through the transformer to get output tokens. The output tokens are then passed to the transformer decoder. We use a ViT-H/16, the “Huge” variant with 16×16 input patch size”; Note: the image is divided into patches. By “patchifying” the image, it is implied to be defined into a grid of patches). Goel does not teach wherein said detector is configured to detect a primary keypoint in each of the N respective regions; wherein each generated 3D mesh is centered around the primary keypoint; wherein each of the primary keypoints comprises a human head, torso, midsection, spine, or pelvis; and wherein said detector is configured to generate, for each of the plurality of regions, a probability that the primary keypoint is present within the region; and determine that the primary keypoint is present by comparing the generated probability to a threshold. However, Sun teaches wherein said detector is configured to detect a primary keypoint in each of the N respective regions (Fig. 2, Paragraph 3 in 2nd Col. of Page 4 – “For stable parameter sampling, we need an explicit body center. Therefore, we calculate each body center from the ground truth 2D pose”; Note: the body center is equivalent to the primary keypoint. Fig. 2, shown above, demonstrates how a body center is calculated for each region with a human in it. The part of the model that determines the centers is equivalent to the detector); wherein each generated 3D mesh is centered around the primary keypoint (Paragraph 3 in 2nd Col. of Page 3, Paragraph 1 in 1st Col. of Page 4, Paragraph 3 in 2nd Col. of Page 4 – “In the Body Center heatmap, we predict the probability of each position being a human body center… we sample the 3D body mesh parameter results from the Mesh Parameter map at the 2D body center locations parsed from the Body Center heatmap. Finally, we put the sampled parameters into the SMPL model to generate the 3D body meshes…For stable parameter sampling, we need an explicit body center. Therefore, we calculate each body center from the ground truth 2D pose”; Note: the body center is the primary keypoint so it is implied that the mesh of the body is centered at the keypoint); and wherein each of the primary keypoints comprises a human head, torso, midsection, spine, or pelvis (Paragraph 3 in 2nd Col. of Page 4 – “we calculate each body center from the ground truth 2D pose. Considering that any body joint may be occluded in general cases, we define the body center as the center of visible torso joints (neck, left/right shoulders, pelvis, and left/right hips)”); and wherein said detector is configured to generate, for each of the plurality of regions, a probability that the primary keypoint is present within the region (Fig. 2 Caption on Page 3 – “the Body Center heatmap predicts the probability of each position being a body center”; Note: the body center is the primary keypoint. The part of the model that predicts the probabilities and determines the centers is equivalent to the detector); and determine that the primary keypoint is present by comparing the generated probability to a threshold (Paragraph 4-5 in 1st Col. of Page 5, Paragraph 2 in 1st Col. of Page 6, Paragraph 3 in 2nd Col. of Page 12 – “To parse the 3D body meshes from the estimated maps, we need to first parse the 2D body center coordinates c ∈ RK×2 from Cm, where K is the number of the detected people, and then use them to sample Pm for the SMPL parameters… Cm is a probability map whose local maxima are regarded as the body centers …Let c be the 2D coordinates of a local maximum with confidence score larger than a threshold tc. We rank the confidence score at each c and take the top N as the final centers…The threshold tc of the Body Center heatmap is 0.2…The confidence threshold tc is used to filter out the detected people with the confidence value lower than tc”; Note: only the body centers above a threshold are considered present). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Goel to incorporate the teachings of Sun to detect a primary keypoint that is the center of each mesh because it creates a uniform and consistent way to identify a person and their location in the image and place the mesh in the correct location. It also would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Goel to incorporate the teachings of Sun to have the primary keypoint be a torso, midsection, or pelvis because those areas represent the center of the human body, and thus would represent the center of the mesh. Finally, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Goel to incorporate the teachings of Sun to determine the primary keypoint based on probabilities and comparison to a threshold because the probabilities would help with identifying the most accurate point for the primary keypoint.
Regarding claim 28, Goel in view of Tan and Sun teaches the system of claim 27. Goel does not teach wherein said detector is further configured to determine a location of each of the N primary keypoints within their respective region by regressing an offset from a center of the respective region; wherein said mesh positioning module is configured to: for each generated 3D mesh, determine the 3D spatial location based on the determined location of the primary keypoint and the predicted depth; and place the generated 3D mesh at the determined 3D spatial location in the scene; and wherein the 3D spatial location is in camera space. However, Sun teaches wherein said detector is further configured to determine a location of each of the N primary keypoints within their respective region by regressing an offset from a center of the respective region (Paragraph 4-5 in 1st Col. of Page 5, Paragraph 1 in 2nd Col. of Page 5 – “To parse the 3D body meshes from the estimated maps, we need to first parse the 2D body center coordinates c ∈ RK×2 from Cm, where K is the number of the detected people, and then use them to sample Pm for the SMPL parameters… Cm is a probability map whose local maxima are regarded as the body centers …Let c be the 2D coordinates of a local maximum with confidence score larger than a threshold tc. We rank the confidence score at each c and take the top N as the final centers. During inference, we directly sample the parameters from Pm at c. During training, the estimated c are matched with the nearest ground truth body center according to the L2 distance”; Note: L2 regression loss is used to compute a distance/offset from a ground truth body center. The part of the model that determines the centers is equivalent to the detector); wherein said mesh positioning module is configured to: for each generated 3D mesh, determine the 3D spatial location based on the determined location of the primary keypoint and the predicted depth (Paragraph 4-5 in 1st Col. of Page 4, Paragraph 4 in 1st Col. of Page 5, Paragraph 1 in 2nd Col. of Page 5 – “Pm ∈ R145× H× W consists of two parts, the Camera map and SMPL map… we employ a weak-perspective camera model to project K 3D body joints J = (xk,yk,zk),k = 1···K of the estimated 3D mesh back to the 2D joints J = (xk, yk) on the image plane…Camera map: Am ∈ R3× H× W contains the 3-dim camera parameters (s,tx,ty) that describe the 2D scale s and translation t = (tx,ty) of the person in the image. The scale s reflects the body size and the depth…To parse the 3D body meshes from the estimated maps, we need to first parse the 2D body center coordinates c ∈ RK×2 from Cm, where K is the number of the detected people, and then use them to sample Pm for the SMPL parameters… we approximate the depth order between multiple people by using the center confidence from Cm and the 2D body scale s of the camera parameters from Am”; Note: the 2D coordinate of the primary keypoint (body center) and depth from the body scale s are used to determine SMPL parameters, which includes the 3D location of the mesh. The part of the model that places/projects the meshes is equivalent to the mesh positioning module); and place the generated 3D mesh at the determined 3D spatial location in the scene (Paragraph 4 in 1st Col. of Page 4, Paragraph 1 in 1st Col. of Page 13 – “we employ a weak-perspective camera model to project K 3D body joints J = (xk,yk,zk),k = 1···K of the estimated 3D mesh back to the 2D joints J = (xk, yk) on the image plane…For better visualization, we attempt to approximate the depth ordering between the estimated multi-person body meshes to render the meshes onto the original 2D images. In detail, we construct a depth ordering map to determine the visible meshes in front, using 2D body scale and center confidence as the cue…In this way, we bring to the front, the body mesh that occupies a larger area on the image plane”; Note: the meshes are placed in the scene based on the determined 3D position); wherein the 3D spatial location is in camera space (Paragraph 4-5 in 1st Col. of Page 4 – “Pm ∈ R145× H× W consists of two parts, the Camera map and SMPL map…we employ a weak-perspective camera model to project K 3D body joints J = (xk,yk,zk),k = 1···K of the estimated 3D mesh back to the 2D joints J = (xk, yk) on the image plane… Am ∈ R3× H× W contains the 3-dim camera parameters (s,tx,ty) that describe the 2D scale s and translation t = (tx,ty) of the person in the image. The scale s reflects the body size and the depth to some extent. tx and ty, ranging in (−1,1), reflect the normalized translation of the human body relative to the image center on the x and y axis, respectively. The 2D projection J of 3D body joints J can be derived as xk = sxk + tx, yk = syk + ty”; Note: it is implied that the 3D location is in camera space because the 3D location includes a translation of the human body to be in front of the camera and thus in the camera coordinate system). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Goel to incorporate the teachings of Sun to regress an offset from a center of the respective region for the benefit of improving the model to learn to identify the correct location of the humans. It also would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Goel to incorporate the teachings of Sun to determine the 3D spatial location in camera space and place the mesh at the location for the benefit of accurately projecting the mesh onto the image so that it can be in alignment with image.
Claims 16-17 are rejected under 35 U.S.C. 103 as being unpatentable over Goel in view of Pavlakos et al. (Expressive Body Capture: 3D Hands, Face, and Body from a Single Image), hereinafter Pavlakos.
Regarding claim 16, Goel teaches the method of claim 1. Goel does not teach wherein each generated 3D mesh is an expressive human pose, and each generated 3D mesh comprises human faces, hands, and feet. However, Pavlakos teaches wherein each generated 3D mesh is an expressive human pose, and each generated 3D mesh comprises human faces, hands, and feet (Fig. 2 – The figure shows each generated 3D mesh with expression pose and faces, hands, and feet; see screenshot below). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Goel to incorporate the teachings of Pavlakos to have an expressive pose with faces, hands, and feet for the 3D mesh because it would provide a more detailed and realistic visualization that is more appealing to the eye.
PNG
media_image7.png
555
1132
media_image7.png
Greyscale
Screenshot of Fig. 2 (taken from Pavlakos)
Regarding claim 17, Goel teaches the method of claim 1. Goel further teaches wherein said encoding the received image uses a Vision Transformer (Paragraph 4 in 2nd Col. of Page 3 – “The Vision Transformer, or ViT [15] is a transformer [74] that has been modified to operate on an image. The input image is first patchified into input tokens and passed through the transformer to get output tokens. The output tokens are then passed to the transformer decoder”; Note: the vision transformer encodes the input image). Goel does not teach wherein the 3D parametric model comprises a SMPL-X model. However, Pavlakos teaches wherein the 3D parametric model comprises a SMPL-X model (Paragraph 5 in 2nd Col. of Page 6, Paragraph 2 in 2nd Col. of Page 8 – “we fit SMPL-X to the EHF images to evaluate both qualitatively and quantitatively… In this work we present SMPL-X, a new model that jointly captures the body together with face and hands. We additionally present SMPLify-X, an approach to fit SMPL-X toa single RGB image and 2D OpenPose joint detections”). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Goel to incorporate the teachings of Pavlakos to use SMPL-X for the 3D parametric model because it provides a more realistic mesh representation than SMPL, which is what Goel uses. “Compared to SMPLify [10], SMPLify-X uses a better pose prior (Section 3.3), a more detailed collision penalty (Section 3.4), gender detection (Section 3.5), and a faster PyTorch implementation… A strong holistic model like SMPL-X results in natural and expressive reconstruction of bodies, hands and faces” (Pavlakos: Paragraph 2 in 1st Col. of Page 4, Fig. 4 Caption on Page 8).
Claims 18-19 are rejected under 35 U.S.C. 103 as being unpatentable over Goel in view of Sun et al. (Putting People in their Place: Monocular Regression of 3D People in Depth) and Patel et al. (AGORA: Avatars in Geography Optimized for Regression Analysis), hereinafter Sun 2 and Patel.
Regarding claim 18, Goel teaches the method of claim 1. Goel further teaches wherein the computer-implemented method is implemented by a neural model (Paragraph 1 in 2nd Col. of Page 1 – “In this paper, we present a fully transformer-based approach for recovering 3D meshes of human bodies from single images, and tracking them over time in video”; Note: the transformer-based approach is a neural model since transformers are a type of neural network). Goel does not teach wherein the neural model is trained using a dataset comprising a synthetic dataset; wherein the synthetic dataset comprises a generated plurality of images, each including a single human having visible hands positioned in a hand pose; wherein among the generated plurality of images, the hand poses are diverse. However, Sun 2 teaches wherein the neural model is trained using a dataset comprising a synthetic dataset (Paragraph 2 in 2nd Col. of Page 6, Paragraph 6 in 2nd Col. of Page 12 – “For basic training, we use two 3D pose datasets (Human3.6M [10] and MuCo-3DHP [24]) and four 2D pose datasets (COCO [21], MPII [3], LSP [12], and CrowdPose [20]). We also use the pseudo SMPL annotations from [14] and WST on RH. Most samples in RH are collected from 2D pose datasets [20,21,46]. For a fair comparison, we only use the samples that are also used for training in compared methods [11, 18, 19, 25, 34, 48]. To compare with [18,28], we further fine-tune our model and ROMP on AGORA… AGORA[27] is a synthetic dataset” ; Note: the model is trained on AGORA, which is a synthetic dataset); wherein the synthetic dataset comprises a generated plurality of images (Paragraph 6 in 2nd Col. of Page 12, Paragraph 1 in 1st Col. of Page 13 – “AGORA[27] is a synthetic dataset with accurate annotations of body meshes and 3D translations, with 4,240 high realism textured scans in diverse poses and clothes”). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Goel to incorporate the teachings of Sun 2 to train the model using synthetic image data because in cases where there is not enough ground truth real-world data to effectively train the model, synthetic data provides additional data for helping the model learn and produce accurate results. Additionally, synthetic data may be more diverse and have more privacy since they do not show real people and the type of data within the dataset can be controlled.
Goel modified by Sun 2 still does not teach each image including a single human having visible hands positioned in a hand pose; wherein among the generated plurality of images, the hand poses are diverse. However, Patel teaches each image including a single human having visible hands positioned in a hand pose (Fig. 12 – The figure shows examples of the images with humans having hands from AGORA; see screenshot below); wherein among the generated plurality of images, the hand poses are diverse (Fig. 12 – The figure shows diverse hand poses in the images from AGORA; see screenshot below).
PNG
media_image8.png
729
1319
media_image8.png
Greyscale
Screenshot of Fig. 12 (taken from Patel)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Goel to incorporate the teachings of Patel to have visible hands and diverse hand poses in the synthetic image dataset for the benefit of being able to train the mesh recovery model to produce meshes with detailed hands, which would help make the meshes more realistic.
Regarding claim 19, Goel in view of Sun 2 and Patel teaches the method of claim 18. Goel does not teach wherein the synthetic dataset is a supplement to a dataset including ground truths for the 3D parametric model. However, Sun 2 teaches wherein the synthetic dataset is a supplement to a dataset including ground truths for the 3D parametric model (Paragraph 2 in 2nd Col. of Page 6, Paragraph 6 in 2nd Col. of Page 12, Paragraph 2 in 1st Col. of Page 13, Paragraph 1 in 2nd Col. of Page 13 – “For basic training, we use two 3D pose datasets (Human3.6M [10] and MuCo-3DHP [24]) and four 2D pose datasets (COCO [21], MPII [3], LSP [12], and CrowdPose [20]). We also use the pseudo SMPL annotations from [14] and WST on RH. Most samples in RH are collected from 2D pose datasets [20,21,46]. For a fair comparison, we only use the samples that are also used for training in compared methods [11, 18, 19, 25, 34, 48]. To compare with [18,28], we further fine-tune our model and ROMP on AGORA… AGORA[27] is a synthetic dataset…Human3.6M [8] is a single-person 3D pose dataset. It contains videos of 9 professional actors performing activities in 17 scenarios. It provides 3D pose annotations for each frame…we also use the pseudo-3D annotations [12] for training”; Note: the model is trained on multiple datasets. The synthetic dataset, AGORA, supplements the other datasets like Human3.6M, which includes annotations (ground truths) to be used for the parametric model). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Goel to incorporate the teachings of Sun 2 to supplement a dataset of ground truths for the 3D parametric model with a synthetic dataset because when there is not enough real-world ground truth data for effective training, synthetic data can be used instead and help the model learn to produce more accurate results.
Claim 21 is rejected under 35 U.S.C. 103 as being unpatentable over Goel in view of Kolen et al. (US 20200306640 A1), hereinafter Kolen.
Regarding claim 21, Goel teaches the method of claim 1. Goel further teaches performing a downstream task using each generated 3D mesh (Paragraph 2 in 2nd Col. of Page 14 – “As an alternative way to assess the quality of 3D human reconstruction, we evaluate various human mesh recovery systems on the downstream task of action recognition on AVA (please refer to [19] for more details on the task definition). More specifically, we take the tracklets from [19], which were generated by running PHALP [21] on the Kinetics [8] and AVA [3] datasets. Then, we replace the poses from various human mesh recovery models (i.e., PyMAF [28], PyMAF-X [27], PARE [9], CLIFF [12], HMAR[21], HMR2.0) and evaluate their performance on the action recognition task. In this pose-only setting, the action recognition model has access only to the 3D poses (in the SMPL format) and 3D location and is trained to predict the action of each person”; Note: a downstream task is performed on the meshes). Goel does not teach storing each generated 3D mesh; and wherein the downstream task comprises one or more of: generating a virtual 3D avatar; operating a virtual 3D avatar; controlling movement of an autonomous device; performing collision avoidance between a human and an autonomous device based on the generated 3D meshes; performing an interaction between a human and an autonomous device; and predicting a response to an interaction between a human and an autonomous device based on the generated 3D meshes. However, Kolen teaches storing each generated 3D mesh (Paragraph 0025 – “The custom character system may ultimately generate and store various custom data for the specific person, such as 3D mesh”); and wherein the downstream task comprises generating a virtual 3D avatar (Paragraph 0082, 0091 – “the custom character system 130 or an associated 3D rendering engine may apply a 3D rendering or shading procedure in which the generating texture information and/or other visual style information (from block 406) is applied to the generic 3D mesh (retrieved at block 402), posed in 3D virtual space with one or more virtual lighting sources, and a particular virtual camera location. The custom character system 130 may initially attempt to match a pose of the 3D model and the model's position relative to the virtual camera to the pose and position of the real person relative to a real camera in at least one input image or input video frame… the custom character system 130 may cause in-game presentation of the custom 3D virtual character”; Note: a 3D mesh is used to generate a virtual character); and operating a virtual 3D avatar (Paragraph 0046, 0082 – “The animation generator 138, which may operate in conjunction or cooperation with the behavior learning system 134, may be configured to generate animation data for a virtual character to move in a manner that mimics or approximates specific movements performed by the real life person depicted in input media…the custom character system 130 or an associated 3D rendering engine may apply a 3D rendering or shading procedure in which the generating texture information and/or other visual style information (from block 406) is applied to the generic 3D mesh (retrieved at block 402), posed in 3D virtual space with one or more virtual lighting sources, and a particular virtual camera location. The custom character system 130 may initially attempt to match a pose of the 3D model and the model's position relative to the virtual camera to the pose and position of the real person relative to a real camera in at least one input image or input video frame”; Note: the 3D mesh is used to generate a virtual character, which becomes animated (for operating)). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Goel to incorporate the teachings of Kolen to store the 3D meshes for the benefit of being able to use them or display them again in the future. It also would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Goel to incorporate the teachings of Kolen to have the downstream tasks be creating or operating a 3D avatar because meshes provide a basis for avatars. So having a mesh would make it easier and more efficient to create, edit, and animate the avatar.
Claim 22 is rejected under 35 U.S.C. 103 as being unpatentable over Goel in view of Zeng et al. (3D Human Mesh Regression with Dense Correspondence), hereinafter Zeng.
Regarding claim 22, Goel teaches the method of claim 1. Goel further teaches wherein the computer-implemented method is implemented by a neural model (Paragraph 1 in 2nd Col. of Page 1 – “In this paper, we present a fully transformer-based approach for recovering 3D meshes of human bodies from single images, and tracking them over time in video”; Note: the transformer-based approach is a neural model since transformers are a type of neural network); wherein the neural model is end-to-end trained (Paragraph 2 in 2nd Col. of Page 13 – “For our main model HMR2.0b, we train the network end-to-end”) using a loss comprising at least one regression loss (Paragraph 1 in 1st Col. of Page 4 – “Given an input image I, the model predicts Θ = [θ,β,π] = f(I). Whenever we have access to the ground-truth SMPL pose parameters θ∗ and shape parameters β∗, we bootstrap the model predictions using an MSE loss:
PNG
media_image9.png
38
321
media_image9.png
Greyscale
”; Note: the MSE loss is a regression loss); and wherein the at least one regression loss comprises one or more of a parameters loss, an image plane reprojection loss, or a loss for human-centered output meshes (Paragraph 1 in 1st Col. of Page 4 – “Given an input image I, the model predicts Θ = [θ,β,π] = f(I). Whenever we have access to the ground-truth SMPL pose parameters θ∗ and shape parameters β∗, we bootstrap the model predictions using an MSE loss:
PNG
media_image9.png
38
321
media_image9.png
Greyscale
”; Note: the MSE regression loss is a parameters loss). Goel does not teach a human detection loss wherein the detection loss comprises a cross-entropy loss. However, Zeng teaches a human detection loss wherein the detection loss comprises a cross-entropy loss (Paragraph 1 in 1st Col. of Page 5 – “The decoder first generates a mask of the human body, which distinguishes fore pixels (i.e. human body) from those at the back… The loss function for the CNet contains two terms,
PNG
media_image10.png
28
224
media_image10.png
Greyscale
where Lc is a dense binary cross-entropy loss for classifying each pixel as ‘fore’ or ‘back’…”; Note: Lc is a human detection loss and is a cross-entropy loss). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Goel to incorporate the teachings of Zeng to have a human detection loss that is cross-entropy because the human detection loss helps ensure that the model is accurately identifying people, and cross-entropy works well with classification to identify if an area in the image corresponds to a human or not.
Claims 24-25 are rejected under 35 U.S.C. 103 as being unpatentable over Goel in view of Tan and Meinhardt.
Regarding claim 24, Goel in view of Tan teaches the system of claim 23. Goel further teaches wherein N is greater than one (Fig. 2 – The figure shows that there is more than one human; see screenshot above), the body model parameters comprise pose and shape parameters (Paragraph 2 in Col. 2 of Page 3 – “we model f to predict Θ = [θ,β,π] = f(I) where θ and β are the SMPL pose and shape parameters and π is the camera translation”), and the 3D mesh is a whole-body mesh (Fig. 2 – The figure shows that the generated 3D meshes represent the whole body; see screenshot above). Goel does not teach wherein the decoder is configured to generate a query from the embedded features for each of the N respective regions to provide N generated cross-attention queries and input the N generated cross-attention queries and the embedded features for each of the plurality of regions into the cross-attention module; and wherein the embedded features for each of the plurality of regions provide cross-attention keys and values for the cross-attention module. However, Meinhardt teaches wherein the decoder is configured to generate a query from the embedded features for each of the N respective regions to provide N generated cross-attention queries (Paragraph 4-5 in 2nd Col. of Page 3, Paragraph 7 in 1st Col. of Page 6 – “we introduce the concept of track queries to the decoder. Track queries follow objects through a video sequence carrying over their identity information while adapting to their changing position in an autoregressive manner. For this purpose, each new object detection initializes a track query with the corresponding output embedding of the previous frame…for the decoder we stack the feature maps of the previous and current frame and compute cross-attention with queries over both frames”; Note: the queries are generated/initialized with embeddings. There are N cross-attention queries because the cross-attention is computed with those queries and there are N queries (one per detected object and each object corresponds to a region in the frame)) and input the N generated cross-attention queries and the embedded features for each of the plurality of regions into the cross-attention module (Paragraph 5 in 2nd Col. of Page 3, Paragraph 7 in 1st Col. of Page 6 – “The Transformer encoder-decoder performs attention on frame features and decoder queries continuously updating the instance-specific representation of an object‘s identity and location in each track query embedding… for the decoder we stack the feature maps of the previous and current frame and compute cross-attention with queries over both frames”); and wherein the embedded features for each of the plurality of regions provide cross-attention keys and values for the cross-attention module (Paragraph 3 and 5 in 2nd Col. of Page 3, Paragraph 1 in 1st Col. of Page 4, Paragraph 7 in 1st Col. of Page 6 – “each object query learns to predict objects with certain spatial proper ties, such as bounding box size and position…The Transformer encoder-decoder performs attention on frame features and decoder queries continuously updating the instance-specific representation of an object‘s identity and location in each track query embedding… queries (white) are decoded to output embeddings for potential track initializations. Each valid object detection {b00, b10, . . . } with a classification score above σobject, i.e., output embedding not predicting the background class (crossed), initializes a new track query embedding… for the decoder we stack the feature maps of the previous and current frame and compute cross-attention with queries over both frames”; Note: the embeddings of the object position/location and actual object identity are the keys and values respectively for cross-attention). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Goel to incorporate the teachings of Meinhardt to generate cross-attention queries for each region for the benefit of finding features for each human in the image so that parameters can be predicted for each one. Additionally, cross-attention helps connect the object and position data from the image, which improves parameter predictions. It also would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Goel to incorporate the teachings of Meinhardt to have the embedded features be the keys and values for the cross-attention module because logically, the features are what the model is trying to find and process.
Regarding claim 25, Goel in view of Tan teaches the system of claim 23. Goel further teaches wherein the decoder comprises a transformer model (Paragraph 2 in 1st Col. of Page 13 – “Our transformer decoder is a standard transformer decoder architecture [23] with 6 layers, each containing multi-head self-attention, multi-head cross-attention, and feed-forward blocks, with layer normalization”); wherein said decoder further comprises a multi-layer perceptron (MLP) configured to regress the body model and depth parameters from the updated queries (Paragraph 1 in 2nd Col. of Page 4, Paragraph 2 in 1st Col. of Page 13 – “A tracklet representation is incrementally built up for each individual person over time. The recursion step is to predict for each tracklet, the pose, location and appearance of the person in the next frame, all in 3D, and then find best matches between these top-down predictions and the bottom-up detections of people in that frame after lifting them to 3D. The state represented by each tracklet is then updated by the incoming observation, and the process is iterated. It is possible to track through occlusions because the 3D representation of a tracklet continues to be updated based on past history… Our transformer decoder is a standard transformer decoder architecture [23] with 6 layers, each containing multi-head self-attention, multi-head cross-attention, and feed-forward blocks, with layer normalization [2]. It has a 2048 hidden dimension, 8 (64-dim) heads for self- and cross-attention, and a hidden dimension of 1024 in the feed-forward MLP block. It operates on a single learnable 2048-dimensional SMPL query token as input and cross-attends to the 16 × 12 image tokens. Finally, a linear readout on the output token from the transformer decoder gives pose θ, shape β, and camera”; Note: the 3D pose, location, and appearance, which are equivalent to the body model and depth parameters, are regressed/predicted based on past history that gets updated, which is the updated queries in this case. The process goes through an MLP). Goel does not teach wherein the cross-attention module generates updated queries; wherein said decoder further comprises a self-attention module for further updating the queries. However, Meinhardt teaches wherein the cross-attention module generates updated queries (Paragraph 5 in 2nd Col. of Page 3, Paragraph 7 in 1st Col. of Page 6 – “The Transformer encoder-decoder performs attention on frame features and decoder queries continuously updating the instance-specific representation of an object‘s identity and location in each track query embedding…For track queries, the deformable reference points for the current frame are dynamically adjusted to the previous frame bounding box centers. Furthermore, for the decoder we stack the feature maps of the previous and current frame and compute cross-attention with queries over both frames”; Note: encoder-decoder attention is the same as cross-attention, and it updates the queries); wherein said decoder further comprises a self-attention module for further updating the queries (Paragraph 5 in 2nd Col. of Page 3 – “The Transformer encoder-decoder performs attention on frame features and decoder queries continuously updating the instance-specific representation of an object‘s identity and location in each track query embedding. Self-attention over the joint set of both query types allows for the detection of new objects while simultaneously avoiding re-detection of already tracked objects”; Note: self-attention further updates the queries). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Goel to incorporate the teachings of Meinhardt to update the queries with cross-attention because it “gives queries global access to the visual information of the encoded features. The output embeddings accumulate bounding box and class information over multiple decoding layers” (Meinhardt: Paragraph 5 in 1st Col. of Page 3). In other words, cross-attention helps connect the object and position data from the image, which improves parameter predictions. It also would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Goel to incorporate the teachings of Meinhardt to update the queries with self-attention because self-attention “allows for joint reasoning about the objects in a scene… Self-attention over the joint set of both query types allows for the detection of new objects while simultaneously avoiding re-detection of already tracked objects” (Meinhardt: Paragraph 5 in 1st Col. of Page 3, Paragraph 5 in 2nd Col. of Page 3). In other words, it helps with identifying and tracking the humans in the image.
Claim 30 is rejected under 35 U.S.C. 103 as being unpatentable over Goel in view of Tan and Pavlakos.
Regarding claim 30, Goel in view of Tan teaches the system of claim 23. Goel further teaches wherein said image encoder comprises a Vision Transformer (Paragraph 4 in 2nd Col. of Page 3 – “The Vision Transformer, or ViT [15] is a transformer [74] that has been modified to operate on an image. The input image is first patchified into input tokens and passed through the transformer to get output tokens. The output tokens are then passed to the transformer decoder”; Note: the vision transformer encodes the input image). Goel does not teach wherein the 3D parametric model comprises a SMPL-X model; and wherein the generated 3D meshes comprise human faces, hands, and feet. However, Pavlakos teaches wherein the 3D parametric model comprises a SMPL-X model (Paragraph 5 in 2nd Col. of Page 6, Paragraph 2 in 2nd Col. of Page 8 – “we fit SMPL-X to the EHF images to evaluate both qualitatively and quantitatively… In this work we present SMPL-X, a new model that jointly captures the body together with face and hands. We additionally present SMPLify-X, an approach to fit SMPL-X toa single RGB image and 2D OpenPose joint detections”); and wherein the generated 3D meshes comprise human faces, hands, and feet (Fig. 2 – The figure shows each generated 3D mesh with expression pose and faces, hands, and feet; see screenshot above). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Goel to incorporate the teachings of Pavlakos to use SMPL-X for the 3D parametric model because it provides a more realistic mesh representation than SMPL, which is what Goel uses. “Compared to SMPLify [10], SMPLify-X uses a better pose prior (Section 3.3), a more detailed collision penalty (Section 3.4), gender detection (Section 3.5), and a faster PyTorch implementation… A strong holistic model like SMPL-X results in natural and expressive reconstruction of bodies, hands and faces” (Pavlakos: Paragraph 2 in 1st Col. of Page 4, Fig. 4 Caption on Page 8). It also would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Goel to incorporate the teachings of Pavlakos to have faces, hands, and feet for the 3D mesh because it would provide a more detailed and realistic visualization that is more appealing to the eye.
Claims 31-32 are rejected under 35 U.S.C. 103 as being unpatentable over Goel in view of Tan, Kolen, and Kikuchi et al. (US 11222549 B2), hereinafter Kikuchi.
Regarding claim 31, Goel in view of Tan teaches the system of claim 23. Goel further teaches wherein the instructions executable by the processor further implement performing a downstream task using the generated 3D meshes (Paragraph 2 in 2nd Col. of Page 14 – “As an alternative way to assess the quality of 3D human reconstruction, we evaluate various human mesh recovery systems on the downstream task of action recognition on AVA (please refer to [19] for more details on the task definition). More specifically, we take the tracklets from [19], which were generated by running PHALP [21] on the Kinetics [8] and AVA [3] datasets. Then, we replace the poses from various human mesh recovery models (i.e., PyMAF [28], PyMAF-X [27], PARE [9], CLIFF [12], HMAR[21], HMR2.0) and evaluate their performance on the action recognition task. In this pose-only setting, the action recognition model has access only to the 3D poses (in the SMPL format) and 3D location and is trained to predict the action of each person”; Note: a downstream task is performed on the meshes). Goel does not teach a memory for storing the generated 3D meshes; a controller for performing a downstream task using the generated 3D meshes; and an actuator coupled to said controller for actuating an autonomous device; wherein the downstream task comprises one or more of: controlling movement of an autonomous device using the actuator including collision avoidance based on the generated 3D meshes, and performing an interaction between a human and an autonomous device. However, Kolen teaches a memory for storing the generated 3D meshes (Paragraph 0025 – “The custom character system may ultimately generate and store various custom data for the specific person, such as 3D mesh”; Note: it is implied that there is a memory since mesh data is stored). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Goel to incorporate the teachings of Kolen to have store the 3D meshes for the benefit of being able to use or display the meshes again in the future. Goel modified by Kolen still does not teach a controller for performing a downstream task using the generated 3D meshes; an actuator coupled to said controller for actuating an autonomous device; wherein the downstream task comprises one or more of: controlling movement of an autonomous device using the actuator including collision avoidance based on the generated 3D meshes, and performing an interaction between a human and an autonomous device. However, Kikuchi teaches a controller for performing a downstream task using the generated 3D meshes (Col. 4 lines 24-25, Col. 11 lines 39-50– “The physics engine 102 creates a three-dimensional mesh representing a virtual world…the collision avoidance system determines whether the predicted trajectory will result in a collision of the virtual UAV model with the virtual world model… the collision avoidance system performs evasive actions by the UAV based on the capability parameters of the UAV, in response to determining that the predicted trajectory will result in a collision. In some implementations, performing evasive actions comprises overriding controller inputs from a controller device to control the UAV to perform at least one of a braking or a banking maneuver by adjusting at least one of a roll, pitch, yaw, or throttle of the UAV”; Note: the collision avoidance system is a controller that controls the UAV for performing the downstream task of collision avoidance using generated virtual models, which correspond to 3D meshes in this case); an actuator coupled to said controller for actuating the autonomous device (Col. 9 lines 48-56 – “The collision avoidance system 100 performs evasive actions by the UAV…the evasive actions may include overriding controller inputs from a controller device to control the UAV to perform braking or a banking maneuver by adjusting at least one of a roll, pitch, yaw, or throttle of the UAV to avoid an obstacle in the physical world”; Note: the collision avoidance system is a controller that controls the UAV, which is the autonomous device. It is implied to be coupled to an actuator since it causes the UAV to perform a mechanical action, like braking); wherein the downstream task comprises one or more of: controlling movement of an autonomous device using the actuator including collision avoidance based on the generated 3D meshes (Col. 4 lines 24-25, Col. 11 lines 39-50– “The physics engine 102 creates a three-dimensional mesh representing a virtual world…the collision avoidance system determines whether the predicted trajectory will result in a collision of the virtual UAV model with the virtual world model… the collision avoidance system performs evasive actions by the UAV based on the capability parameters of the UAV, in response to determining that the predicted trajectory will result in a collision. In some implementations, performing evasive actions comprises overriding controller inputs from a controller device to control the UAV to perform at least one of a braking or a banking maneuver by adjusting at least one of a roll, pitch, yaw, or throttle of the UAV”); and performing an interaction between a human and an autonomous device (Col. 4 lines 24-25, Col. 11 lines 39-50– “The physics engine 102 creates a three-dimensional mesh representing a virtual world…the collision avoidance system determines whether the predicted trajectory will result in a collision of the virtual UAV model with the virtual world model… the collision avoidance system performs evasive actions by the UAV based on the capability parameters of the UAV, in response to determining that the predicted trajectory will result in a collision. In some implementations, performing evasive actions comprises overriding controller inputs from a controller device to control the UAV to perform at least one of a braking or a banking maneuver by adjusting at least one of a roll, pitch, yaw, or throttle of the UAV”; Note: the UAV, which is the autonomous device, performs an action avoiding an object in its path, which is a type of interaction. The human was previously taught by Goel in the rejection of claim 23 and is represented by the object in this case). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Goel to incorporate the teachings of Kikuchi to have a controller and actuator for a downstream task of collision avoidance of an autonomous device because “HMR 2.0 is such a large pre-trained model which could potentially be useful not just in computer vision, but also in robotics” (Goel: Paragraph 2 in 1st Col. of Page 2). In other words, it would be beneficial to use the HMR system in an autonomous device, like a robot vehicle, to help control the device for reasons like safety between the device and the human, as well as efficiency for the device. Additionally, a controller and actuator are required for controlling the device to perform the task.
Regarding claim 32, Goel in view of Tan teaches the system of claim 23. Goel further teaches wherein the instructions executable by the processor further implement performing a downstream task using the generated 3D meshes (Paragraph 2 in 2nd Col. of Page 14 – “As an alternative way to assess the quality of 3D human reconstruction, we evaluate various human mesh recovery systems on the downstream task of action recognition on AVA (please refer to [19] for more details on the task definition). More specifically, we take the tracklets from [19], which were generated by running PHALP [21] on the Kinetics [8] and AVA [3] datasets. Then, we replace the poses from various human mesh recovery models (i.e., PyMAF [28], PyMAF-X [27], PARE [9], CLIFF [12], HMAR[21], HMR2.0) and evaluate their performance on the action recognition task. In this pose-only setting, the action recognition model has access only to the 3D poses (in the SMPL format) and 3D location and is trained to predict the action of each person”; Note: a downstream task is performed on the meshes). Goel does not teach a memory for storing the generated 3D meshes; a controller for performing a downstream task using the generated 3D meshes; an actuator coupled to said controller for actuating the autonomous device; wherein the downstream task comprises one or more of: generating a virtual 3D avatar; and operating a virtual 3D avatar; wherein said controller is coupled to a display for displaying a 3D avatar in an executed virtual reality or augmented reality application. However, Kolen teaches a memory for storing the generated 3D meshes (Paragraph 0025 – “The custom character system may ultimately generate and store various custom data for the specific person, such as 3D mesh”; Note: it is implied that there is a memory since mesh data is stored); a controller for performing a downstream task using the generated 3D meshes (Paragraph 0010, 0082, 0115, 0124 – “The instructions may further configure the computing system to render, within a 3D virtual environment of a video game, a series of frames for display that depict the custom virtual character… the custom character system 130 or an associated 3D rendering engine may apply a 3D rendering or shading procedure in which the generating texture information and/or other visual style information (from block 406) is applied to the generic 3D mesh… Display I/O 36 provides input/output functions that are used to display images from the game being played… the processor can be a controller”; Note: the processor is a controller that creates and displays a virtual character from a 3D mesh, as a downstream task); and wherein the downstream task comprises generating a virtual 3D avatar (Paragraph 0082, 0091 – “the custom character system 130 or an associated 3D rendering engine may apply a 3D rendering or shading procedure in which the generating texture information and/or other visual style information (from block 406) is applied to the generic 3D mesh (retrieved at block 402), posed in 3D virtual space with one or more virtual lighting sources, and a particular virtual camera location. The custom character system 130 may initially attempt to match a pose of the 3D model and the model's position relative to the virtual camera to the pose and position of the real person relative to a real camera in at least one input image or input video frame… the custom character system 130 may cause in-game presentation of the custom 3D virtual character”; Note: a 3D mesh is used to generate a virtual character); and operating a virtual 3D avatar (Paragraph 0046, 0082 – “The animation generator 138, which may operate in conjunction or cooperation with the behavior learning system 134, may be configured to generate animation data for a virtual character to move in a manner that mimics or approximates specific movements performed by the real life person depicted in input media…the custom character system 130 or an associated 3D rendering engine may apply a 3D rendering or shading procedure in which the generating texture information and/or other visual style information (from block 406) is applied to the generic 3D mesh (retrieved at block 402), posed in 3D virtual space with one or more virtual lighting sources, and a particular virtual camera location. The custom character system 130 may initially attempt to match a pose of the 3D model and the model's position relative to the virtual camera to the pose and position of the real person relative to a real camera in at least one input image or input video frame”; Note: the 3D mesh is used to generate a virtual character, which becomes animated (for operating)); wherein said controller is coupled to a display for displaying a 3D avatar in an executed virtual reality or augmented reality application (Fig. 8, Paragraph 0010, 0055, 0057, 0115, 0124 – “The instructions may further configure the computing system to render, within a 3D virtual environment of a video game, a series of frames for display that depict the custom virtual character…player computing system 102 may include any type of computing device(s), such as desktops, laptops, game application platforms, virtual reality systems, augmented reality systems… The player computing system 102 is capable of executing one or more game applications…Display I/O 36 provides input/output functions that are used to display images from the game being played… the processor can be a controller”; Note: Fig. 8 shows the display coupled to the processing unit (controller)). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Goel to incorporate the teachings of Kolen to have a controller and store the 3D meshes for the benefit of processing specific tasks related to the meshes and being able to use or display the meshes again in the future. It also would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Goel to incorporate the teachings of Kolen to have the downstream tasks be creating or operating a 3D avatar because meshes provide a basis for avatars. So having a mesh would make it easier and more efficient to create, edit, and animate the avatar. Goel modified by Kolen still does not teach an actuator coupled to said controller for actuating the autonomous device. However, Kikuchi teaches an actuator coupled to said controller for actuating the autonomous device (Col. 9 lines 48-56 – “The collision avoidance system 100 performs evasive actions by the UAV…the evasive actions may include overriding controller inputs from a controller device to control the UAV to perform braking or a banking maneuver by adjusting at least one of a roll, pitch, yaw, or throttle of the UAV to avoid an obstacle in the physical world”; Note: the collision avoidance system is a controller that controls the UAV, which is the autonomous device. It is implied to be coupled to an actuator since it causes the UAV to perform a mechanical action, like braking). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Goel to incorporate the teachings of Kikuchi to have an actuator for actuating an autonomous device because “HMR 2.0 is such a large pre-trained model which could potentially be useful not just in computer vision, but also in robotics” (Goel: Paragraph 2 in 1st Col. of Page 2). In other words, it would be beneficial to use the HMR system in an autonomous device, like a robot vehicle, to help control the device.
Allowable Subject Matter
Claim 33 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Claim 33 would be allowable for disclosing wherein the instructions executable by the processor further implementing a downstream task control module that controls an operation of the image capturing device based on the generated 3D meshes; an actuator controlled using said downstream task control module; and a display controlled using said downstream task control module.
Regarding claim 33, Goel in view of Tan teaches the system of claim 23. Goel further teaches an image capturing device comprising a camera; at least one image capturing device for obtaining the 2D image (Paragraph 4 in 1st Col. of Page 3, Paragraph 1 and 4 in 2nd Col. of Page 3 – “Each camera π = (R,t) consists of a global orientation R ∈ R3×3 and translation t ∈ R3. Given these parameters, points in the SMPL space (e.g., joints X) can be projected to the image as x =π(X)=Π(K(RX+t)), where Π is a perspective projection with camera intrinsics K…The input image is first patchified into input tokens...We use a ViT-H/16, the ‘Huge’ variant with 16×16 input patch size”; Note: there is a camera for obtaining an image); wherein the instructions executable by the processor further implementing a downstream task control module (Paragraph 2 in 2nd Col. of Page 14 – “As an alternative way to assess the quality of 3D human reconstruction, we evaluate various human mesh recovery systems on the downstream task of action recognition on AVA (please refer to [19] for more details on the task definition). More specifically, we take the tracklets from [19], which were generated by running PHALP [21] on the Kinetics [8] and AVA [3] datasets. Then, we replace the poses from various human mesh recovery models (i.e., PyMAF [28], PyMAF-X [27], PARE [9], CLIFF [12], HMAR[21], HMR2.0) and evaluate their performance on the action recognition task. In this pose-only setting, the action recognition model has access only to the 3D poses (in the SMPL format) and 3D location and is trained to predict the action of each person”; Note: the action recognition model is equivalent to the downstream task control module). However, none of the prior art teaches the claim limitation of “implementing a downstream task control module that controls an operation of the image capturing device based on the generated 3D meshes; an actuator controlled using said downstream task control module; and a display controlled using said downstream task control module” nor would it have been obvious to incorporate these features into Goel. Therefore, the combination of features is considered allowable.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Guler et al. (US 20220358770 A1) teaches a method of generating a 3D scene including multiple objects from 2D images by identifying and reconstructing each object in 3D. Kulon et al. (US 20230070008 A1) teaches a method of generating 3D models from 2D images using an embedding neural network and a learned decoder model.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MICHELLE HAU MA whose telephone number is (571)272-2187. The examiner can normally be reached M-Th 7-5:30.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, King Poon can be reached at (571) 270-0728. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MICHELLE HAU MA/ Examiner, Art Unit 2617
/KING Y POON/Supervisory Patent Examiner, Art Unit 2617
Prosecution Insights