Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
This action is in response to the amendment filed on 1/21/2026. Claims 3, 13, 14, and 19 have been amended while claims 21-23 have been added. Amendments have been fully considered but are not persuasive. Claims 1-23 remain rejected in the application.
Response to Arguments
In response to applicant’s arguments regarding Wang failing to disclose the limitation “for each image, computing a set of keypoint projections in the image plane of the image, the keypoint projections being 2D projections of 3D semantic keypoints of the initial 3D model”. Argument’s fully considered but is not persuasive because Wang explicitly teaches the limitation, [Wang: 0041 “A 3D point S, in the facial surface is computed by back-projecting po based on the initial pose estimates Mo in the first frame. The 3D point S, is on the 1-th planar facet in the triangulated mesh structure of the morphable 3D face model 202. With the correct camera poses M and M in the last two frames, the p; and p2 can be predicted based on the 3D point.”][Wang: 0042 “the re-projection error”](teaches computing 2d predictions (projections) of 3d points from the 3d model using object and camera poses which correspond to keypoint projections in the image plane).
In response to applicant’s arguments regarding Wang failing to disclose the limitation “determining a new 3D model and a new object pose for each image based on an aggregate reprojection error between the 2D semantic keypoint detections and the keypoint projections across the time sequence of multiple images”. Argument’s fully considered but is not persuasive because Wang explicitly teaches the limitation [Wang: 0031 “The optimization effected in the feature estimator 220 may integrate batch processing and model-based bundle adjustment, which results in robust 3D information recovery from the video clip 104.”][Wang: 0032 “2D point matching, head poses, and the 3D face model are iteratively estimated and refined by the feature estimator 220 and the feature refiner 222.”][Wang: 0033 “the feature estimator 220 is able to efficiently estimate matched features across all the frames of the video clip 104”][Wang: 0041 “the essential optimization function for one feature matching result in this segment, which can be implemented using the re-projection constraint. It can be assumed that the index of the planar facet is not changed when the 3D point S, is refined with the model parameters b, until the 2D-3D re-matcher 232 performs.”][Wang: 0042 “the re-projection error in the second image X,d(p.Hpi) is used as a unit cost function that may be minimized in order to estimate parameters for overdetermined solutions”](teaches minimizing reprojection error to jointly refine the 3d model and per frame head pose, using bundle adjustment over frames, aggregate reprojection across the time sequence).
In response to applicant’s arguments regarding dependent claims being allowable, Examiner kindly suggests explaining in detail how those claims differ compared to the cited reference. All arguments have been fully considered but are not persuasive. Claims 1-23 remain rejected in the application.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1, 2, 3, 4, 9, 10, 15, 16, 17, 18, 19, and 20 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Wang et al. (U.S. Patent Publication No. 2007/0091085).
Regarding claim 1, Wang discloses a computer-implemented method of locating and modelling a moving 3D object captured in a time sequence of multiple images, the method comprising (interpreted as a software method that builds a 3D model of a moving object using images over time)[Wang: 0017 “This disclosure describes systems and methods for automatically obtaining an accurate 3-dimensional (3D) face model from a video sequence.”](teaches automatic 3D modeling from a video sequence of a moving face which corresponds to a 3D object): applying, to each image, semantic keypoint detection, in order to generate a set of 2D semantic keypoint detections in an image plane of the image (interpreted as for every frame, detect labeled 2D landmarks (keypoints) in the image)[Wang: 0024 “The layer iterator 206 recursively estimates and refines 2-dimensional (2D) feature matching”][Wang: 0033 “for each frame instead of each segment. When expanded to the entire video clip 104, the feature estimator 220 is able to efficiently estimate matched features across all the frames of the video clip 104”][Wang: 0036 “The face aligner 227 then extracts salient facial features”][Wang: 0037 “The initializer 216 then establishes semantic 2D-3D feature correspondences and uses”](teaches salient facial features and semantic 2d-3d feature which are semantic keypoints, further teaches applying detection/estimation on a per frame basis and yields 2d features in the image plane); receiving, for each image, a camera pose and an initial object pose, each pose comprising a 3D location and 3D orientation [Wang: 0024 “A 3D face model 224“in-the-making and a camera pose 226 lie intermediate between the feature estimator 220 and the feature refiner 222.”][Wang: 0037 “The initializer 216 then establishes semantic 2D-3D feature correspondences and uses, for example, the POSIT technique to obtain an approximate initial face pose”][Wang: 0032 “the underlying 3D face model and camera poses for each frame are determined.”](explicitly teaches managing a per frame camera pose and computes an initial object (face) pose via POSIT. Although wang determines rather than externally receives, per frame poses are available to and used by the pipeline); determining an initial 3D model of the 3D object [Wang: 0021 “engine 110 also includes or communicates with a morphable 3D face model 202, which can be a conventional “3D mesh' model that begins as a generic 3D face mesh that is changeable into a universe of other 3D faces by providing enough parameters to adjust the various “triangular grids' making up the 3D model.”](teaches providing an initial generic morphable 3D face model that is then fitted which corresponds to an initial 3d model of the object); based on the initial 3D model, and the initial object pose and the camera pose for each image, computing a set of keypoint projections in the image plane of the image, the keypoint projections being 2D projections of 3D semantic keypoints of the initial 3D model [Wang: 0041 “A 3D point S, in the facial surface is computed by back-projecting po based on the initial pose estimates Mo in the first frame. The 3D point S, is on the 1-th planar facet in the triangulated mesh structure of the morphable 3D face model 202. With the correct camera poses M and M in the last two frames, the p; and p2 can be predicted based on the 3D point.”][Wang: 0042 “the re-projection error”](teaches computing 2d predictions (projections) of 3d points from the 3d model using object and camera poses which correspond to keypoint projections in the image plane); and determining a new 3D model and a new object pose for each image based on an aggregate reprojection error between the 2D semantic keypoint detections and the keypoint projections across the time sequence of multiple images (interpreted as optimize shape and per frame pose by minimizing summed reprojection error over frames comparing detected 2d keypoints vs projected 3d keypoints) [Wang: 0031 “The optimization effected in the feature estimator 220 may integrate batch processing and model-based bundle adjustment, which results in robust 3D information recovery from the video clip 104.”][Wang: 0032 “2D point matching, head poses, and the 3D face model are iteratively estimated and refined by the feature estimator 220 and the feature refiner 222.”][Wang: 0033 “the feature estimator 220 is able to efficiently estimate matched features across all the frames of the video clip 104”][Wang: 0041 “the essential optimization function for one feature matching result in this segment, which can be implemented using the re-projection constraint. It can be assumed that the index of the planar facet is not changed when the 3D point S, is refined with the model parameters b, until the 2D-3D re-matcher 232 performs.”][Wang: 0042 “the re-projection error in the second image X,d(p.Hpi) is used as a unit cost function that may be minimized in order to estimate parameters for overdetermined solutions”](teaches minimizing reprojection error to jointly refine the 3d model and per frame head pose, using bundle adjustment over frames, aggregate reprojection across the time sequence).
Regarding claim 2, Wang discloses the method of claim 1, wherein the new 3D model and the new object pose are determined via optimization of a cost function (interpreted as shape and pose are obtained by minimizing a defined objective (cost) function)[Wang: 0041 “where I is the essential optimization function for one feature matching result in this segment, which can be implemented using the re-projection constraint.”][Wang: 0044 “The final function to be minimized by an IRLS (iterative reweighed least square) procedure way is, as shown in Equation (5):”](teaches that the process as optimization of a function and states that a final function is minimized to recover model parameters and pose corresponding to determine via optimization of a cost function), the cost function comprising a reprojection error term (interpreted as the objective includes a term measuring 2d reprojection error between predicted and observed points)[Wang: 0042 “In one implementation, the re-projection error in the second image X,d(p.Hpi) is used as a unit cost function that may be minimized in order to estimate parameters for overdetermined solutions.”](teaches that the optimization includes a reprojection error term used as a cost to be minimized corresponding to cost function comprising a reprojection error term).
Regarding claim 3, Wang discloses the method of claim 1, wherein each semantic keypoint detection pertains to a semantic keypoint type, and is in a form of a distribution over possible 2D semantic keypoint locations in the image plane for that semantic keypoint type, the distribution encoding detection confidence for that semantic keypoint type (interpreted as each detected landmark is of a labeled type like eye corner or nose tip, the detector outputs a distribution over 2d image coordinates for that type. Values in the distribution represent detection confidence)[Wang: 0036 “The face aligner 227 then extracts salient facial features”][Wang: 0054 “small-baseline features matcher 240 performs a block matching search in the neighborhood of the window in I. 304, denoted as a confidence region, based on the new narrow baseline pair I', 306 and I, 304.”](teaches salient facial features which are the semantic keypoint types. Further teaches confidence region in which the 2d location of a feature is searched, conveying uncertainty over possible positions).
Regarding claim 4, Wang discloses the method of claim 2, wherein the aggregate reprojection error for each semantic keypoint type is weighted in the reprojection error term according to a detection confidence for that semantic keypoint type (interpreted as, in the reprojection loss, give each landmark type a weight based on how confident the detector is for that type, higher confidence means higher weight and lower confidence means lower weight)[Wang: 0042 “In one implementation, the re-projection error in the second image X,d(p.Hpi) is used as a unit cost function that may be minimized in order to estimate parameters for overdetermined solutions.”][Wang: 0044 “The final function to be minimized by an IRLS (iterative reweighed least square) procedure way is, as shown in Equation (5):”][Wang: 0054 “small-baseline features matcher 240 performs a block matching search in the neighborhood of the window in I. 304, denoted as a confidence region, based on the new narrow baseline pair I', 306 and I, 304.”](teaches minimizing a reprojection error cost, IRLS (a weighted least squares scheme), and detected confidence region).
Regarding claim 9, Wang discloses the method of claim 1, wherein the initial 3D model encodes expected location information about the 3D keypoints (interpreted as the starting 3D model carries prior knowledge of where semantic 3D landmarks are on the surface)[Wang: 0053 “"Model based' flow means that at least a rough face shape (model) is already known and prediction is made (or refined) with feature-based data. In the model-based flow used by the feature refiner 222, image features are constrained by the geometry of the underlying 3D face shape.”][Wang: 0037 “The initializer 216 then establishes semantic 2D-3D feature correspondences”][Wang: 0041 “A 3D point S, in the facial surface is computed by back-projecting po based on the initial pose estimates Mo in the first frame. The 3D point S, is on the 1-th planar facet in the triangulated mesh structure of the morphable 3D face model 202. With the correct camera poses M and M in the last two frames, the p; and p2 can be predicted based on the 3D point.”](teaches using a generic morphable face mesh and states that features are constrained by the model’s geometry. It establishes semantic 2d-3d correspondences and predicts image features from known 3d surface points. Together, this shows the initial model encodes expected 3D keypoint locations).
Regarding claim 10, Wang discloses the method of claim 9, wherein the 3D model has a set of one or more shape parameters that define a shaped 3D object surface [Wang: 0021 “also includes or communicates with a morphable 3D face model 202, which can be a conventional “3D mesh' model that begins as a generic 3D face mesh that is changeable into a universe of other 3D faces by providing enough parameters to adjust the various “triangular grids' making up the 3D model.”], wherein the semantic keypoints are defined relative to the shaped 3D object surface [Wang: 0041 “A 3D point S, in the facial surface is computed by back-projecting po based on the initial pose estimates Mo in the first frame. The 3D point S, is on the 1-th planar facet in the triangulated mesh structure of the morphable 3D face model 202. With the correct camera poses M and M in the last two frames, the p; and p2 can be predicted based on the 3D point. Since the face shape is represented by a triangular mesh, any point on one of the triangles is a linear combination of the three triangle vertices, which is a function of model parameters, and any point on a triangle is also a function of model parameters.”](teaches predicting image locations from 3d point that place the keypoints relative to the shaped surface (mesh)), wherein the new 3D model is determined by tuning the shape parameters (interpreted as solve for updated shape parameters to get the new model)[Wang: 0026 “The feature estimator 220 may further include a (head) pose estimator 228, a model (deformation) parameter estimator 230, a 2D-3D re-matcher 232, and a convergence comparator 234.”][Wang: 0050 “The optimal value of model coefficients can be computed as in Equation (8):”][Wang: 0051 “The regularization term is used to normalize the dimension size of the two terms when minimizing the objective function. In one implementation, the adaptive regularization term is applied after the objective function is decreased in the first several optimization iterations. In one implementation, the minimum value of the regularization term is set to maintain the power of the regularization term in smoothing the face surface.”](teaches computing optimal model coefficients via optimization which is tuning the shape parameters to obtain the new 3d model).
Regarding claim 15, Wang discloses the method of claim 1, wherein a single 3D model is determined for the time sequence of images (interpreted as video) for modelling a rigid object (interpreted as fixed shape)[Wang: 0018 “estimate the final 3D face shape for frames of the video clip”][Wang: 0029 “A consideration when face-modeling from the video clip 104 is how to match the generic morphable 3D face model 202 to all frames of the video clip 104 accurately and in an automatic and efficient way.”][Wang: 0084 “the feature matching can be efficiently estimated across all frames with adaptive refinement of face shape. The refined feature correspondences for each segment are combined together in the last post-processing step to further refine the 3D face model”][Wang: 0029 “The coordinate system of the morphable 3D model 202 is assumed fixed in 3D”](teaches fitting one morphable 3d face model across the clip and computing a final 3d face shape corresponding to single 3d model, further teaches processing all frames of the video sequence to obtain the model corresponding to time sequence of images, and finally, further teaching fixed shape corresponding to modelling a rigid object).
Regarding claim 16, Wang discloses the method of claim 1, wherein the 3D model is a deformable model (interpreted as the model is parameterized and can change) [Wang: 0021 “The exemplary 3D face-modeling engine 110 also includes or communicates with a morphable 3D face model 202, which can be a conventional “3D mesh' model that begins as a generic 3D face mesh that is changeable into a universe of other 3D faces by providing enough parameters to adjust the various “triangular grids' making up the 3D model”](teaches the 3D face mesh is changeable corresponding to deformable), and wherein the 3D model varies across different images of the time sequence of images (interpreted as the model parameters are updated per image, not held constant across all frames)[Wang: 0033 “the 3D face geometry and the motion parameters (head poses) are now more efficiently estimated in the feature estimator 220, but this time for each frame instead of each segment.”](teaches per frame estimation and adaptive refinement of face shape).
Regarding claim 20, Wang discloses the computer program of claim 18, wherein the initial 3D model encodes expected location information about the 3D keypoints (interpreted as the model provides predicted positions for 3d feature points)[Wang: 0041 “A 3D point S, in the facial surface is computed by back-projecting po based on the initial pose estimates Mo in the first frame.” “p; and p2 can be predicted based on the 3D point.”](teaches initial pose estimates where image features should appear in other frames).
Claims 17 and 18 are computer system and program claims corresponding to claim 1 without any additional limitations. Thus, claims 17 and 18 are rejected for the same reasons as claim 1 above.
Claim 19 is a computer program claim corresponding to claim 3 above without any additional limitations. Thus, claim 19 is rejected for the same reasons as claim 3 above.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over Wang et al. (U.S. Patent Publication No. 2007/0091085), in view of Lee et al. (U.S. Patent No. 10,657,376), in further view of Bao et al. (U.S. 2020/0043189).
Regarding claim 5, Wang discloses the method of claim 4, but fails to explicitly disclose wherein the distribution over possible 2D semantic keypoint locations is Gaussian, and the aggregate reprojection error is weighted by covariance for each keypoint type.
However, Lee discloses wherein the distribution over possible 2D semantic keypoint locations is Gaussian (Lee: Col. 8, Lines 10-13 “Each keypoint ground truth can be represented by a 2D Gaussian heat map centered at the true keypoint location as one of the channels in the output layer. In some embodiments, the keypoint heat maps 320r0-320”)(teaches representing keypoint location uncertainty as a 2d gaussian heatmap which corresponds to the claimed gaussian distribution).
However, Bao discloses and the aggregate reprojection error is weighted by covariance for each keypoint type [Bao: 0017 “the camera pose C ; is estimated by minimizing a relative error and a reprojection error”][Bao: 0018 “Here , log ( T ) maps a 3D rigid transformation TESE ( 3 ) to se ( 3 ) and returns a vector representation in in Rº . Mells ? = e72-1e is standard Mahalanobis distance .”](teaches Mahalanobis distance which equals covariance weighted squared error).
Wang, Lee, and Bao are considered analogous to the claimed invention because they are in the same field of keypoint based object modeling and reprojection based optimization. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify the system of Wang to incorporate Lee an Bao’s teachings of modeling each landmark as a Gaussian distribution with a learned covariance and weighting reprojection errors via Mahalanobis. The motivation for such a combination would provide the benefit of uncertainty aware Gaussian keypoint distributions supply principled covariance weights for the reprojection error objective, improving robustness and accuracy across frames.
Claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over Wang et al. (U.S. Patent Publication No. 2007/0091085), in view of Cootes et al. (EP 2 893 491).
Regarding claim 6, Wang discloses the method of claim 2, but fails to explicitly disclose wherein the cost function comprises a regularization term that penalizes deviation between the 3D semantic keypoints and a 3D semantic keypoint prior.
However, Cootes discloses wherein the cost function comprises a regularization term that penalizes deviation between the 3D semantic keypoints and a 3D semantic keypoint prior (interpreted as the objective includes a prior based penalty that keeps 3D keypoints close to a prior model)[Cootes: 0002 “A widely used approach is to use a statistical shape model to regularise the output of independent feature detectors trained to locate each model point within two-dimensional (2D) or three-dimensional (3D) image data.”][Cootes: 0024 “The first term encodes the shape constraints, the second the image matching information. The cost function Q(p) can then be optimised either using a general purpose optimiser [4], or using the mean-shift approach advocated by Saragih et al. [5].”](teaches regularizer in a cost, further teaches the penalty on model points relative to a shape prior).
Wang and Cootes are both considered analogous to the claimed invention because they are in the same field of model based landmark fitting and 3D morphable model priors. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify the system of Wang to incorporate Cootes teachings of using a 3D shape prior and to apply the regularization term to penalize deviation. The motivation for such a combination would provide the benefit of improved robustness and plausibility by constraining optimization to prior consistent 3D keypoint configurations.
Claim 7 and 8 are rejected under 35 U.S.C. 103 as being unpatentable over Wang et al. (U.S. Patent Publication No. 2007/0091085), in view of Cootes et al. (EP 2 893 491), in further view of Marks et al. (U.S. Patent Publication No. 2021/0104068).
Regarding claim 7, Wang and Cootes disclose the method of claim 6, but fail to explicitly disclose wherein the 3D semantic keypoint prior comprises, a distribution over possible 3D semantic keypoint locations for each semantic keypoint type, the distribution encoding an extent of expected variation in the 3D semantic keypoint locations, wherein a penalty for each semantic keypoint type in the regularization term is weighted according to the extent of expected variation for that semantic keypoint type.
However, Marks discloses wherein the 3D semantic keypoint prior comprises, a distribution over possible 3D semantic keypoint locations for each semantic keypoint type (interpreted as a probability distribution for each keypoints 3D location) [Marks: 0012 “An example of a parametric probability distribution defined by values of parameters for a location of each landmark in each processed image is a Gaussian distribution , wherein the parameters determine a mean and a covariance matrix of the Gaussian distribution.”](teaches a per landmark probability distribution over location), the distribution encoding an extent of expected variation in the 3D semantic keypoint locations (interpreted as the distribution carries variance)[Marks: 0012 “the covariance matrix defines the uncertainty . Note that the covariance matrix defines the multidimensional ( e.g. , two dimensional ) shape and structure of the Gaussian probability distribution over the landmark's location .”](teaches that covariance encodes uncertainty for each landmarks location corresponding to extent of expected variation), wherein a penalty for each semantic keypoint type in the regularization term is weighted according to the extent of expected variation for that semantic keypoint type (interpreted as the loss weighs each keypoint by its covariance) [Marks: 0016 “the neural network can be trained using negative log likelihood as a loss function .”][Marks: 0017 “Gaussian log - likelihood loss function enables simultaneous estimation of landmark locations and their uncertainty”](teaches Gaussian NLL applies weighting and ties the penalty to each landmarks covariance corresponding to the claimed limitation).
Wang, Cootes, and Marks are considered analogous to the claimed invention because they are in the same field of statistical shape modeling and probabilistic, per landmark priors. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify the system of Wang and Cootes to incorporate Mark’s teachings of using a per keypoint probability distribution with covariance that encodes expected variation and weighting the regularization penalty by that covariance. The motivation for such a combination would provide the benefit of improved robustness and statistically consistent estimation by variance-aware weighting of each keypoint.
Regarding claim 8, Wang and Cootes disclose the method of claim 7, but fail to explicitly disclose wherein the distribution over possible 3D semantic keypoint locations is Gaussian, and the penalty in the regularization term for each semantic keypoint type is weighted by covariance.
However, Marks discloses wherein the distribution over possible 3D semantic keypoint locations is Gaussian (interpreted each keypoint has a gaussian prior over its location) [Marks: 0012 “An example of a parametric probability distribution defined by values of parameters for a location of each landmark in each processed image is a Gaussian distribution , wherein the parameters determine a mean and a covariance matrix of the Gaussian distribution . In this example , the mean defines the point of location of the landmark , and the covariance matrix defines the uncertainty . Note that the covariance matrix defines the multidimensional ( e.g. , two dimensional ) shape and structure of the Gaussian probability distribution over the landmark's location .”][Wang: 0015 “Gaussian probability distribution for each landmark”][Marks: 0019 “Cholesky Estimator Network ( CEN ) , which estimates Cholesky coefficients of a covariance matrix of a 2D Gaussian probability distribution for each landmark location . Estimation of a Gaussian log - likelihood loss ( GLL ) where the estimated landmark location is used as the mean of the Gaussian distribution , and the estimated covariance matrix is used as the covariance of the Gaussian distribution”](teaches Gaussian distribution per landmark location), and the penalty in the regularization term for each semantic keypoint type is weighted by covariance [Marks: 0072 “Maximizing the likelihood ( 1 ) is equivalent to minimizing negative log likelihood . Therefore , the negative log likelihood is utilized as the loss function . The loss function Li ; at each hourglass i for the landmark j can be written as the sum of two terms , T , and Tz”].
Wang, Cootes, and Marks are considered analogous to the claimed invention because they are in the same field of statistical shape modeling and probabilistic, per landmark priors. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify the system of Wang and Cootes to incorporate Mark’s teachings of using a Gaussian distribution for each landmark location and weighting the penalty by the landmarks covariance. The motivation for such a combination would provide the benefit of stable fitting, noise tolerance, and accuracy through covariance aware penalization.
Claim 11 is rejected under 35 U.S.C. 103 as being unpatentable over Wang et al. (U.S. Patent Publication No. 2007/0091085), in view of Jin (U.S. Patent No. 8,693,734).
Regarding claim 11, Wang discloses the method of claim 2, but with the camera poses fixed [Wang: 0029 “with using a fixed camera position”] and without any assumption that the moving 3D object remains static [Wang: 0056 “The video clip 104 can be obtained by capturing a head turn of the user in front of a static camera”], but fails to explicitly disclose wherein the new 3D model and the new object pose are determined using a structure from motion algorithm applied to the cost function.
However, Jin discloses wherein the new 3D model and the new object pose are determined using a structure from motion algorithm applied to the cost function (Jin: Col. 5, Lines 16-19 “Embodiments of the technique may be used to detect poorly conditioned points during the bundle adjustment process or portion of a feature-based 3D reconstruction pipeline (e.g., an SFM pipeline).”)(Jin: Col. 6, Lines 62-64 “Bundle adjustment may generally be formulated as a large nonlinear optimization.”)(ties the bundle adjustment and cost minimization to SfM corresponding to structure from motion algorithm).
Wang and Jin are considered analogous to the claimed invention because they are in the same field of multi frame 3d reconstruction. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify the system of Wang to incorporate Jin’s teachings of utilizing the SfM algorithm. The motivation for such a combination would provide the benefit of stable convergence.
Claim 12 is rejected under 35 U.S.C. 103 as being unpatentable over Wang et al. (U.S. Patent Publication No. 2007/0091085), in view of Zhang et al. (U.S. Patent Publication No. 2006/0078172).
Regarding claim 12, Wang discloses the method of claim 1, wherein the 3D semantic keypoints of the 3D model correspond to only a first subset of the 2D semantic keypoint detections for each image (interpreted as use only a selected set of 2D detections per frame to anchor 3D points) [Wang: 0038 “The feature rough-matching engine 218 selects relevant features in the first frame of each segment and then, for example, applies the KLT technique”](teaches selecting relevant features which is a subset of 2D detections used for correspondence to model points), wherein the aggregate reprojection error is determined between (i) the 3D semantic keypoints and the first subset of 2D semantic keypoint detections, and (ii) the reflected 3D semantic keypoints and the second subset of 2D semantic keypoint detections (interpreted as sum reprojection errors for original first subset and mirrored second subset) [Wang: 0042 “the re-projection error in the second image X,d(p.Hpi) is used as a unit cost function that may be minimized in order to estimate parameters for overdetermined solutions.”](teaches the reprojection error formula), but fails to explicitly disclose and a set of reflected 3D semantic keypoints, corresponding to a second subset of the 2D semantic keypoint detections, is determined by reflecting the 3D semantic keypoints about a symmetry plane of the 3D object model.
However, Zhang discloses and a set of reflected 3D semantic keypoints, corresponding to a second subset of the 2D semantic keypoint detections, is determined by reflecting the 3D semantic keypoints about a symmetry plane of the 3D object model (interpreted as mirror the models 3D keypoints across a symmetry plane to create a second 3D set)[Wang: Abstract “the approach first automatically extracts the bilateral symmetry plane of the face Surface.”][Wang: 0055 “They first mirror the whole dataset of a face, which is represented in a 3D triangular mesh, at an arbitrary plane. Then the ICP (Iterative Closest Point) algorithm was used to align the original and the mirrored facial Surfaces”](teaches finding symmetry plane and mirroring the 3d facial mesh which yields reflected 3D points).
Wang and Zhang are considered analogous to the claimed invention because they are in the same field of image based 3D face modeling. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify the system of Wang to incorporate Zhang’s teachings of extracting facial symmetry plane. The motivation for such a combination would provide the benefit of improving fitting accuracy by exploiting facial bilateral symmetry.
Claims 13, 14, and 21-23 are rejected under 35 U.S.C. 103 as being unpatentable over Wang et al. (U.S. Patent Publication No. 2007/0091085), in view of Cootes et al. (EP 2 893 491), in further view of Blanz et al. (U.S. Patent No. 6,556,196).
Regarding claim 13, Wang and Cootes disclose the method of claim 6, but fail to explicitly disclose wherein the object belongs to a known object class, and the 3D semantic keypoint prior or expected location information has been learned from a training set comprising example objects of the known object class.
However, Blanz discloses wherein the object belongs to a known object class (Blanz: Col. 1, Lines 8-10 “images of three-dimensional objects, Such as human faces”)(teaches faces as a class for objects), and the 3D semantic keypoint prior or expected location information has been learned from a training set comprising example objects of the known object class (Blanz: Col. 5, Line 32 “The morphable model is based on a data set of 3D faces.”).
Wang, Cootes, and Blanz are considered analogous to the claimed invention because they are in the same field of statistical shape modeling. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify the system of Wang and Cootes to incorporate Blanz’s teachings of learning the prior/expected landmark locations from a training set of example faces of that class. The motivation for such a combination would provide the benefit of improved accuracy and plausibility.
Regarding claim 14, Wang, Cootes, and Blanz disclose the method of claim 13, comprising: using an object classifier to determine the known object class from multiple available object classes, the multiple object classes associated with respective expected 3D shape information [Wang: 0025 “The initializer 216 further includes a face detector 225, which may comprise a conventional face detection technique, and a face aligner 227.”](teaches using the initializer which includes a detector which is a classifier to detect faces which is an object class).
Regarding claim 21, Wang and Cootes disclose the method of claim 14, but fails to explicitly disclose wherein the object class corresponds to a car and the respective expected 3D shape information corresponds to an expected 3D surface of a car.
However, Blanz discloses wherein the object class corresponds to a car and the respective expected 3D shape information corresponds to an expected 3D surface of a car (Blanz: Col. 14, Lines 8-10 “applications of the invention are given in the field of modelling images of three-dimensional objects other than human faces”)(teaches that the objects can also be 3D objects other than human faces which means it could be a car).
Wang, Cootes, and Blanz are considered analogous to the claimed invention because they are in the same field of statistical shape modeling. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify the system of Wang and Cootes to incorporate Blanz’s teachings of utilizing 3D objects that are not human faces. The motivation for such a combination would provide the benefit of having a larger variety of objects to utilize.
Regarding claim 22, Wang and Blanz disclose the method of claim 14, but fail to explicitly disclose wherein the object class corresponds to a pedestrian, the respective expected 3D shape information corresponds to an expected 3D surface of a human.
However, Cootes discloses wherein the object class corresponds to a pedestrian, the respective expected 3D shape information corresponds to an expected 3D surface of a human [Cootes: 0008 “the Implicit Shape Model [20] uses local patches located on an object to vote for the object position, and Poselets [21] match patches to detect human body parts”].
Wang, Cootes, and Blanz are considered analogous to the claimed invention because they are in the same field of statistical shape modeling. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify the system of Wang and Blanz to incorporate Cootes’s teachings of modeling human body parts. The motivation for such a combination would provide the benefit of detecting the human body.
Regarding claim 23, Wang and Blanz disclose the method of claim 1, but fail to explicitly disclose wherein the multiple images are images of a driving scenario captured by one or more sensor of an ego vehicle, and the moving 3D object corresponds to an agent in the driving scenario.
However, Cootes discloses wherein the multiple images are images of a driving scenario captured by one or more sensor of an ego vehicle, and the moving 3D object corresponds to an agent in the driving scenario [Cootes: 0086 “A related application would be to monitor and model how a user interacts with machine controls, such as a driver in a car or a pilot in an aeroplane, by tracking where the person is looking to perform various operations and as various events occur”].
Wang, Cootes, and Blanz are considered analogous to the claimed invention because they are in the same field of statistical shape modeling. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to modify the system of Wang and Blanz to incorporate Cootes’s teachings of monitoring and modeling how a user interacts with a driver in a car. The motivation for such a combination would provide the benefit of improving modeling and tracking of agents in driving scenarios.
Conclusion
THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a). A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action. Any inquiry concerning this communication or earlier communications from the examiner should be directed to AHMED TAHA whose telephone number is (571)272-6805. The examiner can normally be reached 8:30 am - 5 pm, Mon - Fri. Examiner interviews are available via telephone, in person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, XIAO WU can be reached at (571)272-7761. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786- 9199 (IN USA OR CANADA) or 571-272-1000.
/AHMED TAHA/Examiner, Art Unit 2613
/XIAO M WU/Supervisory Patent Examiner, Art Unit 2613
Prosecution Insights