DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant's arguments filed 26 January 2026 have been fully considered but they are not persuasive. Applicant’s arguments with respect to the prior art have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Objections
Claim 1 is objected to because of the following informalities: Lines 25-26 recite “the second modified the key point information”. The second “the” appears to be a typographical error. Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 9 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 9 recites the limitations “the current iteration" and “the next iteration” in lines 1-2. There is insufficient antecedent basis for these limitations in the claim.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1, 6, 7, 15, 16, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Wang (U.S. Publication 2023/0298204) in view of Wang ‘731 (U.S. Publication 2021/0390731), and Georgakis (U.S. Publication 2021/0183097).
As to claim 1, Wang discloses a method of estimating an object posture, the method comprising:
determining key point information in an image, the key point information comprising two-dimensional coordinates, in the image, of key points of an object in the image (p. 1, sections 0020-0021; p. 6, sections 0055-0059; key point information including 2D coordinates is determined for objects such as joints in a skeleton);
determining two-dimensional modified key point information (p. 7, sections 0063-0064; the key point information is modified via normalization);
estimating an object posture of an object in the image, based on the modified key point information, the object posture comprising an orientation and location of the object (p. 1, section 0020; p. 7, sections 0065-0066; pose/posture, including position/location and orientation of an object are estimated based on the key point information modified via normalization) and estimating a second object posture of the object based on second modified key point information (p. 9, sections 0082-0083; p. 16-17, section 0147; keypoint information is updated, further modified, and pose/posture is estimated).
Wang does not disclose, but Wang ‘731 discloses obtaining key point feature maps of the key points respectively, each key point feature map defined according to the coordinates of its corresponding key point (p. 1, section 0003; p. 3, sections 0052-0053; key point features, which read on “key point feature maps” since they are extracted portions of larger feature maps, are extracted according to corresponding positions/coordinates of the key point); determining key point offsets of the respective key points from the key point feature maps of the respective key points and determining modified key point information by adjusting the coordinates of the respective key points according to the respectively corresponding key point offsets (p. 3, section 0054-0057; p. 4, section 0060; offsets are determined from the individual features/feature maps and used to adjust positions/coordinates of each corresponding key point). The motivation for this is to learn different types of features and improve key point positioning accuracy (p. 2, section 0020). It would have been obvious to one skilled in the art before the effective filing date of the claimed invention to modify Wang to extract from the image key point feature maps of the key points respectively, each key point feature map defined according to the coordinates of its corresponding key point, determine key point offsets of the respective key points from the key point feature maps of the respective key points and determine modified key point information by adjusting the coordinates of the respective key points according to the respectively corresponding key point offsets determined from the feature maps corresponding to the key points in order to learn different types of features and improve key point positioning accuracy as taught by Wang ‘731.
Wang does not disclose, but Georgakis discloses a key point feature map comprising a depth feature map (p. 2, section 0020-p. 3, section 0021; based on 2D coordinates of the keypoints, depth images are annotated, and depth feature maps are generated from them; the depth feature maps are then provided to the network along with keypoint data) and that offsets are determined by inputting each key point and its corresponding key point feature map to a refinement network that predicts the key point offsets therefrom, wherein the key point offsets comprise residuals of the refinement network (p. 4, section 0029; p. 4, sections 0032-0033; an error/residual is generated based on each key point’s distance/offset to corresponding matching 3D points; the method using the trained network can proceed iteratively, making this a refinement network), updating the key points based on the estimated object posture, determining second key point offsets of the respective key points by inputting each updated key point and its corresponding key point feature map to the refinement network, which predicts the second key point offsets therefrom, wherein the second key point offsets comprise residuals of the refinement network, determining second two-dimensional modified key point information by adjusting the 2D coordinates of the respective updated key points according to the respective updated key points according to the respectively corresponding second key point offsets (p. 4, section 0029; p. 4, sections 0032-0033; the key points are updated based on estimated pose/posture and, assuming the algorithm has not converged, another iteration of the same process described above is run). The motivation for this is to avoid inaccurate feature representations that would occur if only the depth images were used (p. 2, section 0014). It would have been obvious to one skilled in the art to modify Wang and Wang ‘731 to use a key point feature map comprising a depth feature map, have offsets be determined by inputting each key point and its corresponding key point feature map to a refinement network that predicts the key point offsets therefrom, wherein the key point offsets comprise residuals of the refinement network, update the key points based on the estimated object posture, determine second key point offsets of the respective key points by inputting each updated key point and its corresponding key point feature map to the refinement network, which predicts the second key point offsets therefrom, wherein the second key point offsets comprise residuals of the refinement network, and determine second two-dimensional modified key point information by adjusting the 2D coordinates of the respective updated key points according to the respective updated key points according to the respectively corresponding second key point offsets in order to avoid inaccurate feature representations as taught by Georgakis.
As to claim 6, Wang discloses wherein the second estimated object posture is more accurate than the estimated object posture (p. 16-17, section 0147; models that generate the pose are refined to improve pose/posture accuracy, meaning that second estimates would be more accurate than first estimates, third estimates would be more accurate than second, etc.).
As to claim 7, Wang discloses a method further comprising determining 3D key point mapping information of the object in the based on the estimated object posture (p. 8, sections 0073-0075; 3D keypoint coordinate information, reading on mapping information, is determined from an estimated 3D pose/posture) and updating the key point information based on the 3D key point mapping information (p. 9, sections 0082-0083; keypoint information is updated, further modified, and pose/posture is estimated).
As to claim 15, see the rejection to claim 1. Further, Wang discloses a non-transitory computer-readable storage medium storing instructions that, when executed by a processor, cause the processor to perform the method (p. 16, sections 0144-0145).
As to claim 16, see the rejection to claim 1. Further, Wang discloses an apparatus for estimating an object posture, the apparatus comprising: one or more processors; storage storing instructions configured to, when executed by the one or more processors, cause the one or more processors to perform the method (p. 16, sections 0144-0145).
As to claim 20, see the rejection to claim 6.
Claims 2 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Wang, Wang ‘731, and Georgakis in view of Kalra (U.S. Publication 2022/0044441).
As to claim 2, Wang does not disclose, but Kalra discloses prior to the determining the key point information in the image, receiving an input image; and obtaining the image by converting the input image to an image having a preset image style (p. 20, sections 0216-0217; p. 21, section 0223; style transfer is used to convert the input image to a different style, such as an unrealistic painterly style, before keypoint determination is performed). The motivation for this is to have images appear more consistent for training. It would have been obvious to one skilled in the art before the effective filing date of the claimed invention to modify Wang, Wang ‘731, and Georgakis to, prior to determining the key point information in the image, receive an input image, and obtain the image by converting the input image to an image having a preset image style in order to have images appear more consistent for training as taught by Kalra.
As to claim 17, Wang does not disclose, but Kalra discloses wherein the image is converted from an input image to be a version of the input image having an image style that the input image does not have (p. 20, sections 0216-0217; images are given a characteristic, such as more artificiality, that was not present in the original image). Motivation for the combination is given in the rejection to claim 2.
Claims 3, 4, and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Wang, Wang ‘731, and Georgakis in view of Kalra and further in view of Li (U.S. Publication 2023/0082050).
As to claim 3, Kalra discloses style conversion in general but does not explicitly disclose the steps of extracting an image-content feature of the input image by a content neural network, obtaining a preset image-style feature, generating an integrated feature by integrating the image-content feature of the input image with the preset image-style feature, and obtaining the image by rendering the input image with a renderer, wherein the rendering is based on the integrated feature. Li, however, does disclose extracting an image-content feature of the input image by a content neural network, obtaining a preset image-style feature (fig. 4, elements 404 and 406; p. 4, section 0037; p. 6, section 0051; neural networks are used to extract a content feature from an input image and a style feature from some other image that would read on a preset image), generating an integrated feature by integrating the image-content feature of the input image with the preset image-style feature (fig. 4, elements 408 and 410), and obtaining the image by rendering the input image with a renderer, wherein the rendering is based on the integrated feature (p. 6, section 0053; an image with integrated content and style features is generated/rendered). The motivation for this is to allow a content creator to creatively manipulate images to generate expressive artwork (p. 1, section 0001). It would have been obvious to one skilled in the art before the effective filing date of the claimed invention to modify Wang, Wang ‘731, Georgakis and Kalra to extract an image-content feature of the input image by a content neural network, obtain a preset image-style feature, generate an integrated feature by integrating the image-content feature of the input image with the preset image-style feature, and obtain the image by rendering the input image with a renderer, wherein the rendering is based on the integrated feature in order to allow a content creator to creatively manipulate images to generate expressive artwork as taught by Li.
As to claim 4, Li discloses wherein the image-content feature of the input image has a resolution lower than the image, and wherein the preset image-style identifies the image-content feature as corresponding to the resolution (p. 4, section 0037; content and style features in an image are extracted at various resolutions/scales lower than an original image; the style of the corresponding feature is identified at a corresponding resolution for feature transform; transform takes place at 56x56, 112x112, etc.). Motivation for the combination is given in the rejection to claim 3.
As to claim 18, see the rejection to claim 3.
Claims 5 and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Wang in view of Wang ‘731 and Georgakis and further in view of Ayush (U.S. Publication 2021/0142539).
As to claim 5, Wang does not disclose, but Ayush discloses wherein the determining the modified key point information comprises: obtaining key point offsets by performing offset regression on the key point feature map (figs. 3-4; p. 2, section 0025; p. 3, section 0036; p. 5, section 0053-p. 6, section 0060; p. 6, section 0063; p. 7, sections 0067-0068; information sent over a residual connection, as shown in fig. 3, is obtained by performing coarse offset regression, generating a coarse offset matrix based on priors including keypoint information; the output is an image with warped/modified pose, and thus associated keypoints). The motivation for this is to use warping to efficiently and accurately generate try-on images (p. 1, section 0006). It would have been obvious to one skilled in the art before the effective filing date of the claimed invention to modify Wang, Wang ‘731, and Georgakis to obtain key point offsets by performing offset regression on the key point feature map in order to use warping to efficiently and accurately generate try-on images as taught by Ayush.
As to claim 19, see the rejection to claim 5.
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Wang in view of Wang ‘731 and Georgakis and further in view of Yang (U.S. Publication 2022/0148243).
As to claim 8, Wang does not disclose, but Yang discloses wherein the determining the 3D key point mapping information is based on the estimated object posture, a preset 3D model set corresponding to the object, and a camera eigen matrix (p. 3, section 0037-p. 4, section 0042; pose/posture, a 3D shape model and a camera rotation eigenmatrix are used to map 3D points in an image to another image). The motivation for this is to support real-time search for image editing (p. 4, section 0042). It would have been obvious to one skilled in the art before the effective filing date of the claimed invention to modify Wang, Wang ‘731, and Georgakis to determine the 3D key point mapping information based on the estimated object posture, a preset 3D model set corresponding to the object, and a camera eigen matrix in order to support real-time search for image editing as taught by Yang.
Claims 9 and 10 are rejected under 35 U.S.C. 103 as being unpatentable over Wang, Wang ‘731, and Georgakis and further in view of North (U.S. Publication 2021/0110603).
As to claim 9, Wang does not disclose, but North discloses wherein the current iteration and the next iteration are part of an iterative refinement process that comprises: iteratively estimating object postures of the object in the input image until a termination condition is satisfied, wherein the termination condition comprises either: a difference of a key point before and after modification being less than a threshold, or a number of modifications of the key point reaching a predetermined number of times (fig. 3; p. 3, section 0030; p. 5-6, section 0050; for a surface of an input image, if vertex/keypoint position change, which would define an object pose, is less than a precision threshold, the vertex/keypoint refinement is terminated; otherwise the method loops back to the start and further iterations are performed). The motivation for this is to improve continuity and smoothing. It would have been obvious to one skilled in the art before the effective filing date of the claimed invention to modify Wang, Wang ‘731, and Georgakis to iteratively perform the refinement process on the object posture until a termination condition is satisfied, wherein the termination condition comprises a difference of a key point before and after modification being less than a threshold in order to improve continuity and smoothing as taught by North.
As to claim 10, Wang does not disclose, but North discloses wherein the difference of a key point before and after modification being less than a threshold comprises: either a sum of differences of at least one key point before and after modification being less than the threshold, or a difference of each key point of the at least one key point before and after modification being less than the threshold (p. 5-6, section 0050; it is seen if each vertex difference is less than the precision threshold). Motivation for the combination is given in the rejection to claim 9.
Claim 11 is rejected under 35 U.S.C. 103 as being unpatentable over Wang in view of Wang ‘731 and Georgakis and further in view of Wang ‘967 (U.S. Publication 2022/0405967).
As to claim 11, Wang does not disclose, but Wang ’967 does disclose wherein the determining the key point information comprises: determining the two dimensional coordinates and key point visible information through a detector network that processes the image; and determining the key point information based on the two-dimensional coordinates and the key point visible information (figs. 1-3; p. 7, sections 0106-0113; human body key points and their visibility vs. occludedness is determined by a detector network; determined key point info includes both the key point position in a 2D image as shown and key point confidence based on visibility info). The motivation for this is to more accurately judge hand/body connections in complex scenes as compared to prior art methods (p. 1, section 0003). It would have been obvious to one skilled in the art before the effective filing date of the claimed invention to modify Wang, Wang ‘731, and Georgakis to determine two-dimensional coordinates and key point visible information through a detector network that processes the image and determine the key point information based on the two-dimensional coordinates and the key point visible information in order to more accurately judge hand/body connections in complex scenes as compared to prior art methods as taught by Wang ‘967.
Claim 12 is rejected under 35 U.S.C. 103 as being unpatentable over Wang in view of Wang ‘731 and Georgakis and further in view of Vajda (U.S. Publication 2019/0172224).
As to claim 12, Wang does not disclose, but Vajda discloses wherein the determining the key point information further comprises: obtaining a region of interest (Rol) feature map in the image through the detector network (p. 4, sections 0040-0041; p. 5, sections 0046-0047; feature maps are derived for the RoIs using a network for detection, reading on a detector network) and prior to the determining the modified key point information: determining the key point feature map corresponding the key point information in the Rol feature map, based on the key point information (fig. 3; p. 5, section 0046; p. 5-6, section 0049; p. 7, section 0056; keypoint masks and heatmaps, reading on feature maps, are determined corresponding to the RoI feature maps; the keypoint maps are later refined/modified by comparing errors vs. ground truth). The motivation for this is to provide a more efficient method for keypoint detections by identifying most likely regions for joints (p. 1, sections 0006-0009). It would have been obvious to one skilled in the art before the effective filing date of the claimed invention to modify Wang, Wang ‘731, and Georgakis to obtain a region of interest (Rol) feature map in the image through the detector network, and prior to determining the modified key point information, determine the key point feature map corresponding the key point information in the Rol feature map, based on the key point information in order to provide a more efficient method for keypoint detections by identifying most likely regions for joints as taught by Vajda.
Claim 13 is rejected under 35 U.S.C. 103 as being unpatentable over Wang in view of Wang ‘731, Georgakis, and Wang ‘967 and further in view of Vajda.
As to claim 13, Wang ‘967 discloses determining an object category of the object in the image through the detector network (p. 6, section 0098; p. 7, section 0106; it is determined whether an object is a hand or wrist; this can be done with the detection network). Wang ‘967 does not disclose, but Vajda discloses prior to the determining the modified key point information: determining the key point feature map based on the object category and the key point information (fig. 3; p. 5, section 0045; p. 5-6, section 0049; p. 7, section 0056; a classifier determines object categories, used with RoI keypoint info to determine the keypoint masks and heat maps; the keypoint maps are later refined/modified by comparing errors vs. ground truth). Motivation for the combination is given in the rejection to claim 12.
Claim 14 is rejected under 35 U.S.C. 103 as being unpatentable over Gupta (U.S. Publication 2022/0215580) in view of Kalra, Wang ‘731, and Georgakis.
As to claim 14, Gupta discloses a method of estimating an object posture, the method comprising:
obtaining, from a converted image, local feature maps of the respective key points (p. 2, sections 0027-0031; p. 3, section 0034; p. 4, sections 0055-0056; a number of maps are generated from a source image; the source image can be augmented, reading on a converted image);
determining modified key points based on the key point feature maps and their respectively corresponding key points, the modified key points respectively corresponding to the key points, wherein each key point is displaced according to its keypoint offset (fig. 2; fig. 4; p. 1, sections 0017-0018; p. 2, sections 0027-0031; p. 3, section 0034; p. 3, sections 0040-0042; transformed/modified key point information is generated using a feature map corresponding to the key point information; different maps are used for different source images with different key points; the displacement is with a per-location vector that reads on an offset);
Gupta does not disclose, but Kalra discloses converting an input image to a converted image having a preset image style (p. 20, sections 0216-0217; style transfer is used to convert the input image to a different style, such as an unrealistic painterly style), determining key points of an object information in the converted image having the preset image style, and for a current iteration, estimating an object posture of an object in the input image having the preset style based on the 2D modified key points and 3D key points corresponding to modified key points (p. 21, section 0223; p. 23, section 0251; keypoint determination is performed on the converted image for a pose estimator; the keypoints can be modified reprojected 2D keypoints and associated 3D keypoints of 3D objects) and for a next iteration, estimating a second object posture of the object in the input image having the preset style based on the object posture of the object estimated for the current iteration (figs. 4-5; the process refines pose and is iterative, repeating previous steps). Motivation for the combination of references is similar to that given in the rejection to claim 2.
Gupta does not disclose, but Wang ‘731 discloses that each local feature map is determined according to its corresponding key point and determining key point offsets of the respective key points, wherein each key point’s key point offset is determined from its corresponding local feature map (p. 1, section 0003; p. 3, sections 0052-0057; p. 4, section 0060; key point features, which read on “key point feature maps” since they are extracted portions of larger feature maps, are extracted according to corresponding positions/coordinates of the key point; offsets are determined from the individual features/feature maps and used to adjust positions/coordinates of each corresponding key point). Motivation for the combination of references is similar to that given in the rejection to claim 1.
Gupta does not disclose, but Georgakis discloses a key point feature map comprising a depth feature map from a converted image (p. 2, section 0017; p. 2, section 0020-p. 3, section 0021; images are converted to depth images; based on 2D coordinates of the keypoints, depth images are annotated, and depth feature maps are generated from them; the depth feature maps are then provided to the network along with keypoint data) and that offsets are determined by inputting each key point and its corresponding key point feature map to a refinement network that predicts the key point offsets therefrom, wherein the key point offsets comprise residuals of the refinement network (p. 4, section 0029; p. 4, sections 0032-0033; an error/residual is generated based on each key point’s distance/offset to corresponding matching 3D points; the method using the trained network can proceed iteratively, making this a refinement network). Motivation for the addition of the Georgakis reference is similar to that given in the rejection to claim 1.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to AARON M RICHER whose telephone number is (571)272-7790. The examiner can normally be reached 9AM-5PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, King Poon can be reached at (571)272-7440. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/AARON M RICHER/Primary Examiner, Art Unit 2617
Prosecution Insights