DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-7 are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Yamazaki et al. (US 20240320850) and Wang et al. (“Graph-PCNN: Two Stage Human Pose Estimation with Graph Pose Refinement”).
Regarding claim 1, Yamazaki et al. discloses an information processing apparatus comprising:
set determination means for determining a set including a plurality of keypoints for recognizing a pose of an object, on a basis of a three-dimensional model of the object (“In flow diagram 400, a projected keypoint is obtained for a feature of an input image 402. The projected keypoint may be obtained by applying a 3D pose and shape estimation process 404 (for example, by 3D pose and shape regression techniques as described above or any other 3D estimation techniques known in the art) on the feature of the input image 402 using to generate a 3D rendering 406, and then applying a 3D-2D keypoint projection process 408 on the 3D rendering 406 to obtain coordinates of the projected keypoint associated with the feature. The projected keypoint comprises a set of coordinates of the feature that is projected from the 3D rendering 406 of the input image 402” at paragraph 0041, line 7);
candidate determination means for determining one or a plurality of candidate keypoints that are candidates (“Further, a direct keypoint is obtained based on a 2D rendering of a feature of the input image 402. Specifically, the direct keypoint may be obtained by applying a 2D keypoint estimation process 412 (for example, by using heatmap estimation as described in FIG. 3 or any other 2D estimation techniques known in the art) on the feature of the input image 402” at paragraph 0042, line 1);
reliability determination means for determining reliability of each of the keypoints included in the set and the candidate keypoints from information (“Thereafter, at 416, a confidence score is computed based on the projected keypoint and the direct keypoint, wherein a higher confidence score indicates a higher accuracy of the projected and direct keypoints” at paragraph 0043, line 1) that is output when a captured image is input to a machine learning model trained and that indicates positions of the keypoints included in the set and the candidate keypoints, the machine learning model being configured to receive the captured image as an input and output information indicating the positions of the keypoints included in the set and information indicating the positions of the candidate keypoints (“3D pose and shape regressor refers to a module or process which estimates three-dimensional locations of human meshes (vertexes and surfaces), and camera parameters including 3D location of a camera and/or angles to render the 3D human meshes that align with a 2D body shape and pose that is identified in the input image. The module may be, for example, a trainable neural network model” at paragraph 0031, line 1).
Yamazaki et al. does not explicitly disclose that the plurality of candidate keypoints are candidates to replace at least some of the plurality of keypoints included in the set and replacement means for replacing the at least some of the keypoints included in the set with at least some of the candidate keypoints on a basis of the reliability determined.
Wang et al. teaches an apparatus in the same field of endeavor of human pose keypoint determination, comprising:
set determination means for determining a set including a plurality of keypoints for recognizing a pose of an object (“In the top-down manner pose estimation methods, single person pose estimator aims to locate K keypoints P = {p1, p2, ..., pk} from an image I of size W × H × 3, where pk is a 2D-coordinates. Heatmap based methods transform this problem to estimating K heatmaps {H1,H2, ...,Hk} of size Wi × Hi × K, where each heatmap Hk will be decoded to the corresponding coordinates pk during the test phase” at section 3, line 1);
candidate determination means for determining one or a plurality of candidate keypoints that are candidates to replace at least some of the plurality of keypoints included in the set
replacement means for replacing the at least some of the keypoints included in the set with at least some of the candidate keypoints on a basis of the reliability determined (“During training, N guided points {s1 k, s2 k, ..., sNk } are sampled for each heatmap Hk, while the best guided points s∗ k is selected for heapmap Hk during testing. For sake of simplification, we omit the superscript in the following formula. For any guided point sk, guided feature fk = F[sk] at the corresponding location and its confidence score hk = Hk[sk] can be extracted” at page 496, second to last paragraph, line 7; “Finally, the refined classification result ck and offset regression result rk are achieved based on the refined feature gk.” at page 497, line 2).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to utilize the keypoint refinement as taught by Wang et al. with the keypoints of Yamazaki et al. “to improve the accuracy of keypoint localization” (Wang et al. at section 1, second to last paragraph, last sentence).
Regarding claim 2, Yamazaki et al. discloses an apparatus wherein the machine learning model to which the captured image has been input outputs a plurality of images each indicating the positions of the keypoints included in the set and the candidate keypoints (Figure 4, numerals 410 and 414).
Regarding claim 3, Yamazaki et al. discloses an apparatus wherein,
in each of the plurality of images output by the machine learning model to which the captured image has been input, each point indicates a positional relation with any of the keypoints included in the set and the candidate keypoints (as seen in Figure 4 at numerals 410 and 414, the keypoints and their relative positions to each other are shown), and
the reliability determination means determines, regarding any of the plurality of images output, on a basis of a variation of a plurality of position candidates that are candidates for a position of any of a plurality of keypoints and candidate keypoints corresponding to the any of the images and that are obtained from different points included in the any of the images, reliability of the any of the plurality of keypoints and the candidate keypoints (“The above formula is based on the obtained positions or coordinates of the projected and direct keypoints, as well as a visibility value v of the direct keypoint. The above formula first calculates a consistency score based on the projected keypoint and direct keypoint, then calculates a confidence score by multiplying the consistency score with the visibility value v. The tuning parameter a in the consistency score calculation may be a pre-fixed value during the calculation which can be manually tuned based on experiments to get more accurate scores” at paragraph 0049, line 1).
Regarding claim 4, the Yamazaki et al. and Wang et al. combination discloses an apparatus further comprising:
pose determination means for determining the pose of the object from information that has been output when the captured image is input to the machine learning model and that indicates positions of some of the keypoints included in the set and any of the candidate keypoints (the output keypoints are representative of the object’s pose in the input image),
wherein the reliability determination means determines the reliability of the keypoints and the candidate keypoints on a basis of positions of the keypoints and the candidate keypoints according to the pose determined and the positions of the keypoints and the candidate keypoints indicated by the information output (“The above formula is based on the obtained positions or coordinates of the projected and direct keypoints, as well as a visibility value v of the direct keypoint. The above formula first calculates a consistency score based on the projected keypoint and direct keypoint, then calculates a confidence score by multiplying the consistency score with the visibility value v. The tuning parameter a in the consistency score calculation may be a pre-fixed value during the calculation which can be manually tuned based on experiments to get more accurate scores” at paragraph 0049, line 1).
The Yamazaki et al. and Wang et al. combination does not explicitly disclose re-projecting the keypoints and candidate keypoints to the determined pose.
However, both Yamazaki et al. and Wang et al. evaluate the keypoints of the determined pose for accuracy (“A plurality of projected keypoints 812 are obtainable from a plurality of extracted features of the input image 802. Further, the 3D rendering undergoes a 3D pose and shape loss calculation process 814 to obtain a 3D keypoint loss L3D for each extracted feature, and a 2D projected keypoint loss Lproj is calculated for each feature through a 2D projected keypoint loss calculation process 816” Yamazaki et al. at paragraph 0054, third to last sentence; see section 3.3 for Wang et al.). Yamazaki et al. further defines a consistency accuracy metric for comparing the individual projection image keypoints with each other (“Thereafter, at 824, a consistency loss value is computed based on the projected keypoint and the direct keypoint, wherein a lower consistency loss value indicates a higher accuracy of the projected and direct keypoints” at paragraph 0056, line 1).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to re-project the keypoints and candidate keypoints to the final pose to evaluate the performance of the individual keypoint prediction models.
Regarding claim 5, Yamazaki et al. discloses an apparatus further comprising:
pose determination means for determining the pose of the object from information that has been output when the captured image is input to the machine learning model and that indicates positions of some of the keypoints included in the set and any of the candidate keypoints (the output keypoints are representative of the object’s pose in the input image),
wherein the reliability determination means determines estimated reliability of each of the keypoints included in the set and the candidate keypoints on a basis of the pose determined and ground truth data on the pose of the object in the captured image (“A plurality of projected keypoints 812 are obtainable from a plurality of extracted features of the input image 802. Further, the 3D rendering undergoes a 3D pose and shape loss calculation process 814 to obtain a 3D keypoint loss L3D for each extracted feature, and a 2D projected keypoint loss Lproj is calculated for each feature through a 2D projected keypoint loss calculation process 816” at paragraph 0054, third to last sentence).
Regarding claim 6, Yamazaki et al. discloses an information processing method comprising:
determining a set including a plurality of keypoints for recognizing a pose of an object, on a basis of a three-dimensional model of the object (“In flow diagram 400, a projected keypoint is obtained for a feature of an input image 402. The projected keypoint may be obtained by applying a 3D pose and shape estimation process 404 (for example, by 3D pose and shape regression techniques as described above or any other 3D estimation techniques known in the art) on the feature of the input image 402 using to generate a 3D rendering 406, and then applying a 3D-2D keypoint projection process 408 on the 3D rendering 406 to obtain coordinates of the projected keypoint associated with the feature. The projected keypoint comprises a set of coordinates of the feature that is projected from the 3D rendering 406 of the input image 402” at paragraph 0041, line 7);
determining one or a plurality of candidate keypoints that are candidates (“Further, a direct keypoint is obtained based on a 2D rendering of a feature of the input image 402. Specifically, the direct keypoint may be obtained by applying a 2D keypoint estimation process 412 (for example, by using heatmap estimation as described in FIG. 3 or any other 2D estimation techniques known in the art) on the feature of the input image 402” at paragraph 0042, line 1);
determining reliability of each of the keypoints included in the set and the candidate keypoints from information (“Thereafter, at 416, a confidence score is computed based on the projected keypoint and the direct keypoint, wherein a higher confidence score indicates a higher accuracy of the projected and direct keypoints” at paragraph 0043, line 1) that is output when a captured image is input to a machine learning model trained and that indicates positions of the keypoints included in the set and the candidate keypoints, the machine learning model being configured to receive the captured image as an input and output information indicating the positions of the keypoints included in the set and information indicating the positions of the candidate keypoints (“3D pose and shape regressor refers to a module or process which estimates three-dimensional locations of human meshes (vertexes and surfaces), and camera parameters including 3D location of a camera and/or angles to render the 3D human meshes that align with a 2D body shape and pose that is identified in the input image. The module may be, for example, a trainable neural network model” at paragraph 0031, line 1).
Yamazaki et al. does not explicitly disclose that the plurality of candidate keypoints are candidates to replace at least some of the plurality of keypoints included in the set and replacing the at least some of the keypoints included in the set with at least some of the candidate keypoints on a basis of the reliability determined.
Wang et al. teaches a method in the same field of endeavor of human pose keypoint determination, comprising:
determining a set including a plurality of keypoints for recognizing a pose of an object (“In the top-down manner pose estimation methods, single person pose estimator aims to locate K keypoints P = {p1, p2, ..., pk} from an image I of size W × H × 3, where pk is a 2D-coordinates. Heatmap based methods transform this problem to estimating K heatmaps {H1,H2, ...,Hk} of size Wi × Hi × K, where each heatmap Hk will be decoded to the corresponding coordinates pk during the test phase” at section 3, line 1);
determining one or a plurality of candidate keypoints that are candidates to replace at least some of the plurality of keypoints included in the set
replacing the at least some of the keypoints included in the set with at least some of the candidate keypoints on a basis of the reliability determined (“During training, N guided points {s1 k, s2 k, ..., sNk } are sampled for each heatmap Hk, while the best guided points s∗ k is selected for heapmap Hk during testing. For sake of simplification, we omit the superscript in the following formula. For any guided point sk, guided feature fk = F[sk] at the corresponding location and its confidence score hk = Hk[sk] can be extracted” at page 496, second to last paragraph, line 7; “Finally, the refined classification result ck and offset regression result rk are achieved based on the refined feature gk.” at page 497, line 2).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to utilize the keypoint refinement as taught by Wang et al. with the keypoints of Yamazaki et al. “to improve the accuracy of keypoint localization” (Wang et al. at section 1, second to last paragraph, last sentence).
Regarding claim 7, Yamazaki et al. discloses a non-transitory, computer readable storage medium containing a computer program, which when executed by a computer, causes the computer to execute an information processing method by carrying out actions (“In an implementation, the apparatus 1104 may be generally described as a physical device comprising at least one processor 1106 and at least one memory 1108 including computer program code. The at least one memory 1108 and the computer program code are configured to, with the at least one processor 1106, cause the physical device to perform the operations described in FIGS. 7 and/or 10” at paragraph 0062, line 7), comprising:
determining a set including a plurality of keypoints for recognizing a pose of an object, on a basis of a three-dimensional model of the object (“In flow diagram 400, a projected keypoint is obtained for a feature of an input image 402. The projected keypoint may be obtained by applying a 3D pose and shape estimation process 404 (for example, by 3D pose and shape regression techniques as described above or any other 3D estimation techniques known in the art) on the feature of the input image 402 using to generate a 3D rendering 406, and then applying a 3D-2D keypoint projection process 408 on the 3D rendering 406 to obtain coordinates of the projected keypoint associated with the feature. The projected keypoint comprises a set of coordinates of the feature that is projected from the 3D rendering 406 of the input image 402” at paragraph 0041, line 7);
determining one or a plurality of candidate keypoints that are candidates (“Further, a direct keypoint is obtained based on a 2D rendering of a feature of the input image 402. Specifically, the direct keypoint may be obtained by applying a 2D keypoint estimation process 412 (for example, by using heatmap estimation as described in FIG. 3 or any other 2D estimation techniques known in the art) on the feature of the input image 402” at paragraph 0042, line 1);
determining reliability of each of the keypoints included in the set and the candidate keypoints from information (“Thereafter, at 416, a confidence score is computed based on the projected keypoint and the direct keypoint, wherein a higher confidence score indicates a higher accuracy of the projected and direct keypoints” at paragraph 0043, line 1) that is output when a captured image is input to a machine learning model trained and that indicates positions of the keypoints included in the set and the candidate keypoints, the machine learning model being configured to receive the captured image as an input and output information indicating the positions of the keypoints included in the set and information indicating the positions of the candidate keypoints (“3D pose and shape regressor refers to a module or process which estimates three-dimensional locations of human meshes (vertexes and surfaces), and camera parameters including 3D location of a camera and/or angles to render the 3D human meshes that align with a 2D body shape and pose that is identified in the input image. The module may be, for example, a trainable neural network model” at paragraph 0031, line 1).
Yamazaki et al. does not explicitly disclose that the plurality of candidate keypoints are candidates to replace at least some of the plurality of keypoints included in the set and replacing the at least some of the keypoints included in the set with at least some of the candidate keypoints on a basis of the reliability determined.
Wang et al. teaches a method in the same field of endeavor of human pose keypoint determination, comprising:
determining a set including a plurality of keypoints for recognizing a pose of an object (“In the top-down manner pose estimation methods, single person pose estimator aims to locate K keypoints P = {p1, p2, ..., pk} from an image I of size W × H × 3, where pk is a 2D-coordinates. Heatmap based methods transform this problem to estimating K heatmaps {H1,H2, ...,Hk} of size Wi × Hi × K, where each heatmap Hk will be decoded to the corresponding coordinates pk during the test phase” at section 3, line 1);
determining one or a plurality of candidate keypoints that are candidates to replace at least some of the plurality of keypoints included in the set
replacing the at least some of the keypoints included in the set with at least some of the candidate keypoints on a basis of the reliability determined (“During training, N guided points {s1 k, s2 k, ..., sNk } are sampled for each heatmap Hk, while the best guided points s∗ k is selected for heapmap Hk during testing. For sake of simplification, we omit the superscript in the following formula. For any guided point sk, guided feature fk = F[sk] at the corresponding location and its confidence score hk = Hk[sk] can be extracted” at page 496, second to last paragraph, line 7; “Finally, the refined classification result ck and offset regression result rk are achieved based on the refined feature gk.” at page 497, line 2).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to utilize the keypoint refinement as taught by Wang et al. with the keypoints of Yamazaki et al. “to improve the accuracy of keypoint localization” (Wang et al. at section 1, second to last paragraph, last sentence).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KATRINA R FUJITA whose telephone number is (571)270-1574. The examiner can normally be reached Monday - Friday 9:30-5:30 pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Sumati Lefkowitz can be reached at 5712723638. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KATRINA R FUJITA/ Primary Examiner, Art Unit 2672
Prosecution Insights