DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Remarks
This office action is responsive to the preliminary amendment filed on 05/10/2024.
Claim(s) 26-45 is/are pending in the application.
Claim(s) 26-45 was/were added.
Claim(s) 1-25 was/were canceled.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 26-29, 32-33 is/are rejected under 35 U.S.C. 102(a)(1)/(a)(2) as being anticipated by Nie et al. (US 2019/0251341 A1).
In regards to claim 26, Nie teaches a computer-implemented system comprising:
memory (e.g. Fig.6: memory 620; [0181]: memory 620 may mainly include a program storage area and a data storage area; data storage area may store data … created according to use of the skeleton posture determining apparatus 600, and the like) to store a plurality of video sequences of images of a plurality of perspectives of a same scene with at least one person (e.g. [0061]-[0063],Fig.3A: electronic device obtains a human body motion video; a stereoscopic camera may be disposed in the electronic device, in this case, the electronic device may photograph a human body motion using the stereoscopic camera disposed in the electronic device, to obtain a human body motion video; the motion video may include a plurality of video frames, and each video frame may include a human body posture image);
instructions (e.g. further in [0181]: memory 620 may be configured to store a software program and a module); and
at least one processor circuit (e.g. further in [0181]: processor 680 runs the software program and module stored in the memory 620, to execute various function applications and process data) to be programmed by the instructions to:
generate a three-dimensional (3D) skeleton with a joint point at respective joint locations of the skeleton based on the images (e.g. [0064]-[0067]: electronic device obtains a first skeleton posture based on a first depth image corresponding to a first video frame; the first video frame may be any video frame of the plurality of video frames included in the motion video; first skeleton posture obtained by the electronic device may correspond to a plurality of joints, and the first skeleton posture includes location information of the plurality of joints; see also [0057]: abstracts a human body posture image in a depth image; separates different regions … from the human body posture image … where each region is a region in which a human joint may be located; projects the different regions onto three orthogonal planes xy, yz, and zx in a world coordinate system; locations of centers of projected different regions are extracted on each of the orthogonal planes using a mean shift algorithm, the locations of the centers are used as locations of joints, and then the electronic device may combine the locations of the joints on the three orthogonal planes to obtain location information of joints included in a skeleton posture; Examiner’s note: shows skeleton posture may be determined from a human body posture corresponding to each frame, where the skeleton is 3D);
refine one or more distances from one or more of the joint locations to at least one other one or the joint locations of the skeleton based on a comparison of the one or more distances to a criterion (e.g. [0088]: the electronic device may remove an erroneous joint from the at least one target joint based on the description parameter of the at least one target joint in one or at least two of the following four manners, and then the electronic device may determine an unremoved target joint in the at least one target joint as the correct joint; [0103]: the electronic device determines a length of a skeleton between a third joint and a parent joint of the third joint based on location information of the third joint and location information of the parent joint of the third joint, where the third joint is any target joint, having a parent joint, of the at least one target joint, when the length of the skeleton between the third joint and the parent joint of the third joint is beyond a pre-obtained skeleton length range, the electronic device determines that the third joint is an erroneous joint, and then the electronic device may remove the erroneous joint from the at least one target joint), the criterion based on one or more datasets of measured skeletons of people (e.g. [0105]-[0106]: before the erroneous joint is removed in the second manner, the electronic device may obtain a skeleton length range corresponding to each skeleton; after obtaining a human body motion video, the electronic device recognizes a human face image included in the motion video, to obtain a user identity corresponding to the human face image, and then the electronic device obtains, from a database of a prestored skeleton length range, a skeleton length range set corresponding to the user identity; Examiner’s note: this shows one or more datasets of a plurality of people); and
modify one or more of the joint locations associated with at least one distance that does not pass the criterion (e.g. as above, [0103]: when the length of the skeleton between the third joint and the parent joint of the third joint is beyond a pre-obtained skeleton length range, the electronic device determines that the third joint is an erroneous joint, and then the electronic device may remove the erroneous joint from the at least one target joint).
In regards to claim 27, Nie teaches a system, wherein the criterion is associated with a range of acceptable joint-to-joint distances based on the one or more datasets (e.g. as above, [0103]: the electronic device determines a length of a skeleton between a third joint and a parent joint of the third joint based on location information of the third joint and location information of the parent joint of the third joint … beyond a pre-obtained skeleton length range; [0105]-[0106]: obtains, from a database of a prestored skeleton length range).
In regards to claim 28, Nie teaches a system, wherein the one or more datasets include data for multiple different joint connections on the skeleton (e.g. as above, [0105]-[0106]: obtains, from a database of a prestored skeleton length range, a skeleton length range set corresponding to the user identity; Examiner’s note: this shows one or more datasets of a plurality of people, where each skeleton length range in the set corresponds to multiple different joint connections).
In regards to claim 29, Nie teaches a system, wherein the one or more datasets include at least one of a mean distance of different joint pair connections, a maximum distance of different joint pair connections, or a minimum distance of different joint pair connections (e.g. as above, [0105]-[0106]: obtains, from a database of a prestored skeleton length range, a skeleton length range set corresponding to the user identity; Examiner’s note: this shows one or more datasets of a plurality of people, where each skeleton length range in the set corresponds to multiple different joint connections; range can be viewed as the minimum and maximum lengths (distances)).
In regards to claim 32, Nie teaches at least one non-transitory machine-readable medium comprising instructions to cause at least one processor circuit to at least:
determine joint clusters of candidate three-dimensional (3D) points based on images of a plurality of perspectives of a scene with people, the joint clusters corresponding to respective different joints on a 3D skeleton of a person in the scene (e.g. [0061]-[0063],Fig.3A: electronic device obtains a human body motion video; a stereoscopic camera may be disposed in the electronic device, in this case, the electronic device may photograph a human body motion using the stereoscopic camera disposed in the electronic device, to obtain a human body motion video; the motion video may include a plurality of video frames, and each video frame may include a human body posture image; [0064]-[0067]: electronic device obtains a first skeleton posture based on a first depth image corresponding to a first video frame; the first video frame may be any video frame of the plurality of video frames included in the motion video; first skeleton posture obtained by the electronic device may correspond to a plurality of joints, and the first skeleton posture includes location information of the plurality of joints; see also [0057]: abstracts a human body posture image in a depth image; separates different regions … from the human body posture image … where each region is a region in which a human joint may be located; projects the different regions onto three orthogonal planes xy, yz, and zx in a world coordinate system; locations of centers of projected different regions are extracted on each of the orthogonal planes using a mean shift algorithm, the locations of the centers are used as locations of joints, and then the electronic device may combine the locations of the joints on the three orthogonal planes to obtain location information of joints included in a skeleton posture; [0088]: the electronic device may remove an erroneous joint from the at least one target joint based on the description parameter of the at least one target joint in one or at least two of the following four manners, and then the electronic device may determine an unremoved target joint in the at least one target joint as the correct joint; Examiner’s note: shows correct joints are may be determined through multiple target joints (clusters); skeleton posture may be determined from a human body posture corresponding to each frame, where the skeleton is 3D; Fig.2 shows scene may comprise multiple people);
determine whether distances between pairs of candidate 3D points of two clusters of the skeleton satisfy a first criterion (e.g. [0103]: the electronic device determines a length of a skeleton between a third joint and a parent joint of the third joint based on location information of the third joint and location information of the parent joint of the third joint, where the third joint is any target joint, having a parent joint, of the at least one target joint, when the length of the skeleton between the third joint and the parent joint of the third joint is beyond a pre-obtained skeleton length range, the electronic device determines that the third joint is an erroneous joint, and then the electronic device may remove the erroneous joint from the at least one target joint), the first criterion based on measured joint distances of people (e.g. [0105]-[0106]: before the erroneous joint is removed in the second manner, the electronic device may obtain a skeleton length range corresponding to each skeleton; after obtaining a human body motion video, the electronic device recognizes a human face image included in the motion video, to obtain a user identity corresponding to the human face image, and then the electronic device obtains, from a database of a prestored skeleton length range, a skeleton length range set corresponding to the user identity; Examiner’s note: this shows one or more datasets of a plurality of people);
generate joint points of the skeleton based on the candidate 3D points that satisfy the first criterion (e.g. as above, [0088]: the electronic device may remove an erroneous joint from the at least one target joint based on the description parameter of the at least one target joint … determine an unremoved target joint in the at least one target joint as the correct joint; see also [0135]: after obtaining, through screening, the correct joint from the at least one target joint, the electronic device may perform fitting based on the location information of the correct joint to obtain the second skeleton posture); and
refine locations of the joint points of the skeleton based on a second criterion (e.g. [0089]: determines whether an angle indicated by angular information of the first joint falls within a preset angle range, and when the angle indicated by the angular information of the first joint is beyond the preset angle range, the electronic device may determine that the first joint is an erroneous joint, and then remove the first joint from the at least one target joint).
In regards to claim 33, Nie teaches a medium, wherein the first criterion and the second criterion are based on an acceptable range of distances between joints established based
on a dataset (e.g. as above, [0103]: beyond a pre-obtained skeleton length range; [0089]: determines whether an angle indicated by angular information (angular distance) of the first joint falls within a preset angle range).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 30 is/are rejected under 35 U.S.C. 103 as being unpatentable over Nie as applied to claim 26 above, and further in view of Nishimoto et al. (US 2011/0306422 A1).
In regards to claim 30, Nie teaches the system of claim 26, but does not explicitly teach the method, wherein one or more of the at least one processor circuit is to replace a joint-to-joint distance of the skeleton based on a mean distance, the mean distance based on the one or more datasets.
However, Nishimoto teaches a method, wherein one or more of the at least one processor circuit is to replace a joint-to-joint distance of the skeleton based on a mean distance, the mean distance based on the one or more datasets (e.g. [0164]: the correction section 108 may perform the correction process using distance information obtained from joint-to-joint distance information acquired over a plurality of frames; for example, the correction section 108 averages the joint-to-joint distance information acquired over a plurality of frames (dataset), and performs the correction process using the resulting distance information).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings/combination of Nie to replace distances, in the same conventional manner as taught by Nishimoto as both deal with determining skeletal information from an image. The motivation to combine the two would be that it would allow the correction of joint-to-joint distances using an averaged (mean) value.
Claim(s) 35, 37-38, 43-45 is/are rejected under 35 U.S.C. 103 as being unpatentable over Nie et al. (US 2019/0251341 A1) in view of Sali et al. (US 2011/0292036 A1).
In regards to claim 35, Nie teaches a method comprising:
obtaining joint clusters of candidate three-dimensional (3D) points based on images of video sequences of a same scene, the joint clusters corresponding respectively to different joints on a 3D skeleton (e.g. [0061]-[0063],Fig.3A: electronic device obtains a human body motion video; a stereoscopic camera may be disposed in the electronic device, in this case, the electronic device may photograph a human body motion using the stereoscopic camera disposed in the electronic device, to obtain a human body motion video; the motion video may include a plurality of video frames, and each video frame may include a human body posture image; [0064]-[0067]: electronic device obtains a first skeleton posture based on a first depth image corresponding to a first video frame; the first video frame may be any video frame of the plurality of video frames included in the motion video; first skeleton posture obtained by the electronic device may correspond to a plurality of joints, and the first skeleton posture includes location information of the plurality of joints; see also [0057]: abstracts a human body posture image in a depth image; separates different regions … from the human body posture image … where each region is a region in which a human joint may be located; projects the different regions onto three orthogonal planes xy, yz, and zx in a world coordinate system; locations of centers of projected different regions are extracted on each of the orthogonal planes using a mean shift algorithm, the locations of the centers are used as locations of joints, and then the electronic device may combine the locations of the joints on the three orthogonal planes to obtain location information of joints included in a skeleton posture; [0088]: the electronic device may remove an erroneous joint from the at least one target joint based on the description parameter of the at least one target joint in one or at least two of the following four manners, and then the electronic device may determine an unremoved target joint in the at least one target joint as the correct joint; Examiner’s note: shows correct joints are may be determined through multiple target/candidate joints (clusters); skeleton posture may be determined from a human body posture corresponding to each frame, where the skeleton is 3D; Fig.2 shows scene may comprise multiple people); and
determining whether a joint value representative of distances between pairs of the candidate 3D points of two of the joint clusters passes at least one criterion (e.g. [0103]: the electronic device determines a length of a skeleton between a third joint and a parent joint of the third joint based on location information of the third joint and location information of the parent joint of the third joint, where the third joint is any target joint, having a parent joint, of the at least one target joint, when the length of the skeleton between the third joint and the parent joint of the third joint is beyond a pre-obtained skeleton length range, the electronic device determines that the third joint is an erroneous joint, and then the electronic device may remove the erroneous joint from the at least one target joint), the at least one criterion based on a dataset, the dataset based on measured joint distances of human beings (e.g. [0105]-[0106]: before the erroneous joint is removed in the second manner, the electronic device may obtain a skeleton length range corresponding to each skeleton; after obtaining a human body motion video, the electronic device recognizes a human face image included in the motion video, to obtain a user identity corresponding to the human face image, and then the electronic device obtains, from a database of a prestored skeleton length range, a skeleton length range set corresponding to the user identity; Examiner’s note: this shows one or more datasets of a plurality of people),
but does not explicitly teach the method,
wherein the joint value is a joint confidence value.
However, Sali teaches a method,
wherein the joint value is a joint confidence value (e.g. [0013]: processing the depth map includes computing a confidence value associated with an identification of an element in the scene; [0031]: other information provided by the API regarding the skeleton may include, for example, confidence values associated with joint coordinates; confidence values can be useful in making application-level decisions under conditions of conflicting input information due to noise or other uncertainty factors).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings/combination of Nie to use confidence values, in the same conventional manner as taught by Sali as both deal with determining a skeleton of a humanoid subject in scene using a depth map. The motivation to combine the two would be that it would help making application-level decisions (see [0031]), such as correct joint locations.
In regards to claim 37, the combination of Nie and Sali teaches a method, wherein the dataset includes at least one of average distances between skeleton joints, maximum distances between skeleton joints, or minimum distances between skeleton joints (e.g. Nie as above, [0105]-[0106]: obtains, from a database of a prestored skeleton length range, a skeleton length range set corresponding to the user identity; Examiner’s note: this shows one or more datasets of a plurality of people, where each skeleton length range in the set corresponds to multiple different joint connections; range can be viewed as the minimum and maximum lengths (distances)).
In regards to claim 38, the combination of Nie and Sali teaches a method, further including performing skeleton fitting to determine joint points corresponding respectively to the joint clusters (e.g. Nie, [0135]: after obtaining, through screening, the correct joint from the at least one target joint, the electronic device may perform fitting based on the location information of the correct joint to obtain the second skeleton posture).
In regards to claim 43, the combination of Nie and Sali teaches a method, wherein the determining includes keeping one or more candidate 3D points in a cluster associated with a first joint based on a candidate 3D point in the cluster of the first joint having a distance to an established joint point that passes the at least one criterion (e.g. Nie as above, [0088]: the electronic device may remove an erroneous joint from the at least one target joint based on the description parameter of the at least one target joint …, and then the electronic device may determine an unremoved target joint in the at least one target joint as the correct joint; [0103]: the electronic device determines a length of a skeleton between a third joint and a parent joint of the third joint based on location information of the third joint and location information of the parent joint of the third joint, where the third joint is any target joint, having a parent joint, of the at least one target joint, when the length of the skeleton between the third joint and the parent joint of the third joint is beyond a pre-obtained skeleton length range, the electronic device determines that the third joint is an erroneous joint, and then the electronic device may remove the erroneous joint from the at least one target joint).
In regards to claim 44, the combination of Nie and Sali teaches a method, wherein the determining includes determining a single joint point of a cluster of candidate 3D points based on a mean-shift algorithm (e.g. Nie as above, [0057]: abstracts a human body posture image in a depth image; separates different regions … from the human body posture image … where each region is a region in which a human joint may be located; projects the different regions onto three orthogonal planes xy, yz, and zx in a world coordinate system; locations of centers of projected different regions are extracted on each of the orthogonal planes using a mean shift algorithm, the locations of the centers are used as locations of joints, and then the electronic device may combine the locations of the joints on the three orthogonal planes to obtain location information of joints included in a skeleton posture).
In regards to claim 45, the combination of Nie and Sali teaches a method, further including refining locations of joint points at respective individual joints of the skeleton based on the dataset (e.g. Nie as above, [0088]: the electronic device may remove an erroneous joint from the at least one target joint based on the description parameter of the at least one target joint …, and then the electronic device may determine an unremoved target joint in the at least one target joint as the correct joint; [0103]: the electronic device determines a length of a skeleton between a third joint and a parent joint of the third joint based on location information of the third joint and location information of the parent joint of the third joint, where the third joint is any target joint, having a parent joint, of the at least one target joint, when the length of the skeleton between the third joint and the parent joint of the third joint is beyond a pre-obtained skeleton length range, the electronic device determines that the third joint is an erroneous joint, and then the electronic device may remove the erroneous joint from the at least one target joint).
Claim(s) 36 is/are rejected under 35 U.S.C. 103 as being unpatentable over the combination of Nie and Sali as applied to claim 35 above, and further in view of Singh (US 2020/0394384 A1).
In regards to claim 36, the combination of Nie and Sali teaches the method of claim 35, but does not explicitly teach the method, wherein the dataset is based on images of at least a thousand people.
However, Singh teaches a method, wherein the dataset is based on images of at least a thousand people (e.g. [0031]: pose estimation of the individuals using a ScatterNet Hybrid Deep Learning (SHDL) Network to determine whether anomalies exist in the captured/recorded images; identifying fourteen key-points of a human body to form a skeleton structure of the detected individuals; and classifying of the estimated pose using a three dimensional (3D) ResNet, wherein the ScatterNet Hybrid Deep Learning (SHDL) Network is trained with an Aerial Violent Individual (AVI) Dataset to perform analysis of the identified key-points, where the Aerial Violent Individual (AVI) Dataset is composed of thousands of images and thousands of individuals).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings/combination of Nie and Sali to use a database, in the same conventional manner as taught by Singh as both deal with determining posture/skeleton information of a person in an image. The motivation to combine the two would be that it would help analyze identified key-points (joints) using data from thousands of individuals.
Allowable Subject Matter
Claim(s) 34, 39-42 is/are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims. To note, claims 40-42 are included as they depend on claim 39.
The following is a statement of reasons for the indication of allowable subject matter:
Claim(s) 31, 34, 39-42 was/were carefully reviewed and a search with regards to independent claim(s) 26, 32, 35 has been made. Accordingly, those claim(s) are believed to be distinct from the prior art searched.
Regarding claim(s) 31 (and specifically independent claim(s) 26), the prior art search was found to neither anticipate nor suggest the system of claim 26, wherein one or more of the at least one processor circuit is to increment a joint error indicator based on a connection to a joint not meeting the criterion (emphasis added).
Regarding claim(s) 34 (and specifically independent claim(s) 32), the prior art search was found to neither anticipate nor suggest the medium of claim 32, wherein the instructions are to cause one or more of the at least one processor circuit to increment a joint confidence value of a candidate 3D point of a first cluster based on a point of a second cluster having a distance to the candidate 3D point that satisfies the first criterion, the increment is a fraction of one over a number of points in the second cluster such that a total confidence value of the candidate 3D point of the first cluster is a proportion of the points on the second cluster that satisfy the first criterion (emphasis added).
Regarding claim(s) 39-42 and specifically independent claim(s) 35, the prior art search was found to neither anticipate nor suggest the method of claim 35, further including generating a second joint confidence value of an individual candidate 3D point in a first cluster, the second joint confidence value indicating how many points in a second cluster are a distance from the individual candidate 3D point that passes the at least one criterion, and keeping the individual candidate 3D point in the first cluster when the second joint confidence value passes at least another criterion (emphasis added).
It is viewed that any of the previously cited references or any of the prior art searched, in part or in whole, cannot be combined in such a way to render the claimed invention obvious.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JED-JUSTIN IMPERIAL whose telephone number is (571)270-5807. The examiner can normally be reached Monday to Friday, 9am - 6pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Daniel Hajnik can be reached at (571) 272-7642. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JED-JUSTIN IMPERIAL/Examiner, Art Unit 2616
/DANIEL F HAJNIK/Supervisory Patent Examiner, Art Unit 2616