DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-15, and 18-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Hazeli et al. (US-20240257392-A1) hereinafter referenced as Hazeli, in view of Ludwig et al. (Recognition of Freely Selected Keypoints on Human Limbs) hereinafter referenced as Ludwig.
Regarding claim 1, Hazeli discloses: A computer-implemented method performed by a computer system comprising a memory and at least one hardware processor (Hazeli: Figure 1b; 0030), the computer-implemented method comprising: accessing a first set of keypoints generated by processing at least one image of a body area of a person (Hazeli: 0049-0050: “The pose detection 600 and image segmentation 650 processes (e.g., included in Mediapipe Pose) infer landmarks 460 (i.e., estimated locations of joints of the user 301) and a segmentation mask 465 (i.e., the estimated human contour 370 of the user 301 ) from the RGB image frames 224. FIG. 6 A is a diagram of the landmarks 460 (identified in Table 1 below) generated by the pose detection process 600 according to exemplary embodiments. FIG. 6 B is an example image 224 with the landmarks 460.”);
generating a body mask by processing the at least one image of the body area of the person (Hazeli: 0049-0050: “The pose detection 600 and image segmentation 650 processes (e.g., included in Mediapipe Pose) infer landmarks 460 (i.e., estimated locations of joints of the user 301) and a segmentation mask 465 (i.e., the estimated human contour 370 of the user 301 ) from the RGB image frames 224…FIG. 6 C is an example segmentation mask 465 identified by the image segmentation process 650 based on the example image 224 of FIG. 6 B and the landmarks 460 identified using the pose detection process 600.”);
processing the body mask to identify a body contour of the body (Hazeli: 0049-0050: “The pose detection 600 and image segmentation 650 processes (e.g., included in Mediapipe Pose) infer landmarks 460 (i.e., estimated locations of joints of the user 301) and a segmentation mask 465 (i.e., the estimated human contour 370 of the user 301 ) from the RGB image frames 224.”;
0053: “The pose estimation process 400 is performed individually for each stream of video images 224 received from each image capture system 120 . Accordingly, using either or both of the processes 401 and 402 described above, the fall prevention system captures a two-dimensional silhouette 415 and/or segmentation mask 465 indicative of the human contour 370 of the user 301 from the point of view of the image capture system 120 providing the video images 224.”); and
tracking the first set of keypoints to generate motion tracking data (Hazeli: Abstract: “A fall prevention system that monitors the real-time pose of a user and provides alerts in response to a determination that the user may be likely to fall. To accurately determine whether the user is in an unstable pose, the fall prevention system receives video images of the user (and, in some instances, depth information) captured by multiple image capture systems from multiple angles.”;
0063: “The stability metrics may also include the base of support 380 and the geometric centerline 390 of the captured human contour 370 . In embodiments that use pose estimation 600 to capture a segmentation mask 465, the base of base of support 380 may be identified based on the landmarks 460 indicative of the toes, feet, and heels. (Additionally, when there is no contact between the feet of the user 301 and the ground, the fall detection system includes activity detection algorithms that detect contact between human body and other supporting surfaces, such as a chair, a bed, a wall, etc.) ”; Wherein for activity detection, the landmarks are used for detecting contact between the user’s body and surfaces, including the ground, chairs, beds, and walls.).
Hazeli does not disclose expressly: processing the body mask to identify a second set of keypoints corresponding to a body contour of the body;
in response to identifying the second set of keypoints, executing a predetermined function to identify a new keypoint based on the first set of keypoints and the second set of keypoints;
generating a third set of keypoints based on the first set of keypoints and the new keypoint; and
tracking the third set of keypoints to generate motion tracking data.
Ludwig discloses: a method for generating keypoints on limbs based on a first set of key points, described as fixed keypoints, and segmentation body masks (Ludwig: Figure 2: “Examples for the keypoint generation process on COCO images. The body part is visualized with a red overlay and the fixed keypoints enclosing the body part in yellow.”;
3.1. Keypoint Generation: “we use segmentation masks of upper arms, forearms, thighs and lower legs. As we want to generate keypoints that are distributed over the complete body part, we use the following generation scheme: Let bi and bj be the coordinates of two fixed keypoints (e.g., left shoulder and left elbow joints) that enclose the body part B (e.g., left upper arm).”).
Wherein the method comprises: processing the body mask to identify a second set of keypoints corresponding to a body contour of the body; in response to identifying the second set of keypoints, executing a predetermined function to identify a new keypoint based on the first set of keypoints and the second set of keypoints (Ludwig: Figure 2: “The randomly selected projection point on the line between the fixed keypoints is displayed in green and the orthogonal line in blue. The intersection points of the line with the edge of the body part are visualized in blue, while the red points visualize the final generated keypoints.”;
3.1. Keypoint Generation: “At first, we uniformly sample a percentage pb of the line between bi and bj, which results in the projection point bp…Next, we generate the line f that is orthogonal to the line between bi and bj and fits through bp. This line has two intersection points c1 and c2 with the boundary of the body part segmentation mask B. Then, we sample ˜pt from a normal distribution and define pt = max(0,1 − |˜pt|) ∈ [0, 1]. This ratio pt corresponds to the distance from the projection point bp to the body part boundary, referred to as the thickness. With pt, we create the final keypoint bt as follows:
PNG
media_image1.png
85
572
media_image1.png
Greyscale
˜pt is drawn from a normal distribution in order to generate more keypoints on the body part boundaries”; Wherein the final keypoint bt comprises keypoints from the body contour.; Wherein the second set of keypoints are identified to be points on the mask boundary.);
generating a third set of keypoints based on the first set of keypoints and the new keypoint; and tracking the third set of keypoints to generate motion tracking data (Ludwig: 5. Conclusion: “This paper proposes two representations for freely selectable keypoints on the limbs of humans. The first approach, called vectorized keypoints, represents each key point as a combination of the projection point encoded in a keypoint vector and the thickness encoded in a thickness vector. The projection point is the point on the line between the two fixed keypoints that enclose the body part, while the thickness indicates the distance of the desired keypoint from the projection point to the body part boundary.”).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to implement the keypoint generation method taught by Ludwig using the landmarks disclosed by Hazeli. The suggestion/motivation for doing so would have been “We propose and evaluate two different key point representations. The first approach splits the representation into two parts. One part encodes the position of the projection of the desired keypoint onto the straight line between the fixed keypoints that enclose the corresponding body part. The second part encodes the distance of the key point from this projection point relative to the distance of the boundary of the body part. We refer to this approach as the vectorized keypoint approach…Both approaches open the possibility to design the keypoint representation such that desired arbitrary points on the limbs can be represented and therefore also detected by our model without any additional annotations or postprocessing steps” (Ludwig: 1. Introduction). Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Hazeli with Ludwig to obtain the invention as specified in claim 1.
Regarding claim 2, Hazeli in view of Ludwig discloses: The method of claim 1, further comprising generating the first set of keypoints by processing the at least one image of the body area of the person via a pose estimation model (Hazeli: 0049-0050: “In some embodiments, the fall detection system may estimate the human contour 370 of the user 301 using pose detection 600 and image segmentation 650. The pose detection 600 and image segmentation 650 processes may be performed, for example, using a pre-trained machine learning model for human pose estimation (for example, algorithms included in Mediapipe Pose, which are rapidly deployable python API applications from the TensorFlow-based Mediapipe Open Source Project). The pose detection 600 and image segmentation 650 processes (e.g., included in Mediapipe Pose) infer landmarks 460 (i.e., estimated locations of joints of the user 301 ) and a segmentation mask 465 (i.e., the estimated human contour 370 of the user 301 ) from the RGB image frames 224. FIG. 6 A is a diagram of the landmarks 460 (identified in Table 1 below) generated by the pose detection process 600 according to exemplary embodiments.”).
Regarding claim 3, Hazeli in view of Ludwig discloses: The method of claim 1, wherein the first set of keypoints comprises at least one of a joint landmark or a head region landmark (Hazeli: Figure 6A:
PNG
media_image2.png
368
265
media_image2.png
Greyscale
;
0050: “FIG. 6 A is a diagram of the landmarks 460 (identified in Table 1 below) generated by the pose detection process 600 according to exemplary embodiments.”).
Regarding claim 4, Hazeli in view of Ludwig discloses: The method of claim 1, wherein the body mask corresponds to a segmentation mask generated by processing the at least one image of the body area of the person via a segmentation model (Ludwig: 4.2.Triple and Long Jump: “The dataset does not contain body part segmentation masks. Therefore, we use the DensePose [17] model with a ResNet101 [8] backbone and DeepLabV3[4] as well as PanopticFPN [10] heads from detectron2 [17] to generate them.”).
Regarding claim 5, Hazeli in view of Ludwig discloses: The method of claim 1, wherein: the body mask corresponds to a segmentation mask comprising pixels with corresponding numerical values associated with the body contour; and identifying the second set of keypoints corresponding to the body contour comprises determining a subset of the pixels whose corresponding numerical values are determined to fall within a predefined range (Ludwig: 3.1. Keypoint Generation: “we generate the line f that is orthogonal to the line between bi and bj and fits through bp. This line has two intersection points c1 and c2 with the boundary of the body part segmentation mask B.”; Wherein the predefined range corresponds to a boundary of the mask.).
Regarding claim 6, Hazeli in view of Ludwig discloses: The method of claim 1, wherein the identifying of the new keypoint further comprises: determining an area based on the first set of keypoints (Ludwig: 3.1. Keypoint Generation: “As we want to generate keypoints that are distributed over the complete body part, we use the following generation scheme: Let bi and bj be the coordinates of two fixed keypoints (e.g., left shoulder and left elbow joints) that enclose the body part B (e.g., left upper arm).”); generating, using the first set of keypoints, a set of intermediate points in the area (Ludwig: 3.1. Keypoint Generation: “At first, we uniformly sample a percentage pb of the line between bi and bj, which results in the projection point bp”); computing for each keypoint of the second set of keypoints a value of a predetermined measure based on the respective keypoint and the set of intermediate points (Ludwig: 3.1. Keypoint Generation: “Next, we generate the line f that is orthogonal to the line between bi and bj and fits through bp. This line has two intersection points c1 and c2 with the boundary of the body part segmentation mask B.”); and selecting the new keypoint to be a keypoint of the second set of keypoints whose associated value optimizes the predetermined measure with respect to a predetermined criterion (Ludwig: 3.1. Keypoint Generation: “Then, we sample ˜pt from a normal distribution and define pt = max(0,1 − |˜pt|) ∈ [0, 1]. This ratio pt corresponds to the distance from the projection point bp to the body part boundary, referred to as the thickness. With pt, we create the final keypoint bt as follows:
PNG
media_image1.png
85
572
media_image1.png
Greyscale
˜pt is drawn from a normal distribution in order to generate more keypoints on the body part boundaries”; Wherein the final keypoint bt comprises keypoints from the body contour.).
Regarding claim 7, Hazeli in view of Ludwig discloses: The method of claim 6, wherein: generating the set of intermediate points further comprises generating a segment based on a plurality of landmarks retrieved from the first set of keypoints (Ludwig: 3.1. Keypoint Generation: “As we want to generate keypoints that are distributed over the complete body part, we use the following generation scheme: Let bi and bj be the coordinates of two fixed keypoints (e.g., left shoulder and left elbow joints) that enclose the body part B (e.g., left upper arm). At first, we uniformly sample a percentage pb of the line between bi and bj, which results in the projection point bp:”); and the predetermined measure is a distance measure based on each keypoint of the second set of keypoints and one or more points on the generated segment or an extension of the generated segment (Ludwig: 3.1. Keypoint Generation: “Next, we generate the line f that is orthogonal to the line between bi and bj and fits through bp. This line has two intersection points c1 and c2 with the boundary of the body part segmentation mask B . Then, we sample ˜pt from a normal distribution and define pt = max(0,1 − |˜pt|) ∈ [0, 1]. This ratio pt corresponds to the distance from the projection point bp to the body part boundary, referred to as the thickness. With pt, we create the final keypoint bt as follows:
PNG
media_image1.png
85
572
media_image1.png
Greyscale
˜pt is drawn from a normal distribution in order to generate more keypoints on the body part boundaries”; Wherein the generation of the final keypoint on the orthogonal line constitutes a distance measure based on each keypoint of the second set of keypoints and one or more points on the generated segment).
Regarding claim 8, Hazeli in view of Ludwig discloses: The method of claim 7, wherein computing, for each keypoint of the second set of keypoints, the value of the predetermined measure further comprises: generating a reference vector based on the segment (Ludwig: 3.1. Keypoint Generation: “Let bi and bj be the coordinates of two fixed keypoints (e.g., left shoulder and left elbow joints) that enclose the body part B (e.g., left upper arm).”; Wherein the line segment comprises vectors between the 2 points.); generating a candidate vector based on the keypoint and a keypoint of the segment; and computing an angle associated with the keypoint based on the candidate vector and the reference vector (Ludwig: 3.1. Keypoint Generation: “we uniformly sample a percentage pb of the line between bi and bj, which results in the projection point bp…we generate the line f that is orthogonal to the line between bi and bj and fits through bp. This line has two intersection points c1 and c2 with the boundary of the body part segmentation mask B.”).
Regarding claim 9, Hazeli in view of Ludwig discloses: The method of claim 8, wherein selecting the new keypoint further comprises selecting a keypoint of the second set of keypoints associated with an angle of a set of computed angles associated with the second set of keypoints, wherein the angle satisfies a predefined selection criterion (Ludwig: 3.1. Keypoint Generation: “Then, we sample ˜pt from a normal distribution and define pt = max(0,1 − |˜pt|) ∈ [0, 1]. This ratio pt corresponds to the distance from the projection point bp to the body part boundary, referred to as the thickness. With pt, we create the final keypoint bt as follows:
PNG
media_image1.png
85
572
media_image1.png
Greyscale
˜pt is drawn from a normal distribution in order to generate more keypoints on the body part boundaries”; Wherein the final keypoint bt comprises keypoints from the body contour that are orthogonal, 90 degrees, to the line segment.).
Regarding claim 10, Hazeli in view of Ludwig discloses: The method of claim 1, wherein the identifying of the new keypoint further comprises: generating a first segment based on at least a first landmark of the first set of keypoints and one of at least a first contour point of the second set of keypoints or a first coordinate axis; generating a second segment based on at least a second landmark of the first set of keypoints and one of at least a second contour point of the second set of keypoints or a second coordinate axis (Ludwig: Figure 2: “Examples for the keypoint generation process on COCO images. The body part is visualized with a red overlay and the fixed keypoints enclosing the body part in yellow.”;
Figure 3:
PNG
media_image3.png
402
635
media_image3.png
Greyscale
;
3.1. Keypoint Generation: “we use segmentation masks of upper arms, forearms, thighs and lower legs. As we want to generate keypoints that are distributed over the complete body part, we use the following generation scheme: Let bi and bj be the coordinates of two fixed keypoints (e.g., left shoulder and left elbow joints) that enclose the body part B (e.g., left upper arm).”;
As more explicitly displayed in Figure 3, the keypoints define segments, using mask boundaries, for enclosing body parts.); and selecting the new keypoint to correspond to a determined intersection of the first segment and the second segment (Ludwig: 3.1. Keypoint Generation: “Then, we sample ˜pt from a normal distribution and define pt = max(0,1 − |˜pt|) ∈ [0, 1]. This ratio pt corresponds to the distance from the projection point bp to the body part boundary, referred to as the thickness. With pt, we create the final keypoint bt as follows:
PNG
media_image1.png
85
572
media_image1.png
Greyscale
˜pt is drawn from a normal distribution in order to generate more keypoints on the body part boundaries”; Wherein the final keypoint bt comprises keypoints on the orthogonal/intersecting lines, including a keypoint to correspond to a determined intersection.).
Regarding claim 11, Hazeli in view of Ludwig discloses: The method of claim 1, wherein the identifying of the new keypoint further comprises: computing for each keypoint of the second set of keypoints a value of a predetermined measure based on the respective keypoint and the first set of keypoints (Ludwig: 3.1. Keypoint Generation: “Next, we generate the line f that is orthogonal to the line between bi and bj and fits through bp. This line has two intersection points c1 and c2 with the boundary of the body part segmentation mask B.”; Wherein an angle based on each point on the contour and point bp is measured.); and selecting the new keypoint to be a keypoint of the second set of keypoints whose associated value optimizes the predetermined measure with respect to a predetermined criterion (Ludwig: 3.1. Keypoint Generation: “Then, we sample ˜pt from a normal distribution and define pt = max(0,1 − |˜pt|) ∈ [0, 1]. This ratio pt corresponds to the distance from the projection point bp to the body part boundary, referred to as the thickness. With pt, we create the final keypoint bt as follows:
PNG
media_image1.png
85
572
media_image1.png
Greyscale
˜pt is drawn from a normal distribution in order to generate more keypoints on the body part boundaries”; Wherein the final keypoint bt comprises keypoints from the body contour that are orthogonal, 90 degrees, to the line segment at point bp.).
Regarding claim 12, Hazeli in view of Ludwig discloses: The method of claim 1, wherein generating the third set of keypoints based on the first set of keypoints and the new keypoint comprises at least one of: augmenting the first set of keypoints using the new keypoint; or replacing one of the keypoints of the first set of keypoints with the new keypoint (Ludwig: 1. Introduction: “We propose and evaluate two different key point representations. The first approach splits the representation into two parts. One part encodes the position of the projection of the desired keypoint onto the straight line between the fixed keypoints that enclose the corresponding body part. The second part encodes the distance of the key point from this projection point relative to the distance of the boundary of the body part. We refer to this approach as the vectorized keypoint approach…Both approaches open the possibility to design the keypoint representation such that desired arbitrary points on the limbs can be represented and therefore also detected by our model without any additional annotations or postprocessing steps”;
5. Conclusion: “This paper proposes two representations for freely selectable keypoints on the limbs of humans. The first approach, called vectorized keypoints, represents each key point as a combination of the projection point encoded in a keypoint vector and the thickness encoded in a thickness vector. The projection point is the point on the line between the two fixed keypoints that enclose the body part, while the thickness indicates the distance of the desired keypoint from the projection point to the body part boundary.”).
Regarding claim 13, Hazeli in view of Ludwig discloses: The method of claim 5, wherein the numerical values associated with the body contour correspond to at least one of probabilities, grayscale range values, or RGB scale values (Hazeli: 0049: “In some embodiments, the fall detection system may estimate the human contour 370 of the user 301 using pose detection 600 and image segmentation 650. The pose detection 600 and image segmentation 650 processes may be performed, for example, using a pre-trained machine learning model for human pose estimation (for example, algorithms included in Mediapipe Pose, which are rapidly deployable python API applications from the TensorFlow-based Mediapipe Open Source Project). The pose detection 600 and image segmentation 650 processes (e.g., included in Mediapipe Pose) infer landmarks 460 (i.e., estimated locations of joints of the user 301 ) and a segmentation mask 465 (i.e., the estimated human contour 370 of the user 301 ) from the RGB image frames 224.”).
Regarding claim 14, Hazeli in view of Ludwig discloses: The method of claim 1, further comprising capturing the at least one image via a camera (Hazeli: 0031: “Each image capture device 120 includes a camera 124 that capture two-dimensional video images of the environment 101 of the user.”).
Regarding claim 15, Hazeli in view of Ludwig discloses: The method of claim 1, wherein the new keypoint is automatically identified based on the first set of keypoints, the second set of keypoints (Ludwig: “As we want to generate keypoints that are distributed over the complete body part, we use the following generation scheme: Let bi and bj be the coordinates of two fixed keypoints (e.g., left shoulder and left elbow joints) that enclose the body part B (e.g., left upper arm). At first, we uniformly sample a percentage pb of the line between bi and bj, which results in the projection point bp…Next, we generate the line f that is orthogonal to the line between bi and bj and fits through bp. This line has two intersection points c1 and c2 with the boundary of the body part segmentation mask B.”), and a physical activity to be performed by the person (Hazeli: Abstract: “A fall prevention system that monitors the real-time pose of a user and provides alerts in response to a determination that the user may be likely to fall. To accurately determine whether the user is in an unstable pose, the fall prevention system receives video images of the user (and, in some instances, depth information) captured by multiple image capture systems from multiple angles.”;
0063: “In embodiments that use pose estimation 600 to capture a segmentation mask 465 , the base of base of support 380 may be identified based on the landmarks 460 indicative of the toes, feet, and heels. (Additionally, when there is no contact between the feet of the user 301 and the ground, the fall detection system includes activity detection algorithms that detect contact between human body and other supporting surfaces, such as a chair, a bed, a wall, etc.)”;
Wherein the keypoints are generated as a result of the pose/activity monitoring system running.), the method further comprising: capturing additional images of body areas of the person (Hazeli: Abstract: “A fall prevention system that monitors the real-time pose of a user and provides alerts in response to a determination that the user may be likely to fall. To accurately determine whether the user is in an unstable pose, the fall prevention system receives video images of the user (and, in some instances, depth information) captured by multiple image capture systems from multiple angles.”); and tracking the third set of keypoints across the additional images while the person performs the physical activity (Hazeli: 0063: “In embodiments that use pose estimation 600 to capture a segmentation mask 465 , the base of base of support 380 may be identified based on the landmarks 460 indicative of the toes, feet, and heels. (Additionally, when there is no contact between the feet of the user 301 and the ground, the fall detection system includes activity detection algorithms that detect contact between human body and other supporting surfaces, such as a chair, a bed, a wall, etc.)”; Wherein determined/generated keypoints are tracked in subsequent frames.).
Regarding claim 18, Hazeli in view of Ludwig discloses: The method of claim 1, wherein each keypoint in the first set of keypoints and the second set of keypoints is associated with X-axis, Y-axis and Z-axis coordinates (Hazeli: 0054: “In embodiments where the image capture system 120 also captures depth information 226 , a depth incorporation process 470 may be performed to incorporate the captured depth information 226 into the human contour 370 of the user 301 from the point of view of that image capture system 120 . For example, the captured human contour 370 may include both the captured two-dimensional silhouette 415 and/or segmentation mask 465 and the depth of each pixel of the captured two-dimensional silhouette 415 and/or segmentation mask 465.”).
As per claim(s) 19, arguments made in rejecting claim(s) 1 are analogous.
As per claim(s) 20, arguments made in rejecting claim(s) 1 are analogous. In addition, paragraph 0030 of Hazeli discloses the claim 20 limitation “At least one non-transitory computer-readable storage medium, the at least one computer-readable storage medium including instructions”.
Claim(s) 16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Hazeli in view of Ludwig, and further in view of Guo et al (GaitContour: Efficient Gait Recognition based on a Contour-Pose Representation), hereinafter referenced as Guo.
Regarding claim 16, Hazeli in view of Ludwig discloses: The method of claim 1, wherein: executing the predetermined function further comprises identifying a plurality of new keypoints; and
generating the third set of keypoints is further based on the plurality of new keypoints (Ludwig: Figure 2: “Examples for the keypoint generation process on COCO images. The body part is visualized with a red overlay and the fixed keypoints enclosing the body part in yellow. The randomly selected projection point on the line between the fixed keypoints is displayed in green and the orthogonal line in blue. The intersection points of the line with the edge of the body part are visualized in blue, while the red points visualize the final generated keypoints.”;
3.1. Keypoint Generation: “we use segmentation masks of upper arms, forearms, thighs and lower legs. As we want to generate keypoints that are distributed over the complete body part”; Wherein the generated keypoints are added to the third set for motion detection).
Hazeli in view of Ludwig does not disclose expressly: wherein: executing the predetermined function further comprises identifying a plurality of new keypoints, the new keypoints being selected to correspond to at least a majority of the second set of keypoints.
Guo discloses: A method for performing gait recognition of a subject based on contour keypoints and pose keypoints (Guo: Abstract). Wherein the method comprises: identifying a plurality of new keypoints, the new keypoints being selected to correspond to at least a majority of a second set of keypoints (Guo: Figure 2: “The construction of Contour-Pose. The pose is combined with the contour points sampled from the silhouette edge. In particular, contour points are sampled based on their distances from neighborhood poses. As shown in the zoomed area, Contour-Pose is the ten nearby contour points from each pose with bidirectional connections.”;
1. Introduction: “we propose a novel gait representation, called Contour-Pose, as illustrated in Fig. 1. We argue that the key features expressed by a human silhouette can be compactly represented by a series of contour points around the silhouette, e.g., as approximated by Teh-Chin algorithm [40]; however, naively using contour points cannot achieve good gait recognition, as the correspondence and ordering between frames are not consistent. To circumvent this issue, we use pose keypoints as the anchors to select relevant contour points”; Wherein the contour points, selected based on the pose keypoints, correspond to new keypoints being selected to correspond to at least a majority of a second set of keypoints).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to substitute the keypoint generation algorithm based on thickness disclosed by Hazeli in view of Ludwig with the Contour-Pose keypoint selection method taught by Guo. The suggestion/motivation for doing so would have been “Silhouettes preserve the maximum amount of gait information as a series of 2D binary masks; as such, they require significant computation to process…Pose keypoints are usually two orders of magnitude smaller than silhouettes in input size; such sparsity allows keypoint-based methods to be significantly more efficient than their silhouette-based counterparts. …We propose a novel gait representation, called Contour Pose, which augments pose keypoints with contour points extracted from silhouettes; this representation contains rich information, is compact in size, and can directly improve current point-based gait recognition methods” (Guo; Wherein the selection of pose and contour keypoints makes use of each method’s strengths.). Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Hazeli in view of Ludwig with Guo to obtain the invention as specified in claim 16.
Claim(s) 17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Hazeli in view of Ludwig, and further in view of Namboodiri (US-20210001172-A1).
Regarding claim 17, Hazeli in view of Ludwig discloses: The method of claim 1, further comprising: generating feedback for the person based on the motion tracking data; and presenting the generated feedback in real-time to the person (Hazeli: 0006: “a fall prevention system is disclosed that monitors the real-time pose of a user and provides alerts in response to a determination that the user may be likely to fall.”).
Hazeli in view of Ludwig does not disclose expressly: presenting, at a UI, the generated feedback in real-time to the person.
Namboodiri discloses: A method for counting repetitions, and providing form guidance, of exercises performed by a user (Namboodiri: Abstract). Wherein the method comprises: generating feedback for the person based on motion tracking data; and presenting, at a UI, the generated feedback in real-time to the person (Namboodiri: 0163: “The display screen 1104 , via the graphical user interface (GUI) 1104 a, displays the number of repetitions of the exercises performed or calorie burnt, time elapsed during an exercise, speed of performing exercises, halt times, etc…The exercise counting repetition and form guidance system 1102 provides the GUI 1104 a on the display screen 1104. The GUI 1104 a is, for example, an online web interface, a web based downloadable application interface, a mobile based downloadable application interface, etc. The display screen 1104 displays the GUI 1104 a. The input devices 1120 are used for inputting data into the exercise counting repetition and form guidance system 1102 . The input commands, are for example, selection or navigation through user interfaces, selection of specific exercise that the user intends to perform, etc. The input devices 1120 are, for example, a keyboard such as an alphanumeric keyboard, a microphone, a joystick, a pointing device such as a computer mouse, a touch pad, a light pen, a physical button, a touch sensitive display device, a track ball, a pointing stick, any device capable of sensing a tactile input, etc.”;
0172-0173: “The outputs of the execution of the modules…of the exercise counting repetition and form guidance system 1102 are displayed to a user of the exercise counting repetition and form guidance system 1102 on the output device 1121…The non-transitory computer readable storage medium having embodied thereon, computer program codes comprising instructions executable by at least one processor 1103 for automatically counting multiple repetitions and providing form guidance of an exercise performed by the user in real time.”).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to implement the GUI taught by Namboodiri as a form of feedback/alert for the fall prevention system disclosed by Hazeli in view of Ludwig. The suggestion/motivation for doing so would have been “The display screen 1104 , via the graphical user interface (GUI) 1104 a, displays the number of repetitions of the exercises performed or calorie burnt, time elapsed during an exercise, speed of performing exercises, halt times, etc…The exercise counting repetition and form guidance system 1102 provides the GUI 1104 a on the display screen 1104. The GUI 1104 a is, for example, an online web interface, a web based downloadable application interface, a mobile based downloadable application interface, etc. The display screen 1104 displays the GUI 1104 a. The input devices 1120 are used for inputting data into the exercise counting repetition and form guidance system 1102 . The input commands, are for example, selection or navigation through user interfaces, selection of specific exercise that the user intends to perform, etc” (Namboodiri: 0163; Wherein the user is able to instantly view current information and interact with the GUI to control information displayed). Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Hazeli in view of Ludwig with Namboodiri to obtain the invention as specified in claim 17.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ANTHONY J RODRIGUEZ whose telephone number is (703)756-5821. The examiner can normally be reached Monday-Friday 10am-7pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Sumati Lefkowitz can be reached at (571) 272-3638. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ANTHONY J RODRIGUEZ/Examiner, Art Unit 2672
/SUMATI LEFKOWITZ/Supervisory Patent Examiner, Art Unit 2672
Prosecution Insights