DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The method of claim 1 is directed to a process, which is one of the statutory categories of invention, and passes Step 1: Statutory Category- MPEP § 2106.03. However, the following elements of Claim 1 recite steps that can be performed in the human mind or with pen and paper, therefore failing Step 2A Prong One. These steps constitute mental processes because they describe acts of observation, evaluation, and judgement that a human can perform mentally. For example, a human can watch each frame and identify posture (e.g., lying, sitting, crawling), watch a video and recognize when posture is changing, determine a posture before movement starts and after movement ends, assign a label to a posture, and infer an infant action based on the posture labels.
determining, using a posture classification model, posture classification data representing a posture prediction for each frame of the plurality of frames;
determining a first subset of the plurality of frames representing a transition segment between two stable posture segments, wherein the transition segment includes a first frame in time and a last frame in time;
determining, based on the posture classification data and the first frame in time of the transition segment, a second subset of the plurality of frames representing a start posture segment;
determining, based on the posture classification data and the last frame in time of the transition segment, a third subset of the plurality of frames representing an end posture segment;
determining a start posture label for the start posture segment;
determining an end posture label for the end posture segment; and
determining, based on the start posture label and the end posture label, an infant action label for the video segment.
Claim 1 fails Step 2A Prong Two because the additional elements beyond the judicial exception,
including a posture classification model, do not integrate the judicial exception into a practical application. Receiving a video segment of a video recording that captures movement of an infant is insignificant extra-solution activity (MPEP § 2106.05(g)), and the posture classification model does not improve the functioning of a computer or any other technology or technical field (MPEP § 2106.05(a)) as it merely applies the abstract idea on a computer (MPEP § 2106.05(f)). The claim also does not impose meaningful limits on the computer components such that the method is tied to a particular machine; the posture classification model may operate on any generic computing system (MPEP § 2106.05(b)).
Claim 1 also fails Step 2B, as these generic elements are well-understood, routine, and conventional (WURC), adding nothing significantly more than the abstract idea itself (MPEP § 2106.07(a)(III)). Receiving a video segment of a video recording that captures movement of an infant is WURC (see MPEP § 2106.05(d)), and the posture classification model is also WURC (see Abstract and Introduction section of Caliskan, “Detecting human activity types from 3D posture data using deep learning models”). Therefore, claim 1 is rejected. Claim 10 recites this identical ineligible subject matter, with the only additional elements beyond the judicial exception being a processor and memory. The processor and memory do not integrate the judicial exception into a practical application (see claim 1 analysis above) and are WURC (see MPEP § 2106.05(d)). Therefore, claim 10 is rejected.
Regarding Claim 19, the method is directed to a process, which is one of the statutory categories of invention, and passes Step 1: Statutory Category- MPEP § 2106.03. However, the following elements of Claim 19 recite steps that can be performed in the human mind or with pen and paper, therefore failing Step 2A Prong One. These steps constitute mental processes because they describe acts of observation, evaluation, and judgement that a human can perform mentally. For example, a human can watch a video and determine an action that the infant is performing, look at a frame and focus on where the infant is, observe body posture and interpret limb positions, identify joints, infer infant action from posture, and assign a label to a plurality of video recordings.
determining an infant action label for each video recording of the plurality of video recordings, wherein determining the infant action label for a video recording further comprises:
determining a region of interest for each frame of the video recording, wherein the region of interest corresponds to detection of an infant;
determining, using the region of interest for each frame, a skeletal pose;
determining, using the skeletal pose, a set of skeleton keypoints corresponding to an adult skeleton;
and determining, using an action recognition model with the set of skeleton keypoints as input, the infant action label;
labeling each video of the plurality of video recordings with the infant action label corresponding to the video recording; and
Claim 19 fails Step 2A Prong Two because the additional elements beyond the judicial exception,
including an action recognition model, do not integrate the judicial exception into a practical application. Receiving a plurality of video recordings that capture actions of human infants and storing the plurality of videos labeled with the infant action label in a database are insignificant extra-solution activity (MPEP § 2106.05(g)), and the action recognition model does not improve the functioning of a computer or any other technology or technical field (MPEP § 2106.05(a)) as it merely applies the abstract idea on a computer (MPEP § 2106.05(f)). The claim also does not impose meaningful limits on the computer components such that the method is tied to a particular machine; the action recognition model may operate on any generic computing system (MPEP § 2106.05(b)).
Claim 19 also fails Step 2B, as these generic elements are well-understood, routine, and conventional (WURC), adding nothing significantly more than the abstract idea itself (MPEP § 2106.07(a)(III)). Receiving a plurality of video recordings that capture actions of human infants and storing the plurality of videos labeled with the infant action label in a database are WURC (see MPEP § 2106.05(d)), and the action recognition model is also WURC (see Abstract and Introduction section of Zhang et. al, “A Comprehensive Survey of Vision-Based Human Action Recognition Methods”). Therefore, claim 19 is rejected.
Claims 2 and 3 fail Step 2A Prong One as they further recite mental processes as well as mathematical concepts. For example, a human can estimate confidence (claim 2), identify uncertain frames and decide that they are unclear/ambiguous (claim 2), match two groups based on criteria (claim 2), and observe multiple frames and choose the most frequent posture (claim 3). Furthermore, claims 2 and 3 recite Mathematical Concepts, which are defined as mathematical relationships, mathematical formulas or equations, or mathematical calculations. The claim must recite (i.e. set forth or describe) a mathematical concept rather than include limitations that are merely based on math.
These claims, alongside claims 4-9 and 20, fail Step 2A Prong Two because the additional elements beyond the judicial exception, including a pose estimation model, transition segmentor model, recurrent neural network, graph convolutional network, and three-dimensional convolutional network do not integrate the judicial exception into a practical application (see claim 1 analysis above). These claims also fail Step 2B as they are WURC. For reference, see Abstract and Introduction sections of Dubey and Dixit, “A comprehensive survey on human pose estimation approaches,” Introduction section of Yan et. al, “Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition,” Introduction section of Du et. al, “Hierarchical Recurrent Neural Network for Skeleton Based Action Recognition,” and Related Works section of Duan et. al, “Revisiting Skeleton-based Action Recognition.” Therefore, claims 2-9 and 20 are rejected. Claims 11-18 recite this identical ineligible subject matter and are also rejected.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1, 4, 10, and 13 are rejected under 35 U.S.C. 102 (a)(1) as being anticipated by Zhu and Wang (CN116092187A).
Regarding Claim 1, Zhu and Wang teach a computer-implemented method for recognizing an infant action in a video recording, comprising:
Paragraph [0003]: “The invention aims to provide an infant dangerous action detection and identification method based on pose estimation and YOLOX _tiny improvement, so as to solve the problems in the prior art.”
receiving a video segment of said video recording that captures movement of an infant, wherein the video segment includes a plurality of frames;
Paragraph [0006]: “acquiring a video frame from a camera, and preprocessing the input video frame.”
Paragraph [0007]: “Inputting the preprocessed video frames into a pose estimation algorithm to obtain the number of skeleton points in each frame of picture…”
determining, using a posture classification model, posture classification data representing a posture prediction for each frame of the plurality of frames;
Paragraph [0007]: “Inputting the preprocessed video frames into a pose estimation algorithm to obtain the number of skeleton points in each frame of picture, and inputting the number of skeleton points into a YOLOX _tiny target detection algorithm which is improved based on model pruning and fusion space attention mechanisms to obtain an interested target area and category detection result data of each frame of picture, wherein the target detection algorithm is obtained by training a self-built dataset and a transfer learning mode.”
Paragraph [0008]: “The pose estimation algorithm uses a lightweight openpose pose estimation algorithm to obtain the key bone points of the infant body in each frame.”
Explanation: YOLOX _tiny target detection algorithm is the classification model, and “category detection result data” is the posture prediction per frame. Pose estimation provides per-frame posture-related data.
determining a first subset of the plurality of frames representing a transition segment between two stable posture segments, wherein the transition segment includes a first frame in time and a last frame in time;
Paragraph [0016]: “Detecting the number K i of bone points of each frame to a list B by using a pose estimation algorithm.”
Paragraph [0019]: “For list B, C, D, E, when the length is greater than N, the first element in list B, C, D, E is deleted, the list element is updated, and the iteration is performed, always keeping the list length N.’
Explanation: The lists of sequential frames is the temporal segmentation, where transitions are inferred via changes in frame data. The sliding window of frames (length N) defines the first frame and last frame.
determining, based on the posture classification data and the first frame in time of the transition segment, a second subset of the plurality of frames representing a start posture segment;
Paragraph [0028]: “judging 5, when the first element in the list E is 0 and the last element is 1, judging that the baby kicks the quilt”
Explanation: “First element” is the start of the segment. Identifying posture condition is the start posture segment.
determining, based on the posture classification data and the last frame in time of the transition segment, a third subset of the plurality of frames representing an end posture segment;
Paragraph [0028]: “judging 5, when the first element in the list E is 0 and the last element is 1, judging that the baby kicks the quilt”
Explanation: “Last element” is the end of the segment which corresponds to end posture.
determining a start posture label for the start posture segment;
Paragraph [0014]: “initializing an infant dangerous action recognition algorithm by utilizing the previous N frames of video data so as to judge whether an infant is in a sleeping state or a non-sleeping state of a covered quilt at present”
Explanation: “Sleeping/not sleeping” = posture label
determining an end posture label for the end posture segment; and
Paragraph [0017]: “detecting each frame of interested target by utilizing an improved target detection algorithm, sequentially storing the detection result of each frame into a list D, adding 1 into the list C when the frame detection result comprises climbing, otherwise adding 0, and similarly, adding 1 into the list E when the frame detection result comprises the lower body in a lying state, otherwise adding 0”
Explanation: Labels like climbing and lying correspond to posture labels.
determining, based on the start posture label and the end posture label, an infant action label for the video segment.
Paragraph [0020]: “The process of processing the fused data and designing rules to judge the normal and dangerous action states of the infants and give an alarm comprises the following steps.”
Paragraph [0024]: “Judging 2, when the occurrence times of 1 in the list C are larger than N/2, judging that the baby climbs”
Paragraph [0025]: “Judgment 3, except judgment 1 and judgment 2, is regarded as a normal state”
Paragraph [0028]: “judging 5, when the first element in the list E is 0 and the last element is 1, judging that the baby kicks the quilt”
Paragraph [0047]: “A3, processing and fusing the obtained data to improve the accurate recognition capability of different dangerous actions.”
Explanation: Final classification (e.g., climbing, kicking, normal) = action label
Regarding Claim 4, Zhu and Wang teach the computer-implemented method of claim 1, further comprising:
prior to determining the posture classification, determining, using a pose estimation model, pose estimation data representing a human skeleton pose for each frame of the plurality of frames, wherein the human skeleton pose is based on joint locations and joint angles of the infant; and
Paragraph [0008]: “The pose estimation algorithm uses a lightweight openpose pose estimation algorithm to obtain the key bone points of the infant body in each frame.”
Explanation: “Key bone points” is the skeleton joints, and OpenPose is a standard skeletal pose model. Bone points represent joint locations and skeletal structure (angles).
providing the pose estimation data as input to the posture classification model.
Paragraph [0007]: “Inputting the preprocessed video frames into a pose estimation algorithm to obtain the number of skeleton points in each frame of picture, and inputting the number of skeleton points into a YOLOX _tiny target detection algorithm which is improved based on model pruning and fusion space attention mechanisms to obtain an interested target area and category detection result data of each frame of picture, wherein the target detection algorithm is obtained by training a self-built dataset and a transfer learning mode.”
Explanation: Pose estimation performs feature extraction which is fed into detection/classification model.
Regarding Claim 10, Zhu and Wang teach all of the limitations of claim 1 above. Zhu and Wang further teach the at least one processor and at least one memory including instructions that, when executed by the at least one processor, cause the system to perform the same steps as claim 1 as the execution of the pose estimation and target detection algorithms requires processing hardware and storage.
Regarding Claim 13, Zhu and Wang teach the system of claim 10, and additional limitations are met as in the consideration of claim 4 above.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 2, 3, 5, 6, 9, 11, 12, 14, 15, and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Zhu and Wang in view of Huang et. al (“Appearance-Independent Pose-Based Posture Classification in Infants”).
Regarding Claim 2, Zhu and Wang teach the computer-implemented method of claim 1, but fail to teach that it further comprises:
determining probability values corresponding to each frame of the plurality of frames and representing a confidence score for the posture prediction of the corresponding frame; wherein determining the first subset of the plurality of frames representing the transition segment further comprises:
determining a fourth subset of the plurality of frames representing a period of uncertainty, wherein the probability values of frames corresponding to the fourth subset fail to exceed a threshold value; and
determining the fourth subset corresponds with the first subset.
However, Huang teaches determining probability values corresponding to each frame of the plurality of frames and representing a confidence score for the posture prediction of the corresponding frame, stating that “the confidence score of each posture is listed at left-bottom corner in order” (Figure 5 caption). Huang also explicitly teaches ambiguous classification during transitions, stating that “it is difficult to judge which classes these transitional poses belong to” (4.2 Posture Network using 2D Poses, pg. 9), and that “we observe that the most classification failure cases in mini-MIMM data are the sitting postures that are recognized as supine due to the ambiguity of infant transition poses” (3D Pose-based Posture Classification Performance, pg. 12). Huang further provides numerical evidence of competing probabilities, stating that “the transition posture from sitting to standing is getting a confidence score of 0.55 for sitting and the score of 0.44 for standing” (4.2 Posture Network using 2D Poses, pg. 9). These disclosures teach competing class probabilities, lack of a clearly dominant classification, and uncertainty in classification outcomes. This corresponds to probability values not clearly exceeding a threshold and frames representing a period of uncertainty. In addition, it also teaches that transition frames correspond to uncertain classifications, which directly maps to the limitation: “determining the fourth subset corresponds with the first subset.”
Thus, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify Zhu and Wang to incorporate these teachings. A person of ordinary skill in the art would have been motivated to apply Huang’s confidence-based scoring to Zhu and Wang in order to identify low-confidence frames, improve detection of transition segments, and reduce misclassification during ambiguous posture changes. This represents the use of a known technique to improve similar devices (methods or products) in the same way, which would have yielded predictable results, namely identification of uncertain frames, improved segmentation of transition periods, and enhanced classification robustness.
Regarding Claim 3, Zhu and Wang in view of Huang teaches the computer-implemented method of claim 2, and Zhu and Wang further teach that the method comprises:
determining the start posture label further comprises determining a first stable posture by performing majority voting of the probability values corresponding to the second subset;
Paragraph [0024]: “Judging 2, when the occurrence times of 1 in the list C are larger than N/2, judging that the baby climbs.”
Paragraph [0025]: “Judgment 3, except judgment 1 and judgment 2, is regarded as a normal state.”
Explanation: “Number of occurrences times of 1 in the list C are larger than N/2” explicitly teaches majority voting (> 50%). This is performed over a set of frames (list C, size N) which corresponds to the second subset. The output that the baby is climbing corresponds to determining a stable/start posture label.
wherein determining the end posture label further comprises determining a second stable posture by performing majority voting of the probability values corresponding to the third subset.
Paragraph [0024]: “Judging 2, when the occurrence times of 1 in the list C are larger than N/2, judging that the baby climbs.”
Paragraph [0025]: “Judgment 3, except judgment 1 and judgment 2, is regarded as a normal state.”
Paragraph [0028]: “judging 5, when the first element in the list E is 0 and the last element is 1, judging that the baby kicks the quilt”
Explanation: The reference performs decision-making over temporal frame subsets (lists B, C, D, E). Majority voting shown through “greater than N/2.” Applying to another subset (e.g., list E or another temporal window) corresponds to third subset (end posture segment). Classification of the final state (e.g., kicking, climbing, normal) corresponds to end posture label.
Regarding Claim 5, Zhu and Wang teach the computer-implemented method of claim 4, but fail to teach that the pose estimation model is trained using an adult pose dataset and an augmented dataset including real-world infant pose data and synthetic infant pose data.
However, Huang teaches cross-domain pose training and augmentation, including synthetic data. Huang teaches an adult pose dataset and an augmented dataset with real and synthetic infant data, stating that “the strategy of FiDIP method is to transfer the pose learning of the existing adult pose estimation models into the infant poses… the FiDIP network is trained on an infant hybrid synthetic and real infant pose (SyRIP) dataset built based on a cross-domain inspired augmentation approach presented in [12]” (3.1 2D Pose Feature Extraction, pg. 4).
Thus, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify Zhu and Wang to train the pose estimation model using an adult pose dataset and an augmented dataset with real and synthetic infant data. Zhu and Wang explicitly identify that “images or video data sets for infants are lacking” (paragraph [0002]). Huang provides a known and established solution, which is transfer learning from adult dataset and an augmented dataset with real and synthetic infant data. One of ordinary skill in the art would have been motivated to incorporate Huang’s training approach into Zhu and Wang to address the lack of infant data and improve model robustness, with a reasonable expectation of success. It would have been obvious to apply Huang’s augmentation and transfer learning techniques to Zhu and Wang’s system to improve performance in the same way, which would have yielded predictable results such as data sufficiency, model generalization, and classification accuracy.
Regarding Claim 6, Zhu and Wang teach the computer-implemented method of claim 4, but fail to teach that the posture classification model is trained using a two-dimensional infant pose dataset and a three-dimensional infant pose dataset.
However, Huang teaches 2D and 3D pose datasets used for training, stating that they use a “novel deep neural network classifier of infant posture, based on an input 2D or 3D poses… we combine our previous 2D infant pose estimation model [12] with our new posture classifier to obtain a 2D-based pipeline for posture estimation from infant images… we retrain our 3D infant pose estimation model [20] on newly released 3D ground truth poses for the SyRIP dataset [13], and then combine this pose estimation with a 3D pose-based posture classifier, again trained with the 3D SyRIP data” (Introduction, pg. 2).
Thus, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify Zhu and Wang to train the posture classification model using both 2D and 3D pose datasets. Huang provides performance motivation, stating that the 3D pipeline “offers improved performance over the 2D pipeline because our pose estimation model takes advantage of depth data during training” (Introduction, pg. 2). Thus, Huang teaches that incorporating 3D pose data improves posture classification accuracy. Accordingly, a person of ordinary skill in the art would have been motivated to incorporate both 2D and 3D training datasets in order to improve robustness and accuracy in infant action recognition. This represents the use of a known technique to improve similar devices (methods or products) in the same way, which would have yielded predictable results.
Regarding Claim 9, Zhu and Wang teach the computer-implemented method of claim 1, but fail to teach that the posture classification model classifies a posture as one of supine, prone, sitting, standing, or all-fours. Zhu and Wang teach climbing behavior, which corresponds to an all-fours posture as infant climbing involves hands and knees on an all-fours posture, but fail to teach supine, prone, sitting, and standing.
However, Huang teaches classifying infant postures into the claimed categories, stating that “the postures considered are the supine, prone, sitting, and standing postures, which are critical in standardized motor evaluation paradigms” (Introduction, pg. 2).
Thus, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify Zhu and Wang to include supine, prone, sitting, and standing postures as additional classifications. A person of ordinary skill in the art would have been motivated to incorporate Huang’s known posture categories because both references classify infant body states from pose data, posture labels are known interchangeable outputs, and combining known classification yields predictable results. As shown above (Introduction, pg. 2), Huang provides explicit motivation for using these posture categories as these posture classes are standardized, and they improve clinical assessment and interpretability. A person of ordinary skill in the art would have modified Zhu and Wang to include Huang’s posture classes to improve classification usefulness and align with standardized infant motor evaluation. This represents the use of a known technique to improve similar devices (methods or products) in the same way, which would have yielded predictable results.
Regarding Claim 11, Zhu and Wang teach the system of claim 10, and additional limitations are met as in the consideration of claim 2 above.
Regarding Claim 12, Zhu and Wang in view of Huang teaches the system of claim 11, and additional limitations are met as in the consideration of claim 3 above.
Regarding Claim 14, Zhu and Wang teach the system of claim 13, and additional limitations are met as in the consideration of claim 5 above.
Regarding Claim 15, Zhu and Wang teach the system of claim 13, and additional limitations are met as in the consideration of claim 6 above.
Regarding Claim 18, Zhu and Wang teach the system of claim 10, and additional limitations are met as in the consideration of claim 9 above.
Claims 7 and 8 are rejected under 35 U.S.C. 103 as being unpatentable over Zhu and Wang in view of Yan et. al (“Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition”).
Regarding Claim 7, Zhu and Wang in view teach the computer-implemented method of claim 4, wherein determining the first subset of the plurality of frames representing the transition segment further comprises:
extracting, based on the pose estimation data, a set of feature vectors corresponding to the plurality of frames;
Paragraph [0008]: “The pose estimation algorithm uses a lightweight openpose pose estimation algorithm to obtain the key bone points of the infant body in each frame.”
Explanation: Keypoints = structured feature representation (feature vectors).
However, Zhu and Wang fail to teach the limitation: “determining, using a transition segmentor model with the set of feature vectors as input, the first subset, wherein the transition segmentor model is trained using vectors representing posture transitions.”
However, Yan explicitly teaches trained temporal modeling of posture transitions, stating that “we propose to design a generic representation of skeleton sequences for action recognition by extending graph neural networks to a spatial-temporal graph model,” (Introduction, pg. 2) and “as ST-GCN’s input, the feature vector on a node F(vti) consists of coordinate vectors, as well as estimation confidence, of the i-th joint on frame t” (3.2 Skeleton Graph Construction, pg. 3). This directly teaches feature vectors as input to a trained transition (spatial-temporal GCN) model.
Thus, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify Zhu and Wang to include a transition segmentor model that is trained using vectors representing posture transitions. Zhu and Wang use rule-based temporal segmentation while Yan teaches a trained model operating on skeleton feature vectors. This demonstrates that learned models were known alternatives to rule-based approaches for processing pose sequences. A person of ordinary skill in the art would have been motivated to substitute the rule-based temporal processing of Zhu and Wang with the trained spatio-temporal model of ST-GCN because both techniques perform the same function of analyzing temporal pose data to determine actions. Because both references operate on skeleton/keypoint data and address human action recognition from pose sequences, a substitution would have yielded predictable results, namely improved transition detection using learned temporal patterns. Additionally, Yan further teaches that modeling temporal dependencies improves action recognition. Thus, one of ordinary skill in the art would have been motivated to adopt this modification in order to improve accuracy and robustness in identifying posture transitions, representing the use of a known technique to improve similar devices (methods or products) in the same way.
Regarding Claim 8, Zhu and Wang in view of Yan teaches the computer-implemented method of claim 7, and Yan further teaches that the set of feature vectors are extracted from a penultimate layer of the posture classification model.
Fig. 2 Caption: “Multiple layers of spatial-temporal graph convolution (ST-GCN) will be applied and gradually generate higher-level feature maps on the graph.”
3.4 Partition Strategies: “In this strategy, feature vectors on every neighboring node will have a inner product with the same weight vector.”
3.6 Implementing ST-GCN, pg. 6: “After that, a global pooling was performed on the resulting tensor to get a 256-dimension feature vector for each sequence. Finally, we feed them to a SoftMax classifier.”
Explanation: ST-GCN teaches that multiple layers generate higher level feature maps which are subsequently pooled into a 256-dimension feature vector prior to classification by a softmax layer, thereby disclosing feature vectors extracted from a penultimate layer of the classification model.
Regarding Claim 16, Zhu and Wang teach the system of claim 13, and additional limitations are met as in the consideration of claim 7 above.
Regarding Claim 17, Zhu and Wang in view of Yan teaches the system of claim 16, and additional limitations are met as in the consideration of claim 8 above.
Claim 19 is rejected under 35 U.S.C. 103 as being unpatentable over Zhu and Wang in view of Duan et. al (“Revisiting Skeleton-based Action Recognition”).
Regarding Claim 19, Zhu and Wang teach a computer-implemented method of generating a dataset of a plurality of infant actions, comprising:
Paragraph [0001]: “The invention relates to the technical field of pose estimation and target detection based on computer vision, in particular to an infant dangerous action detection and identification method based on pose estimation and improvement YOLOX _tiny.”
Paragraph [0031]: “The invention is characterized in that the construction of the target detection algorithm related data set mainly comprises the steps of data acquisition, labeling and division, and the detailed steps are as follows…”
receiving a plurality of video recordings that capture actions of human infants;
Paragraph [0006]: “acquiring a video frame from a camera, and preprocessing the input video frame.”
Paragraph [0007]: “Inputting the preprocessed video frames into a pose estimation algorithm to obtain the number of skeleton points in each frame of picture…”
Paragraph [0032]: “a1, collecting state pictures of infants in indoor environments, wherein the state pictures comprise five state pictures of standing, lying, climbing, sitting and climbing.”
determining an infant action label for each video recording of the plurality of video recordings, wherein determining the infant action label for a video recording further comprises:
Paragraph [0007]: “Inputting the preprocessed video frames into a pose estimation algorithm to obtain the number of skeleton points in each frame of picture, and inputting the number of skeleton points into a YOLOX _tiny target detection algorithm which is improved based on model pruning and fusion space attention mechanisms to obtain an interested target area and category detection result data of each frame of picture, wherein the target detection algorithm is obtained by training a self-built dataset and a transfer learning mode.”
Paragraph [0024]: “Judging 2, when the occurrence times of 1 in the list C are larger than N/2, judging that the baby climbs”
Paragraph [0025]: “Judgment 3, except judgment 1 and judgment 2, is regarded as a normal state”
Paragraph [0028]: “judging 5, when the first element in the list E is 0 and the last element is 1, judging that the baby kicks the quilt”
determining a region of interest for each frame of the video recording, wherein the region of interest corresponds to detection of an infant;
Paragraph [0007]: “Inputting the preprocessed video frames into a pose estimation algorithm to obtain the number of skeleton points in each frame of picture, and inputting the number of skeleton points into a YOLOX _tiny target detection algorithm which is improved based on model pruning and fusion space attention mechanisms to obtain an interested target area and category detection result data of each frame of picture, wherein the target detection algorithm is obtained by training a self-built dataset and a transfer learning mode.”
Explanation: “Interested target area” = region of interest
determining, using the region of interest for each frame, a skeletal pose;
Paragraph [0008]: “The pose estimation algorithm uses a lightweight openpose pose estimation algorithm to obtain the key bone points of the infant body in each frame.”
Explanation: Key bone points = skeletal pose representation
determining, using an action recognition model with the set of skeleton keypoints as input, the infant action label;
Paragraph [0007]: “Inputting the preprocessed video frames into a pose estimation algorithm to obtain the number of skeleton points in each frame of picture, and inputting the number of skeleton points into a YOLOX _tiny target detection algorithm which is improved based on model pruning and fusion space attention mechanisms to obtain an interested target area and category detection result data of each frame of picture, wherein the target detection algorithm is obtained by training a self-built dataset and a transfer learning mode.”
Paragraph [0047]: “A3, processing and fusing the obtained data to improve the accurate recognition capability of different dangerous actions.”
labeling each video of the plurality of video recordings with the infant action label corresponding to the video recording; and
Paragraph [0031]: “The invention is characterized in that the construction of the target detection algorithm related data set mainly comprises the steps of data acquisition, labeling and division, and the detailed steps are as follows…”
storing the plurality of videos labeled with the infant action label in a database.
Paragraph [0034]: “a3, dividing the data set into a training set, a verification set and a test set, wherein the ratio is 8:1:1.”
Explanation: Dataset storage and organization (training/validation/testing) requires storage in a memory/database.
Zhu and Wang fail to teach the limitation: “determining, using the skeletal pose, a set of skeleton keypoints corresponding to an adult skeleton,” as the key points they extract correspond to infant skeletons.
However, Duan teaches this limitation, stating that “human skeletons in a video are mainly represented as a sequence of joint coordinate lists, where the coordinates are extracted by pose estimators” (Introduction, pg. 1). Here, pose estimators output standardized human (adult) skeleton keypoints.
Thus, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify Zhu and Wang to determine, using the skeletal pose, a set of skeleton keypoints corresponding to an adult skeleton. While Zhu and Wang determine, using the skeletal pose, a set of skeleton keypoints, it corresponds to an infant instead of an adult. A person of ordinary skill in the art would have been motivated to represent the detected skeletal pose using a standardized adult skeleton keypoint structure in order to improve computability with existing action recognition models and leverage well-established skeletal representations. This represents the use of a known technique to improve similar devices (methods or products) in the same way, which would have yielded predictable results such as enhanced processing and recognition performance.
Claim 20 is rejected under 35 U.S.C. 103 as being unpatentable over Zhu and Wang in view of Duan et. al, further in view of Du et. al (“Hierarchical Recurrent Neural Network for Skeleton Based Action Recognition”) and Yan et. al (“Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition”).
Regarding Claim 20, Zhu and Wang in view of Duan et. al teaches the computer-implemented method of claim 19, and Duan further teaches that the action recognition model is one of:
a three-dimensional convolutional network with the skeleton keypoints from each frame converted into a heatmap.
Introduction, pg. 2: “The 2D poses are represented by stacks of heatmaps of skeleton joints rather than coordinates operated on a human skeleton graph. The heatmaps at different timesteps will be stacked along the temporal dimension to form a 3D heatmap volume.”
Figure 2 Caption: “Finally, we use a 3D-CNN to classify the 3D heatmap volumes.”
Conclusion, pg. 8: “a 3D-CNN based approach for skeleton-based action recognition, which takes 3D heatmap volumes as input.”
Neither Zhu and Wang nor Duan teach that the action recognition model is one of a recurrent neural network (RNN) with the skeleton keypoints separated into body part groups and a graph convolutional network with the skeleton keypoints represented as a graph, wherein joints are nodes of the graph and connections between the joints are edges of the graph.
However, Du teaches a recurrent neural network with the skeleton keypoints separated into body part groups, stating that the “human body can be roughly decomposed into five parts, e.g., two arms, two legs and one trunk, and human actions are composed of the movements of these body parts…given this fact, we divide the human skeleton into the five corresponding parts, and feed them into five bidirectionally recurrently connected subnets (BRNNs) in the first layer” (Introduction, pg. 2).
Thus, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify Zhu and Wang to include part-based RNN modeling. Du explains the benefit of part-based modeling, stating that “from a viewpoint of feature learning, these stacked BRNNs can be considered to extract the spatial and temporal features of the skeleton sequences” (3.2. Hierarchical RNN for Skeleton Based Action Recognition, pg. 4). A person of ordinary skill in the art would have applied this technique because it improves the modeling of spatial and temporal dynamics, leading to improved accuracy in skeleton-based action recognition. This represents the use of a known technique to improve similar devices (methods or products) in the same way, which would have yielded predictable results.
Neither Zhu and Wang, Duan, or Du teach that the action recognition model is one of a graph convolutional network with the skeleton keypoints represented as a graph, wherein joints are nodes of the graph and connections between the joints are edges of the graph.
However, Yan teaches a graph convolutional network with the skeleton keypoints represented as a graph, wherein joints are nodes of the graph and connections between the joints are edges of the graph, stating that “given the sequences of body joints in the form of 2D or 3D coordinates, we construct a spatial temporal graph with the joints as graph nodes and natural connectivities in both human body structures and time as graph edges” (3.1 Pipeline Overview, pg. 2). In addition, Yan states that “we construct an undirected spatial temporal graph G =(V,E) on a skeleton sequence with N joints and T frames featuring both intra-body and inter-frame connection…the edge set E is composed of two subsets, the first subset depicts the intra-skeleton connection at each frame, denoted as ES = {vtivtj|(i,j) ∈ H}, where H is the set of naturally connected human body joints” (3.2 Skeleton Graph Construction, pg. 3). This defines the nodes as (V) and edges (E), where edges are directly connected to joint connectivity.
Thus, it would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to modify Zhu and Wang to include a graph convolutional network with the skeleton keypoints represented as a graph. Yan explains the advantage of such representation, stating that “the dynamic skeleton modality can be naturally represented by a time series of human joint locations, in the form of 2D or 3D coordinates” (Introduction, pg. 1). In view of these teachings, a person of ordinary skill in the art would have been motivated to apply Yan’s graph-based representation and corresponding GCN to the skeleton keypoints of Zhu and Wang in order to better model spatial relationships between joints and improve action recognition performance. This represents the use of a known technique to improve similar devices (methods or products) in the same way, which would have yielded predictable results.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Okuno et. al (“Rollover Detection of Infants Using Posture Estimation Model”) compares the accuracy of infant posture estimation by two posture estimation models, OpenPose and Cascaded Pyramid Network (CPN). They also introduce a system for estimating infant's sleep turn using the posture estimation results.
McCay et. al (“Abnormal Infant Movements Classification With Deep Learning on Pose-Based Features”) teaches pose-based feature sets for automated classification within a deep learning framework by carrying out extensive experiments on five new deep learning architectures.
Dechemi et. al (“BabyNet: A Lightweight Network for Infant Reaching Action Recognition in Unconstrained Environments to Support Future Pediatric Rehabilitation Applications”) teaches BabyNet, a light-weight (in terms of trainable parameters) network structure to recognize infant reaching action from off-body stationary cameras.
Pachecho et. al (“A Detection-based Approach to Multiview Action Classification in Infants”) teaches activity recognition in children and infants using a multiview action classification system based on Faster R-CNN and LSTM networks, which fuses information from different views by using learnable fusion coefficients derived from detection confidence scores.
Zhou et. al (“HIERARCHICAL POSE CLASSIFICATION FOR INFANT ACTION ANALYSIS AND MENTAL DEVELOPMENT ASSESSMENT”) teaches a systematic image-based pose classifier to classify infant actions based on AIMS to provide early diagnosis of a potential develop mental disorder such as Autism.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to WILLIAM ADU-JAMFI whose telephone number is (571)272-9298. The examiner can normally be reached M-T 8:00-6:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Bee can be reached at (571) 270-5183. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/WILLIAM ADU-JAMFI/Examiner, Art Unit 2677
/ANDREW W BEE/Supervisory Patent Examiner, Art Unit 2677
Prosecution Insights