DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on June 4, 2024 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Objections
Claims 17 and 19-20 are objected to because of the following informalities:
Claim 17, line 2 ends with an erroneous period which should be removed
Claim 19 and claim 20, “is configured” should read “are configured”
Appropriate correction is required.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are:
“feature access module” in claims 16-17
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-3, 7, 14-16 and 19-21 are rejected under 35 U.S.C. 103 as being unpatentable over Xu, Bingjie, et al. "Interact as you intend: Intention-driven human-object interaction detection." IEEE Transactions on Multimedia 22.6 (2019): 1423-1432. (hereinafter Xu), and further in view of Qiu, Lingteng, et al. "Peeking into occluded joints: A novel framework for crowd pose estimation." European conference on computer vision. Cham: Springer International Publishing, 2020. (hereinafter Qiu).
Regarding independent claim 1, Xu discloses A person intention reasoning method (abstract, “In this work, we focus on detecting human-object interactions (HOIs) in social scene images, which is demanding in terms of research and increasingly useful for practical applications.”… “the proposed human intention driven HOI detection (iHOI) framework models human pose with the relative distances from body joints to the object instances.”), comprising:
performing target detection on a to-be-reasoned image to obtain a corresponding target detection result (page 3, left column, “First, given an input image I, we adopt Faster R-CNN [2] from Detectron [40] to detect all humans and objects, generating a set of detected bounding boxes b = (b1,...,bm) where m denotes the total number of detected instances.”);
determining a detection bounding box of each person in the to-be-reasoned image based on the target detection result (page 3, left column, “First, given an input image I, we adopt Faster R-CNN [2] from Detectron [40] to detect all humans and objects, generating a set of detected bounding boxes b = (b1,...,bm) where m denotes the total number of detected instances. The detected bounding boxes for a person and an object are denoted as bh and bo, respectively.”), determining that an image portion corresponding to each detection bounding box in the to-be-reasoned image is a to-be-reasoned sub-image of a corresponding person respectively (page 3, left column, “The detected bounding boxes for a person and an object are denoted as bh and bo, respectively;” bh is read as person boxes),
performing person intention reasoning by using the target detection result and the correction feature of the corresponding joint of the corresponding person in each to-be-reasoned sub-image to obtain a corresponding person intention reasoning result (page 4, left column, “the feature space x for each bh and bo contains visual appearance, relative spatial layout, and object semantic likelihood, referred as vα, vl and vc, respectively. vα is a 2048-d vector, extracted from fc7 layer in the object detector to capture the appearance of each bo. vl is a 4-d vector consisted of {lx,ly,lw,lh}. {lx,ly} specifies the bounding box coordinates distances, and {lw,lh} specifies the log-space height/width shift, all relative to a counterpart as parameterized in Faster R-CNN. vc is a 81-d vector of object classification scores over MS-COCO object categories, generated by the object detectors. In contrast to the general visual relationships, we extend the feature space with human pose information since our task is intrinsically human-centric. Human pose bridges the human body with the interacting object. For example, the up stretching arms, jumping posture and the relative distances to the ball possibly reveal that the person is hitting a sports ball. Since body pose ground-truth is not available, we use the pose estimation network in [41] to extract body joints locations for each human. The output of the pose estimation network is the locations of 18 body joints. We consider eight representative body joints1 that are more frequently detected, which cover the head, upper and lower body.”).
Xu fails to explicitly disclose as further recited. However, Qiu discloses
and acquiring a joint feature and an occlusion probability of a joint of the corresponding person in each to-be-reasoned sub-image (page 494, “ After that, we employ a grid sample method that obtains the jth joint feature by excavating the feature located in on the related coordinate weight feature map. Every pose leads to three node feature vectors , , and extracted following this process. Finally, these node features are fed into the ResGCN attention blocks accordingly;” see paper to note the variables that can’t copy over)
performing prediction and analysis on the joint feature of corresponding joint based on the occlusion probability to obtain a corresponding prediction feature (abstract, “our framework estimates the invisible joints from an inference perspective by proposing an Image-Guided Progressive GCN module which provides a comprehensive understanding of both image context and pose structure.”), and
performing correction based on the joint feature and the corresponding prediction feature of the corresponding joint of the corresponding person in each to-be-reasoned sub-image to obtain a correction feature of the corresponding joint of the corresponding person in each to-be-reasoned sub-image (page 489, “The first stage generates heatmaps to produce an initial pose and the subsequent correction stage adjusts the initial pose obtained from the heatmaps by an Image-Guided Progressive GCN (IGP-GCN) module;” page 493, “we propose an Image-Guided graph network for correction which takes the initial pose generated from the above modules and adjusts the estimation results according to the implicit relationship of joints.”).
Xu is directed toward, “In this work, we focus on detecting human-object interactions (HOIs) in social scene images, which is demanding in terms of research and increasingly useful for practical applications. To undertake social tasks interacting with objects, humans direct their attention and move their body based on their intention. Based on this observation, we provide a unique computational perspective to explore human intention in HOI detection. Specifically, the proposed human intention driven HOI detection (iHOI) framework models human pose with (abstract).” Qiu is directed toward, “our framework estimates the invisible joints from an inference perspective by proposing an Image-Guided Progressive GCN module which provides a comprehensive understanding of both image context and pose structure. Moreover, existing benchmarks contain limited occlusions for evaluation. Therefore, we thoroughly pursue this problem and propose a novel OPEC-Net framework together with a new Occluded Pose (OCPose) dataset with 9k annotated images (abstract).” As can be easily seen by one of ordinary skill in the art Xu and Qiu are directed toward similar methods of endeavor of image analysis and person action reasoning. Further, one of ordinary skill in the art before the effective filing date of the claimed invention would easily be aware that when humans perform actions, different body parts can occlude others, leading to confusing the system and generating an inaccurate output. Thus, it would have been obvious to a person having ordinary skill in the art before the effective filing date to incorporate the teaching of Qiu in order to ensure accurate joint analysis is performed, using constraints, to determine an accurate action output.
Regarding dependent claim 2, the rejection of claim 1 is incorporated herein. Additionally, Qiu in the combination further discloses herein the performing prediction and analysis on the joint feature of the corresponding joint based on the occlusion probability to obtain the corresponding prediction feature comprises:
taking an arbitrary to-be-reasoned sub-image as a current sub-image, and performing coding fusion on the joint feature and corresponding occlusion probability of each joint in the current sub-image to obtain corresponding fused feature information (page 494, “excavate the related image features for each joints position and fuse them into the module. In another word, we improve the pose estimation results by incorporating image feature maps , and . S;” page 495, “ Moreover, the low-level feature and high-level feature are fused in the cascaded design in order to enlarge their respective fields resulting the updated feature are more informative. The details of Conv Blocks and Fusion Blocks used in this module is in supplementary materials;”); and
inputting the corresponding fused feature information of the current sub-image into an occluded joint prediction network to obtain a prediction feature of each joint in the current sub- image outputted by the occluded joint prediction network (page 490, “The IGP-GCN feeds both the coordinate of joints and also the image features extracted at the location of joints as input to each graph node. Therefore, the multi-scale image features from the heatmap modules are fed into the IGP-GCN in a progressive way, so that large displacements can be learned steadily. ”), wherein the occluded joint prediction network is obtained by pre-training based on a plurality of pieces of the corresponding fused feature information of a known prediction feature (page 496, “We build a new dataset, called Occluded Pose (OCPose), that includes more heavy occlusions to evaluate the MPPE. It contains challenging invisible joints and complex intertwined human poses. We mostly consider the couple pose scenes, such as dancing, skating, and wrestling, because they have more reliable annotations and practical utility. This section gives details of data collection, data annotation, and data statistics.;” page 497, “Data Statistics. In total, our dataset contains 9000 images and 18000 fully annotated persons. For the training process, the training dataset consists of 5000 images, whereas validation and test dataset each contains 2000 images.”).
Regarding dependent claim 3, the rejection of claim 2 is incorporated herein. Additionally, Qiu in the combination further discloses wherein the performing coding fusion on the joint feature and the corresponding occlusion probability of each joint in the current sub-image to obtain the corresponding fused feature information comprises:
splicing the joint feature of the current sub-image and the corresponding occlusion probability of the current sub-image directly into a corresponding multi-dimensional vector as the corresponding fused feature information of the current sub-image (page 490, “Considering that, our GCN network is specially designed in an Image-Guided way: The IGP-GCN feeds both the coordinate of joints and also the image features extracted at the location of joints as input to each graph node. Therefore, the multi-scale image features from the heatmap modules are fed into the IGP-GCN in a progressive way, so that large displacements can be learned steadily;” inputting to values is read as splicing (i.e. they are input to be combined or analyzed together); page 495, “Moreover, the low-level feature and high-level feature are fused in the cascaded design in order to enlarge their respective fields resulting the updated feature are more informative;” page 494, “For every node, the input feature is the joint estimation result , where i is ith pose and j is the jth joint of the skeleton. We denote as the input feature of the ith pose in the training set, where L is the feature dimension.”).
Regarding dependent claim 7, the rejection of claim 2 is incorporated herein. Additionally, Qiu in the combination further discloses wherein the method further comprises:
acquiring a plurality of images as training images respectively, wherein each of the training images comprises a single person (page 492, “This is a top-down approach, which first detects a bounding box for each person and then performs instance-level human pose estimation.;” page 497, “the training set;” page 497, “ In total, our dataset contains 9000 images and 18000 fully annotated persons. For the training process, the training dataset consists of 5000 images, whereas validation and test dataset each contains 2000 images.”);
acquiring fused feature information and a corresponding prediction feature of each of the training images (page 490, “The IGP-GCN feeds both the coordinate of joints and also the image features extracted at the location of joints as input to each graph node;” page 495, “ a binary mask where the element in M corresponds to 1 when the related joint has a ground truth label, otherwise it is 0. ”); and
inputting the fused feature information and the corresponding prediction feature of each of the training images into a graph convolutional network (page 489, “the correction module is designed as a GCN-based network, which offers an explicit way of modeling the body structural information that is advantageous for correcting the joints.”), and training the graph convolutional network to obtain a trained graph convolutional network (page 490, “ our framework is trained in an end-to-end fashion ”), wherein the trained graph convolutional network is the occluded joint prediction network (page 489, “the correction module is designed as a GCN-based network, which offers an explicit way of modeling the body structural information that is advantageous for correcting the joints.”).
Regarding dependent claim 14, the rejection of claim 1 is incorporated herein. Additionally, Xu in the combination further discloses wherein the performing target detection on the to-be-reasoned image to obtain the corresponding target detection result comprises:
performing feature extraction on the to-be-reasoned image by using a target detection network to obtain the detection bounding box comprising each person in the to-be-reasoned image and the target detection result corresponding to each detection bounding box (page 5, right column, “Our implementation is based on Faster R-CNN [2] with a Feature Pyramid Network (FPN) [3] backbone built on ResNet-50 [1], from Detectron [40];” target detection network is read as deep learning to identify objects in image/video ).
Regarding dependent claim 15, the rejection of claim 1 is incorporated herein. Additionally, Xu in the combination further discloses wherein the performing person intention reasoning by using the target detection result and the correction feature of the corresponding joint of the corresponding person in each to-be-reasoned sub-image to obtain the corresponding person intention reasoning result comprises:
inputting features of other entities, except the corresponding person, in the target detection result and the correction feature of the corresponding joint of the corresponding person in each to-be-reasoned sub-image into an intention prediction network to obtain the corresponding person intention reasoning result outputted by the intention prediction network (page 4, left column, “for general visual relationships among objects, the feature space x for each bh and bo contains visual appearance, relative spatial layout, and object semantic likelihood” … “The pairwise embedding is passed through a fully-connected layer to produce the pairwise action scores”).
Regarding dependent claim 16, the rejection of claim 15 is incorporated herein. Additionally, Xu in the combination further discloses wherein the method further comprises:
storing the features of other entities, except the corresponding person, in the to-be- reasoned image comprised in the target detection result to a feature access module (page 4, left column, “for general visual relationships among objects, the feature space x for each bh and bo contains visual appearance, relative spatial layout, and object semantic likelihood”… “vc is a 81-d vector of object classification scores over MS-COCO object categories, generated by the object detectors.”).
Regarding independent claim 19, the rejection of claim 1 applies directly. Additionally, Xu discloses A person intention reasoning device, comprising a memory and one or more processors, wherein the memory stores computer-readable instructions, and the computer-readable instructions, upon execution by the one or more processors (abstract, “In this work, we focus on detecting human-object interactions (HOIs) in social scene images, which is demanding in terms of research and increasingly useful for practical applications. To undertake social tasks interacting with objects, humans direct their attention and move their body based on their intention. Based on this observation, we provide a unique computational perspective to explore human intention in HOI detection.;” Figure 2 outlines the entire method, which being based on neural networking is required to be executed on a computer, thus programmed and stored in memory as well), is configured to cause the one or more processors to:
perform target detection on a to-be-reasoned image to obtain a corresponding target detection result (page 3, left column, “First, given an input image I, we adopt Faster R-CNN [2] from Detectron [40] to detect all humans and objects, generat ing a set of detected bounding boxes b = (b1,...,bm) where m denotes the total number of detected instances.”);
determine a detection bounding box of each person in the to-be-reasoned image based on the target detection result (page 3, left column, “First, given an input image I, we adopt Faster R-CNN [2] from Detectron [40] to detect all humans and objects, generating a set of detected bounding boxes b = (b1,...,bm) where m denotes the total number of detected instances. The detected bounding boxes for a person and an object are denoted as bh and bo, respectively.”), determine that an image portion corresponding to each detection bounding box in the to-be-reasoned image is a to-be-reasoned sub-image of a corresponding person respectively (page 3, left column, “The detected bounding boxes for a person and an object are denoted as bh and bo, respectively;” bh is read as person boxes),
perform person intention reasoning by using the target detection result and the correction feature of the corresponding joint of the corresponding person in each to-be-reasoned sub-image to obtain a corresponding person intention reasoning result (page 4, left column, “the feature space x for each bh and bo contains visual appearance, relative spatial layout, and object semantic likelihood, referred as vα, vl and vc, respectively. vα is a 2048-d vector, extracted from fc7 layer in the object detector to capture the appearance of each bo. vl is a 4-d vector consisted of {lx,ly,lw,lh}. {lx,ly} specifies the bounding box coordinates distances, and {lw,lh} specifies the log-space height/width shift, all relative to a counterpart as parameterized in Faster R-CNN. vc is a 81-d vector of object classification scores over MS-COCO object categories, generated by the object detectors. In contrast to the general visual relationships, we extend the feature space with human pose information since our task is intrinsically human-centric. Human pose bridges the human body with the interacting object. For example, the up stretching arms, jumping posture and the relative distances to the ball possibly reveal that the person is hitting a sports ball. Since body pose ground-truth is not available, we use the pose estimation network in [41] to extract body joints locations for each human. The output of the pose estimation network is the locations of 18 body joints. We consider eight representative body joints1 that are more frequently detected, which cover the head, upper and lower body.”).
Xu fails to explicitly disclose as further recited. However, Qiu discloses acquire a joint feature and an occlusion probability of a joint of the corresponding person in each to-be-reasoned sub-image ; (page 494, “ After that, we employ a grid sample method that obtains the jth joint feature by excavating the feature located in on the related coordinate weight feature map. Every pose leads to three node feature vectors , , and extracted following this process. Finally, these node features are fed into the ResGCN attention blocks accordingly;” see paper to note the variables that can’t copy over)
perform prediction and analysis on the joint feature of corresponding joint based on the occlusion probability to obtain a corresponding prediction feature (abstract, “our framework estimates the invisible joints from an inference perspective by proposing an Image-Guided Progressive GCN module which provides a comprehensive understanding of both image context and pose structure.”), and perform correction based on the joint feature and the corresponding prediction feature of the corresponding joint of the corresponding person in each to-be-reasoned sub-image to obtain a correction feature of the corresponding joint of the corresponding person in each to-be-reasoned sub-image (page 489, “The first stage generates heatmaps to produce an initial pose and the subsequent correction stage adjusts the initial pose obtained from the heatmaps by an Image-Guided Progressive GCN (IGP-GCN) module;” page 493, “we propose an Image-Guided graph network for correction which takes the initial pose generated from the above modules and adjusts the estimation results according to the implicit relationship of joints.”).
Xu is directed toward, “In this work, we focus on detecting human-object interactions (HOIs) in social scene images, which is demanding in terms of research and increasingly useful for practical applications. To undertake social tasks interacting with objects, humans direct their attention and move their body based on their intention. Based on this observation, we provide a unique computational perspective to explore human intention in HOI detection. Specifically, the proposed human intention driven HOI detection (iHOI) framework models human pose with (abstract).” Qiu is directed toward, “our framework estimates the invisible joints from an inference perspective by proposing an Image-Guided Progressive GCN module which provides a comprehensive understanding of both image context and pose struc ture. Moreover, existing benchmarks contain limited occlusions for evalu ation. Therefore, we thoroughly pursue this problem and propose a novel OPEC-Net framework together with a new Occluded Pose (OCPose) dataset with 9k annotated images (abstract).” As can be easily seen by one of ordinary skill in the art Xu and Qiu are directed toward similar methods of endeavor of image analysis and person action reasoning. Further, one of ordinary skill in the art before the effective filing date of the claimed invention would easily be aware that when humans perform actions, different body parts can occlude others, leading to confusing the system and generating an inaccurate output. Thus, it would have been obvious to a person having ordinary skill in the art before the effective filing date to incorporate the teaching of Qiu in order to ensure accurate joint analysis is performed, using constraints, to determine an accurate action output.
Regarding independent claim 20, the rejection of claim 1 applies directly. Additionally, Xu discloses One or more non-volatile computer-readable storage medium, storing computer-readable instructions, wherein the computer-readable instructions , upon execution by the one or more processors (abstract, “In this work, we focus on detecting human-object interactions (HOIs) in social scene images, which is demanding in terms of research and increasingly useful for practical applications. To undertake social tasks interacting with objects, humans direct their attention and move their body based on their intention. Based on this observation, we provide a unique computational perspective to explore human intention in HOI detection.;” Figure 2 outlines the entire method, which being based on neural networking is required to be executed on a computer, thus programmed and stored in memory as well), is configured to:
perform target detection on a to-be-reasoned image to obtain a corresponding target detection result (page 3, left column, “First, given an input image I, we adopt Faster R-CNN [2] from Detectron [40] to detect all humans and objects, generating a set of detected bounding boxes b = (b1,...,bm) where m denotes the total number of detected instances.”);
determine a detection bounding box of each person in the to-be-reasoned image based on the target detection result (page 3, left column, “First, given an input image I, we adopt Faster R-CNN [2] from Detectron [40] to detect all humans and objects, generating a set of detected bounding boxes b = (b1,...,bm) where m denotes the total number of detected instances. The detected bounding boxes for a person and an object are denoted as bh and bo, respectively.”), determine that an image portion corresponding to each detection bounding box in the to-be-reasoned image is a to-be-reasoned sub-image of a corresponding person respectively (page 3, left column, “The detected bounding boxes for a person and an object are denoted as bh and bo, respectively;” bh is read as person boxes)
perform person intention reasoning by using the target detection result and the correction feature of the corresponding joint of the corresponding person ineach to-be-reasoned sub-image to obtain a corresponding person intention reasoning result (page 4, left column, “the feature space x for each bh and bo contains visual appearance, relative spatial layout, and object semantic likelihood, referred as vα, vl and vc, respectively. vα is a 2048-d vector, extracted from fc7 layer in the object detector to capture the appearance of each bo. vl is a 4-d vector consisted of {lx,ly,lw,lh}. {lx,ly} specifies the bounding box coordinates distances, and {lw,lh} specifies the log-space height/width shift, all relative to a counterpart as parameterized in Faster R-CNN. vc is a 81-d vector of object classification scores over MS-COCO object categories, generated by the object detectors. In contrast to the general visual relationships, we extend the feature space with human pose information since our task is intrinsically human-centric. Human pose bridges the human body with the interacting object. For example, the up stretching arms, jumping posture and the relative distances to the ball possibly reveal that the person is hitting a sports ball. Since body pose ground-truth is not available, we use the pose estimation network in [41] to extract body joints locations for each human. The output of the pose estimation network is the locations of 18 body joints. We consider eight representative body joints1 that are more frequently detected, which cover the head, upper and lower body.”).
Xu fails to explicitly disclose as further recited. However, Qiu discloses acquire a joint feature and an occlusion probability of a joint of the corresponding person in each to-be-reasoned sub-image (page 494, “ After that, we employ a grid sample method that obtains the jth joint feature by excavating the feature located in on the related coordinate weight feature map. Every pose leads to three node feature vectors , , and extracted following this process. Finally, these node features are fed into the ResGCN attention blocks accordingly;” see paper to note the variables that can’t copy over);
perform prediction and analysis on the joint feature of corresponding joint based on the occlusion probability to obtain a corresponding prediction feature (abstract, “our framework estimates the invisible joints from an inference perspective by proposing an Image-Guided Progressive GCN module which provides a comprehensive understanding of both image context and pose structure.”), and perform correction based on the joint feature and the corresponding prediction feature of the corresponding joint of the corresponding person in each to-be-reasoned sub-image to obtain a correction feature of the corresponding joint of the corresponding person in each to-be-reasoned sub-image (page 489, “The first stage generates heatmaps to produce an initial pose and the subsequent correction stage adjusts the initial pose obtained from the heatmaps by an Image-Guided Progressive GCN (IGP-GCN) module;” page 493, “we propose an Image-Guided graph network for correction which takes the initial pose generated from the above modules and adjusts the estimation results according to the implicit relationship of joints.”).
Xu is directed toward, “In this work, we focus on detecting human-object interactions (HOIs) in social scene images, which is demanding in terms of research and increasingly useful for practical applications. To undertake social tasks interacting with objects, humans direct their attention and move their body based on their intention. Based on this observation, we provide a unique computational perspective to explore human intention in HOI detection. Specifically, the proposed human intention driven HOI detection (iHOI) framework models human pose with (abstract).” Qiu is directed toward, “our framework estimates the invisible joints from an inference perspective by proposing an Image-Guided Progressive GCN module which provides a comprehensive understanding of both image context and pose struc ture. Moreover, existing benchmarks contain limited occlusions for evalu ation. Therefore, we thoroughly pursue this problem and propose a novel OPEC-Net framework together with a new Occluded Pose (OCPose) dataset with 9k annotated images (abstract).” As can be easily seen by one of ordinary skill in the art Xu and Qiu are directed toward similar methods of endeavor of image analysis and person action reasoning. Further, one of ordinary skill in the art before the effective filing date of the claimed invention would easily be aware that when humans perform actions, different body parts can occlude others, leading to confusing the system and generating an inaccurate output. Thus, it would have been obvious to a person having ordinary skill in the art before the effective filing date to incorporate the teaching of Qiu in order to ensure accurate joint analysis is performed, using constraints, to determine an accurate action output.
Regarding dependent claim 21, the rejection of claim 14 is incorporated herein. Additionally, Xu in the combination further discloses wherein the target detection network is trained by Visual Genome (VG) or Common Objects in Context (COCO) (page 5, right column, “The weights are pre trained on MS-COCO dataset.”).
Claim(s) 4-6 are rejected under 35 U.S.C. 103 as being unpatentable over Xu further in view of Qiu as applied to claim 2 above, and further in view of Yan, S., Xiong, Y., & Lin, D. (2018). Spatial Temporal Graph Convolutional Networks for Skeleton-Based Action Recognition. Proceedings of the AAAI Conference on Artificial Intelligence, 32(1). https://doi.org/10.1609/aaai.v32i1.12328 (hereinafter Yan).
Regarding dependent claim 4, the rejection of claim 2 is incorporated herein. Additionally, Xu and Qiu in the combination fails to explicitly disclose wherein the performing coding fusion on the joint feature and the corresponding occlusion probability of each joint in the current sub-image to obtain the corresponding fused feature information comprises:
extending the corresponding occlusion probability of the current sub-image into a d- dimensional sub-probability, and adding the d-dimensional sub-probability to a d-dimensional joint feature of the current sub-image in one-to-one correspondence to obtain the corresponding fused feature information of the current sub-image.
However, Yan discloses wherein the performing coding fusion on the joint feature and the corresponding occlusion probability of each joint in the current sub-image to obtain the corresponding fused feature information comprises:
extending the corresponding occlusion probability of the current sub-image into a d- dimensional sub-probability (page 7446, left column, “the feature vector on a node F(vti) consists of coordinate vectors, as well as estimation confidence, of the i-th joint on frame t.”), and adding the d-dimensional sub-probability to a d-dimensional joint feature of the current sub-image in one-to-one correspondence to obtain the corresponding fused feature information of the current sub-image (page 7446, left column, “the feature vector on a node F(vti) consists of coordinate vectors, as well as estimation confidence, of the i-th joint on frame t.” page 7449, right column, “The toolbox gives 2D coordinates (X,Y) in the pixel coordinate system and confidence scores C for the 18 human joints. We thus represent each joint with a tuple of (X,Y,C) and a skeleton frame is recorded as an array of 18 tuples. For the multi-person cases, we select the people with the highest average joint confidence in each clip. In this way, one clip with T frames is transformed into a skeleton sequence of these tuples. In practice, we represent the clips with tensors of (18,3,T) dimensions”).
As noted above, Xu and Qiu are directed toward similar methods of endeavor of image analysis and person action reasoning. Additionally, Yan is directed toward, “we propose a novel model of dynamic skeletons called Spatial Temporal Graph Convolutional Networks (ST-GCN), which moves beyond the limitations of previous methods by automatically learning both the spatial and temporal patterns from data (abstract).” As can be easily seen by one of ordinary skill in the art before the effective filing date of the claimed invention, Xu, Qiu and Yan are directed toward similar methods of endeavor of person action reasoning through joint analysis. Additionally, one of ordinary skill in the art before the effective filing date of the claimed invention would easily understand there are both spatial and temporal limitations on joints which can effect a joint model. Thus, it would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to incorporate the teaching of Yan in order to take into account both spatial and temporal features to ensure the joint model is both realistic and accurate.
Regarding dependent claim 5, the rejection of claim 4 is incorporated herein. Additionally, Yan in the combination further discloses wherein the adding the d- dimensional sub-probability to the d-dimensional joint feature of the current sub-image in one-to-one correspondence to obtain the corresponding fused feature information of the current sub- image comprises:
splicing the d-dimensional joint feature and one-dimensional occlusion sub-probability into a (d+1)-dimensional vector to obtain the corresponding fused feature information of the current sub-image (page 7446, left column, “the feature vector on a node F(vti) consists of coordinate vectors, as well as estimation confidence, of the i-th joint on frame t.” page 7449, right column, “The toolbox gives 2D coordinates (X,Y) in the pixel coordinate system and confidence scores C for the 18 human joints. We thus represent each joint with a tuple of (X,Y,C) and a skeleton frame is recorded as an array of 18 tuples. For the multi-person cases, we select the people with the highest average joint confidence in each clip. In this way, one clip with T frames is transformed into a skeleton sequence of these tuples. In practice, we represent the clips with tensors of (18,3,T) dimensions”).
Regarding dependent claim 6, the rejection of claim 4 is incorporated herein. Additionally, Yan in the combination further discloses wherein the adding the d- dimensional sub-probability to the d-dimensional joint feature of the current sub-image in one- to-one correspondence to obtain the corresponding fused feature information of the current sub- image comprises:
extending occlusion sub-probability into d dimensions, and then adding to the d- dimensional joint feature of the current sub-image in one-to-one correspondence to obtain the corresponding fused feature information of the current sub-image (page 7446, left column, “the feature vector on a node F(vti) consists of coordinate vectors, as well as estimation confidence, of the i-th joint on frame t.” page 7449, right column, “The toolbox gives 2D coordinates (X,Y) in the pixel coordinate system and confidence scores C for the 18 human joints. We thus represent each joint with a tuple of (X,Y,C) and a skeleton frame is recorded as an array of 18 tuples. For the multi-person cases, we select the people with the highest average joint confidence in each clip. In this way, one clip with T frames is transformed into a skeleton sequence of these tuples. In practice, we represent the clips with tensors of (18,3,T) dimensions”).
Allowable Subject Matter
Claims 8-13 and 17 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims, as well as correcting objections as noted above.
Claims 8-12:
The following is a statement of reasons for the indication of allowable subject matter: the closest prior arts of record teach methods of determining human intention when reviewing imaging data based on joint locations.
However, none of them alone or in any combination teaches compressing the image using a CNN into a multi-dimensional vector comprising data by compressing length and width, and then obtaining average pooling of the data to obtain a vector of the joint feature for each joint.
The closest prior art being Xu discloses the use of vectors to store data (see page 4, left column).
However, Xu fails to disclose compressing the image using a CNN into a multi-dimensional vector comprising data by compressing length and width, and then obtaining average pooling of the data to obtain a vector of the joint feature for each joint.
Claim 13:
The following is a statement of reasons for the indication of allowable subject matter: the closest prior arts of record teach methods of determining human intention when reviewing imaging data based on joint locations. However, none of them alone or in any combination teaches determining a prediction feature of a joint as the correction feature of the image when the occlusion probability is higher than an occlusion threshold, and if less than the threshold determining the joint feature as the correction feature.
The closest prior art being Qiu discloses performing pose correction at page 489, “To achieve this goal, two stages are proposed in our framework: Initial Pose Estimation and GCN-based Pose Correction. The first stage generates heatmaps to produce an initial pose and the subsequent correction stage adjusts the initial pose obtained from the heatmaps by an Image-Guided Progressive GCN (IGP-GCN) module.”
However, Qiu fails to disclose performing different operations as related to the correction feature based on thresholding.
Claim 17:
The following is a statement of reasons for the indication of allowable subject matter: the closest prior arts of record teach methods of determining human intention when reviewing imaging data based on joint locations. However, none of them alone or in any combination teaches determining if the occlusion probability is less than a threshold, outputting the joint feature in the manner encompassed with the 35 USC 112(f) interpretation as noted above.
The closest prior art being Qiu discloses performing pose correction at page 489, “To achieve this goal, two stages are proposed in our framework: Initial Pose Estimation and GCN-based Pose Correction. The first stage generates heatmaps to produce an initial pose and the subsequent correction stage adjusts the initial pose obtained from the heatmaps by an Image-Guided Progressive GCN (IGP-GCN) module.”
However, Qiu fails to disclose determining if the occlusion probability is less than a threshold, outputting the joint feature in the manner encompassed with the 35 USC 112(f) interpretation as noted above.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
U.S. Publication No. 2021/0104067 to Fu et al. discloses, “Embodiments provide functionality for identifying joints and limbs in images. An embodiment extracts features from an image to generate feature maps and, in turn, processes the feature maps using a single convolutional neural network trained based on a target model that includes joints and limbs (abstract).”
U.S. Patent No. 11,275,931 to Zhang et al. discloses, “A human pose prediction method is provided for an electronic device. The method includes using a basic neural network based on image-feature-based prediction to perform prediction on an inputted target image (abstract)”
Contact
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Courtney J. Nelson whose telephone number is (571)272-3956. The examiner can normally be reached Monday - Friday 8:00 - 4:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, John Villecco can be reached at 571-272-7319. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/COURTNEY JOAN NELSON/Primary Examiner, Art Unit 2661
Prosecution Insights