Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim(s) 1-6, 8, and 14- 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over US 2020/0394384 A1 to Singh in view of Deep Fully Connected Model for Collective Activity Recognition to Liu et al., hereinafter, “Liu” and A Short Review of Deep Learning Methods for Understanding Group and Crowd Activities to Borja-Borja et al., hereinafter, “Borja-Borja”.
Claim 1. Singh teaches A method for adaptive identification of suspicious activities, the method comprising: [0018] a real-time aerial suspicious analysis (ASANA) system that can detect one or more individuals engaged in suspicious activities from aerial images.
receiving image data captured by an image capturing device, [0020] at least one drone configured for capturing/recording one or more aerial images
wherein the image data comprises a plurality of frames; [0020] … one or more aerial images
for each frame of the plurality of frames, processing the frame, wherein processing comprises: [0052] the SHDL network 108 is trained with the Aerial Violent Individual (AVI) Dataset 116….where each image contains at least two individuals.
detecting, via a deep learning network, one or more individuals in the frame;
[0052] the SHDL network 108 is trained with the Aerial Violent Individual (AVI) Dataset 116….where each image contains at least two individuals.
[0027] detect individuals from the images recorded by the drone
and estimating, through the deep learning network, a pose of each of the one or more individuals; [0027] a YOLO detector is used to detect individuals from the images recorded by the drone and then a ScatterNet Hybrid Deep Learning (SHDL) Network performs the individual pose estimation.
and classifying, via a 3-dimensional (3D) Residual Network (ResNet), the behaviour dynamics as one of suspicious and non-suspicious. [0024] the orientations of the limbs of the estimated pose are used to identify suspicious individuals using the 3D ResNet (residual neural network). Preferably, 3D ResNet identifies the posture as one of the violent poses from the dataset and flags the individuals engaged as violent or suspicious.
[0026] the 3D ResNet classifies the individuals as either neutral (interpreted to be non-suspicious) or assigns a most likely suspicious or violent activity label using the estimated poses.
Singh fails to explicitly teach determining (deep learning) behaviour dynamics of the one or more individuals based on the pose estimated in each of the plurality of frames. Liu, in the same field of using deep learning for action recognition in image data teaches determining, via the deep learning network, FIGURE 2: CNN (deep learning) behaviour dynamics of the one or more individuals based on the pose estimated in each of the plurality of frames, [A. DATASETS INFORMATION AND ITS SPLIT METHOD] … It has five scene action labels: Crossing, Queuing, Walking, Talking, Waiting, which are performed by N people in each frame
[I. Introduction] …using CNN and LSTM network to obtain the dynamic information of the single-person behavior as well as the preliminary pre diction of the group activity; then using Conditional Random Field graphical model to describe the interactions between people in the group; finally integrating the knowledge from both CRF and deep learning network to refine the individual and group activity recognition
Thus, before the effective filing date of the present application, it would have been obvious to one of ordinary skill in the art to combine the teachings of Singh with the teachings of Liu [Abstract] to improve recognition accuracy over baseline methods.
Singh fails to explicitly teach the behaviour dynamics comprise at least one of: interaction patterns amongst the one or more individuals. Borja-Borja in the same field of using deep learning for action recognition in image data teaches wherein the behaviour dynamics comprise at least one of: interaction patterns amongst the one or more individuals; [II. Group Analysis] Group activity behavior analysis involves the analysis of people in a specific scene along with their interactions…
a deep neural-network-based hierarchical graphical model for individual and group activity recognition in surveillance scenes. This type of network is used to recognize the individual actions of people in a specific scene.
… understand how each person’s action is changing over time can be used to infer the group activity.
and an activity pattern of each of the one or more individuals; [II. Group Analysis] Person level actions: person action recognition is a first step toward recognizing group activities (low-semantic); temporal dynamics of a person’s action: understand how each person’s action is changing over time can be used to infer the group activity. Examiner interprets “temporal dynamics…person’s action is changing over time ” to be an activity pattern.
Thus, before the effective filing date of the present application, it would have been obvious to one of ordinary skill in the art to combine the teachings of Singh with the teachings of Borja-Borja [Introduction] to maintain public safety in crowds.
Claim 2. Singh and Liu teach wherein the image data is one of thermal image data, Infrared (IR) image data, or visible light image data. Singh [0020] … one or more aerial images. Specifying an image type is merely a design choice.
Liu [III. Conditional Random Field Model Based on Deep Learning Network] …the input RGB video images… Specifying an image type is merely a design choice.
Claim 3. Singh teaches wherein processing further comprises extracting, via the deep learning network, a set of features from each of the plurality of frames using a feature extraction technique. [0031] using a YOLO detector by performing analysis on the aerial images for extracting features from the captured/recorded image
[0052] the SHDL network 108 is trained with the Aerial Violent Individual (AVI) Dataset 116….where each image contains at least two individuals.
Claim 4. Singh teaches wherein the deep learning network is a hybrid deep learning network based on regression network. [0031] pose estimation of the individuals using a ScatterNet Hybrid Deep Learning (SHDL) Network to determine whether anomalies exist in the captured/recorded images
Claim 5. Singh and Liu teach further comprising identifying, via the deep learning network, at least one group of individuals from the one or more individuals in each of the plurality of frames. Singh [0052] the SHDL network 108 is trained with the Aerial Violent Individual (AVI) Dataset 116….where each image contains at least two individuals.
Liu [A. DATASETS INFORMATION AND ITS SPLIT METHOD] The first data-set used in our experiment is Collective Activity Dataset1 containing 44 video clips acquired using a low resolution hand-held camera. It has five scene action labels: Crossing, Queuing, Walking, Talking, Waiting, which are performed by N people in each frame
Claim 6. Liu teaches wherein determining the interaction patterns comprises: [Abstract] we propose a deep fully-connected model for group recognition, … to capture the dynamic features of each person. Then, we use the fully-connected conditional random field (FCCRF) to learn the interactions between people.
[Introduction] interaction information among individuals plays an important role in terms of collective activity recognition
for each group of the at least one group of individuals, [III. CONDITIONAL RANDOM FIELD MODEL BASED ON DEEP LEARNING NETWORK] the input RGB video images are fed to a spatial- temporal model based on CNN and LSTM, aiming to obtain posing and moving features for each person in the group
comparing, via the deep learning network, FIGURE 2: CNN (deep learning)
the pose of each individual of the group with the pose of each of remaining individuals of the group, [C. FULLY CONNECTED CONDITIONAL RANDOM FIELD MODEL] For example, the 3 persons in Figure 2 with red bounding boxes are observed. Their posture and the temporal information indicate they are all ‘‘talking’’. Besides, their relative positions are close, therefore, they have stronger interactive relationship pair-wisely compared (comparing) to the person in the yellow bounding box (interpreted as the remaining individuals) because his spacial-temporal information portrays ‘‘walking’’ which is different from ‘‘talking’’
in each of the plurality of frames; [A. DATASETS INFORMATION AND ITS SPLIT METHOD] The first data-set used in our experiment is Collective Activity Dataset1 containing 44 video clips acquired using a low resolution hand-held camera. It has five scene action labels: Crossing, Queuing, Walking, Talking, Waiting, which are performed by N people in each frame
and determining, via the deep learning network, the interaction patterns amongst the one or more individuals of the group based on the comparing. [C. FULLY CONNECTED CONDITIONAL RANDOM FIELD MODEL] That is, when two people are of similar feature information in the same group, they may have a bigger potential function value indicating a strong interaction (interaction patterns) between them
For example, the 3 persons in Figure 2 with red bounding boxes are observed. Their posture and the temporal information indicate they are all ‘‘talking’’. Besides, their relative positions are close, therefore, they have stronger interactive relationship (interaction patterns) pair-wisely compared (comparing) to the person in the yellow bounding box (interpreted as the remaining individuals) because his spacial-temporal information portrays ‘‘walking’’ which is different from ‘‘talking’’
Claim 8. Singh teaches wherein estimating, through the deep learning network, a pose of an individual from the one or more individuals comprises: identifying, via the deep learning network, a plurality of key points of the individual in the frame, [0020] a ScatterNet Hybrid Deep Learning (SHDL) Network for pose estimation of the detected individuals, where the ScatterNet Hybrid Deep Learning (SHDL) Network identifies fourteen key-points of a human body to form a skeleton structure of the detected individuals
wherein the plurality of key points corresponds to a plurality of body parts of the individual; [0020] where the ScatterNet Hybrid Deep Learning (SHDL) Network identifies fourteen key-points of a human body to form a skeleton structure of the detected individuals
[0031] identifying fourteen key-points of a human body to form a skeleton structure of the detected individuals
and estimating, via the deep learning network, the pose of the individual in the frame based on a position of each of the plurality of key points in the frame. [0029] each individual in the aerial image frame is annotated with several key-points which are utilized by the proposed ScatterNet Hybrid Deep Learning (SHDL) network as labels for learning pose estimation
Claim 14. Reviewed and analyzed in the same way as claim 1. See the above analysis and rationale.
Claim 15. Reviewed and analyzed in the same way as claim 2. See the above analysis and rationale.
Claim 16. Reviewed and analyzed in the same way as claim 3. See the above analysis and rationale.
Claim 18. Reviewed and analyzed in the same way as claim 5. See the above analysis and rationale.
Claim 19. Reviewed and analyzed in the same way as claim 6. See the above analysis and rationale.
Claim 17. Reviewed and analyzed in the same way as claim 4. See the above analysis and rationale.
Claim 20. Reviewed and analyzed in the same way as claim 1. See the above analysis and rationale.
Claim(s) 7 is/are rejected under 35 U.S.C. 103 as being unpatentable over US 2020/0394384 A1 to Singh in view of Deep Fully Connected Model for Collective Activity Recognition to Liu et al., hereinafter, “Liu” and A Short Review of Deep Learning Methods for Understanding Group and Crowd Activities to Borja-Borja et al., hereinafter, “Borja-Borja” to US 2014/0270707 A1 to Fathi et al., hereinafter, “Fathi”.
Claim 7. Singh fails to explicitly teach for each group of the at least one group of individuals. Fathi in the same field of using deep learning for action recognition in image data teaches further comprising: for each group of the at least one group of individuals, [0010] the environment 120 are other persons 131, 132 and 133. The illustration of three people besides the user of the camera 110 is exemplary; environment 120 may include any number of people. The persons 131, 132, and 133 may be part of the same social group
[0039] an interaction type is assigned to the sequence based on the state labels and their dependencies. A state can mean a particular pattern within a scene. For example, a state can correspond to four individuals A, B, C and D, where A, B and C are looking at D, and D is looking at A.
for each of the plurality of frames, [0014] Many of the actions described in method 200 are performed on a frame-by-frame basis, meaning that the steps described herein are performed on individual frames
comparing, via the 3D ResNet, [0039] In 255, the changes in the feature vectors determined in 250 are analyzed to characterize the social interactions present in the video… uses a Hidden Conditional Random Field ("HCRF") model, but other methods (3D ResNet is interpreted to be other method) of analyzing the change in the feature vectors…each frame (understood to be previous, current or next frame) is assigned a hidden state label based on its features, and then an interaction type is assigned
the interaction patterns amongst the one or more individuals in a current frame with the interaction patterns amongst the one or more individuals in a previous frame; [0020] In 270, a face is characterized using various clues that can be derived from the image, including patterns of attention of persons depicted in the image and first-person head movement. These clues can be interpreted for a single frame as well as interpreted over time using information from multiple frames (interpreted as current and previous frame) in a sequence.
[0021] Patterns of attention refers to recurring mannerisms in which people express interest in each other or with objects by paying attention to them. These recurring mannerisms can be identified as patterns in the imagery captured of a social interaction.
[0022] By analyzing the changes of the roles over time (understood to be the comparing), one can describe the patterns of turn taking and attention shift that are important indicators of social interactions. Social interactions are characterized by patterns of attention between individuals over time…
and classifying, via the 3D ResNet, the interaction patterns as one of suspicious or non-suspicious based on the comparing. 0022] the people in a scene can be classified into apparent roles in 240, and the change in roles over time can be used to characterize the social interactions (understood to be interaction patterns) depicted in the scene
[0042] the sequences of images are classified as to the social interactions depicted therein in a different manner, the recording captured in 210 has been classified as one or more types of social interactions (understood to be interaction patterns). Once the interactions have been classified, they are output (understood to be classifying where Singh
teaches the output to be suspicious or non-suspicious [0026] the 3D ResNet classifies the individuals as either neutral or assigns a most likely suspicious or violent activity label using the estimated poses.
Thus, before the effective filing date of the present application, it would have been obvious to one of ordinary skill in the art to combine the teachings of Singh with the teachings of Fathi [0002] to automatically or semi-automatically record the activity to the members of a group.
Claim(s) 9 is/are rejected under 35 U.S.C. 103 as being unpatentable over
US 2020/0394384 A1 to Singh in view of Deep Fully Connected Model for Collective Activity Recognition to Liu et al., hereinafter, “Liu” and A Short Review of Deep Learning Methods for Understanding Group and Crowd Activities to Borja-Borja et al., hereinafter, “Borja-Borja” to US 2020/0074165 A1 to Ghafoor et al., hereinafter, “Ghafoor”.
Claim 9. Singh fails to explicitly teach iteratively comparing, by the deep learning network. Ghafoor in the same field of using deep learning for action recognition in image data teaches further comprising: iteratively comparing, by the deep learning network, [0078] … the neural network could be a recursive neural network (e.g. a long-short term memory (LSTM) network) to analyse the identified pose estimates over a sequence of images…That is, the recurrent neural network (understood to be iterative processing) could operate to analyse the identified pose estimates in each of a plurality of (sequential) images to identify categorised actions from the change in the pose estimates over the images (identifying the change over a series of images/time is understood to be the comparing).
the plurality of key points of the individual in a current frame with the plurality of key points of the individual in a next frame, wherein the plurality of frames comprises the current frame and the next frame; [0078] The second class of examples involves using a neural network to analyse the identified pose estimates directly and identifying an action performed by a person depicted in the one or more images in dependence on their pose estimates in the one or more images. The pose estimates may be analysed for a single image, or over a sequence of multiple images (interpreted to be next frame, current frame and previous frames) . For example, the neural network could be a recursive neural network (e.g. a long-short term memory (LSTM) network) to analyse the identified pose estimates over a sequence of images. An action performed by a person depicted in those images can then be identified from changes in their pose estimate over the series of images
and determining, through the deep learning network, activity pattern of the individual based on the comparing. [0078] The second class of examples involves using a neural network to analyse the identified pose estimates directly and identifying an action (understood to be the activity pattern) performed by a person depicted in the one or more images in dependence on their pose estimates in the one or more images… An action performed by a person depicted in those images can then be identified from changes in their pose estimate over the series of images (identifying the change over a series of images/time is understood to be the comparing). That is, the recurrent neural network (deep learning network) could operate to analyse the identified pose estimates in each of a plurality of (sequential) images to identify categorised actions (understood to be the activity pattern) from the change in the pose estimates over the images (understood to be the comparing).
Thus, before the effective filing date of the present application, it would have been obvious to one of ordinary skill in the art to combine the teachings of Singh with the teachings of Ghafoor [0003] detection of humans within images can also be used to perform activity recognition.
Claim(s) 10-12 is/are rejected under 35 U.S.C. 103 as being unpatentable over US 2020/0394384 A1 to Singh in view of Deep Fully Connected Model for Collective Activity Recognition to Liu et al., hereinafter, “Liu” and A Short Review of Deep Learning Methods for Understanding Group and Crowd Activities to Borja-Borja et al., hereinafter, “Borja-Borja” to Deep Learning Approach for Suspicious Activity Detection from Surveillance Video to Amrutha et al., hereinafter, “Amrutha”.
Claim 10. Singh fails to explicitly teach the behaviour dynamics classified as suspicious is one of pre-trained suspicious behaviour dynamics or new suspicious behaviour dynamics. Amrutha, in the same field of using deep learning for action recognition in image data wherein the behaviour dynamics classified as suspicious is one of pre-trained suspicious behaviour dynamics or new suspicious behaviour dynamics. [D. Video Pre-processing] The proposed system is using a pre-trained model called VGG-16(Visual Geometry Group), which is trained on the ImageNet dataset…The model is able to predict suspicious or normal human behavior in the footage which is used to aid the monitoring process.
[D. Video Pre-processing] The system classifies the videos as suspicious (students using mobile phone, fighting, fainting) or normal (walking, running).
Thus, before the effective filing date of the present application, it would have been obvious to one of ordinary skill in the art to combine the teachings of Singh with the teachings of Amrutha [Introduction] a system which will automatically detects any unusual or abnormal situation in advance and a mechanism to alert the respective authority.
Claim 11. Borja-Borja teaches further comprising performing, by the 3D ResNet, incremental learning based on the new suspicious behavior dynamics [I. Introduction] recurrent neural networks are a family of neural networks for recurrent neural networks are a family of neural networks for the processing of sequential data (equivalent of incremental learning), where connections between units form a directed cycle, this permits you to show dynamic temporal behavior, also used to analyze sequences of actions. Examiner understands RNNs are specifically designed to process sequential or incremental data, where each new input depends on previous ones.
Claim 12. Singh teaches, wherein the 3D ResNet is deployed on a primary edge device. [0044] FIG. 1, the Aerial Suspicious Analysis (ASANA) system includes a drone 102 (interpreted to be an edge device) configured …a 3D ResNet 112…
Claim(s) 13 is/are rejected under 35 U.S.C. 103 as being unpatentable over US 2020/0394384 A1 to Singh in view of Deep Fully Connected Model for Collective Activity Recognition to Liu et al., hereinafter, “Liu” and A Short Review of Deep Learning Methods for Understanding Group and Crowd Activities to Borja-Borja et al., hereinafter, “Borja-Borja” to US 2021/0056387 A1 Eilert et al., hereinafter, “Eilert”.
Claim 13. The method of claim 12, wherein performing the incremental learning comprises: sharing information associated with the new suspicious behavior dynamics with a master 3D ResNet deployed on a cloud; [0027] the drone is configured with a processing device for onboard processing or a cloud server is used to perform computations in real-time
[0025] The estimated poses are used by the 3D ResNet (residual neural network) to identify suspicious individuals.
Singh fails to explicitly teach sharing information. Eilert, in the field of using distributed machine learning teaches [0019] …described herein that are transmitted from the user devices to the central device, server, or cloud hosting the master version of the ANN (equivalent of an ResNet)
Eilert teaches and disseminating, by the master 3D ResNet, the information associated with the new suspicious behavior dynamics to a plurality of 3D ResNets deployed on secondary edge devices. [0011] results outputted from the ANN can be communicated to other devices hosting other versions of the ANN, such as a device hosting a master version of the ANN.
Thus, before the effective filing date of the present application, it would have been obvious to one of ordinary skill in the art to combine the teachings of Singh with the teachings of Eilert [0004] to increase the effectiveness of such networks, complexities and challenges have been met by distributing training of artificial neural networks using multiple processors and/or distributed computing.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DELOMIA L GILLIARD whose telephone number is (571)272-1681. The examiner can normally be reached 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, John Villecco can be reached at (571) 272-7319. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DELOMIA L GILLIARD/Primary Examiner, Art Unit 2661
Prosecution Insights