CTNF 18/912,088 CTNF 86621 Notice of Pre-AIA or AIA Status 07-03-aia AIA 15-10-aia The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA. Note: Double patenting analysis was done with co-owned application number 17/328,356, there are extra features in the current claim language that in the Examiner’s opinion does not warrant a double patenting rejection. Such as “second number of people” and “a deep learning based analytic and a second number of people using a computer vision based analytic” which does only come out in dependent claims however only as “deep learning” and not the combination of computer vision and deep learning. Claim Rejections - 35 USC § 102 07-07-aia AIA 07-07 The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action: A person shall be entitled to a patent unless – 07-08-aia AIA (a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention. 07-15 AIA Claim s 1-3, 12-13 and 20 are rejected under 35 U.S.C. 102( a)(1 ) as being anticipated by Mlybari (US 2016/0259980) . As per claims 1 and 20, Mlybari teaches, method and system for determining a number of people in a space of a building (Mlybari, ¶[0048] “A people count is updated (e.g., increment, decrement) as a function of the direction of a movement of the track (e.g., exiting or entering a building).” This represents determining a number of people in a space of a building by being able to people count), the method comprising: capturing a video feed from each of one or more video cameras of the building (Mlybari, ¶[0053] “For example, each video camera (or other input source) may be associated with a geographic location identified using longitudinal and latitude coordinates. The geographic location of each video camera along with a unique camera identifier may be stored in the server 110.” This clearly represents capturing a video feed from each of one or more video cameras of the building ); processing one or more of the video feeds to identify a first number of people using a deep learning based analytic (Mlybari, ¶[0038] “Methods for feature learning include, but are not limited to, artificial neural networks, deep learning, genetic programming, granular computing, evolutionary computation, probabilistic heuristics, and metaheuristic programming. The extensive learning process is applied to decide which visual human features should be extracted in order to identify them.” This represents that a deep learning gets used and can be used. And ¶[0039] “he hybrid technique includes applying a granular computing process (e.g., fuzzy and rough sets) for modelling and evaluations, and applying deep learning and meta-heuristics for feature searching. Features includes, but are not limited to, head, shoulders, head peaks, covered head, western wear, Muslim wear, and Asian wear.” ) and a second number of people using a computer vision based analytic (Mlybari, fig.1 and ¶ [0056] In one example, an external device may access the server 110 and/or the computer 100 to obtain a people count at a specific location. For example, a user may check the crowd level at the specific location. The external device may include a computer, a tablet a smartphone or the like.” This clearly represent having computer vision with 106 and using computer 100); and determining a most probably number of people in the space based on the first number of people and the second number of people (Mlybari, ¶[0057] In one example, when the system include a single input system the camera identifier is automatically selected. A “result” pane 510 may show the people count when in a “Real-time” mode. And ¶ [0089] (10) The method of any one of features (1) to (9), further including defining a virtual line in a field of view of the video data; determining a movement direction of each tracked individual; and updating the people counter as a function of the virtual line and the movement direction of each tracked individual.” And ¶[0019] “[0019] The methodology described herein may be applied to live images sequences captured by surveillance video cameras. Analysis can be performed in real- time using a computer while the surveillance video cameras are capturing a live crowded environment. In one example, the methodologies described herein may be applied to the analysis of recorded or time-delayed videos.” And ¶[0031] “A Deep Neural Network (DNN) may be used. The DNN outputs a fixed number of bounding boxes. In addition, the DNN outputs a score for each box expressing the network confidence of this box containing an object. Each object box and its associated confidence are encoded as node values of the last net layer.” Deep learning is also helping in the counting ). As per claim 2, Mlybari teaches, the method of claim 1, wherein the deep learning based analytic comprises: detecting objects (Mlybari, ¶ [0036] “At step S302, the CPU 600 detects objects in the one or more videos.”); analyzing an overall shape of the detected objects to identify objects that correspond to people; and determine the first number of people based on the detected people (Mlybari, ¶ [0050] The virtual reference line may be of arbitrary shape, which may be user-defined and may be integrated into the one or more frames using video processing techniques as would be understood by one of ordinary skill in the art.” And ¶[0014] “[0014] Referring now to the drawings, wherein like reference numerals designate identical or corresponding parts throughout several views, the following description relates to systems and associated methodologies for real-time people detection, people tracking, and people counting in a crowd.” And people counting in a particular area). As per claim 3, Mlybari teaches, the method of claim 2, wherein analyzing the overall shape of the detected objects includes identifying one or more of heads, limbs, and shoulders to identify objects that correspond to people (Mlybari, ¶[0039] “[0039] In one example, the CPU 600 applies a hybrid technique. The hybrid technique includes applying a granular computing process (e.g., fuzzy and rough sets) for modelling and evaluations, and applying deep learning and meta-heuristics for feature searching. Features includes, but are not limited to, head, shoulders, head peaks, covered head, western wear, Muslim wear, and Asian wear.” This identifies heads, limbs, and shoulders to identify objects that correspond to people). As per claim 12, Mlybari teaches, the method of claim 1, comprising: tracking a track of one or more people in the building; and determining the most probably number of people in the space based on the first number of people, the second number of people and the track of one or more people in the building (Mlybari, ¶[0052] “For example, a building may have a plurality of entrances each equipped with the system described herein. The server 110 may acquire data from each of the plurality of entrances and analyze the data to determine, in a real-time, a total people count for the building.” Real time there would be a first and second number). As per claim 13, Mlybari teaches, the method of claim 12, wherein tracking the track of one or more people in the building comprises tracking a location of a device carried by each of the one or more people (Mlybari, fig.1, at least one user would have a device, as it is one or more people, such as a smartphone, as there is share processing ¶[0073] “The distributed components may include one or more client and server machines, which may share processing in addition to various human interface and communication devices (e.g., display monitors, smart phones, tablets, personal digital assistants (PDAs)).” And at least one of these persons would have a device) . Claim Rejections - 35 USC § 103 07-20-aia AIA The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action: A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made. 07-21-aia AIA Claim s 4-9 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over by Mlybari (US 2016/0259980) in view of Marciniak (T. Marciniak, A. Dąbrowski, A. Chmielewska and M. Nowakowski, "Real-time bi-directional people counting using video blob analysis," 2012 Joint Conference New Trends In Audio & Video And Signal Processing: Algorithms, Architectures, Arrangements And Applications (NTAV/SPA), Lodz, Poland, 2012, pp. 161-166. ) . As per claims 4 and 7, Mlybari teaches, the method of claim 2. Mlybari doesn’t clearly teach, wherein the computer vision based analytic comprises: subtracting a background image from one or more frames of a respective video feed to identify one or more blobs; comparing a size of each of the one or more blobs to an expected size of a person when the size of the blob is greater than the expected size of a person by more than a predetermined threshold, counting the blob as two or more people, otherwise counting the blob as one person or no person; and determining the second number of people based on the one or more blobs . However, Marciniak teaches, wherein the computer vision based analytic comprises: subtracting a background image from one or more frames of a respective video feed to identify one or more blobs; comparing a size of each of the one or more blobs to an expected size of a person (Marciniak, page 162 background represents background storage since it is being generated ) ; when the size of the blob is greater than the expected size of a person by more than a predetermined threshold (Marciniak, fig.6 a showing two people close together and C dividing them) , counting the blob as two or more people (Marciniak, fig.6 C blob being counted as 2 people, B no person ) , otherwise counting the blob as one person or no person; and determining the second number of people based on the one or more blobs (Marciniak, page 166 right column The results of operations on a 30 min sequence of the algorithm indicated that the 186 people entered the room and 175 came out, this represents a count in the field of view ). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of Marciniak with those of Mlybari to be able to tell the difference in blob size. The motivation would have been to improve the recognition count in a crowd. As per claim 5, Mlybari in view of Marciniak teaches, the method of claim 4, wherein determining the most probably number of people in the space includes adjusting the first number of people based on the second number of people (Mlybari, ¶[0051] In the real-time mode, the system can continue to update the learning parameters using the methodology described in FIG. 3. Thus, the system may re-compute and adjust the learned features in real-time.” This would change the number of people in real time). As per claim 6, Mlybari in view of Marciniak teaches, the method of claim 4, wherein determining the most probably number of people in the space includes incrementing or decrementing the first number of people based on the second number of people (Mlybari, ¶[0051] In the real-time mode, the system can continue to update the learning parameters using the methodology described in FIG. 3. Thus, the system may re-compute and adjust the learned features in real-time.” This would change the number of people in real time). As per claim 8, Mlybari in view of Marciniak teaches, the method of claim 7, wherein the computer vision based analytic comprises: determining a distance of each blob from a respective video camera based on a location of the blob in a field of view of the respective camera; and wherein the expected size of a person is dependent on the distance the blob is from the respective camera (Marciniak, fig.6 (c ) showing 1.5 the expected size of a person recognizing the blog are 2 persons). As per claims 9 and 17, Mlybari teaches, the method of claim 1. Mlybari doesn’t teach, however, Marciniak teaches, wherein the computer vision based analytic comprises: storing a background image of a field of view of a respective video feed (Marciniak, page 162 background represents background storage since it is being generated ) ; subtracting the background image from one or more frames of the respective video feed to identify one or more blobs (Marciniak, fig.5 Input on the left then the background then the blob without the background); determining a distance between the respective video camera and each of the one or more blobs (Marciniak, page 162 right column “We assume the typical parameters: an average human height of approx. 175cm and entrance of a building with width equal 2.5m and the min height of the camera mounted 3.9m, the greater the height the greater the accuracy” see fig.2, by having these parameters and the known height of the camera then the blobs distance to the camera can and is calculated ); comparing a size of each of the one or more blobs to an expected size of a person at the determined distance of the corresponding blob ( Marciniak, page 165, fig.5 the distance between the two subject blobs is taken and the box is the expected size of a person and above page 162 right column “We assume the typical parameters: an average human height of approx. 175cm and entrance of a building with width equal 2.5m and the min height of the camera mounted 3.9m, the greater the height the greater the accuracy” there is an assumption for an average ); when the size of the blob is greater than the expected size of a person at the determined distance of the corresponding blob by more than a predetermined threshold (Marciniak, fig.6 a showing two people close together and C dividing them), counting the blob as two or more people, otherwise counting the blob as one person or no person (Marciniak, fig.6 C blob being counted as 2 people, B no person ); and determining the second number of people based at least in part on the count of people assigned to each of the one or more blobs (Marciniak, page 166 right column The results of operations on a 30 min sequence of the algorithm indicated that the 186 people entered the room and 175 came out, this represents a count in the field of view ). Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of Marciniak with those of Mlybari to be able to tell the difference in blob size. The motivation would have been to improve the recognition count in a crowd . 07-21-aia AIA Claim s 10-11 are rejected under 35 U.S.C. 103 as being unpatentable over by Mlybari (US 2016/0259980) in view of Marciniak (T. Marciniak, A. Dąbrowski, A. Chmielewska and M. Nowakowski, "Real-time bi-directional people counting using video blob analysis," 2012 Joint Conference New Trends In Audio & Video And Signal Processing: Algorithms, Architectures, Arrangements And Applications (NTAV/SPA), Lodz, Poland, 2012, pp. 161-166. ) and in view of Xu (US 2010/0322516) . As per claim 10, Mlybari in view of Marciniak teaches the method of claim 1. Mlybari in view of Marciniak doesn’t clearly teach, wherein the computer vision based analytic processes a video feed captured by a non-overhead video camera of the building. However Xu teaches wherein the computer vision based analytic processes a video feed captured by a non-overhead video camera of the building (Xu, 4a and 4b the view from a non-overhead camera and ¶[0145] “a 4500 frames train approaches the platform from far side of the camera's 15:22:14-15:25:22 field of view (FOV)” and ¶[0106] “ If a large portion of a blob, say over 95%, is contained in the specified ROI for train detection, then the blob is incorporated into the calculations and a weight is assigned according to a scale variation model, or the weight is obtained by multiplying the percentage of pixels of the blob falling within the ROI and the distance between the blob's centre and the side of the image close to the camera's mounting position. This is shown in FIG. 15a and FIG. 15b, wherein blobs further away from the camera obtain more weight compared to the blobs close to the camera. As in the platform congestion estimation approach, a blob can be either dynamically congested or statically congested and the same respective procedures that are used for crowd analysis may also be applied to train detection.” This represents that a blob is found). At the time of the effective filing date, it would have been obvious to one of ordinary skill in the art to modify Mlybari in view of Marciniak to have the cameras be mounted on the side as be non-overhead as in Xu fig.4a and then the programming would change as well by implementing the camera’s position. The motivation would have been to reduce the number of cameras and getting a fuller picture with a side view. As per claim 11, Mlybari in view of Marciniak and Xu teaches, the method of claim 10, wherein the deep learning based analytic processes a video feed captured by an overhead video camera of the building (Mlybari, fig.1 106 showing overhead camera) . 07-21-aia AIA Claim s 14-19 are rejected under 35 U.S.C. 103 as being unpatentable over by Mlybari (US 2016/0259980) in view of Marciniak in view of Xu (US 2010/0322516) and Marciniak (T. Marciniak, A. Dąbrowski, A. Chmielewska and M. Nowakowski, "Real-time bi-directional people counting using video blob analysis," 2012 Joint Conference New Trends In Audio & Video And Signal Processing: Algorithms, Architectures, Arrangements And Applications (NTAV/SPA), Lodz, Poland, 2012, pp. 161-166. ) . As per claim 14, Mlybari teaches, a method for determining a number of people in a space of a building, the method comprising: capturing an overhead video feed from an overhead video camera having an overhead field of view (Mlybari, fig.1, 106 overhead camera); processing the overhead video feed to identify a first number of people using a deep learning based analytic (Mlybari, ¶[0083] “(4) The method of feature (3), in which the hybrid technique includes at least one of a granular computing process and a deep learning and meta-heuristics process.” This represents overhead video feed to identify a first number of people using a deep learning based analytic); and using the first number of people and the second number of people to determine a most probably number of people in the space (Mlybari, ¶[0051] In the real-time mode, the system can continue to update the learning parameters using the methodology described in FIG. 3. Thus, the system may re-compute and adjust the learned features in real-time.” This would change the number of people in real time). Mlybari doesn’t clearly teach, capturing a non-overhead video feed from a non-overhead video camera having a perspective field of view; processing the non-overhead video feed to identify a second number of people using a computer vision based analytic. However Xu teaches, capturing a non-overhead video feed from a non-overhead video camera having a perspective field of view; processing the non-overhead video feed to identify a second number of people using a computer vision based analytic (Xu, 4a and 4b the view from a non-overhead camera and ¶[0145] “a 4500 frames train approaches the platform from far side of the camera's 15:22:14-15:25:22 field of view (FOV)” and ¶[0106] “ If a large portion of a blob, say over 95%, is contained in the specified ROI for train detection, then the blob is incorporated into the calculations and a weight is assigned according to a scale variation model, or the weight is obtained by multiplying the percentage of pixels of the blob falling within the ROI and the distance between the blob's centre and the side of the image close to the camera's mounting position. This is shown in FIG. 15a and FIG. 15b, wherein blobs further away from the camera obtain more weight compared to the blobs close to the camera. As in the platform congestion estimation approach, a blob can be either dynamically congested or statically congested and the same respective procedures that are used for crowd analysis may also be applied to train detection.” This represents that a blob is found). At the time of the effective filing date, it would have been obvious to one of ordinary skill in the art to modify Mlybari to have the cameras be mounted on the side as be non-overhead as in Xu fig.4a and then the programming would change as well by implementing the camera’s position. The motivation would have been to reduce the number of cameras and getting a fuller picture with a side view. As per claim 15, Mlybari in view of Xu teaches, the method of claim 14, wherein the deep learning based analytic comprises: detecting objects in the overhead video feed; analyzing an overall shape of the detected objects to identify objects that correspond to people; and determine the first number of people based on the detected people (Mlybari, ¶[0051] In the real-time mode, the system can continue to update the learning parameters using the methodology described in FIG. 3. Thus, the system may re-compute and adjust the learned features in real-time.” This would change the number of people in real time). As per claim 16, Mlybari in view of Xu teaches, the method of claim 15 , wherein analyzing the overall shape of the detected objects includes identifying one or more of heads, limbs, and shoulders to identify objects that correspond to people (Mlybari, ¶[0039] “[0039] In one example, the CPU 600 applies a hybrid technique. The hybrid technique includes applying a granular computing process (e.g., fuzzy and rough sets) for modelling and evaluations, and applying deep learning and meta-heuristics for feature searching. Features includes, but are not limited to, head, shoulders, head peaks, covered head, western wear, Muslim wear, and Asian wear.” This identifies heads, limbs, and shoulders to identify objects that correspond to people). As per claim 17, Mlybari in view of Marciniak and Xu teaches the method of claim 15, wherein the computer vision based analytic comprises: comparing a background image for the non-overhead video feed to one or more frames of the non-overhead video feed to identify one or more blobs (Xu, 4a and 4b the view from a non-overhead camera and ¶[0145] “a 4500 frames train approaches the platform from far side of the camera's 15:22:14-15:25:22 field of view (FOV)” and ¶[0106] “ If a large portion of a blob, say over 95%, is contained in the specified ROI for train detection, then the blob is incorporated into the calculations and a weight is assigned according to a scale variation model, or the weight is obtained by multiplying the percentage of pixels of the blob falling within the ROI and the distance between the blob's centre and the side of the image close to the camera's mounting position. This is shown in FIG. 15a and FIG. 15b, wherein blobs further away from the camera obtain more weight compared to the blobs close to the camera. As in the platform congestion estimation approach, a blob can be either dynamically congested or statically congested and the same respective procedures that are used for crowd analysis may also be applied to train detection.” This represents that a blob is found); determining a distance between the non-overhead video camera and each of the one or more blobs (Marciniak, page 162 background represents background storage since it is being generated); comparing a size of each of the one or more blobs to an expected size of a person at the determined distance of the corresponding blob; when the size of the blob is greater than the expected size of a person at the determined distance of the corresponding blob by more than a predetermined threshold ( Marciniak, page 165, fig.5 the distance between the two subject blobs is taken and the box is the expected size of a person and above page 162 right column “We assume the typical parameters: an average human height of approx. 175cm and entrance of a building with width equal 2.5m and the min height of the camera mounted 3.9m, the greater the height the greater the accuracy” there is an assumption for an average and the determined distance of the corresponding blob by more than a predetermined threshold ), counting the blob as two or more people, otherwise counting the blob as one person or no person (Marciniak, page 166 right column The results of operations on a 30 min sequence of the algorithm indicated that the 186 people entered the room and 175 came out, this represents a count in the field of view ); and determining the second number of people based at least in part on the count of people assigned to each of the one or more blobs (Marciniak, page 166 right column The results of operations on a 30 min sequence of the algorithm indicated that the 186 people entered the room and 175 came out, this represents a count in the field of view ). As per claim 18, Mlybari in view of Marciniak and Xu teaches, the method of claim 14, comprising: tracking a track of one or more people in the building; and using the first number of people, the second number of people and the track of one or more people to determine the most probably number of people in the space (Mlybari, ¶[0051] In the real-time mode, the system can continue to update the learning parameters using the methodology described in FIG. 3. Thus, the system may re-compute and adjust the learned features in real-time.” This would change the number of people in real time). As per claim 19, Mlybari in view of Marciniak and Xu teaches, the method of claim 14, wherein using the first number of people and the second number of people to determine the most probably number of people in the space comprises incrementing or decrementing the first number of people based on the second number of people (Mlybari, ¶[0051] In the real-time mode, the system can continue to update the learning parameters using the methodology described in FIG. 3. Thus, the system may re-compute and adjust the learned features in real-time.” This would change the number of people in real time any sort of arrangement can happen if real time counting is happening and would be obvious to try.) Close Prior Art Sulzer (US 2020/0364468) is a close prior art. Abstract “An intelligent video surveillance system is disclosed which performs real-time analytics on a live video stream. The system includes a training database populated with frames of actual video of objects of interest taken in a relevant environment. A subset of the frames include bounding boxes and/or bounding polygons which can be augmented. The training database also includes classification/annotation of data/labels relevant to the object of interest, a person carrying the object of interest, and/or the background or environment. The training database is searchable by the classification/annotation of data/labels.” Which would be able to yield a total count of persons in a building. This prior art should be considered if amending the claims. Chan (US 2020/0387718) Abstract “A system and a method for counting objects includes the steps of: obtaining a plurality of images representing the objects to be counted in a target area; generating a map for each of the plurality of images representing an identification of each of the corresponding objects; and fusing the plurality of maps being generated to obtain a scene-level density map representing a count of the objects in the target area.” This would clearly read on the counting of people in a building and first and second count as well. This prior art should also be considered if amending the claim language. Conclusion Any inquiry concerning this communication or earlier communications from the examiner should be directed to SANTIAGO GARCIA whose telephone number is (571)270-5182. The examiner can normally be reached Monday-Friday 9:30am-5:30pm. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chineyere Wills-Burns can be reached at (571) 272-9752. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. /SANTIAGO GARCIA/Primary Examiner, Art Unit 2673 /SG/ Application/Control Number: 18/912,088 Page 2 Art Unit: 2673 Application/Control Number: 18/912,088 Page 3 Art Unit: 2673 Application/Control Number: 18/912,088 Page 4 Art Unit: 2673 Application/Control Number: 18/912,088 Page 5 Art Unit: 2673 Application/Control Number: 18/912,088 Page 6 Art Unit: 2673 Application/Control Number: 18/912,088 Page 7 Art Unit: 2673 Application/Control Number: 18/912,088 Page 8 Art Unit: 2673 Application/Control Number: 18/912,088 Page 9 Art Unit: 2673 Application/Control Number: 18/912,088 Page 10 Art Unit: 2673 Application/Control Number: 18/912,088 Page 11 Art Unit: 2673 Application/Control Number: 18/912,088 Page 12 Art Unit: 2673 Application/Control Number: 18/912,088 Page 13 Art Unit: 2673 Application/Control Number: 18/912,088 Page 14 Art Unit: 2673 Application/Control Number: 18/912,088 Page 15 Art Unit: 2673
Prosecution Insights