DETAILED ACTION
Notice of Pre-AIA or AIA Status
Claims 1-11 are pending in this application.
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Specification
The title of the invention is not descriptive. A new title is required that is clearly indicative of the invention to which the claims are directed.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent may not be obtained though the invention is not identically disclosed or described as set forth in section 102 of this title, if the differences between the subject matter sought to be patented and the prior art are such that the subject matter as a whole would have been obvious at the time the invention was made to a person having ordinary skill in the art to which said subject matter pertains. Patentability shall not be negatived by the manner in which the invention was made.
Claims 1-11 are rejected under 35 U.S.C. 103 as being unpatentable over Kulandai-Samy et al. (US PGPub US20240135687A1), hereby referred to as “Kulandai-Samy”, in view of Lin et al. (US Patent 9,251,410), hereby referred to as “Lin”.
Consider Claims 1, 10 and 11.
Kulandai-Samy teaches:
1. [Claim 1] (Currently amended) A traffic flow analysis device comprising: / [Claim 10] (Currently amended) A traffic flow analysis method comprising: / 11. [Claim 11] (Currently amended) A non-transitory computer-readable program recording medium storing a program for causing a computer to execute: (Kulandai-Samy: abstract, Disclosed herein is an object detection system, including apparatuses and methods for object detection. An implementation may include receiving a first class of a first object depicted in an image frame from a classification model and subsequently receiving a second image frame. The implementation further includes predicting, using a classification tracking model, that the classification model will output the first class for the second image frame and then detecting whether the first class is in fact outputted. The implementation includes determining that the second image frame should be added to a training dataset for the classification model when detecting that the classification model did not generate the first class for the second image frame as predicted and re-training the classification model using the training dataset. [0010] [0112])
1. a memory configured to store instructions; and one or more processors configured to execute the instructions to: (KULANDAI-SAMY: [0047] FIG. 3 is a block diagram of computing device 300 executing detection training component 315, in accordance with exemplary aspects of the present disclosure. FIG. 4 is a flowchart illustrating method 400 of re-training a region of interest (ROI) detection model to fix detection misses, in accordance with exemplary aspects of the present disclosure. Referring to FIG. 3 and FIG. 4 , in operation, computing device 300 may perform method 400 of re-training a region of interest (ROI) detection model to fix detection misses via execution of detection training component 315 by one or more processors 305 and/or one or more memories 310. [0048]-[0049])
1. acquire an image from a camera installed at a location where a moving object subject to traffic flow analysis is configured to be imaged; (KULANDAI-SAMY: [0048] At block 402, the method 400 includes receiving a first image frame from an ROI detection model that is configured to detect an object in an image and generate an ROI boundary around the object, wherein the first image frame comprises a first ROI boundary around a first object. For example, in an aspect, computer device 300, one or more processors 305, one or more memories 310, detection training component 315, and/or receiving component 320 may be configured to or may comprise means for receiving image 100 from an ROI detection model that is configured to detect persons in an image and generate an ROI boundary around the object. The first image frame may include ROI boundary 112 around person 110.)
1.store a plurality of types of identification methods for identifying an attribute of the moving object captured by the camera; 1.select an identification method suitable for a tendency of the moving object captured by the camera from the plurality of types of stored identification methods; / 10. selecting, from a plurality of types of identification methods for identifying an attribute of a moving object captured by a camera installed at a location where the moving object subject to traffic flow analysis is configured to be imaged, an identification method suitable for a tendency of the moving object captured by the camera; / 11. selecting, from a plurality of types of identification methods for identifying an attribute of a moving object captured by the camera installed at a location where the moving object subject to traffic flow analysis is configured to be imaged, an identification method suitable for a tendency of the moving object captured by the camera; (KULANDAI-SAMY: [0052] At block 408, the method 400 includes detecting whether the first ROI boundary is present in the second image frame. For example, in an aspect, computer device 300, one or more processors 305, one or more memories 310, detection training component 315, and/or detecting component 330 may be configured to or may comprise means for detecting whether ROI boundary 112 is present in image 120. For example, detecting component 330 may search for a set of pixels resembling a boundary (e.g., of any shape) that is found in image 100 around person 110 in image 120.[0053] If detecting component 330 determines that the first ROI boundary is not present, method 400 advances to block 410. If the first ROI boundary is detected in the second image frame, method 400 advances 414. [0054] At block 410, the method 400 includes determining that the second image frame should be added to a training dataset for the ROI detection model. For example, in an aspect, computer device 300, one or more processors 305, one or more memories 310, detection training component 315, and/or determining component 335 may be configured to or may comprise means for determining that image 120 should be added to a training dataset for the ROI detection model. In some aspects, an ROI boundary is added to image 120 around person 110 where person 110 is located. This updated image is then added to the training dataset.)
10. acquiring an image from the camera; / 11. acquiring an image from the camera; (KULANDAI-SAMY: [0048] At block 402, the method 400 includes receiving a first image frame from an ROI detection model that is configured to detect an object in an image and generate an ROI boundary around the object, wherein the first image frame comprises a first ROI boundary around a first object. For example, in an aspect, computer device 300, one or more processors 305, one or more memories 310, detection training component 315, and/or receiving component 320 may be configured to or may comprise means for receiving image 100 from an ROI detection model that is configured to detect persons in an image and generate an ROI boundary around the object. The first image frame may include ROI boundary 112 around person 110. [0049] At block 404, the method 400 includes receiving, from the ROI detection model, a second image frame that is a subsequent frame to the first image frame in a video. For example, in an aspect, computer device 300, one or more processors 305, one or more memories 310, detection training component 315, and/or receiving component 320 may be configured to or may comprise means for receiving, from the ROI detection model, image 120 that is a subsequent frame to image 100 in a security surveillance stream.)
1.and identify a moving object appearing in the acquired image and an attribute of the moving object using the selected identification method. / 10. and identifying an attribute of a moving object appearing in the acquired image using the selected identification method. / 11. and identifying an attribute of a moving object appearing in the acquired image using the selected identification method. (KULANDAI-SAMY: [0052]-[0055] At block 412, the method 400 includes re-training the ROI detection model using the training dataset comprising the second image frame. For example, in an aspect, computer device 300, one or more processors 305, one or more memories 310, detection training component 315, and/or re-training component 340 may be configured to or may comprise means for re-training the ROI detection model using the training dataset comprising image 120. For example, re-training component 340 may execute a training algorithm to update the weights in the ROI detection model that are used to classify objects. This training algorithm may use techniques such as gradient descent. Because the images in the training dataset include examples of objects that the ROI detection model failed to detect previously, the updated weights will enable the ROI detection model to learn how to detect the missed objects. Accordingly, for example, the re-trained ROI detection model will generate the first ROI boundary (e.g., ROI boundary 112) around the first object (e.g., person 110) in any subsequently inputted image frame depicting the first object. [0058] FIG. 5 is a flowchart illustrating method 500 of selecting frames for a training dataset, in accordance with exemplary aspects of the present disclosure. Method 500 may be executed by detection training component 315 when, at block 408, detecting component 330 determines that the first ROI boundary is not present in the second image frame. Prior to determining that the second image frame should be added to the training set, method 500 may be initiated at either block 502, block 506, block 508, or block 510.)
Even if KS does not teach: select an identification method suitable for a tendency of the moving object captured by the camera from the plurality of types of stored identification methods
Lin teaches:
1. [Claim 1] (Currently amended) A traffic flow analysis device comprising: / [Claim 10] (Currently amended) A traffic flow analysis method comprising: / 11. [Claim 11] (Currently amended) A non-transitory computer-readable program recording medium storing a program for causing a computer to execute: (Lin: abstract, A people counting system includes: a top-view, a first and a second side-view image-capturing device, capturing a top-view, a first and a second side-view image respectively; an image stitching module, stitching the top-view, the first and the second side-view image into an ultra wide-angle image; a ROI selecting module, selecting at least one recognition zone and a counting zone; a face recognition module, monitoring the recognition zone to determine a face location corresponding to a face through analyzing the recognition zone; a head recognition module, monitoring the counting zone to determine a head location corresponding to a head through analyzing the counting zone; an object tracking module, the head recognition module, generating a face track and a head track; and a people counting module, counting a first number of face tracks and a second number of head tracks passing through the counting zone and generating a counting result.)
1. a memory configured to store instructions; and one or more processors configured to execute the instructions to: (Lin: column 2 lines 55-67, column 3 lines 1-12, Figure 1, It should be noted further that, unless indicated otherwise, all functions described herein may be performed in hardware or as software instructions for enabling a computer to perform predetermined operations, where the software instructions are embodied on a computer-readable storage medium, such as RAM, a hard drive, flash memory or other type of computer-readable storage medium known to a person of ordinary skill in the art. In certain embodiments, the predetermined operations of the computer are performed by a processor such as a computer or an electronic data processor in accordance with code such as computer program code, software, firmware, and, in some embodiments, integrated circuitry that is coded to perform such functions. Furthermore, it should be understood that various operations described herein as being performed by a user may be operations manually performed by the user, or may be automated processes performed either with or without instructions provided by the user. FIG. 1 is an environment schematic diagram including a people counting system 100 according to an embodiment of the present invention. The people counting system 100 at least comprises a top-view image-capturing device 110, a first side-view image-capturing device 120, a second side-view image-capturing device 130 and a processor 140. The processor 140 is electrically coupled to the top-view image-capturing device 110, the first side-view image-capturing device 120 and the second side-view image-capturing device 130.)
1. acquire an image from a camera installed at a location where a moving object subject to traffic flow analysis is configured to be imaged; (Lin: column 3 lines 5-34, Figure 1, FIG. 1 is an environment schematic diagram including a people counting system 100 according to an embodiment of the present invention. The people counting system 100 at least comprises a top-view image-capturing device 110, a first side-view image-capturing device 120, a second side-view image-capturing device 130 and a processor 140. The processor 140 is electrically coupled to the top-view image-capturing device 110, the first side-view image-capturing device 120 and the second side-view image-capturing device 130. As shown in FIG. 1, the top-view image-capturing device 110 is installed between the first side-view image-capturing device 120 and the second side-view image-capturing device 130 and is used to capture a top-view image. The first side-view image-capturing device 120 and the second side-view image-capturing device 130 are used to capture a first side-view image and a second side-view image, respectively, wherein the first side-view image-capturing device 120 and the second side-view image-capturing device 130 are installed at angles where the first side-view image-capturing device 120 and the second side-view image-capturing device 130 can capture the face. The first side-view image-capturing device 120 and the second side-view image-capturing device 130 may be a device or an apparatus, such as a webcam, that can capture images or videos. In addition, the field of view (FOV) of the top-view image-capturing device 110 is a top-view area 112. The fields of view of the first side-view image-capturing device 120 and the second side-view image-capturing device 130 are a first side-view area 122 and a second side-view area 132, respectively.)
1.store a plurality of types of identification methods for identifying an attribute of the moving object captured by the camera; 1.select an identification method suitable for a tendency of the moving object captured by the camera from the plurality of types of stored identification methods; / 10. selecting, from a plurality of types of identification methods for identifying an attribute of a moving object captured by a camera installed at a location where the moving object subject to traffic flow analysis is configured to be imaged, an identification method suitable for a tendency of the moving object captured by the camera; / 11. selecting, from a plurality of types of identification methods for identifying an attribute of a moving object captured by the camera installed at a location where the moving object subject to traffic flow analysis is configured to be imaged, an identification method suitable for a tendency of the moving object captured by the camera; (Lin: column 3 lines 35-41, FIG. 2 is a schematic diagram of the processor 140 according to an embodiment of the present invention. As shown in FIG. 2, the processor 140 comprises an image stitching module 210, a region of interest (ROI) selecting module 220, a face recognition module 230, a head recognition module 240, an object tracking module 250 and a people counting module 260. Column 4 lines 5-35 The ROI selecting module 220 is coupled to the image stitching module 210 and is used to select at least one recognition zones and a counting zone from the ultra wide-angle image. In a preferred embodiment, the ROI selecting module 220 selects two recognition zones and a counting zone so that the ROI selecting module 220 can further determine a people moving direction (a person moves from left to right or from right to left). If the ROI selecting module 220 selects a single recognition zone, the ROI selecting module 220 merely determines a single direction (from left to right or from right to left). As shown in FIG. 4, the counting zone 420 is between the two recognition zones 420 and 430. In addition, the ranges of the counting zone and the recognition zone can be determined by the people counting system, or can also be determined by a user. The face recognition module 230 is coupled to the ROI selecting module 220, receives the images in the recognition zone and monitors the recognition zone to determine a face location corresponding to a face through analyzing the recognition zone, wherein the face detection technology can be well-known technologies, such as the Viola-Jones algorithm or another technology. These technologies do not need to be illustrated elaborately. The head recognition module 240 is coupled to the ROI selecting module 220 and receives the images in the counting zone, as shown in FIG. 5.)
10. acquiring an image from the camera; / 11. acquiring an image from the camera; (Lin: column 3 lines 35-67, column 4 lines 1-30, Figures 2 and 5; FIG. 2 is a schematic diagram of the processor 140 according to an embodiment of the present invention. As shown in FIG. 2, the processor 140 comprises an image stitching module 210, a region of interest (ROI) selecting module 220, a face recognition module 230, a head recognition module 240, an object tracking module 250 and a people counting module 260. The image stitching module 210 is coupled to the top-view image-capturing device 110, the first side-view image-capturing device 120 and the second side-view image-capturing device 130 and is used to receive the top-view image, the first side-view image and the second side-view image captured by the image-capturing devices. Then, the image stitching module 210 stitches the top-view image, the first side-view image and the second side-view image into a wide-angle image or an ultra wide-angle image.)
1.and identify a moving object appearing in the acquired image and an attribute of the moving object using the selected identification method. / 10. and identifying an attribute of a moving object appearing in the acquired image using the selected identification method. / 11. and identifying an attribute of a moving object appearing in the acquired image using the selected identification method. (Lin: column 3 lines 35-67, column 4 lines 1-30, Figures 2-3 and 5; The image stitching module 210 is coupled to the top-view image-capturing device 110, the first side-view image-capturing device 120 and the second side-view image-capturing device 130 and is used to receive the top-view image, the first side-view image and the second side-view image captured by the image-capturing devices. Then, the image stitching module 210 stitches the top-view image, the first side-view image and the second side-view image into a wide-angle image or an ultra wide-angle image. The image stitching technology can use an image features extraction method, a spatial relation method or other technology to stitch the image. For example, as shown in FIG. 3, the image features extraction method is used to extract feature points from the top-view image 310, the first side-view image 320 and the second side-view image 330, respectively, such as the asterisks in the top-view image 310 and the first side-view image 320, and the diamonds in the top-view image 310 and the second side-view image 330. The image stitching module 210 obtains the corresponding relationship of the feature points by comparing the feature points, and stitches the top-view image 310, the first side-view image 320 and the second side-view image 330 into the ultra wide-angle image 340 according to the corresponding relationship. For another example, the spatial relation method is used to obtain a corresponding relationship according to positions in which the top-view image-capturing device 110, the first side-view image-capturing device 120 and the second side-view image-capturing device 130 are installed and stitch the top-view image 310, the first side-view image 320 and the second side-view image 330 into the ultra wide-angle image according to the corresponding relationship. The ROI selecting module 220 is coupled to the image stitching module 210 and is used to select at least one recognition zones and a counting zone from the ultra wide-angle image. In a preferred embodiment, the ROI selecting module 220 selects two recognition zones and a counting zone so that the ROI selecting module 220 can further determine a people moving direction (a person moves from left to right or from right to left). If the ROI selecting module 220 selects a single recognition zone, the ROI selecting module 220 merely determines a single direction (from left to right or from right to left). As shown in FIG. 4, the counting zone 420 is between the two recognition zones 420 and 430. In addition, the ranges of the counting zone and the recognition zone can be determined by the people counting system, or can also be determined by a user.)
It would have been obvious before the effective filing date of the claimed invention to one of ordinary skill in the art to modify the people counting system and method of Kulandai-Samy and refine it with features from the object detection and learning algorithm of Lin in order to ensure enhanced accuracy in traffic control machine learning algorithms. The determination of obviousness is predicated upon the following findings: both references are directed towards the overall field of image processing analysis using machine learning and one skilled in the art would have been motivated to modify Kulandai-Samy with Lin in order to ensure a more accurate and enhanced person count in traffic and pedestrian control applications and ensure a more robust machine learning model. Furthermore, the prior art collectively includes each element claimed (though not all in the same reference), and one of ordinary skill in the art could have combined the elements in the manner explained above using known engineering design, interface and/or programming techniques, without changing a “fundamental” operating principle of Kulandai-Samy, while the teaching of Lin continues to perform the same function as originally taught prior to being combined, in order to produce the repeatable and predictable result of leveraging a series of features pertinent to crowd control and person counting in the overall learning process. It is for at least the aforementioned reasons that the examiner has reached a conclusion of obviousness with respect to the claim in question.
Consider Claim 2.
The combination of Kulandai-Samy and Lin teaches:
[Claim 2] (Currently amended) The traffic flow analysis device according to claim 1, wherein the one or more processors are further configured to: select the identification method based on a position of the camera. (KULANDAI-SAMY: [0052]-[0055] At block 412, the method 400 includes re-training the ROI detection model using the training dataset comprising the second image frame. For example, in an aspect, computer device 300, one or more processors 305, one or more memories 310, detection training component 315, and/or re-training component 340 may be configured to or may comprise means for re-training the ROI detection model using the training dataset comprising image 120. For example, re-training component 340 may execute a training algorithm to update the weights in the ROI detection model that are used to classify objects. This training algorithm may use techniques such as gradient descent. Because the images in the training dataset include examples of objects that the ROI detection model failed to detect previously, the updated weights will enable the ROI detection model to learn how to detect the missed objects. Accordingly, for example, the re-trained ROI detection model will generate the first ROI boundary (e.g., ROI boundary 112) around the first object (e.g., person 110) in any subsequently inputted image frame depicting the first object. [0058] FIG. 5 is a flowchart illustrating method 500 of selecting frames for a training dataset, in accordance with exemplary aspects of the present disclosure. Method 500 may be executed by detection training component 315 when, at block 408, detecting component 330 determines that the first ROI boundary is not present in the second image frame. Prior to determining that the second image frame should be added to the training set, method 500 may be initiated at either block 502, block 506, block 508, or block 510. Lin: column 3 lines 35-67, column 4 lines 1-30, Figures 2-3 and 5; The image stitching module 210 is coupled to the top-view image-capturing device 110, the first side-view image-capturing device 120 and the second side-view image-capturing device 130 and is used to receive the top-view image, the first side-view image and the second side-view image captured by the image-capturing devices. Then, the image stitching module 210 stitches the top-view image, the first side-view image and the second side-view image into a wide-angle image or an ultra wide-angle image. The image stitching technology can use an image features extraction method, a spatial relation method or other technology to stitch the image. For example, as shown in FIG. 3, the image features extraction method is used to extract feature points from the top-view image 310, the first side-view image 320 and the second side-view image 330, respectively, such as the asterisks in the top-view image 310 and the first side-view image 320, and the diamonds in the top-view image 310 and the second side-view image 330. The image stitching module 210 obtains the corresponding relationship of the feature points by comparing the feature points, and stitches the top-view image 310, the first side-view image 320 and the second side-view image 330 into the ultra wide-angle image 340 according to the corresponding relationship. For another example, the spatial relation method is used to obtain a corresponding relationship according to positions in which the top-view image-capturing device 110, the first side-view image-capturing device 120 and the second side-view image-capturing device 130 are installed and stitch the top-view image 310, the first side-view image 320 and the second side-view image 330 into the ultra wide-angle image according to the corresponding relationship. The ROI selecting module 220 is coupled to the image stitching module 210 and is used to select at least one recognition zones and a counting zone from the ultra wide-angle image. In a preferred embodiment, the ROI selecting module 220 selects two recognition zones and a counting zone so that the ROI selecting module 220 can further determine a people moving direction (a person moves from left to right or from right to left). If the ROI selecting module 220 selects a single recognition zone, the ROI selecting module 220 merely determines a single direction (from left to right or from right to left). As shown in FIG. 4, the counting zone 420 is between the two recognition zones 420 and 430. In addition, the ranges of the counting zone and the recognition zone can be determined by the people counting system, or can also be determined by a user.)
Consider Claim 3.
The combination of Kulandai-Samy and Lin teaches:
[Claim 3] (Currently amended) The traffic flow analysis device according to claim 1, wherein the one or more processors are further configured to: select the identification method based on a time zone in which the image is captured. (Lin: column 4 lines 59-67, column 5 lines 1-12, The people counting module 260 is coupled to the object tracking module 250 and is used to count a first number of face tracks passing through the counting zone during a time interval and count a second number of head tracks passing through the counting zone during the time interval. Then, the people counting module 260 compares the first number with the second number to generate a people counting result. For example, during the time interval T, when the people counting module 260 determines that the first number is equal to the second number, it represents the number of heads passing through the counting zone being the same as the number of faces passing through the counting zone, and the number can be adopted. When the people counting module 260 finds that the first number is 1 and the second number is 0 during the time interval T, the number cannot be adopted. For example, the object tracking module 250 detects that a person passes through the counting zone from right to left, but the object tracking module 250 does not detect any head track. Since the first and second numbers are different because of errors caused by the people counting system, the number cannot be adopted. KULANDAI-SAMY: [0031] In some aspects, the detection training component 315 may identify detection misses using data acquired from sensors such as an audio sensor, a thermal camera, an RFID sensor or an occupancy sensor. For example with respect to an audio sensor, if an audio clue suggests that a person is present in an environment (e.g., a conversation in a voice clip captured by a security camera) despite an ROI boundary not existing in a frame captured at the same time, the detection training component 315 may determine that the frame should be selected for training Likewise, in an example with respect to an occupancy sensor, if an occupancy schedule of a building or real-time occupancy data feed from an occupancy sensing (e.g., Lidar, Wi-Fi, Bluetooth, etc.) suggests that an untracked person is present in the environment at a given time (e.g., an employee is in his/her office) despite an ROI boundary not existing in one or more frames captured at the same time, the corresponding frames may be selected for training Furthermore, in another example, thermal cameras can highlight body temperature, which can indicate that a person is in the environment even though the person is not detected in an image, and consequently corresponding frames may be selected for training In yet another example, in crowd scenes, the number of head/face detections may be compared with number of ROI boundary detections to identify ROI detection misses (e.g., more heads than boundaries indicates detection misses, fewer heads than boundaries indicates false positives). In an additional example, if a crowd heat map or density estimation region is larger than a person detection region in a frame, the detection training component 315 may select the frame for inclusion in the training dataset. [0036] The additional criteria used by the detection training component 315 to prevent duplicate/similar images may also include selecting frames with a certain level of illumination (e.g., morning, afternoon, evening, night, etc.) Alternatively or in addition, the additional criteria may include selecting frames captured during a specific season/weather. Balanced composition of training data captured at different times of the day—morning, afternoon, evening, night—and covering various seasons can eliminate any bias in detection accuracy on time of the day or season. If timestamps are not available with videos, the detection training component 315 may use image features to estimate seasons and timings. For example, the following attributes may be associated with the different times in a day: morning—low contrast, less brightness/illumination, afternoon—low contrast, high brightness/illumination, evening—high contrast (due to lights), very less brightness/illumination, raining/snowing—motion throughout images.)
Consider Claim 4.
The combination of Kulandai-Samy and Lin teaches:
[Claim 4] (Currently amended) The traffic flow analysis device according to claim 1,wherein the one or more processors are further configured to: select the identification method based on a tendency of the identified attribute. (KULANDAI-SAMY: [0052]-[0055] At block 412, the method 400 includes re-training the ROI detection model using the training dataset comprising the second image frame. For example, in an aspect, computer device 300, one or more processors 305, one or more memories 310, detection training component 315, and/or re-training component 340 may be configured to or may comprise means for re-training the ROI detection model using the training dataset comprising image 120. For example, re-training component 340 may execute a training algorithm to update the weights in the ROI detection model that are used to classify objects. This training algorithm may use techniques such as gradient descent. Because the images in the training dataset include examples of objects that the ROI detection model failed to detect previously, the updated weights will enable the ROI detection model to learn how to detect the missed objects. Accordingly, for example, the re-trained ROI detection model will generate the first ROI boundary (e.g., ROI boundary 112) around the first object (e.g., person 110) in any subsequently inputted image frame depicting the first object. [0058] FIG. 5 is a flowchart illustrating method 500 of selecting frames for a training dataset, in accordance with exemplary aspects of the present disclosure. Method 500 may be executed by detection training component 315 when, at block 408, detecting component 330 determines that the first ROI boundary is not present in the second image frame. Prior to determining that the second image frame should be added to the training set, method 500 may be initiated at either block 502, block 506, block 508, or block 510. Lin: column 3 lines 35-67, column 4 lines 1-30, Figures 2-3 and 5; The image stitching module 210 is coupled to the top-view image-capturing device 110, the first side-view image-capturing device 120 and the second side-view image-capturing device 130 and is used to receive the top-view image, the first side-view image and the second side-view image captured by the image-capturing devices. Then, the image stitching module 210 stitches the top-view image, the first side-view image and the second side-view image into a wide-angle image or an ultra wide-angle image. The image stitching technology can use an image features extraction method, a spatial relation method or other technology to stitch the image. For example, as shown in FIG. 3, the image features extraction method is used to extract feature points from the top-view image 310, the first side-view image 320 and the second side-view image 330, respectively, such as the asterisks in the top-view image 310 and the first side-view image 320, and the diamonds in the top-view image 310 and the second side-view image 330. The image stitching module 210 obtains the corresponding relationship of the feature points by comparing the feature points, and stitches the top-view image 310, the first side-view image 320 and the second side-view image 330 into the ultra wide-angle image 340 according to the corresponding relationship. For another example, the spatial relation method is used to obtain a corresponding relationship according to positions in which the top-view image-capturing device 110, the first side-view image-capturing device 120 and the second side-view image-capturing device 130 are installed and stitch the top-view image 310, the first side-view image 320 and the second side-view image 330 into the ultra wide-angle image according to the corresponding relationship. The ROI selecting module 220 is coupled to the image stitching module 210 and is used to select at least one recognition zones and a counting zone from the ultra wide-angle image. In a preferred embodiment, the ROI selecting module 220 selects two recognition zones and a counting zone so that the ROI selecting module 220 can further determine a people moving direction (a person moves from left to right or from right to left). If the ROI selecting module 220 selects a single recognition zone, the ROI selecting module 220 merely determines a single direction (from left to right or from right to left). As shown in FIG. 4, the counting zone 420 is between the two recognition zones 420 and 430. In addition, the ranges of the counting zone and the recognition zone can be determined by the people counting system, or can also be determined by a user.)
Consider Claim 5.
The combination of Kulandai-Samy and Lin teaches:
[Claim 5] (Currently amended) The traffic flow analysis device according to claim 1, wherein the one or more processors are further configured to: select the identification method based on a color of lighting of a traffic signal installed around the camera. (KULANDAI-SAMY: [0041] FIG. 2 is block diagram 200 of a clustering approach to select training images, in accordance with exemplary aspects of the present disclosure. In some aspects, the images selected by the detection training component (e.g., selected images 201) may be clustered into different buckets (e.g., buckets 1-7 in FIG. 2 ), wherein each bucket contains similar images. Each bucket may also represent a certain type of additional criteria mentioned above. For example, bucket 1 may include low-light images taken during the night and bucket 2 may include daylight images. Bucket 3 may include images with larger crowds. Bucket 4 may include images with no objects of interest. Bucket 5 may include images where persons are in a certain pose. Bucket 6 may include images where persons are occluded. Bucket 7 may include images where persons are holding items. [0042] In one example, the detection training component 315 may extract pre-trained Deep Neural Network (DNN) generated image features 202, an image histogram (to capture color information), and low level features 204 such as lines and edges. These extractions are input as features for a clustering component 208 that executes DBscan or Hierarchical clustering. In some aspects, frame timestamps 206 are used as an additional feature such that images that have closer timestamps, similar ROI boundaries (e.g., size and location in an image) and background features are grouped together. The required number of images, which may be pre-determined, can be aggregated by the detection training component from each bucket for training the ROI detection model (e.g., 6 images from bucket 1, 3 images from bucket 2, etc.). By this way, a variety of data is collected including varying background, colors, lines, etc. [0043] In some aspects, the detection training component 315 may automatically annotate training data using ROI detection and ROI tracking along with sensor data fusion as described previously. An additional annotation approach is discussed below. [0064] At block 508, the method 500 includes determining whether more than a threshold number of images in the training dataset include a given light setting, background, or environment. For example, in an aspect, computer device 300, one or more processors 305, one or more memories 310, detection training component 315, and/or environment analysis component 352 may be configured to or may comprise means for determining whether more than a threshold number of images in the training dataset include a given light setting, background, or environment. Environment analysis component 352 may use computer vision and machine learning techniques to classify different types of lighting and environments. Based on the classifications (e.g., “night,” “low-light,” “busy background,” etc.), environment analysis component 352 may add a tag to each image that is identified as a potential training image. Detection training component 315 may query these tags to determine how many images in the training dataset have a specific tag.)
Consider Claim 6.
The combination of Kulandai-Samy and Lin teaches:
[Claim 6] (Currently amended) The traffic flow analysis device according to claim 1,wherein the identification method includes a classification model or a processing algorithm for identifying an attribute of a moving object captured by a camera. (KULANDAI-SAMY: [0052]-[0055] At block 412, the method 400 includes re-training the ROI detection model using the training dataset comprising the second image frame. For example, in an aspect, computer device 300, one or more processors 305, one or more memories 310, detection training component 315, and/or re-training component 340 may be configured to or may comprise means for re-training the ROI detection model using the training dataset comprising image 120. For example, re-training component 340 may execute a training algorithm to update the weights in the ROI detection model that are used to classify objects. This training algorithm may use techniques such as gradient descent. Because the images in the training dataset include examples of objects that the ROI detection model failed to detect previously, the updated weights will enable the ROI detection model to learn how to detect the missed objects. Accordingly, for example, the re-trained ROI detection model will generate the first ROI boundary (e.g., ROI boundary 112) around the first object (e.g., person 110) in any subsequently inputted image frame depicting the first object. [0058] FIG. 5 is a flowchart illustrating method 500 of selecting frames for a training dataset, in accordance with exemplary aspects of the present disclosure. Method 500 may be executed by detection training component 315 when, at block 408, detecting component 330 determines that the first ROI boundary is not present in the second image frame. Prior to determining that the second image frame should be added to the training set, method 500 may be initiated at either block 502, block 506, block 508, or block 510. Lin: column 3 lines 35-67, column 4 lines 1-30, Figures 2-3 and 5; The image stitching module 210 is coupled to the top-view image-capturing device 110, the first side-view image-capturing device 120 and the second side-view image-capturing device 130 and is used to receive the top-view image, the first side-view image and the second side-view image captured by the image-capturing devices. Then, the image stitching module 210 stitches the top-view image, the first side-view image and the second side-view image into a wide-angle image or an ultra wide-angle image. The image stitching technology can use an image features extraction method, a spatial relation method or other technology to stitch the image. For example, as shown in FIG. 3, the image features extraction method is used to extract feature points from the top-view image 310, the first side-view image 320 and the second side-view image 330, respectively, such as the asterisks in the top-view image 310 and the first side-view image 320, and the diamonds in the top-view image 310 and the second side-view image 330. The image stitching module 210 obtains the corresponding relationship of the feature points by comparing the feature points, and stitches the top-view image 310, the first side-view image 320 and the second side-view image 330 into the ultra wide-angle image 340 according to the corresponding relationship. For another example, the spatial relation method is used to obtain a corresponding relationship according to positions in which the top-view image-capturing device 110, the first side-view image-capturing device 120 and the second side-view image-capturing device 130 are installed and stitch the top-view image 310, the first side-view image 320 and the second side-view image 330 into the ultra wide-angle image according to the corresponding relationship. The ROI selecting module 220 is coupled to the image stitching module 210 and is used to select at least one recognition zones and a counting zone from the ultra wide-angle image. In a preferred embodiment, the ROI selecting module 220 selects two recognition zones and a counting zone so that the ROI selecting module 220 can further determine a people moving direction (a person moves from left to right or from right to left). If the ROI selecting module 220 selects a single recognition zone, the ROI selecting module 220 merely determines a single direction (from left to right or from right to left). As shown in FIG. 4, the counting zone 420 is between the two recognition zones 420 and 430. In addition, the ranges of the counting zone and the recognition zone can be determined by the people counting system, or can also be determined by a user.)
Consider Claim 7.
The combination of Kulandai-Samy and Lin teaches:
[Claim 7] (Currently amended) The traffic flow analysis device according to claim 6, wherein a plurality of types of the classification models or the processing algorithms is created based on a tendency of the moving object captured by the camera, the tendency being investigated in advance, and the one or more processors are further configured to: select, using a selection rule determined to select a classification model or a processing algorithm suitable for the tendency of the moving object captured by the camera from the plurality of types of classification models or processing algorithms, the classification model or the processing algorithm. (KULANDAI-SAMY: [0052]-[0055] At block 412, the method 400 includes re-training the ROI detection model using the training dataset comprising the second image frame. For example, in an aspect, computer device 300, one or more processors 305, one or more memories 310, detection training component 315, and/or re-training component 340 may be configured to or may comprise means for re-training the ROI detection model using the training dataset comprising image 120. For example, re-training component 340 may execute a training algorithm to update the weights in the ROI detection model that are used to classify objects. This training algorithm may use techniques such as gradient descent. Because the images in the training dataset include examples of objects that the ROI detection model failed to detect previously, the updated weights will enable the ROI detection model to learn how to detect the missed objects. Accordingly, for example, the re-trained ROI detection model will generate the first ROI boundary (e.g., ROI boundary 112) around the first object (e.g., person 110) in any subsequently inputted image frame depicting the first object. [0058] FIG. 5 is a flowchart illustrating method 500 of selecting frames for a training dataset, in accordance with exemplary aspects of the present disclosure. Method 500 may be executed by detection training component 315 when, at block 408, detecting component 330 determines that the first ROI boundary is not present in the second image frame. Prior to determining that the second image frame should be added to the training set, method 500 may be initiated at either block 502, block 506, block 508, or block 510. Lin: column 3 lines 35-67, column 4 lines 1-30, Figures 2-3 and 5; The image stitching module 210 is coupled to the top-view image-capturing device 110, the first side-view image-capturing device 120 and the second side-view image-capturing device 130 and is used to receive the top-view image, the first side-view image and the second side-view image captured by the image-capturing devices. Then, the image stitching module 210 stitches the top-view image, the first side-view image and the second side-view image into a wide-angle image or an ultra wide-angle image. The image stitching technology can use an image features extraction method, a spatial relation method or other technology to stitch the image. For example, as shown in FIG. 3, the image features extraction method is used to extract feature points from the top-view image 310, the first side-view image 320 and the second side-view image 330, respectively, such as the asterisks in the top-view image 310 and the first side-view image 320, and the diamonds in the top-view image 310 and the second side-view image 330. The image stitching module 210 obtains the corresponding relationship of the feature points by comparing the feature points, and stitches the top-view image 310, the first side-view image 320 and the second side-view image 330 into the ultra wide-angle image 340 according to the corresponding relationship. For another example, the spatial relation method is used to obtain a corresponding relationship according to positions in which the top-view image-capturing device 110, the first side-view image-capturing device 120 and the second side-view image-capturing device 130 are installed and stitch the top-view image 310, the first side-view image 320 and the second side-view image 330 into the ultra wide-angle image according to the corresponding relationship. The ROI selecting module 220 is coupled to the image stitching module 210 and is used to select at least one recognition zones and a counting zone from the ultra wide-angle image. In a preferred embodiment, the ROI selecting module 220 selects two recognition zones and a counting zone so that the ROI selecting module 220 can further determine a people moving direction (a person moves from left to right or from right to left). If the ROI selecting module 220 selects a single recognition zone, the ROI selecting module 220 merely determines a single direction (from left to right or from right to left). As shown in FIG. 4, the counting zone 420 is between the two recognition zones 420 and 430. In addition, the ranges of the counting zone and the recognition zone can be determined by the people counting system, or can also be determined by a user.)
Consider Claim 8.
The combination of Kulandai-Samy and Lin teaches:
[Claim 8] (Original) The traffic flow analysis device according to claim 7, wherein the moving object includes a person, and the classification model includes a classification model created by machine learning using statistical data of a flow of people at a position where the camera is installed as teacher data. (KULANDAI-SAMY: [0054] At block 410, the method 400 includes determining that the second image frame should be added to a training dataset for the ROI detection model. For example, in an aspect, computer device 300, one or more processors 305, one or more memories 310, detection training component 315, and/or determining component 335 may be configured to or may comprise means for determining that image 120 should be added to a training dataset for the ROI detection model. In some aspects, an ROI boundary is added to image 120 around person 110 where person 110 is located. This updated image is then added to the training dataset. [0055] At block 412, the method 400 includes re-training the ROI detection model using the training dataset comprising the second image frame. For example, in an aspect, computer device 300, one or more processors 305, one or more memories 310, detection training component 315, and/or re-training component 340 may be configured to or may comprise means for re-training the ROI detection model using the training dataset comprising image 120. For example, re-training component 340 may execute a training algorithm to update the weights in the ROI detection model that are used to classify objects. This training algorithm may use techniques such as gradient descent. Because the images in the training dataset include examples of objects that the ROI detection model failed to detect previously, the updated weights will enable the ROI detection model to learn how to detect the missed objects. Accordingly, for example, the re-trained ROI detection model will generate the first ROI boundary (e.g., ROI boundary 112) around the first object (e.g., person 110) in any subsequently inputted image frame depicting the first object. [0056] At block 416, the method 400 includes operating the object detection system using the re-trained ROI detection model, wherein the re-trained ROI detection model generates the first ROI boundary around the first object in any subsequently inputted image frame depicting the first object. For example, in an aspect, computer device 300, one or more processors 305, one or more memories 310, detection training component 315, and/or re-training component 340 may be configured to operate the object detection system using the re-trained ROI detection model, wherein the re-trained ROI detection model generates the first ROI boundary around the first object in any subsequently inputted image frame depicting the first object. In some aspects, the re-trained ROI detection model being operated does not generate the second ROI boundary around the second object in any subsequently inputted image frame depicting the second object (discussed in FIG. 5 ). [0057] At block 414, the method 400 includes determining that the second image frame should not be added to a training dataset for the ROI detection model. For example, in an aspect, computer device 300, one or more processors 305, one or more memories 310, detection training component 315, and/or determining component 335 may be configured to or may comprise means for determining that image 120 should not be added to a training dataset for the ROI detection model. In this case, image 120 is skipped and the next frame is considered. If the next frame is identified as an image that should be added to the training dataset, re-training component 340 may re-training the ROI detection model using the training dataset comprising the next frame. [0058] FIG. 5 is a flowchart illustrating method 500 of selecting frames for a training dataset, in accordance with exemplary aspects of the present disclosure. Method 500 may be executed by detection training component 315 when, at block 408, detecting component 330 determines that the first ROI boundary is not present in the second image frame. Prior to determining that the second image frame should be added to the training set, method 500 may be initiated at either block 502, block 506, block 508, or block 510. [0059] At block 502, the method 500 includes assigning a first tracking identifier to the first ROI boundary around the first object. For example, in an aspect, computer device 300, one or more processors 305, one or more memories 310, detection training component 315, and/or tracking ID component 350 may be configured to or may comprise means for assigning a first tracking identifier (e.g., a set of characters such as “ABC123”) to ROI boundary 108 around person 106.)
Consider Claim 9.
The combination of Kulandai-Samy and Lin teaches:
[Claim 9] (Original) The traffic flow analysis device according to claim 7, wherein the moving object includes a person, and the classification model is created by machine learning using statistical data of a flow of people for each time zone, the flow of people being imaged by the camera, as teacher data. (KULANDAI-SAMY: [0036] The additional criteria used by the detection training component 315 to prevent duplicate/similar images may also include selecting frames with a certain level of illumination (e.g., morning, afternoon, evening, night, etc.) Alternatively or in addition, the additional criteria may include selecting frames captured during a specific season/weather. Balanced composition of training data captured at different times of the day—morning, afternoon, evening, night—and covering various seasons can eliminate any bias in detection accuracy on time of the day or season. If timestamps are not available with videos, the detection training component 315 may use image features to estimate seasons and timings. For example, the following attributes may be associated with the different times in a day: morning—low contrast, less brightness/illumination, afternoon—low contrast, high brightness/illumination, evening—high contrast (due to lights), very less brightness/illumination, raining/snowing—motion throughout images. Lin: column 5 lines 33-67, column 6 lines 1-28, FIG. 7 is a flow diagram 700 illustrating a people counting method according to an embodiment of the present invention with reference to FIGS. 1-2. First, in step S705, a top-view image-capturing device captures a top-view image. A first side-view image-capturing device and a second side-view image-capturing device capture a first side-view image and a second side-view image, respectively. Then, in step S710, an image stitching module stitches the top-view image, the first side-view image and the second side-view image into an ultra wide-angle image. In step S715, a region of interest (ROI) selecting module selects a plurality of recognition zones and a counting zone from the ultra wide-angle image. Then, in step S720, the face recognition module monitors the recognition zone to determine a face location corresponding to a face, and a head recognition module monitors the counting zone to determine a head location corresponding to a head. In step S725, an object tracking module generates a face track and a head track according to the face location and the head location, respectively. Finally, in step S730, a people counting module counts a first number of face tracks passing through the counting zone during a time interval, counting a second number of head tracks passing through the counting zone during the time interval and compares the first number with the second number to generate a people counting result. FIGS. 8A-8B are schematic diagrams illustrating that the object tracking module generates a face track and a head track according to an embodiment of the present invention. FIG. 8A shows a face track. As shown in FIG. 8A, the object tracking module generates solid face tracks 842 and 844 in recognition zones 820 and 830 according to face locations corresponding to the face monitored by the face recognition module, and derives a dotted head track 840 in a counting zone 810 according to the solid face tracks 842 and 844. FIG. 8B shows a head track. As shown in FIG. 8B, the object tracking module generates a head track 850 in the counting zone 810 according to a head location corresponding to the head monitored by the head recognition module. After the object tracking module generates the head track 850 and the face tracks 842 and 844, the people counting module counts a first number of solid face tracks 840 passing through the counting zone 810 during a time interval, counts a second number of head tracks 850 passing through the counting zone 810 during the time interval and compares the first number with the second number to generate a people counting result.)
Conclusion
The prior art made of record in form PTO-892 and not relied upon is considered pertinent to applicant's disclosure.
PNG
media_image1.png
206
908
media_image1.png
Greyscale
Any inquiry concerning this communication or earlier communications from the examiner should be directed to TAHMINA ANSARI whose telephone number is 571-270-3379. The examiner can normally be reached on IFP Flex - Monday through Friday 9 to 5.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, O’NEAL MISTRY can be reached on 313-446-4912. The fax phone numbers for the organization where this application or proceeding is assigned are 571-273-8300 for regular communications and 571-273-8300 for After Final communications. TC 2600’s customer service number is 571-272-2600.
Any inquiry of a general nature or relating to the status of this application or proceeding should be directed to the receptionist whose telephone number is 571-272-2600.
2674
/Tahmina Ansari/
June 4, 2026
/TAHMINA N ANSARI/Primary Examiner, Art Unit 2674
Prosecution Insights