DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 11/24/2025 has been entered.
Response to Arguments
Applicant’s arguments, see Remarks pages 10-14, filed 11/24/2025, with respect to the rejection of amended claim(s) 1, 3, and 20 under 35 U.S.C. 103 have been fully considered and are moot in view of the new grounds of rejection (detailed in the rejections below) necessitated by Applicant’s amendment to the claim(s).
Claim Objections
Claims 1, 3, and 20 are objected to because of the following informalities:
Regarding claims 1, 3, and 20, the disclosed limitation: “wherein features corresponding to a persistent background are given a high attention weight and features corresponding to non-persistent background are given a low attention weight,” should be corrected to “wherein the features corresponding to a persistent background are given a high attention weight and the features corresponding to a non-persistent background are given a low attention weight.”
Appropriate correction is required.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 3, 5, 9-10, and 18-20, are rejected under 35 U.S.C. 103 as being unpatentable over Franklin et al. (US 20190259278 A1), hereinafter referenced as Franklin, in view of Li et al. (Seamless Positioning and Navigation by Using Geo-Referenced Images and Multi-Sensor Data) hereinafter referenced as Li, Stumpe et al. (US 11132416 B1) hereinafter referenced as Stumpe, and Panboonyuen et al. (Semantic Segmentation on Remotely Sensed Images Using an Enhanced Global Convolutional Network with Channel Attention and Domain Specific Transfer Learning), hereinafter referenced as Panboonyuen.
Regarding claim 1, Franklin discloses: A system for parking monitoring in an urban area (Franklin: Abstract), the system comprising: at least one camera, wherein the at least one camera is positioned to capture images of the urban area (Franklin: 0064: “the camera used to capture the parking stall sign may also be used to capture other scene information…surrounding parking stall identification, road surface markers representing stall demarcation (such as white paint on the ground), etc.”);
a computing device in communication with the at least one camera via a network to receive image data comprising the captured images from the at least one camera (Franklin: 0014: “determining the parking stall identifier comprises capturing one or more images of the occupied parking stall using an image detector mounted on the patrol vehicle; and processing the one or more images to identify the parking stall identifier.”; 0037: “These embodiments may be implemented in computer programs executing on programmable computers”);
the computing device comprising at least one processor and a memory accessible to the at least one processor (Franklin: 0037: “each computer including at least one processor, a data storage system (including volatile memory or non-volatile memory or other data storage elements or a combination thereof),”);
wherein the memory comprises a library of reference background images and metadata for each reference background image, wherein the metadata comprises parking location information and parking condition information (Franklin: 0056: “A survey of the parking area may also be performed…for the parking stalls. The survey data may be stored in a machine readable format such as XML, KML or CSV. Additionally, reference images of empty stalls may additionally be obtained in association with each stall to indicate a field of view in which the parking stall is empty to facilitate identification of the correct parking stall.”);
wherein the memory stores program code that when executed by the at least one processor causes the at least one processor to: process a first captured image to determine a licence plate number corresponding to a target vehicle in the first captured image (Franklin: 0065: “The vision system of the mobile enforcement vehicle may also comprise a number of cameras to obtain vehicle-identifying information including the license plate number,”);
process a second captured image using a background matching module to identify a matching reference background image (Franklin: 0089: “Identification of the correct parking stall may be further enhanced using reference photo(s) background (e.g. those that are images of the parking stall only, without the presence of a parked vehicle) and compared to those acquired by the mobile enforcement vehicle during its patrols.”);
determine an identified parking location of the target vehicle and at least one parking condition based on the metadata of the matching reference background image (Franklin: Figures 6A & 6B; 0067: “the mobile enforcement vehicle 310 may proceed to determine which parking stall the vehicle is occupying. To do so, the position (e.g. GPS coordinates) of the parked vehicle should preferably be determined as precisely as possible so that the parked vehicle can be “placed” into the boundaries of the parking stall as defined by the four corners of the stall.”);
determine compliance of the target vehicle with the determined at least one parking condition (Franklin: 0095: “At decision step 754, the central processor may query the database to determine whether or not, at the time of observation, the permitted parking time at the identified parking stall has expired…If the parking stall should not be occupied…the method may proceed to steps 770 and 772 to issue a citation…”).
Franklin does not disclose expressly: wherein the background matching module comprises: a background feature extractor neural network trained to identify background descriptors in images that correspond to permanent structures, and the at least one processor is further configured to identify the matching reference background image by: extracting background descriptors corresponding to permanent structures from the second captured image using the trained background feature extractor neural network; selecting one or more candidate matching images from the library of background images based on the extracted background descriptors; and performing geometric matching between the second captured image and the candidate matching images to select the matching reference background image;
wherein the background feature extractor neural network comprises at least one convolutional neural network trained to extract the background feature descriptors corresponding to permanent structures from captured images, each at least one convolutional neural network comprising an attention determination layer trained to determine attention weights for features in the captured image, wherein features corresponding to a persistent background are given a high attention weight and features corresponding to non-persistent background are given a low attention weight, wherein the permanent structures are identified using the high attention weight.
Li discloses: at least one processor configured to identify a matching reference background image by: extracting background descriptors corresponding to structures in a captured image (Li: 3.2. Image Retrieval Using SIFT-Based Voting Strategy: “We first use a voting strategy to find candidate reference images, then check the geometric consistency to detect mismatches and remove mismatched reference images from the image space. First, the SIFT features are extracted for the query image (e.g., Figure 8).”); selecting one or more candidate matching images from a library of background images based on the extracted background descriptors (Li: 3.2. Image Retrieval Using SIFT-Based Voting Strategy: “Secondly, SIFT matching is performed between the query image and the newly generated reference feature database (Figure 10) to find corresponding reference images. A K-NN (here k = 3) search function is used to find the k nearest neighbours from the feature database for the feature points in the query image. Each correspondence adds one vote to the reference image it belongs to…the top m (5 in our case) reference images are chosen as ones corresponding to the query scene and retrieved from the candidate image space.”); and performing geometric matching between the second captured image and the candidate matching images to select a matching reference background image (Li: 3.2. Image Retrieval Using SIFT-Based Voting Strategy: “To improve the robustness of the system, a further step is to check to the correctness of the top voted images based on pair-wise geometric consistency. This process can detect any falsely ranked/selected reference images as well as remove mismatches. First RANSAC is used to estimate the homography (projective transformation) between the two images, and remove mismatches (Figure 11). A new method we proposed in previous research [24] that utilizes cross-correlation information to check the quality of the homography model built by RANSAC is used here to further ensure the correctness of matching…An average correlation ρ̄ is calculated for all the matched (reliable) points produced by one matching (one H is generated). If ρ̄ is close to 1, the estimated homography model H is very accurate and the two images are a correctly matched pair; the reverse would also apply. The threshold for ρ̄ is set to 0.75 in the system. In Figure 11 the score is higher than the threshold, which indicates it is a correct matched pair; while in Figure 12 the score shows the inverse situation…Therefore in this way the candidate image space is further filtered, so that it contains only the reference images with corresponding views in the query image.”);
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to incorporate the reference image retrieval algorithms disclosed by Li into the parking stall identification process disclosed by Franklin. The suggestion/motivation for doing so would have been “ To improve the robustness of the system, a further step is to check to the correctness of the top voted images based on pair-wise geometric consistency. This process can detect any falsely ranked/selected reference images as well as remove mismatches…Therefore in this way the candidate image space is further filtered, so that it contains only the reference images with corresponding views in the query image. ” (Li: 3.2. Image Retrieval Using SIFT-Based Voting Strategy; Wherein the SIFT based matching and RANSAC based candidate filtering allows for a robust and accurate process.). Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results.
Franklin in view of Li does not disclose expressly: wherein the background matching module comprises: a background feature extractor neural network trained to identify background descriptors in images that correspond to permanent structures, and the at least one processor is further configured to identify the matching reference background image by: extracting background descriptors corresponding to permanent structures from the second captured image using the trained background feature extractor neural network;
Stumpe discloses: a background feature extractor neural network trained to identify background descriptors in images that correspond to permanent structures, and the at least one processor is configured to identify a matching reference background image by: extracting background descriptors corresponding to permanent structures from a captured image using the trained background feature extractor neural network (Stumpe: Col 10: Lines 20-36: “the new images may be identified as being of the business location based on visual features. Visual features may include colors, shapes, and locations of other businesses, building structures, or other persistent features...The number of matching visual features may be determined using a computer vision approach such as a deep neural network. The visual features surrounding an image region depicting the business in the reference image, or image context, may first be identified and then may be matched with visual features in the new images. In this way, visual features may be used to define the image context and the image regions associated with the business location in the reference image and comparison image.”).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to implement the deep neural network for identifying persistent features taught by Stumpe prior to performing the SIFT based reference image retrieval as disclosed by Franklin in view of Li in order to extract SIFT features from identified persistent features. The suggestion/motivation for doing so would have been “If an image has the same or similar image context as the reference image, the image is probably depicting the same location as the reference image. For example, if the business is depicted in the reference image as surrounded by two other businesses and a traffic light, then an image region in the comparison image may be identified by identifying where the two businesses and the traffic light are in the comparison image.” (Stumpe: Col 10: Lines 37-44). Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Franklin in view of Li with Stumpe to obtain the invention as specified in claim 1.
Franklin in view of Li and Stumpe does not disclose expressly: wherein the background feature extractor neural network comprises at least one convolutional neural network trained to extract the background feature descriptors corresponding to permanent structures from captured images, each at least one convolutional neural network comprising an attention determination layer trained to determine attention weights for features in the captured image, wherein features corresponding to a persistent background are given a high attention weight and features corresponding to non-persistent background are given a low attention weight, wherein the permanent structures are identified using the high attention weight.
Thus Franklin in view of Li and Stumpe does not disclose expressly: the persistent feature identifying deep neural network comprising at least one convolutional neural network, wherein each at least one convolutional neural network comprises an attention determination layer trained to determine attention weights for features in the captured image, wherein the features corresponding to persistent features are given a high attention weight and features corresponding to non-persistent features are given a low attention weight, wherein the persistent features are identified using the high attention weight.
Panboonyuen discloses: a feature extractor neural network comprising at least one convolutional neural network trained to extract feature descriptors corresponding to structures from captured images (Panboonyuen: Abstract: “In the remote sensing domain, it is crucial to complete semantic segmentation on the raster images, e.g., river, building, forest, etc., on raster images…In this paper, we aim to propose a novel CNN for semantic segmentation particularly for remote sensing corpora with three main contributions. First, we propose applying a recent CNN called a global convolutional network (GCN), since it can capture different resolutions by extracting multi-scale features from different stages of the network.”), each at least one convolutional neural network comprising an attention determination layer trained to determine attention weights for features in the captured image (Panboonyuen: 5. Experimental Results and Discussion: “The implementation is based on a deep learning framework, called “Tensorflow-Slim” [36], which is extended from Tensorflow...All models are trained for 50 epochs with a mini-batch size of 4, and each batch contains the cropped images that are randomly selected from training patches. These patches are resized to 521 x 521 pixels. The statistics of BN is updated on the whole mini-batch.” ; Wherein the deep-learning implementation is trained.), wherein features corresponding to important features are given a high attention weight and features corresponding to non-important features are given a low attention weight, wherein the structures are identified using the high attention weight (Panboonyuen: 3.3. The Channel Attention Block: “To apply this atttentional layer to our network, the channel attention block is shown in Block A in Figure 2 and its detailed architecture is shown in Figure 4. It is designed to change the weights of the remote sensing features on each stage (level), so that the weights are assigned more values on important features adaptively.”; Wherein the features with higher attention weights are used, and play a larger role, for the identification and classification of structures/objects within the images.).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to substitute the persistent feature identifying deep neural network disclosed by Franklin in view of Li and Stumpe with the convolutional neural network containing attention blocks as taught by Panboonyuen. The suggestion/motivation for doing so would have been “Attention mechanisms [16,17] in neural networks are very loosely based on the visual attention mechanism found in humans and equips a neural network with the ability to focus on a subset of its inputs (or features): it selects specific inputs...It is designed to change the weights of the remote sensing features on each stage (level), so that the weights are assigned more values on important features adaptively.” (Panboonyuen: 3.3. The Channel Attention Block). Further, one skilled in the art could have substituted the elements as described above by known methods with no change in their respective functions, and the substitution would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Franklin in view of Li and Stumpe with Panboonyuen to obtain the invention as specified in claim 1.
Regarding claim 3, Franklin discloses: A system for parking monitoring in an urban area (Franklin: Abstract), the system comprising: at least one camera, wherein the at least one camera is positioned to capture images of the urban area (Franklin: 0064: “the camera used to capture the parking stall sign may also be used to capture other scene information…surrounding parking stall identification, road surface markers representing stall demarcation (such as white paint on the ground), etc.”);
a computing device in communication with the at least one camera over a network to receive image data comprising the captured images (Franklin: 0014: “determining the parking stall identifier comprises capturing one or more images of the occupied parking stall using an image detector mounted on the patrol vehicle; and processing the one or more images to identify the parking stall identifier.”; 0037: “These embodiments may be implemented in computer programs executing on programmable computers”);
the computing device comprising at least one processor and a memory accessible to the at least one processor (Franklin: 0037: “each computer including at least one processor, a data storage system (including volatile memory or non-volatile memory or other data storage elements or a combination thereof)”);
wherein the memory comprises a library of reference background images and metadata for each reference background image, wherein the metadata comprises parking location information (Franklin: 0056: “A survey of the parking area may also be performed…for the parking stalls. The survey data may be stored in a machine readable format such as XML, KML or CSV. Additionally, reference images of empty stalls may additionally be obtained in association with each stall to indicate a field of view in which the parking stall is empty to facilitate identification of the correct parking stall.”);
the memory also comprises program code that when executed by the at least one processor causes the at least one processor to:
process the captured images using a parking indicator detection machine learning model to identify a parking indicator in at least one of the captured images (Franklin: 0063: “The captured image of the stall sign can then be processed using optical character recognition (OCR) to identify the stall number.”; Wherein the stall sign is a parking indicator);
on identifying the parking indicator, process the captured images using a background matching module to identify a matching reference background image that matches one of the captured images (Franklin: 0089 “Identification of the correct parking stall may be further enhanced using reference photo(s) background (e.g. those that are images of the parking stall only, without the presence of a parked vehicle) and compared to those acquired by the mobile enforcement vehicle during its patrols.);
determine a parking location based on the metadata associated with the matching reference background image (Franklin: 0067: “the mobile enforcement vehicle 310 may proceed to determine which parking stall the vehicle is occupying. To do so, the position (e.g. GPS coordinates) of the parked vehicle should preferably be determined as precisely as possible so that the parked vehicle can be “placed” into the boundaries of the parking stall as defined by the four corners of the stall.”);
determine parking conditions based on the identified parking indicator (Franklin: Figure 5; 0079-0080: “…the database may comprise a “Parking stall No.” field 510 which stores the unique parking stall numbers corresponding to each parking stall being managed…The “Occupied?” field 540 may be used to specify whether a given parking stall should be occupied (i.e. it may be occupied if it has been paid for).”);
process the captured images to determine a licence plate number corresponding to a target vehicle (Franklin: 0065: “The vision system of the mobile enforcement vehicle may also comprise a number of cameras to obtain vehicle-identifying information including the license plate number”);
and determine compliance of the target vehicle to the determined parking conditions (Franklin: 0095: “At decision step 754, the central processor may query the database to determine whether or not, at the time of observation, the permitted parking time at the identified parking stall has expired…If the parking stall should not be occupied…the method may proceed to steps 770 and 772 to issue a citation…”).
Franklin does not disclose expressly: wherein the at least one processor is further configured to identify the matching reference background image by: extracting background descriptors corresponding to permanent structures from at least one of the captured images using the trained background feature extractor neural network; selecting one or more candidate matching images from the library of background images based on the extracted background descriptors; and performing geometric matching between the at least one captured image and the candidate matching images to select the matching reference background image.
Li discloses: at least one processor configured to identify a matching reference background image by: extracting background descriptors corresponding to structures in a captured image (Li: 3.2. Image Retrieval Using SIFT-Based Voting Strategy: “We first use a voting strategy to find candidate reference images, then check the geometric consistency to detect mismatches and remove mismatched reference images from the image space. First, the SIFT features are extracted for the query image (e.g., Figure 8).”); selecting one or more candidate matching images from a library of background images based on the extracted background descriptors (Li: 3.2. Image Retrieval Using SIFT-Based Voting Strategy: “Secondly, SIFT matching is performed between the query image and the newly generated reference feature database (Figure 10) to find corresponding reference images. A K-NN (here k = 3) search function is used to find the k nearest neighbours from the feature database for the feature points in the query image. Each correspondence adds one vote to the reference image it belongs to…the top m (5 in our case) reference images are chosen as ones corresponding to the query scene and retrieved from the candidate image space.”); and performing geometric matching between the second captured image and the candidate matching images to select a matching reference background image (Li: 3.2. Image Retrieval Using SIFT-Based Voting Strategy: “To improve the robustness of the system, a further step is to check to the correctness of the top voted images based on pair-wise geometric consistency. This process can detect any falsely ranked/selected reference images as well as remove mismatches. First RANSAC is used to estimate the homography (projective transformation) between the two images, and remove mismatches (Figure 11). A new method we proposed in previous research [24] that utilizes cross-correlation information to check the quality of the homography model built by RANSAC is used here to further ensure the correctness of matching…An average correlation ρ̄ is calculated for all the matched (reliable) points produced by one matching (one H is generated). If ρ̄ is close to 1, the estimated homography model H is very accurate and the two images are a correctly matched pair; the reverse would also apply. The threshold for ρ̄ is set to 0.75 in the system. In Figure 11 the score is higher than the threshold, which indicates it is a correct matched pair; while in Figure 12 the score shows the inverse situation…Therefore in this way the candidate image space is further filtered, so that it contains only the reference images with corresponding views in the query image.”);
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to incorporate the reference image retrieval algorithms disclosed by Li into the parking stall identification process disclosed by Franklin. The suggestion/motivation for doing so would have been “To improve the robustness of the system, a further step is to check to the correctness of the top voted images based on pair-wise geometric consistency. This process can detect any falsely ranked/selected reference images as well as remove mismatches…Therefore in this way the candidate image space is further filtered, so that it contains only the reference images with corresponding views in the query image. ” (Li: 3.2. Image Retrieval Using SIFT-Based Voting Strategy; Wherein the SIFT based matching and RANSAC based candidate filtering allows for a robust and accurate process.). Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results.
Franklin in view of Li does not disclose expressly: wherein the background matching module comprises: a background feature extractor neural network trained to identify background descriptors in images that correspond to permanent structures, and the at least one processor is further configured to identify the matching reference background image by: extracting background descriptors corresponding to permanent structures from at least one of the captured images using the trained background feature extractor neural network.
Stumpe discloses: a background feature extractor neural network trained to identify background descriptors in images that correspond to permanent structures, and the at least one processor is configured to identify a matching reference background image by: extracting background descriptors corresponding to permanent structures from a captured image using the trained background feature extractor neural network (Stumpe: Col 10: Lines 20-36: “the new images may be identified as being of the business location based on visual features. Visual features may include colors, shapes, and locations of other businesses, building structures, or other persistent features...The number of matching visual features may be determined using a computer vision approach such as a deep neural network. The visual features surrounding an image region depicting the business in the reference image, or image context, may first be identified and then may be matched with visual features in the new images. In this way, visual features may be used to define the image context and the image regions associated with the business location in the reference image and comparison image.”).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to implement the deep neural network for identifying persistent features taught by Stumpe prior to performing the SIFT based reference image retrieval as disclosed by Franklin in view of Li in order to extract SIFT features from identified persistent features. The suggestion/motivation for doing so would have been “If an image has the same or similar image context as the reference image, the image is probably depicting the same location as the reference image. For example, if the business is depicted in the reference image as surrounded by two other businesses and a traffic light, then an image region in the comparison image may be identified by identifying where the two businesses and the traffic light are in the comparison image.” (Stumpe: Col 10: Lines 37-44). Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results.
Franklin in view of Li and Stumpe does not disclose expressly: wherein the feature extractor module comprises at least one convolutional neural network trained to extract the background feature descriptors corresponding to permanent structures from captured images, each at least one convolutional neural network comprising an attention determination layer trained to determine attention weights for features in the captured image, wherein features corresponding to a persistent background are given a high attention weight and features corresponding to non-persistent background are given a low attention weight, wherein the permanent structures are identified using the high attention weight.
Thus Franklin in view of Li and Stumpe does not disclose expressly: the persistent feature identifying deep neural network comprising at least one convolutional neural network, wherein each at least one convolutional neural network comprises an attention determination layer trained to determine attention weights for features in the captured image, wherein the features corresponding to persistent features are given a high attention weight and features corresponding to non-persistent features are given a low attention weight, wherein the persistent features are identified using the high attention weight.
Panboonyuen discloses: a feature extractor neural network comprising at least one convolutional neural network trained to extract feature descriptors corresponding to structures from captured images (Panboonyuen: Abstract: “In the remote sensing domain, it is crucial to complete semantic segmentation on the raster images, e.g., river, building, forest, etc., on raster images…In this paper, we aim to propose a novel CNN for semantic segmentation particularly for remote sensing corpora with three main contributions. First, we propose applying a recent CNN called a global convolutional network (GCN), since it can capture different resolutions by extracting multi-scale features from different stages of the network.”), each at least one convolutional neural network comprising an attention determination layer trained to determine attention weights for features in the captured image (Panboonyuen: 5. Experimental Results and Discussion: “The implementation is based on a deep learning framework, called “Tensorflow-Slim” [36], which is extended from Tensorflow...All models are trained for 50 epochs with a mini-batch size of 4, and each batch contains the cropped images that are randomly selected from training patches. These patches are resized to 521 x 521 pixels. The statistics of BN is updated on the whole mini-batch.” ; Wherein the deep-learning implementation is trained.), wherein features corresponding to important features are given a high attention weight and features corresponding to non-important features are given a low attention weight, wherein the structures are identified using the high attention weight (Panboonyuen: 3.3. The Channel Attention Block: “To apply this atttentional layer to our network, the channel attention block is shown in Block A in Figure 2 and its detailed architecture is shown in Figure 4. It is designed to change the weights of the remote sensing features on each stage (level), so that the weights are assigned more values on important features adaptively.”; Wherein the features with higher attention weights are used, and play a larger role, for the identification and classification of structures/objects within the images.).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to substitute the persistent feature identifying deep neural network disclosed by Franklin in view of Li and Stumpe with the convolutional neural network containing attention blocks as taught by Panboonyuen. The suggestion/motivation for doing so would have been “Attention mechanisms [16,17] in neural networks are very loosely based on the visual attention mechanism found in humans and equips a neural network with the ability to focus on a subset of its inputs (or features): it selects specific inputs...It is designed to change the weights of the remote sensing features on each stage (level), so that the weights are assigned more values on important features adaptively.” (Panboonyuen: 3.3. The Channel Attention Block). Further, one skilled in the art could have substituted the elements as described above by known methods with no change in their respective functions, and the substitution would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Franklin in view of Li and Stumpe with Panboonyuen to obtain the invention as specified in claim 3.
Regarding claim 5, Franklin in view of Li, Stumpe, and Panboonyuen discloses: The system of claim 3, wherein the parking indicators comprise parking signs or licence plates and the parking indicator detection machine learning model detects parking signs or licence plates in the captured images (Franklin: 0063: “The captured image of the stall sign can then be processed using optical character recognition (OCR) to identify the stall number.”; Wherein the parking sign is a parking indicator and OCR is a machine learning algorithm).
Regarding claim 9, Franklin in view of Li, Stumpe, and Panboonyuen discloses: The system of claim 1, wherein the geometric matching is performed using a random sample consensus process (Li: 3.2. Image Retrieval Using SIFT-Based Voting Strategy: “To improve the robustness of the system, a further step is to check to the correctness of the top voted images based on pair-wise geometric consistency. This process can detect any falsely ranked/selected reference images as well as remove mismatches. First RANSAC is used to estimate the homography (projective transformation) between the two images, and remove mismatches (Figure 11).”).
Regarding claim 10, Franklin in view of Li, Stumpe, and Panboonyuen discloses: The system of claim 1, wherein: the parking location information comprises a street name and a street number; or the parking location information comprises a street name, a street number and a parking bay identifier; or the parking location information comprises a longitude coordinate and a latitude coordinate associated with the parking location (Franklin: 0055: “The locations of each parking stall can be further identified using a number of location markers such as GPS coordinate points 230 to specify the four corners of each parking stall. These location markers may assist a mobile enforcement vehicle, discussed below, to identify individual parking stalls. Alternatively, a single location marker such as a GPS coordinate point corresponding to the center of the stall (not shown) may be used instead.”; Wherein the GPS Coordinate point comprises a longitude coordinate and a latitude coordinate).
Regarding claim 18, Franklin in view of Li, Stumpe, and Panboonyuen discloses: The system of claim 3, wherein the parking conditions are determined based on characters recognised by processing a portion of the at least one captured image corresponding to the identified parking signs using a character recognition module (Franklin: 0063: “The captured image of the stall sign can then be processed using optical character recognition (OCR) to identify the stall number.”; 0079-0080: “In some embodiments, the database may comprise a “Parking stall No.” field 510 which stores the unique parking stall numbers corresponding to each parking stall being managed…The “Occupied?” field 540 may be used to specify whether a given parking stall should be occupied”).
Regarding claim 19, Franklin in view of Li, Stumpe, and Panboonyuen discloses: The system of claim 1, wherein at least one reference background image relates to a parking zone start location and another at least one reference background image relates to a parking zone end location (Franklin: 0089: “Identification of the correct parking stall may be further enhanced using reference photo(s) background (e.g. those that are images of the parking stall only, without the presence of a parked vehicle) and compared to those acquired by the mobile enforcement vehicle during its patrols.”; Wherein the reference photos contains the parking stall’s start and end locations), and determination of compliance of the target vehicle to the determined at least one parking condition is based on: distance between the identified parking location and the parking zone start location; or distance between the identified parking location and the parking zone end location (Franklin: 0067: “given the proximity of parking stalls and parked vehicles, the mobile enforcement vehicle may be required to “place” a parked vehicle into the correct parking stall…the position (e.g. GPS coordinates) of the parked vehicle should preferably be determined as precisely as possible so that the parked vehicle can be “placed” into the boundaries of the parking stall as defined by the four corners of the stall...the mobile enforcement vehicle 310 may therefore use its own GPS coordinates to determine the GPS coordinates of the parked vehicle and associate that parked vehicle to a given parking stall.”; Wherein the vehicle’s parking location is determined by its location/distance from the parking stall’s boundaries. The vehicle’s parking conditions being determined by stall it is within.).
As per claim(s) 20, arguments made in rejecting claim(s) 3 are analogous.
Claim(s) 2 is/are rejected under 35 U.S.C. 103 as being unpatentable over Franklin in view of Li, Stumpe, and Panboonyuen, and further in view of Cohen et al. (US 20170124395 A1) hereinafter referenced as Cohen.
Regarding claim 2, Franklin in view of Li, Stumpe, and Panboonyuen discloses: The system of claim 1.
Franklin in view of Li, Stumpe, and Panboonyuen does not disclose expressly: wherein the first captured image is the same captured image as the second captured image.
Cohen discloses: wherein the image used to determine a license plate number is the same as the image used to determine if a parking space is occupied (Cohen: 0003: “An “occupancy and identity image” is understood herein to mean an image from which either a human operator or a computer equipped with appropriate image processing software can decide whether a parking space is occupied and also can determine the identity of a vehicle that occupies an occupied parking space.”; 0008: “…according to the present invention there is provided a method of managing a plurality of parking spaces, including: (a) acquiring at least one occupancy and identity image”;).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to use the same captured image for identifying a licence plate and to identify a reference background image as taught by Cohen for determining vehicle compliance in Franklin in view of Li, Stumpe, and Panboonyuen. The suggestion/motivation for doing so would have been for the purposes of increasing the speed of image processing by capturing and processing one image as opposed to two. Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Franklin in view of Li, Stumpe, and Panboonyuen with Cohen to obtain the invention as specified in claim 2.
Claim(s) 4, 6, 7, and 17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Franklin in view of Li, Stumpe, and Panboonyuen, and further in view of Nerayoff et al. (US 20140249896 A1) hereinafter referenced as Nerayoff.
Regarding claim 4, Franklin in view of Li, Stumpe, and Panboonyuen discloses: The system of claim 1.
Franklin in view of Li, Stumpe, and Panboonyuen does not disclose expressly: wherein the licence plate number corresponding to the target vehicle is determined using a licence plate detection machine learning model.
Nerayoff discloses: wherein the licence plate number corresponding to the target vehicle is determined using a licence plate detection machine learning model (Nerayoff: 0042: “OCR (optical character recognition) may be performed on any license plate image(s) in order to obtain an alphanumeric license plate number along with state/province of issue.”)
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to substitute the license plate OCR character detection process taught by Nerayoff for the license plate detector disclosed by Franklin in view of Li, Stumpe, and Panboonyuen. The suggestion/motivation for doing so would have been for the purposes of automating the extraction and verification of the license plate number. Further, one skilled in the art could have substituted the elements as described above by known methods with no change in their respective functions, and the substitution would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Franklin in view of Li, Stumpe, and Panboonyuen with Nerayoff to obtain the invention as specified in claim 4.
Regarding claim 6, Franklin in view of Li, Stumpe, and Panboonyuen discloses: The system of claim 1.
Franklin in view of Li, Stumpe, and Panboonyuen does not disclose expressly: wherein the memory further comprises parking perimeter metadata associated with each reference background image, and the at least one processor is further configured to: process the at least one captured image to identify an image portion corresponding to the target vehicle in one of the captured images; and determine compliance of the target vehicle to the determined parking conditions based on the parking perimeter metadata associated with the matching reference background image and the image portion corresponding to the target vehicle.
Nerayoff discloses: parking perimeter metadata associated with each image feed (Nerayoff: Figure 6A; 0049: “The GUI described below in connection with FIGS. 6A and 6B may also be configured to provide user interface elements which may be used to specify, with an overlay on an image feed, the position and extent of destination locations within the field of view of a camera.”), and at least one processor configured to:
process at least one captured image to identify an image portion corresponding to a target vehicle in one of the captured images (Nerayoff: Figure 6A; 0050: “server system 140 may be configured to: (1) be aware of any and all vehicles present in the field of view of destination camera 125… (4) identify which destination location(s) each stationary vehicle is occupying; (5) keep track of how long each vehicle has occupied its respective destination location(s);”);
determine compliance of the target vehicle to the determined parking conditions based on the parking perimeter metadata associated with the matching image feed and the image portion corresponding to the target vehicle (Nerayoff: 0050: “…when a vehicle appears in the field of view of destination camera 125…server system 140 may be configured to…(7) determine when use of a destination location conflicts with restrictions established for the destination location; (8) identify which street or block a vehicle is improperly parked; (9) track an amount of time each vehicle has been improperly parked; and (10) track a time at which a vehicle ceases to be illegally parked…Server system 140 may also be configured to detect double parked vehicles”; Wherein the perimeter data is used to determine whether the target vehicle is parked compliantly).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to incorporate the perimeter metadata taught by Nerayoff into the reference background images disclosed by Franklin in view of Li, Stumpe, and Panboonyuen. The suggestion/motivation for doing so would have been that they “may be used to specify…the position and extent of destination locations within the field of view of a camera.” (Nerayoff: 0049). Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Franklin in view of Li, Stumpe, and Panboonyuen with Nerayoff to obtain the invention as specified in claim 6.
Regarding claim 7, Franklin in view of Li, Stumpe, Panboonyuen, and Nerayoff discloses: The system of claim 6.
Franklin in view of Li, Stumpe, Panboonyuen, and Nerayoff does not disclose expressly: wherein the image portion corresponding to the target vehicle is identified using a vehicle detection machine learning model.
Nerayoff further discloses: wherein the image portion corresponding to the target vehicle is identified using a vehicle detection machine learning model (Nerayoff: 0072: “Many machine vision techniques are known in the art which may be used for detecting the presence of vehicles in video frames. For example, there are various known edge detection algorithms which may be used for detecting the presence of vehicles. As an example, an algorithm may detect the presence of a vehicle based on differences in color between the vehicle and the background, and whether a size of a total shape is larger than a minimum vehicle size expected in consideration of zoom level and/or location in a field of vision.”).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to substitute the vehicle detection technique disclosed by Franklin in view of Li, Stumpe, Panboonyuen, and Nerayoff with the edge detection algorithm further taught by Nerayoff. The suggestion/motivation for doing so would have been “The algorithm may further accommodate "holes" or imperfections in the shape, which may occur due to specular reflections or highlights, for example” (Nerayoff: 0072). Further, one skilled in the art could have substituted the elements as described above by known methods with no change in their respective functions, and the substitution would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Franklin in view of Li, Stumpe, Panboonyuen, and Nerayoff with the further teaching of Nerayoff to obtain the invention as specified in claim 7.
Regarding claim 17, Franklin in view of Li, Stumpe, Panboonyuen, and Nerayoff discloses: The system of claim 4, wherein the license plate detection machine learning model is configured to identify a portion of the captured image corresponding to a license plate of the target vehicle (Nerayoff: 0039: “server system 140 identifies license plates and/or additional identifications that are moving versus license plates and/or additional identifications which are stationary, based on multiple images captured by identification camera 120. In an example, server system 140 may be configured to: (1) based on license plates and/or additional identification information obtained, such as via OCR, from the current image, find corresponding matches in previous images”), and the license plate number is determined based on processing the portion of the captured image corresponding to the license plate by a character recognition module (Nerayoff: 0042: “OCR (optical character recognition) may be performed on any license plate image(s) in order to obtain an alphanumeric license plate number along with state/province of issue.”).
Claim(s) 11-16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Franklin in view of Li, Stumpe, and Panboonyuen, and further in view of Franklin et al. (US2008218383A1) hereinafter referenced as Franklin(2).
Regarding claim 11, Franklin in view of Li, Stumpe, and Panboonyuen discloses: The system of claim 1, wherein the one or more cameras are mounted on a surveillance vehicle (Franklin: 0065: “The vision system of the mobile enforcement vehicle may also comprise a number of cameras to obtain vehicle-identifying information including the license plate number…”).
Franklin in view of Li, Stumpe, and Panboonyuen does not disclose expressly: the computing device is carried by the surveillance vehicle, and the system further comprises a communication module to enable wireless communication between the computing device and a remote computer system.
Franklin(2) discloses: a computing device used to detect parking violations carried by the surveillance vehicle (Franklin(2): 0037: “The parking enforcement vehicle 18 is equipped with an infraction detection device 20…The device 20 determines whether parking infractions have occurred, informs the enforcement officer when an infraction has been detected, records information relating to the infraction, issues infraction notices, and transmits infraction information to a database for processing and storage, as is further described.” ), and the system further comprises a communication module to enable wireless communication between the computing device and a remote computer system (Franklin(2): 0045: “The communication application 48 allows for a communication link between the detection device 20 and the back office.”; 0048: “it is not necessary that the detection device 20 download the databases stored upon the backend office 60 prior to the commencement of a patrol, as the detection device can access the contents of such databases and write to such databases in real time upon the backend office 60”).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to implement the known technique taught by Franklin(2) of installing the processor to identify parking violations into the surveillance vehicle disclosed by Franklin in view of Li, Stumpe, and Panboonyuen. The suggestion/motivation for doing so would have been to improve performance and resource utilization without overwhelming the central processor as the number of surveillance vehicles increase. Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Franklin in view of Li, Stumpe, and Panboonyuen with Franklin(2) to obtain the invention as specified in claim 11.
Regarding claim 12, Franklin in view of Li, Stumpe, Panboonyuen, and Franklin(2) discloses: The system of claim 11, wherein the system is configured to perform parking monitoring in real time as the surveillance vehicle moves in the urban area (Franklin: 0059: “The mobile enforcement vehicle may also transmit vehicle and stall information in real time or in batch-mode. Vehicles for which citations are required may be determined immediately and a citation may be generated immediately (i.e. issuance of citations in real-time) or mailed in batches at a later time (i.e. issuance of citations in batch mode).”).
Regarding claim 13, Franklin in view of Li, Stumpe, Panboonyuen, and Franklin(2) discloses: The system of claim 11, wherein the system comprises at least two cameras (Franklin: 0065: “The vision system of the mobile enforcement vehicle may also comprise a number of cameras to obtain vehicle-identifying information including the license plate number…a dedicated license plate camera may be used as a part of a license plate reader (LPR) system. This dedicated camera may also be used to capture additional information including stall demarcation to provide redundancy.”).
Franklin in view of Li, Stumpe, Panboonyuen, and Franklin(2) does not disclose expressly: with at least one camera positioned on each side of the surveillance vehicle to perform parking monitoring on both sides of the surveillance vehicle.
Franklin(2) further discloses: with at least one camera positioned on each side of the surveillance vehicle to perform parking monitoring on both sides of the surveillance vehicle (Franklin(2): 0060: “the enforcement vehicle 18 as illustrated in FIG. 13 may be outfitted with a proximity detection system 24 located on either side of the enforcement vehicle 18, and with a vision system 26 that is able to capture images of vehicles parked on both sides of the street,”).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to implement the known technique of positioning one camera on each of the left and right sides of the surveillance vehicle disclosed in Franklin in view of Li, Stumpe, Panboonyuen, and Franklin(2) as further taught by Franklin(2). The suggestion/motivation for doing so would have been “and with a vision system 26 that is able to capture images of vehicles parked on both sides of the street, as one would find on a one way street” (Franklin(2): 0060). Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Franklin in view of Li, Stumpe, Panboonyuen, and Franklin(2) with the further teaching of Franklin(2) to obtain the invention as specified in claim 13.
Regarding claim 14, Franklin in view of Li, Stumpe, Panboonyuen, and Franklin(2) discloses: The system of claim 11, wherein the system comprises at least two cameras, the at least two cameras positioned to capture images on one side of the surveillance vehicle (Franklin: 0064-0065: “the camera used to capture the parking stall sign may also be used to capture other scene information, such as, for example, background and foreground information around the parking stall sign…The vision system of the mobile enforcement vehicle may also comprise a number of cameras to obtain vehicle-identifying information including the license plate number... a dedicated license plate camera may be used as a part of a license plate reader (LPR) system. This dedicated camera may also be used to capture additional information including stall demarcation to provide redundancy.”; Wherein both cameras must be facing the same direction); and the background matching module is configured to perform background matching using captured images from each of the at least two cameras to identify a matching reference background image (Franklin: 0089: “Identification of the correct parking stall may be further enhanced using reference photo(s) background (e.g. those that are images of the parking stall only, without the presence of a parked vehicle) and compared to those acquired by the mobile enforcement vehicle during its patrols.”; Wherein the cameras used to capture the parking stall sign and the vehicle-identification capture portions of the parking stall for the purposes of redundancy.).
Regarding claim 15, Franklin in view of Li, Stumpe, Panboonyuen, and Franklin(2) discloses: The system of claim 11, wherein the at least one processor is further configured to track the target vehicle across the captured images as the surveillance vehicle moves in the urban area (Franklin: Figure 7B; 0095: “If the parking stall is paid for (i.e. answering “Yes” at decision step 754), the method may proceed to decision step 756 where a comparison of the vehicle-identifying information obtained for the given parking stall during the most recent patrol and the immediately preceding patrol is made. If no difference is observed in the comparison, then it may be concluded that the same vehicle has been occupying the paid-for parking stall during the two patrols and the vehicle is validly parked (i.e. answering “No” at step 756) so the method may proceed to step 775 in which the method may end.”; Wherein the processor keeps track of target vehicles for consecutive patrols).
Regarding claim 16, Franklin in view of Li, Stumpe, Panboonyuen, and Franklin(2) discloses: The system of claim 11, wherein the at least one processor is further configured to transmit to the remote computer system via the communication module one or more of: the determined compliance of the target vehicle with the determined parking conditions; the determined license plate number corresponding to the target vehicle; the determined parking location; or captured images of the target vehicle (Franklin: 0090: “the mobile enforcement vehicle may also scan and transmit vehicle-identifying information of the vehicle occupying a given stall, the stall number and the time of observation to the central processor. The information may include one or more identifies, such as the license plate number…The central processor may proceed to associate the vehicle-identifying information to the identified parking stall, along with any previously received information (e.g. those obtained at time of payment by the parker or an earlier patrol).”; Wherein the central processor this association entails sending this information to a remote server.).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ANTHONY J RODRIGUEZ whose telephone number is (703)756-5821. The examiner can normally be reached Monday-Friday 10am-7pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Sumati Lefkowitz can be reached at (571) 272-3638. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ANTHONY J RODRIGUEZ/
Examiner, Art Unit 2672
/SUMATI LEFKOWITZ/Supervisory Patent Examiner, Art Unit 2672