Notice of Pre-AIA or AIA Status
1. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
2. Claims 1-20 are pending.
Claim Objections
3. Applicant is advised that should claim 10 be found allowable, claim 12 will be objected to under 37 CFR 1.75 as being a substantial duplicate thereof. When two claims in an application are duplicates or else are so close in content that they both cover the same thing, despite a slight difference in wording, it is proper after allowing one claim to object to the other as being a substantial duplicate of the allowed claim. See MPEP § 608.01(m).
Claim 12 is objected to because of the following informalities: said claim states “a” sign type and “a” sign value which are substantially identically claimed in claim 10 of which claim 12 depends via claim 11. Appropriate correction is required.
Claim Rejections - 35 USC § 102
4. In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 10 and 15-20 is/are rejected under 35 U.S.C. 102a1 as being anticipated by Kabkab et al. (US Patent Application Publication 2023/0119634), herein after referred to as Kabkab.
Regarding independent claim 10, Kabkab discloses a computer-implemented method of sign detection and classification for an autonomous vehicle (Figure 1 autonomous vehicle 100 including systems 110+120+140 as described in paragraphs [0025]-[0032]. Figures 7-8 depicts methods for perception system 130 of an autonomous vehicle as described in paragraph [0077]. Figure 9 depicts a computer device 900 for performing the methods of 700 and 800 as described in paragraph [0092]. The following reference numerals will be cited in regards to the above figures unless otherwise stated.), the method comprising:
receiving, from at least one first sensor (110), first sensor data (Figure 1 output of sensing system 110 sent to data processing system 120 for processing as described in paragraph [0031]. First sensor data herein after referred to as 110 output.);
detecting at least one region of interest (ROI) based on the first sensor data (110 output) (Paragraph [0032] describes SIM 132 to receive a region of a driving environment 101 which may be a cropped portion of camera and lidar images.), wherein the at least one ROI includes a street sign (422) (Paragraph [0043] describes the SIM 132 to combine the lidar and camera data for a particular region of interest that contains images of signs. Paragraphs [0018] and [0031] examples the objects within a frame including road (street) signs. Figure 2 and paragraph [0033] examples signs such as a stop sign etc. Paragraphs [0054]-[0055] describes the region of interest in regards to a patch of interest to include candidate signs 422.);
extracting ROI data (output of 320) of the at least one ROI from the first sensor data (110 output) (Paragraph [0055] describes the MLM-I 320 to include a number of convolutional layers to extract local and global context of the image of the candidate signs 422. Figure 3 depicts 320 to receive input from the first sensor data including Lidar 3310 and camera 312 (depicted in figure 1 as 110).); and
classifying (330), using a classification machine learning model (Figures 3 and 4B describes using machine learning model MLM 330 with filter 440 to classify signs based on input to 449 (ROI from MLM 320), as described in paragraphs [0062] and [0076].), the street sign (422) based on the ROI data (Paragraph [0062] describes SVM 134 with MLM-V 330 to include sign viability filter 440 (figure 4a) that performs sign classification based on the output of 320 (which is input to 134).), wherein the classification machine learning model is configured to output a sign type and a sign value of the street sign (Paragraphs [0032] and [0055] describes MLM320 to output sign type 430. The current application’s originally filed specification support for “sign value” is found in paragraph [0056] that states sign values may include a numerical value such as a speed limit, symbols, and/or text. Prior art Kabkab figures 4A-4B depicts sign type 430 is performed based on camera input 412. Paragraph [0020] describes object/text recognition is performed on the camera images, describing sign values.).
Regarding claim 15, Kabkab discloses the method of claim 10, further comprising:
detecting, using a detection machine learning model, the at least one ROI ([0054] MLM used for ROI).
Regarding claim 16, Kabkab discloses the method of claim 10, further comprising:
detecting the at least one ROI by:
determining a depth map based on the first sensor data (figure 3 310 or figure 4 410 [0041] Lidar intensity map); and
detecting the at least one ROI based on the depth map ([0054] ROI identified via images 410 and 412).
Regarding claim 17, Kabkab discloses the method of claim 10, further comprising:
receiving, from at least one second sensor (if 312 or 412 is first sensor), second sensor data, the at least one second sensor including at least one of an infrared sensor or a Light Detection and Ranging (LiDAR) sensor (310 or 410); and
augmenting detection of the at least one ROI with the second sensor data ([0054] ROI identified via images 410 and 412).
Regarding claim 18, Kabkab discloses the method of claim 10, further comprising:
detecting the at least one ROI based on at least one of a texture map or a hue-saturation-value (HSV) color space representation of the first sensor data ([0042] camera color images).
Regarding claim 19, Kabkab discloses the method of claim 10, further comprising:
receiving LiDAR data from the at least one first sensor (310 or 410); and
detecting the at least one ROI based on LiDAR data ([0054] ROI identified via images 410 and 412).
Regarding claim 20, Kabkab discloses the method of claim 10 further comprising:
reducing false positives in detection and/or classification based on external data (Figure 2 depict image-false signs in which mirror images are invalid. Paragraph [0062] describes using filter 440 to classify if the sign is an image false sign such as a reflection. Paragraph [0022] describes to reduce the false signals.).
Claim Rejections - 35 USC § 103
5. In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-9 and 11-14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kabkab in view of Chung (US Patent Application Publication 2016/0117562).
Regarding independent claim 1, Kabkab discloses an autonomy computing system of an autonomous vehicle (Figure 1 autonomous vehicle 100 including systems 110+120+140 as described in paragraphs [0025]-[0032]. Figures 7-8 depicts methods for perception system 130 of an autonomous vehicle as described in paragraph [0077]. Figure 9 depicts a computer device 900 for performing the methods of 700 and 800 as described in paragraph [0092]. The following reference numerals will be cited in regards to the above figures unless otherwise stated.), the autonomy computing system (105+900) comprising at least one processor (902) in communication with ([0093]) at least one memory device (904), the at least one processor (902) programmed to:
receive, from at least one first sensor (110), first sensor data (Figure 1 output of sensing system 110 sent to data processing system 120 for processing as described in paragraph [0031]. First sensor data herein after referred to as 110 output.);
detect at least one region of interest (ROI) based on the first sensor data (110 output) (Paragraph [0032] describes SIM 132 to receive a region of a driving environment 101 which may be a cropped portion of camera and lidar images.), wherein the at least one ROI includes a street sign (422) (Paragraph [0043] describes the SIM 132 to combine the lidar and camera data for a particular region of interest that contains images of signs. Paragraphs [0018] and [0031] examples the objects within a frame including road (street) signs. Figure 2 and paragraph [0033] examples signs such as a stop sign etc. Paragraphs [0054]-[0055] describes the region of interest in regards to a patch of interest to include candidate signs 422.), the at least one processor (902) further programmed to:
detect the at least one ROI by performing computations on measured dimensions of the at least one ROI to predefined types of the street sign at a depth corresponding to the at least one ROI (Paragraph [0041] descries depth information is acquired from lidar (figure 3 310, figure 4 410), or radar or sonar. Paragraph [0051] describes the outputted Lidar depth data is in 3D or 2D format (measured dimensions). Paragraph [0054] describes a trained MLM to identify regions, from the inputted lidar and camera images, contain signs of interest (making the identified region a region of interest or ROI). The ROI/patches of interest are then described in paragraph [0055] to be used to identify and determine the sign via computations that enable identification. Paragraph [0062] describes a look-up table of all known (predefined) types of signs.);
extract ROI data (output of 320) of the at least one ROI from the first sensor data (110 output) (Paragraph [0055] describes the MLM-I 320 to include a number of convolutional layers to extract local and global context of the image of the candidate signs 422. Figure 3 depicts 320 to receive input from the first sensor data including Lidar 3310 and camera 312 (depicted in figure 1 as 110).); and
classify (330) the street sign (422) based on the ROI data (Paragraph [0062] describes SVM 134 with MLM-V 330 to include sign viability filter 440 (figure 4a) that performs sign classification based on the output of 320 (which is input to 134).).
Kabkab does not specifically disclose to detect the at least one ROI by comparing measured dimensions of the at least one ROI to predefined dimensions of the street sign.
Chung discloses computations including to detect the at least one ROI by comparing measured dimensions of the at least one ROI to predefined dimensions of the street sign (Paragraphs [0019] and [0067]-[0072] describes determining an ROI via extracting a valid area information of at least one of size and shape which is compared with pre-stored data of a traffic sign’s size and shape.).
It would have been obvious to one skilled in the art before the effective filing date of the current application to enable Kabkab’s performed computations on measured dimensions of the at least one ROI with the known technique of comparing measured dimensions of the at least one ROI to predefined dimensions of the street sign yielding the predictable results of performing a calculated similarity score that improves performance of recognizing a traffic sign and decreasing the amount of calculation as disclosed by Chung (paragraph [0073]).
Regarding claim 2, Kabkab discloses the autonomy computing system of claim 1, wherein the at least one processor is further programmed to:
classify, using a classification machine learning model, the street sign based on the ROI data (Figures 3 and 4B describes using machine learning model MLM 330 with filter 440 to classify signs based on input to 449 (ROI from MLM 320), as described in paragraphs [0062] and [0076].).
Regarding claim 3, Kabkab discloses the autonomy computing system of claim 2, wherein the at least one processor is further programmed to:
classify, using the classification machine learning model, a sign type and a sign value of the street sign, wherein the sign type and the sign value are outputs from the classification machine learning model (Paragraphs [0032] and [0055] describes MLM320 to output sign type 430. The current application’s originally filed specification support for “sign value” is found in paragraph [0056] that states sign values may include a numerical value such as a speed limit, symbols, and/or text. Prior art Kabkab figures 4A-4B depicts sign type 430 is performed based on camera input 412. Paragraph [0020] describes object/text recognition is performed on the camera images, describing sign values.).
Regarding claim 4, Kabkab discloses the autonomy computing system of claim 2, wherein the at least one processor is further programmed to:
train the classification machine learning model using training data, the training data including synthetic data (Paragraph [0061] describes MLM to be trained using a plurality of variety of training images including real images and reference images of signs. Synthetic training data regards artificially created dataset and non-real reference images of signs is within the scope.).
Regarding claim 5, Kabkab discloses the autonomy computing system of claim 4, wherein the synthetic data includes manipulated images of street signs, image manipulation of images of the street signs including at least one of image distortion ([0061] training includes images during variety of conditions including clear, overcast, rainy, foggy etc. describing distorted images.), noise addition ([0070] confidence in view of noise added to images.), intensity manipulation, or color manipulation.
Regarding claim 6, Kabkab discloses the autonomy computing system of claim 1, wherein the at least one processor is further programmed to:
detect, using a detection machine learning model, the at least one ROI ([0054] MLM used for ROI).
Regarding claim 7, Kabkab discloses the autonomy computing system of claim 1, wherein the at least one processor is further programmed to:
extract the ROI data by:
transforming the first sensor data to a two-dimensional (2D) image ([0030] camera data is 2D, [0051] Lidar is 2D); and
extracting the ROI data by cropping (420) the 2D image at the at least one ROI ([0054]).
Regarding claim 8, Kabkab discloses the autonomy computing system of claim 1, wherein the at least one first sensor includes a stereo camera, the at least one processor further programmed to:
detect the at least one ROI by:
determining a depth map based on the first sensor data (figure 3 310 or figure 4 410 [0041] Lidar intensity map); and
detect the at least one ROI based on the depth map ([0054] ROI identified via images 410 and 412).
Regarding claim 9, Kabkab discloses the autonomy computing system of claim 1, wherein the at least one processor is further programmed to:
receive, from at least one second sensor (if 312 or 412 is first sensor), second sensor data, the at least one second sensor including at least one of an infrared sensor or a Light Detection and Ranging (LiDAR) sensor (310 or 410); and
augment detection of the at least one ROI with the second sensor data ([0054] ROI identified via images 410 and 412).
Regarding claim 11, Kabkab discloses the method claim 10, further comprising:
detecting the at least one ROI by performing computations on measured dimensions of the at least one ROI to predefined types of the street sign at a depth corresponding to the at least one ROI (Paragraph [0041] descries depth information is acquired from lidar (figure 3 310, figure 4 410), or radar or sonar. Paragraph [0051] describes the outputted Lidar depth data is in 3D or 2D format (measured dimensions). Paragraph [0054] describes a trained MLM to identify regions, from the inputted lidar and camera images, contain signs of interest (making the identified region a region of interest or ROI). The ROI/patches of interest are then described in paragraph [0055] to be used to identify and determine the sign via computations that enable identification. Paragraph [0062] describes a look-up table of all known (predefined) types of signs.).
Kabkab does not specifically disclose to detect the at least one ROI by comparing measured dimensions of the at least one ROI to predefined dimensions of the street sign.
Chung discloses computations including to detect the at least one ROI by comparing measured dimensions of the at least one ROI to predefined dimensions of the street sign (Paragraphs [0019] and [0067]-[0072] describes determining an ROI via extracting a valid area information of at least one of size and shape which is compared with pre-stored data of a traffic sign’s size and shape.).
It would have been obvious to one skilled in the art before the effective filing date of the current application to enable Kabkab’s performed computations on measured dimensions of the at least one ROI with the known technique of comparing measured dimensions of the at least one ROI to predefined dimensions of the street sign yielding the predictable results of performing a calculated similarity score that improves performance of recognizing a traffic sign and decreasing the amount of calculation as disclosed by Chung (paragraph [0073]).
Regarding claim 12, Kabkab discloses the method of claim 11, further comprising:
classifying, using the classification machine learning model, a sign type and a sign value of the street sign, wherein the sign type and the sign value are outputs from the classification machine learning model (Paragraphs [0032] and [0055] describes MLM320 to output sign type 430. The current application’s originally filed specification support for “sign value” is found in paragraph [0056] that states sign values may include a numerical value such as a speed limit, symbols, and/or text. Prior art Kabkab figures 4A-4B depicts sign type 430 is performed based on camera input 412. Paragraph [0020] describes object/text recognition is performed on the camera images, describing sign values.).
Regarding claim 13, Kabkab discloses the method of claim 11, further comprising:
training the classification machine learning model using training data, the training data including synthetic data (Paragraph [0061] describes MLM to be trained using a plurality of variety of training images including real images and reference images of signs. Synthetic training data regards artificially created dataset and non-real reference images of signs is within the scope.).
Regarding claim 14, Kabkab discloses the method of claim 13, further comprising:
training the classification machine learning model by:
training based on training data including manipulated images of street signs, image manipulation of images of the street signs including at least one of image distortion ([0061] training includes images during variety of conditions including clear, overcast, rainy, foggy etc. describing distorted images.), noise addition ([0070] confidence in view of noise added to images.), intensity manipulation, or color manipulation.
Conclusion
6. Any inquiry concerning this communication or earlier communications from the examiner should be directed to CHRISTOPHER E LEIBY whose telephone number is (571)270-3142. The examiner can normally be reached 11-7.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Amr Awad can be reached at 571-272-7764. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CHRISTOPHER E LEIBY/ Primary Examiner, Art Unit 2621
Prosecution Insights