Notice of Pre-AIA or AIA Status
1. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
2. Claims 1, 3-5, 7-10, 12-14, and 16-24 are pending. Bolded claim language below regards newly amended subject matter with a corresponding new rejection citation. Newly amended subject matter that is not bolded does not comprise a new rejection citation (utilizes previous interpretation that is unchanged in view of the new language) or is a newly added claim.
Claim Rejections - 35 USC § 103
3. In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 3-4, 7-10, 12-13, and 16-24 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kabkab et al. (US Patent Application Publication 2023/0119634), herein after referred to as Kabkab, in view of Chung (US Patent Application Publication 2016/0117562) and further in view of Sprunk et al. (2022/0063662), herein after referred to as Sprunk.
Regarding independent claim 1, Kabkab discloses an autonomy computing system of an autonomous vehicle (Figure 1 autonomous vehicle 100 including systems 110+120+140 as described in paragraphs [0025]-[0032]. Figures 7-8 depicts methods for perception system 130 of an autonomous vehicle as described in paragraph [0077]. Figure 9 depicts a computer device 900 for performing the methods of 700 and 800 as described in paragraph [0092]. The following reference numerals will be cited in regards to the above figures unless otherwise stated.), the autonomy computing system (105+900) comprising at least one processor (902) in communication with ([0093]) at least one memory device (904), the at least one processor (902) programmed to:
receive, from at least one first sensor (110), first sensor data (Figure 1 output of sensing system 110 sent to data processing system 120 for processing as described in paragraph [0031]. First sensor data herein after referred to as 110 output.);
detect at least one region of interest (ROI) based on the first sensor data (110 output) (Paragraph [0032] describes SIM 132 to receive a region of a driving environment 101 which may be a cropped portion of camera and lidar images.), wherein the at least one ROI includes a street sign (422) (Paragraph [0043] describes the SIM 132 to combine the lidar and camera data for a particular region of interest that contains images of signs. Paragraphs [0018] and [0031] examples the objects within a frame including road (street) signs. Figure 2 and paragraph [0033] examples signs such as a stop sign etc. Paragraphs [0054]-[0055] describes the region of interest in regards to a patch of interest to include candidate signs 422.), the at least one processor (902) further programmed to:
detect the at least one ROI by determining the at least one ROI by:
performing computations on measured dimensions of the at least one ROI to predefined types of the street sign at a depth corresponding to the at least one ROI (Paragraph [0041] descries depth information is acquired from lidar (figure 3 310, figure 4 410), or radar or sonar. Paragraph [0051] describes the outputted Lidar depth data is in 3D or 2D format (measured dimensions). Paragraph [0054] describes a trained MLM to identify regions, from the inputted lidar and camera images, contain signs of interest (making the identified region a region of interest or ROI). The ROI/patches of interest are then described in paragraph [0055] to be used to identify and determine the sign via computations that enable identification. Paragraph [0062] describes a look-up table of all known (predefined) types of signs.); and
[ ];
extract ROI data (output of 320) of the at least one ROI from the first sensor data (110 output) (Paragraph [0055] describes the MLM-I 320 to include a number of convolutional layers to extract local and global context of the image of the candidate signs 422. Figure 3 depicts 320 to receive input from the first sensor data including Lidar 3310 and camera 312 (depicted in figure 1 as 110).); and
classify (330), by the classification machine learning model (Figure 4A SIM 132 that classifies signs [0033] using MLM 320 [0044], Figures 3 and 4B describes using machine learning model MLM 330 with filter 440 to classify signs as reflections or not based on input to 449 (ROI from MLM 320), as described in paragraphs [0062] and [0076].), the street sign (422) based on the ROI data (Paragraph [0062] describes SVM 134 with MLM-V 330 to include sign viability filter 440 (figure 4a) that performs sign classification based on the output of 320 (which is input to 134).).
Kabkab does not specifically disclose to detect the at least one ROI by comparing measured dimensions of the at least one ROI to predefined dimensions of the street sign.
Chung discloses computations including to detect the at least one ROI by comparing measured dimensions of the at least one ROI to predefined dimensions of the street sign (Paragraphs [0019] and [0067]-[0072] describes determining an ROI via extracting a valid area information of at least one of size and shape which is compared with pre-stored data of a traffic sign’s size and shape.).
It would have been obvious to one skilled in the art before the effective filing date of the current application to enable Kabkab’s performed computations on measured dimensions of the at least one ROI with the known technique of comparing measured dimensions of the at least one ROI to predefined dimensions of the street sign yielding the predictable results of performing a calculated similarity score that improves performance of recognizing a traffic sign and decreasing the amount of calculation as disclosed by Chung (paragraph [0073]).
Kabhab does not specifically disclose determining the street sign is contained in the at least one ROI when the measure dimensions match the predefined dimensions.
Sprunk discloses determining the street sign is contained in the at least one ROI when the measure dimensions match the predefined dimensions ([0105] data can indicate that an area (ROI) of an environment 400 corresponds to an object of interest/pedestrian 402 by the system 412 matching/not matching expected (predefined) detections of sizes of the object in the environment 400 based on stored surfel data. [0080] examples that the object may be a street sign. [0024] emphasizes the surfel data to include size and shape parameters.).
It would have been obvious to one skilled in the art before the effective filing date of the current application to enable Kabkab’s ROI with the known technique of determining the street sign is contained in the at least one ROI when the measure dimensions match the predefined dimensions yielding the predictable results of provide output of unexpected objects to be within a sensed environment as disclosed by Sprunk ([0105]).
Regarding claim 3, Kabkab discloses the autonomy computing system of claim 1, wherein the at least one processor is further programmed to:
classify, using the classification machine learning model, a sign type and a sign value of the street sign, wherein the sign type and the sign value are outputs from the classification machine learning model (Paragraphs [0032] and [0055] describes MLM320 to output sign type 430. The current application’s originally filed specification support for “sign value” is found in paragraph [0056] that states sign values may include a numerical value such as a speed limit, symbols, and/or text. Prior art Kabkab figures 4A-4B depicts sign type 430 is performed based on camera input 412. Paragraph [0020] describes object/text recognition is performed on the camera images, describing sign values.).
Regarding claim 4, Kabkab discloses the autonomy computing system of claim 1, wherein the at least one processor is further programmed to:
train the classification machine learning model using training data, the training data including synthetic data (Paragraph [0061] describes MLM to be trained using a plurality of variety of training images including real images and reference images of signs. Synthetic training data regards artificially created dataset and non-real reference images of signs is within the scope.).
Regarding claim 7, Kabkab discloses the autonomy computing system of claim 1, wherein the at least one processor is further programmed to:
extract the ROI data by:
transforming the first sensor data to a two-dimensional (2D) image ([0030] camera data is 2D, [0051] Lidar is 2D); and
extracting the ROI data by cropping (420) the 2D image at the at least one ROI ([0054]).
Regarding claim 8, Kabkab discloses the autonomy computing system of claim 1, wherein the at least one first sensor includes a stereo camera, the at least one processor further programmed to:
detect the at least one ROI by:
determining a depth map based on the first sensor data (figure 3 310 or figure 4 410 [0041] Lidar intensity map); and
detect the at least one ROI based on the depth map ([0054] ROI identified via images 410 and 412).
Regarding claim 9, Kabkab discloses the autonomy computing system of claim 1, wherein the at least one processor is further programmed to:
receive, from at least one second sensor (if 312 or 412 is first sensor), second sensor data, the at least one second sensor including at least one of an infrared sensor or a Light Detection and Ranging (LiDAR) sensor (310 or 410); and
augment detection of the at least one ROI with the second sensor data ([0054] ROI identified via images 410 and 412).
Regarding independent claim 10, Kabkab discloses a computer-implemented method of sign detection and classification for an autonomous vehicle (Figure 1 autonomous vehicle 100 including systems 110+120+140 as described in paragraphs [0025]-[0032]. Figures 7-8 depicts methods for perception system 130 of an autonomous vehicle as described in paragraph [0077]. Figure 9 depicts a computer device 900 for performing the methods of 700 and 800 as described in paragraph [0092]. The following reference numerals will be cited in regards to the above figures unless otherwise stated.), the method comprising:
receiving, from at least one first sensor (110), first sensor data (Figure 1 output of sensing system 110 sent to data processing system 120 for processing as described in paragraph [0031]. First sensor data herein after referred to as 110 output.);
detecting at least one region of interest (ROI) based on the first sensor data (110 output) (Paragraph [0032] describes SIM 132 to receive a region of a driving environment 101 which may be a cropped portion of camera and lidar images.), wherein the at least one ROI includes a street sign (422) (Paragraph [0043] describes the SIM 132 to combine the lidar and camera data for a particular region of interest that contains images of signs. Paragraphs [0018] and [0031] examples the objects within a frame including road (street) signs. Figure 2 and paragraph [0033] examples signs such as a stop sign etc. Paragraphs [0054]-[0055] describes the region of interest in regards to a patch of interest to include candidate signs 422.), detecting the at least one ROI further comprising determining the at least one ROI by:
performing computations on measured dimensions of the at least one ROI to predefined types of the street sign at a depth corresponding to the at least one ROI (Paragraph [0041] descries depth information is acquired from lidar (figure 3 310, figure 4 410), or radar or sonar. Paragraph [0051] describes the outputted Lidar depth data is in 3D or 2D format (measured dimensions). Paragraph [0054] describes a trained MLM to identify regions, from the inputted lidar and camera images, contain signs of interest (making the identified region a region of interest or ROI). The ROI/patches of interest are then described in paragraph [0055] to be used to identify and determine the sign via computations that enable identification. Paragraph [0062] describes a look-up table of all known (predefined) types of signs.); and
[ ];
extracting ROI data (output of 320) of the at least one ROI from the first sensor data (110 output) (Paragraph [0055] describes the MLM-I 320 to include a number of convolutional layers to extract local and global context of the image of the candidate signs 422. Figure 3 depicts 320 to receive input from the first sensor data including Lidar 3310 and camera 312 (depicted in figure 1 as 110).); and
classifying (330), using a classification machine learning model (Figures 3 and 4B describes using machine learning model MLM 330 with filter 440 to classify signs based on input to 449 (ROI from MLM 320), as described in paragraphs [0062] and [0076].), the street sign (422) based on the ROI data (Paragraph [0062] describes SVM 134 with MLM-V 330 to include sign viability filter 440 (figure 4a) that performs sign classification based on the output of 320 (which is input to 134).), wherein the classification machine learning model is configured to output a sign type and a sign value of the street sign (Paragraphs [0032] and [0055] describes MLM320 to output sign type 430. The current application’s originally filed specification support for “sign value” is found in paragraph [0056] that states sign values may include a numerical value such as a speed limit, symbols, and/or text. Prior art Kabkab figures 4A-4B depicts sign type 430 is performed based on camera input 412. Paragraph [0020] describes object/text recognition is performed on the camera images, describing sign values.).
Kabkab does not specifically disclose to detect the at least one ROI by comparing measured dimensions of the at least one ROI to predefined dimensions of the street sign.
Chung discloses computations including to detect the at least one ROI by comparing measured dimensions of the at least one ROI to predefined dimensions of the street sign (Paragraphs [0019] and [0067]-[0072] describes determining an ROI via extracting a valid area information of at least one of size and shape which is compared with pre-stored data of a traffic sign’s size and shape.).
It would have been obvious to one skilled in the art before the effective filing date of the current application to enable Kabkab’s performed computations on measured dimensions of the at least one ROI with the known technique of comparing measured dimensions of the at least one ROI to predefined dimensions of the street sign yielding the predictable results of performing a calculated similarity score that improves performance of recognizing a traffic sign and decreasing the amount of calculation as disclosed by Chung (paragraph [0073]).
Kabhab does not specifically disclose determining the street sign is contained in the at least one ROI when the measure dimensions match the predefined dimensions.
Sprunk discloses determining the street sign is contained in the at least one ROI when the measure dimensions match the predefined dimensions ([0105] data can indicate that an area (ROI) of an environment 400 corresponds to an object of interest/pedestrian 402 by the system 412 matching/not matching expected (predefined) detections of sizes of the object in the environment 400 based on stored surfel data. [0080] examples that the object may be a street sign. [0024] emphasizes the surfel data to include size and shape parameters.).
It would have been obvious to one skilled in the art before the effective filing date of the current application to enable Kabkab’s ROI with the known technique of determining the street sign is contained in the at least one ROI when the measure dimensions match the predefined dimensions yielding the predictable results of provide output of unexpected objects to be within a sensed environment as disclosed by Sprunk ([0105]).
Regarding claim 12, Kabkab discloses the method of claim 10, further comprising:
classifying, using the classification machine learning model, a sign type and a sign value of the street sign, wherein the sign type and the sign value are outputs from the classification machine learning model (Paragraphs [0032] and [0055] describes MLM320 to output sign type 430. The current application’s originally filed specification support for “sign value” is found in paragraph [0056] that states sign values may include a numerical value such as a speed limit, symbols, and/or text. Prior art Kabkab figures 4A-4B depicts sign type 430 is performed based on camera input 412. Paragraph [0020] describes object/text recognition is performed on the camera images, describing sign values.).
Regarding claim 13, Kabkab discloses the method of claim 10, further comprising:
training the classification machine learning model using training data, the training data including synthetic data (Paragraph [0061] describes MLM to be trained using a plurality of variety of training images including real images and reference images of signs. Synthetic training data regards artificially created dataset and non-real reference images of signs is within the scope.).
Regarding claim 16, Kabkab discloses the method of claim 10, further comprising:
detecting the at least one ROI by:
determining a depth map based on the first sensor data (figure 3 310 or figure 4 410 [0041] Lidar intensity map); and
detecting the at least one ROI based on the depth map ([0054] ROI identified via images 410 and 412).
Regarding claim 17, Kabkab discloses the method of claim 10, further comprising:
receiving, from at least one second sensor (if 312 or 412 is first sensor), second sensor data, the at least one second sensor including at least one of an infrared sensor or a Light Detection and Ranging (LiDAR) sensor (310 or 410); and
augmenting detection of the at least one ROI with the second sensor data ([0054] ROI identified via images 410 and 412).
Regarding claim 18, Kabkab discloses the method of claim 10, further comprising:
detecting the at least one ROI based on at least one of a texture map or a hue-saturation-value (HSV) color space representation of the first sensor data ([0042] camera color images).
Regarding claim 19, Kabkab discloses the method of claim 10, further comprising:
receiving LiDAR data from the at least one first sensor (310 or 410); and
detecting the at least one ROI based on LiDAR data ([0054] ROI identified via images 410 and 412).
Regarding claim 20, Kabkab discloses the method of claim 10 further comprising:
reducing false positives in detection and/or classification based on external data (Figure 2 depict image-false signs in which mirror images are invalid. Paragraph [0062] describes using filter 440 to classify if the sign is an image false sign such as a reflection. Paragraph [0022] describes to reduce the false signals.).
Regarding claim 21, Kabkab discloses the autonomy computing system of claim 4, wherein the synthetic data includes manipulated images of street signs, image manipulation of images of the street signs including noise addition ([0070] confidence in view of noise added to images).
Regarding claim 22, Kabkab discloses the autonomy computing system of claim 1, wherein the at least one processor is further programmed to:
classify the street sign by:
cropping the first sensor data at the at least one ROI ( [0032] and [0054] describes SIM 132 to receive a region of a driving environment 101 which may be a cropped portion of camera and lidar images via imaging cropping module 420); and
using the cropped first sensor data as inputs to the classification machine learning model (Figure 4A reference output of image cropping module 420 input to MLM 320 as described in [0054]-[0055]), thereby reducing input data sizes to the classification machine learning model (means for language such that performing the first limitation of “using the cropped first sensor data as inputs to the MLM” enables the means for language of “reducing…”; prior art Kabkab discloses using inputs of cropped images (patches) rather than the entire image as inputs to MLM 320. This inherently comprises reduction in data size of a patch compared to the entire image in addition to the patch.).
Regarding claim 23, Kabkab discloses the method of claim 13, wherein training the classification machine learning model further comprises:
training the classification machine learning model using the training data including manipulated images of street signs, image manipulation of images of the street signs including noise addition ([0070] confidence in view of noise added to images).
Regarding claim 24, Kabkab discloses the method of claim 10, wherein classifying the street sign further comprises:
cropping the first sensor data at the at least one ROI ( [0032] and [0054] describes SIM 132 to receive a region of a driving environment 101 which may be a cropped portion of camera and lidar images via imaging cropping module 420); and
using the cropped first sensor data as inputs to the classification machine learning model (Figure 4A reference output of image cropping module 420 input to MLM 320 as described in [0054]-[0055]), thereby reducing input data sizes to the classification machine learning model (means for language such that performing the first limitation of “using the cropped first sensor data as inputs to the MLM” enables the means for language of “reducing…”; prior art Kabkab discloses using inputs of cropped images (patches) rather than the entire image as inputs to MLM 320. This inherently comprises reduction in data size of a patch compared to the entire image in addition to the patch.).
4. Claim(s) 5 and 14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kabkab-Chung-Sprunk in view of Georgescu et al. (2025/0095340), herein after referred to as Georgescu.
Regarding claim 5, Kabkab discloses the autonomy computing system of claim 4.
Kabkab does not specifically disclose wherein the synthetic data includes manipulated images of street signs, image manipulation of images of the street signs including at least one of image distortion, intensity manipulation, or color manipulation.
Georgescu discloses wherein synthetic data includes color manipulation ([0035]).
It would have been obvious to one skilled in the art before the effective filing date of the current application to enable Kabkab-Chung-Sprunk’s synthetic data includes manipulated images of color manipulation yielding the predictable results of augmenting real training data as disclosed by Sprunk ([0035]).
Regarding claim 14, Kabkab discloses the method of claim 13.
Kabkab does not specifically disclose training the classification machine learning model using the training data including manipulated images of street signs, image manipulation of images of the street signs including at least one of image distortion, intensity manipulation, or color manipulation.
Georgescu discloses wherein synthetic data includes color manipulation ([0035]).
It would have been obvious to one skilled in the art before the effective filing date of the current application to enable Kabkab-Chung-Sprunk’s synthetic data includes manipulated images of color manipulation yielding the predictable results of augmenting real training data as disclosed by Sprunk ([0035]).
Response to Arguments
5. Applicant's arguments filed 9/3/2026 have been fully considered and relate to newly amended subject matter. Newly cited art Sprunk and Georgescu are utilized above in combination with Kabkab and Chung to reject the newly amended subject matter. Please refer to the above office action as the rebuttal.
Conclusion
7. The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Shetty et al. (US Patent Application Publication 2026/0100042) a VLM to identify ROIs within an image (abstract)
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CHRISTOPHER E LEIBY whose telephone number is (571)270-3142. The examiner can normally be reached 11-7.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Amr Awad can be reached at 571-272-7764. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CHRISTOPHER E LEIBY/ Primary Examiner, Art Unit 2621
