DETAILED ACTION
Claims 1-3, 5-14, 16-17, 19-21, 23-26 are currently pending and have been examined in this application. Claims 4, 15, 18, & 22 have been Cancelled. Claims 25-26 are New.
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This action is mad FINAL in response to the “amendment” and “remarks” filed 12/22/2025.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-2, 3, 5, 11-14, 23-26 is/are rejected under 35 U.S.C. 103 as being unpatentable over Banvait (US20170248955) in view of Kim (US20170305427) further in view of Reiff (US20170096138).
Claim 1:
Banvait explicitly teaches:
A method of detecting and responding to sounds for a vehicle having an autonomous driving mode, the method comprising:
(Banvait) – “A controller for an autonomous vehicle receives audio signals from one or more microphones. The audio signals are input to a machine learning model that classifies the source of the audio features. For example, features may be classified as originating from a vehicle. A direction to a source of the audio features is determined based on relative delays of the audio features in signals from multiple microphones. Where audio features are classified with an above-threshold confidence as originating from a vehicle, collision avoidance is performed with respect to the direction to the source of the audio features.” (Abstract)
receiving, by one or more processors of the vehicle, an audible signal corresponding to a sound received at one or more microphones of the vehicle;
(Banvait) – “A controller for an autonomous vehicle receives audio signals from one or more microphones. The audio signals are input to a machine learning model that classifies the source of the audio features. For example, features may be classified as originating from a vehicle. A direction to a source of the audio features is determined based on relative delays of the audio features in signals from multiple microphones. Where audio features are classified with an above-threshold confidence as originating from a vehicle, collision avoidance is performed with respect to the direction to the source of the audio features.” (Abstract)
“The present invention is described below with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions or code. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.” (Para 0016)
“Referring to FIG. 1, a controller 102 may be housed within a vehicle.” (Para 0019)
“FIG. 2 is a block diagram illustrating an example computing device 200. Computing device 200 may be used to perform various procedures, such as those discussed herein. The controller 102 may have some or all of the attributes of the computing device 200.” (Para 0030)
“Computing device 200 includes one or more processor(s) 202” (Para 0031)
Examiner Note: It will be hereinafter understood that the processes taught in Banvait are implemented using at least a processor.
determining, by the one or more processors, a type of the sound and a corresponding likelihood value for the type of the sound, [wherein the type of the sound is selected from a plurality of sound types including different vehicle types];
(Banvait) – “A controller for an autonomous vehicle receives audio signals from one or more microphones. The audio signals are input to a machine learning model that classifies the source of the audio features. For example, features may be classified as originating from a vehicle. A direction to a source of the audio features is determined based on relative delays of the audio features in signals from multiple microphones. Where audio features are classified with an above-threshold confidence as originating from a vehicle, collision avoidance is performed with respect to the direction to the source of the audio features.” (Abstract)
“The audio detection module 110a may further include a machine learning module 112b that implements a model that evaluates features in processed audio streams from the pre-processing module 112a and attempts to classify the audio features. The machine learning module 112b may output a confidence score indicating a likelihood that a classification is correct.” (Para 0023)
“the outputs of the noise cancelation modules 400a-400d may be input to a machine learning model 402 that classifies features in the outputs as corresponding to a particular vehicle. The machine learning model 112b may further output a confidence in the classification.” (Para 0041)
Examiner Note: Classification corresponds to type. Confidence score is a measure of likelihood.
[determining, by the one or more processors, whether the corresponding likelihood value meets a threshold; and when the corresponding likelihood value is determined to meet the by searching the map information and sensor data for the one or more objects of the determined type of the sound; and
(Banvait) – “The map correlation module 112d evaluates map data to determine whether a parking stall, driveway, or other parking area is located within the angular tolerance from the direction to the source of a sound corresponding to a vehicle with its engine running, particularly a parked vehicle. If so, then the confidence that the sound corresponds to a parked vehicle with its engine running is increased.” (Para 0025)
“The method 500 may further include attempting to validate the classification performed at step 508 using one or more other sources of information. For example, the method 500 may include attempting to correlate the direction to the sound origin with a vehicle image located within an output of an imaging sensor 104 at a position corresponding to the direction to the sound origin. For example, any vehicle image located within an angular region including the direction may be identified at step 512. If a vehicle image is found in the image stream of the imaging sensor 104 within the angular region, then a confidence value may be increased.” (Para 0052)
“The direction to the source of the audio features may be correlated with vehicle images and/or map data to increase a confidence score that the source of the audio features is a parked vehicle with its engine running.” (Abstract)
“The vehicle 300 may be close enough to detect sound generated by an occluded vehicle 304 or a parked vehicle 304. Although the methods disclosed herein are particularly useful where there is an occluding object, the identification of obstacles as described herein may be performed where image data is available and may, for example, confirm the location of an obstacle that is also visible to imaging devices 104. Likewise, the existence of a parked vehicle with its engine running may be confirmed using imaging devices 104 as described in greater detail below.” (Para 0040)
“The direction as estimated by the microphone array processing module 404 and the classification and confidence score as generated by the machine learning model 402 may then be provided as an output 406 from the machine learning module 112b. For example, the obstacle identification module 110b may add a vehicle having the identified class located at the estimated direction to a set of potential obstacles, the set of potential obstacles including any obstacles identified by other means, such as using the imaging devices 104.” (Para 0048)
Examiner Note: Bracketed text not explicitly taught by the primary reference, but is taught by non-primary reference later in the rejection.
controlling, by the one or more processors, the vehicle in the autonomous driving mode in order to:
(Banvait) – “A controller for an autonomous vehicle receives audio signals from one or more microphones. The audio signals are input to a machine learning model that classifies the source of the audio features. For example, features may be classified as originating from a vehicle. A direction to a source of the audio features is determined based on relative delays of the audio features in signals from multiple microphones. Where audio features are classified with an above-threshold confidence as originating from a vehicle, collision avoidance is performed with respect to the direction to the source of the audio features.” (Abstract)
respond to the sound based on the type of the sound,
(Banvait) – “The method 500 may include evaluating 516 whether the confidence score of step 508 exceeds a threshold. For example, where no classifications at step 508 have a confidence score above a threshold, the method 500 may include determining that the audio features that were the basis of the classification likely do not correspond to a vehicle. Otherwise, if the confidence score does exceed a threshold, then the method 500 may include adding 518 a potential obstacle to a set of obstacles identified by other means, such as using imaging devices 104. The potential obstacle may be defined as a potential obstacle located in the direction or range of angles determined at step 510.” (Para 0055)
“A controller for an autonomous vehicle receives audio signals from one or more microphones. The audio signals are input to a machine learning model that classifies the source of the audio features. For example, features may be classified as originating from a vehicle. A direction to a source of the audio features is determined based on relative delays of the audio features in signals from multiple microphones. Where audio features are classified with an above-threshold confidence as originating from a vehicle, collision avoidance is performed with respect to the direction to the source of the audio features.” (Abstract)
the corresponding likelihood value, and the identified one or more objects
(Banvait) – “The method 500 may include evaluating 516 whether the confidence score of step 508 exceeds a threshold. For example, where no classifications at step 508 have a confidence score above a threshold, the method 500 may include determining that the audio features that were the basis of the classification likely do not correspond to a vehicle. Otherwise, if the confidence score does exceed a threshold, then the method 500 may include adding 518 a potential obstacle to a set of obstacles identified by other means, such as using imaging devices 104. The potential obstacle may be defined as a potential obstacle located in the direction or range of angles determined at step 510.” (Para 0055)
“The method 500 may further include attempting to validate the classification performed at step 508 using one or more other sources of information. For example, the method 500 may include attempting to correlate the direction to the sound origin with a vehicle image located within an output of an imaging sensor 104 at a position corresponding to the direction to the sound origin. For example, any vehicle image located within an angular region including the direction may be identified at step 512. If a vehicle image is found in the image stream of the imaging sensor 104 within the angular region, then a confidence value may be increased.” (Para 0052)
“In either outcome of step 516, obstacles are detected using other sensing systems, such as the imaging devices 104, and obstacles detected using these sensing systems are added 520 to the obstacle set. Collision avoidance is performed 522 with respect to the obstacle set.” (Para 0057)
“The audio detection module 110a may further include a machine learning module 112b that implements a model that evaluates features in processed audio streams from the pre-processing module 112a and attempts to classify the audio features. The machine learning module 112b may output a confidence score indicating a likelihood that a classification is correct.” (Para 0023)
Banvait does not explicitly teach:
wherein the type of the sound is selected from a plurality of sound types including different vehicle types …determining, by the one or more processors, whether the corresponding likelihood value meets a threshold; and when the corresponding likelihood value is determined to meet the
Kim, in the same field of endeavor of vehicle control, teaches:
determining, by the one or more processors, whether the corresponding likelihood value meets a threshold; and when the corresponding likelihood value is determined to meet the
(Kim) – “The sound recognizer 130 may generate a determination result according to whether the confidence level is higher than or equal to a threshold (e.g., 0.7) and include the same in the sound classification result. That is, if the confidence level is higher than or equal to the threshold, the sound recognizer 130 may determine the type of sound of a class corresponding to the confidence level as the type of the present sound data.” (Para 0050)
“The sound recognizer 130 may classify the feature values of the acquired sound data using a classifier and determine whether the acquired sound data corresponds to a sound in which the user is interested.” (Para 0046)
“the results of sound detection may be information about the probabilities of presence of a vehicle (hereinafter, a “neighboring vehicle”) travelling around an object (the vehicle 10) (hereinafter, a “host vehicle”)” (Para 0061)
“Therefore, with a method for providing sound detection information, an apparatus for detecting sound around a vehicle, and a vehicle including the same according to an exemplary embodiment in the present disclosure configured as above, peripheral devices of the vehicle may be controlled to be set to modes appropriate for tunnel travel by accurately detecting entry into or exit from a tunnel based only on information about sound around the vehicle.” (Para 0105)
Therefore, it would be obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the system of Banvait with the method for providing sound detection information of Kim. One of ordinary skill in the art would have been motivated to make these modifications, with a reasonable expectation of success, because “there is a need for a technology to report information about a specific sound such as the identity of the sound generated around the vehicle and the direction of the sound, without undermining driving safety.” (Kim Para 0004)
Kim does not explicitly teach:
wherein the type of the sound is selected from a plurality of sound types including different vehicle types
Reiff, in the same field of endeavor of vehicle control, teaches:
wherein the type of the sound is selected from a plurality of sound types including different vehicle types
(Reiff) – “A controller for an autonomous vehicle receives audio signals from one or more microphones and identifies sounds. The controller further identifies an estimated location of the sound origin and the type of sound, i.e. whether the sound is a vehicle and/or the type of vehicle. The controller analyzes map data and attempts to identify a landmark within a tolerance from the estimated location. If a landmark is found corresponding to the estimated location and type of the sound origin, then the certainty is increased that the source of the sound is at that location and is that type of sound source. Collision avoidance is then performed with respect to the location of the sound origin and its type with the certainty as augmented using the map data. Collision avoidance may include automatically actuating brake, steering, and accelerator actuators in order to avoid the location of the sound origin.” (Abstract)
“The controller 102 may include or access a database 104 housed in the vehicle or otherwise accessible by the controller 102. The database 104 may include data sufficient to enable identification of an obstacle using map data. For example, sound data 106 may contain data describing sounds generated by one or more types of vehicles or other potential obstacles. For example, sound data 106 may include samples of the sounds made by one or more types of vehicles, animals (e.g. a dog barking), people conversing, and the like.” (Para 0020)
Therefore, it would be obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the system of Banvait with the Collision Avoidance Using Auditory Data Augmented With Map Data of Reiff. One of ordinary skill in the art would have been motivated to make these modifications, with a reasonable expectation of success, in order to “provide an improved approach for detecting obstacles.” (Reiff Para 0005)
Claim 2:
Banvait in combination with the references relied upon in Claim 1 teach those respective limitations. Banvait further teaches:
selecting, by the one or more processors, at least one signal from a plurality of sets of additional signals based on the type of the sound, [each one of the plurality of sets being associated with a different type of sound]
(Banvait) – “The collision avoidance module 108 may further include an obstacle identification module 110b, a collision prediction module 110c, and a decision module 110d. The obstacle identification module 110b analyzes the one or more image streams and identifies potential obstacles, including people, animals, vehicles, buildings, curbs, and other objects and structures. In particular, the obstacle identification module 110b may identify vehicle images in the image stream.” (Para 0026)
“The collision prediction module 110c predicts which obstacle images are likely to collide with the vehicle based on its current trajectory or current intended path. The collision prediction module 110c may evaluate the likelihood of collision with objects identified by the obstacle identification module 110b as well as obstacles detected using the audio detection module 110a…The decision module 110d may make a decision to stop, accelerate, turn, etc. in order to avoid obstacles. The manner in which the collision prediction module 110c predicts potential collisions and the manner in which the decision module 110d takes action to avoid potential collisions may be according to any method or system known in the art of autonomous vehicles.” (Para 0027)
Examiner Note: The claimed limitation, in light of the specification, gives no criteria for selection. As such any identified relevant signal may be considered to be selected.
Banvait does not explicitly teach:
each one of the plurality of sets of additional signals being associated with a different type of sound
Reiff, in the same field of endeavor of vehicle control, teaches:
each one of the plurality of sets of additional signals being associated with a different type of sound
(Reiff) – “The method 400 may include one or both of estimating 406 a distance to the origin of the sound and estimating 408 a direction to the origin of the sound.” (Para 0044)
“The method 400 may include retrieving 410 map data in a region including the estimated location of the sound origin as determined at steps 406 and 408. And evaluating 412 whether the map data includes a landmark corresponding to the location and candidate source of the sound origin. For example, a landmark closest to the location of the sound origin may be identified 412. For example, where the candidate sound source is determined at step 404 to be a vehicle and a parking garage is within a specified tolerance from the location determined at steps 406 and 408, then it may be determined that the parking garage is the landmark corresponding to the sound detected at step 402. As noted above, the tolerance may be a region or range of angles and distances corresponding to the uncertainty in determining the location, direction, and distance, respectively of the sound origin. In another scenario, an emergency vehicle station is within the tolerance from the sound origin and the candidate sound source is an emergency vehicle, then it may be determined at step 412 that the landmark corresponding to the sound detected at step 402 is the emergency vehicle station.” (Para 0045)
Examiner Note: Reiff shows different signals (map data/landmarks) representing different types of sounds (vehicle/emergency vehicle).
Therefore, it would be obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the system of Banvait with the Collision Avoidance Using Auditory Data Augmented With Map Data of Reiff. One of ordinary skill in the art would have been motivated to make these modifications, with a reasonable expectation of success, in order to “provide an improved approach for detecting obstacles.” (Reiff Para 0005)
Claim 3:
Banvait in combination with the references relied upon in Claim 1 teach those respective limitations. Banvait further teaches:
receiving, by the one or more processors, sensor data from one or more sensors of the vehicle,
(Banvait) – “The method 500 may include evaluating 516 whether the confidence score of step 508 exceeds a threshold. For example, where no classifications at step 508 have a confidence score above a threshold, the method 500 may include determining that the audio features that were the basis of the classification likely do not correspond to a vehicle. Otherwise, if the confidence score does exceed a threshold, then the method 500 may include adding 518 a potential obstacle to a set of obstacles identified by other means, such as using imaging devices 104. The potential obstacle may be defined as a potential obstacle located in the direction or range of angles determined at step 510.” (Para 0055)
“In either outcome of step 516, obstacles are detected using other sensing systems, such as the imaging devices 104, and obstacles detected using these sensing systems are added 520 to the obstacle set. Collision avoidance is performed 522 with respect to the obstacle set.” (Para 0057)
Claim 4: Cancelled
Claim 5:
Banvait in combination with the references relied upon in Claim 1 teach those respective limitations. Banvait further teaches:
further comprising inputting the audible signal into a classifier.
(Banvait) – “A controller for an autonomous vehicle receives audio signals from one or more microphones. The audio signals are input to a machine learning model that classifies the source of the audio features. For example, features may be classified as originating from a vehicle. A direction to a source of the audio features is determined based on relative delays of the audio features in signals from multiple microphones. Where audio features are classified with an above-threshold confidence as originating from a vehicle, collision avoidance is performed with respect to the direction to the source of the audio features.” (Abstract)
“The audio detection module 110a may further include a machine learning module 112b that implements a model that evaluates features in processed audio streams from the pre-processing module 112a and attempts to classify the audio features.” (Para 0023)
“the outputs of the noise cancelation modules 400a-400d may be input to a machine learning model 402 that classifies features in the outputs as corresponding to a particular vehicle. The machine learning model 112b may further output a confidence in the classification.” (Para 0041)
Examiner Note: Machine learning model corresponds to classifier.
Claim 11:
Banvait explicitly teaches:
A system for detecting and responding to sounds for a vehicle having an autonomous driving mode, the system comprising:
(Banvait) – “A controller for an autonomous vehicle receives audio signals from one or more microphones. The audio signals are input to a machine learning model that classifies the source of the audio features. For example, features may be classified as originating from a vehicle. A direction to a source of the audio features is determined based on relative delays of the audio features in signals from multiple microphones. Where audio features are classified with an above-threshold confidence as originating from a vehicle, collision avoidance is performed with respect to the direction to the source of the audio features.” (Abstract)
one or more microphones: and one or more processors configured to:
(Banvait) – “A controller for an autonomous vehicle receives audio signals from one or more microphones. The audio signals are input to a machine learning model that classifies the source of the audio features. For example, features may be classified as originating from a vehicle. A direction to a source of the audio features is determined based on relative delays of the audio features in signals from multiple microphones. Where audio features are classified with an above-threshold confidence as originating from a vehicle, collision avoidance is performed with respect to the direction to the source of the audio features.” (Abstract)
“The present invention is described below with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions or code. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.” (Para 0016)
Examiner Note: It will be hereinafter understood that the processes taught in Banvait are implemented using at least a processor.
receive an audible signal corresponding to a sound received at the one or more microphonesdetermine a type of the sound and a corresponding likelihood value for the type of the sound, [wherein the type of the sound is selected from a plurality of sound types including different vehicle types];
(Banvait) – “A controller for an autonomous vehicle receives audio signals from one or more microphones. The audio signals are input to a machine learning model that classifies the source of the audio features. For example, features may be classified as originating from a vehicle. A direction to a source of the audio features is determined based on relative delays of the audio features in signals from multiple microphones. Where audio features are classified with an above-threshold confidence as originating from a vehicle, collision avoidance is performed with respect to the direction to the source of the audio features.” (Abstract)
“The audio detection module 110a may further include a machine learning module 112b that implements a model that evaluates features in processed audio streams from the pre-processing module 112a and attempts to classify the audio features. The machine learning module 112b may output a confidence score indicating a likelihood that a classification is correct.” (Para 0023)
Examiner Note: Confidence score is a measure of likelihood.
[determine, by the one or more processors, whether the corresponding likelihood value meets a threshold; and when the corresponding likelihood value is determined to meet the ], use, by the one or more processors, map information and sensor data to identifyone or more objects associated with a source of the by searching the map information and sensor data for the one or more objects of the determined type of the sound; and
(Banvait) – “The map correlation module 112d evaluates map data to determine whether a parking stall, driveway, or other parking area is located within the angular tolerance from the direction to the source of a sound corresponding to a vehicle with its engine running, particularly a parked vehicle. If so, then the confidence that the sound corresponds to a parked vehicle with its engine running is increased.” (Para 0025)
“The method 500 may further include attempting to validate the classification performed at step 508 using one or more other sources of information. For example, the method 500 may include attempting to correlate the direction to the sound origin with a vehicle image located within an output of an imaging sensor 104 at a position corresponding to the direction to the sound origin. For example, any vehicle image located within an angular region including the direction may be identified at step 512. If a vehicle image is found in the image stream of the imaging sensor 104 within the angular region, then a confidence value may be increased.” (Para 0052)
“The direction to the source of the audio features may be correlated with vehicle images and/or map data to increase a confidence score that the source of the audio features is a parked vehicle with its engine running.” (Abstract)
“The vehicle 300 may be close enough to detect sound generated by an occluded vehicle 304 or a parked vehicle 304. Although the methods disclosed herein are particularly useful where there is an occluding object, the identification of obstacles as described herein may be performed where image data is available and may, for example, confirm the location of an obstacle that is also visible to imaging devices 104. Likewise, the existence of a parked vehicle with its engine running may be confirmed using imaging devices 104 as described in greater detail below.” (Para 0040)
“The direction as estimated by the microphone array processing module 404 and the classification and confidence score as generated by the machine learning model 402 may then be provided as an output 406 from the machine learning module 112b. For example, the obstacle identification module 110b may add a vehicle having the identified class located at the estimated direction to a set of potential obstacles, the set of potential obstacles including any obstacles identified by other means, such as using the imaging devices 104.” (Para 0048)
Examiner Note: Bracketed text not explicitly taught by the primary reference, but is taught by non-primary reference later in the rejection.
control the vehicle in the autonomous driving mode in order to:
(Banvait) – “A controller for an autonomous vehicle receives audio signals from one or more microphones. The audio signals are input to a machine learning model that classifies the source of the audio features. For example, features may be classified as originating from a vehicle. A direction to a source of the audio features is determined based on relative delays of the audio features in signals from multiple microphones. Where audio features are classified with an above-threshold confidence as originating from a vehicle, collision avoidance is performed with respect to the direction to the source of the audio features.” (Abstract)
(Banvait) – “The method 500 may include evaluating 516 whether the confidence score of step 508 exceeds a threshold. For example, where no classifications at step 508 have a confidence score above a threshold, the method 500 may include determining that the audio features that were the basis of the classification likely do not correspond to a vehicle. Otherwise, if the confidence score does exceed a threshold, then the method 500 may include adding 518 a potential obstacle to a set of obstacles identified by other means, such as using imaging devices 104. The potential obstacle may be defined as a potential obstacle located in the direction or range of angles determined at step 510.” (Para 0055)
“A controller for an autonomous vehicle receives audio signals from one or more microphones. The audio signals are input to a machine learning model that classifies the source of the audio features. For example, features may be classified as originating from a vehicle. A direction to a source of the audio features is determined based on relative delays of the audio features in signals from multiple microphones. Where audio features are classified with an above-threshold confidence as originating from a vehicle, collision avoidance is performed with respect to the direction to the source of the audio features.” (Abstract)
“The audio detection module 110a may further include a machine learning module 112b that implements a model that evaluates features in processed audio streams from the pre-processing module 112a and attempts to classify the audio features.” (Para 0023)
“the outputs of the noise cancelation modules 400a-400d may be input to a machine learning model 402 that classifies features in the outputs as corresponding to a particular vehicle. The machine learning model 112b may further output a confidence in the classification.” (Para 0041)
the corresponding likelihood value, and the identified one or more objects
(Banvait) – “The method 500 may include evaluating 516 whether the confidence score of step 508 exceeds a threshold. For example, where no classifications at step 508 have a confidence score above a threshold, the method 500 may include determining that the audio features that were the basis of the classification likely do not correspond to a vehicle. Otherwise, if the confidence score does exceed a threshold, then the method 500 may include adding 518 a potential obstacle to a set of obstacles identified by other means, such as using imaging devices 104. The potential obstacle may be defined as a potential obstacle located in the direction or range of angles determined at step 510.” (Para 0055)
“The method 500 may further include attempting to validate the classification performed at step 508 using one or more other sources of information. For example, the method 500 may include attempting to correlate the direction to the sound origin with a vehicle image located within an output of an imaging sensor 104 at a position corresponding to the direction to the sound origin. For example, any vehicle image located within an angular region including the direction may be identified at step 512. If a vehicle image is found in the image stream of the imaging sensor 104 within the angular region, then a confidence value may be increased.” (Para 0052)
“In either outcome of step 516, obstacles are detected using other sensing systems, such as the imaging devices 104, and obstacles detected using these sensing systems are added 520 to the obstacle set. Collision avoidance is performed 522 with respect to the obstacle set.” (Para 0057)
“The audio detection module 110a may further include a machine learning module 112b that implements a model that evaluates features in processed audio streams from the pre-processing module 112a and attempts to classify the audio features. The machine learning module 112b may output a confidence score indicating a likelihood that a classification is correct.” (Para 0023)
Banvait does not explicitly teach:
wherein the type of the sound is selected from a plurality of sound types including different vehicle types …determine, by the one or more processors, whether the corresponding likelihood value meets a threshold; and when the corresponding likelihood value is determined to meet the
Kim, in the same field of endeavor of vehicle control, teaches:
determine, by the one or more processors, whether the corresponding likelihood value meets a threshold; and when the corresponding likelihood value is determined to meet the
(Kim) – “The sound recognizer 130 may generate a determination result according to whether the confidence level is higher than or equal to a threshold (e.g., 0.7) and include the same in the sound classification result. That is, if the confidence level is higher than or equal to the threshold, the sound recognizer 130 may determine the type of sound of a class corresponding to the confidence level as the type of the present sound data.” (Para 0050)
“The sound recognizer 130 may classify the feature values of the acquired sound data using a classifier and determine whether the acquired sound data corresponds to a sound in which the user is interested.” (Para 0046)
“the results of sound detection may be information about the probabilities of presence of a vehicle (hereinafter, a “neighboring vehicle”) travelling around an object (the vehicle 10) (hereinafter, a “host vehicle”)” (Para 0061)
“Therefore, with a method for providing sound detection information, an apparatus for detecting sound around a vehicle, and a vehicle including the same according to an exemplary embodiment in the present disclosure configured as above, peripheral devices of the vehicle may be controlled to be set to modes appropriate for tunnel travel by accurately detecting entry into or exit from a tunnel based only on information about sound around the vehicle.” (Para 0105)
Therefore, it would be obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the system of Banvait with the method for providing sound detection information of Kim. One of ordinary skill in the art would have been motivated to make these modifications, with a reasonable expectation of success, because “there is a need for a technology to report information about a specific sound such as the identity of the sound generated around the vehicle and the direction of the sound, without undermining driving safety.” (Kim Para 0004)
Kim does not explicitly teach:
wherein the type of the sound is selected from a plurality of sound types including different vehicle types
Reiff, in the same field of endeavor of vehicle control, teaches:
wherein the type of the sound is selected from a plurality of sound types including different vehicle types
(Reiff) – “A controller for an autonomous vehicle receives audio signals from one or more microphones and identifies sounds. The controller further identifies an estimated location of the sound origin and the type of sound, i.e. whether the sound is a vehicle and/or the type of vehicle. The controller analyzes map data and attempts to identify a landmark within a tolerance from the estimated location. If a landmark is found corresponding to the estimated location and type of the sound origin, then the certainty is increased that the source of the sound is at that location and is that type of sound source. Collision avoidance is then performed with respect to the location of the sound origin and its type with the certainty as augmented using the map data. Collision avoidance may include automatically actuating brake, steering, and accelerator actuators in order to avoid the location of the sound origin.” (Abstract)
“The controller 102 may include or access a database 104 housed in the vehicle or otherwise accessible by the controller 102. The database 104 may include data sufficient to enable identification of an obstacle using map data. For example, sound data 106 may contain data describing sounds generated by one or more types of vehicles or other potential obstacles. For example, sound data 106 may include samples of the sounds made by one or more types of vehicles, animals (e.g. a dog barking), people conversing, and the like.” (Para 0020)
Therefore, it would be obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the system of Banvait with the Collision Avoidance Using Auditory Data Augmented With Map Data of Reiff. One of ordinary skill in the art would have been motivated to make these modifications, with a reasonable expectation of success, in order to “provide an improved approach for detecting obstacles.” (Reiff Para 0005)
Claim 12:
Rejected using the same rationale as Claim 2
Claim 13:
Rejected using the same rationale as Claim 5
Claim 14:
Rejected using the same rationale as Claim 3
Claim 15: Cancelled
Claim 23:
Banvait in combination with the references relied upon in Claim 1 teach those respective limitations. Banvait does not explicitly teach the following limitations. However, Reiff, in the same field of endeavor of vehicle control, teaches:
the identified one or more objects being located within a predetermined distance of a current location of the vehicle
(Reiff) – “The method 400 may include one or both of estimating 406 a distance to the origin of the sound and estimating 408 a direction to the origin of the sound.” (Para 0044)
“The method 400 may include retrieving 410 map data in a region including the estimated location of the sound origin as determined at steps 406 and 408. And evaluating 412 whether the map data includes a landmark corresponding to the location and candidate source of the sound origin. For example, a landmark closest to the location of the sound origin may be identified 412. For example, where the candidate sound source is determined at step 404 to be a vehicle and a parking garage is within a specified tolerance from the location determined at steps 406 and 408, then it may be determined that the parking garage is the landmark corresponding to the sound detected at step 402. As noted above, the tolerance may be a region or range of angles and distances corresponding to the uncertainty in determining the location, direction, and distance, respectively of the sound origin. In another scenario, an emergency vehicle station is within the tolerance from the sound origin and the candidate sound source is an emergency vehicle, then it may be determined at step 412 that the landmark corresponding to the sound detected at step 402 is the emergency vehicle station.” (Para 0045)
Examiner Note: The distance from the vehicle to the detected sound is determined before searching the map data for corresponding landmarks within a tolerance of the detected sound. This corresponds with “objects being located within a predetermined distance of a current location of the vehicle.”
Therefore, it would be obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the system of Banvait with the Collision Avoidance Using Auditory Data Augmented With Map Data of Reiff. One of ordinary skill in the art would have been motivated to make these modifications, with a reasonable expectation of success, in order to “provide an improved approach for detecting obstacles.” (Reiff Para 0005)
Claim 24:
Rejected based on the same rationale as Claim 23
Claim 25:
Banvait in combination with the references relied upon in Claim 1 teach those respective limitations. Banvait does not explicitly teach the following limitations. However, Reiff, in the same field of endeavor of vehicle control, teaches:
wherein the plurality of sound types further includes different infrastructure types.
(Reiff) – “A controller for an autonomous vehicle receives audio signals from one or more microphones and identifies sounds. The controller further identifies an estimated location of the sound origin and the type of sound, i.e. whether the sound is a vehicle and/or the type of vehicle. The controller analyzes map data and attempts to identify a landmark within a tolerance from the estimated location. If a landmark is found corresponding to the estimated location and type of the sound origin, then the certainty is increased that the source of the sound is at that location and is that type of sound source. Collision avoidance is then performed with respect to the location of the sound origin and its type with the certainty as augmented using the map data. Collision avoidance may include automatically actuating brake, steering, and accelerator actuators in order to avoid the location of the sound origin.” (Abstract)
“The controller 102 may include or access a database 104 housed in the vehicle or otherwise accessible by the controller 102. The database 104 may include data sufficient to enable identification of an obstacle using map data. For example, sound data 106 may contain data describing sounds generated by one or more types of vehicles or other potential obstacles. For example, sound data 106 may include samples of the sounds made by one or more types of vehicles, animals (e.g. a dog barking), people conversing, and the like.” (Para 0020)
“The database 104 may further include map data 108. The map data 108 may include maps in the region of the vehicle, such as the city, state, or country in which the vehicle is located. The maps may include data describing roads, landmarks, businesses, public buildings, etc. In particular, the map data 108 may include the locations of emergency vehicle stations (fire stations, hospitals with ambulance service, police stations, etc.).” (Para 0021)
“The method 400 may include retrieving 410 map data in a region including the estimated location of the sound origin as determined at steps 406 and 408. And evaluating 412 whether the map data includes a landmark corresponding to the location and candidate source of the sound origin. For example, a landmark closest to the location of the sound origin may be identified 412. For example, where the candidate sound source is determined at step 404 to be a vehicle and a parking garage is within a specified tolerance from the location determined at steps 406 and 408, then it may be determined that the parking garage is the landmark corresponding to the sound detected at step 402. As noted above, the tolerance may be a region or range of angles and distances corresponding to the uncertainty in determining the location, direction, and distance, respectively of the sound origin. In another scenario, an emergency vehicle station is within the tolerance from the sound origin and the candidate sound source is an emergency vehicle, then it may be determined at step 412 that the landmark corresponding to the sound detected at step 402 is the emergency vehicle station.” (Para 0045)
Therefore, it would be obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the system of Banvait with the Collision Avoidance Using Auditory Data Augmented With Map Data of Reiff. One of ordinary skill in the art would have been motivated to make these modifications, with a reasonable expectation of success, in order to “provide an improved approach for detecting obstacles.” (Reiff Para 0005)
Claim 26:
Banvait explicitly teaches:
One or more computer-readable storage media collectively storing machine- readable instructions, that when executed by one or more processors, performs a method of detecting and responding to sounds for a vehicle having an autonomous driving mode, the method comprising:
(Banvait) – “A controller for an autonomous vehicle receives audio signals from one or more microphones. The audio signals are input to a machine learning model that classifies the source of the audio features. For example, features may be classified as originating from a vehicle. A direction to a source of the audio features is determined based on relative delays of the audio features in signals from multiple microphones. Where audio features are classified with an above-threshold confidence as originating from a vehicle, collision avoidance is performed with respect to the direction to the source of the audio features.” (Abstract)
“These computer program instructions may also be stored in a non-transitory computer-readable medium that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable medium produce an article of manufacture including instruction means which implement the function/act specified in the flowchart and/or block diagram block or blocks.” (Para 0017)
receiving, by the one or more processors of the vehicle, an audible signal corresponding to a sound received at one or more microphones of the vehicle;
(Banvait) – “A controller for an autonomous vehicle receives audio signals from one or more microphones. The audio signals are input to a machine learning model that classifies the source of the audio features. For example, features may be classified as originating from a vehicle. A direction to a source of the audio features is determined based on relative delays of the audio features in signals from multiple microphones. Where audio features are classified with an above-threshold confidence as originating from a vehicle, collision avoidance is performed with respect to the direction to the source of the audio features.” (Abstract)
“The present invention is described below with reference to flowchart illustrations and/or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and/or block diagrams, and combinations of blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions or code. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions/acts specified in the flowchart and/or block diagram block or blocks.” (Para 0016)
“Referring to FIG. 1, a controller 102 may be housed within a vehicle.” (Para 0019)
“FIG. 2 is a block diagram illustrating an example computing device 200. Computing device 200 may be used to perform various procedures, such as those discussed herein. The controller 102 may have some or all of the attributes of the computing device 200.” (Para 0030)
“Computing device 200 includes one or more processor(s) 202” (Para 0031)
Examiner Note: It will be hereinafter understood that the processes taught in Banvait are implemented using at least a processor.
determining, by the one or more processors, a type of the sound and a corresponding likelihood value for the type of the sound, [wherein the type of the sound is selected from a plurality of sound types including different vehicle types];
(Banvait) – “A controller for an autonomous vehicle receives audio signals from one or more microphones. The audio signals are input to a machine learning model that classifies the source of the audio features. For example, features may be classified as originating from a vehicle. A direction to a source of the audio features is determined based on relative delays of the audio features in signals from multiple microphones. Where audio features are classified with an above-threshold confidence as originating from a vehicle, collision avoidance is performed with respect to the direction to the source of the audio features.” (Abstract)
“The audio detection module 110a may further include a machine learning module 112b that implements a model that evaluates features in processed audio streams from the pre-processing module 112a and attempts to classify the audio features. The machine learning module 112b may output a confidence score indicating a likelihood that a classification is correct.” (Para 0023)
“the outputs of the noise cancelation modules 400a-400d may be input to a machine learning model 402 that classifies features in the outputs as corresponding to a particular vehicle. The machine learning model 112b may further output a confidence in the classification.” (Para 0041)
Examiner Note: Classification corresponds to type. Confidence score is a measure of likelihood.
[determining, by the one or more processors, whether the corresponding likelihood value meets a threshold; and when the corresponding likelihood value is determined to meet the threshold] using, by the one or more processors, map information and sensor data to identify one or more objects associated with a source of the type of the sound by searching the map information and sensor data for the one or more objects of the determined type of the sound; and
(Banvait) – “The map correlation module 112d evaluates map data to determine whether a parking stall, driveway, or other parking area is located within the angular tolerance from the direction to the source of a sound corresponding to a vehicle with its engine running, particularly a parked vehicle. If so, then the confidence that the sound corresponds to a parked vehicle with its engine running is increased.” (Para 0025)
“The method 500 may further include attempting to validate the classification performed at step 508 using one or more other sources of information. For example, the method 500 may include attempting to correlate the direction to the sound origin with a vehicle image located within an output of an imaging sensor 104 at a position corresponding to the direction to the sound origin. For example, any vehicle image located within an angular region including the direction may be identified at step 512. If a vehicle image is found in the image stream of the imaging sensor 104 within the angular region, then a confidence value may be increased.” (Para 0052)
“The direction to the source of the audio features may be correlated with vehicle images and/or map data to increase a confidence score that the source of the audio features is a parked vehicle with its engine running.” (Abstract)
“The vehicle 300 may be close enough to detect sound generated by an occluded vehicle 304 or a parked vehicle 304. Although the methods disclosed herein are particularly useful where there is an occluding object, the identification of obstacles as described herein may be performed where image data is available and may, for example, confirm the location of an obstacle that is also visible to imaging devices 104. Likewise, the existence of a parked vehicle with its engine running may be confirmed using imaging devices 104 as described in greater detail below.” (Para 0040)
“The direction as estimated by the microphone array processing module 404 and the classification and confidence score as generated by the machine learning model 402 may then be provided as an output 406 from the machine learning module 112b. For example, the obstacle identification module 110b may add a vehicle having the identified class located at the estimated direction to a set of potential obstacles, the set of potential obstacles including any obstacles identified by other means, such as using the imaging devices 104.” (Para 0048)
Examiner Note: Bracketed text not explicitly taught by the primary reference, but is taught by non-primary reference later in the rejection.
controlling, by the one or more processors, the vehicle in the autonomous driving mode to:
(Banvait) – “A controller for an autonomous vehicle receives audio signals from one or more microphones. The audio signals are input to a machine learning model that classifies the source of the audio features. For example, features may be classified as originating from a vehicle. A direction to a source of the audio features is determined based on relative delays of the audio features in signals from multiple microphones. Where audio features are classified with an above-threshold confidence as originating from a vehicle, collision avoidance is performed with respect to the direction to the source of the audio features.” (Abstract)
respond to the sound based on the type of the sound,
(Banvait) – “The method 500 may include evaluating 516 whether the confidence score of step 508 exceeds a threshold. For example, where no classifications at step 508 have a confidence score above a threshold, the method 500 may include determining that the audio features that were the basis of the classification likely do not correspond to a vehicle. Otherwise, if the confidence score does exceed a threshold, then the method 500 may include adding 518 a potential obstacle to a set of obstacles identified by other means, such as using imaging devices 104. The potential obstacle may be defined as a potential obstacle located in the direction or range of angles determined at step 510.” (Para 0055)
“A controller for an autonomous vehicle receives audio signals from one or more microphones. The audio signals are input to a machine learning model that classifies the source of the audio features. For example, features may be classified as originating from a vehicle. A direction to a source of the audio features is determined based on relative delays of the audio features in signals from multiple microphones. Where audio features are classified with an above-threshold confidence as originating from a vehicle, collision avoidance is performed with respect to the direction to the source of the audio features.” (Abstract)
the corresponding likelihood value, and the identified one or more objects
(Banvait) – “The method 500 may include evaluating 516 whether the confidence score of step 508 exceeds a threshold. For example, where no classifications at step 508 have a confidence score above a threshold, the method 500 may include determining that the audio features that were the basis of the classification likely do not correspond to a vehicle. Otherwise, if the confidence score does exceed a threshold, then the method 500 may include adding 518 a potential obstacle to a set of obstacles identified by other means, such as using imaging devices 104. The potential obstacle may be defined as a potential obstacle located in the direction or range of angles determined at step 510.” (Para 0055)
“The method 500 may further include attempting to validate the classification performed at step 508 using one or more other sources of information. For example, the method 500 may include attempting to correlate the direction to the sound origin with a vehicle image located within an output of an imaging sensor 104 at a position corresponding to the direction to the sound origin. For example, any vehicle image located within an angular region including the direction may be identified at step 512. If a vehicle image is found in the image stream of the imaging sensor 104 within the angular region, then a confidence value may be increased.” (Para 0052)
“In either outcome of step 516, obstacles are detected using other sensing systems, such as the imaging devices 104, and obstacles detected using these sensing systems are added 520 to the obstacle set. Collision avoidance is performed 522 with respect to the obstacle set.” (Para 0057)
“The audio detection module 110a may further include a machine learning module 112b that implements a model that evaluates features in processed audio streams from the pre-processing module 112a and attempts to classify the audio features. The machine learning module 112b may output a confidence score indicating a likelihood that a classification is correct.” (Para 0023)
Banvait does not explicitly teach:
wherein the type of the sound is selected from a plurality of sound types including different vehicle types …determining, by the one or more processors, whether the corresponding likelihood value meets a threshold; and when the corresponding likelihood value is determined to meet the threshold
Kim, in the same field of endeavor of vehicle control, teaches:
determining, by the one or more processors, whether the corresponding likelihood value meets a threshold; and when the corresponding likelihood value is determined to meet the
(Kim) – “The sound recognizer 130 may generate a determination result according to whether the confidence level is higher than or equal to a threshold (e.g., 0.7) and include the same in the sound classification result. That is, if the confidence level is higher than or equal to the threshold, the sound recognizer 130 may determine the type of sound of a class corresponding to the confidence level as the type of the present sound data.” (Para 0050)
“The sound recognizer 130 may classify the feature values of the acquired sound data using a classifier and determine whether the acquired sound data corresponds to a sound in which the user is interested.” (Para 0046)
“the results of sound detection may be information about the probabilities of presence of a vehicle (hereinafter, a “neighboring vehicle”) travelling around an object (the vehicle 10) (hereinafter, a “host vehicle”)” (Para 0061)
“Therefore, with a method for providing sound detection information, an apparatus for detecting sound around a vehicle, and a vehicle including the same according to an exemplary embodiment in the present disclosure configured as above, peripheral devices of the vehicle may be controlled to be set to modes appropriate for tunnel travel by accurately detecting entry into or exit from a tunnel based only on information about sound around the vehicle.” (Para 0105)
Therefore, it would be obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the system of Banvait with the method for providing sound detection information of Kim. One of ordinary skill in the art would have been motivated to make these modifications, with a reasonable expectation of success, because “there is a need for a technology to report information about a specific sound such as the identity of the sound generated around the vehicle and the direction of the sound, without undermining driving safety.” (Kim Para 0004)
Kim does not explicitly teach:
wherein the type of the sound is selected from a plurality of sound types including different vehicle types
Reiff, in the same field of endeavor of vehicle control, teaches:
wherein the type of the sound is selected from a plurality of sound types including different vehicle types
(Reiff) – “A controller for an autonomous vehicle receives audio signals from one or more microphones and identifies sounds. The controller further identifies an estimated location of the sound origin and the type of sound, i.e. whether the sound is a vehicle and/or the type of vehicle. The controller analyzes map data and attempts to identify a landmark within a tolerance from the estimated location. If a landmark is found corresponding to the estimated location and type of the sound origin, then the certainty is increased that the source of the sound is at that location and is that type of sound source. Collision avoidance is then performed with respect to the location of the sound origin and its type with the certainty as augmented using the map data. Collision avoidance may include automatically actuating brake, steering, and accelerator actuators in order to avoid the location of the sound origin.” (Abstract)
“The controller 102 may include or access a database 104 housed in the vehicle or otherwise accessible by the controller 102. The database 104 may include data sufficient to enable identification of an obstacle using map data. For example, sound data 106 may contain data describing sounds generated by one or more types of vehicles or other potential obstacles. For example, sound data 106 may include samples of the sounds made by one or more types of vehicles, animals (e.g. a dog barking), people conversing, and the like.” (Para 0020)
Therefore, it would be obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the system of Banvait with the Collision Avoidance Using Auditory Data Augmented With Map Data of Reiff. One of ordinary skill in the art would have been motivated to make these modifications, with a reasonable expectation of success, in order to “provide an improved approach for detecting obstacles.” (Reiff Para 0005)
Claim(s) 6, 9, 16, 19, 21 is/are rejected under 35 U.S.C. 103 as being unpatentable over Banvait (US20170248955) in view of Kim (US20170305427) further in view of Reiff (US20170096138) further in view of Myers (US20190277986).
Claim 6:
Banvait in combination with the references relied upon in Claim 1 teach those respective limitations. Banvait does not explicitly teach the following limitations. However, Myers, in the same field of endeavor of vehicle control, teaches:
wherein the type of the sound is a train whistle.
(Myers) – “Other vehicular traffic, and particularly emergency vehicles and motorcycles 212, pose some of the greatest safety threats to vehicles 202 on the road. For this reason, almost all vehicles 202 are equipped with mechanisms capable of producing distinct noises to warn other vehicles 202 of potential danger. For example, sirens on police cars 206, fire engines 208, ambulances, and other emergency vehicles readily identify such vehicles and warn other vehicles of potential danger. Likewise, the loud engine sounds produced by a motorcycle 212 are discernable almost immediately, while the bells of a railroad crossing barrier 210 are widely recognized as announcing an impending train.” (Para 0022)
“Embodiments of the present invention may utilize onboard or ancillary ultrasonic sensors to detect and identify potentially dangerous obstacles and situations, including the impending train 510 and the motorcycle 508.” (Para 0036)
“information from the ultrasonic sensors 204 of the first vehicle 506 may override information received from other data sources. For example, sensor 204 information from the first vehicle 506 indicating that a train 510 is approaching may override information from other GPS sources indicating that the railroad barrier 512 is up.” (Para 0039)
Therefore, it would be obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the system of Banvait with the method for processing audible sounds of Myers. One of ordinary skill in the art would have been motivated to make these modifications, with a reasonable expectation of success, in order “to detect objects or obstacles corresponding to such sounds that may be obstructed or not directly visible” (Myers Para 0005)
Claim 9:
Banvait in combination with the references relied upon in Claim 1 teach those respective limitations. Banvait further teaches:
wherein the controlling further comprises automatically slowing down the vehicle in response to identifying the type of the sound [as a train whistle].
(Banvait) – “The collision prediction module 110c predicts which obstacle images are likely to collide with the vehicle based on its current trajectory or current intended path. The collision prediction module 110c may evaluate the likelihood of collision with objects identified by the obstacle identification module 110b as well as obstacles detected using the audio detection module 110a…The decision module 110d may make a decision to stop, accelerate, turn, etc. in order to avoid obstacles. The manner in which the collision prediction module 110c predicts potential collisions and the manner in which the decision module 110d takes action to avoid potential collisions may be according to any method or system known in the art of autonomous vehicles.” (Para 0027)
“The decision module 110d may control the trajectory of the vehicle by actuating one or more actuators 114 controlling the direction and speed of the vehicle. For example, the actuators 114 may include a steering actuator 116a, an accelerator actuator 116b, and a brake actuator 116c. The configuration of the actuators 116a-116c may be according to any implementation of such actuators known in the art of autonomous vehicles.” (Para 0028)
“A controller for an autonomous vehicle receives audio signals from one or more microphones. The audio signals are input to a machine learning model that classifies the source of the audio features. For example, features may be classified as originating from a vehicle. A direction to a source of the audio features is determined based on relative delays of the audio features in signals from multiple microphones. Where audio features are classified with an above-threshold confidence as originating from a vehicle, collision avoidance is performed with respect to the direction to the source of the audio features.” (Abstract)
Examiner Note: Bracketed text not explicitly taught by the primary reference, but it is taught by non-primary reference later in the rejection.
Banvait does not explicitly teach:
as train whistle.
Myers, in the same field of endeavor of vehicle control, teaches:
as train whistle.
(Myers) – “Other vehicular traffic, and particularly emergency vehicles and motorcycles 212, pose some of the greatest safety threats to vehicles 202 on the road. For this reason, almost all vehicles 202 are equipped with mechanisms capable of producing distinct noises to warn other vehicles 202 of potential danger. For example, sirens on police cars 206, fire engines 208, ambulances, and other emergency vehicles readily identify such vehicles and warn other vehicles of potential danger. Likewise, the loud engine sounds produced by a motorcycle 212 are discernable almost immediately, while the bells of a railroad crossing barrier 210 are widely recognized as announcing an impending train.” (Para 0022)
“Embodiments of the present invention may utilize onboard or ancillary ultrasonic sensors to detect and identify potentially dangerous obstacles and situations, including the impending train 510 and the motorcycle 508.” (Para 0036)
“information from the ultrasonic sensors 204 of the first vehicle 506 may override information received from other data sources. For example, sensor 204 information from the first vehicle 506 indicating that a train 510 is approaching may override information from other GPS sources indicating that the railroad barrier 512 is up.” (Para 0039)
Therefore, it would be obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the system of Banvait with the method for processing audible sounds of Myers. One of ordinary skill in the art would have been motivated to make these modifications, with a reasonable expectation of success, in order “to detect objects or obstacles corresponding to such sounds that may be obstructed or not directly visible” (Myers Para 0005)
Claim 16:
Rejected using the same rationale as Claim 6
Claim 19:
Rejected using the same rationale as Claim 9
Claim 21:
Banvait in combination with the references relied upon in Claim 1 teach those respective limitations. Banvait does not explicitly teach the following limitations. However, Myers, in the same field of endeavor of vehicle control, teaches:
wherein the source of the sound is a train or railroad warning bell or whistle, and the one or more objects include at least one of a flashing light, a gate, a train station or a railroad crossing.
(Myers) – “Other vehicular traffic, and particularly emergency vehicles and motorcycles 212, pose some of the greatest safety threats to vehicles 202 on the road. For this reason, almost all vehicles 202 are equipped with mechanisms capable of producing distinct noises to warn other vehicles 202 of potential danger. For example, sirens on police cars 206, fire engines 208, ambulances, and other emergency vehicles readily identify such vehicles and warn other vehicles of potential danger. Likewise, the loud engine sounds produced by a motorcycle 212 are discernable almost immediately, while the bells of a railroad crossing barrier 210 are widely recognized as announcing an impending train.” (Para 0022)
“Embodiments of the present invention may utilize onboard or ancillary ultrasonic sensors to detect and identify potentially dangerous obstacles and situations, including the impending train 510 and the motorcycle 508.” (Para 0036)
“information from the ultrasonic sensors 204 of the first vehicle 506 may override information received from other data sources. For example, sensor 204 information from the first vehicle 506 indicating that a train 510 is approaching may override information from other GPS sources indicating that the railroad barrier 512 is up.” (Para 0039)
Therefore, it would be obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the system of Banvait with the method for processing audible sounds of Myers. One of ordinary skill in the art would have been motivated to make these modifications, with a reasonable expectation of success, in order “to detect objects or obstacles corresponding to such sounds that may be obstructed or not directly visible” (Myers Para 0005)
Claim 22: Canceled
Claim(s) 7, 10, 17, 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Banvait (US20170248955) in view of Kim (US20170305427) further in view of Reiff (US20170096138) further in view of Jones (US20180074162).
Claim 7:
Banvait in combination with the references relied upon in Claim 1 teach those respective limitations. Banvait does not fully, explicitly teach the following limitations. However, Jones, in the same field of endeavor of sound detection, teaches:
wherein the type of the sound is a reverse beeping sound.
(Jones) – “a microphone (out of the array of microphones) can detect a sound of a truck backing up toward the loading dock. The microphone can detect a sound of vehicle motion alarm (also known as backup alarm, which emits beeps or chirps as a truck backs up) generated by the truck.” (Para 0029)
Therefore, it would be obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the system of Banvait with the systems for identifying actions based on detected sounds of Jones. One of ordinary skill in the art would have been motivated to make these modifications, with a reasonable expectation of success, in order to “determine the action being performed causing the sounds” (Jones Abstract)
Claim 10:
Banvait in combination with the references relied upon in Claim 1 teach those respective limitations. Banvait further teaches:
wherein the controlling further comprises automatically yielding to all larger vehicles in response to identifying the type of the sound [as a reverse beeping sound.]
(Banvait) – “The collision prediction module 110c predicts which obstacle images are likely to collide with the vehicle based on its current trajectory or current intended path. The collision prediction module 110c may evaluate the likelihood of collision with objects identified by the obstacle identification module 110b as well as obstacles detected using the audio detection module 110a…The decision module 110d may make a decision to stop, accelerate, turn, etc. in order to avoid obstacles. The manner in which the collision prediction module 110c predicts potential collisions and the manner in which the decision module 110d takes action to avoid potential collisions may be according to any method or system known in the art of autonomous vehicles.” (Para 0027)
“The decision module 110d may control the trajectory of the vehicle by actuating one or more actuators 114 controlling the direction and speed of the vehicle. For example, the actuators 114 may include a steering actuator 116a, an accelerator actuator 116b, and a brake actuator 116c. The configuration of the actuators 116a-116c may be according to any implementation of such actuators known in the art of autonomous vehicles.” (Para 0028)
“A controller for an autonomous vehicle receives audio signals from one or more microphones. The audio signals are input to a machine learning model that classifies the source of the audio features. For example, features may be classified as originating from a vehicle. A direction to a source of the audio features is determined based on relative delays of the audio features in signals from multiple microphones. Where audio features are classified with an above-threshold confidence as originating from a vehicle, collision avoidance is performed with respect to the direction to the source of the audio features.” (Abstract)
Examiner Note: Bracketed text not explicitly taught by the primary reference, but it is taught by non-primary reference later in the rejection. Banvait teaches avoiding collisions with all vehicles, including larger vehicles.
Banvait does not explicitly teach:
as a reverse beeping sound.
Jones, in the same field of endeavor of vehicle control, teaches:
as a reverse beeping sound.
(Jones) – “a microphone (out of the array of microphones) can detect a sound of a truck backing up toward the loading dock. The microphone can detect a sound of vehicle motion alarm (also known as backup alarm, which emits beeps or chirps as a truck backs up) generated by the truck.” (Para 0029)
Therefore, it would be obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the system of Banvait with the systems for identifying actions based on detected sounds of Jones. One of ordinary skill in the art would have been motivated to make these modifications, with a reasonable expectation of success, in order to “determine the action being performed causing the sounds” (Jones Abstract)
Claim 17:
Rejected using the same rationale as Claim 7
Claim 20:
Rejected using the same rationale as Claim 10
Claim(s) 8 is/are rejected under 35 U.S.C. 103 as being unpatentable over Banvait (US20170248955) in view of Kim (US20170305427) further in view of Reiff (US20170096138) further in view of Cullinane (US20140032032).
Claim 8:
Banvait in combination with the references relied upon in Claim 1 teach those respective limitations. Banvait does not explicitly teach the following limitations. However, Cullinane, in the same field of endeavor of vehicle control, teaches:
wherein the type of the sound is a crosswalk chirp.
(Cullinane) – “The computing device may be configured to receive, from the one or more sensors, audio information relating to an audible crosswalk signal for an intersection that the vehicle is approaching. The computing device may also be configured to determine a likelihood associated with a presence of a pedestrian in a crosswalk of the intersection, based on the audio information.” (Para 0004)
“the computing device 111 may be configured to receive from the microphone 156 an audio signal associated with a sound emitted by an audible crosswalk signal at an intersection that the automobile 100 may be approaching.” (Para 0048)
“the audible crosswalk signal may emit a long "cuckoo" sound when a north-south walking direction is allowed and may emit a short "chirp" sound when an east-west walking direction of travel is allowed. The computing device may be configured to determine a type of sound (e.g., a cuckoo or chirp) by analyzing the audio signal associated with the sound.” (Para 0076)
Therefore, it would be obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention, to have modified the system of Banvait with the systems for control of vehicles based on auditory signals are described of Cullinane. One of ordinary skill in the art would have been motivated to make these modifications, with a reasonable expectation of success, for the purpose of “determining a likelihood associated with a presence of a pedestrian in a crosswalk” (Cullinane Para 0003) in order “to safely maintain a distance with other objects (including pedestrians) and select a trajectory that is considered safest.” (Cullinane Para 0084)
Claim 18: Canceled
Response to Arguments
Applicant's arguments with respect to the 35 U.S.C. 103 rejection mailed 09/23/2025 have been fully considered but they are not persuasive. Rationale has been updated to reflect amendment
Specifically, all claims are now rejected further in view of Reiff as necessitated by amendment. Examiner maintains that Reiff resolves any alleged deficiencies of Banvait and Kim as evidenced in the updated rejection rationale.
Therefore, Claims all remaining claims remain rejected under 35 USC 103.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DAVID RUBEN PEDERSEN whose telephone number is (571)272-9696. The examiner can normally be reached M-Th: 07:00 -16:00 Eastern.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Ramon Mercado can be reached on 571-272-5109. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/DAVID RUBEN PEDERSEN/Examiner, Art Unit 3658
/Ramon A. Mercado/Supervisory Patent Examiner, Art Unit 3658