DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Claims 3, 4, and 5 are objected to because they state “determining presence of an emergency vehicle.” All of the stated claims are dependent on claim 1 which already states “an emergency vehicle” in line 14. Therefore claims 3, 4, and 5 should be rewritten so that “determining the presence of an emergency vehicle” reads as “determining the presence of the emergency vehicle.”
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
Claim 1 is rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 1 recites the limitation "the different modalities" in line 11. There is insufficient antecedent basis for this limitation in the claim.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-20 are rejected under USC 103 for being obvious over US20200241552 (Hereinafter referred to as Leenayongwut) in view of “Single-modal Incremental Terrain Clustering from Self-Supervised Audio-Visual Feature Learning” (hereinafter referred to as Ishikawa).
Regarding claim one, Leenayongwut teaches a system comprising one or more processors; and one or more non-transitory computer-readable media storing instructions executable by the one or more processors [see paragraph 0091 which discloses a processor and memory that execute the instructions for the system],
PNG
media_image1.png
168
1217
media_image1.png
Greyscale
wherein the instructions, when executed, cause the system to perform operations comprising: obtaining, visual data acquired by a first image sensor and associated with a traffic scene [see paragraph 0054 where image sensors are used to create a visual scene, which in this case would be a traffic scene as the scene is take from an autonomous vehicle] ;
PNG
media_image2.png
208
1206
media_image2.png
Greyscale
obtaining, audio data acquired by a first audio sensor and associated with the traffic scene, [see above 0054 where ambient noise and sound sources are detected using a microphone array ] wherein the first audio sensor and the first image sensor are mutually independent [see paragraph 0131 where sound and image information is obtained separately];
PNG
media_image3.png
125
1195
media_image3.png
Greyscale
determining, based at least in part on the visual data and the audio data, embeddings for the traffic scene that represent a joint representation space for the different modalities [see paragraph 0136 where the sound context is stored as a vector, and correlated with activations of the visual neural network. Embeddings are essentially context information stored in a numerical form (i.e. vectors) Therefore embeddings of the traffic data are determined. See also that the information from the sound data as well as the image data is then processed in the VBSL (vision based sound localizer) module which is a unified CNN indicating a joint representation space of the combined traffic data];
PNG
media_image4.png
231
889
media_image4.png
Greyscale
and determining, by a first machine learning model, and based at least in part on the embeddings, presence of an emergency vehicle in the traffic scene [See paragraph 0142 below where an emergency vehicle is detected using the information output by the VBSL module, indicating that the previously stated embeddings are used in the determination].
PNG
media_image5.png
127
1285
media_image5.png
Greyscale
Although Leenayongwut does disclose embeddings to process the audio information related to the traffic scene, It does not explicitly state that there are embeddings for the other sensory inputs as well. Additionally, though Leenayongwut discloses a unified CNN, It does not explicitly state this space is a multidimensional framework where different datatypes are present in the same structure.
Ishikawa does teach embedding both visual and auditory data (which make up the data required to determine a traffic scene). [See Figure 3 of Ishikawa where the audio and visual components are embedded as latent vectors which provide the neural network context grouping the context into categories in order to identify distinct information about the surroundings] Ishikawa also teaches a joint representation space [as seen in Fig 3 since heterogenous data types are projected into the same structure, where the distance between the vectors represent the semantic similarity].
PNG
media_image6.png
336
734
media_image6.png
Greyscale
Therefore it would have been obvious to one with ordinary skill in the art before the effective filing date to have combined the system of Leenayongwut where autonomous vehicle can be controlled using multiple sensory input in the case of the detection of an emergency vehicle, with the embedding of the information into a neural network as disclosed in Ishikawa. The two inventions are in the same field of endeavor of detecting information in the environment for autonomous vehicles. The motivation to combine would be so that the model can work accurately even one of the sensors is not working as expected as discussed in the conclusion of Ishikawa where “the result demonstrated that even if one of the modality is missing due to the extreme conditions, our model can receive partial input and predict the correct terrain type.”
Regarding claim 2, Leenayongwut and Ishikawa teach the system of claim 1, and Leeenayongwut teaches wherein the instructions, when executed, cause the system to perform operations further comprising: obtaining, range data acquired by a first range sensor and associated with the traffic scene, wherein the range sensor is a radar sensor or a lidar sensor [see paragraph 0106 of Leenayongwut which discloses using LIDAR to construct a representation of the environment. Range data is being read as distance measurements by the examiner, therefore the creation a reconstructed environment from point clouds indicates that range data was obtained];
PNG
media_image7.png
111
1247
media_image7.png
Greyscale
Leenayongwut does teach including the range data as part of the inputs used in the perception model [see 0106 above], But it does not explicitly state determining of the embeddings.
Ishiwaka teaches determining, the embeddings [See the clusters formed by the audio and visual data of figure 3 of Ishikawa above to detect a scenario in the environment]
It would have been obvious to one with ordinary skill in the art before the effective filing date to have include collected range data as a vector in the model to more accurately determine embeddings of traffic data, an account for physical distances sensed in the environment. See also Ishikawa equation 4 where the model is built to be able to include multiple modalities, indicating that the Lidar data mentioned above could be used to determine the embeddings.
PNG
media_image8.png
459
709
media_image8.png
Greyscale
Regarding claim 3, Leenayongwut and Ishikawa teach the system of claim 1 wherein the instructions, when executed, cause the system to perform operations further comprising: labeling, based at least in part on at least one of the embeddings or determining presence of an emergency vehicle, the visual data thereby providing labeled visual data [See paragraph 0103 of Leenayongwut where visual information such as pedestrian/ signs/ cars are classified based on the visual data embeddings];
PNG
media_image9.png
209
1075
media_image9.png
Greyscale
and training, based at least in part on the labeled visual data, a second machine learning model configured to accept at least visual data as input to detect emergency vehicles in traffic scenes [see paragraph 0136 of Leenayongwut above where there is a separate visual neural network for processing the visual data. See also Table two of page 9405 of Ishikawa where both image and audio data are used for training the model].
PNG
media_image10.png
685
740
media_image10.png
Greyscale
Regarding claim 4, Leenayongwut and Ishikawa teach the system of claim 1 wherein the instructions, when executed, cause the system to perform operations further comprising: labeling, based at least in part on at least one of the embeddings or determining presence of an emergency vehicle, the audio data thereby providing labeled audio data [See paragraph 0135 of Leenayongwut where the sounds are classified into various categories such as footsteps, engines, and traffic based on the audio embeddings];
PNG
media_image11.png
190
1020
media_image11.png
Greyscale
and training, based at least in part on the labeled audio data, a third machine learning model configured to accept at least audio data as input to detect emergency vehicles in traffic scenes. [see paragraph 0136 of Leenayongwut above where there is a separate sound neural network for processing sound data and Table two of page 9405 of Ishikawa above where both image and audio data are used for training the model] .
Regarding claim 5, Leenayongwut and Ishikawa teach the system of claim 1 wherein the instructions, when executed, cause the system to perform operations further comprising: controlling, based at least in part on determining presence of an emergency vehicle in the traffic scene, operation of an autonomous vehicle [See paragraph 0143 of Leenayongwut where the planning modules can give directions for the autonomous vehicle to perform an action when an emergency vehicle is detected].
PNG
media_image12.png
204
1071
media_image12.png
Greyscale
Regarding Claim 6, Leenayongwut and Ishikawa teach a method comprising: obtaining, first sensor data acquired by a first sensor and associated with a traffic scene; obtaining, second sensor data acquired by a second sensor and associated with the traffic scene, wherein the first sensor and the second sensor are mutually independent and of different modalities [see paragraph 0054 of Leenayongwut above where visual and auditory images sensor are used to obtain data regarding a traffic scene]; determining, based at least in part on the first sensor data and the second sensor data, embeddings for the traffic scene that represent a joint representation space for the different modalities; [see paragraph 0136 of Leenayongwut and Fig 3 of Ishikawa above where the data is vectorized to create embeddings in shared space between multiple data types, indicating a joint representation space] and labeling, based at least in part on the embeddings, at least one of the first sensor data or the second sensor data thereby providing labeled sensor data [See paragraph 0135 and 0103 of Leenayongwut where the sounds and objects in an image are classified];.
Regarding Claim 7, Leenayongwut and Ishikawa teach the method of claim 6 further comprising training, based at least in part on the labeled sensor data, a second machine learning model configured to detect objects of a specific type in traffic scenes, [see paragraph 0136 of Leenayongwut above where there is a separate visual neural network for processing the visual data. See also Table two of page 9405 of Ishikawa where both image and audio data are used for training the model]. wherein an accepted modality of the second machine learning model is at least a modality of the labeled sensor data, and the objects of the specific type are emergency vehicles [see paragraph 0142 of Leenayongwut where both the visual information, and the sound information (a siren) are labeled to detect an emergency vehicle].
PNG
media_image13.png
122
1205
media_image13.png
Greyscale
Regarding Claim 8, Leenayongwut teaches the method of claim 6, wherein the second sensor is an audio sensor [see paragraph 0054 above which discloses microphones].
Regarding Claim 9, Leenayongwut and Ishikawa teach the method of claim 6, wherein the first sensor is one of an optical sensor or a radar sensor [see paragraph 0054 above where an image sensor obtains optical data of the environment].
Regarding claim 10, Leenayongwut and Ishikawa teach the method of claim 6, wherein the first sensor is an image sensor, the second sensor is an audio sensor and the method further comprises: obtaining, range data acquired by a third sensor and associated with the traffic scene, wherein the third sensor is a radar sensor or lidar sensor; [See paragraph 106 above where LIDAR data is additionally obtained] and determining, and based at least in part on the range data, the embeddings [see Ishikawa equation 4 where the model is built to include multiple modalities, indicating that the Lidar data could be used to determine the embeddings].
Regarding claim 12, Leenayongwut and Ishikawa teach the method of claim 6, further comprising: obtaining, additional data acquired by an additional sensor and associated with the traffic scene, wherein the additional data is of the same modality as one of the first sensor data or the second sensor data and wherein the additional sensor is independent from the first sensor and the second sensor [See paragraph 0108 of Ishikawa where the vision data includes information from multiple cameras i.e. vision sensors, or additional camera such as a stereopsis camera where the depth of the scene can be better perceived);
PNG
media_image14.png
188
1054
media_image14.png
Greyscale
and determining, based at least in part on the additional data, the embeddings [See fig 3 of Ishikawa above where the data can be vectorized to determine embeddings based on this additional data]
Regarding claim 11, Leenayongwut and Ishikawa teach the method of claim 6, further comprising: obtaining previously labeled sensor data, wherein the previously labeled data is audio data and/or visual data labeled by a second or third machine learning model [see paragraph 0136 above where there is a separate sound neural network for processing sound data and a separate visual neural network for processing the visual data] labeling, based at least in part on the embeddings, the previously labeled data, thereby providing relabeled sensor data; comparing labels of the previously labeled data to labels of the relabeled sensor data [see page 9404 part F. of Ishikawa where the model used the method of a previous model. The method incorporated determining output labels from the clustering results in a self- supervised manner indicating that the embedded vectors were used when outputting the label, and calculating metrics based on the output label and the previously known ground truth label];
PNG
media_image15.png
563
745
media_image15.png
Greyscale
and training, based at least in part on the comparison, one or more of the second or third machine learning models [See above where the training is done in a self-supervised manner, it would have been obvious to one with ordinary skill in the art to perform this training on both the visual and audio neural network. See also Leenayongwut 0133 where machine learning is used to classify sounds, and 0103 where a neural network is used to classify image data].
PNG
media_image16.png
102
1097
media_image16.png
Greyscale
Regarding claim 13, Leenayongwut and Ishikawa teach the method of claim 6, further comprising: determining, by a first machine learning model, and based at least in part on the embeddings, presence of an object of a specific type in the traffic scene [see paragraph 0142 of Leenayongwut above where specific objects such as an emergency vehicle can be determined by the VBSL model].
Regarding claim 14, Leenayongwut and Ishikawa teach the method of claim 13 wherein the first machine learning model is configured to detect presence of one or more emergency vehicles, accidents, ice cream trucks, electric vehicles, high-performance vehicles, motorcycles, heavy-duty trucks, military vehicles or rail crossings [see paragraph 0142 of Leenayongwut above where emergency vehicles can be detected].
Regarding claim 15, Leenayongwut and Ishikawa teach the method of claim 13 further comprising: labeling, based at least in part on determining presence of the object of the specific type, the at least one of the first sensor data or the second sensor data [see paragraph 0103 of Leenayongwut above where when an object is detected, bounding boxes and labels can be provided].
Regarding claim 16, Leenayongwut and Ishikawa teach the method of claim 13, further comprising: training, based at least in part on ground truth data of different modalities associated with the same traffic scene, the first machine learning model [See page 9404 part F. of Ishikawa above where the ground truth information is used in order to label the data of relevant scenarios in the environment]; and training, based at least in part on ground truth data of a single modality associated with a traffic scene, the first machine learning model [See the conclusion of Ishikawa where the clustering can be done incrementally as additional data is input into the model, therefore it utilized ground truth data. Incremental clustering is a form of training, This process can be done on the VBSL model of Leenayongwut].
Regarding claim 17, Leenayongwut and Ishikawa teach the method of claim 13, further comprising: controlling, based at least in part on determining presence of objects of the specific type in the traffic scene, operation of an autonomous vehicle [See paragraph 0143 of Leenayongwut above where the presence of objects can cause the autonomous vehicle to move appropriately].
Claim 18 is similarly analyzed to claim 6
Claim 19 is similarly analyzed to claim 7
Claim 20 is similarly analyzed to claim 13
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Contact Information
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ANUSHA KASHYAPA whose telephone number is (571)272-8766. The examiner can normally be reached Monday-Friday 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chan Park can be reached at (571) 272-7409. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ANUSHA KASHYAPA/Examiner, Art Unit 2669
/CHAN S PARK/Supervisory Patent Examiner, Art Unit 2669
Prosecution Insights