DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 12/29/2025 has been entered.
Response to Arguments
Applicant’s arguments, see Remarks pages 9-12, filed 12/29/2025, with respect to the rejections of claims 1-28 under 35 U.S.C. 101 have been fully considered and are persuasive. The rejections of claims 1-28 have been withdrawn.
Applicant’s arguments, see Remarks pages 13-16, filed 12/29/2025, with respect to the prior art rejection of amended claim(s) 1 and 15 under 35 U.S.C. 103 have been fully considered and are moot in view of the new grounds of rejection (detailed in the rejections below) necessitated by Applicant’s amendment to the claim(s).
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 3-5, 8-9, 11-15, 17-19, 22-23, and 25-28 is/are rejected under 35 U.S.C. 103 as being unpatentable over Wolf et al. (Camera-First Form Filling: Reducing the Friction in Climate Hazard Reporting) hereinafter referenced as Wolf, in view of Gong et al. (Listen, Think, and Understand) hereinafter referenced as Gong.
Regarding claim 1, Wolf discloses: A computer-implemented method executed on data processing hardware that causes the data processing hardware to perform operations (Wolf: Section 4: “we demonstrate how such data pipelines can be built using off-the-shelf AI services, including Google’s Teachable Machine, Microsoft Azure Cognitive Image API, and OpenAI’s GPT models.”) comprising: receiving data representing an emergency scene, the data comprising image data captured by a camera in communication with a user device present at the emergency scene (Wolf: Figure 1: Input data; Section 2.2: “Figure 1 shows the flow of the individual steps from a climate hazard (Hurricane Ian used here) occurs by a citizen who then captures the hazard in a photo which includes meta-information such as an image description, date, time, and the location of the incident.”);
processing, using a classifier model, the received data to identify aspects at the emergency scene; detecting an emergency at the emergency scene based on the identified aspects at the emergency scene (Wolf: Figure 1: Data transformation; Section 2.2: “An image classifier using MobileNet via Google’s no-code Teachable Machine tool and a collection of images of different climate hazards. This allows us to classify the incoming images into different categories of natural hazards, such as large-scale or small-scale hazards, and different types, including floods and earthquakes. An off-the-shelf Microsoft Azure Cognitive Image API to detect and extract objects from images indicating if urban assets, environmental infrastructure, and people might be located in spatial proximity and could be impacted by the hazard…The next step, data transformation, involves reverse geocoding the incident coordinates, classifying the event type, and recognizing objects.”);
and in response to detecting the emergency at the emergency scene: generating, using a generative model configured to receive the image data representing the emergency scene, a descriptive summary of the emergency scene (Wolf: Figure 1: Output data; Tables 2 & 3; Section 2.2: “The next step, data transformation, involves reverse geocoding the incident coordinates, classifying the event type, and recognizing objects. This transformed data is then fed through generative natural langugage models to create the final report.”), the generated descriptive summary of the emergency scene providing detailed information about the emergency scene to assist an emergency responder in preparing for the emergency scene upon arrival (Wolf: 1 INTRODUCTION: “we demonstrate how combining off-the-shelf tools and services including Google Teachable Machine, Microsoft Azure Cognitive Image API, and OpenAI’s GPT models can accelerate the process of analysing incoming incident data, extract valuable information from submitted images, and help generate an automated hazard impact report. With this approach, we aim to reduce friction for citizens in time critical situations when reporting incidents and support emergency services in their response through (near-) real-time image analysis showing the impact of the hazard.”); and
providing the descriptive summary of the emergency scene to the emergency responder (Wolf: Figure 1: Action; Section 3.1: “While existing M/ETHANE reports provide vital information to the responding agencies, we show how we can further enhance their impact assessment through the analysis of imagery data. AI-assisted image analysis can provide a quicker assessment of the scope of response required, which would have taken more time and effort if responders had extracted and analysed image data manually.”; Wherein the final report is provided to the emergency responders), wherein the generative model comprises a large language model (Wolf: Figure 1: Output data; Tables 2 & 3; Section 2.2: “Our proof-of-concept workflow is as follows…OpenAI’s GPT models (accessed through Lex.page) to generate structured text descriptions based on keywords and predicted class labels provided by the Azure Cognitive Image API. This involves providing the model’s content and posing a series of specific questions.”; Wherein transformed image data are processed by the OpenAI GPT generative model, which is a large language model).
Wolf does not disclose expressly: the data comprising sound data captured by a microphone in communication with a user device present at the emergency scene; and in response to detecting the emergency at the emergency scene: generating, using a generative model configured to receive the sound data representing the emergency scene, a descriptive summary of the emergency scene, and wherein the classifier model and the generative model comprising the large language model are trained on a training data set that comprises sounds and corresponding descriptions of emergencies.
Thus, Wolf does not disclose expressly: a classifier model and a generative model comprising a large language model, wherein the classifier model and the generative model comprising the large language model are configured to process captured sound data and are trained on a training data set that comprises sounds and corresponding descriptions of emergencies.
Gong discloses: a model titled Listen, Think, and Understand (LTU), wherein the model processes audio data and performs audio perception and reasoning tasks (Gong: Abstract: “Therefore, we ask the question: can we build an AI model that has both audio perception and a reasoning ability? In this paper, we propose a novel audio foundation model, called LTU (Listen, Think, and Understand)…it exhibits remarkable reasoning and comprehension abilities in the audio domain.”). Wherein the LTU model is a generative AI system (Gong: 3 The OpenAQA Dataset: “Since LTU is a generative AI system and is anticipated to possess a wide range of capabilities for diverse audio tasks, existing datasets are not sufficient to train LTU.”) comprising a large language model (Gong: 2 LTU Model Architecture: “LLaMA Large Language Model. In this paper, we use the LLaMA-7B large language model [5] with Vicuna [9] instruction following training. LLaMA is pretrained on a combination of natural and programming language corpora in a self-supervised manner.”), wherein the LTU model processes the received data in order to perform audio classification (Gong: 3.1 Close-Ended Audio Question-Answer Generation: “We include four closed-ended audio tasks…Classification: The question for this task is “classify the sound events in the audio clip” and its GPT-assisted paraphrases; The answer is a list of sound class names that the audio clip contains.”) and descriptive summary generation (Gong: 5.1 Closed-Ended Audio Task Experiments: “Summarization: As an open-ended AQA model, LTU also exhibits a strong generalization ability across closed-ended datasets and tasks. Notably, LTU directly outputs its prediction in text, automatically filters out non-prominent sounds in the prediction, and requires no pre-defined label set for inference. These all make LTU a more practical system for real-world applications.”) tasks. The LTU model is trained on a training data set that comprises sounds and corresponding descriptions of the sounds, including emergencies (Gong: 1 Introduction: “In order to train LTU, we create a new dataset called OpenAQA-5M that combines 8 mainstream audio datasets and formats all data as (audio, question, answer) tuples.”;
Table 2: “Question: What mood or atmosphere does the audio clip convey? Answer: The audio clip conveys a sense of urgency, due to the sound of the ambulance siren, mixed with a sense of a busy, noisy environment, due to the sound of traffic noise.”).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to implement the audio processing LTU model taught by Gong into the emergency report generation system disclosed by Wolf, in order to incorporate audio captured by the user’s smartphone into the emergency report generation. The suggestion/motivation for doing so would have been “LTU understands the scene and can connect sounds to actions. In Table 5…Sample 3, LTU understands it is a dangerous scene and suggests seeking shelter when a gunshot is heard” (Gong: 5.2 Open-ended Audio Tasks Experiment). Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Wolf with Gong to obtain the invention as specified in claim 1.
Regarding claim 3, Wolf in view of Gong discloses: The computer-implemented method of claim 1, wherein providing the descriptive summary of the emergency scene to the emergency responder comprises providing, using a conversational assistant, the descriptive summary of the emergency scene to the emergency responder (Gong: 5.2 Open-ended Audio Tasks Experiment: “We observe LTU has superior performance in answering open-ended questions. In Table 5, we show LTU’s answers to questions asking about realistic audios. The audios are sampled from AudioSet evaluation set, which are extracted from YouTube videos and have not been used in LTU training.”)
(Wolf: Figure 1: Output data; Tables 2 & 3; Section 2.2: “OpenAI’s GPT models (accessed through Lex.page) to generate structured text descriptions based on keywords and predicted class labels provided by the Azure Cognitive Image API. This involves providing the model’s content and posing a series of specific questions” Section 3.2: “the suggested M/ETHANE report template can be enhanced and customized by providing additional questions relevant to other emergency agencies”; Wherein the GPT model disclosed by Wolf and the LTU model disclosed by Gong creating a report based on a template of questions asked regarding their respective data constitutes using a conversational assistant.).
Regarding claim 4, Wolf in view of Gong discloses: The computer-implemented method of claim 1, wherein the generative model is further configured to receive the aspects of the emergency scene when generating the descriptive summary of the emergency scene (Wolf: Figure 1: Input data; Section 1: “With this approach, we aim to reduce friction for citizens in time critical situations when reporting incidents and support emergency services in their response through (near-) real-time image analysis showing the impact of the hazard”; Wherein data submitted by a user, including images and audio, for the purposes of incident reporting is processed when generating the descriptive summary).
Regarding claim 5, Wolf in view of Gong discloses: The computer-implemented method of claim 1, wherein the operations further comprise: receiving, from a person, an indication of the emergency, wherein detecting the emergency at the emergency scene is further based on receiving the indication of the emergency (Wolf: Figure 1: Input data; Section 1: “With this approach, we aim to reduce friction for citizens in time critical situations when reporting incidents and support emergency services in their response through (near-) real-time image analysis showing the impact of the hazard”; Wherein a user submits captured data, including images and audio, for the purposes of reporting an incident, wherein the data is analyzed in order to extract information regarding the emergency).
Regarding claim 8, Wolf in view of Gong discloses: The computer-implemented method of claim 1, wherein the descriptive summary is generated to assist the emergency responder in preparing for the emergency scene upon arrival (Wolf: Figure 1: Action; Abstract: “To quickly and correctly assess the situation and deploy resources, emergency services often rely on citizen reports that must be timely, comprehensive, and accurate…By building on existing computer vision and natural language models, we demonstrate the automated generation of a full-form hazard impact assessment report from a single photograph.”; Section 4: “Building shared situational awareness is critical for emergency response agencies to respond effectively to an incident…This paper aims to simplify and accelerate this process to gather higher-quality information and reduce the burden on emergency services.”).
Regarding claim 9, Wolf in view of Gong discloses: The computer-implemented method of claim 1, wherein the descriptive summary comprises at least one of: states of one or more vehicles involved in an accident at the emergency scene; states of one or more airbags; locations of one or more vehicles at the emergency scene; a health status of one or more persons involved in the accident at the emergency scene; locations of one or more persons involved in the accident; a presence of fire; a presence of water; a presence of a roadway; damage to a roadway; a terrain topology; a presence of a leaking fluid; debris on a roadway; a roadblock condition; sounds at the emergency scene comprising speaking, crying, moaning, or barking; a description of surroundings; or a presence of weapons (Wolf: Tables 2 & 3; Wherein the reports generated based on the transformed input image’s data includes descriptions regarding vehicle states and locations, people present and their conditions, flood, road locations, roadblocks, and surroundings.)
(Gong: Table 5: “Real samples of LTU open-ended question answering. Answers are directly taken from the model output without modification. Notes in grey square brackets are not input to the model…Sample 3. Gun Shot Sound Question: Describe the audio clip with a sentence. [Closed-Ended Question] Answer: Gunshots and footsteps are heard, followed by a man speaking and a beep.”; Wherein audio and image data submitted by the user is processed when generating the descriptive summary).
Regarding claim 11, Wolf in view of Gong discloses: The computer-implemented method of claim 1, wherein the received capturing the data representing the emergency scene further comprises image data captured by a camera in communication with the user device present at the emergency scene (Wolf: Figure 1: Input data; Section 2.2: “A demonstration workflow for generating a pre-filled report using only a smartphone camera photograph is described herein and visualized in Figure 1. We assume that photographs are geotagged – either the smartphone embeds this information in the EXIF (the standard for metadata in media containers) fields or attaches the coordinates when uploading the photograph.”; Wherein the usage of geotagged photograph locations for report generation discloses a user present at the scene.).
Regarding claim 12, Wolf in view of Gong discloses: The computer-implemented method of claim 11, wherein providing the descriptive summary of the emergency scene to the emergency responder further comprises providing the image data captured by the camera to the emergency responder (Wolf: Section 3.3: “Third, we believe that reports containing images are more useful. If either cloud models or image-including reports are used, then both can rely on the same uploaded image.”).
Regarding claim 13, Wolf in view of Gong discloses: The computer-implemented method of claim 11, wherein the image data comprises an image of a person associated with the user device to facilitate identification of the person by the emergency responder (Wolf: Tables 2 & 3: “Can you identify people?”; Section 3.3: “Third, we believe that reports containing images are more useful. If either cloud models or image-including reports are used, then both can rely on the same uploaded image.”; Wherein the input images are used to identify people, including the user, as is disclosed in the report generating template questions shown in Tables 2 & 3, and wherein the final report includes the input images.).
Regarding claim 14, Wolf in view of Gong discloses: The computer-implemented method of claim 1, wherein the received data representing the emergency scene further comprises sensor data from one or more sensors in communication with the data processing hardware, the one or more sensors comprising at least one of a speed sensor, an altitude sensor, an accelerometer, a braking sensor, a position sensor, a temperature sensor, or a light sensor (Wolf: Table 1; Section 2.2: “We assume that photographs are geotagged – either the smartphone embeds this information in the EXIF (the standard for metadata in media containers) fields or attaches the coordinates when uploading the photograph.”; Wherein the smartphone geotagging photographs constitutes the usage of a position sensor.).
As per claim(s) 15, arguments made in rejecting claim(s) 1 are analogous. In addition, Abstract of Wolf discloses the usage of computer vision and natural language model, implying the usage of a system comprising data processing hardware and memory hardware, storing instructions, in communication with the data processing hardware.
As per claim(s) 17, arguments made in rejecting claim(s) 3 are analogous.
As per claim(s) 18, arguments made in rejecting claim(s) 4 are analogous.
As per claim(s) 19, arguments made in rejecting claim(s) 5 are analogous.
As per claim(s) 22, arguments made in rejecting claim(s) 8 are analogous.
As per claim(s) 23, arguments made in rejecting claim(s) 9 are analogous.
As per claim(s) 25, arguments made in rejecting claim(s) 11 are analogous.
As per claim(s) 26, arguments made in rejecting claim(s) 12 are analogous.
As per claim(s) 27, arguments made in rejecting claim(s) 13 are analogous.
As per claim(s) 28, arguments made in rejecting claim(s) 14 are analogous.
Claim(s) 6-7 and 20-21 is/are rejected under 35 U.S.C. 103 as being unpatentable over Wolf in view of Gong, and further in view of Gross (US10699580B1).
Regarding claim 6, Wolf in view of Gong discloses: The computer-implemented method of claim 4.
Wolf in view of Gong does not disclose expressly: wherein the operations further comprise: receiving, from a vehicle involved in the emergency, an indication of the emergency, wherein detecting the emergency at the emergency scene is further based on receiving the indication of the emergency.
Gross discloses: a vehicle detecting an emergency, and generating an incident report containing collected sensor data to send to an emergency responder (Gross: Col 12: Lines: 23-28: “Activities that can activate the accident reporting system include, but are not limited to, other vehicles hitting the vehicle 103, humans and/or animals touching and/or damaging the vehicle 103, vandalism to the vehicle 103, theft of the vehicle 103, and/or foreign objects falling on the vehicle 103.”;
Col 12: Lines: 58-61: “When an incident is detected, the vision system of the vehicle 103 takes pictures and/or video recordings of surroundings and the audio system records sounds made around the time of detection or interference. The data collected from detection of the incident can be recorded analyzed and used to generate an incident report. This report is sent to the user via the user device 106. The incident report can contain screen shots and/or video of the incident with probable perpetrators along with any audio that was recorded using an installed microphone during the incident. The incident report also can be sent to an emergency responder user device 106 a.”).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to implement the vehicle with automatic incident detection and incident reporting system taught by Gross as an additional input device for the hazard reporting system disclosed by Wolf in view of Gong. The suggestion/motivation for doing so would have been “But if the owner is not available, not available to respond quickly, or is incapacitated, it will be necessary to be able to transfer or hand off control of the automated vehicle to an emergency responder without receiving approval from the owner. Waiting to receive approval from the owner could result in injured persons not receiving timely attention and care. It also could create risk of further accidents and injuries, where a vehicle cannot be promptly moved out of the way of traffic.” (Gross: Col 2-3: Lines: 62-03: Wherein if a user is not able to contact emergency services for help, the vehicle may do so in order to the people involved to receive help as soon as possible.). Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Wolf in view of Gong with Gross to obtain the invention as specified in claim 6.
Regarding claim 7, Wolf in view of Gong discloses: The computer-implemented method of claim 1.
Wolf in view of Gong does not disclose expressly: wherein providing the descriptive summary of the emergency scene to the emergency responder further comprises providing the sound data captured by the microphone to the emergency responder.
Gross discloses: the capturing of audio data during a detected incident for the generation of an incident report, wherein the generated incident report, containing captured audio and image data, is provided to an emergency responder (Gross: Col 12: Lines: 52-61: “When an incident is detected, the vision system of the vehicle 103 takes pictures and/or video recordings of surroundings and the audio system records sounds made around the time of detection or interference. The data collected from detection of the incident can be recorded analyzed and used to generate an incident report. This report is sent to the user via the user device 106. The incident report can contain screen shots and/or video of the incident with probable perpetrators along with any audio that was recorded using an installed microphone during the incident. The incident report also can be sent to an emergency responder user device 106 a.”).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to implement the known technique of providing an emergency responder collected audio and image data disclosed by Gross into the emergency report generation system disclosed by Wolf in view of Gong by providing the emergency responder the generated descriptive summary along with collected image and audio data. The suggestion/motivation for doing so would have been “The incident report can contain screen shots and/or video of the incident with probable perpetrators along with any audio that was recorded using an installed microphone during the incident.” (Gross: Col 12: Lines: 58-61; Wherein the collected data may contain information that may be better expressed as audio/images rather than text.). Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Wolf in view of Gong with Gross to obtain the invention as specified in claim 7.
As per claim(s) 20, arguments made in rejecting claim(s) 6 are analogous.
As per claim(s) 21, arguments made in rejecting claim(s) 7 are analogous.
Claim(s) 10 and 24 is/are rejected under 35 U.S.C. 103 as being unpatentable over Wolf in view of Gong, and further in view of Do et al. (Virtual assistant for first responders using natural language understanding and optical character recognition) hereinafter referenced as Do.
Regarding claim 10, Wolf in view of Gong discloses: The computer-implemented method of claim 1, wherein: the descriptive summary comprises text (Wolf: Figure 1: Output data; Section 2.2: “OpenAI’s GPT models (accessed through Lex.page) to generate structured text descriptions based on keywords and predicted class labels provided by the Azure Cognitive Image API. This involves providing the model’s content and posing a series of specific questions”)
(Gong: 5.1 Closed-Ended Audio Task Experiments: “Audio Classification: Since LTU directly outputs audio label names or descriptions in text form instead of label index, in order to benchmark and compare it with existing models, we encode the LTU output and the evaluation dataset label names using a text encoder (gpt-text-embedding-ada), and compute the cosine similarity between the text embedding of LTU output and each label…We tested two prompts “classify the sound events in the audio clip” and ’write an audio caption describing the sound’, which let the model output a list of sound classes and a sentence describing the sound, respectively (examples shown in Table 5).”).
Wolf in view of Gong does not disclose expressly: the operations further comprise providing the text to a text-to-speech (TTS) system, the TTS system configured to convert the text into TTS audio data that conveys the descriptive summary as synthetic speech; and providing the descriptive summary of the emergency scene to the emergency responder comprises providing the TTS audio data to the emergency responder.
Do discloses: a virtual assistant for first responders, allowing for first responders to ask questions to, and receive information from, a chatbot regarding certain scenarios. Wherein the chatbot is able to respond to the user using text and text-to-speech (Do: Abstract: “We have developed a user-friendly application through understanding the hazardous material database, first aid safety guidelines and observing the process of first responders who access this information in the field. We created the Trusted and Explainable Artificial Intelligence for Saving Lives (TruePAL) virtual assistant using Dialogflow1 and TensorFlow2 paired with EasyOCR.3 The chatbot supports first responders by providing voice interaction which helps limit additional steps such as browsing through multiple categories when searching for information.”; 4.3.3 Chatbot: “The Chatbot allows first responders to talk using colloquial and domain specific language using either text or voice to provide information and instructions for dealing with various scenarios. Responses by the chatbot are given both as text and text to speech.”).
Before the effective filing date of the claimed invention, it would have been obvious to a person of ordinary skill in the art to process the final report disclosed by Wolf in view of Gong with the text-to-speech system disclosed by Do. The suggestion/motivation for doing so would have been “…to generate a visual and text-to-speech response. The functionality of the TruePAL virtual assistant allows for flexible use in both noisy and less noisy environments. This accommodation creates a better and trusted user experience.” (Do: 5.1 Conversation Design). Further, one skilled in the art could have combined the elements as described above by known methods with no change in their respective functions, and the combination would have yielded nothing more than predictable results. Therefore, it would have been obvious to combine Wolf in view of Gong with Do to obtain the invention as specified in claim 10.
As per claim(s) 24, arguments made in rejecting claim(s) 10 are analogous.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ANTHONY J RODRIGUEZ whose telephone number is (703)756-5821. The examiner can normally be reached Monday-Friday 10am-7pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Sumati Lefkowitz can be reached at (571) 272-3638. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ANTHONY J RODRIGUEZ/
Examiner, Art Unit 2672
/SUMATI LEFKOWITZ/Supervisory Patent Examiner, Art Unit 2672