DETAILED ACTION
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on XXXXXXXXXXXXXX has been entered.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Claims
Claims X are canceled.
Claims 4-6 are new.
Claims 1-6 are pending and have been examined.
This action is in reply to the papers filed on 08/20/2026 (effective filing date 10/21/2024).
Information Disclosure Statement
The information disclosure statements submitted: 10/17/2025 and 08/20/2026, have been considered by the Examiner and made of record in the application file.
Amendment
The present Office Action is based upon the original patent application filed on 10/17/2025 as modified by the amendment filed on 08/20/2026.
Reasons For Allowance
Prior-Art Rejection withdrawn
Claims 1-6 are potentially allowable over the prior-art, but, are still subject to the 35 USC §101 and/or 35 USC §112 rejections herein. The closest prior art (See PTO-892, Notice of References Cited) does not teach the claimed: Claim 1. (Currently Amended) A system comprising: a server; and a terminal; wherein the terminal comprises a camera, a microphone, a processor, storage, and random access memory and is configured to: capture, using the camera, images of building damage associated with a disaster scene; record, using the microphone, audio describing circumstances, of a user of the terminal, and requesting aid; and transmit, to the server, a structured data package comprising the captured images, the recorded audio, and location data; and wherein the server comprises a processor, storage, and random access memory and is configured to: receive the structured data package from the terminal; verify integrity of the structured data package; perform multimodal analysis by estimating, based on the recorded audio, an emotional state of the user and by analyzing the captured images to evaluate severity of the building damage; generate, based on the severity of the building damage, based on the aid requested in the recorded audio, and based on the estimated emotional state of the user, a prompt sentence for a generative artificial intelligence (Al) model, wherein the generated prompt sentence requests disaster relief resources suitable to the severity of the building damage and to the circumstances of the user; transmit, to the generative Al model, the generated prompt sentence; receive, from the generative Al model, a response to the prompt sentence, wherein the response recognizes the estimated emotional state of the user and recommends disaster relief resources; transmit, to the terminal for output to the user, the recommended disaster relief resources; and transmit, to a local government, results of the multimodal analysis and the recommended disaster relief resources.
The closest prior-art (Kadambi et al. 2023/0233091, Sharma et al. 2023/0005391, Sohum et al. 2021/0118009, Oliner et al. 2023/0069958, Sankaran et al. 2019/0147366) teach the features as disclosed in Non-final Rejection (06/08/2026), however, these cited references do not teach and the prior-art does not teach at least the following combination of features and/or elements:
Claim 1. (Currently Amended) A system comprising: … capture, using the camera, images of building damage associated with a disaster scene; record, using the microphone, audio describing circumstances, of a user of the terminal, and requesting aid; and transmit, to the server, a structured data package comprising the captured images, the recorded audio, and location data; and wherein the server comprises a processor, storage, and random access memory and is configured to: receive the structured data package from the terminal; verify integrity of the structured data package; perform multimodal analysis by estimating, based on the recorded audio, an emotional state of the user and by analyzing the captured images to evaluate severity of the building damage; generate, based on the severity of the building damage, based on the aid requested in the recorded audio, and based on the estimated emotional state of the user, a prompt sentence for a generative artificial intelligence (Al) model, wherein the generated prompt sentence requests disaster relief resources suitable to the severity of the building damage and to the circumstances of the user; transmit, to the generative Al model, the generated prompt sentence; receive, from the generative Al model, a response to the prompt sentence, wherein the response recognizes the estimated emotional state of the user and recommends disaster relief resources; transmit, to the terminal for output to the user, the recommended disaster relief resources; and transmit, to a local government, results of the multimodal analysis and the recommended disaster relief resources.
Claim Rejections - 35 USC §101 - Withdrawn
Per Applicant’s amendments and arguments and considering new guidance in the MPEP, the rejections are withdrawn. Specifically, in Applicant’s Remarks (dated 03/14/2017, pgs. 8-11), Applicant traverses the 35 USC §101 rejections arguing that the amended claims recite new limitations that are not abstract, amount to significantly more, are directed to a practical application, etc… For example, Applicant argues….
In support of their arguments, Applicant cites to the following recent Fed. Cir. court cases (i.e., Alice Corp. v. CLS Bank Int’l, SRI Int’l, Inc. v. Cisco Systems, Inc., Ultramercial, Inc. v. Hulu, LLC, Berkheimer, Core Wireless, McRO, Enfish, Bascom, DDR, etc…).
Claim Rejections - 35 USC § 101
35 U.S.C. § 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1–6 are rejected under 35 U.S.C. § 101 because the claimed invention is directed to a judicial exception (an abstract idea) without significantly more.
Step 1 (MPEP § 2106.03) asks whether a claim is directed to a process, machine, manufacture, or composition of matter. Claim 1 recites “A system comprising: a server; and a terminal,” and is therefore directed to a machine, one of the four statutory categories of invention. Claims 1–6 satisfy Step 1.
Having passed Step 1, the claims are evaluated under the two-step framework of Alice Corp. Pty. Ltd. v. CLS Bank International, 573 U.S. 208 (2014), and Mayo Collaborative Services v. Prometheus Laboratories, Inc., 566 U.S. 66 (2012), as implemented in MPEP § 2106.04 (Step 2A) and § 2106.05 (Step 2B).
Step 2A, Prong One – Does the claim recite a judicial exception?
Claim 1 - Under its broadest reasonable interpretation, claim 1 recites limitations that describe (i) a mental process — concepts performed in the human mind, or with the aid of pen and paper, including an observation, evaluation, judgment, or opinion — and (ii) a certain method of organizing human activity — managing personal behavior or relationships or interactions between people, such as matching a disaster victim's needs to available aid — both of which are abstract-idea groupings identified in MPEP § 2106.04(a)(2):
“verify integrity of the structured data package” — a person can manually check that received information is complete and unaltered, which is an evaluation;
“perform multimodal analysis by estimating, based on the recorded audio, an emotional state of the user and by analyzing the captured images to evaluate severity of the building damage” — a person (e.g., an emergency dispatcher or relief caseworker) can listen to a caller's voice and look at photographs to judge how distressed the caller sounds and how badly a building is damaged; this is a classic observation, evaluation, and judgment;
“generate ... a prompt sentence ... wherein the generated prompt sentence requests disaster relief resources suitable to the severity of the building damage and to the circumstances of the user” — deciding what aid to ask for, based on an assessment of the victim's damage and emotional state, mirrors the judgment a caseworker exercises when matching a victim to a relief program; and
the overall scheme of collecting a victim's information, assessing the victim's situation, and recommending/reporting appropriate aid is, at its core, a way of organizing the human activity of administering disaster relief — comparable to processing an application for assistance and evaluating eligibility, i.e., “certain methods of organizing human activity” under MPEP § 2106.04(a)(2)(II).
Because claim 1 recites these concepts, it recites an abstract idea under Step 2A, Prong One — consistent with the prior Office Action's characterization, which the amendment narrows in description but does not remove. See CyberSource Corp. v. Retail Decisions, Inc., 654 F.3d 1366, 1372–73 (Fed. Cir. 2011) (a method of analyzing information and forming a conclusion about a transaction is an unpatentable mental process even where a computer is nominally recited, because the underlying steps can be performed using human judgment).
Step 2A, Prong Two – Is the exception integrated into a practical application?
The additional elements of claim 1 — beyond the abstract idea identified above — are: a “server”; a “terminal” comprising “a camera, a microphone, a processor, storage, and random access memory”; the capturing of images and recording of audio; transmission of a “structured data package” (including “location data”) to the server; transmission of a “prompt sentence” to, and receipt of a “response” from, a “generative artificial intelligence (AI) model”; and transmission of results to the terminal and to “a local government.”
These additional elements do not integrate the abstract idea into a practical application. Each device and step is recited functionally and used for its known, conventional purpose: a camera captures images, a microphone records audio, a processor/storage/RAM run software, and a network connection carries data to a server and, from the server, to a third-party AI model and a government recipient. Having the handheld terminal gather data (because it has a camera and microphone) while the server performs the heavier processing (because servers typically have more computing capacity) reflects the ordinary, well-understood division of labor in client-server computing — it is not a technical improvement to the terminal, the server, the network, or the AI model. See In re TLI Communications LLC Patent Litig., 823 F.3d 607, 612–13 (Fed. Cir. 2016) (claims reciting a camera-equipped telephone that records, attaches classification data to, and transmits digital images to a server were ineligible because the components operated “in their normal, expected manner” and provided only “a generic environment in which to carry out the abstract idea”); Electric Power Group, LLC v. Alstom S.A., 830 F.3d 1350, 1354–55 (Fed. Cir. 2016) (collecting, analyzing, and communicating information from multiple sources remains abstract where the claim does not require any particular technical means of performing those functions beyond generic computer/network technology).
Nor does claim 1 specify any particular technical way of “verify[ing] integrity,” “estimating ... an emotional state,” “evaluat[ing] severity,” or “generat[ing] ... a prompt sentence” — these are recited only as results to be reached, without reciting the underlying technical mechanism for reaching them. Claiming a desired result without a specific technical means of achieving it is not indicative of a practical application. Electric Power Group, 830 F.3d at 1356. Sending a “prompt sentence” to a separate “generative AI model” and receiving back a “response” likewise reflects invoking a generic, off-the-shelf AI tool for its ordinary function (accepting a text prompt, returning a text output) — an “apply it” instruction to use existing technology for its known purpose, which MPEP § 2106.05(f) explains does not integrate an abstract idea into a practical application.
For these reasons, claim 1 does not integrate the abstract idea into a practical application, and claim 1 is directed to the identified abstract idea.
Step 2B – Does the claim recite significantly more?
The same additional elements evaluated in Step 2A, Prong Two — a generic server; a generic terminal with a generic camera, microphone, processor, storage, and RAM; conventional data capture, storage, and network transmission; and use of an off-the-shelf generative AI model — are well-understood, routine, and conventional computer functions when claimed at this level of generality. See MPEP § 2106.05(d); TLI Communications, 823 F.3d at 613 (receiving, storing, and transmitting digital images over a network are basic, conventional functions of generic computer components). Automating collection of data from a mobile-type device and transmitting it to a server for generic AI-assisted processing does not supply an inventive concept; it is a routine application of existing technology. See Alice, 573 U.S. at 225–26 (“the mere recitation of a generic computer cannot transform a patent-ineligible abstract idea into a patent-eligible invention”). Considered as an ordered combination, the elements add nothing beyond what each contributes individually: data is gathered on a mobile-type device, sent to a server, evaluated, used to prompt an AI model, and the results are relayed to the user and a government recipient — a sequence of conventional data-gathering, analysis, and transmission steps that has repeatedly been found insufficient to supply an inventive concept. Electric Power Group, 830 F.3d at 1355.
Claim 1 therefore does not include an inventive concept sufficient to transform the abstract idea into patent-eligible subject matter and claim 1 is rejected under 35 U.S.C. § 101.
Claims 2–6 (Dependent Claims)
Claim 2 (“wherein the terminal is a smartphone, a tablet, or a personal computer”): Claim 2 narrows the terminal to well-known, general-purpose computing devices. Reciting a particular but conventional off-the-shelf device merely limits the abstract idea to a particular technological environment or field of use, which does not integrate the exception into a practical application or supply an inventive concept. MPEP § 2106.05(h). Claim 2 is rejected under 35 U.S.C. § 101 for the reasons given for claim 1.
Claim 3 (“wherein the multimodal analysis employs a natural language processing algorithm and an image processing algorithm”): Claim 3 recites generic categories of software used for their known purpose (extracting meaning from audio/speech; analyzing image content), without any specific improved technique. Invoking generic NLP and image-processing software to implement the abstract idea of evaluating emotional state and damage severity is an “apply it” instruction using existing generic technology and reflects well-understood, routine, conventional computer functionality. MPEP §§ 2106.05(d), (f); see TLI Communications, 823 F.3d at 613. Claim 3 is rejected under 35 U.S.C. § 101.
Claim 4 (“wherein the terminal comprises smart glasses”) and Claim 5 (“wherein the terminal comprises a headset”): Each substitutes a different, but still known, consumer-electronics form factor for the same generic camera/microphone/processor/storage/RAM components already recited in claim 1, without adding any new function. This does not change the eligibility analysis. MPEP § 2106.05(h). Claims 4 and 5 are rejected under 35 U.S.C. § 101.
Claim 6 (“wherein the terminal comprises a robot”): Although a robot can be a more complex machine than a smartphone, claim 6 does not recite any additional structure or function of the robot (e.g., autonomous navigation, manipulation, or sensing) beyond the generic camera, microphone, processor, storage, and RAM already recited in claim 1 and used for the same generic capture/record/transmit functions. Reciting a generic robot as an alternative housing for the same generic components does not integrate the abstract idea into a practical application or add significantly more. MPEP §§ 2106.05(d), (h). Claim 6 is rejected under 35 U.S.C. § 101.
For the foregoing reasons, claims 1–6 are rejected under 35 U.S.C. § 101 as being directed to a judicial exception without significantly more.
Additional Eligibility Considerations – Signal-Per-Se / Software-Per-Se (35 U.S.C. § 101)
The claims were also reviewed under MPEP § 2106.03 to determine whether any claim is directed to no more than a transitory, propagating electromagnetic signal per se (see In re Nuijten, 500 F.3d 1346, 1356–57 (Fed. Cir. 2007) (a claim to a signal, standing alone and without any physical, structural limitation, is not a “process, machine, manufacture, or composition of matter” and is therefore ineligible under § 101)), or is directed to software per se (i.e., a computer program or set of instructions untethered from any recited hardware or tangible medium).
No rejection on either ground is warranted here. Claim 1 is drafted as a “system” and affirmatively recites physical, structural hardware — “a server” and “a terminal,” with the terminal reciting “a camera, a microphone, a processor, storage, and random access memory” and the server reciting “a processor, storage, and random access memory.” These structural recitations place the claim within the statutory “machine” category, not a bare signal or a bare set of instructions. Dependent claims 2 and 4–6 reinforce this by reciting specific, tangible terminal embodiments (a smartphone, tablet, or personal computer; smart glasses; a headset; or a robot). Accordingly, claims 1–6 do not present a signal-per-se or software-per-se eligibility problem.
Claim Rejections/Objections – 35 U.S.C. § 112
35 U.S.C. § 112(b) – Definiteness / Antecedent Basis
Claims 1–6 were reviewed for antecedent-basis and clarity issues. See MPEP § 2173.05(e). Every “the [term]” recitation in claim 1 (e.g., “the camera,” “the captured images,” “the recorded audio,” “the user,” “the building damage,” “the structured data package,” “the generative AI model,” “the generated prompt sentence,” “the prompt sentence,” “the estimated emotional state of the user,” and “the recommended disaster relief resources”) traces back to an earlier positive recitation of that term, including where the earlier recitation uses a participle (e.g., “captured images” ties back to “capture ... images”). No outright lack-of-antecedent-basis defect is identified in claim 1 or in dependent claims 2–6.
Two clarity points are nonetheless flagged, though neither alone is believed to rise to a formal indefiniteness rejection under the “reasonable certainty” standard of Nautilus, Inc. v. Biosig Instruments, Inc., 572 U.S. 898, 910 (2014):
1. The phrase “record, using the microphone, audio describing circumstances, of a user of the terminal, and requesting aid” contains a comma after “circumstances” that could be read as modifying “audio” rather than “circumstances.” Applicant may wish to amend to “audio describing circumstances of a user of the terminal and requesting aid” to remove the ambiguity.
2. “Results of the multimodal analysis,” transmitted to the local government, is a new noun phrase built on the previously defined term “multimodal analysis”; it is reasonably clear, but Applicant may wish to clarify whether “results” means the estimated emotional state and evaluated severity, the recommended resources, or both.
35 U.S.C. § 112(f) – Means- (or Step-) Plus-Function
Claim 1 recites the terminal and the server as each “configured to” perform a series of functions. Because claim 1 does not use the word “means,” there is a rebuttable presumption that 35 U.S.C. § 112(f) does not apply; that presumption is overcome only if the claim otherwise fails to recite sufficiently definite structure for performing the recited functions. Williamson v. Citrix Online, LLC, 792 F.3d 1339, 1348–49 (Fed. Cir. 2015) (en banc); see also MPEP § 2181 (three-prong analysis for identifying a § 112(f) limitation).
Applying that standard, the terminal and server limitations of claim 1 are not believed to invoke § 112(f), because “terminal” and “server” are not used as unbounded, generic “nonce” placeholders (e.g., “module for,” “mechanism for,” “means for”) — each is expressly modified by recited structural hardware (a camera, a microphone, a processor, storage, and random access memory for the terminal; a processor, storage, and random access memory for the server) that conveys to a person of ordinary skill in the art the structure understood to perform the recited functions. Accordingly, § 112(f) is not currently invoked for claim 1, and no means-plus-function construction has been applied in the § 101 analysis above. This determination should be revisited if the claims are amended to remove the structural recitations.
35 U.S.C. § 112(a) – Written Description
Section 112(a) requires that the specification, as filed, reasonably convey to a person of ordinary skill in the art that the inventor had possession of the claimed invention as of the filing date. Ariad Pharmaceuticals, Inc. v. Eli Lilly & Co., 598 F.3d 1336, 1351 (Fed. Cir. 2010) (en banc); MPEP § 2163.
Applicant's remarks identify paragraphs [0015]–[0025], [0041]–[0047], [0060]–[0070], [0073], [0080]–[0091], [0094], [0101]–[0113], and [0116], and FIGS. 1–8 and 13, as support for the amendments. Based on a review of the publicly available specification, the following appear to have reasonably clear support: the terminal's camera/microphone-based collection of images, audio, and location/text data, and transmission of that data to a server; the server's use of a speech-to-text and emotion-recognition (“emotion identification”) model to estimate a user's emotional state from audio, and image-analysis techniques to assess building damage; the server's generation of a “prompt sentence” for a generative AI model that incorporates the assessed damage, the aid requested, and the user's emotional state; and the server's transmission of recommended relief measures to both the terminal and a local-government (or equivalent public-institution) system. The specification also appears to describe smart-glasses, headset-type, and robot terminal embodiments (supporting dependent claims 4–6) and speech-to-text/image-recognition software (supporting claim 3's “natural language processing algorithm and an image processing algorithm”).
Two limitations warrant closer comparison against the exact, certified specification text before the written-description issue is considered fully resolved:
3. “verify integrity of the structured data package” (claim 1) — the specification appears to describe the server logging incoming data and “ensuring completeness and authenticity,” together with secure-transmission protocols (e.g., HTTPS/TLS), but it is not clear from the available excerpt whether the specification describes an “integrity” check (e.g., a checksum, hash, or similar tamper-evidence mechanism) with enough specificity to show possession of this particular claim term, as opposed to more general secure-transmission language. The precise supporting passage(s) should be identified and confirmed.
4. “wherein the response recognizes the estimated emotional state of the user” (claim 1) — i.e., that the response received back from the generative AI model itself recognizes the user's emotional state — should be checked against the specification to confirm it describes the AI model's response as reflecting or acknowledging the emotional state, as opposed to describing only the server (prior to invoking the AI model) as estimating the emotional state and feeding it into the prompt. If the specification shows only the latter, Applicant should be prepared to point to the specific supporting passage for the former.
----- Examiner’s Response to Arguments -----
Per Applicants’ amendments/arguments, the rejections are withdrawn.
Applicant's arguments have been considered but are moot in view of the new ground(s) of rejection.
Applicants’ amendments have necessitated the new grounds of rejection noted above.
Examiner’s Response: Claim Rejections – 35 USC § 103
Per Applicants’ amendments/arguments, the rejections are withdrawn. See notes above for additional reasoning and rationale for dropping prior-art rejection including Applicant’s amendments and arguments and unique combination of features and elements not taught by the prior-art without hindsight reasoning.
Applicant's arguments have been considered but are moot in view of the new ground(s) of rejection.
Applicants’ amendments have necessitated the new grounds of rejection noted above.
Regarding Claim X, on page(s) 8-9 of Applicant’s Remarks / After Final Amendments (dated 07/15/2011), Applicant(s) argues that the cited reference(s) (Ellis and Vandermolen) fails to teach, describe, or suggest the amended features. Specifically, Applicant(s) argues that cited reference(s) do not teach, describe, or suggest the following: . With respect, Applicant’s arguments are deemed unpersuasive and the amended feature(s) remain rejected as follows.
With respect, Applicant’s arguments are deemed unpersuasive and the amended feature(s) remain rejected as follows.
Examiner’s Response: Claim Rejections – 35 USC §112
Per Applicants’ amendments/arguments, the rejections are withdrawn.
Applicant's arguments have been considered but are moot in view of the new ground(s) of rejection.
Applicants’ amendments have necessitated the new grounds of rejection noted above.
Examiner’s Response: Claim Rejections – 35 USC §101
Per Applicants’ amendments/arguments, the rejections are withdrawn. See notes above for additional reasoning and rationale for dropping 35 USC 101 rejection including Applicant’s amendments, arguments, lack of abstract idea, and practical integration.
Applicant's arguments have been considered but are moot in view of the new ground(s) of rejection.
Applicant's remarks, filed in response to the Non-Final Office Action dated June 8, 2026, have been fully considered but are not persuasive.
Applicant argues that amended claim 1 is not directed to a judicial exception because it recites “additional elements that incorporate any such judicial exception into a practical application” — specifically, the division of the claimed steps between a terminal (which captures images and records audio using its camera and microphone) and a server (which performs the multimodal analysis, prompt generation, and communication with the generative AI model and local government). Applicant further argues, quoting specification paragraph [0042], that this arrangement solves problems of conventional disaster-victim-support systems, which have had difficulty “accurately recogniz[ing] users' emotional states and needs” and lack “efficient mechanisms to rapidly analyze multimodal data ... make tailored support program recommendations, and share these determinations promptly with local authorities.”
This argument is not persuasive. The problem Applicant identifies — accurately recognizing a disaster victim's needs and emotional state and efficiently matching and communicating appropriate aid — is a problem of organizing information and coordinating aid, not a technical problem rooted in computer technology. And the claimed “solution” is simply to have each generic device do what it already does best: a handheld terminal, because it has a camera and a microphone, gathers image/audio/location data, while a server, because it typically has more computing capacity, performs data processing and communicates with an external AI model and a government system. That kind of task allocation is standard, well-understood client-server architecture — not a technical improvement to the terminal, the server, the network, or the AI model. See In re TLI Communications LLC Patent Litig., 823 F.3d 607, 612–13 (Fed. Cir. 2016) (a claim reciting a camera-equipped phone that captures, classifies, and transmits images to a server for storage was ineligible because the components operated in their normal, expected manner and the claim did not improve the functioning of the phone, server, or network); Electric Power Group, LLC v. Alstom S.A., 830 F.3d 1350, 1354 (Fed. Cir. 2016) (distributing conventional data-gathering, analysis, and communication functions across a network of generic devices does not, by itself, integrate an abstract idea into a practical application).
Applicant's related argument — that a server can be “more easily configured with additional computational resources” than a terminal and can therefore “facilitate more consistent emotion-based responses from an AI model” — describes a general, well-known advantage of using a server for compute-intensive tasks. It does not identify any specific technical improvement to how the server, the terminal, or the AI model actually performs its function, nor any specific technical mechanism (e.g., a particular algorithm, data structure, or protocol) that achieves the improvement. A claim to a desired result — more consistent, better-tailored recommendations — without a claimed technical means of achieving that result is not indicative of eligibility. Electric Power Group, 830 F.3d at 1356.
For at least these reasons, the rejection of claims 1–6 under 35 U.S.C. § 101 is maintained. Applicant is invited to contact the Examiner to discuss potential amendments — for example, claim language reciting a specific technical mechanism for the integrity check, image analysis, or emotional-state estimation, or a specific technical improvement to how the terminal, server, and AI model interact — that might advance prosecution.
Any comments considered necessary by applicant must be submitted no later than the payment of the issue fee and, to avoid processing delays, should preferably accompany the issue fee. Such submissions should be clearly labeled “Comments on Statement of Reasons for Allowance.”
----- Conclusion -----
PERTINENT PRIOR ART – Patent Literature
The prior-art made of record and considered pertinent to applicant's disclosure.
Oliner et al. 2023/0069958 [0016 - the client module 122, the server 104 collects, stores, manages, analyzes, evaluates, indexes, monitors, learns from, visualizes, and transmits information to the client module 122 based upon data collected from unstructured data in images, video, audio, text, and graphs originally resident on data sources]
Sankaran et al. 2019/0147366 [0019 - perform deep classification analysis on multimodal data of a registered user (which is collected and stored in the database of multimodal user data 137) to extract various features that are indicative of a user's preferences, activities, interests, etc., which may not be self-identified. For example, in one embodiment, different types of deep learning classification techniques are utilized to process different types of multimodal user data (e.g., time-series data, unstructured text data, multimedia data, etc.) to generate feature sets for each type of multimodal user data]
PERTINENT PRIOR ART – Non-Patent Literature (NPL)
The NPL prior-art made of record and considered pertinent to applicant's disclosure.
Peng Z, Ma R, Zhang Y, Yan M, Lu J, Cheng Q, Liao J, Zhang Y, Wang J, Zhao Y, Zhu J, Qin B, Jiang Q, Shi F, Qian J, Chen X and Zhao C (2023) Development and evaluation of multimodal AI for diagnosis and triage of ophthalmic diseases using ChatGPT and anterior segment images: protocol for a two-stage cross-sectional study. Front. Artif. Intell. 6:1323924. doi: 10.3389/frai.2023.1323924.
THIS ACTION IS MADE FINAL
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action.
Contact Information
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MATTHEW T. SITTNER whose telephone number is (571) 270-7137 and email: matthew.sittner@uspto.gov. The examiner can normally be reached on Monday-Friday, 8:00am - 5:00pm (Mountain Time Zone). Please schedule interview requests via email: matthew.sittner@uspto.gov
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Sarah M. Monfeldt can be reached on (571) 270-1833.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MATTHEW T SITTNER/
Primary Examiner, Art Unit 3629b
Claims 1 and 2 are rejected under 35 U.S.C. 103 as being unpatentable over: Kadambi et al. 2023/0233091; in view of Sharma et al. 2023/0005391.
19/361,636 – Claim 1. Kadambi et al. 2023/0233091 teaches A system comprising a processor, wherein the processor is configured to collect audio data, image data, and text data from a device (Kadambi et al. 2023/0233091 [0017 - system includes an RF sensor that captures RF data; and an audio sensor that captures audio data; wherein the disease diagnosis application direct the processor to perform an acoustic features process that: divides audio data captured by the audio sensor into a first section with continuous speech and a second section with forced coughs][0052 - a multimodal health (MMHealth) sensor platform for remote diagnosis of various medical conditions. In many embodiments, the MMHealth platform can include (but are not limited to) 3D and/or 2D image sensors][0056 - mobile device 120 may be a mobile phone, Personal Digital Assistant (PDA), a tablet, a smartphone, or any other type of device that connects to network 160 via wireless connection without departing from this invention. The mobile device and/or personal device can include one or more cameras for capturing images, recording videos and also one or more sensors for capturing vital sign data. In many embodiments, mobile device 120 and/or personal device 180 can include 3D and/or 2D image sensors, audio sensors, among other vital sign sensors][0064-0065 - processors][0068 - multimedia data can include (but is not limited to) video, images, and/or audio. Multimedia data in accordance with some embodiments of the invention can include video captured by an individual to measure their vital signs to be transmitted to a physician for review. In some embodiments, multimedia data can include video captured of multiple individuals, where vital signs can be measured for one or more of the individuals in the video. As is discussed further below, specialized hardware can be utilized to capture additional sensor data including multi-modal image data.]), perform multimodal analysis on the data collected from the device (Kadambi et al. 2023/0233091 [0015 - includes a multi-modal system for diagnosing disease, including: several different types of sensors, including: an RGB camera that captures image data…][0040 - FIG. 11 conceptually illustrates a multi-modal health (MMHealth) device…][0052; 0092; 0096; 0128][0123 - enables an effective analysis of these multimodal image data for vital sign estimation]), diagnose applicable support programs based on analysis results obtained by the multimodal analysis (Kadambi et al. 2023/0233091 [0092 - the multimodal platform can be applied to a variety of different applications, including include infectious disease diagnosis (e.g., COVID-19, other viral and bacterial pneumonias), mental health disorders (e.g., via ocular cues, facial expression cues), sleep disorders, heart arrhythmias, stroke, among numerous other applications]), notify a user of a diagnosis result (Kadambi et al. 2023/0233091 [0054 – communications network][0061 - measured vitals can then be displayed for the user on the user's device. In a number of embodiments, measured vital signs can be further analyzed to screen for potential conditions and can provide notifications or alerts to a user to seek treatment][0082 - provide a variety of outputs to a user, including (but not limited to) charts, images, vital sign data, notifications, instructions, and/or alerts. In numerous embodiments, output engines can communicate with other devices to transmit vital sign data][0090 - an output based on the signal processing and extraction. Outputs in accordance with numerous embodiments of the invention can include visual data (e.g., charts, images, etc.), notifications (e.g., alerts, reminders, warnings, etc.) and/or text data (e.g., vital sign data, instructions, recommendations, among others). In various embodiments, the generated outputs can be transmitted to a physician for a review or as part of a telemedicine session.]), and share the diagnosis result with a local government (Kadambi et al. 2023/0233091 [0128 - data decode and multi-unit inference pipeline, alerts (e.g., notification for inference result), device management and telemetry, and local data storage][0090 - an output based on the signal processing and extraction. Outputs in accordance with numerous embodiments of the invention can include visual data (e.g., charts, images, etc.), notifications (e.g., alerts, reminders, warnings, etc.) and/or text data (e.g., vital sign data, instructions, recommendations, among others). In various embodiments, the generated outputs can be transmitted to a physician for a review or as part of a telemedicine session.]).
Kadambi et al. 2023/0233091 may not expressly disclose the “share the diagnosis result with a local government” features, however, Sharma et al. 2023/0005391 teaches (Sharma et al. 2023/0005391 [0037 - medical test and treatment results for a patient are shared with government officials…] In one example, a credit card applicant sees her own PII (Personally Identifiable Information) including her SSN (Social Security Number) in plain text. However, a customer support agent sees only the last 4 digits of her SSN for identity verification purposes. The rest is masked. In another example, a front-office staffer leverages the power of encrypted computation to match the SSN of a customer without ever viewing the entire SSN column in plain text. Only a confirmation of the match is displayed. In the United States, the SSN is a unique nine-digit number assigned to a person for taxpayer identification purposes. It is also used for banking, medical, and insurance transactions. In another example, a physician can view and edit medication information of only those patients she treats and not the entire patient database. In another example, medical test and treatment results for a patient are shared with government officials only if the patient has given consent to share their data.). Before the effective filing date of the claimed invention, it would have been obvious for one of ordinary skill in the art to have modified Kadambi et al. 2023/0233091 to include the features as taught by Sharma et al. 2023/0005391. One of ordinary skill in the art would have been motivated to do so to utilize well known tools and features useful for implement a system for multimodal analysis, diagnosis and reporting which should prove to improve user experience, maximize profits, and optimize revenue (i.e., advertisement optimization / improve user experience).
19/361,636 – Claim 2. Kadambi et al. 2023/0233091 further teaches The system according to claim 1, wherein the device is a smartphone, a tablet, or a personal computer (Kadambi et al. 2023/0233091 [0056-0057; 0067; 0145; 0165]).
Claim 3 is rejected under 35 U.S.C. 103 as being unpatentable over: Kadambi et al. 2023/0233091; in view of Sharma et al. 2023/0005391; in further view of Sohum et al. 2021/0118009.
19/361,636 – Claim 3. Kadambi et al. 2023/0233091 further teaches The system according to claim 1, wherein the multimodal analysis employs a natural language processing algorithm and an image processing algorithm (Kadambi et al. 2023/0233091 [0015; 0048; 0072; 0076; 0077; 0121]).
Kadambi et al. 2023/0233091 may not expressly disclose the “multimodal analysis employs a natural language processing algorithm” features, however, Sohum et al. 2021/0118009 teaches (Sohum et al. 2021/0118009 [0008 - the computer-implemented method enables a multi-modal natural language analysis…][0049 - the multi-modal natural language analysis is run using the natural language processing module 114 for dynamic transcription…]). Before the effective filing date of the claimed invention, it would have been obvious for one of ordinary skill in the art to have modified Kadambi et al. 2023/0233091 to include the features as taught by Sohum et al. 2021/0118009. One of ordinary skill in the art would have been motivated to do so to utilize well known tools and features useful for implement a system for multimodal analysis, diagnosis and reporting which should prove to improve user experience, maximize profits, and optimize revenue (i.e., advertisement optimization / improve user experience).
Prosecution Insights