DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office Action has been withdrawn pursuant to 37 CFR 1.114. Applicant’s submission filed on March 3, 2026 has been entered.
Status of Claims
Claims 1-3, 5-12, 18-19, 21-22, and 25-29 are pending.
Claims 4, 13-17, 20, 23-24, and 30-31 are cancelled.
Response to Arguments
Applicant’s arguments regarding “The Office Action’s “motivation to combine” Rao and Xiao”, presented on pages 9-10. In response to applicant’s argument that there is no teaching, suggestion, or motivation to combine the references, the examiner recognizes that obviousness may be established by combining or modifying the teachings of the prior art to produce the claimed invention where there is some teaching, suggestion, or motivation to do so found either in the references themselves or in the knowledge generally available to one of ordinary skill in the art. See In re Fine, 837 F.2d 1071, 5 USPQ2d 1596 (Fed. Cir. 1988), In re Jones, 958 F.2d 347, 21 USPQ2d 1941 (Fed. Cir. 1992), and KSR International Co. v. Teleflex, Inc., 550 U.S. 398, 82 USPQ2d 1385 (2007). In this case, Rao teaches using machine learning to analyze visual and audio information to diagnose a patient with a neurological condition, and Xiao teaches a method to combine features of visual data and audio data at different levels. The suggestion/motivation to combine the Rao and Xiao references would have been that methods of using video data to analyze visual and audio features are lacking. As disclosed in Xiao page 1 right hand column paragraph 2 “Given its high potential in facilitating video understanding, researchers have attempted to utilize audio in videos [41, 24, 2, 5, 58, 59, 3, 65, 23]. However, there are a few challenges in making effective use of audio. First, audio does not always correspond to the visual frames (e.g., in a “dunking basketball” video, there can be class-unrelated background music playing). Conversely, audio does not always contain information that can help understand the video (e.g., “shaking hands” does not have a particular sound signature). There are also challenges from a technical perspective. Specifically, we identify the incompatibility of “learning dynamics” between the visual and audio pathways – audio pathways generally train much faster than visual ones, which can lead to generalization issues during joint audiovisual training. Due in part to these various difficulties, a principled approach for audiovisual modeling is currently lacking. Many previous methods adopt an ad-hoc scheme that consists of a separate audio network that is integrated with the visual pathway via “late-fusion” [24, 2, 58]”. Accordingly, “this approach enforces strong temporal alignment between audio and visual features, as audio featured are fused into the Fast pathway which preserves fine temporal resolution” as noted by Xiao disclosure in page 4 left hand column paragraph 4. One of ordinary skill in the art would recognize combining Rao and Xiao would enable visual features to align with audio features which would allow for improved analysis of whether an individual has a neurological condition. For example, analysis of stroke considered both whether the individuals fast is drooping and speech is slurred, which contains both visual and audio features.
Applicant’s amendments of independent claims 1, 12, and 21 which have altered the scope of the claims of the instant application, have necessitated the new ground(s) of rejection presented in this office action with respect to claims of the instant application. Accordingly, in response to Applicant’s arguments that are merely directed to the amended portion of the claims, new analyses have been presented below, which make Applicant’s arguments moot.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 1-3, 6-9, 11-12, 18, 21-22, 25-27, and 29 are rejected under 35 U.S.C. 103 as being unpatentable over Rao et al. (US 2019/0110754 A1), in view of Xiao (“Audiovisual SlowFast Network for Video Recognition” – Published 2020), in further view of Eichler et al. (US 2022/0044821 A1).
Consider claim 12, Rao teaches “A system, comprising: a trained model (Rao paragraph [0086] "A supervised machine learning algorithm that is trained from data to produce a desired output. In the case of classification, the system's goal is to determine which of a set of diagnoses is most likely given the input") ,
at least one processing device (Rao paragraph [0034] "system for diagnosing a patient, the system comprising: at least one sensor in communication with a processor and a memory"); and
a non-transitory, processor-readable storage medium comprising programming instructions thereon that, when executed, cause the at least one processing device (Rao paragraph [0034] "a data processing module in communication with the processor and the memory; wherein said data processing module converts said raw patient data into processed diagnostic data; a diagnosis module in communication with the data processing module" and Rao paragraph [0070] "software "module" comprises a program or set of programs executable on a processor and configured to accomplish the designated task. A module may operate autonomously, or may require a user to input certain commands") to:
receive [[raw]] video of a subject presenting for a possible stroke, wherein the video captures the subject under instruction to perform the predetermined language task (Rao paragraph [0110] "Record close-up video (with audio) of the patient's face while they say a prompted sentence or perform an alternative method of speech analysis")
preprocess the image stream into a spatiotemporal facial frame sequence; preprocess the audio stream into a preprocessed audio component (Rao paragraph [0117] " The raw video and audio data usually needs to go through several stages of preparation before it can be used to train models. These stages include data preprocessing ( e.g., trimming video/audio, cropping video, adjusting audio gain, subsampling or supersampling time series, temporal smoothing, etc.), normalization (e.g., aligning audio clips to standard template, transforming face image to canonical view, detecting object of interest and cropping around it, etc.), and feature extraction (e.g., deriving Mel Frequency Cepstral Coefficients (MFCC) from acoustic data, computing optical flow features for video data, extracting and representing actions such as blinks or finger taps, etc.)");
analyze the spatiotemporal facial frame sequence and the preprocessed audio component using the trained model to of whether the subject is experiencing a stroke (Rao paragraph [0086] "A supervised machine learning algorithm that is trained from data to produce a desired output. In the case of classification, the system's goal is to determine which of a set of diagnoses is most likely given the input")
output the binary indication (Rao paragraph [0138] "The system would output the final diagnostic prediction to the patient along with intermediate model predictions. The system may display such an output on the screen of the device used to collect the initial senor data, or may transmit it to the relevant parties via other means, such as SMS messaging to a mobile device or sending an email to a designated party"),
Rao is not relied on to teach “comprising a deep neural network having an audio branch, a video branch, and a lateral connection between a convolutional block of the audio branch and a convolutional block of the video branch, wherein the trained model is trained with (1) training data comprising audio and video of individuals experiencing a clinical condition and instructed to perform a predetermined language task that requires either (i) describing a scene from a printed image or (ii) retrieving, thinking, organizing, and vocally expressing information, and wherein at least some of the individuals were experiencing a stroke, and (2) a binary label indicating whether each individual of the individuals were having a stroke or not having a stroke”, “split the [[raw]] video into an image stream and an audio stream”, “generate a binary [[an]] indication”, and “wherein the output is employed to trigger treatment of the subject as an acute ischemic stroke”.
However, Xiao teaches “a deep neural network having an audio branch (Xiao page 3 left hand column paragraph 3 "A key property of the Audio pathway is that it has an even finer temporal structure than the Slow and Fast pathways (with waveform sampling rate on the order of kHz)"), a video branch (Xiao page 2 left hand column paragraph 2 "The Slow pathway (Fig. 1, top row) is a convolutional network that processes videos with a large temporal stride (i.e., it samples one frame out of frames). The primary goal of the Slow pathway is to produce features that capture semantic contents of the video, which has a low refresh rate (semantics do not change all of a sudden). The Fast pathway (Fig. 1, middle row) is another convolutional model with three key properties"), and a lateral connection between a convolutional block of the audio branch and a convolutional block of the video branch (Xiao page 3 left hand column paragraph 4 "In addition to the lateral connections between the Slow and Fast pathways in [16], we add lateral connections between the Audio, Slow, and Fast pathways to fuse audio and visual features")”.
It would have been obvious to a person having ordinary skill in the art before effective filing date of the claimed invention of the instant to combine a system for detection of neurological condition using audio and visual analysis as taught by Rao to include lateral connection to fuse the audio and visual branch as taught by Xiao.
The suggestion/motivation to combine the Rao and Xiao references would have been that methods of using video data to analyze visual and audio features are lacking. As disclosed in Xiao page 1 right hand column paragraph 2 “Given its high potential in facilitating video understanding, researchers have attempted to utilize audio in videos [41, 24, 2, 5, 58, 59, 3, 65, 23]. However, there are a few challenges in making effective use of audio. First, audio does not always correspond to the visual frames (e.g., in a “dunking basketball” video, there can be class-unrelated background music playing). Conversely, audio does not always contain information that can help understand the video (e.g., “shaking hands” does not have a particular sound signature). There are also challenges from a technical perspective. Specifically, we identify the incompatibility of “learning dynamics” between the visual and audio pathways – audio pathways generally train much faster than visual ones, which can lead to generalization issues during joint audiovisual training. Due in part to these various difficulties, a principled approach for audiovisual modeling is currently lacking. Many previous methods adopt an ad-hoc scheme that consists of a separate audio network that is integrated with the visual pathway via “late-fusion” [24, 2, 58]”. Accordingly, “this approach enforces strong temporal alignment between audio and visual features, as audio featured are fused into the Fast pathway which preserves fine temporal resolution” as noted by Xiao disclosure in page 4 left hand column paragraph 4. One of ordinary skill in the art would recognize combining Rao and Xiao would enable visual features to align with audio features which would allow for improved analysis of whether an individual has a neurological condition. For example, analysis of stroke considered both whether the individuals fast is drooping and speech is slurred, which contains both visual and audio features.
The combination of Rao and Xiao is not relied on to teach “wherein the trained model is trained with (1) training data comprising audio and video of individuals experiencing a clinical condition and instructed to perform a predetermined language task that requires either (i) describing a scene from a printed image or (ii) retrieving, thinking, organizing, and vocally expressing information, and wherein at least some of the individuals were experiencing a stroke, and (2) a binary label indicating whether each individual of the individuals were having a stroke or not having a stroke”, “split the [[raw]] video into an image stream and an audio stream”, “generate a binary [[an]] indication”, and “wherein the output is employed to trigger treatment of the subject as an acute ischemic stroke”, “split the [[raw]] video into an image stream and an audio stream”, “generate a binary [[an]] indication”, and “wherein the output is employed to trigger treatment of the subject as an acute ischemic stroke”.
However, Eichler teaches “the trained model is trained with (1) training data comprising audio and video of individuals experiencing a clinical condition (Eichler paragraph [0063- 0064] "Tables 1-12 potential stroke features are extracted, via processors 104 (FIG. lA) and 104S (FIG. 18) as follows. Spatial ROIs 1602 , 1603 , 1604 , 1605 , 1606 , and 1607 from a time POI, i.e., image 1304 are extracted from image data 130. ROI 1621 is extracted from sound data 134. Multi-dimensional ROI 1641 is extracted from movement data 138. ROI 1661 is extracted from tactile data 142. The potential stroke features are extracted according to at least one predetermined stroke assessment criterion in Tables 1-12. In procedure 256, the potential stroke features are compared with classified sampled data acquired from a plurality of subjects, each positively diagnosed with at least one stroke condition, defining a positive stroke dataset […] Optionally additionally, the potential stroke features are compared with classified sampled data in a patient database acquired from a plurality of subjects, each negatively diagnosed with a stroke condition") and instructed to perform a predetermined language task (Eichler paragraph [0051] "Tables 1-12 hereinbelow show examples of predetermined stroke assessment criteria based on NIHSS, a computerized version of which according to the disclosed technique is denoted interchangeably herein as "modified NIHSS" (mNIHSS), and "adopted NIHSS"") that requires either (i) describing a scene from a printed image or (ii) retrieving, thinking, organizing, and vocally expressing information (Eichler Table 9 "Record the POI and ROI extraction: patient's responses with camera and microphone while the instructor is guiding the patient to read sentences and describe a picture of several objects, which is presented to the patient on the mobile device screen"), and wherein at least some of the individuals were experiencing a stroke, and (2) a binary label indicating whether each individual of the individuals were having a stroke or not having a stroke (Eichler paragraph [0043] "machine learning classification/classifier (MLC)) (also denoted herein as "pre-trained") so as to be enable to classify input data (e.g., distinguish, identify) among two main classes of potential stroke features stored in two different and main datasets, namely, a positive stroke dataset, and a negative stroke dataset. The positive stroke dataset includes a plurality of entries (labeled data) that are sampled from individuals positively diagnosed with at least one stroke condition. The negative stroke dataset includes a plurality of entries that are sampled from individuals negatively diagnosed for a stroke condition (i.e., are verified not to have a stroke condition)");”
“split the [[raw]] video (Eichler paragraph [0066] "Acquisition unit 106C1 of client device 101C1 captures clinical measurement data ( e.g., video data that includes image data 130 and sound data 134) pertaining to subject 10") into an image stream and an audio stream (Eichler Figure 4 and paragraph [0049] "extraction of potential stroke features from various types of clinical measurement data at various times, according to the disclosed technique. FIG. 4 shows the extraction of potential stroke features from image data 130 (images 1301, 1302, 1033, …) (as detailed in FIG.3), as well as sound data 134 in the time domain");”
PNG
media_image1.png
892
671
media_image1.png
Greyscale
Eichler Figure 4
“generate a binary [[an]] indication (Eichler paragraph [0043] "Given a tested potential stroke feature input, systems 1011 and 1012 are configured and/or trained to classify, i.e., associate the input potential stroke feature with either one of the positive stroke dataset (with a particular probability of match), the negative stroke dataset (with a particular probability of match)")” and
“herein the output is employed to trigger treatment of the subject as an acute ischemic stroke (Eichler paragraph [0068] "The mobile patient management console 220 enables the physician ( e.g., located remotely from subject 10, such as at a hospital, clinic, etc.) to view only the relevant ROIs and POIs (i.e., and not the entire clinical measurement data, such as the entire video), thereby saving time in the treatment of a stroke event")”.
It would have been obvious to a person having ordinary skill in the art before effective filing date of the claimed invention of the instant to combine a system for detection of neurological condition using audio and visual analysis as taught by Rao and Xiao to include binary classification and training of the classifier using data of patients with signs of stroke and patients with no signs of stroke as taught by Eichler.
The suggestions/motivation for doing so would have been “In ischemic stroke there is a deficiency or insufficiency of blood flow to cells, so as to meet the oxygen requirements, which leads to cerebral hypoxia and consequently to brain cell death also
known as cerebral infarction. Blood flow irregularities may be caused by a partial or complete blockage of blood vessels or arteries and is known to be caused by several factors which include thrombus (blood clot), embolus, and stenosis (internal narrowing of a blood vessel due to atheroma also known as plaque). In hemorrhagic stroke there is intracranial bleeding (due to a blood vessel rupture, leak, aneurysm), which can lead to an increase of intracranial pressure. Since brain cells die quickly after the onset of a stroke, treatment should begin as early as possible, given that stroke is currently one of the main causes of worldwide medical related death as well as disability. Therefore, there is a need to reduce the time to first treatment of stroke once it is detected” as noted by Eichler paragraph 2.
Therefore, it would have been obvious to combine the disclosure of Rao and Xiao with the Eichler disclosure to obtain the invention as specified in claim 12 as there is a reasonable expectation of success and/or because doing so merely combines prior art elements according to known methods to yield predictable results.
Claim 1 recites a method with steps corresponding to the system elements
recited in claim 12. Therefore, the recited steps of this claim are mapped to the
proposed combination in the same manner as the corresponding elements of system
claim 12. Additionally, the rationale and motivation to combine the Rao, Xiao and, Eichler references, presented in rejection of claim 12 apply to this claim.
Claim 21 recites a computer readable medium including computer executable instructions corresponding to the elements of the system recited in claim 12. Therefore, the recited instructions of the computer readable medium of claim 21 are mapped to the proposed combination in the same manner as the corresponding elements of the system claim 12. Additionally, the rationale and motivation to combine Rao, Xiao and, Eichler presented in rejection of claim 12, apply to this claim.
Regarding claim 2, the combination of Rao, Xiao and, Eichler teaches “The method of claim 1, wherein analyzing the spatiotemporal facial frame sequence (Rao paragraph [0088] "Video analysis of the patient may include analysis of the patient's face and facial movements, mouth specific movements, arm movements, full body movement, gait analysis, finger tapping") and the preprocessed audio component (Rao paragraph [0089] "Audio analysis (from video or microphone): Throughout the course of video recording, the audio signal may also be recorded. alternately, a microphone may be used to acquire audio data independently of a video. In some cases, when the focus is purely on movement, the audio data will not be used. However, in other aspects of the test, the audio signal may include speech from the patient or other sounds that are relevant to the task being performed and may provide diagnostic information ( e.g., Zhang, Y. (2017). Can a Smartphone Diagnose Parkinson Disease? A Deep Neural Network Method and Telediagnosis System Implementation. Parkinson's Disease, vol. 2017.)") to determine whether the subject is exhibiting signs of stroke the neurological condition includes analyzing additional information (Rao paragraph [0078] "the machine learning system as a whole will take the data acquired during these tests and use them to produce the desired output. In other embodiments, the system may also integrate background information about a patient including but not limited to age, sex, prior medical history, family history, and results from any additional or alternate medical tests").”
Regarding claim 3 (similarly claim 22), the combination of Rao, Xiao, and Eichler teaches “The method of claim 1, wherein the preprocessed audio component comprises one or more spectrograms, each of the one or more spectrograms representing an amplitude at each of a plurality of frequency levels over a period of time (Eichler Figure 4 and paragraph [0045] "Sound sensor 122C1 (e.g., a microphone) in client device 101C1 is configured and operative to acquire sound produced by subject 10 (i.e., typically voice, speech, and the like) and to produce corresponding sound data 134 that is graphically represented in FIG. 2 as a sound waveform (shown as a variation of amplitude in the time domain)").”
Regarding claim 6 (similarly claim 25), the combination of Rao, Xiao, and Eichler teaches “The method of claim 1, wherein capturing the [[raw]] video of the subject comprises receiving a video feed from a mobile device (Rao paragraph [0168] "First, the user instructs a mobile device, such as a cell phone or tablet computer, to run an application that can execute the program of the present invention (401). The user is then prompted to perform a series of tests on the subject to be diagnosed (402)") that has captured the subject repeating a predetermined sentence and describing a scene from a printed image (Eichler Table 9 "Record the POI and ROI extraction: patient's responses with camera and microphone while the instructor is guiding the patient to read sentences and describe a picture of several objects, which is presented to the patient on the mobile device screen").”
The proposed combination as well as the motivation for combining Rao, Xiao, and Eichler references presented in the rejection of claim 12, applies to claim 6. Finally the method recited in claim 6 is met by Rao, Xiao, and Eichler.
Regarding claim 7 (similarly claim 26), the combination of Rao, Xiao and, Eichler teaches “The method of claim 1, wherein preprocessing the image stream into the spatiotemporal facial frame sequence comprises extracting frontal face sequences from the [[raw]] video (Rao paragraph [0088] "The system may aid the user in collecting the appropriate images by providing an on-screen prompt, such as a frame on the video display of the device. Given a video sequence of the specific body location being observed, initial processing may be done to accurately localize the body part and its sub components ( e.g., the face and parts of the face such as eye and mouth locations)")..”
Regarding claim 8, the combination of Rao, Xiao and, Eichler teaches “The method of claim 1, further comprising receiving a 3D depth
data stream, wherein preprocessing the image stream comprises analyzing the 3D depth data stream to generate the spatiotemporal facial frame sequence (Rao paragraph [0090] " Range imaging systems record information about the structure of objects in view. Typically they record a depth value for every pixel in the image (though in the case of LiDAR, they may produce a full 3D point cloud for the visible scene). 2D depth data or 3D point cloud data can be integrated into the machine learning system to assist in object localization, keypoint detection, motion feature extraction, and classification/regression decisions. In many instances, this data is processed in a similar manner to image and audio data in that it often requires preprocessing, normalization, and feature extraction").”
Regarding claim 9 (similarly claim 18 and claim 27), the combination of Rao, Xiao, and Eichler teaches “The method of claim 1, wherein preprocessing the image stream into the spatiotemporal facial frame sequence (Rao paragraph [0091] "The temporal movement data can be processed in a similar way to the video data using preprocessing stages to prepare the data and feature extraction to obtain a discriminative representation that can be passed to the machine learning algorithm") comprises:
detecting a face of the subject with a face detection algorithm (Rao paragraph [0122] The first stage in processing the raw video data is to find a continuous region(s) within the video where the face is present, unobstructed, and at rest");
placing a square bounding box around the face of the subject (Rao paragraph [0125] "These techniques may rely on the change in the detected face box region from one frame to the next or similarly the change in the location of specific facial landmarks");
tracking the face of the subject as the face moves (Eichler Figure 3 and paragraph [0070] "Processor 104S is configured and operative to operate a program (e.g., an algorithm) that analyzes image data 130 as well as and sound data 134 typically in the form of video for each subject, such that facial landmarks (e.g., lips, face contour nose, etc.) in individual image frames from the video are identified and tracked in time so as to identify potential stroke features such as smile asymmetry, speech irregularities such as irregular connection between word pronunciation and lip movements (e.g., checked with respect to subject's baseline profile), and the like");
PNG
media_image2.png
925
687
media_image2.png
Greyscale
Eichler Figure 3
estimating a pose of the face (Eichler Table 3 "The algorithm analysis of the patient's gaze is quantified by calculating the head pose of the patient relative to the camera during this test" and "Eye coordination and the facial symmetry axis position for all video frames (including calculation of more measurements from these data such as variance, average speed, distance, etc.)" and paragraph [0069] "FIG. 12 shows a graph of an amalgamated position of right facial landmarks as well as a graph of an amalgamated position of left facial landmarks that are related to smiling of a subject of FIG. 11, and their interrelationship");
excluding any frame sequences outside one or more predetermined limits (Rao paragraph [0122] "Regions of the video where a face is not present will be discarded");
using a video stabilizer with a sliding window over a trajectory of between-frame affine transformations to smooth out pixel-level vibrations on one or more sequences (Rao paragraph [0125] "During the extraction of the regions of interest, image stabilization techniques are used to assure a smooth view of the object of interest within the cropped video sequence"); and
passing data corresponding to the one or more sequences to an encoder
(Rao paragraph [0081] "The machine learning algorithms typically involve
several stages of processing to obtain the output including: data preprocessing, data normalization, feature extraction, and classification/regression").”
The proposed combination as well as the motivation for combining Rao, Xiao, and Eichler references presented in the rejection of claim 12, applies to claim 9. Finally the method recited in claim 9 is met by Rao, Xiao, and Eichler.
Regarding claim 11 (similarly claim 29), the combination of Rao, Xiao and, Eichler teaches “The method of claim 1, wherein providing the binary indication (Eichler paragraph [0068] "The mobile patient management console 220 enables the physician ( e.g., located remotely from subject 10, such as at a hospital, clinic, etc.) to view only the relevant ROIs and POIs (i.e., and not the entire clinical measurement data, such as the entire video), thereby saving time in the treatment of a stroke event") further comprises providing supplemental information (Rao paragraph [0138] "The system might present additional information relevant to the diagnostic prediction (e.g., confidence scores, assessment of recording quality, recommendations for follow up tests, etc.). The app may also log relevant information and data from the tests and could pass along information regarding the diagnosis to a selected medical professional").”
The proposed combination as well as the motivation for combining Rao, Xiao, and Eichler references presented in the rejection of claim 12, applies to claim 11. Finally the method recited in claim 11 is met by Rao, Xiao, and Eichler.
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over Rao, Xiao and, Eichler, in view of Lakhani et al. (US 2015/0050010 A1).
Regarding claim 5, the combination of Rao, Xiao and, Eichler teaches the method of claim 1. However combination of Rao, Xiao and, Eichler is not relied on to teach “the preprocessed audio component further comprises a speech transcription”.
However, Lakhani teaches “the preprocessed audio component further comprises a speech transcription (Lakhani paragraph [0039] "At 430, text for a plurality of segments can then be combined. The combination can result in segmented transcripts and/or a full transcripts of the audio data").”
It would have been obvious to a person having ordinary skill in the art before
effective filing date of the claimed invention of the instant application to combine an
detection of neurological condition using audio and visual analysis taught as by Rao, Xiao and, Eichler to process the audio from speech to text as taught by Lakhani.
The suggestion/motivation for doing so would have been " Audio-to-text algorithms can be used to transcribe text from audio. An exemplary application is note-taking software. Audio-to-text, however, lacks semantic and contextual language understanding" and “At 150, an optional step of natural language processing can be applied to the text. For example, based on dictionary, grammar, and/or a knowledge database, the text extract from the audio stream of a video can be given context, an applied sentiment, and topical weightings” as noted by the Lakhani disclosure in paragraph 3 and paragraph 41.
Therefore, it would have been obvious to combine the disclosure of Rao, Xiao and, Eichler with the Lakhani disclosure to obtain the invention as specified in claim 5 as there is a reasonable expectation of success and/or because doing so merely combines prior art elements according to known methods to yield predictable results.
Claims 10, 19. 28 are rejected under 35 U.S.C. 103 as being unpatentable over Rao, Xiao and, Eichler, in view of Arik et al. (US 2019/0355347 A1).
Regarding claim 10 (similarly claim 19 and claim 28) the combination of Rao, Xiao and, Eichler teaches “The method of claim 1, wherein preprocessing the audio stream into the preprocessed audio component comprises: using a video processing tool to extract the audio stream from [[the]] raw video (Eichler Figure 4 and paragraph [0049] "extraction of potential stroke features from various types of clinical measurement data at various times, according to the disclosed technique. FIG. 4 shows the extraction of potential stroke features from image data 130 (images 1301, 1302, 1033, …) (as detailed in FIG.3), as well as sound data 134 in the time domain") and saving (Eichler paragraph [0053] "The subject specific data is part of the user (subject's) account stored in server database 102S of server 101S. Alternatively, all or at least part of the subject-specific data is stored in a memory storage (i.e., at least one of the hardware device, in software, firmware, removable storage medium, etc.) of client device 101 associated with the C1 subject (e.g., an owner, a user, of the client device). For example, baseline(s) dataset 186 is stored in memory storage of the client device (i.e., "in-memory database") and the extracted potential stroke features are stored on server database 102S") the audio stream as an audio file having a particular bit rate and frequency (Eichler paragraph [0045] "Alternatively, sound data is in a frequency domain (i.e., an amplitude value for each frequency in the frequency range of sound sensor 112C. Sound sensor 112C1 outputs sound data 134 to a preprocessor 136 that is configured and operative to preprocess sound data 134 by various techniques, which include for example, equalization, frequency band-pass filtering, level compression, noise reduction, etc");
using an audio analysis software package to load the audio file and trim one or more silent edges (Rao paragraph [0115] "trimming the ends of a video recording to remove irrelevant data, marking the beginning and end of speech");
generating a spectrogram from the converted data (Eichler Figure 4 and paragraph [0045] "Sound sensor 122C1 (e.g., a microphone) in client device 101C1 is configured and operative to acquire sound produced by subject 10 (i.e., typically voice, speech, and the like) and to produce corresponding sound data 134 that is graphically represented in FIG. 2 as a sound waveform (shown as a variation of amplitude in the time domain)".
The combination of Rao, Xiao and, Eichler is not relied on to teach “performing a short-time Fourier transform on a sound waveform of the audio file; converting (i) a scale of the audio file to decibel and (ii) a y-axis into a Mel-scale to produce converted data”.
However, Arik teaches “performing a short-time Fourier transform on a sound waveform of the audio file (Arik paragraph [0052] "Scaling with a trainable scalar may be used at the last layer to match the scaling of inverse STFT operation");
converting (i) a scale of the audio file to decibel and (ii) a y-axis into a Mel-scale to produce converted data (Arik paragraph [0045] "Neural networks to convert Mel spectrogram to linear spectrogram can be based on simple architectures such as fully-connected layers")”.
It would have been obvious to a person having ordinary skill in the art before
effective filing date of the claimed invention of the instant application to combine an
detection of neurological condition using audio and visual analysis taught as by Rao, Xiao and, Eichler to manipulate audio using Fourier transformation, Mel-scaling, and spectrogram storage as taught by Arik.
The suggestion/motivation for doing so would have been "Recent achievements of deep neural networks in generative modeling have been striking in many applications, from synthesis of high-resolution photorealistic images to highly natural speech samples. Three perennial goals remain towards their widespread adaptation: (i) matching the distribution of generated samples with the distribution of ground truth samples; (ii) enabling high-level controls by conditioning generation on some inputs; and (iii) improving the inference speed of generative models for hardware deployment. In this patent document, these goals are a focus for the application of waveform synthesis from a spectrogram. This problem is also known as spectrogram inversion in signal processing literature" as noted by the Arik disclosure in paragraph 37.
Therefore, it would have been obvious to combine the disclosure of Rao, Xiao and, Eichler with the Arik disclosure to obtain the invention as specified in claim 19 as there is a reasonable expectation of success and/or because doing so merely combines prior art elements according to known methods to yield predictable results.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JASPREET KAUR whose telephone number is (571)272-5534. The examiner can normally be reached Monday - Friday 9:30 am - 5:30 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Amandeep Saini can be reached at (571)272-3382. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JASPREET KAUR/Examiner, Art Unit 2662
/AMANDEEP SAINI/Supervisory Patent Examiner, Art Unit 2662
Prosecution Insights