DETAILED ACTION
This communication is in response to the Application filed on 10/17/2023 (provisional). Claims 1-20 are pending and have been examined.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Specification
The disclosure is objected to because of the following informalities:
in ¶ [0036]: “Thus, in at least one aspect, system 100 can be used to produced synthetic speech for users that are unable to speak (e.g., laryngectomees).” should read:
Thus, in at least one aspect, system 100 can be used to produce synthetic speech for users that are unable to speak (e.g., laryngectomees).
in ¶ [0069]: “In another example, communications interface 118 can include a Wi-Fi transceiver and/or cellular or mobile phone communications transceivers for wireless communications.” should read:
In another example, communications interface 118 can include a Wi-Fi® transceiver and/or cellular or mobile phone communications transceivers for wireless communications.
in ¶ [0098]: “FIGS. 9D and 9D are DTW aligned trajectories for y and z dimensions, respectively…” should read:
FIGS. 9C and 9D are DTW aligned trajectories for y and z dimensions, respectively…
in ¶ [0103]: “In this experiment, the input data from the commercial EMA were 18-dimensonal.” should read:
In this experiment, the input data from the commercial EMA were 18-dimensional.
Appropriate correction is required.
The use of the term WI-FI®, which is a trade name or a mark used in commerce, has been noted in this application. The term should be accompanied by the generic terminology; furthermore, the term should be capitalized wherever it appears or, where appropriate, include a proper symbol indicating use in commerce such as ™, SM, or ® following the term.
Although the use of trade names and marks used in commerce (i.e., trademarks, service marks, certification marks, and collective marks) are permissible in patent applications, the proprietary nature of the marks should be respected and every effort made to prevent their use in any manner which might adversely affect their validity as commercial marks.
The lengthy specification has not been checked to the extent necessary to determine the presence of all possible minor errors. Applicant’s cooperation is requested in correcting any errors of which applicant may become aware in the specification.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 1, 9, and 11 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Ghovanloo (US 2014/0342324 A1).
Regarding claim 1, Ghovanloo teaches a system for generating synthesized sound or speech, the system comprising:
a sensor positioned on a user’s tongue to generate position data and orientation data associated with a position and orientation of the user’s tongue (see [0031] where, a method for tracking a subject's tongue movement and position is provided. The method can comprise positioning in the mouth of a subject a tracer unit non-obstructively carried by the tongue, such that a change in position or gesture of the tongue changes the position or orientation of the tracer unit; detecting the position and orientation of the tracer unit; generating a sensor signal and tracking information based on the detected position and orientation of the tracer unit; and transmitting the sensor signal and tracking information to a computing platform, wherein the computer program processes the received data into a graphical representation, such as for example and not limitation, a tracing, line drawing, or 3D drawing. Also see [0081], where an assistive system and apparatus for tracking a subject's tongue in real-time and in 3D during speech is provided. In some embodiments, the system and apparatus can comprise a tracer unit adapted for non-obstructive affixation to the tongue of the subject, such that a change in position of the tongue changes the position and/or orientation of the tracer; a plurality of sensors for detecting a position and/or orientation and/or movement of the tracer unit and adapted for non-obstructive placement proximal to the tracer unit; and a sensor control unit for transmitting a sensor signal to a computing platform based on the detected position of the tracer unit);
one or more processors (see [0105], where the FPGA controller can also comprise a processor/controller unit, for example and not limitation, to perform further processing of the data received from the sensor unit or plurality of sensor units. Also see element 13 in Figure 1); and
memory having instructions stored thereon that, when executed by the one or more processors (see [0118], where the FPGA controller 32 can comprise one or more of a preprocessor 34, memory, and/or a parallel sensor interface 35. The FPGA controller 32 can implement fast pre-processing of the sensor data before sending it to a central computing platform 5 for, for example and not limitation, additional processing, localization, recording, and/or display), cause the system to: receive the generated position data and orientation data from the sensor (see [0031], a method for tracking a subject's tongue movement and position is provided. The method can comprise positioning in the mouth of a subject a tracer unit non-obstructively carried by the tongue, such that a change in position or gesture of the tongue changes the position or orientation of the tracer unit; detecting the position and orientation of the tracer unit; generating a sensor signal and tracking information based on the detected position and orientation of the tracer unit; and transmitting the sensor signal and tracking information to a computing platform, wherein the computer program processes the received data into a graphical representation, such as for example and not limitation, a tracing, line drawing, or 3D drawing);
generate, via an articulation conversion model, synthesizable sound or speech data using the generated position data and orientation data (see [0092], where the production of tongue trajectories by embodiments of the TTS system and apparatus does not necessarily require perfect and legible vocalization. Therefore, only tongue motion or magnetic signature with or without an imperfect or severely disordered voice can be sufficient to for the TTS to recognize words, phrases, and sentences. Thus, the TTS system can be an effective silent speech interface. Also see [0092], where the TTSSI signal processing algorithm can also map the user's tongue trajectories or magnetic/acoustic signatures to their identical vocalized words for natural real-time speech synthesis, by using a pre-recorded voice of the user or for example and not limitation, synthesized speech using Siri, Dragon Naturally Speaking, Microsoft Speech application programming interface (API), and/or Web Speech API. In these examples, TTS and TTSSI refer to Tongue Tracking System and Tongue Tracking Silent Speech Interface, respectively); and
output the synthesizable sound or speech data as at least one of: (i) audio of synthesized voice or speech (see [0004], the TTS can also be used as a silent speech interface by using the acoustic-kinematic recordings of a subject who may have a voice disorder, for example, to automatically build, index, and reproduce speech from a database of phonemes, words, phrases, and commands based on that user's specific acoustic-kinematic recording. In this example TTS refers to Tongue Tracking System), which fully corresponds to the first alternative of the “memory having instructions stored thereon that, when executed by the one or more processors, cause the system to: output the synthesizable sound or speech data as at least one of: (i) audio of synthesized voice or speech, or (ii) a textual representation of the synthesizable sound or speech data” limitation in claim 1.
Regarding claim 9, Ghovanloo teaches the system of claim 1, wherein the sensor is further configured to detect three-dimensional (3D) magnetic field signals based on detected variations in a local magnetic field corresponding to movement of the tongue (see [0025], where the sensor can comprise an array of magnetic field sensors or magnetometers that are capable of detecting 3D magnetic fields. Also see [0149], where the sensors 12 convert the magnetic field intensity (B) from the magnet to a proportional analog output voltage), wherein the instructions further cause the system to:
convert the 3D magnetic field signals into additional 3D position information and additional 2D orientation information (see [0089], where to find the position and orientation of the tracer unit, the Tongue Tracking System signal processing algorithm can minimize a cost function that correlates with the difference between the measured, B, and the estimated, B (sᵢ, a, m), magnetic flux densities at the location of each magnetometer in the plurality of sensors using the following equation:
f
a
,
m
=
∑
i
=
1
N
B
ᵢ
,
m
e
a
s
-
B
(
s
ᵢ
,
a
,
m
)
where
a
=
<
a
x
,
a
y
,
a
z
>
is the magnet’s location,
m
=
<
m
sin
θ
cos
Φ
,
m
sin
θ
sin
Φ
,
m
cos
θ
>
is the tracer's orientation and
s
i
=
<
s
i
x
,
s
i
y
,
s
i
z
>
is the location of the magnetic sensor); and
augment the position data and the orientation data with the additional 3D position information and the additional 2D orientation information (see [0090], where the magnetic sensor data can be fused with additional sensor data, such as for example and not limitation, acoustic, video, touch, pressure, and flow data, in the post-processing stage of embodiments of the system, thus creating a comprehensive analytical tool for speech and language therapy and research. Also see [0109], where the system and apparatus can comprise a computing platform receiving the sensor signal from the sensor control unit, translating and processing the sensor signal to obtain tracking information from the tracer unit, generating, for example and not limitation, a graphical and/or audiovisual representation of the tracking information, and displaying the graphical and/or audiovisual representation on the graphical interface of the computing platform or a graphical display for review by the SLP, clinician, or tutor. In some embodiments, the graphical and/or audiovisual representation can comprise, for example and not limitation, a tracing, line drawing, or 3D drawing of the tongue movement and position based on the movement of the tracer unit during speech. … The signal processing constitutes of multimodal sensor data fusion, machine learning, and pattern recognition methods. Also see [0128], where algorithm 111 can localize the magnetic tracer 1 within the 3D oral space and can fuse it with additional sensor data, which can be, for example and not limitation, from a camera 6 and/or microphone 7) for generating the synthesizable sound or speech data (see [0004], where the Tongue Tracking System can also be used as a silent speech interface by using the acoustic-kinematic recordings of a subject who may have a voice disorder, for example, to automatically build, index, and reproduce speech from a database of phonemes, words, phrases, and commands based on that user's specific acoustic-kinematic recordings).
Regarding claim 11, Ghovanloo teaches a method for generating synthesized sound or speech, the method comprising:
obtaining, from a sensor positioned on a tongue of a user, position data and orientation data associated with a position and orientation of the tongue (see [0031] where, a method for tracking a subject's tongue movement and position is provided. The method can comprise positioning in the mouth of a subject a tracer unit non-obstructively carried by the tongue, such that a change in position or gesture of the tongue changes the position or orientation of the tracer unit; detecting the position and orientation of the tracer unit; generating a sensor signal and tracking information based on the detected position and orientation of the tracer unit; and transmitting the sensor signal and tracking information to a computing platform, wherein the computer program processes the received data into a graphical representation, such as for example and not limitation, a tracing, line drawing, or 3D drawing. Also see [0081], where an assistive system and apparatus for tracking a subject's tongue in real-time and in 3D during speech is provided. In some embodiments, the system and apparatus can comprise a tracer unit adapted for non-obstructive affixation to the tongue of the subject, such that a change in position of the tongue changes the position and/or orientation of the tracer; a plurality of sensors for detecting a position and/or orientation and/or movement of the tracer unit and adapted for non-obstructive placement proximal to the tracer unit; and a sensor control unit for transmitting a sensor signal to a computing platform based on the detected position of the tracer unit);
generating, via an articulation conversion model, synthesizable sound or speech data using the generated position data and orientation data (data (see [0092], where the production of tongue trajectories by embodiments of the TTS system and apparatus does not necessarily require perfect and legible vocalization. Therefore, only tongue motion or magnetic signature with or without an imperfect or severely disordered voice can be sufficient to for the TTS to recognize words, phrases, and sentences. Thus, the TTS system can be an effective silent speech interface. Also see [0092], where the TTSSI signal processing algorithm can also map the user's tongue trajectories or magnetic/acoustic signatures to their identical vocalized words for natural real-time speech synthesis, by using a pre-recorded voice of the user or for example and not limitation, synthesized speech using Siri, Dragon Naturally Speaking, Microsoft Speech application programming interface (API), and/or Web Speech API. In these examples, TTS and TTSSI refer to Tongue Tracking System and Tongue Tracking Silent Speech Interface, respectively); and
outputting the synthesizable sound or speech data as at least one of: (i) audio of synthesized voice or speech (see [0004], the TTS can also be used as a silent speech interface by using the acoustic-kinematic recordings of a subject who may have a voice disorder, for example, to automatically build, index, and reproduce speech from a database of phonemes, words, phrases, and commands based on that user's specific acoustic-kinematic recording. In this example TTS refers to Tongue Tracking System), which fully corresponds to the first alternative of the “method for generating synthesized sound or speech, the method comprising: outputting the synthesizable sound or speech data as at least one of: (i) audio of synthesized voice or speech, or (ii) a textual representation of the synthesizable sound or speech data” limitation in claim 11.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 2, and 12 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ghovanloo in view of Cao et al. (2021). "Investigating speech reconstruction for laryngectomees for silent speech interfaces," Proceedings of Interspeech, 2021, pp. 651-655 .
Regarding claim 2, Ghovanloo teaches all the limitations as in claim 1 but fails to teach generating the synthesizable sound or speech data includes to: generate phoneme data from the generated position data and orientation data; and convert the phoneme data to the synthesizable sound or speech data.
However, Cao et al. (2021) does teach generating the synthesizable sound or speech data which includes to: generate phoneme data from the generated position data and orientation data (see Page 652, Section 3.2, where rather than a traditional SSR that converts articulatory to text [11, 10], the real-time SSR (RT SSR) convert articulation movement frames into numeric textual frames (the one-hot encoding of phonemes). Then the TTS component directly converts the recognized text frames to speech. Also see Page 652, Section 3.3, where articulatory movement is the input of ATS and SSR. Text labels are the output of SSR and the input of TTS. For mngu0 data and the EL speech data, the alphabetic phoneme labels of the sentences were aligned to their audio samples using the Festival speech synthesis system [34]. The articulatory data was synchronously recorded with the audio; thus, each data frame (acoustic and articulatory) was labeled with a phoneme. There are 48 phonemes represent the textual information includes: the 39 English phonemes in the CMU dictionary, silence (“sil”), and extra phonemes: [“ax”, “axr”, “dx”, “el”, “em”, “en”, “hv”, “nx”], which are same to that used in the Merlin and HTS toolkits [32, 35]. The phoneme frame labels were finally converted to 48-dimensional numeric vectors which indicate the phoneme of current frames (one-hot encoding, one for the current phoneme, zeros for the other 47 phonemes)), and convert the phoneme data to the synthesizable sound or speech data (see Page 652, Section 3.2, where rather than a traditional SSR that converts articulatory to text [11, 10], the real-time SSR (RT SSR) convert articulation movement frames into numeric textual frames (the one-hot encoding of phonemes). Then the TTS component directly converts the recognized text frames to speech. Also see Figure 1. proposed RT-SSR+TTS design (b) Real-time silent speech recognition and text-to-speech).
Ghovanloo and Cao et al. (2021) are both considered to be analogous to the claimed invention because they are in the same field of silent speech interfaces and tongue tracking analysis. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Ghovanloo to incorporate the teachings of Cao et al. (2021) to include generating phoneme data from the generated position data and orientation data; and converting the phoneme data to the synthesizable sound or speech data in order to allow for motion-to-phoneme conversion for the purpose of synthesizing speech without the need for the user to input audio (see Page 651, Section 1., where the SSR+TTS design is known to be able to generate quality speech without requiring audio data from the users [10, 11, 12], thus could be more suitable for clinical applications if its high latency issue could be resolved).
Regarding claim 12 which depends from claim 11 and recites a method, this claim is rejected as unpatentable over the same combination of prior art applied against claim 2 (i.e., Ghovanloo in view of Cao et al. (2021)). Ghovanloo teaches all the limitations of claim 11 as noted above. As detailed in the rejection of claim 2, the disclosed system teaches or renders obvious each step of the method recited in claim 12. Accordingly, claim 12 is rejected for the same reasons set forth in the rejection of claim 2.
Claim(s) 3, and 13 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ghovanloo in view of Wang et al. (2014). "Preliminary test of a real-time, interactive silent speech interface based on electromagnetic articulograph," Proceedings of the 5th Workshop on Speech and Language Processing for Assistive Technologies, pp. 38-45.
Regarding claim 3, Ghovanloo teaches all the limitations in claim 1 but fails to teach generating text associated with sound or speech from the generated position data and orientation data; and converting the text to the synthesizable sound or speech data using a text-to-speech conversion model.
However, Wang et al. (2014) does teach generating the synthesizable sound or speech data includes to: generating text associated with sound or speech from the generated position data and orientation data (see Section 2.1, where Figure 1 illustrates the three-component design of the SSI: (a) real-time articulatory motion tracking using a commercial EMA, (b) online silent speech recognition (converting articulation information to text), and (c) text-to-speech synthesis for speech output); and converting the text to the synthesizable sound or speech data using a text-to-speech conversion model (see Section 2.1, where the third component played back either pre-recorded or synthesized sounds using a text-to-speech synthesizer. Also see Figure 1. Design of the real-time silent speech interface. Also see Section 3.2, where Figure 2 (right) demonstrates a participant is using the SSI during the test. After the participant silently articulated “Good afternoon”, the SSI displayed the phrase on the screen, and played the corresponding synthesized sound simultaneously).
Ghovanloo and Wang et al. (2014) are both considered to be analogous to the claimed invention because they are in the same field of silent speech interfaces and tongue tracking analysis. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Ghovanloo to incorporate the teachings of Wang et al. (2014) to include generating text associated with sound or speech from the generated position data and orientation data; and converting the text to the synthesizable sound or speech data using a text-to-speech conversion model in order to allow for motion-to-text-to-speech conversion for the purpose of synthesizing speech of natural quality (see Section 4, where the voice output quality (determined by the text to-speech synthesizer) was natural, which strongly supports the major motivation of SSI research: to produce speech with natural voice quality that current treatments cannot provide).
Regarding claim 13 which depends from claim 11 and recites a method, this claim is rejected as unpatentable over the same combination of prior art applied against claim 3 (i.e., Ghovanloo in view of Wang et al. (2014)). Ghovanloo teaches all the limitations of claim 11 as noted above. As detailed in the rejection of claim 3, the disclosed system teaches or renders obvious each step of the method recited in claim 13. Accordingly, claim 13 is rejected for the same reasons set forth in the rejection of claim 3.
Claim(s) 4, and 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ghovanloo in view of Kim et al. (2017). "Speaker-independent silent speech recognition from flesh-point articulatory movements using an LSTM neural network," IEEE/ACM Transactions on Audio, Speech, and Language Processing, 25(12), pp. 2323-2336.
Regarding claim 4, Ghovanloo teaches all the limitations in claim 1 but fails to teach the articulation conversion model comprising a Gaussian mixture model (GMM) combined with a hidden Markov model (HMM) or a deep neural network (DNN) combined with a hidden Markov model (HMM).
However, Kim et al. (2017) does teach the articulation conversion model comprising a Gaussian mixture model (GMM) combined with a hidden Markov model (HMM) or a deep neural network (DNN) combined with a hidden Markov model (HMM) (see Page 2329, Section III, Subsection F, where the following SSR systems were compared to evaluate the proposed method: GMM-HMM, DNN-HMM, LSTM-HMM, and BLSTM-HMM. We used hidden Markov model (HMM)-based silent speech recognition systems where each state can be modeled by GMM or neural network).
Ghovanloo and Kim et al. (2017) are both considered to be analogous to the claimed invention because they are in the same field of silent speech interfaces and tongue tracking analysis. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Ghovanloo to incorporate the teachings of Kim et al. (2017) to include the articulation conversion model comprising a Gaussian mixture model (GMM) combined with a hidden Markov model (HMM) or a deep neural network (DNN) combined with a hidden Markov model (HMM) in order to allow for more accurate silent speech recognition (see Page 2324, Section I, where the promise of deep acoustic models to improve the speech recognition accuracy motivates the application of deep articulatory models in silent speech recognition).
Regarding claim 15 which depends from claim 11 and recites a method, this claim is rejected as unpatentable over the same combination of prior art applied against claim 4 (i.e., Ghovanloo in view of Kim et al. (2017)). Ghovanloo teaches all the limitations of claim 11 as noted above. As detailed in the rejection of claim 4, the disclosed system teaches or renders obvious each step of the method recited in claim 15. Accordingly, claim 15 is rejected for the same reasons set forth in the rejection of claim 4.
Claim(s) 5, 10, 16, and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ghovanloo in view of Cao et al. (2018). “Articulation-to-speech synthesis using articulatory flesh point sensors' orientation information,” Proceedings of Interspeech, pp. 3152-3156.
Regarding claim 5, Ghovanloo teaches all the limitations in claim 1 but fails to teach the articulation conversion model comprises a long short-term memory (LSTM) recurrent neural network (RNN).
However, Cao et al. (2018) does teach the articulation conversion model comprises a long short-term memory (LSTM) recurrent neural network (RNN) (see Page 3153, Section 3.1, where ATS models that predict acoustic parameters from articulatory position and orientation data with a trained DNN or LSTM-RNN were implemented. Also see Page 3154, Section 3.3, where the best combination of position and orientation components were fed into an LSTM-RNN-based ATS to determine if LSTM-RNN outperforms DNN. We hypothesized that LSTM-RNN might show better performance than DNN for the combination of orientation and position information [12]).
Ghovanloo and Cao et al. (2018) are both considered to be analogous to the claimed invention because they are in the same field of silent speech interfaces and tongue tracking analysis. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Ghovanloo to incorporate the teachings of Cao et al. (2018) to include an articulation conversion model comprising a long short-term memory (LSTM) recurrent neural network (RNN) in order to better model the temporal relationship between acoustic parameters and the tongue’s position and orientation (see Page 3153, Section 3.1, where long short-term memory-recurrent neural networks (LSTM-RNNs) can model long-range temporal information by overcoming the vanishing gradient problem in conventional recurrent neural networks (RNNs). LSTM-RNN based models have been successfully used in ATS with only articulatory position information by outperforming DNN-based ATS [12,10,24]. Therefore, we adopted LSTM-RNN-based ATS to model the long-range temporal relationship between acoustic parameters and both the articulatory position and orientation).
Regarding claim 10, Ghovanloo teaches all the limitations in claim 1 but fails to teach a pair of second sensors positioned on the user’s lips to generate lip movement data by tracking movement of the lips, wherein the instructions further cause the system to: receive the lip movement data from the pair of second sensors, and wherein the synthesizable sound or speech data is generated using lip movement data in addition to the generated position data and orientation data.
However, Cao et al. (2018) does teach a pair of second sensors positioned on the user’s lips to generate lip movement data by tracking movement of the lips, wherein the instructions further cause the system to: receive the lip movement data from the pair of second sensors (see Page 3153, Section 2, where the 3D position vector P [xyz] of 6 sensors (Figure1): upper lip (UL), lower lip (LL), lower incisor (LI), tongue tip (TT), tongue body (TB), tongue dorsum (TD) extracted from raw EMA data, and the 3D orientation vector of each sensor provided by the dataset were used), and wherein the synthesizable sound or speech data is generated using lip movement data in addition to the generated position data and orientation data (see Page 3155, Section 5, where the effectiveness of applying orientation information of sensors attached to flesh points on articulators (tongue, lips, and jaw) in articulation-to-speech (ATS) synthesis was investigated. EMA sensors’ position information with and without orientation information were used as input to DNN-based ATS and LSTM-RNN-based ATS).
Ghovanloo and Cao et al. (2018) are both considered to be analogous to the claimed invention because they are in the same field of silent speech interfaces and tongue tracking analysis. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Ghovanloo to incorporate the teachings of Cao et al. (2018) to include a pair of second sensors positioned on the user’s lips to generate lip movement data by tracking movement of the lips, wherein the instructions further cause the system to: receive the lip movement data from the pair of second sensors, and wherein the synthesizable sound or speech data is generated using lip movement data in addition to the generated position data and orientation data in order to capture better orientational information of the articulators on top of positional data to further improve speech synthesis (see Page 3152, Section I, where most of current ATS works used only the articulatory movement information (spatial coordinates), although magnetic tracking technologies provide orientation information of sensors attached to the articulators (e.g., tongue, lips, and jaw) and recent studies suggest the sensor orientation information is significant in speech production [16]).
Regarding claim 16 which depends from claim 11 and recites a method, this claim is rejected as unpatentable over the same combination of prior art applied against claim 5 (i.e., Ghovanloo in view of Cao et al. (2018)). Ghovanloo teaches all the limitations of claim 11 as noted above. As detailed in the rejection of claim 5, the disclosed system teaches or renders obvious each step of the method recited in claim 16. Accordingly, claim 16 is rejected for the same reasons set forth in the rejection of claim 5.
Regarding claim 20 which depends from claim 11 and recites a method, this claim is rejected as unpatentable over the same combination of prior art applied against claim 10 (i.e., Ghovanloo in view of Cao et al. (2018)). Ghovanloo teaches all the limitations of claim 11 as noted above. As detailed in the rejection of claim 10, the disclosed system teaches or renders obvious each step of the method recited in claim 20. Accordingly, claim 20 is rejected for the same reasons set forth in the rejection of claim 10.
Claim(s) 6, 7, 8 , 17, 18, and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ghovanloo in view of Sebkhi et al. (2021). "Inertial measurements for tongue motion tracking based on magnetic localization with orientation compensation," IEEE Sensors Journal, 21(6), pp. 7964-7971.
Regarding claim 6, Ghovanloo teaches all the limitations in claim 1 but fails to teach wherein the position data comprises left-right (x), superior-inferior (y), and anterior-posterior (z) coordinates.
However, Sebkhi et al. (2021) does teach wherein the position data comprises left-right (x), superior-inferior (y), and anterior-posterior (z) coordinates (see Page 7967, Section III, where the linear stage comprises three motorized XSlides (Velmex Inc., Bloomfield, NY, USA) that can position the tracer in the 3D space (X, Y, and Z) with a reported accuracy of 76 μm).
Ghovanloo and Sebkhi et al. (2021) are both considered to be analogous to the claimed invention because they are in the same field of silent speech interfaces and tongue tracking analysis. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Ghovanloo to incorporate the teachings of Sebkhi et al. (2021) to include the position data comprising left-right (x), superior-inferior (y), and anterior-posterior (z) coordinates in order to capture the 3-dimensional position of the tongue’s tracer for the purpose of speech recognition (see Page 7965, Section I, where permanent magnet localization (PML) has the potential to overcome many of the shortcomings of the current tongue tracking technologies since the tracer (i.e. magnet) is small enough to not be obtrusive, provides continuous tracking in the whole 3D space, and is capable of providing millimetric tracking accuracy that is required for typical tongue tracking applications such as speech recognition [10]).
Regarding claim 7, Ghovanloo teaches all the limitations in claim 1 but fails to teach wherein the orientation data comprises pitch and roll.
However, Sebkhi et al. (2021) does teach the orientation data comprises pitch and roll (see Page 7967, Section III, where the rotational stage is capable of orienting the tracer along its pitch and roll thanks to a pulley/belt system driven by two stepper motors in half-step mode, resulting in an accuracy of 0.9◦).
Ghovanloo and Sebkhi et al. (2021) are both considered to be analogous to the claimed invention because they are in the same field of silent speech interfaces and tongue tracking analysis. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Ghovanloo to incorporate the teachings of Sebkhi et al. (2021) to include the orientation data comprising pitch and roll in order to improve accuracy and correctly interpret the sensor data using orientation compensation (see Page 7970, Section IV, Part B, where using an orientation compensation method to reduce the complexity of the model to be learned results in more accurate tracking).
Regarding claim 8, Ghovanloo teaches all the limitations in claim 1 but fails to teach wherein the sensor comprises an inertial measurement unit (IMU), wherein the position data is three-dimensional and the generated orientation data is two-dimensional.
However, Sebkhi et al. (2021) does teach wherein the sensor comprises an inertial measurement unit (IMU) (see Page 7965, Section I, where the technology centers around the use of an inertial measurement unit (IMU), as the tongue tracer, that moves in a local magnetic field generated by a magnet strip composed of an array of permanent magnets. An IMU combines into one package a magnetometer, an accelerometer, and a gyroscope), wherein the position data is three-dimensional (see Page 7967, Section III, where the linear stage comprises three motorized XSlides (Velmex Inc., Bloomfield, NY, USA) that can position the tracer in the 3D space (X, Y, and Z) with a reported accuracy of 76 μm), and the generated orientation data is two-dimensional (see Page 7967, Section III, where the rotational stage is capable of orienting the tracer along its pitch and roll thanks to a pulley/belt system driven by two stepper motors in half-step mode, resulting in an accuracy of 0.9◦).
Ghovanloo and Sebkhi et al. (2021) are both considered to be analogous to the claimed invention because they are in the same field of silent speech interfaces and tongue tracking analysis. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Ghovanloo to incorporate the teachings of Sebkhi et al. (2021) to include the sensor comprising an inertial measurement unit (IMU), wherein the position data is three-dimensional and the generated orientation data is two-dimensional in order to measure the magnetic field over its position and accurately track the position and orientation of the tongue’s tracer while also improving portability (see Page 7970, Section V, where an IMU is used as the tracer and tracked inside a local magnetic field generated by a magnet strip. This allows the system to not only be wearable but also more practical for tongue tracking applications since the magnet strip can be hidden from view by being placed further from the user’s face. The tracer is designed to be as unobtrusive as possible with a current size of 6 ×6×0.8 mm³, and with the potential to be further reduced. Our novel tracking method builds upon our previous work by using the IMU’s magnetometer to measure the varying magnetic field over its position. These magnetic measurements are then compensated for the tracer’s orientation and fed into a magnetic localization algorithm).
Regarding claim 17 which depends from claim 11 and recites a method, this claim is rejected as unpatentable over the same combination of prior art applied against claim 6 and claim 7 (i.e., Ghovanloo in view of Sebkhi (2021)). Ghovanloo teaches all the limitations of claim 11 as noted above. As detailed in the rejection of claim 6 and claim 7, the disclosed system teaches or renders obvious each step of the method recited in claim 17. Accordingly, claim 17 is rejected for the same reasons set forth in the rejection of claim 6 and claim 7.
Regarding claim 18 which depends from claim 11 and recites a method, this claim is rejected as unpatentable over the same combination of prior art applied against claim 8 (i.e., Ghovanloo in view of Sebkhi (2021)). Ghovanloo teaches all the limitations of claim 11 as noted above. As detailed in the rejection of claim 8, the disclosed system teaches or renders obvious each step of the method recited in claim 8. Accordingly, claim 18 is rejected for the same reasons set forth in the rejection of claim 8.
Regarding claim 19, Sebkhi et al. (2021) teaches all the limitations in claim 18 but fails to teach wherein the sensor is further configured to detect three-dimensional (3D) magnetic signals based on detected variations in a local magnetic field corresponding to movement of the tongue, the method further comprising: converting the 3D magnetic signals into additional 3D position information and additional 2D orientation information; and augmenting the position data and the orientation data with the additional 3D position information and the additional 2D orientation information for generating the synthesizable sound or speech data.
However, Ghovanloo does teach all the limitations in claim 11 and also does teach wherein the sensor is further configured to detect three-dimensional (3D) magnetic field signals based on detected variations in a local magnetic field corresponding to movement of the tongue (see [0025], where the sensor can comprise an array of magnetic field sensors or magnetometers that are capable of detecting 3D magnetic fields. Also see [0149], where the sensors 12 convert the magnetic field intensity (B) from the magnet to a proportional analog output voltage), wherein the instructions further cause the system to:
convert the 3D magnetic field signals into additional 3D position information and additional 2D orientation information (see [0089], where to find the position and orientation of the tracer unit, the TTS signal processing algorithm can minimize a cost function that correlates with the difference between the measured, B, and the estimated, B (sᵢ, a, m), magnetic flux densities at the location of each magnetometer in the plurality of sensors using the following equation:
f
a
,
m
=
∑
i
=
1
N
B
ᵢ
,
m
e
a
s
-
B
(
s
ᵢ
,
a
,
m
)
where
a
=
<
a
x
,
a
y
,
a
z
>
is the magnet’s location,
m
=
<
m
sin
θ
cos
Φ
,
m
sin
θ
sin
Φ
,
m
cos
θ
>
is the tracer's orientation and
s
i
=
<
s
i
x
,
s
i
y
,
s
i
z
>
is the location of the magnetic sensor); and
augment the position data and the orientation data with the additional 3D position information and the additional 2D orientation information (see [0090], where the magnetic sensor data can be fused with additional sensor data, such as for example and not limitation, acoustic, video, touch, pressure, and flow data, in the post-processing stage of embodiments of the system, thus creating a comprehensive analytical tool for speech and language therapy and research. Also see [0109], where the system and apparatus can comprise a computing platform receiving the sensor signal from the sensor control unit, translating and processing the sensor signal to obtain tracking information from the tracer unit, generating, for example and not limitation, a graphical and/or audiovisual representation of the tracking information, and displaying the graphical and/or audiovisual representation on the graphical interface of the computing platform or a graphical display for review by the SLP, clinician, or tutor. In some embodiments, the graphical and/or audiovisual representation can comprise, for example and not limitation, a tracing, line drawing, or 3D drawing of the tongue movement and position based on the movement of the tracer unit during speech. … The signal processing constitutes of multimodal sensor data fusion, machine learning, and pattern recognition methods. Also see [0128], where algorithm 111 can localize the magnetic tracer 1 within the 3D oral space and can fuse it with additional sensor data, which can be, for example and not limitation, from a camera 6 and/or microphone 7) for generating the synthesizable sound or speech data (see [0004], where the Tongue Tracking System can also be used as a silent speech interface by using the acoustic-kinematic recordings of a subject who may have a voice disorder, for example, to automatically build, index, and reproduce speech from a database of phonemes, words, phrases, and commands based on that user's specific acoustic-kinematic recordings).
Sebkhi et al. (2021) and Ghovanloo are both considered to be analogous to the claimed invention because they are in the same field of silent speech interfaces and tongue tracking analysis. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Sebkhi et al. (2021) to incorporate the teachings of Ghovanloo to include the sensor which is further configured to detect three-dimensional (3D) magnetic signals based on detected variations in a local magnetic field corresponding to movement of the tongue, the method further comprising: converting the 3D magnetic signals into additional 3D position information and additional 2D orientation information; and augmenting the position data and the orientation data with the additional 3D position information and the additional 2D orientation information for generating the synthesizable sound or speech data in order to create a comprehensive tool capable of analyzing speech and language therapy (see [0090], where the combination of the magnetic sensors with the surface electrode array and pressure sensors can form a wireless hybrid Tongue Tracking System and electropalatography system. The magnetic sensor data can be fused with additional sensor data, such as for example and not limitation, acoustic, video, touch, pressure, and flow data, in the post-processing stage of embodiments of the system, thus creating a comprehensive analytical tool for speech and language therapy and research).
Claim(s) 14 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ghovanloo in view of Cao et al. (2022). “Speaker Adaptation on Articulation and Acoustics for Articulation-to-Speech Synthesis.” Sensors (Basel, Switzerland) [Switzerland], vol. 22, no. 16, pp. 6056.
Regarding claim 14, Ghovanloo teaches all the limitations in claim 11 but fails to teach wherein the output synthesizable sound or speech data is generated using a model that is trained using recorded speech obtained from the user and/or one or more other target individuals.
However, Cao et al. (2022) does teach wherein the output synthesizable sound or speech data is generated using a model that is trained using recorded speech obtained from the user and/or one or more other target individuals (see Page 1, Abstract, where current ATS studies focus on speaker-dependent (SD) models to avoid large variations of articulatory patterns and acoustic features across speakers. However, these designs are limited by the small data size from individual speakers. Speaker adaptation designs that include multiple speakers’ data have the potential to address the issue of limited data size from single speakers. We used Procrustes matching and voice conversion for articulation and voice adaptation, respectively. Also see Page 2, Introduction, where the speaker-dependent ATS (SD-ATS) is where training and testing data are from the same speakers; Speaker-independent ATS (SI-ATS) is where training and testing data are from different speakers; Speaker-adaptive ATS (SA-ATS) is where training data are from other speakers and the target speaker).
Ghovanloo and Cao et al. (2022) are both considered to be analogous to the claimed invention because they are in the same field of silent speech interfaces and tongue tracking analysis. Therefore, it would have been obvious to someone of ordinary skill in the art before the effective filing date of the claimed invention to have modified Ghovanloo to incorporate the teachings of Cao et al. (2022) to include the output synthesizable sound or speech data is generated using a model that is trained using recorded speech obtained from the user and/or one or more other target individuals in order to make the synthesized speech sound more natural and more customizable for those who have lost their ability to speak (see Page 1, Abstract, where this technology has the potential to recover the speech ability of individuals who have lost their voice but can still articulate (e.g., laryngectomees). Also see Page 14, Conclusions, where a framework of using voice conversion for ATS voice adaptation was proposed and validated, in which voice conversion (VC) models were trained for reducing the acoustic variations between training and testing speakers. The experimental results have shown the effectiveness of both Procrustes matching and voice conversion; the performance was further improved when both were used in conjunction).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JOHN HONG FANG-WU whose telephone number is (571)270-0607. The examiner can normally be reached Monday - Friday, 8AM to 5PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Paras Shah can be reached at (571)-270-1650. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JOHN HONG FANG-WU/Examiner, Art Unit 2653
/Paras D Shah/Supervisory Patent Examiner, Art Unit 2653
06/01/2026
Prosecution Insights