DETAILED ACTION
This office action is a First Action on the Merits (FAOM) for the claim set entered on 04/30/2025. Claims 1-18 are pending and have been considered. The examiner would like to note that the claims are eligible under 35 U.S.C. 101 due to the independent claims containing subject matter which cannot be reasonably performed in the mind with or without the aid of pen and paper, namely, separating encoded feature sequences into separated feature sequences. See “a speaker separation unit…” of claims 1/13 and “providing, by a speaker separation unit…” of claims 17/18 . One will not be able to meaningfully separate encoded, mixed speech into separate feature sequences (Step 2A, Prong 1, NO). Further, should this be interpreted to be a standard transformer operation, the examiner asserts that this element improves the functioning of speech separation in a computing environment as disclosed in [0007] of the instant app, “…a speech separation device capable of not only improving system performance but also reducing complexity by providing a plurality of separated feature sequences, each separated for a plurality of speakers, by using a speaker separation unit disposed between and encoder and a decoder.” (Step 2A, Prong 2, YES).
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Acknowledgment is made of applicant’s claim for foreign priority under 35 U.S.C. 119
(a)-(d). The certified copy has been filed for the parent Application No. KR10-2024-0076723, filed on 06/13/2024.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are: “a speaker separation unit [configured to]…” in claims [1, 13], 17, 18, “a feature compression unit” in claim 2, “a convolution unit configured to…” in claim 3, “a feature extension unit” (wherein “the feature extension unit is configured to…” in claim 12) and “an upsampling unit configured to…” in claim 7, “a feature fusion unit configured to…” in claim 8, “a loss calculation unit configured to…” in claim 14, “an auxiliary extension unit” in claim 15.
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
Specifically, Applicant’s specification provides no definition as to how a “unit” is to be interpreted. Figs. 6-7 are described to be showing “a speaker separation unit included in the speech separation device”. Fig. 6 shows convolution of input with no physical components. Fig. 7 shows the speaker separation unit as a black box which takes input and results in output. Fig. 14 is described to be describing the loss calculation unit. Again, both the loss calculation unit 600 and auxiliary loss calculation units are shown as black boxes which take input and result in output. Fig. 15 is described to be showing an auxiliary signal and auxiliary loss calculation unit of Fig. 14. Again, the units are merely described to be black boxes capable of performing the claimed functions. Fig. 16 describes the auxiliary signals and auxiliary loss calculation unit of Fig. 15, Again, the units are merely described to be black boxes capable of performing the claimed functions. This can be extended to all units (feature compression unit of Fig. 2, convolution unit of Fig. 3, etc. As there is no structure associated with performing the operations of the units, they are being interpreted to be software per se. The only structure provided in the application is with reference to the encoder and decoder, software components. The speaker separation unit is not defined beyond the “unit” which, as previously disclosed, is also being interpreted to be software.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 1-18 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention.
Claims 1-18 are also rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the enablement requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to enable one skilled in the art to which it pertains, or with which it is most nearly connected, to make and/or use the invention.
Such claim limitation(s) is/are: “a speaker separation unit [configured to]…” in claims [1, 13], 17, 18 ([Fig. 1, 200]), “a feature compression unit [110]” in claim 2 ([Fig. 2, 110]), “a convolution unit configured to…” in claim 3 ([Fig. 3, 122]), “a feature extension unit” (wherein “the feature extension unit is configured to…” in claim 12), ([Fig. 8, 330]) and “an upsampling unit configured to…” in claim 7 ([Fig. 9, 311]), “a feature fusion unit configured to…” in claim 8 ([Fig. 9, 312]), “a loss calculation unit configured to…” in claim 14 ([Fig. 14, 600]), and “an auxiliary extension unit” in claim 15 ([Fig. 16, 711]).
As to the written description requirement, MPEP 2161.01 I. details determining whether there is adequate written description for a computer-implemented functional claim limitation, noting:
Similarly, original claims may lack written description when the claims define the invention in functional language specifying a desired result but the specification does not sufficiently describe how the function is performed or the result is achieved. For software, this can occur when the algorithm or steps/procedure for performing the computer function are not explained at all or are not explained in sufficient detail (simply restating the function recited in the claim is not necessarily sufficient). In other words, the algorithm or steps/procedure taken to perform the function must be described with sufficient detail so that one of ordinary skill in the art would understand how the inventor intended the function to be performed. See MPEP §§ 2163.02 and 2181, subsection IV.
Further as to the written description requirement, MPEP 2163.03 V details written description circumstances arising from original claims not sufficiently described, noting:
An original claim may lack written description support when (1) the claim defines the invention in functional language specifying a desired result but the disclosure fails to sufficiently identify how the function is performed or the result is achieved or (2) a broad genus claim is presented but the disclosure only describes a narrow species with no evidence that the genus is contemplated. See Ariad Pharms., Inc. v. Eli Lilly & Co., 598 F.3d 1336, 1349-50 (Fed. Cir. 2010) (en banc). The written description requirement is not necessarily met when the claim language appears in ipsis verbis in the specification. "Even if a claim is supported by the specification, the language of the specification, to the extent possible, must describe the claimed invention so that one skilled in the art can recognize what is claimed. The appearance of mere indistinct words in a specification or a claim, even an original claim, does not necessarily satisfy that requirement. "Enzo Biochem, Inc. v. Gen-Probe, Inc., 323 F.3d 956, 968, 63 USPQ2d 1609, 1616 (Fed. Cir. 2002).
Further as to the written description requirement, MPEP 2163.03 VI details written description circumstances arising from indefiniteness of a means plus function limitation, noting:
A claim limitation expressed in means- (or step-) plus-function language "shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof." 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. If the specification fails to disclose sufficient corresponding structure. materials. or acts that perform the entire claimed function. then the claim limitation is indefinite because the applicant has in effect failed to particularly point out and distinctly claim the invention as required by 35 U.S.C. 112(b) or pre-A/A 35 U.S.C. 112, second paragraph. In re Donaldson Co., 16 F.3d 1189, 1195, 29 USPQ2d 1845, 1850(Fed. Cir. 1994) (en banc). Such a limitation also lacks an adequate written description as required by 35 U.S. C. 112(a) or pre-A/A 35 U.S. C. 112. first
paragraph, because an indefinite, unbounded functional limitation would cover all ways of performing a function and indicate that the inventor has not provided sufficient disclosure to show possession of the invention. See also MPEP § 2181.
As to the enablement requirement, MPEP 2163.02 details examples of enablement issues in computer programing cases, in particular in section II. Regarding block elements within a computer, noting:
While no specific universally applicable rule exists for recognizing an insufficiently disclosed application involving computer programs, an examining guideline to generally follow is to challenge the sufficiency of disclosures that fail to include the programmed steps. algorithms or procedures that the computer performs necessary to produce the claimed function. These can be described in any way that would be understood by one of ordinary skill in the art, such as with a reasonably detailed flowchart which delineates the sequence of operations the program must perform. In programming applications where the software disclosure only includes a flowchart, as the complexity of functions and the generality of the individual components of the flowchart increase, the basis for challenging the sufficiency of such a flowchart becomes more reasonable because the likelihood of more than routine experimentation being required to generate a working program from such a flowchart also increases.
The claimed units are illustrated by “a speaker separation unit [configured to]…” in claims [1, 13], 17, 18 ([Fig. 1, 200]), “a feature compression unit [110]” in claim 2 ([Fig. 2, 110]), “a convolution unit configured to…” in claim 3 ([Fig. 3, 122]), “a feature extension unit” (wherein “the feature extension unit is configured to…” in claim 12), ([Fig. 8, 330]) and “an upsampling unit configured to…” in claim 7 ([Fig. 9, 311]), “a feature fusion unit configured to…” in claim 8 ([Fig. 9, 312]), “a loss calculation unit configured to…” in claim 14 ([Fig. 14, 600]), and “an auxiliary extension unit” in claim 15 ([Fig. 16, 711]). Fig. 6 shows convolution of input with no physical components. Fig. 7 shows the speaker separation unit as a black box which takes input and results in output. [0009], [0034] of the spec discloses “The speaker separation unit may provide a plurality of separated feature sequences by separating the encoded feature sequence for each of a plurality of speakers included in the speech signal”, further described in [0067]-[0068]. This does not describe how the separation is performed. Fig. 14 is described to be describing the loss calculation unit. Again, both the loss calculation unit 600 and auxiliary loss calculation units are shown as black boxes which take input and result in output. Fig. 15 is described to be showing an auxiliary signal and auxiliary loss calculation unit of Fig. 14. Again, the units are merely described to be black boxes capable of performing the claimed functions. Fig. 16 describes the auxiliary signals and auxiliary loss calculation unit of Fig. 15, Again, the units are merely described to be black boxes capable of performing the claimed functions. Further, flowcharts are provided in Figs. 17-18. The speaker separation steps S200 are performed by “separating encoded feature sequence and global-local encoding sequence for each stage for each of plurality of speakers included in speech signal”. This does not explain how the separation is performed, merely what components are input and output. This can be extended to all units (feature compression unit of Fig. 2, convolution unit of Fig. 3, etc.). The specification does not provide details as to how any of the units function. Further, there is no physical structure recited anywhere in the instant application; therefore, the units can be reasonably assumed to be representing software implementations. In view of this, there is no flow chart, nor program code defining the operations of these units as required for software implementation. The step of “separating encoded feature sequence and global-local encoding sequence” does not describe how the separation is performed. Further, it is unreasonable to assume the separation step is generic as Applicant claims this to be related to the inventive concept, see [0007] of instant app: “An aspect of the present disclosure may provide a speech separation device capable of not only improving system performance but also reducing system complexity by providing a plurality of separated feature sequences, each separated for a plurality of speakers, by using a speaker separation unit disposed between an encoder and a decoder”.
The specification only discloses the software, algorithm, and/or flow chart of these operations of the modules/units at a high level, and does not provide the requisite level of detail (e.g. code/algorithm) necessary that would indicate to one of ordinary skill in the art:
1) That the inventor(s) at the time the application was filed, had possession of the claimed invention, or
2) How to make or use the invention without undue experimentation
All claims dependent upon rejected base claims are also rejected for failing to meet the written description and enablement requirements.
In sum, claims 1-18 fail to meet the enablement requirement of 35 U.S.C. 112(a). The lack of disclosure of the code/algorithm to implement the speech separation unit (and/or any other claimed units) (as described above) in a manner understandable to a person of ordinary skill in the art results in claimed subject matter which was not described in the specification in such a way as to enable one skilled in the art to which it pertains, or with which it is most nearly connected, to make and/or use the invention.
Further, claims 1-18 fail to meet the written description requirement of 35 U.S.C. 112(a). The lack of disclosure of the code/algorithms to implement the claimed speech separation unit (and/or any other claimed units, as detailed above) in a manner understandable to a person of ordinary skill in the art results in a failure to reasonably convey that the inventor(s) at the time the application was filed, had possession of the claimed invention.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-18 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim limitations “speaker separation unit”, “feature compression unit”, “convolution unit”, “feature extension unit”, “upsampling unit”, “feature fusion unit”, “loss calculation unit”, and “auxiliary feature extension unit” in claims 1-8, 12-15, 17-18 invokes 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. However, the written description fails to disclose the corresponding structure, material, or acts for performing the entire claimed function and to clearly link the structure, material, or acts to the function. Claims 1-8, 12-15, 17-18 recite a “speaker separation unit”, “feature compression unit”, “convolution unit”, “feature extension unit”, “upsampling unit”, “feature fusion unit”, “loss calculation unit”, and “auxiliary feature extension unit”, interpreted under 112(f) as noted above. The claimed units are illustrated by “a speaker separation unit” in claims 1, 13, 17, 18 ([Fig. 1, 200]), “a feature compression unit [110]” in claim 2 ([Fig. 2, 110]), “a convolution unit configured to…” in claim 3 ([Fig. 3, 122]), “a feature extension unit” (wherein “the feature extension unit is configured to…” in claim 12), ([Fig. 8, 330]) and “an upsampling unit configured to…” in claim 7 ([Fig. 9, 311]), “a feature fusion unit configured to…” in claim 8 ([Fig. 9, 312]), “a loss calculation unit configured to…” in claim 14 ([Fig. 14, 600]), and “an auxiliary extension unit” in claim 15 ([Fig. 16, 711]). Therefore, the claim is indefinite and is rejected under 35 U.S.C. 112(b) or pre-AIA 35 U.S.C. 112, second paragraph.
Applicant may:
(a) Amend the claim so that the claim limitation will no longer be interpreted as a limitation under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph;
(b) Amend the written description of the specification such that it expressly recites what structure, material, or acts perform the entire claimed function, without introducing any new matter (35 U.S.C. 132(a)); or
(c) Amend the written description of the specification such that it clearly links the structure, material, or acts disclosed therein to the function recited in the claim, without introducing any new matter (35 U.S.C. 132(a)).
If applicant is of the opinion that the written description of the specification already implicitly or inherently discloses the corresponding structure, material, or acts and clearly links them to the function so that one of ordinary skill in the art would recognize what structure, material, or acts perform the claimed function, applicant should clarify the record by either:
(a) Amending the written description of the specification such that it expressly recites the corresponding structure, material, or acts for performing the claimed function and clearly links or associates the structure, material, or acts to the claimed function, without introducing any new matter (35 U.S.C. 132(a)); or
(b) Stating on the record what the corresponding structure, material, or acts, which are implicitly or inherently set forth in the written description of the specification, perform the claimed function. For more information, see 37 CFR 1.75(d) and MPEP §§ 608.01(o) and 2181.
All claims dependent upon rejected base claims are also rejected; therefore, claims 9-11 and 16 are also rejected under 35 U.S.C. 112(b) as being indefinite.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1-18 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Anonymous (“Separate and Reconstruct: Asymmetric Encoder-Decoder for Speech Separation”), hereinafter Anon, published on 6/11/2024 to Github, located using the WaybackMachine (https://web.archive.org/web/20240613054858/https:/fordemopage.github.io/SepReformer/). The examiner notes that, though the date on the snapshot appears to be 06/13/2024, it is clear that there is a version/snapshot of the document originally posted on 06/11/2024 as can be seen on the WaybackMachine archive. The examiner would like to note that the “anonymous” source appears to be the same document as that published by the inventors after the EFD of the instant app (in addition to two extra authors on the published demo). Both the anonymous and deanonymized versions of this publication will be attached. The examiner believes this art can be disqualified as an exception, under 102(b)(1)(A), should Applicant prove that the anonymous version is from themselves and/or that the other authors are not contributing inventors; therefore, the examiner has also included additional rejections under 35 U.S.C. 103 using art which is from different authors and is not exception-eligible.
Regarding claim 1, Anon discloses: a speech separation device ([Title, “…Speech Separation”]) comprising:
a separation encoder configured to provide an encoded feature sequence by downsampling an input representation generated based on a speech signal ([Third image, Separation Encoder featuring Down Conv. layers based on input speech signal X, an input representation after being passed through the input layer]);
a speaker separation unit configured to provide a plurality of separated feature sequences by separating the encoded feature sequence for each of a plurality of speakers included in the speech signal ([Third image, Spk. Split], [pg. 3, Early Split] The encoded feature sequence is split early in the network, corresponding to the number of speakers, which allows the network to handle each speaker's features separately from an early stage); and
a reconstruction decoder configured to provide an output representation for each speaker by upsampling the separated feature sequence ([pg. 4, Decoder] Decoder: Reconstructs the separated speech signals from the encoded features, [Third image, Reconstruction Decoder resulting in output Y, wherein Upsample is performed using Concat + Linear layers]).
Regarding claim 2, Anon discloses: the device of claim 1.
Anon further discloses:
wherein the separation encoder includes a feature compression unit and a plurality of encoding stages ([Third image, Down Conv. in a Separation Encoder which results in smaller samples of speech (as seen with window length T below structure) indicates the down convolution to be a feature compression]), and
each of the plurality of encoding stages further includes a global-local transformer configured to provide a global-local encoding sequence generated based on all components included in an encoding input sequence input into each of the encoding stages and components included in a preset region corresponding to a predetermined region ([Third image, Global/Local transformer pairs in every step/stage of encoding]).
Regarding claim 3, Anon discloses: the device of claim 2.
Anon further discloses:
wherein each of the encoding stages further includes a convolution unit configured to downsample an output of the global-local encoding sequence ([Third image, Down Conv. performed after global-local transform layers], [The examiner assert that down convolution and downsampling are synonymous terms in the art]).
Regarding claim 4, Anon discloses: the device of claim 3,
Anon further discloses:
wherein the encoding input sequence of a first encoding stage among the plurality of encoding stages is an input feature sequence provided by the feature compression unit ([Third image, output from Down Conv. passed into global-local transformers], [Sending output from the down convolution, i.e. compression, into the encoding stages indicates a compressed, first encoding input sequence]).
Regarding claim 5, Anon discloses: the device of claim 4.
Anon further discloses:
wherein the feature compression unit outputs the input feature sequence based on the input representation ([Third image, output from Down Conv. based on input which has been passed through an Input Layer and Global-Local transformers], [Wherein the input representation is that output from the input layer and/or first global-local transform]).
Regarding claim 6, Anon discloses: the device of claim 5.
Anon further discloses:
wherein the speaker separation unit provides separated skip connection sequences to a corresponding decoding stage based on the global-local encoding sequence ([Third image, single line input to Spk. Split skipping down conv., double line output from Spk. Split skipping cross-spk. transformer], [Wherein a skip connection in each of the encoding/decoding stages suggests a correspondence between the skip connections]).
Regarding claim 7, Anon discloses: the device of claim 6.
Anon further discloses:
wherein the decoder includes the plurality of decoding stages ([Third image, Siam Global-Local Transformers], [Wherein the image suggests at least a second decoding stage following the first labeled stage]) and a feature extension unit ([Third image, Upsample], [The examiner asserts that upsampling tracks to feature extension as seen with the window times/lengths T increasing after each decoding stage]), and
each of the plurality of decoding stages further includes an upsampling unit configured to provide an upsampled sequence by upsampling each of a plurality of decoded input sequences input into each of the decoding stages ([Third image, Concat. + Linear layer], [Considering the window T to be increasing after each decoding stage, this suggests the concatenation and linear transform layer to be an upsampling unit]).
Regarding claim 8, Anon discloses: the device of claim 7.
wherein each of the plurality of decoding stages further includes a feature fusion unit configured to provide a plurality of fused feature sequences based on the respective upsampled sequences and the separated skip connection provided by the speaker separation unit ([Third image, Concat. + Linear layer], [Wherein it can be seen that this layer receives both input from a decoding stage and input from skip connections of an encoding stage resulting in one output, suggestion the fusion of the two inputs to result in one output, wherein it can be seen that this occurs after “Upsample” in the image]).
Regarding claim 9, Anon discloses the device of claim 8.
Anon further discloses:
wherein each of the plurality of decoding stages includes a plurality of Siamese global-local transformers each configured to provide an output of a global-local decoded sequence generated based on each fused feature sequence ([Third image, Siam Global-Local Transformers which receive an output from the Concat. + Linear (fusing) layer]).
Regarding claim 10, Anon discloses: the device of claim 9.
Anon further discloses:
wherein each of the decoding stages further includes a cross-reconstruction transformer configured to provide a reconstructed decoded sequence by extracting feature information among the speakers based on an output of a decoding transformer of each of the Siamese global-local transformers ([Third image, Cross-Spk. Transformer], [pg. 4, Cross-Speaker (CS) Module] This module within the decoder allows interaction between features of different speakers, enhancing the network's ability to separate and reconstruct overlapping speech elements, [Wherein it can be seen in the third image that the cross-spk. layer receives output from the Siam global-local transformers]).
Regarding claim 11, Anon discloses: the device of claim 10.
Anon further discloses:
wherein among the plurality of decoding stages, the plurality of decoded input sequences of a first decoding stage are the separated feature sequences ([Third image, output from Spk. Split input to Reconstruction Decoder], [Wherein inputting a speaker split output into a decoder indicates the input of a first decoding stage to be separated input]).
Regarding claim 12, Anon discloses: the device of claim 11.
Anon further discloses:
wherein the feature extension unit is configured to provide the output representation generated based on the plurality of decoded feature sequences provided from an N-th decoding stage among the plurality of decoding stages ([Third image, Concat. + Linear layer into Output layer], [The output from a final decoding stage (containing feature extension) indicates a provided output representation to the output layer, wherein the decoder suggests at least two decoding stages]).
Regarding claim 13, Anon discloses: a speech separation system ([Title, “…Speech Separation”]), comprising:
an audio encoder configured to provide an input representation based on a mixed speech signal ([Second image, Proposed separator, Encoder]);
a separation encoder configured to provide an encoded feature sequence by downsampling the input representation ([Third image, Separation Encoder featuring Down Conv. layers based on input speech signal X, an input representation after being passed through the input layer]);
a speaker separation unit configured to provide a plurality of separated feature sequences by separating the encoded feature sequence for each of a plurality of speakers included in the speech signal ([Third image, Spk. Split], [pg. 3, Early Split] The encoded feature sequence is split early in the network, corresponding to the number of speakers, which allows the network to handle each speaker's features separately from an early stage);
a decoder configured to provide an output representation for each speaker by upsampling the separated feature sequence ([pg. 4, Decoder] Decoder: Reconstructs the separated speech signals from the encoded features, [Third image, Reconstruction Decoder resulting in output Y, wherein Upsample is performed using Concat + Linear layers]); and
an audio decoder configured to provide a speech signal for each speaker based on the output representation ([Second image, Proposed separator, Decoder]).
Regarding claim 14, Anon discloses: the system of claim 13.
Anon further discloses:
further comprising a loss calculation unit configured to calculate a loss value and an auxiliary loss value based on the sequences provided from the output representation and a plurality of decoding stages ([pg. 4, Multi-Loss Training] The network is trained with a multi-loss objective, where intermediate layers are optimized to progressively refine the separated outputs, [A multi-loss training, wherein each loss is corresponding to one separated speaker (as suggested by discriminative training in the multi-loss section) indicates at least an auxiliary loss in addition to a loss based on the first and second speakers]).
Regarding claim 15, Anon discloses: the system of claim 14.
Anon further discloses:
wherein the loss calculation unit further includes an auxiliary feature extension unit and an auxiliary audio decoder each configured to provide an auxiliary signal to produce the auxiliary loss value based on the sequences provided from the decoding stages ([As previously disclosed in the multi-loss training, optimizing separated output (indicating separated output is what the loss is performed on), wherein the separation process includes feature extension units and decoders as seen in the third image, suggests at least an auxiliary feature extension unit and decoder for the second speaker as compared to the first speaker]).
Regarding claim 16, Anon discloses: the system of claim 15.
Anon further discloses:
wherein a parameter (weight) applied to the speech separation system is adjusted based on the loss value and the auxiliary loss value ([As previously disclosed in the multi-loss training, optimizing intermediate layers based on a multi-loss objective suggests modification of weights of said layers to result in said optimization]).
Regarding claim 17, Anon discloses: A method for operating a speech separation device ([Title, “…Speech Separation”]), the method comprising:
providing, by a separation encoder ([Third image, Separation Encoder]), an encoded feature sequence and a global-local encoding sequence for each encoding stage by downsampling an input representation generated based on a mixed speech signal ([Third image, Output from Global-Local transformers skipping Down Conv. (encoded feature sequence), output from Down Conv. layers of addition decoding stages (global-local encoding sequence)], [Wherein Down Conv. tracks to a downsampling an input representation]);
providing, by a speaker separation unit ([Third image, Spk. Split module]), a plurality of separated feature sequences ([Third image, output Spk. Split into Upsample]) and separated skip connections ([Third image, output from Spk. Split into second upsample, skipping first decoding stage]) by separating the encoded feature sequence and the global-local encoding sequences for each stage for each of a plurality of speakers included in the speech signal ([Passing encoded signals into a speaker split module suggests the outputs from said module to be separated for each of a plurality of speakers in the speech signal]); and
providing, by a reconstruction decoder ([Third image, Reconstruction Decoder resulting in output Y, wherein Upsample is performed using Concat + Linear layers]), an output representation for each speaker by upsampling and fusing the separated feature sequence and the separated skip connection ([pg. 4, Decoder] Decoder: Reconstructs the separated speech signals from the encoded features, [The examiner asserts that two inputs, wherein one is a separated feature sequence and the other is a separated skip connection, to a Concat. + Linear layer resulting in one output indicates fusion of the two inputs to result in the output]).
Regarding claim 18, Anon discloses: a method for operating a speech separation system, the method comprising:
providing, by an audio encoder, an input representation based on a mixed speech signal ([Second image, Proposed separator, Encoder]);
providing, by a separation encoder ([Third image, Separation Encoder]), an encoded feature sequence and a global-local encoding sequence for each encoding stage by downsampling an input representation generated based on the mixed speech signal ([Third image, Output from Global-Local transformers skipping Down Conv. (encoded feature sequence), output from Down Conv. layers of addition decoding stages (global-local encoding sequence)], [Wherein Down Conv. tracks to a downsampling an input representation]);
providing, by a speaker separation unit ([Third image, Spk. Split module]), a plurality of separated feature sequences ([Third image, output Spk. Split into Upsample]) and separated skip connections ([Third image, output from Spk. Split into second upsample, skipping first decoding stage]) by separating the encoded feature sequence and the global-local encoding sequences for each stage for each of a plurality of speakers included in the speech signal ([Passing encoded signals into a speaker split module suggests the outputs from said module to be separated for each of a plurality of speakers in the speech signal]); and
providing, by a reconstruction decoder ([Third image, Reconstruction Decoder resulting in output Y, wherein Upsample is performed using Concat + Linear layers]), an output representation for each speaker by upsampling and fusing the separated feature sequence and the separated skip connection ([pg. 4, Decoder] Decoder: Reconstructs the separated speech signals from the encoded features, [The examiner asserts that two inputs, wherein one is a separated feature sequence and the other is a separated skip connection, to a Concat. + Linear layer resulting in one output indicates fusion of the two inputs to result in the output]); and providing, by an audio decoder, a speech signal separated for each speaker based on the output representation for each speaker ([Second image, Proposed separator, Decoder]).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-8, 13-18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kim et al. (US-20210120355-A1), hereinafter Kim, in view of Wang et al. (“Time-domain adaptive attention network for single-channel speech separation”), hereinafter Wang.
Regarding claim 1, Kim discloses a speech separation device ([Title, “audio source separation”]) comprising:
a separation encoder configured to provide an encoded feature sequence by downsampling an input representation generated based on a speech signal ([Fig. 1, down-transition layer 220], [0061] In the encoding process, the dense block 100 and the down-transition layer 220 may be repeatedly executed to extract a feature of a signal in which only music is separated, [0082] In this case, the audio signal may be a mixed signal of a mono sound source, that is, a signal including voice, music, and noise, [Wherein a “down-transition layer” tracks to a downsampling. The examiner asserts that an audio signal comprised of voice is a speech signal]).
Kim does not disclose:
a speaker separation unit configured to provide a plurality of separated feature sequences by separating the encoded feature sequence for each of a plurality of speakers included in the speech signal.
Wang discloses:
a speaker separation unit configured to provide a plurality of separated feature sequences by separating the encoded feature sequence for each of a plurality of speakers included in the speech signal ([Fig. 1, Separator], [Fig. 1, explanation following figure label/title] Firstly, the encoder converts the speech signal into a feature representation, and then the feature separation and mask estimation are performed by the separator, [In view of the encoding units containing down-sampling operations in Kim. Feature separation and mask estimation which results in separated speech to be input to the decoder (as indicated by the multiple inputs to the decoder) indicates a provision to the decoder of separated feature sequences, i.e. masked feature spectra]).
Kim and Wang are considered analogous art within speaker separation. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Kim to incorporate the teachings of Wang, because of the novel way to construct a single-channel speech separation model which uses local and global attention modules to model the speech sequence and combines the local and global information in the sequence coding, improving the performance of speech separation ([Wang, pg. 2, Main Contribution 1 (bullet point 1)]).
Kim further discloses:
a reconstruction decoder configured to provide an output representation for each speaker by upsampling the separated feature sequence ([Fig. 1, up-transition layer 230], [Fig. 6, S750 “output a plurality of signals”], [0084] the encoded signal may be decoded using at least one second dense block and at least one up-transition layer (S730), [In view of the plurality of output speaker signals of Wang, wherein the outputting step of Kim (S750) is performed after decoding (S730) indicating an output representation for up-sampled, separated feature sequences, i.e. maps]).
Regarding claim 2, Kim in view of Wang discloses: the device of claim 1.
Kim further discloses:
wherein the separation encoder includes a feature compression unit and a plurality of encoding stages ([Fig. 1, dense blocks 100-1 to 100-3, down-transition layers 220], [0061] In the encoding process, the dense block 100 and the down-transition layer 220 may be repeatedly executed, [Wherein a down-transition layer tracks to a compression unit as down sampling compresses an overall audio signal into discrete samples. Further, wherein the plurality of dense blocks used for encoding indicates a plurality of encoding stages]), and
Wang further discloses:
each of the plurality of encoding stages further includes a global-local transformer ([Fig. 2, Adaptive Attention Network (AAN) containing Local Attention and Global Attention]) configured to
provide a global-local encoding sequence generated based on all components included in an encoding input sequence input ([Fig. 2, Encoding and Segmentation output into AAN], [Providing an encoding sequence to the AAN, in view of the down-sampled encoded sequence of Kim which contains a plurality of encoding steps indicates a global-local encoding sequence generated based on all components in an encoding input sequence]) into each of the encoding stages and components ([Fig. 2, Convolving output from the AAN after Overlap-Add with the output from the Encoding and Segmentation]) included in a preset region corresponding to a predetermined region ([Wherein the preset region corresponding to a predetermined region is the entirety of output from the encoding and segmentation module. It is unclear to the examiner what a “preset region corresponding to a predetermined region” is related to when the operation being performed is an encoding sequence generation. The examiner is interpreting the region to be a portion of a signal, wherein that portion can be the entirety]).
Regarding claim 3, Kim in view of Wang discloses: the device of claim 2.
Kim further discloses:
wherein each of the encoding stages further includes a convolution unit configured to downsample an output of the global-local encoding sequence ([Fig. 1, Dense Block 100-1 to 100-3 and down-transition layer 220], [0061] the dense block 100 and the down-transition layer 220 may be repeatedly executed, [0065] The dense block may be composed of k initial feature maps and L convolution layers, [0066] The dense block is a block that can be used in…the down-sampling process, [0071] down-transition layer, 1×1 convolution (S510) may be performed to reduce the number of feature maps 51 increased through the processing by the dense block, [Wherein each iteration of dense blocks/down sampling tracks to a convolution unit. Output from dense blocks which are disclosed to be feature maps are taken in view of the global-local encoding sequence of Wang which the examiner asserts are also feature maps, see section 3.3. It would be obvious to replace the dense blocks of Kim with multiple instances of the AAN as disclosed in Wang to achieve the aforementioned speech separation improvement]).
Regarding claim 4, Kim in view of Wang discloses: the device of claim 3.
Kim further discloses:
wherein the encoding input sequence of a first encoding stage among the plurality of encoding stages is an input feature sequence provided by the feature compression unit ([0059] The initial convolution unit 210 may perform initial convolution and down-sampling on an input signal, [0061] In the encoding process, the dense block 100 and the down-transition layer 220 may be repeatedly executed to extract a feature of a signal, [Wherein performing an initial down-sampling before encoding (in pre-processing) indicates the signal being received to be an input feature sequence provided by the feature compression unit]).
Regarding claim 5, Kim in view of Wang discloses: the device of claim 4.
Kim further discloses:
wherein the feature compression unit outputs the input feature sequence based on the input representation ([Fig. 1, initial convolution 210 input to dense block 100-1 based on mixed input], [0059] The initial convolution unit 210 may perform initial convolution and down-sampling on an input signal, [Wherein down-sampling tracks to a form of feature compression as previously disclosed. An input signal tracks to an input representation]).
Regarding claim 6, Kim in view of Wang discloses: the device of claim 5.
Kim further discloses:
wherein the speaker separation unit provides separated skip connection sequences to a corresponding decoding stage based on the global-local encoding sequence ([0065] an output of each step of the encoding part may be connected to an output of the same layer of the decoding part to provide a skip connection, [In view of the AAN comprising global-local encoding features of Wang to replace the dense blocks of Kim]).
Regarding claim 7, Kim in view of Wang discloses: the device of claim 6.
Kim further discloses:
wherein the decoder includes the plurality of decoding stages ([Fig. 1, dense blocks 100-4, 100-5, up-transition layers 230], [0084] When the encoding process is completed, the encoded signal may be decoded using at least one second dense block and at least one up-transition layer (S730), [Wherein Fig. 1 shows two dense blocks/up-transitions, indicating at least two decoding stages]) and a feature extension unit ([0084] up-sampling layer for the feature maps output by the convolution for reducing the number of feature maps output by each of the at least one second dense block, [0085] the up-sampling may be performed to maintain an initial input length of an initial feature map by increasing a length of a feature vector by a length of the feature vector reduced through the down-transition layer, [Wherein upsampling is a form of feature extension as shown through the feature vector extension]), and
each of the plurality of decoding stages further includes an upsampling unit configured to provide an upsampled sequence by upsampling each of a plurality of decoded input sequences input into each of the decoding stages ([The examiner asserts that up-transition layers track to up-sampling layers, wherein the previously cited portions of Fig. 1 indicate a plurality of upsampling units which up-sample output from dense blocks, i.e. part of a decoding stage]).
Regarding claim 8, Kim in view of Wang discloses: the device of claim 7.
Kim further discloses:
wherein each of the plurality of decoding stages further includes a feature fusion unit configured to provide a plurality of fused feature sequences based on the respective upsampled sequences and the separated skip connection provided by the speaker separation unit ([Fig. 1, © connecting the skip connection to up-sampled, i.e. up-transition layer, output], [Combining/fusing upsampled sequences with separated skip connections to result in one output indicates the “©” to be a feature fusion unit]).
Regarding claim 13, Kim discloses:
a speech separation system ([Title, “audio source separation”], [0099] The computer readable recording medium is any data storage device that may store data which can be thereafter read by a computer system) comprising:
an audio encoder configured to provide an input representation based on a mixed speech signal ([Fig. 1, mixed input into initial convolution (pre-processing) into dense block 100-1], [Wherein the dense block tracks to an encoder and the output from the dense block is an input representation based on a mixed speech signal (which has been convolved)]);
a separation encoder configured to provide an encoded feature sequence by downsampling the input representation ([Fig. 1, down-transition layer 220], [0061] In the encoding process, the dense block 100 and the down-transition layer 220 may be repeatedly executed to extract a feature of a signal in which only music is separated, [0082] In this case, the audio signal may be a mixed signal of a mono sound source, that is, a signal including voice, music, and noise, [Wherein a “down-transition layer” tracks to a downsampling. The examiner asserts that an audio signal comprised of voice is a speech signal]).
Kim does not disclose:
a speaker separation unit configured to provide a plurality of separated feature sequences by separating the encoded feature sequence for each of a plurality of speakers included in the speech signal.
Wang discloses:
a speaker separation unit configured to provide a plurality of separated feature sequences by separating the encoded feature sequence for each of a plurality of speakers included in the speech signal ([Fig. 1, Separator], [Fig. 1, explanation following figure label/title] Firstly, the encoder converts the speech signal into a feature representation, and then the feature separation and mask estimation are performed by the separator, [In view of the encoding units containing down-sampling operations in Kim. Feature separation and mask estimation which results in separated speech to be input to the decoder (as indicated by the multiple inputs to the decoder) indicates a provision to the decoder of separated feature sequences, i.e. masked feature spectra]).
Kim and Wang are considered analogous art within speaker separation. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Kim to incorporate the teachings of Wang, because of the novel way to construct a single-channel speech separation model which uses local and global attention modules to model the speech sequence and combines the local and global information in the sequence coding, improving the performance of speech separation ([Wang, pg. 2, Main Contribution 1 (bullet point 1)]).
Kim further discloses:
a decoder configured to provide an output representation for each speaker by upsampling the separated feature sequence ([Fig. 1, up-transition layer 230], [Fig. 6, S750 “output a plurality of signals”], [0084] the encoded signal may be decoded using at least one second dense block and at least one up-transition layer (S730), [In view of the plurality of output speaker signals of Wang, wherein the outputting step of Kim (S750) is performed after decoding (S730) indicating an output representation for up-sampled, separated feature sequences, i.e. maps]); and
an audio decoder configured to provide a speech signal for each speaker based on the output representation ([Fig. 1, output from dense block 100-5 into final convolution 250 resulting in separated output], [In view of the previously disclosed S750 indicating a plurality of output signals, the dense block is the audio decoder configured to provide a speech signal for each speaker (before a final convolution)]).
Regarding claim 14, Kim in view of Wang discloses: the system of claim 13.
Wang further discloses:
a loss calculation unit configured to calculate a loss value and an auxiliary loss value based on the sequences provided from the output representation and a plurality of decoding stages ([Section 4.3] The final output of the end-to-end speech separation model is the time-domain waveform of the clean signals, with the scale-invariant source-to-noise ratio…therein, s, the estimated target speech and s, the clean target speech are required to be normalized with a mean value of 0 to ensure scale invariance. The larger the SI-SNR value means the better the speech separation performance, but gradient descent is used to train the model during the training process, so the actual loss function is the inverse of SI-SNR, [In the context of Wang being to separate two different speakers (as seen in Figs. 1, 2), calculating a loss value between each estimated target speech and clean target speech indicates at least a loss calculation unit and an auxiliary loss calculation unit, one for each speaker separated]).
Regarding claim 15, Kim in view of Wang discloses: the system of claim 14.
Wang further discloses:
wherein the loss calculation unit further includes an auxiliary feature extension unit ([Fig. 2, Overlap-Add into convolution], [The examiner asserts that the overlap-add operation for discrete features resulting in signals to be decoded after convolution suggests an extension of features into complete signals. Considering at least two separated signals, the second separated signal is an auxiliary signal with auxiliary operations as compared to the first separated signal]) and an auxiliary audio decoder ([Fig. 2, Decoding]) each configured to provide an auxiliary signal to produce the auxiliary loss value based on the sequences provided from the decoding stages ([Fig. 2, separated output from speakers S1 and S2], [Wherein the output separated signals would be used with the loss function of Wang as previously disclosed as “estimated target speech”]).
Regarding claim 16, Kim in view of Wang discloses: the system of claim 15.
Wang further discloses:
wherein a parameter (weight) applied to the speech separation system is adjusted based on the loss value and the auxiliary loss value ([pg. 8, Section 4.3] gradient descent is used to train the model during the training process, so the actual loss function is the inverse of SI-SNR, [Performing gradient descent with an inverse SI-SNR indicates the inverse SI-SNR to be a weight value applied to the speech separation system]).
Regarding claim 17, Kim discloses: A method for operating a speech separation device (Abstract, A method for receiving a mono sound source audio signal including phase information as an input, and separating into a plurality of signals), the method comprising:
providing, by a separation encoder ([Fig. 1, down-transition layer 220]), an encoded feature sequence for each encoding stage by downsampling an input representation generated based on a mixed speech signal ([0061] In the encoding process, the dense block 100 and the down-transition layer 220 may be repeatedly executed to extract a feature of a signal in which only music is separated, [0082] In this case, the audio signal may be a mixed signal of a mono sound source, that is, a signal including voice, music, and noise, [Wherein a “down-transition layer” tracks to a downsampling. The examiner asserts that an audio signal comprised of voice is a speech signal]).
Kim does not disclose:
a global-local encoding sequence for each encoding stage; and
providing, by a speaker separation unit, a plurality of separated feature sequences and separated skip connections by separating the encoded feature sequence and the global-local encoding sequences for each stage for each of a plurality of speakers included in the speech signal.
Wang discloses:
a global-local encoding sequence for each encoding stage ([Fig. 2, Adaptive Attention Network (AAN) containing Local Attention and Global Attention], [In view of the plurality of dense blocks, i.e. encoding stages, of Kim]); and
providing, by a speaker separation unit ([Fig. 1, Separator]), a plurality of separated feature sequences and separated skip connections by separating the encoded feature sequence and the global-local encoding sequences for each stage for each of a plurality of speakers included in the speech signal ([Fig. 1, explanation following figure label/title] Firstly, the encoder converts the speech signal into a feature representation, and then the feature separation and mask estimation are performed by the separator, [Fig. 2, B. Local & Global Attention Network], [In view of the encoding units containing down-sampling operations in Kim. Feature separation and mask estimation which results in separated speech to be input to the decoder (as indicated by the multiple inputs to the decoder) indicates a provision to the decoder of separated feature sequences, i.e. masked feature spectra. Further, consider the arrows of the global-local network of Wang which skip potential processing steps, necessarily skip connections, wherein each separated speech will have its own associated global-local processing, indicating a plurality of separated feature sequences and skip connections by separating multi-stage encoding sequences in view of the multiple decoding steps of Kim]).
Kim and Wang are considered analogous art within speaker separation. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Kim to incorporate the teachings of Wang, because of the novel way to construct a single-channel speech separation model which uses local and global attention modules to model the speech sequence and combines the local and global information in the sequence coding, improving the performance of speech separation ([Wang, pg. 2, Main Contribution 1 (bullet point 1)]).
Kim further discloses:
downsampling an input representation generated based on a mixed speech signal ([Fig. 1, Dense Block 100-1 to 100-3 and down-transition layer 220], [0061] the dense block 100 and the down-transition layer 220 may be repeatedly executed, [0065] The dense block may be composed of k initial feature maps and L convolution layers, [0066] The dense block is a block that can be used in…the down-sampling process, [0071] down-transition layer, 1×1 convolution (S510) may be performed to reduce the number of feature maps 51 increased through the processing by the dense block, [Wherein each iteration of dense blocks/down sampling tracks to a convolution unit. Output from dense blocks which are disclosed to be feature maps are taken in view of the global-local encoding sequence of Wang which the examiner asserts are also feature maps, see section 3.3. It would be obvious to replace the dense blocks of Kim with multiple instances of the AAN as disclosed in Wang to achieve the aforementioned speech separation improvement]);
providing, by a reconstruction decoder ([Fig. 1, up-transition layer 230]), an output representation for each speaker by upsampling and fusing the separated feature sequence and the separated skip connection ([Fig. 1, ©], [Fig. 6, S750 “output a plurality of signals”], [0084] the encoded signal may be decoded using at least one second dense block and at least one up-transition layer (S730), [In view of the plurality of output speaker signals of Wang, wherein the outputting step of Kim (S750) is performed after decoding (S730) indicating an output representation for up-sampled, separated feature sequences, i.e. maps. Further, combining output from up-transition layers with that from skip connections indicates a fusing separated feature sequences and skip connections]).
Regarding claim 18, Kim discloses: A method for operating a speech separation system (Abstract, A method for receiving a mono sound source audio signal including phase information as an input, and separating into a plurality of signals), the method comprising:
providing, by an audio encoder, an input representation based on a mixed speech signal ([Fig. 1, mixed input into initial convolution (pre-processing) into dense block 100-1], [Wherein the dense block tracks to an encoder and the output from the dense block is an input representation based on a mixed speech signal (which has been convolved)]);
providing, by a separation encoder ([Fig. 1, down-transition layer 220]), an encoded feature sequence for each encoding stage by downsampling an input representation generated based on a mixed speech signal ([0061] In the encoding process, the dense block 100 and the down-transition layer 220 may be repeatedly executed to extract a feature of a signal in which only music is separated, [0082] In this case, the audio signal may be a mixed signal of a mono sound source, that is, a signal including voice, music, and noise, [Wherein a “down-transition layer” tracks to a downsampling. The examiner asserts that an audio signal comprised of voice is a speech signal]); and
downsampling an input representation generated based on a mixed speech signal ([Fig. 1, Dense Block 100-1 to 100-3 and down-transition layer 220], [0061] the dense block 100 and the down-transition layer 220 may be repeatedly executed, [0065] The dense block may be composed of k initial feature maps and L convolution layers, [0066] The dense block is a block that can be used in…the down-sampling process, [0071] down-transition layer, 1×1 convolution (S510) may be performed to reduce the number of feature maps 51 increased through the processing by the dense block, [Wherein each iteration of dense blocks/down sampling tracks to a convolution unit. Output from dense blocks which are disclosed to be feature maps are taken in view of the global-local encoding sequence of Wang which the examiner asserts are also feature maps, see section 3.3. It would be obvious to replace the dense blocks of Kim with multiple instances of the AAN as disclosed in Wang to achieve the aforementioned speech separation improvement]).
Kim does not disclose:
a global-local encoding sequence for each encoding stage; and
providing, by a speaker separation unit, a plurality of separated feature sequences and separated skip connections by separating the encoded feature sequence and the global-local encoding sequences for each stage for each of a plurality of speakers included in the speech signal.
Wang discloses:
a global-local encoding sequence for each encoding stage ([Fig. 2, Adaptive Attention Network (AAN) containing Local Attention and Global Attention], [In view of the plurality of dense blocks, i.e. encoding stages, of Kim]); and
providing, by a speaker separation unit ([Fig. 1, Separator]), a plurality of separated feature sequences and separated skip connections by separating the encoded feature sequence and the global-local encoding sequences for each stage for each of a plurality of speakers included in the speech signal ([Fig. 1, explanation following figure label/title] Firstly, the encoder converts the speech signal into a feature representation, and then the feature separation and mask estimation are performed by the separator, [Fig. 2, B. Local & Global Attention Network], [In view of the encoding units containing down-sampling operations in Kim. Feature separation and mask estimation which results in separated speech to be input to the decoder (as indicated by the multiple inputs to the decoder) indicates a provision to the decoder of separated feature sequences, i.e. masked feature spectra. Further, consider the arrows of the global-local network of Wang which skip potential processing steps, necessarily skip connections, wherein each separated speech will have its own associated global-local processing, indicating a plurality of separated feature sequences and skip connections by separating multi-stage encoding sequences in view of the multiple decoding steps of Kim]).
Kim and Wang are considered analogous art within speaker separation. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Kim to incorporate the teachings of Wang, because of the novel way to construct a single-channel speech separation model which uses local and global attention modules to model the speech sequence and combines the local and global information in the sequence coding, improving the performance of speech separation ([Wang, pg. 2, Main Contribution 1 (bullet point 1)]).
Kim further discloses:
providing, by a reconstruction decoder ([Fig. 1, up-transition layer 230]), an output representation for each speaker by upsampling and fusing the separated feature sequence and the separated skip connection ([Fig. 1, ©], [Fig. 6, S750 “output a plurality of signals”], [0084] the encoded signal may be decoded using at least one second dense block and at least one up-transition layer (S730), [In view of the plurality of output speaker signals of Wang, wherein the outputting step of Kim (S750) is performed after decoding (S730) indicating an output representation for up-sampled, separated feature sequences, i.e. maps. Further, combining output from up-transition layers with that from skip connections indicates a fusing separated feature sequences and skip connections]); and
providing, by an audio decoder, a speech signal separated for each speaker based on the output representation for each speaker ([Fig. 1, output from dense block 100-5 into final convolution 250 resulting in separated output], [In view of the previously disclosed S750 indicating a plurality of output signals, the dense block is the audio decoder configured to provide a speech signal for each speaker (before a final convolution)]).
Claim(s) 9-12 is/are rejected under 35 U.S.C. 103 as being unpatentable over Kim in view of Wang, further in view of Li et al. (“An Efficient Encoder-Decoder Architecture for Top-Down Attention for Speech Separation”), hereinafter Li.
Regarding claim 9, Kim in view of Wang discloses: the device of claim 8.
Kim in view of Wang does not disclose:
wherein each of the plurality of decoding stages includes a plurality of Siamese global-local transformers each configured to provide an output of a global-local decoded sequence generated based on each fused feature sequence.
Li discloses: wherein each of the plurality of decoding stages includes a plurality of Siamese global-local transformers each configured to provide an output of a global-local decoded sequence generated based on each fused feature sequence ([Fig. 1, output from Global attention (GA) module input to local attention (LA) layers in the Decoder], [Wherein Fig. 10 of the instant application discloses Siamese global-local transformers. The examiner is interpreting these to be duplicate global-local transformers. Taking this in view of Li, there is disclosed a GA module, output from which is passed in LA layers in a decoder. This indicates both the GA and LA are used for determining features, wherein there are a plurality of layers indicating a plurality of Siamese global-local transformers, wherein decoding is part of a transformation operation. It would be obvious to replace the global-local transformers of Li with the dense blocks of Kim to achieve improvements further described below]).
Kim, Wang, and Li are considered analogous art within speech separation. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the teachings of Kim in view of Wang to incorporate the teachings of Li, because of the novel way to use top-down attention which includes extracting global attention modules and cascaded local attention layers, improving separation performance and efficiency in speech (Li, Abstract).
Regarding claim 10, Kim in view of Wang, further in view of Li discloses: the device of claim 9.
Li further discloses:
wherein each of the decoding stages further includes a cross-reconstruction transformer configured to provide a reconstructed decoded sequence by extracting feature information among the speakers based on an output of a decoding transformer of each of the Siamese global-local transformers ([Fig. 1, Acoustic features output from GA/LA layer], [pg. 4, Section 3.1] the target waveform xi is reconstructed using the target speech embedding Ei through an audio decoder, which can be implemented using a 1-D transposed convolutional layer with the same kernel and stride sizes as the audio encoder, [Wherein the audio decoder as previously disclosed in claim 9 has acoustic features output from each decoding step (the Siamese global-local encoders) indicating a cross-reconstruction transformer to transform the frequency features into acoustic features]).
Regarding claim 11, Kim in view of Wang, further in view of Li discloses: the device of claim 10.
Li further discloses:
wherein among the plurality of decoding stages, the plurality of decoded input sequences of a first decoding stage are the separated feature sequences ([Fig. 2, Output from Audio decoder being SPK A/B separated audio], [Separating feature before decoding, wherein the decoding has a plurality of stages/layers as shown in Kim/Li, indicates a second decoding layer which receives a decoded input sequence from a first decoding stage which are separated features as shown through the separation before decoding]).
Regarding claim 12, Kim in view of Wang, further in view of Li discloses: the device of claim 11.
Kim further discloses:
wherein the feature extension unit is configured to provide the output representation generated based on the plurality of decoded feature sequences provided from an N-th decoding stage among the plurality of decoding stages ([Fig. 1, output from up-transition layer 230 is passed into dense block 100-5], [0061] the exemplary embodiment shown in FIG. 1, a case in which the dense block and the down-transition layer are repeatedly executed two times is illustrated, but the number s of repetitions may be adjusted as necessary, [As shown in Fig. 1, a feature extension is providing output from a dense block, wherein the number of dense block/up-transition layers included is variable, indicating the n=2 decoding layers of this figure is not limiting and could be a greater number]).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Sun et al. (US-20230401429-A1) discloses “Systems, methods, and computer program products for audio processing based on convolutional neural network (CNN) are described. A first CNN architecture may comprise a contracting path of a U-net, a multi-scale CNN, and an expansive path of a U-net. The contracting path may comprise a first encoding layer and may be configured to generate an output representation of the contracting path. The multi-scale CNN may be configured to generate, based on the output representation of the contracting path, an intermediate representation. The multi-scale CNN may comprise at least two parallel convolution paths. The expansive path may comprise a first decoding layer and may be configured to generate a final representation based on the intermediate representation generated by the multi-scale CNN. Within a second CNN architecture, the first encoding layer may comprise a first multi-scale CNN with at least two parallel convolution paths, and the first decoding layer may comprise a second multi-scale CNN with at least two parallel convolution paths” (abstract). See entire document.
Li et al. (US-12190896-B2) discloses “Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for processing an input audio waveform using a generator neural network to generate an output audio waveform. In one aspect, a method comprises: receiving an input audio waveform; processing the input audio waveform using an encoder neural network to generate a set of feature vectors representing the input audio waveform; and processing the set of feature vectors representing the input audio waveform using a decoder neural network to generate an output audio waveform that comprises a respective output audio sample for each of a plurality of output time steps.” (abstract). See entire document.
He et al. (“SE-DPTUNet: Dual-Path Transformer based U-Net for Speech Enhancement”) discloses “a novel transformer-based U-Net model for end-to-end speech enhancement in the time domain, called SE-DPTUNet, which is comprised of an encoding module, a proposed dual-path transformer module and a decoding module. The encoding module extracts low-level features from input noisy speech. The dual-path transformer module exploits transformer encoder-decoder layers to model long-range speech sequence by extracting short- and long-term information of speech sequences, generating the contextual features. Meanwhile,
the down- and up-sampling blocks are adopted in each encoder-decoder layer to obtain the hierarchical features. The decoding module is applied to reconstruct the enhanced speech based on the output from transformer module.” (abstract). See entire document.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to THEODORE JOHN WITHEY whose telephone number is (703)756-1754. The examiner can normally be reached Monday - Friday, 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Flanders can be reached at (571) 272-7516. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/THEODORE WITHEY/Examiner, Art Unit 2655
/ANDREW C FLANDERS/Supervisory Patent Examiner, Art Unit 2655
