DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . This action is responsive to claims filed 8/18/2025. Claims 1-30 are pending.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-2, 5-8, 14-15, 18-21, 27-30 are rejected under 35 U.S.C. 103 as being unpatentable over Gan (US 20220044022 A1) in view of Zhong (US 20190130248 A1).
For claim 1, Gan discloses: a method performed by an artificial neural network (ANN), comprising:
generating, at a first stage of a multi-stage cross-attention model of the ANN, a respective first representation for each modality of a plurality of modalities (figs.2-3 shows a cross-attention model generating respective first representations in the pipeline based on cross-modal input), each input of the sequence of inputs comprising one or both of image content or audio content (fig.2-3 shows operating on input video and audio);
generating a concatenated feature representation (fig.2:226, fig.3:226, 0068-70 discloses generating output representation O_av based on attended visual and audio inputs, O_av being generated based on the CMRA transformer type architecture of fig.5, 0060-61, the latter being an attended concatenated output, see 504-508, hence, O_av being a concatenated feature representation; further, concatenated representations are generated throughout the architecture whenever CMRA used, e.g., 318, 320);
determining, for each input of the sequence of inputs based on the concatenated feature representation, a probability distribution between a first probability of the respective image content and/or the respective audio content of the input including one or more background actions and a second probability of the respective image content and/or the respective audio content of the input including one or more foreground actions (fig.3:326-330, 0071-73: determining, for sequence of audio-video inputs, and based on the above concatenated representations, a probability distribution confidence score during the inference phase); and
localizing one or more actions in one or more inputs of the sequence of inputs based on the probability distribution, the respective second probability being greater than the respective first probability for each input of the one or more inputs (fig.6, 0077: localization based on probabilities calculated in 0071-73).
Gan does not expressly disclose: wherein the first representations are first attended representations; said representations based on determining a first cross-correlation between a first representation of each modality of the plurality of modalities associated with a sequence of inputs;
generating, at each second stage of one or more second stages of the multi-stage cross-attention model, a respective second attended representation for each modality of the plurality of modalities based on determining a second cross-correlation between a respective prior attended representation of each modality;
the concatenated feature representation being associated with a final second stage of the one or more second stages based on the second cross-correlation associated with the final second stage, the respective first attended representation of each modality, and the first representation of each modality.
Zhong discloses: the first representations are first attended representations; said representations based on determining a first cross-correlation between a first representation of each modality of the plurality of modalities associated with a sequence of inputs (figs. 3a-b, 0027-34 gives overview of multi-stage cross-attention structure, with inputs E^1, E^2 being cross-correlated to give an affinity matrix (eq.3) that is then used to attend over the respective inputs E (such as using a row- or column-wise softmax, see 0033, eq.4-5) to generate outputs S^1, S^2, hence, S^1-2 being attended representation based on cross-correlation matrix A);
generating, at each second stage of one or more second stages of the multi-stage cross-attention model, a respective second attended representation for each modality of the plurality of modalities based on determining a second cross-correlation between a respective first prior attended representation of each modality (fig.3A: iterative application of the above process of fig.3B for additional second stages);
the concatenated feature representation being associated with a final second stage of the one or more second stages based on the second cross-correlation associated with the final second stage, the respective first attended representation of each modality, and the first representation of each modality (Zhong fig.3A-B combined with Gan would yield the concatenation before event detection being based on the final second stage attended outputs, furthermore fig.3A:360, fig.3B:360, 0037-38 contemplates residual or skip connections between earlier layers including respective E^1-2 representations and S^1-2 attended representations).
It would have been obvious before the effective filing date to a person of ordinary skill in the art to modify the method of Gan by incorporating the multi-stage co-attention architecture of Zhong. Both concern the art of co-attention networks involving audio and video signals (0015, 0060), and the incorporation would have, according to Zhang, allow the generation of richer codependent representations containing more relevant information and better mutual and self attendance (0029).
For claim 2, Gan modified by Zhong discloses the method of claim 1, as described above. Gan modified by Zhong further discloses: in which the first representation of each modality is a latent representation based on features of each modality extracted from the sequence of inputs (Gan fig.3:306-310: representation via CNN constitutes a latent representation based on features extraction from A/V sequence; Zhong 0031 eq.1-2).
For claim 5, Gan modified by Zhong discloses the method of claim 1, as described above. Gan modified by Zhong further discloses: generating the respective second attended representation of each modality based on a sum of the second cross-correlation and the respective first attended representation of each modality (Zhong 0037-38 contemplates forming skip connections with attended representations S^1-2, these skip connections destined for combination, such as via summation, with the input of later layers, such as after skipping one or more layers, hence, combining with the inputs of a later layer, such as 312a-n, 314a-n yielding combination with the second cross-correlation as it attends to the input (i.e., as the input of the destination layer) to generate the respective second attended representation).
For claim 6, Gan modified by Zhong discloses the method of claim 6, as described above. Gan modified by Zhong further discloses: generating the concatenated feature representation based on the respective second attended representation of each modality (Gao fig.2:226, fig.3:226, 0068-70 discloses generating output representation O_av based on attended visual and audio inputs, O_av being generated based on the CMRA transformer type architecture of fig.5, 0060-61, the latter being an attended concatenated output, see 506-508, hence, O_av being a concatenated feature representation; further, concatenated representations are generated throughout the architecture (i.e., whenever CMRA used, e.g., 318, 320)).
For claim 7, Gan modified by Zhong discloses the method of claim 1, as described above. Gan further discloses: in which determining the probability distribution comprises:
determining a reliability of the first probability (0071-73: magnitude of confidence scores > 0.5 for foreground for each category constitutes a reliability metric); and
determining a reliability of the second probability (ibid: likewise for inverse background probability).
For claim 8, Gan modified by Zhong discloses the method of claim 1, as described above. Gan further discloses: the first modality is a visual modality; the second modality is an audio modality; and the sequence of inputs is a sequence of frames (fig.3:302, 304).
Remaining claims 14-15, 18-21, 27-30 recite analogous devices, media, and ANNs and are rejected for the same reasons.
Claim(s) 3-4, 16-17 are rejected under 35 U.S.C. 103 as being unpatentable over Gan (US 20220044022 A1) in view of Zhong (US 20190130248 A1) in view of Gao ("Co-saliency detection with co-attention fully convolutional network", published 8/20/2020).
For claim 3, Gan modified by Zhong discloses the method of claim 1, as described above. Gan modified by Zhong does not disclose the limitations of claim 3.
Gao discloses: generating the first attended representation of a first modality of the plurality of modalities based on a sum of the first cross-correlation and the first representation of the first modality (fig.2, §III.C Co-Attention Module (p.5) last 2 ¶, eq.4: the output of the co-attention module for the top and bottom routes comprise the original signal added back to the attended signal with a weight gamma); and
generating the first attended representation of a second modality of the plurality of modalities based on a sum of the first cross-correlation and the first representation of the second modality (ibid: likewise for other branch).
It would have been obvious before the effective filing date to a person of ordinary skill in the art to modify the method of Gan modified by Zhong by incorporating the summation of the original signal technique of Gao. Both concern the art of co-attention networks, and the incorporation would have, according to Gao, allow reliance on non-local evidence as produced by the attention mechanism in a gradual manner (§III.C last ¶ (p.6)).
For claim 4, Gan modified by Zhong modified by Gao discloses the method of claim 3, as described above. Gan modified by Zhong modified by Gao further discloses: determining the second cross-correlation based on a product of the first attended representation of each modality and a weight variable (§III.C eq.4 (p.5)).
Remaining claims 16-17 recite analogous devices, media, and ANNs and are rejected for the same reasons.
Claim(s) 9-13, 22-26 are rejected under 35 U.S.C. 103 as being unpatentable over Gan (US 20220044022 A1) in view of Zhong (US 20190130248 A1) in view of Goyal ("Cross-modal learning for multi-modal video categorization", published 3/16/2020).
For claim 9, Gan modified by Zhong discloses the method of claim 1, as described above. Gan modified by Zhong further discloses: creating each skip-connection of a plurality of skip-connections (Zhong 0037-38), in which:
each stage of the multi-stage cross-attention model is associated with a pair of skip-connections of the plurality of skip-connections (Zhong 0037-38, fig.3A-3B:360); and
each skip-connection of the pair of skip-connections is associated with one modality of the plurality of modalities (Zhong 0037-38 contemplates application to E^1-2, S^1-2, hence, either input).
Gan modified by Zhong does not disclose; wherein the creating comprises gating.
Goyal discloses: wherein the creating comprises gating (§3, fig.5 shows the use of a correlation tower to gate a cross-modal skip connection.
It would have been obvious before the effective filing date to a person of ordinary skill in the art to modify the method of Gan by incorporating the gating technique of Goyal. Both concern the art of audio-visual multi-modal fusion and the incorporation would have, according to Goyal, allowed selective shifted cross-modal correlation based on whether the two modes are relevant (§3 ¶1).
For claim 10, Gan modified by Zhong modified by Goyal discloses the method of claim 9, as described above. Gan modified by Zhong modified by Goyal further discloses: gating each skip-connection based on an output of a gating layer associated with the respective stage of the multi-stage cross-attention model associated with the respective skip-connection (Goyal §3 fig.5 shows correlation tower layers associated with a respective stage of the cross-attention model).
For claim 11, Gan modified by Zhong modified by Goyal discloses the method of claim 9, as described above. Gan modified by Zhong modified by Goyal further discloses: in which each skip connection outputs a stage gated feature a plurality of stage gated features, each stage gated feature associated with one modality of the plurality of modalities (Goyal §3 fig.5 shows output of a gated RELU / DNN feature for each mode).
For claim 12, Gan modified by Zhong modified by Goyal discloses the method of claim 9, as described above. Gan modified by Zhong modified by Goyal further discloses: gating each stage-connection of a plurality of stage-connections (Goya’s disclosure in §3 fig.5 of the advantages of gating connections would, combined with Zhong’s disclosure in 0037-38 of skip connections would yield a technique where the gates are applied to the various skip connections), in which:
each stage-connection of the plurality of stage-connections receives an input from a set of skip-connections (fig.3A-B:360, 0037 discloses skip connections for various stages as source), each skip-connection of the set of skip-connections associated with a different stage of the multi-stage cross-attention model (ibid: routing skip connections to various different stages), and each skip-connection of the set of skip-connections being one skip-connection of the plurality of skip-connections (ibid).
For claim 13, Gan modified by Zhong modified by Goyal discloses the method of claim 9, as described above. Gan modified by Zhong modified by Goyal further discloses: each stage-connection of the plurality of stage-connections is gated based on a final attended representation of one modality of the plurality of modalities (Zhong fig.3A-B:360, with 0037-38 contemplating application to attended representation S^1-2 and combination, such as summation, with other inputs at the destination); and the final attended representation of each modality of the plurality of modalities being determined at a final stage of the multi-stage cross-attention model (ibid, applied to the final stage of the multi-stage model).
Remaining claims 22-26 recite analogous devices, media, and ANNs and are rejected for the same reasons.
Response to Arguments
Applicant’s arguments have been fully considered. In the remarks, Applicant argued:
1. Zhong describes each layer’s cross-correlation (affinity) as being computed over then-current encoded sequence representations and not over “prior attended representations” output by an earlier cross attention stage.
Examiner respectfully disagrees. According to figs. 3A-B, the output of the summaries 332-333, the summaries being the attended representations based on softmaxing a cross correlation affinity 331, are being fed as encodings into the next co-attention block. Hence, contrary to Applicants assertion, each layer’s cross correlations are computed over prior attended representations of the prior block.
2. Claim 1 recites that each second stage computes a second cross correlation specifically between prior attended representations generated by an earlier cross-attention stage, and then generates respective second attended representations from that respective second cross correlation. Zhong, in contrast, describes each layer’s affinity computation as operating on the layer’s current encoded inputs (E^1 and E^2) rather than on attended outputs from a prior cross-attention stage that are expressly carried forward as “prior attended representations” for use in a second-stage cross attention operation.
Examiner respectfully disagrees. Applicant argues that the currently encoded inputs of Zhong (e.g., fig.3A-B:312, 314) differs from the attended outputs of the prior cross-attention stage or prior attended representations. However, the current encoded inputs are simply the attended features of the respective E^1 and E^2 according to the Affinity matrix, see 0033, eq.4-5. Hence, Examiner submits that, contrary to Applicant’s assertions, Zhong discloses the respective second attended representation based on prior attended representations, as claimed.
3. The combination of Gan and Zhong is improper. A POSITA would not be motivated to modify Gan’s fusion-based interaction module into the claimed architecture that preserves separate attended representations per modality across stages and performs the claimed second-stage cross correlations. The references do not suggest discarding Gan’s single fused interaction design in favor of a multi-stage cross-attention architecture that maintains and iteratively refines per-modality attended representations based on second-cross-correlation between a respective attended representation, as claimed.
Examiner respectfully disagrees. Gan’s architecture generates various cross-modal interactions prior to the fusion, see figs.2-3, steps before fusion of 226-326. However Gan does not expressly disclose a multi-stage cross-correlation architecture. Zhong discloses the said architecture, and further contemplates its application to various multi-modal sequences including audio and video (0060). According to Zhong, such an architecture could potentially lead to development of higher accuracy models, or improve training, such as for audio and visual sequences (0015).
Furthermore, evidentiary reference Li ("Multimodal fusion with co-attention mechanism", published 9/10/2020) is cited. Li contemplates that fusion processes (§I ¶2-3) can be improved in a modality agnostic way (§I ¶3-4) via stacked transformer or attention layers, see fig.1 showing a single block of a serial multi-head co-attention chain. Hence Li discloses that a POSITA would recognize that such co-attention architectures prior to fusion may be applied to fusion tasks across various modalities without substantial modification. Hence, a POSITA would be motivated to apply the teachings of Zhong or Li’s stacked co-attention blocks to the audio and video fusion of Gan in order to derive the various cited benefits (Gan 0015, Li §I ¶5).
Although not applied in the mapping, Li discloses a stacked co-attention layer similar to the claimed invention (see figs.1-3) and may deserve consideration in potential responses or amendments.
4. Zhong is directed to dual sequence inference for text, such as document-question pairs. Zhong does not concern audio-visual event localization, does not operate on image and audio, and does not describe any probability distribution as cited.
Examiner respectfully disagrees. Gan 0015, 0060 contemplates application to audio and visual modalities. Furthermore, Li is cited as an evidentiary reference for cross-attention to fusion being modality agnostic and readily applicable by a POSITA to various cross-modal fusion architectures, see above and Li §I ¶1-5.
5. The Office Action does not articulate a persuasive reason why a POSITA would start from Gan’s audio-video interaction to import Zhong’s text-oriented co-attention encode including the claimed architectures, and hence constitutes improper hindsight.
Examiner respectfully disagrees for the reason describe above.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Li ("Multimodal fusion with co-attention mechanism", published 9/10/2020) discloses a stacked cross-modal co-attention architecture.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to LIANG LI whose telephone number is (303)297-4263. The examiner can normally be reached Mon-Fri 9-12p, 3-11p MT (11-2p, 5-1a ET).
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor Jennifer Welch can be reached on (571)272-7212. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from Patent Center and the Private Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from Patent Center or Private PAIR. Status information for unpublished applications is available through Patent Center or Private PAIR to authorized users only. Should you have questions about access to Patent Center or the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. The examiner is available for interviews Mon-Fri 6-11a, 2-7p MT (8-1p, 4-9p ET).
/LIANG LI/
Primary examiner AU 2143
Prosecution Insights