DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . This action is responsive to the Amendment filed on 01/06/2026. Claims 1-20 are pending in the case. Claims 1, 9, and 18 are independent claims.
Response to Arguments
Applicant's prior art arguments have been fully considered but are moot in view of the new grounds of rejection presented below.
Claim Rejections - 35 U.S.C. § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. §§ 102 and 103 (or as subject to pre-AIA 35 U.S.C. §§ 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. § 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant are advised of the obligation under 37 C.F.R. § 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. § 102(b)(2)(C) for any potential 35 U.S.C. § 102(a)(2) prior art against the later invention.
Claims 1-20 are rejected under 35 U.S.C. § 103 as being unpatentable over Cai et al. (Cai, Lei, Zhengyang Wang, Hongyang Gao, Dinggang Shen, and Shuiwang Ji. "Deep adversarial learning for multi-modality missing data completion." In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pp. 1158-1166. 2018., hereinafter Cai) in view of Wang Shaonan et al. (Chinese Pat. App. Pub. No. CN-108536735-A, hereinafter Wang Shaonan) and Aronowitz et al. (U.S. Pat. App. Pub. No. 2020/0372899, hereinafter Aronowitz).
As to independent claim 1, Cai teaches:
obtaining a modal information group, wherein the modal information group comprises at least two pieces of modal information (section 3.1, a modal information group is seen as subject I which is composed of two modalities {x, y});
determining, based on an attribute of the modal information group, that a part or all of first modal information is missing from the modal information group, wherein the modal information group further comprises second modal information (section 3. 1: the determination of the test set is based on the fact that a subject I is composed of two modalities, which is seen as an attribute, the determination of the subjects where modality y is missing, which is seen as the first modal information. The modality x is seen as the second modal information);
extracting a feature vector of the second modal information (section 3. 6: MRI modality features);
determining a target feature vector of the first modal information based on a preset feature vector mapping relationship and the feature vector of the second modal information (section 3. 6, second paragraph, the target feature vector is seen as the values of voxels in the PET modal)….
Cai does not appear to expressly teach a modal information completion method comprising.
Wang Shaonan teaches a modal information completion method comprising (Title and abstract. Paragraph 152).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having the multi-modality missing data completion of Cai to include the multi-modal techniques of Wang Shaonan to improve accuracy (see Wang Shaonan at paragraph 4).
Cai does not appear to expressly teach the modal information group is a video, the first modal information is of a text type, the second modal information comprises modal information of a speech type and modal information of an image type.
Wang Shaonan teaches the modal information group is a video, the first modal information is of a text type, the second modal information comprises modal information of a speech type and modal information of an image type (Paragraph 15, "training dataset comprises a plurality of feature vectors computed for the plurality of audio-segments of the plurality of speakers extracted for the entire video". The described entire video reads on the claimed modal information group. The plurality of audio-segments of the plurality of speakers reads on the second modal information. Paragraph 6, "colored text representation in association with a video-segment corresponding to the audio-segment". video-segment corresponding to the audio-segment reads on image type. The colored text reads on the first model information).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having the multi-modality missing data completion of Cai to include the automated subtitle generation techniques of Aronowitz to help a hearing-impaired person to understand who is speaking (see Aronowitz at paragraph 3).
As to dependent claim 2, Cai further teaches the determining a-the target feature vector of the first modal information further comprises: determining a candidate feature vector of the first modal information based on the feature vector mapping relationship and the feature vector of the second modal information (Figure 2, predicted PET G(x)); and determining the target feature vector of the first modal information based on the candidate feature vector of the first modal information (Figure 2, PET after concatenation is no longer just a predicted candidate).
As to dependent claim 3, Cai further teaches the determining the target feature vector of the first modal information further comprises: determining the target feature vector of the first modal information based on a preset machine learning model and the feature vector of the second modal information, wherein the machine learning model learns the feature vector mapping relationship and is used to output a feature vector of other modal information based on an input feature vector of modal information (Figure 2 shows the preset machine learning model determining the PET modality information based on MRI x (i.e., G(x)). Learning occurs with the discriminator feedback loss).
As to dependent claim 4, Cai further teaches the attribute of the modal information group comprises at least one of the following: a quantity of pieces of modal information in the modal information group, and a data volume of each piece of modal information in the modal information group (Section 3.1, "The training set consists of subjects with both x and y as {xi ,yi , ℓi }N i=1 while the test set consists of subjects with only x").
As to dependent claim 5, Cai further teaches obtaining first auxiliary information, and determining the attribute of the modal information group based on the first auxiliary information, wherein the first auxiliary information indicates at least one of the following: a quantity of pieces of modal information in the modal information group, and a data volume of each piece of modal information in the modal information group; determining the attribute of the modal information group based on preset second auxiliary information, wherein the preset second auxiliary information indicates at least one of the following: a quantity of pieces of modal information in any obtained modal information group, and a data volume of each piece of modal information in the any obtained modal information group; or determining the attribute of the modal information group based on an attribute of another modal information group, wherein the another modal information group is a modal information group obtained before the modal information group is obtained (Where the first auxiliary information and second auxiliary information are not distinct from the attribute, then Section 3.1, "The training set consists of subjects with both x and y as {xi ,yi , ℓi }N i=1 while the test set consists of subjects with only x").
As to dependent claim 6, Wang Shaonan further teaches the modal information group further comprises third modal information; and wherein the method further comprises: extracting a feature vector of the third modal information; and determining the target feature vector of the first modal information based on the feature vector mapping relationship, the feature vector of the third modal information, and the feature vector of the second modal information (The third modal information can be seen as audio model information. Since the vector mapping model is trained using audio modal vector (see paragraph 16), it is considered that the target feature vector of the first model information (visual) is also based on the third modal information).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having the multi-modality missing data completion of Cai to include the multi-modal techniques of Wang Shaonan to improve accuracy (see Wang Shaonan at paragraph 4).
As to dependent claim 7, Wang Shaonan further teaches the determining the target feature vector of the first modal information based on the preset feature vector mapping relationship, the feature vector of the third modal information, and the feature vector of the second modal information comprises: determining another candidate feature vector of the first modal information based on the feature vector mapping relationship and the feature vector of the third modal information; and determining the target feature vector of the first modal information based on the candidate feature vector of the first modal information and the another candidate feature vector of the first modal information (Where the another candidate feature vector is not distinct from the candidate feature vector, then The third modal information can be seen as audio model information. Since the vector mapping model is trained using audio modal vector (see paragraph 16), it is considered that the target feature vector of the first model information (visual) is also based on the third modal information).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having the multi-modality missing data completion of Cai to include the multi-modal techniques of Wang Shaonan to improve accuracy (see Wang Shaonan at paragraph 4).
As to dependent claim 8, Cai further teaches each piece of modal information comprised in the modal information group has a different type (section 3.6, MRI and PET are seen as two different modality data types).
As to independent claim 9, Cai teaches:
obtain a modal information group, wherein the modal information group comprises at least two pieces of modal information (section 3.1, a modal information group is seen as subject I which is composed of two modalities {x, y});
determine based on an attribute of the modal information group, that a part or all of first modal information is missing from the modal information group, wherein the modal information group further comprises second modal information (section 3.1: the determination of the test set is based on the fact that a subject I is composed of two modalities, which is seen as an attribute, the determination of the subjects where modality y is missing, which is seen as the first modal information. The modality x is seen as the second modal information);
extract a feature vector of the second modal information (section 3.6: MRI modality features);
determine a target feature vector of the first modal information based on a preset feature vector mapping relationship and the feature vector of the second modal information (section 3.6, second paragraph, the target feature vector is seen as the values of voxels in the PET modal)….
Cai does not appear to expressly teach a computing device comprising: a processor and a memory, wherein the memory is configured to store computer program instructions; and the computer program instructions, upon being executed by the processor, instruct the processor to.
Wang Shaonan teaches a computing device comprising: a processor and a memory, wherein the memory is configured to store computer program instructions; and the computer program instructions, upon being executed by the processor, instruct the processor to (Title and abstract. Paragraph 152).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having the multi-modality missing data completion of Cai to include the multi-modal techniques of Wang Shaonan to improve accuracy (see Wang Shaonan at paragraph 4).
Cai does not appear to expressly teach the modal information group is a video, the first modal information is of a text type, the second modal information comprises modal information of a speech type and modal information of an image type.
Wang Shaonan teaches the modal information group is a video, the first modal information is of a text type, the second modal information comprises modal information of a speech type and modal information of an image type (Paragraph 15, "training dataset comprises a plurality of feature vectors computed for the plurality of audio-segments of the plurality of speakers extracted for the entire video". The described entire video reads on the claimed modal information group. The plurality of audio-segments of the plurality of speakers reads on the second modal information. Paragraph 6, "colored text representation in association with a video-segment corresponding to the audio-segment". video-segment corresponding to the audio-segment reads on image type. The colored text reads on the first model information).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having the multi-modality missing data completion of Cai to include the automated subtitle generation techniques of Aronowitz to help a hearing-impaired person to understand who is speaking (see Aronowitz at paragraph 3).
As to dependent claim 10, Cai further teaches the determining the target feature vector of the first modal information further comprises: determining a candidate feature vector of the first modal information based on the feature vector mapping relationship and the feature vector of the second modal information (Figure 2, predicted PET G(x).); and determining the target feature vector of the first modal information based on the candidate feature vector of the first modal information (Figure 2, PET after concatenation is no longer just a predicted candidate).
As to dependent claim 11, Cai further teaches the determining a target feature vector of the first modal information further comprises: determining the target feature vector of the first modal information based on a preset machine learning model and the feature vector of the second modal information, wherein the machine learning model learns the feature vector mapping relationship and is used to output a feature vector of other modal information based on an input feature vector of modal information (Figure 2 shows the preset machine learning model determining the PET modality information based on MRI x (i.e., G(x)). Learning occurs with the discriminator feedback loss).
As to dependent claim 12, Cai further teaches the attribute of the modal information group comprises at least one of the following: a quantity of pieces of modal information in the modal information group, and a data volume of each piece of modal information in the modal information group (Section 3.1, "The training set consists of subjects with both x and y as {xi ,yi , ℓi }N i=1 while the test set consists of subjects with only x").
As to dependent claim 13, Cai further teaches the attribute of the modal information group comprises at least one of the following: a quantity of pieces of modal information in the modal information group, and a data volume of each piece of modal information in the modal information group (Section 3.1, "The training set consists of subjects with both x and y as {xi ,yi , ℓi }N i=1 while the test set consists of subjects with only x").
As to dependent claim 14, Cai further teaches obtain first auxiliary information, and determining the attribute of the modal information group based on the first auxiliary information, wherein the first auxiliary information indicates at least one of the following: a quantity of pieces of modal information in the modal information group, and a data volume of each piece of modal information in the modal information group; determine the attribute of the modal information group based on preset second auxiliary information, wherein the preset second auxiliary information indicates at least one of the following: a quantity of pieces of modal information in any obtained modal information group, and a data volume of each piece of modal information in the any obtained modal information group; or determine the attribute of the modal information group based on an attribute of another modal information group, wherein another modal information group is a modal information group obtained before the modal information group is obtained (Where the first auxiliary information and second auxiliary information are not distinct from the attribute, then Section 3.1, "The training set consists of subjects with both x and y as {xi ,yi , ℓi }N i=1 while the test set consists of subjects with only x").
As to dependent claim 15, Wang Shaonan further teaches the modal information group further comprises third modal information; and the computer program instructions, upon being executed by the processor, further instruct the processor to: extract a feature vector of the third modal information; and determine the target feature vector of the first modal information based on the feature vector mapping relationship, the feature vector of the third modal information, and the feature vector of the second modal information (The third modal information can be seen as audio model information. Since the vector mapping model is trained using audio modal vector (see paragraph 16), it is considered that the target feature vector of the first model information (visual) is also based on the third modal information).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having the multi-modality missing data completion of Cai to include the multi-modal techniques of Wang Shaonan to improve accuracy (see Wang Shaonan at paragraph 4).
As to dependent claim 16, Wang Shaonan further teaches the determining the target feature vector of the first modal information based on the preset feature vector mapping relationship, the feature vector of the third modal information, and the feature vector of the second modal information comprises: determining another candidate feature vector of the first modal information based on the feature vector mapping relationship and the feature vector of the third modal information; and determining the target feature vector of the first modal information based on the candidate feature vector of the first modal information and the another candidate feature vector of the first modal information (Where the another candidate feature vector is not distinct from the candidate feature vector, then The third modal information can be seen as audio model information. Since the vector mapping model is trained using audio modal vector (see paragraph 16), it is considered that the target feature vector of the first model information (visual) is also based on the third modal information).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having the multi-modality missing data completion of Cai to include the multi-modal techniques of Wang Shaonan to improve accuracy (see Wang Shaonan at paragraph 4).
As to dependent claim 17, Cai further teaches each piece of modal information comprised in the modal information group has a different type (section 3.6, MRI and PET are seen as two different modality data types).
As to independent claim 18, Cai teaches:
obtain a modal information group, wherein the modal information group comprises at least two pieces of modal information (section 3 .1, a modal information group is seen as subject I which is composed of two modalities {x, y});
determine, based on an attribute of the modal information group, that a part or all of first modal information is missing from the modal information group, wherein the modal information group further comprises second modal information (section 3. 1: the determination of the test set is based on the fact that a subject I is composed of two modalities, which is seen as an attribute, the determination of the subjects where modality y is missing, which is seen as the first modal information. The modality x is seen as the second modal information);
extract a feature vector of the second modal information (section 3. 6: MRI modality features);
determine a target feature vector of the first modal information based on a preset feature vector mapping relationship and the feature vector of the second modal information (section 3. 6, second paragraph, the target feature vector is seen as the values of voxels in the PET modal)….
Cai does not appear to expressly teach a computing device cluster comprising: a plurality of computing devices, wherein each computing device comprises a processor and a memory, and a memory in at least one computing device is configured to store computer program instructions; and the computer program instructions, upon being executed by processors in the plurality of computing devices, instruct the processors to.
Wang Shaonan teaches a computing device cluster comprising: a plurality of computing devices, wherein each computing device comprises a processor and a memory, and a memory in at least one computing device is configured to store computer program instructions; and the computer program instructions, upon being executed by processors in the plurality of computing devices, instruct the processors to (Title and abstract. Paragraph 152).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having the multi-modality missing data completion of Cai to include the multi-modal techniques of Wang Shaonan to improve accuracy (see Wang Shaonan at paragraph 4).
Cai does not appear to expressly teach the modal information group is a video, the first modal information is of a text type, the second modal information comprises modal information of a speech type and modal information of an image type.
Wang Shaonan teaches the modal information group is a video, the first modal information is of a text type, the second modal information comprises modal information of a speech type and modal information of an image type (Paragraph 15, "training dataset comprises a plurality of feature vectors computed for the plurality of audio-segments of the plurality of speakers extracted for the entire video". The described entire video reads on the claimed modal information group. The plurality of audio-segments of the plurality of speakers reads on the second modal information. Paragraph 6, "colored text representation in association with a video-segment corresponding to the audio-segment". video-segment corresponding to the audio-segment reads on image type. The colored text reads on the first model information).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, having the multi-modality missing data completion of Cai to include the automated subtitle generation techniques of Aronowitz to help a hearing-impaired person to understand who is speaking (see Aronowitz at paragraph 3).
As to dependent claim 19, Cai further teaches the determining the target feature vector of the first modal information further comprises: determining a candidate feature vector of the first modal information based on the feature vector mapping relationship and the feature vector of the second modal information (Figure 2, predicted PET G(x)); and determining the target feature vector of the first modal information based on the candidate feature vector of the first modal information (Figure 2, PET after concatenation is no longer just a predicted candidate).
As to dependent claim 20, Cai further teaches the determining the target feature vector of the first modal information further comprises: determining the target feature vector of the first modal information based on a preset machine learning model and the feature vector of the second modal information, wherein the machine learning model learns the feature vector mapping relationship and is used to output a feature vector of other modal information based on an input feature vector of modal information (Figure 2 shows the preset machine learning model determining the PET modality information based on MRI x (i.e., G(x)). Learning occurs with the discriminator feedback loss).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Casey R. Garner whose telephone number is 571-272-2467. The examiner can normally be reached Monday to Friday, 8am to 5pm, Eastern Time.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Alexey Shmatov can be reached on 571-270-3428. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from Patent Center and the Private Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from Patent Center or Private PAIR. Status information for unpublished applications is available through Patent Center and Private PAIR to authorized users only. Should you have questions about access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) Form at https://www.uspto.gov/patents/uspto-automated- interview-request-air-form.
/Casey R. Garner/Primary Examiner, Art Unit 2123