DETAILED ACTION
1. This action is responsive to Application no.19/043,365 filed 1/31/2025. All claims have been examined and are currently pending.
Notice of Pre-AIA or AIA Status
2. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
3. The information disclosure statement (IDS) submitted is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Specification
4. The title of the invention is not descriptive. A new title is required that is clearly indicative of the invention to which the claims are directed.
Claim Rejections - 35 USC § 101
5. 35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
6. Claims 11-17 are non-statutory under the most recent interpretation of the Interim Guidelines regarding 35 U.S.C.101 because: the (non-transient) computer-readable storage medium claimed is not positively disclosed in the specification as a statutory only embodiment (Application paragraph 0124). The broadest reasonable interpretation of a claim drawn to a computer readable medium (also called machine readable medium and other such variations) typically covers forms of non-transitory tangible media and transitory propagating signals per se in view of the ordinary and customary meaning of computer readable media, particularly when the specification is silent. See MPEP 2111.01. When the broadest reasonable interpretation of a claim covers a signal per se, the claim must be rejected under 35 U.S.C. § 101 as covering non-statutory subject matter. See In re Nuijten, 500 F.3d 1346, 1356-57 (Fed. Cir. 2007) transitory embodiments are not directed to statutory subject matter) and Interim Examination Instructions for Evaluating Subject Matter Eligibility Under 35 U.S.C. § 101, Aug. 24, 2009; p. 2.
To overcome this rejection, the claim may be amended to recite "a non-transitory computer-readable storage medium ".
Claim Rejections - 35 USC § 102
7. In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
8. The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
9. Claims 1, 11, 18 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Caglayan (Caglayan, Ozan, et al. "Probing the need for visual context in multimodal machine translation." Proceedings of the 2019 conference of the north American chapter of the association for computational linguistics: Human language technologies, volume 1 (long and short papers). 2019.).
Regarding claim 1 Caglayan teaches A method of text translating (Abstract multimodal machine translation), comprising:
obtaining a to-be-translated text, image information associated with the to-be-translated text, and an initial translation corresponding to the to-be-translated text (Section 3 Dataset: English-French, Multi30K; Visual Features;
Table 5 SRC; NMT); and
inputting the to-be-translated text, the image information, and the initial translation into a trained text translation model to obtain a target translation and target description information corresponding to the to-be-translated text
(4160 Section 3 Dataset; Visual Features; Models;
4161 1st column: MMT model; textual and visual context vectors; encoder and decoder;
Results: train all systems
Table 5 MMT),
wherein, the text translation model is configured to obtain first image description information corresponding to the image information based on the image information, and correct the initial translation based on the first image description information and the to-be-translated text to obtain the target translation, and the target description information is used to describe a reason for correcting the initial translation to the target translation (Abstract: multimodal machine translation; combine visual and textual information in order to ground translations; probe the contribution of the visual modality to state-of-the-art MMT models; models are capable of leveraging the virtual input to generate better translations; figure 2; Table 5 REF
Table 5
PNG
media_image1.png
363
634
media_image1.png
Greyscale
Examples pages 4166-4167)
Regarding claim 11 Caglayan teaches A non-transient computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processing apparatus, implements a method of text translating, which comprises
obtaining a to-be-translated text, image information associated with the to-be-translated text, and an initial translation corresponding to the to-be-translated text; and
inputting the to-be-translated text, the image information, and the initial translation into a trained text translation model to obtain a target translation and target description information corresponding to the to-be-translated text,
wherein, the text translation model is configured to obtain first image description information corresponding to the image information based on the image information, and correct the initial translation based on the first image description information and the to-be-translated text to obtain the target translation, and the target description information is used to describe a reason for correcting the initial translation to the target translation.
Claim recites limitations similar to claim 1 and is rejected for similar rationale and reasoning
(4161 4 Results; 4163 Discussion and Conclusions - performance of MMT, which would be accomplished using processing apparatus)
Regarding claim 18 Caglayan teaches An electronic device, comprising:
at least one storage apparatus having at least one computer program stored thereon; and
at least one processing apparatus configured to execute the at least one computer program in the at least one storage apparatus to implement a method of text translating, which comprises
obtaining a to-be-translated text, image information associated with the to-be-translated text, and an initial translation corresponding to the to-be-translated text; and
inputting the to-be-translated text, the image information, and the initial translation into a trained text translation model to obtain a target translation and target description information corresponding to the to-be-translated text,
wherein, the text translation model is configured to obtain first image description information corresponding to the image information based on the image information, and correct the initial translation based on the first image description information and the to-be-translated text to obtain the target translation, and the target description information is used to describe a reason for correcting the initial translation to the target translation.
Claim recites limitations similar to claim 1 and is rejected for similar rationale and reasoning
(4161 4 Results; 4163 Discussion and Conclusions - performance of MMT, which would be accomplished using processing apparatus)
Allowable Subject Matter
10. Claims 2-10, 12-17, 19-20 objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The references discuss multimodal machine translation with image information for improved translation:
Kang, L – BigVideo;
Elliot, “Multi30K: Multilingual English-German Image Descriptions”: 3.2: The Multi30K dataset makes it possible to further develop machine translation in a setting where multimodal data, such as images or video, are observed along side text. The potential advantages of using multi modal information for machine translation include the ability to better deal with ambiguous source text and to avoid (untranslated) out-of-vocabulary words in the target language
Li “On Vision Features in Multimodal Machine Translation”
Abstract: We develop a selective attention model to study the patch-level contribution of an image in MMT. We find that stronger vision models are helpful for learning translation from the visual modality
Ive, “Distilling Translation with Visual Awareness”
Yao- “Multimodal Transformer for Mulitmodal Machine Translation”
Abstract: Multimodal Machine Translation (MMT) aims to introduce information from other modality, generally static images, to improve the translation quality. Previous works propose various incorporation methods, but most of them do not consider the relative importance of multiple modalities. In MMT, equally treating text and images may encode too much irrelevant information from images which may introduce noise. In this paper, we propose the multi modal self-attention in Transformer to solve the issues above. The proposed method learns the representations of images based on the text, which avoids encoding irrelevant information in images.
Gronroos, “The MeMAD Submission to the WMT18 Multimodal Translation Task”
Kwon, “A text-based visual context modulation neural model for multimodal machine translation”
But do not specifically teach:
(claim 2) wherein the text translation model comprises:
a feature extraction module, configured to extract an encoding feature in a text space from the image information;
an embedding layer, configured to obtain a corresponding text feature according to the to-be-translated text and the initial translation; and
a large language model, configured to obtain the first image description information according to the encoding feature, and obtain the target translation and the target description information according to the first image description information and the text feature.
and
(claim 5) wherein the trained text translation model is obtained by the following steps:
obtaining a first training sample, wherein the first training sample is a sample text carrying a first label, a first sample image corresponding to the sample text, and a sample translation corresponding to the sample text, the first label is a first translation and first description information corresponding to the sample text, and the first description information is used to describe a reason for correcting the sample translation to the first translation;
inputting the first training sample into an initial text translation model to obtain a second translation and second description information output by the initial text translation model; and
adjusting a parameter of the initial text translation model based on a first loss between the second translation and the first translation and a second loss between the first description information and the second description information to obtain the trained text translation model.
Conclusion
11. The prior art made of record and not relied upon is considered pertinent to applicant's disclosure: See PTO-892.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SHAUN A ROBERTS whose telephone number is (571)270-7541. The examiner can normally be reached Monday-Friday 9-5 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Andrew Flanders can be reached on 571-272-7516. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov.
For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SHAUN ROBERTS/Primary Examiner, Art Unit 2655
Prosecution Insights