CTNF 19/000,935 CTNF 89039 DETAILED ACTION Notice of Pre-AIA or AIA Status 07-03-aia AIA 15-10-aia The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA. Claim Rejections - 35 USC § 103 07-06 AIA 15-10-15 In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status. 07-20-aia AIA The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action: A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made. 07-21-aia AIA Claim (s) 19-22, 26-29 and 33-36 are rejected under 35 U.S.C. 103 as being unpatentable over Mauser (US 2016/0371256) in view of Aggarwal (US 2020/0380027) . With respect to claim 33 (similarly claims 19 and 26) , Mauser teaches a system (e.g. a computing network 100 Figs 1-2 [0034]), comprising: one or more computers (e.g. server 104 and mobile 108 Fig 1 [0034]) and one or more storage devices (e.g. a memory 208 Fig 2 [0037]) on which are stored instructions that are operable (e.g. configured to store information/instructions at the server 104 [0037]), when executed by the one or more computers, to cause the one or more computers to perform operations comprising: obtaining a first image (e.g. obtaining a first image 300 Fig 3A [0042], [0047]) that includes first text written in a source language (e.g. that includes first text “RIND” written in German, [0042]); extracting (e.g. extracting [0039], [0043]), using a trained feature extractor and the first image (e.g. using server 104 and image 300 of Fig 3A), a first set of image features representing the first text present in the first image (e.g. non-textual context information [0039], [0043] i.e. a first set of image features representing the first text present in the first image); and generating (e.g. generating/obtaining [0044]), using a trained decoder to which the first set of image features are provided as an input (e.g. using trained server 104 as suggested in [0039] to which the first set of image features are provided as an input), a second text (e.g. obtaining “cow” Fig 3B [0044]) in a target language that is a predicted translation of the first text in the source language (e.g. that is in English and is predicted to be a translation of “RIND”, see Fig 3B [0044]), Even though Mauser teaches the trained feature extractor and the trained decoder i.e. server 104 in [0039], he fails to teach wherein the trained feature extractor and the trained decoder are trained using a single loss function. Aggarwal teaches wherein the trained feature extractor and the trained decoder are trained using a single loss function (e.g. a model is trained using a single loss function 132, see Fig 3 [0013] and [0062]). Mauser and Aggarwal are analogous art because they all pertain to processing/translating text from images. Therefore, it would have been obvious to people having ordinary skill in the art before the effective filing date of the claimed invention to modify Mauser with the loss function 132 Fig 3 of Aggarwal to include: wherein the trained feature extractor and the trained decoder are trained using a single loss function, as suggested by Aggarwal in Fig 3 [0013], [0062]. The benefit of the modification would be to achieve a better text classification, as suggested by Aggarwal in [0007]. With respect to claim 34 (similarly claims 20 and 27) , Mauser teaches the system of claim 33, wherein the operations comprise: training the feature extractor using a set of input training images that depict training text in the source language and corresponding sets of training image features (e.g. the server 104 can perform machine learning to train a classifier using labeled training sets and then use the trained classifier to identify the non-textual context information [0039]), wherein each set of training image features is a description of a portion of an input image in which the training text is depicted (e.g. each set of training image features is a description of a portion of the input image in which the training text is depicted, as suggested in Figs 3-4 [0042]-[0046]). With respect to claim 35 (similarly claims 21 and 28) , Mauser teaches the system of claim 33, wherein the trained decoder includes a text-to-text translation model (e.g. the trained server 104 Figs 1-2 includes a text-to-text translation model, as suggested in Figs 3-4 [0042]-[0046]). With respect to claim 36 (similarly claims 22 and 29) , Mauser teaches the system of claim 35, wherein the operations comprise training the decoder (e.g. training server 104 Figs 1-2 as suggested in [0039]), and wherein the training includes: training the text-to-text translation model to translate text written in the source language into text in the target language (e.g. training the text-to-text model, as suggested in [0039], to translate “RIND” in German into “beef”/”cow” in English, see Fig 3 [0043]-[0044]), wherein the text-to-text translation model is trained using a set of input training text data in the source language and a corresponding set of output training text data that is a translation of the input training text data from the source language into the target language (e.g. the text-to-text translation model is trained using a set of input training text data in German and a corresponding set of output training text data that is a translation of the input training text data from German into English, as suggested in Figs 3-4 [0042]-[0046]); and training the trained text-to-text translation model to output text data in the target language that is a predicted translation of text represented by an input set of image features that represent text in an input image (e.g. training the trained text-to-text translation model to output text data in English that is a predicted translation of text represented by an input set of image features that represent text in an input image 300, 350 Fig 3, as suggested in [0039], [0043]-[0044]), wherein the trained text-to-text translation model is trained using a set of input training images that depict training text in the source language and a corresponding set of text data that is a translation in the target language of the training text depicted in the input training images (e.g. the trained text-to-text translation model is trained using a set of input training images that depict training text in German and a corresponding set of text data that is a translation in English of the training text depicted in the input training images, as suggested in Figs 3-4 [0039], [0042]-[0046]) . 07-21-aia AIA Claim (s) 23, 30 and 37 are rejected under 35 U.S.C. 103 as being unpatentable over Mauser (US 2016/0371256) in view of Aggarwal (US 2020/0380027) and further in view of Zhai (US 2021/0215481) . With respect to claim 37 (similarly claims 23 and 30) , Mauser teaches the system of claim 33 including the feature extractor i.e. server 104 Figs 1-2 , However, Mauser fails to teach wherein the feature extractor is a convolution neural network (CNN) with a plurality of layers of at least one, or a combination, of convolution, residual, or pooling. Zhai teaches a feature extractor (e.g. a network layer structure Fig 4 [0059]) is a convolution neural network (CNN) with a plurality of layers of at least one, or a combination, of convolution, residual, or pooling (e.g. is a neural convolutional network with 53 convolutional layers and 22 residual layers, see [0059]). Mauser and Zhai are analogous art because they all pertain to processing images. Therefore, it would have been obvious to people having ordinary skill in the art before the effective filing date of the claimed invention to modify server 104 of Mauser with the network structure of Zhai such that server 104 is a convolution neural network (CNN) with a plurality of layers of at least one, or a combination, of convolution, residual, or pooling, as suggested in Fig 4 [0059] of Zhai. The benefit of the modification would be such that the input image is processed by the convolutional network for output, as suggested by Zhai in [0026] . 07-21-aia AIA Claim (s) 24-25, 31-32 and 38 are rejected under 35 U.S.C. 103 as being unpatentable over Mauser (US 2016/0371256) in view of Aggarwal (US 2020/0380027) and further in view of Xinyuan (NPL: Multi-Encoder-Decoder Transformer for Code-Switching Speech Recognition) . With respect to claim 38 (similarly claims 24 and 31) , Mauser teaches the system of claim 33 , wherein the text-to-text translation model is a transformer neural machine translation model (e.g. performing machine learning to classify non-textual context information as suggested in [0039]). However, Mauser fails to teach wherein the decoder is a multi-layer multi-head transformer decoder. Xinyuan teaches a decoder is a multi- layer multi-head transformer decoder (e.g. The MED Transformer in our experiment contains 12-layer encoder and 6-layer decoder, Section 4.2). Mauser and Xinyuan are analogous art because they all pertain to processing multiple languages. Therefore, it would have been obvious to people having ordinary skill in the art before the effective filing date of the claimed invention to modify the server 104 of Mauser with the teachings of Xinyuan to include: wherein the decoder is a multi- layer multi-head transformer decoder, as suggested by Xinyuan in Section 4.2. The benefit of the modification would be to encapsulate the acoustic and language information jointly in a single network, Xinyuan Introduction. With respect to claim 25 (similarly claims 32) , Mauser teaches the method of claim 21 , wherein the text-to-text translation model is a transformer neural machine translation model (e.g. performing machine learning to classify non-textual context information as suggested in [0039]). However, Mauser fails to teach wherein the decoder is a 6-layer multi-head transformer decoder. Xinyuan teaches a decoder is a 6- layer multi-head transformer decoder (e.g. The MED Transformer in our experiment contains 12-layer encoder and 6-layer decoder, Section 4.2). Mauser and Xinyuan are analogous art because they all pertain to processing multiple languages. Therefore, it would have been obvious to people having ordinary skill in the art before the effective filing date of the claimed invention to modify the server 104 of Mauser with the teachings of Xinyuan to include: wherein the decoder is a 6- layer multi-head transformer decoder, as suggested by Xinyuan in Section 4.2. The benefit of the modification would be to encapsulate the acoustic and language information jointly in a single network, Xinyuan Introduction. Conclusion Any inquiry concerning this communication or earlier communications from the examiner should be directed to IBRAHIM SIDDO whose telephone number is (571)272-4508. The examiner can normally be reached 9:00-5:30PM. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Akwasi M. Sarpong can be reached on 5712703438. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. /IBRAHIM SIDDO/Primary Examiner, Art Unit 2681 Application/Control Number: 19/000,935 Page 2 Art Unit: 2681 Application/Control Number: 19/000,935 Page 3 Art Unit: 2681 Application/Control Number: 19/000,935 Page 4 Art Unit: 2681 Application/Control Number: 19/000,935 Page 5 Art Unit: 2681 Application/Control Number: 19/000,935 Page 6 Art Unit: 2681 Application/Control Number: 19/000,935 Page 7 Art Unit: 2681 Application/Control Number: 19/000,935 Page 8 Art Unit: 2681
Prosecution Insights