DETAILED ACTION
1. This office action is in response to the Application No. 18907466 filed on 05/27/2026. Claims 1-20 are presented for examination and are currently pending.
Notice of Pre-AIA or AIA Status
2. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
3. A request for continued examination under 37 CFR 1.114, including the fee set
forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this
application is eligible for continued examination under 37 CFR 1.114, and the fee set
forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action
has been withdrawn pursuant to 37 CFR 1.114. Applicant’s submission filed on
05/27/2026 has been entered.
Response to Arguments
4. The Applicants argument regarding the prior art have been considered and the Examiner is withdrawing the rejections in the previous Office action because Applicant’s amendment necessitated new grounds of rejection presented in this Office Action.
It is noted that arguments regarding independent claims 1 and 11 have been considered but are moot because new references have now been used to remap the independent claims 1 and 11.
Furthermore, Guo and Shih which were applied in the previous Office Action are still relevant to the instant dependent claims. As a result, their teachings have been used in this Office Action.
The dependent claims 2-10 and 12-20 which depend directly or indirectly from independent claims 1 and 11 are not patentable because the instant claims are still obvious over the prior art of record.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
5. Claims 1, 2, 5, 6 , 8-10, 11, 12, 15, 16, 18-20 are rejected under 35 U.S.C 103 as being unpatentable over Hatamizadeh et al. (US20230145535) in view of Yu et al. (US20240112088 filed 11/27/2023)
Regarding claim 1, Hatamizadeh teaches a system for multimodal data processing and generation using a vector autoencoder (FIG. 39 is a system diagram for an example system for training, adapting, instantiating and deploying machine learning models in an advanced computing pipeline [0052]) and a latent transformer (In at least one embodiment, a decoder transforms a low-dimensional input, such as output of an encoder, to a higher-dimensional output [0065]),
comprising: a computing device comprising at least a memory and a processor (computer system 1300 may include, without limitation, a memory 1320 ... memory 1320 may store instruction(s) 1319 and/or data 1321 represented by data signals that may be executed by processor 1302 [0252]);
a plurality of programming instructions stored in the memory and operable on the processor, wherein the plurality of programming instructions (In at least one embodiment, memory 1320 may store instruction(s) 1319 and/or data 1321 represented by data signals that may be executed by processor 1302 [0252]),
when operating on the processor, cause the computing device to: receive multimodal data comprising at least two different data types selected from: time series data, textual data, image data, audio data, and structured tabular data (an input image 202 and text from a report 204 is projected into [0071] a framework for training one or more neural networks 200 [0070], Fig. 2);
encode each data type into a modality-specific representation using specialized encoders for each data type (image 202 is encoded by image encoder 206 and text report 204 is encoded by language encoder 208, Fig. 2. The Examiner notes the encoder 206 and 208 reads on the claimed encoders that operate on a specific input data. According to the instant specification: “deep learning system comprises a plurality of codeword generator (sometimes referred to as encoders) modules 2926a-c each of which may be specifically configured to operate on a specific input data modality of a plurality of preprocessed multimodal data inputs 2925a-c” [0311]);
fuse the modality-specific representations into a combined representation (fusion by pairwise cross-attention encoder in Fig. 2 and 3) by applying cross-modal attention mechanisms to capture relationships between the different data types (In at least one embodiment, output of these respective portions of a neural network is input to a cross-attention encoder [0089], Fig. 2);
process the discrete latent representation using a transformer to learn relationships (In at least one embodiment, image and language encoders have an architecture enabling them to learn modality-specific (i.e., vision and language) encoding for follow-up learning tasks. In at least one embodiment, a stack of transformer encoders with multi-headed self-attention sub-layers are used [0073])
and generate new discrete representations (In at least one embodiment, for
language embeddings by language encoder 208, diagnostic reports corresponding to these embeddings are first summarized using a sequence-to-sequence model, such as a bidirectional and auto-regressive transformers (“BART”) model, and split into T words {Z1, Z2, . . . , ZT} using a tokenizer, such as a bidirectional encoder representations from transformers (“BERT”) tokenizer [0072]. The Examiner notes T words {Z1, Z2, . . . , ZT} are the new discrete representation);
decode the new discrete representations into output data (In at least one embodiment, this medical interpretation model comprises a neural network 100 that translate learned representations to saliency maps with a dedicated saliency decoder 116 [0065]);
restore information lost during the vector quantization using a neural upsampler (In at least one embodiment, another type of decoder is used to upsample learned representation to a saliency map or other output [0065]) comprising a trained neural network with learnable parameters (wherein the neural network comprises a decoder to generate information indicative of a region [0632]; neural network 100 are trained to perform diagnostic localization [0058]; ... weight parameters and/or input/output data of each layer of a neural network trained or used in conjunction with one or more embodiments during forward propagation of input/output data and/or weight parameters during training [0114]. The Examiner notes this indicates the another type of decoder used to upsample is a neural upsampler comprising a trained neural network),
wherein the neural upsampler receives features derived from each of the at least two different data types and is trained to leverage cross-modal correlations comprising learned relationships (A generated saliency maps is then upsampled to an appropriate image resolution using bilinear interpolation [0082]; In at least one embodiment, a cross-modal learning setting such as this exploits information from both vision and language modalities in a unified manner [0078]) between features of a first one of the at least two different data types and features of a second one of the at least two different data types having a different structural format from the first data type (In at least one embodiment, this model comprises jointly learning visual and textual representations of diagnostic images and diagnostic reports and translating them, using a dedicated decoder, to saliency maps [0060]; diagnostic image 102 ... diagnostic report 106 [0066]); and
Hatamizadeh does not explicitly teach vector quantized variational autoencoder (VQ-VAE), convert the combined representation into a discrete latent representation using vector quantization with a learned codebook, wherein vector quantization comprises mapping continuous latent vectors to nearest discrete codebook vectors via nearest-neighbor lookup; and jointly train the encoding, vector quantization, processing, decoding, and neural upsampling using a combined loss function that includes a vector quantization loss component.
Yu teaches receive multimodal data comprising at least two different data types; encode each data type into a modality-specific representation using specialized encoders for each data type (a set of text and image training examples that each includes a pair of text and image [0088]; The text encoder 402 can receive the input text 334 [0087]; process the plurality of input image patches 12 with a machine-learned image encoder 16 to generate a plurality of image tokens 18 in a latent space [0070]);
a vector quantized variational autoencoder (VQ-VAE) (Another example aspect is directed to a computer-implemented method to perform vector quantization [0022]; The Vector-quantized Variational AutoEncoder (VQVAE) is a CNN-based auto-encoder whose latent space is a matrix of discrete learnable variables [0089]),
convert the combined representation into a discrete latent representation using vector quantization with a learned codebook (an encoder portion configured to encode one or more input image patches into one or more image tokens in a latent space; a quantization portion configured to quantize the one or more image tokens into one or more quantized codes selected from a codebook [0025]; The present disclosure proposes multiple improvements from architecture to codebook learning [0055]),
wherein vector quantization comprises mapping continuous latent vectors to nearest discrete codebook vectors via nearest-neighbor lookup (a visual codebook is learned that snaps a patch embedding to its nearest codebook entry—which is a learned and index-able location in the overall latent space [0104]; some example implementations lookup the closest variable encoded from input on a lower-dimensional lookup space and then project the matched latent code to the high-dimensional embedding space [0094]);
jointly train the encoding, vector quantization, processing, decoding, and neural upsampling using a combined loss function that includes a vector quantization loss component (Some example implementations train with cross-entropy loss and use an int8-quantized Adafactor optimizer [0110]; trained jointly (e.g., using a code prediction loss function) to learn the distribution of tokens over a set of text and image training examples that each includes a pair of text and image [0088]; In some implementations, one or both of the machine-learned image decoder and the codebook were jointly learned with an image encoder model [0031]; Loss balancing weights can be configured with a hyper-parameter sweep to optimize image reconstruction quality [0094]. The Examiner notes that according to the Applicant’s instant specification: “According to an embodiment, the decoders may be trained jointly with the rest of the system, using a combination of reconstruction losses”(US20250190866 [0119])).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of Hatamizadeh to incorporate the teachings of Yu for the benefit of encoding image data to save memory space and network bandwidth and reducing the loss of data experienced when decoding the imagery to a decoded image (Yu [0065]).
Regarding claim 2, Hatamizadeh and Yu teaches the system of claim 1, Hatamizadeh teaches wherein encoding the multimodal data comprises: encoding each data type into a modality-specific representation using specialized encoders (image 202 is encoded by image encoder 206 and text report 204 is encoded by language encoder 208, Fig. 2. The Examiner notes image encoder 206 and language encoder 208 are specialized encoders);
fusing the modality-specific representations into a combined representation (fusion by pairwise cross-attention encoder in Fig. 2 and 3); and
Yu teaches converting the combined representation into discrete codes using vector quantization (a quantization portion configured to quantize the one or more image tokens into one or more quantized codes selected from a codebook [0025]).
The same motivation to combine independent claim 1 applies here.
Regarding claim 5, Hatamizadeh and Yu teaches the system of claim 1, Yu teaches wherein the combined loss function incorporates reconstruction quality across all modalities and latent space consistency (an image quantizer with better reconstruction quality can potentially reduce information loss compared with the original image in pixel space, which is critical for image understanding tasks [0059]).
The same motivation to combine independent claim 1 applies here.
Regarding claim 6, Hatamizadeh and Yu teaches the system of claim 1, Hatamizadeh teaches wherein the computing device is further caused to explore and manipulate the discrete latent representation to generate new or modified multimodal data (A generated saliency maps is then upsampled to an appropriate image resolution using bilinear interpolation [0082]; In at least one embodiment, a saliency map comprises data to indicate notable features of an input, such as features of an image that may be of particular interest [0065]. The Examiner notes that the appropriate image resolution is the modified multimodal data ).
Regarding claim 8, Hatamizadeh and Yu teaches the system of claim 1, Hatamizadeh teaches wherein the multimodal data comprises at least two of: time- series data, textual data, image data, audio data, and structured tabular data (an input image 202 and text from a report 204 is projected into [0071] a framework for training one or more neural networks 200 [0070], Fig. 2).
Regarding claim 9, Hatamizadeh and Yu teaches the system of claim 1, Yu teaches wherein the computing device is further caused to perform conditional generation by adding a condition vector to the input of the transformer (conditioned image generation (e.g., class-conditioned image generation) [0052]; Vector-quantized Image Modeling. Some example implementations can include training a Transformer model to predict rasterized (e.g., 32×32=1024) image tokens autoregressively, where image tokens are encoded by the frozen Stage 1 ViT-VQGAN. ... For conditioned image synthesis (e.g., class-conditioned image synthesis), conditioning data (e.g., a class-id token) can be prepended at the beginning of all image tokens. [0057]).
The same motivation to combine independent claim 1 applies here.
Regarding claim 10, Hatamizadeh and Yu teaches the system of claim 1, Yu teaches wherein the computing device is further caused to quantify uncertainty in the generated output data by using multiple samplings from the discrete latent representation (With a pretrained generative Transformer model, unconditional image generation can be achieved by simply sampling token-by-token from the output softmax distribution. All samples used for both qualitative and quantitative results can be obtained without temperature reduction. The sampled tokens can then be fed into the decoder of ViT-VQGAN to decode output images [0100]; VQGAN relies on top-k and top-p (nucleus) sampling heuristics with a default codebook size of 1024 to obtain best results for image synthesis [0092]).
The same motivation to combine independent claim 1 applies here.
Regarding claim 11, claim 11 is similar to claim 1. It is rejected in same manner and reasoning applying.
Regarding claim 12, claim 12 is similar to claim 2. It is rejected in same manner and reasoning applying.
Regarding claim 15, claim 15 is similar to claim 5. It is rejected in same manner and reasoning applying.
Regarding claim 16, claim 16 is similar to claim 6. It is rejected in same manner and reasoning applying.
Regarding claim 18, claim 18 is similar to claim 8. It is rejected in same manner and reasoning applying.
Regarding claim 19, claim 19 is similar to claim 9. It is rejected in same manner and reasoning applying.
Regarding claim 20, claim 20 is similar to claim 10. It is rejected in same manner and reasoning applying.
6. Claims 3, 4, 13 and 14 are rejected under 35 U.S.C 103 as being unpatentable over Hatamizadeh et al. (US20230145535) in view of Yu et al. (US20240112088 filed 11/27/2023) and further in view of Guo et al. ("MSMC-TTS: Multi-stage multi-codebook VQ-VAE based neural TTS." IEEE/ACM Transactions on Audio, Speech, and Language Processing 31 (2023): 1811-1824, Date of publication 2 May 2023)
Regarding claim 3, Hatamizadeh and Yu teaches the system of claim 1, they do not explicitly teach the limitations of claim 3.
Guo teaches wherein the transformer operates without embedding or positional encoding layers (then fed to 4 FeedForward Transformer blocks. Specifically, the number of heads in multi-head attention is 2, and the feedforward module is composed of two convolutional layers with a kernel size of 3 and a ReLU activation function between them. The Examiner notes that the FeedForward Transformer blocks do not include embedding or positional encoding layers).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of Hatamizadeh and Yu to incorporate the teachings of Guo for the benefit of a system that performs well even with much fewer model parameters or training data, showing lower requirements for the modeling complexity and the data size (Guo, conclusion).
Regarding claim 4, Hatamizadeh and Yu teaches the system of claim 1, Hatamizadeh teaches generating output data for each modality from portions of the continuous latent representation using modality-specific decoders (In at least one embodiment, output of saliency map decoder 212 can include diagnostic localization information 216 and diagnostic indicator information 218 ... In at least one embodiment, a language predictor 214 decoder outputs masked words 222 [0081]; In at least one embodiment, combinations of decoders may be employed to generate a combination of different output types [0065]).
They do not explicitly teach wherein decoding the new discrete representations comprises: converting the new discrete representations into a continuous latent representation
Guo teaches wherein decoding the new discrete representations comprises: converting the new discrete representations into a continuous latent representation (Finally, the output speech sequence xˆ is generated by the decoder D for reconstruction: xˆ = D(z) (pg. 1813, left col., first para.); Specifically, in the quantizer block Qi, e(i) is concatenated with the hidden sequence h(i+1) (except when i = S) from the higher-stage decoder, and then transformed by a projection layer to obtain ˜z(i), pg. 1814, right col., first para.; ˜z refer to latent sequences … after vector quantization “Q” with the codebook c composed of M codewords with the dimension of N, pg. 1813, left col., Fig. 2); and
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of Hatamizadeh and Yu to incorporate the teachings of Guo for the benefit of a system that performs well even with much fewer model parameters or training data, showing lower requirements for the modeling complexity and the data size (Guo, conclusion).
Regarding claim 13, claim 13 is similar to claim 3. It is rejected in same manner and reasoning applying.
Regarding claim 14, claim 14 is similar to claim 4. It is rejected in same manner and reasoning applying.
7. Claims 7 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Hatamizadeh et al. (US20230145535) in view of Yu et al. (US20240112088 filed 11/27/2023) and further in view of Shih et al. (US20210064925)
Regarding claim 7, Hatamizadeh and Yu teaches the system of claim 6, they do not explicitly teach wherein exploring and manipulating the discrete latent representation comprises using techniques including interpolation, extrapolation, and vector arithmetic.
Shih teaches wherein exploring and manipulating the discrete latent representation comprises using techniques including interpolation, extrapolation (achieving high-quality long-range video interpolation and extrapolation through operating on a landmark representation space [0050]), and
vector arithmetic (In at least one embodiment, arithmetic operations on texture data and input geometry data compute pixel color data for each geometric fragment [0329]).
It would have been obvious to a person having ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of Hatamizadeh and Yu to incorporate the teachings of Shih for the benefit of performing inferencing of information, such as speech recognition, or other artificial intelligence services (Shih [0124])
Regarding claim 17, claim 17 is similar to claim 7. It is rejected in same manner and reasoning applying.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MORIAM MOSUNMOLA GODO whose telephone number is (571)272-8670. The examiner can normally be reached Monday-Friday 8:00am-5:00pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michelle T. Bechtold can be reached on (571) 431-0762. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/M.G./Examiner, Art Unit 2148
Prosecution Insights