Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-4, 8-13, and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Huang, Yupan, et al. (“Unifying multimodal transformer for bi-directional image and text generation,” Proceedings of the 29th ACM International Conference on Multimedia. 2021; hereinafter “Huang”) in view of Montalt et al. (U.S. 2024/0378503, hereinafter “Montalt”).
Regarding Claim 1, Huang teaches a computer-implemented method (sections 1 and 3—the unified multimodal framework model performs a method of image and text generation) comprising:
receiving one of 1) an input image, 2) input text, or 3) an input image/text pair (sections 3-3.2 and fig. 2—the framework receives an input image, an input text, or an input image/text pair);
extracting features from the received one of 1) the input image, 2) the input text, or 3) the input image/text pair (sections 3-3.2 and fig. 2—feature and token representations of the image and text are generated);
generating, based on the extracted features and using a trained machine learning based model, one of A) text, B) an image, or C) an image/text pair for the received one of 1) the input image, 2) the input text, or 3) the input image/text pair respectively (sections 2.3 and 3-3.2 and fig. 2—the framework performs text-to-image generation and image-to-text generation, thus generating one of text or an image); and
outputting the generated one of A) the text, B) the image, or C) the image/text pair (sections 3-3.2—the generated image or text is output, as further evidenced by the results reported in the experiments in section 5).
Huang does not specifically teach that the text and images are medical text and images, and that the generated text and images are synthetic medical text and images. However, Montalt teaches text and images that are medical text and images, and generating synthetic medical images ([0060] – [0062] and [0137] – [0138]).
All of the claimed elements were known in Huang and Montalt and could have been combined by known methods with no change in their respective functions. It therefore would have been obvious to a person of ordinary skill in the art at the time of filing of the applicant’s invention to combine the medical text, medical images, and synthetic medical images of Montalt with the text, images, and image/text pairs of Huang to yield the predictable result of receiving one of 1) an input medical image, 2) input medical text, or 3) an input medical image/text pair; extracting features from the received one of 1) the input medical image, 2) the input medical text, or 3) the input medical image/text pair; generating, based on the extracted features and using a trained machine learning based model, one of A) synthetic medical text, B) a synthetic medical image, or C) a synthetic medical image/text pair for the received one of 1) the input medical image, 2) the input medical text, or 3) the input medical image/text pair respectively; and outputting the generated one of A) the synthetic medical text, B) the synthetic medical image, or C) the synthetic medical image/text pair. One would be motivated to make this combination for the purpose of satisfying the need for sufficient training data to train machine learning models that is not subject to data privacy restrictions (Montalt, [0007]).
Regarding Claim 9, Huang teaches an apparatus (sections 1 and 3, and fig. 2—the unified multimodal framework model is an apparatus) comprising:
means for receiving one of 1) an input image, 2) input text, or 3) an input image/text pair (sections 3-3.2 and fig. 2—the framework receives an input image, an input text, or an input image/text pair);
means for extracting features from the received one of 1) the input image, 2) the input text, or 3) the input image/text pair (sections 3-3.2 and fig. 2—feature and token representations of the image and text are generated);
means for generating, based on the extracted features and using a trained machine learning based model, one of A) text, B) an image, or C) an image/text pair for the received one of 1) the input image, 2) the input text, or 3) the input image/text pair respectively (sections 2.3 and 3-3.2 and fig. 2—the framework performs text-to-image generation and image-to-text generation, thus generating one of text or an image); and
means for outputting the generated one of A) the text, B) the image, or C) the image/text pair (sections 3-3.2—the generated image or text is output, as further evidenced by the results reported in the experiments in section 5).
Huang does not specifically teach that the text and images are medical text and images, and that the generated text and images are synthetic medical text and images. However, Montalt teaches text and images that are medical text and images, and generating synthetic medical images ([0060] – [0062] and [0137] – [0138]).
All of the claimed elements were known in Huang and Montalt and could have been combined by known methods with no change in their respective functions. It therefore would have been obvious to a person of ordinary skill in the art at the time of filing of the applicant’s invention to combine the medical text, medical images, and synthetic medical images of Montalt with the text, images, and image/text pairs of Huang to yield the predictable result of means for receiving one of 1) an input medical image, 2) input medical text, or 3) an input medical image/text pair; means for extracting features from the received one of 1) the input medical image, 2) the input medical text, or 3) the input medical image/text pair; means for generating, based on the extracted features and using a trained machine learning based model, one of A) synthetic medical text, B) a synthetic medical image, or C) a synthetic medical image/text pair for the received one of 1) the input medical image, 2) the input medical text, or 3) the input medical image/text pair respectively; and means for outputting the generated one of A) the synthetic medical text, B) the synthetic medical image, or C) the synthetic medical image/text pair. One would be motivated to make this combination for the purpose of satisfying the need for sufficient training data to train machine learning models that is not subject to data privacy restrictions (Montalt, [0007]).
Regarding Claims 2 and 10, Huang/Montalt teaches wherein:
receiving one of 1) an input medical image, 2) input medical text, or 3) an input medical image/text pair comprises:
receiving one of the input medical image or the input medical image/text pair (Huang, section 3.1-3.2 and fig. 2—an input image or an image/text pair is received); and
extracting features from the received one of 1) the input medical image, 2) the input medical text, or 3) the input medical image/text pair comprises:
extracting, from one of the input medical image or an image of the input medical image/text pair, one or more of dense features, tokens embeddings of textual labels of anatomical objects of interest identified in the one of the input medical image or the image of the input medical image/text pair, region features of the anatomical objects of interest, or region coordinates of the anatomical objects of interest (Huang, section 3.2 and fig. 2—dense features are extracted, and tokenization is performed to generate token embeddings).
Regarding Claims 3 and 11, Huang/Montalt teaches wherein:
receiving one of 1) an input medical image, 2) input medical text, or 3) an input medical image/text pair comprises:
receiving one of the input medical text or the input medical image/text pair (Huang, section 3.1-3.2 and fig. 2—an input text or an image/text pair is received); and
extracting features from the received one of 1) the input medical image, 2) the input medical text, or 3) the input medical image/text pair comprises:
extracting an SLS (structured language sequence) representation from one of the input medical text or text of the input medical image/text pair; and encoding the SLS representation into token embeddings (Huang, section 3.1 and fig. 2—sequences are generated, which are encoded into tokens. The sequences generated from the text therefore comprise structured language sequences).
Regarding Claims 4 and 12, Huang/Montalt teaches wherein generating, based on the extracted features and using a trained machine learning based model, one of A) synthetic medical text, B) a synthetic medical image, or C) a synthetic medical image/text pair for the received one of 1) the input medical image, 2) the input medical text, or 3) the input medical image/text pair respectively comprises:
generating synthetic image features using the trained machine learning based model (Montalt, [0069] – [0070] and [0135] – [0136]); and
generating the synthetic medical image based on the synthetic image features using a machine learning based image generator network (Montalt, section 2.3).
Regarding Claim 8, Huang/Montalt teaches wherein the trained machine learning based model is a multimodal transformer network (Huang, section 3.2 and fig. 2).
Regarding Claim 13, Huang teaches a non-transitory computer-readable storage medium comprising instructions (section 5.1—the experimental setup of executing the model with a dataset implies a non-transitory computer-readable storage medium comprising instructions). Huang/Montalt teach the instructions, when executed by a computer, cause the computer to carry out operations of the present claim in the same manner as for claim 1, above.
receiving one of 1) an input medical image, 2) input medical text, or 3) an input medical image/text pair;
extracting features from the received one of 1) the input medical image, 2) the input medical text, or 3) the input medical image/text pair;
generating, based on the extracted features and using a trained machine learning based model, one of A) synthetic medical text, B) a synthetic medical image, or C) a synthetic medical image/text pair for the received one of 1) the input medical image, 2) the input medical text, or 3) the input medical image/text pair respectively; and
outputting the generated one of A) the synthetic medical text, B) the synthetic medical image, or C) the synthetic medical image/text pair.
Regarding Claim 17, Huang teaches a computer-implemented method (sections 1 and 3—the unified multimodal framework model performs a method of image and text generation) comprising:
receiving a training image/text pair (sections 2.3 and 4, and fig. 2—an input image/text pair is received to jointly train an image-to-text generator and a text-to-image generator in the framework);
training a machine learning based model for generating text and an image based on the training image/text pair (sections 2.3 and 4, and fig. 2—the framework model is trained for generating text and images based on the image/text pair); and
outputting the trained machine learning based model (section 4—a trained model is output, as further evidenced by the experiments in section 5).
Huang does not specifically teach that the image/text pair is a medical image/text, and that the generated text and images are synthetic medical text and images. However, Montalt teaches text and images that are medical text and images, and generating synthetic medical images ([0060] – [0062] and [0137] – [0138]).
All of the claimed elements were known in Huang and Montalt and could have been combined by known methods with no change in their respective functions. It therefore would have been obvious to a person of ordinary skill in the art at the time of filing of the applicant’s invention to combine the medical text, medical images, and synthetic medical images of Montalt with the image/text pairs, image generation, and text generation of Huang to yield the predictable result of receiving a training medical image/text pair; and training a machine learning based model for generating synthetic medical text and a synthetic medical image based on the training medical image/text pair. One would be motivated to make this combination for the purpose of satisfying the need for sufficient training data to train machine learning models that is not subject to data privacy restrictions (Montalt, [0007]).
Allowable Subject Matter
Claims 5-7, 14-16, and 18-20 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
None of the prior art of record teaches all of the limitations of claims 5, 14, and 18 in their respective contexts. In particular, the prior art of record does not teach “during a second training stage: training the machine learning based model using 1) a modified version of the training medical image and 2) the training medical text, and training the machine learning based model using 1) a modified version of the training medical text and 2) the training medical image.” Although Montalt teaches generating synthetic training medical images, it does not train a machine learning model using both the combinations of modified and original training images/texts as recited. No other prior art of record teaches these limitations either.
Claims 6-7, 15-16, and 19-20 recite allowable subject matter at least by virtue of their dependence on claims 5, 14, and 18, respectively.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant’s disclosure. This art includes:
Moon, Jong Hak, et al. (“Multi-modal understanding and generation for medical images and text via vision-language pre-training,” IEEE Journal of Biomedical and Health Informatics 26.12 (2022): 6070-6080) teaches a machine learning system that uses multi-modal images and text for medical diagnosis and reporting, but does not generate synthetic data or train the model using synthetic data
Zhou, Hong-Yu, et al. (“A transformer-based representation-learning model with unified processing of multimodal input for clinical diagnostics,” Nature biomedical engineering 7.6 (2023): 743-755) teaches a transformer model with multi-modal input that received text, images, and other modalities to perform unified medical diagnosis; it does not generate images or synthetic data
Li, Qing, et al. (“Towards a unified foundation model: Jointly pre-training transformers on unpaired images and text,” arXiv preprint arXiv:2112.07074 (2021)) teaches a unified transformer model with a modality-specific tokenizer and task-specific attention heads; it is not specific to medical applications
Selivanov, Alexander, et al. (“Medical image captioning via generative pretrained transformers,” Scientific Reports 13.1 (2023): 4171) teaches transformer models that caption medical images using input images and structured patient data; it does not generate images or synthetic data
Pan, Shaoyan, et al. (“2D medical image synthesis using transformer-based denoising diffusion probabilistic model,” Physics in Medicine & Biology 68.10 (2023): 105004) teaches a transformer-based diffusion model that generates synthetic medical images from existing images; it does not include text input
Any inquiry concerning this communication or earlier communications from the examiner should be directed to HAL W SCHNEE whose telephone number is (571) 270-1918. The examiner can normally be reached M-F 7:30 a.m. - 6:00 p.m.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Michael Huntley can be reached at 303-297-4307. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/HAL SCHNEE/ Primary Examiner, Art Unit 2129
Prosecution Insights