DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
The amendment filed 04/30/2026 has been entered. Applicant’s amendments to the claims have overcome each and every objection previously set forth in the Non-Final Office Action mailed 03/10/2026. Claims 1-20 remain pending in the application.
Response to Arguments
Applicant’s arguments have been considered but are moot because the new ground of rejection for claims 1, 8, and 15 does not rely on any combination of references applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Specification
The disclosure is objected to because of the following informalities: The specification filed 04/30/2026 indicates corrections to be made to para 44-45. This should instead be para 43-44, as indicated in the Non-Final Office Action mailed 03/10/2026. Appropriate correction is required.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-7 are rejected under 35 U.S.C. 103 as being unpatentable over Yang et al. (cited in IDS - Yang, C., An, Z., Huang, L., Bi, J., Yu, X., Yang, H., ... & Xu, Y. (2023). CLIP-KD: An Empirical Study of CLIP Model Distillation. arXiv preprint arXiv:2307.12732.), hereinafter Yang, in view of Dong et al. (cited in IDS - Dong, X., Bao, J., Zheng, Y., Zhang, T., Chen, D., Yang, H., ... & Yu, N. (2023). Maskclip: Masked self-distillation advances contrastive language-image pretraining. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition (pp. 10995-11005)), hereinafter Dong, in further view of Zhang et al. (Zhang, S., Liu, H., & He, K. (2023). Knowledge distillation via token-level relationship graph. arXiv preprint arXiv:2306.12442.), hereinafter Zhang, and Singh et al. (Singh, P., De Clercq, O., Lefever, E. (2023). Distilling Monolingual Models from Large Multilingual Transformers. Electronics 2023, 12, 1022. https://doi.org/10.3390/electronics12041022), hereinafter Singh.
Regarding claim 1, Yang teaches a computer-implemented method (Yang, pg. 5, Training details: “All experiments are ran over 8 V100 GPUs”) comprising:
generating, utilizing a vision encoder of a vision-language model (Yang, Student visual encoder of the Masked Feature Distillation, MFD, model in FIG 1c attached below), an image embedding in a unified embedding space of the vision-language model (Yang, see image embedding in FIG 1c; section 3.1 on pg. 2: “CLIP performs an image-text alignment task to push the paired image-text close and unpaired ones apart in the feature embedding space”) from a masked digital image comprising a digital image with one or more masked patches (Yang, masked image input into MFD in FIG 1c; section 3.2.3 on pg. 3-4: “Masked Feature Distillation (MFD) uses masked images as the input to a student”);
generating, utilizing a text encoder of the vision-language model (Yang, Student text encoder of the MFD model in FIG 1c), a text embedding in the unified embedding space (Yang, see text embedding in FIG 1c; see also section 3.1 on pg. 2 cited above) representing a text phrase comprising a text description of the digital image (Yang, text input into MFD in FIG 1c);
generating, utilizing a pretrained model, a teacher text embedding of the text description (Yang, section 4.3 on pg. 6: “Given the pretrained teacher CLIP model, we distill several light-weight student CLIP models with various architectures.”; see Teacher text encoder and embedding in FIG 1c); and
modifying parameters of the vision-language model according to a loss between the teacher text embedding and the text embedding generated by the text encoder (Yang, section 3.2.3 on pg. 4: “we utilize MSE loss to align the student’s and teacher’s visual and text embeddings.”; see sk difference value calculated in the total loss).
PNG
media_image1.png
384
745
media_image1.png
Greyscale
Yang fails to explicitly teach (1) a text embedding comprising one or more tokens in the unified embedding space representing a masked text phrase comprising a text description of the digital image with one or more masked tokens; (2) wherein the pretrained model is a pretrained large language model; (3) wherein the teacher text embedding comprises token-level embeddings of the text description; and (4) a per-token masked distillation loss generated by performing a per-token projection between each token within the teacher text embedding and each token within the text embedding (emphasis added).
However, Dong similarly teaches a CLIP (Contrastive Language Image Pre-training) model (Dong, MaskCLIP). Dong teaches a text embedding comprising one or more tokens in the unified embedding space representing a masked text phrase comprising a text description of the digital image with one or more masked tokens (Dong, Masked text description in FIG 1d, attached below; section 3.3 on pg. 10998: “masked text tokens”; CLIP maps image and text embeddings to a unified embedding space) and modifying parameters of the vision-language model according to a per-token masked distillation loss (Dong, Token-wise distillation loss in FIG 1d; see loss value in section 3.3 on pg. 10997). It would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to have combined the masked text description and associated loss value, as taught by Dong, in the MFD CLIP model in the method of Yang in order to improve the model by jointly learning representations of the image and their corresponding masked descriptions, described by Dong as follows (Dong, last para on pg. 10995: “the learned representation for local patches shall possess semantic meanings, being consistent with the global representation receiving semantic text supervision”; 3rd para on pg. 10996: “we argue that local semantic supervision on the text branch is also helpful for the text encoder and eventually beneficial for zero-shot performance. So we introduce the same mask-data-modeling format supervision into the text branch as well”).
PNG
media_image2.png
517
557
media_image2.png
Greyscale
Additionally, Zhang teaches a method for token-level distillation (Zhang, abstract: “Existing approaches primarily focus on distilling individual information or instance-level relationships, overlooking the valuable information embedded in token-level relationships, which may be particularly affected by the long-tail effects. To address the above limitations, we propose a novel method called Knowledge Distillation with Token-level Relationship Graph (TRG) that leverages the token-wise relational knowledge to enhance the performance of knowledge distillation”), including performing a per-token projection between each token within the teacher representation and each token within the student representation (Zhang, pg. 5, section 4.1.5: “This contrastive learning approach helps the student network capture the underlying relationships and structure present in the teacher’s token graph, enabling a more comprehensive knowledge transfer. Considering that the dimensionality of the student and teacher tokens may differ, we introduce a linear projection, denoted as Proj, for feature alignment. Consequently, the similarity between tokens from the teacher and student can be computed as follows”; top of pg. 3: “We incorporate the contextual similarity loss between tokens within the individual instances”).
Distilling knowledge at the token level can enhance the representation of relationships and similarities across the tokens by incorporating fine-grained information into a relationship representation (Zhang, pg. 2, 3rd para). It would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to have combined the token-level distillation (Zhang, token-level distillation, relationship representation, projection, and loss values), taught by Zhang, with the text embedding-space knowledge distillation method taught by Yang in view of Dong in order to improve the distillation results (Zhang, abstract: “By employing TRG, the student model can effectively emulate higher-level semantic information from the teacher model, resulting in improved distillation results. To further enhance the learning process, we introduce a token-wise contextual loss called contextual loss, which encourages the student model to capture the inner-instance semantic contextual of the teacher model”). Accordingly, the resulting combination of Yang in view of Dong and Zhang teaches wherein the teacher text embedding comprises token-level embeddings of the text description; and modifying parameters of the vision-language model according to a per-token masked distillation loss generated by performing a per-token projection between each token within the teacher text embedding and each token within the text embedding generated by the text encoder (Token-level distillation combined with the text embedding-space distillation taught by Yang in view of Dong).
Lastly, Singh teaches a method wherein a pretrained large language model is utilized as a teacher for a smaller model (Singh, abstract: “apply knowledge
distillation techniques to filter language-specific information from a large multilingual model into a small, fast monolingual model that can often outperform the teacher model”; see Large Language Models in the last para on pg. 3). Accordingly, it would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to have utilized a pretrained large language model to distill knowledge into a smaller model, as taught by Singh, in the method of Yang in view of Dong and Zhang in order to refine the text encoder of the MFD CLIP model using the extensive language knowledge of a large language model (Singh, 1st para in section 7 on pg. 15: “The experimental results confirmed that language-distillation is viable, especially in low-resourced settings, and the resulting students were often able to outperform the teacher multilingual models while being up to four times smaller and six times faster for inference than their respective teachers”). Distilling knowledge from a large language model (LLM) for a student text embedding provides an advantage due to the extensive language knowledge of the LLM, as opposed to pretrained models less focused on language.
Regarding claim 2 (dependent on claim 1), Yang in view of Dong, Zhang, and Singh teaches further comprising modifying the parameters of the vision-language model according to an additional masked distillation loss (Yang, section 3.2.3 on pg. 4: “we utilize MSE loss to align the student’s and teacher’s visual and text embeddings.”; see vk difference value calculated in the total loss) between the image embedding generated by the vision encoder and a teacher image embedding generated by a pretrained vision foundation model (Yang, section 4.3 on pg. 6: “Given the pretrained teacher CLIP model, we distill several light-weight student CLIP models with various architectures.”; see Teacher visual encoder and embedding in FIG 1c).
Regarding claim 3 (dependent on claim 2), Yang in view of Dong, Zhang, and Singh teaches wherein modifying the parameters of the vision-language model according to the additional masked distillation loss comprises distilling features learned by the pretrained vision foundation model into the vision encoder of the vision-language model to encourage the vision encoder to learn to replicate the teacher image embedding of the pretrained vision foundation model from the masked digital image (Yang, section 3.2.3 on pg. 3-4: “The core idea is to recover the masked regions using contextual
information modeling by a vision transformer… In the scenario of distillation, the
teacher is a good supervisor that could provide valuable information to help the student recover the visual semantics given the masked image as input”).
Regarding claim 4 (dependent on claim 1), Yang in view of Dong, Zhang, and Singh teaches wherein modifying the parameters of the vision-language model according to the per-token masked distillation loss comprises distilling features learned by the pretrained large language model into the text encoder of the vision-language model to encourage the text encoder to learn to replicate the teacher text embedding of the pretrained large language model from the masked text phrase (Performed with the student/teacher text embeddings of Yang, see section 3.2.3 on pg. 3-4 citation in claim 3 above; Further supported by the implementation of text masking in Dong, 3rd para on pg. 10996: “So we introduce the same mask-data-modeling format supervision into the text branch as well”).
Regarding claim 5 (dependent on claim 1), Yang in view of Dong, Zhang, and Singh teaches wherein: generating the text embedding comprises utilizing the text encoder of the vision-language model to project features from the unified embedding space of the vision encoder and the text encoder to a dimensionality of the pretrained large language model (Yang, section 3.2.2 on pg. 3: “when the embedding sizes between the teacher and student are different, we apply a linear projection head to student embeddings to match the dimension”; LLM taught in combination with Singh in claim 1); and modifying the parameters of the vision-language model is based on projecting the features (Projection affects the distillation loss value, thus modification of model parameters is based on the projection).
It is recognized that the citations and evidence provided above are derived from potentially different embodiments of a single reference (section 3.2.2 of Yang describes the Feature Distillation approach, not the Masked Feature Distillation). Nevertheless, it would have been obvious, before the effective filing date of the claimed invention, to a person having ordinary skill in the art to which the claimed invention pertains to employ combinations and sub-combinations of these complementary embodiments, because both feature distillation methods described by Yang utilize the same encoding architectures and loss values. Additionally, the projecting of features to a dimensionality of a teacher model is a known-technique in the art.
Regarding claim 6 (dependent on claim 1), Yang in view of Dong, Zhang, and Singh teaches wherein: generating the image embedding comprises utilizing the vision encoder of the vision-language model to project features from the unified embedding space of the vision encoder and the text encoder to a dimensionality of a pretrained vision foundation model (Yang, section 3.2.2 on pg. 3: “when the embedding sizes between the teacher and student are different, we apply a linear projection head to student embeddings to match the dimension”); and modifying the parameters of the vision-language model is based on projecting the features (Projection affects the distillation loss value, thus modification of model parameters is based on the projection).
It is recognized that the citations and evidence provided above are derived from potentially different embodiments of a single reference (section 3.2.2 of Yang describes the Feature Distillation approach, not the Masked Feature Distillation). Nevertheless, it would have been obvious, before the effective filing date of the claimed invention, to a person having ordinary skill in the art to which the claimed invention pertains to employ combinations and sub-combinations of these complementary embodiments, because both feature distillation methods described by Yang utilize the same encoding architectures and loss values. Additionally, the projecting of features to a dimensionality of a teacher model is a known-technique in the art.
Regarding claim 7 (dependent on claim 1), Yang in view of Dong, Zhang, and Singh fails to explicitly teach further comprising modifying the parameters of the vision-language model to learn a projection from multilingual text embeddings of the pretrained large language model to text input embeddings of the text encoder (Yang teaches wherein the student text encoder learns a projection from the teacher encoder using a projection head – see linear projection head in section 3.2.2 on pg. 3, while the combination with Singh teaches wherein the teacher may be a multilingual large language model).
It is recognized that the citations and evidence provided above are derived from potentially different embodiments of a single reference (section 3.2.2 of Yang describes the Feature Distillation approach, not the Masked Feature Distillation). Nevertheless, it would have been obvious, before the effective filing date of the claimed invention, to a person having ordinary skill in the art to which the claimed invention pertains to employ combinations and sub-combinations of these complementary embodiments, because both feature distillation methods described by Yang utilize the same encoding architectures and loss values. Additionally, the projecting of features to a dimensionality of a teacher model is a known-technique in the art.
Claims 8-11 and 13-14 are rejected under 35 U.S.C. 103 as being unpatentable over Yang in view of Dong, in further view of Liu et al. (U.S. Patent No. 2024/0177838 A1), hereinafter Liu, and Singh.
Regarding claim 8, Yang teaches a non-transitory computer readable medium storing executable instructions which, when executed by a processing device (Yang, instructions to execute the training and test the machine-learned models, pg. 5, Training details: “All experiments are ran over 8 V100 GPUs”), cause the processing device to perform operations comprising:
generating, utilizing a vision encoder of a vision-language model (Yang, Student visual encoder of the Masked Feature Distillation, MFD, model in FIG 1c), an image embedding in a unified embedding space of the vision-language model (Yang, see image embedding in FIG 1c; section 3.1 on pg. 2: “CLIP performs an image-text alignment task to push the paired image-text close and unpaired ones apart in the feature embedding space”) from a masked digital image comprising a digital image with one or more masked patches (Yang, masked image input into MFD in FIG 1c; section 3.2.3 on pg. 3-4: “Masked Feature Distillation (MFD) uses masked images as the input to a student”) by projecting features from the unified embedding space to a dimensionality of a pretrained vision foundation model (Yang, vision foundation model, see Teacher in FIG 1c – section 4.3 on pg. 6: “Given the pretrained teacher CLIP model, we distill several light-weight student CLIP models with various architectures.”; features are projected from the students that make up the unified embedding space to the dimension of the teacher, section 3.2.2 on pg. 3: “when the embedding sizes between the teacher and student are different, we apply a linear projection head to student embeddings to match the dimension”);
generating, utilizing a text encoder of the vision-language model (Yang, Student text encoder of the MFD model in FIG 1c) and from a text phrase comprising a text description of the digital image (Yang, text input into MFD in FIG 1c), a text embedding in the unified embedding space (Yang, see text embedding in FIG 1c; see also section 3.1 on pg. 2 cited above) by projecting features from the unified embedding space to a dimensionality of a pretrained model (Yang, see section 3.2.2 citation above);
generating, utilizing the pretrained vision foundation model, a teacher image embedding from patches of the digital image (Yang, section 4.3 on pg. 6: “Given the pretrained teacher CLIP model, we distill several light-weight student CLIP models with various architectures.”; see Teacher visual encoder and embedding in FIG 1c);
generating, utilizing the pretrained large language model, a teacher text embedding from text of the text description of the digital image (Yang, section 4.3 on pg. 6: “Given the pretrained teacher CLIP model, we distill several light-weight student CLIP models with various architectures.”; see Teacher text encoder and embedding in FIG 1c) and
modifying parameters of the vision-language model based on comparing the projected features with the teacher image embedding and the teacher text embedding (Yang, “Similar to FD, we utilize MSE loss to align the student’s and teacher’s visual and text embeddings”; Teacher and student embeddings are compared using MSE loss value. Projection affects the distillation loss value; thus, modification of model parameters is based on the projection.) within the dimensionality of the pretrained vision foundation model and the dimensionality of the pretrained large language model (Yang, the features are projected to the dimensionality of the teachers, see section 3.2.2 citation above).
Regarding the projection of features, it is recognized that the citations and evidence provided above are derived from potentially different embodiments of a single reference (section 3.2.2 of Yang describes the Feature Distillation approach, not the Masked Feature Distillation). Nevertheless, it would have been obvious, before the effective filing date of the claimed invention, to a person having ordinary skill in the art to which the claimed invention pertains to employ combinations and sub-combinations of these complementary embodiments, because both feature distillation methods described by Yang utilize the same encoding architectures and loss values. Additionally, the projecting of features to a dimensionality of a teacher model is a known-technique in the art.
Yang fails to explicitly teach (1) wherein the text phrase is a masked text phrase comprising a text description of the digital image with one or more masked tokens; (2) wherein the pretrained model is a pretrained large language model; and (3) generating, utilizing the pretrained vision foundation model, a teacher image embedding from subsampled patches of the digital image; generating, utilizing the pretrained large language model, a teacher text embedding from subsampled text of the text description of the digital image (emphasis added).
However, Dong similarly teaches a CLIP (Contrastive Language Image Pre-training) model (Dong, MaskCLIP). Dong teaches wherein the text phrase is a masked text phrase comprising a text description of the digital image with one or more masked tokens (Dong, Masked text description in FIG 1d; section 3.3 on pg. 10998: “masked text tokens”). It would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to have combined the masked text description, as taught by Dong, in the MFD CLIP model in the method of Yang in order to improve the model by jointly learning representations of the image and their corresponding masked descriptions, described by Dong as follows (Dong, last para on pg. 10995: “the learned representation for local patches shall possess semantic meanings, being consistent with the global representation receiving semantic text supervision”; 3rd para on pg. 10996: “we argue that local semantic supervision on the text branch is also helpful for the text encoder and eventually beneficial for zero-shot performance. So we introduce the same mask-data-modeling format supervision into the text branch as well”).
Additionally, Liu teaches a method for student/teacher distillation (Liu, para 67-69) wherein the teacher embeddings are generated from subsampled sets of input data (Liu, student and teacher both receive a subset of tokens, para 67: “These augmented views may be split into a series of fixed-size patches and/or may be analogous to image tokens in the MAE method. The student network may receive a subset of these patches, while the teacher network may receive either the same or a different subset, which may include patches that may be withheld from the student network to create a knowledge gap”). Yang in view of Dong teaches wherein image patch and text tokens are input to teacher text and visual encoders, respectively, but fails to explicitly teach wherein a subsampling of patches and text are input (See FIG. 1c of Yang). However, Liu teaches inputting a subsampling of image patches/tokens to a model used for teacher distillation (See citations provided above). One of ordinary skill in the art, before the effective filing date of the claimed invention, could have combined the method of using a subsampled input, taught by Liu, with the vision language model taught by Yang in view of Dong using known methods. In doing so, each element merely would have performed the same functions as it did separately and would achieve the predictable results of generating teacher embeddings from the input digital image and text description. Additionally, utilizing a subset of token inputs to generate the teacher embeddings can focus the model on the most critical features of the image/text. A person of ordinary skill of the art would recognize that the teachings of Liu could be applied to both the image and text teacher encoders.
Lastly, Singh teaches a method wherein a pretrained large language model is utilized as a teacher for a smaller model (Singh, abstract: “apply knowledge
distillation techniques to filter language-specific information from a large multilingual model into a small, fast monolingual model that can often outperform the teacher model”; see Large Language Models in the last para on pg. 3). Accordingly, it would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to have utilized a pretrained large language model to distill knowledge into a smaller model, as taught by Singh, in the method of Yang in view of Dong and Liu in order to refine the text encoder of the MFD CLIP model using the extensive language knowledge of a large language model (Singh, 1st para in section 7 on pg. 15: “The experimental results confirmed that language-distillation is viable, especially in low-resourced settings, and the resulting students were often able to outperform the teacher multilingual models while being up to four times smaller and six times faster for inference than their respective teachers”). Distilling knowledge from a large language model (LLM) for a student text embedding provides an advantage due to the extensive language knowledge of the LLM, as opposed to pretrained models less focused on language.
Regarding claim 9 (dependent on claim 8), Yang in view of Dong, Liu and Singh teaches wherein generating the teacher image embedding comprises: generating a set of patches representing portions of the digital image; and determining, from the set of patches, the subsampled patches as input to the pretrained vision foundation model (Method taught by Liu in combination with the vision foundation model of Yang; see para 67 citation from Liu in claim 8 rejection.).
Regarding claim 10 (dependent on claim 8), Yang in view of Dong, Liu and Singh teaches wherein the operations further comprise: generating, utilizing the vision-language model to process the masked digital image and the masked text phrase, a predicted text embedding of the text description in a unified embedding space of the vision encoder and the text encoder (Yang, operation of CLIP, section 1 on pg. 1: “applies contrastive learning to (image, text) pairs. It guides the model to predict the correct (image, text) pair among the candidate image and text samples”; section 3.1 on pg. 2: “CLIP performs an image-text alignment task to push the paired image-text close and unpaired ones apart in the feature embedding space”; Masking of the text phrase taught in combination with Dong in claim 8); and modifying the parameters of the vision-language model based on a masked distillation loss between the predicted text embedding and the teacher text embedding generated by the pretrained large language model (Masked distillation loss taught in combination with Dong in the same way as described in claim 1– Yang teaches a loss between the predicted student and teacher text embeddings, see sk difference value in section 3.2.3, while Dong teaches applying this loss for masked text, see Token-wise distillation loss in FIG 1d and loss value in section 3.3 on pg. 10997; LLM as the teacher taught by Singh in claim 8).
Regarding claim 11 (dependent on claim 8), Yang in view of Dong, Liu and Singh teaches wherein the operations further comprise: generating, utilizing the vision-language model to process the masked digital image and the masked text phrase, a predicted image embedding of the digital image in a unified embedding space of the vision encoder and the text encoder (Yang, operation of CLIP, section 1 on pg. 1: “applies contrastive learning to (image, text) pairs. It guides the model to predict the correct (image, text) pair among the candidate image and text samples”; section 3.1 on pg. 2: “CLIP performs an image-text alignment task to push the paired image-text close and unpaired ones apart in the feature embedding space”; Masking of the text phrase taught in combination with Dong in claim 8); and modifying the parameters of the vision-language model based on a masked distillation loss between the predicted image embedding and the teacher image embedding generated by the pretrained vision foundation model (Yang teaches a loss between the predicted student and teacher visual embeddings, see vk difference value in section 3.2.3).
Regarding claim 13 (dependent on claim 8), Yang in view of Dong, Liu and Singh teaches wherein the operations further comprise: generating, utilizing the vision-language model to process the masked digital image and the masked text phrase, a predicted text embedding of the text description of the digital image (Taught by the text encoders of Yang in view of Dong, see claim 8); and modifying the parameters of the vision-language model using a contrastive loss to predict correctness of the predicted text embedding (Yang, contrastive loss for image-text alignment in section 3.1 on pg. 2; Further supported by the use of both a contrastive loss and distillation loss in Dong, see section 3.1 on pg. 10997).
Regarding claim 14 (dependent on claim 8), Yang in view of Dong, Liu and Singh teaches wherein the operations further comprise: generating, utilizing the vision-language model to process the masked digital image and the masked text phrase, a predicted image embedding of the digital image (Taught by the vision encoders of Yang in view of Dong, see claim 8); and modifying the parameters of the vision-language model using a contrastive loss to predict correctness of the predicted image embedding (Yang, contrastive loss for image-text alignment in section 3.1 on pg. 2; Further supported by the use of both a contrastive loss and distillation loss in Dong, see section 3.1 on pg. 10997).
Claim 12 is rejected under 35 U.S.C. 103 as being unpatentable over Yang in view of Dong, Liu, Singh, and Sakuma et al. (Sakuma, J., & Yoshinaga, N. (2019, November). Multilingual model using cross-task embedding projection. In Proceedings of the 23rd Conference on Computational Natural Language Learning (CoNLL) (pp. 22-32).), hereinafter Sakuma.
Regarding claim 12 (dependent on claim 8), Yang in view of Dong, Liu and Singh teaches wherein the operations further comprise modifying the parameters of the vision-language model to learn a projection from multilingual text embeddings of the pretrained large language model to text input embeddings of the text encoder (Yang teaches wherein the student text encoder learns a projection from the teacher encoder using a projection head – see linear projection head in section 3.2.2 on pg. 3, while the combination with Singh teaches wherein the teacher may be a multilingual large language model).
Since the model of Yang does not input multilingual training data, it is not explicitly taught that the projection would be without inputting multilingual training data into the vision-language model. However, Sakuma teaches a cross-lingual mapping from one embedding to another embedding of a trained model without inputting multilingual training data into the trained model (Sakuma, see FIG 1 attached below and the cross-task projection based on dimension reduction at the end of pg. 24 into pg. 25; bottom left of pg. 28: “our projection successfully induced task-specific cross-lingual word embeddings”). It would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to have combined the embedding layer projection, taught by Sakuma, in the vision-language model of Yang in view of Dong, Liu, and Singh in order to benefit from the multilingual capabilities of the LLM without requiring the training data that trained the LLM (Sakuma, section 6 on pg. 30: “The locally linear mapping assumes and preserves the local topology across the semantic spaces before and after the projection. Experimental results demonstrated that the locally linear mapping successfully obtains task-specific word embeddings of the target language, and the resulting fully task-specific multilingual model exhibited better model accuracy than the existing multilingual model that fixes its embedding layer to general word embeddings”).
PNG
media_image3.png
254
350
media_image3.png
Greyscale
Claims 15-16 and 19-20 are rejected under 35 U.S.C. 103 as being unpatentable over Yang in view of Burns et al. (Burns, A., Kim, D., Wijaya, D., Saenko, K., & Plummer, B. A. (2020, August). Learning to scale multilingual representations for vision-language tasks. In European Conference on Computer Vision (pp. 197-213). Cham: Springer International Publishing.), hereinafter Burns, and Singh.
Regarding claim 15, Yang teaches a system comprising: one or more memory devices; and one or more processors coupled to the one or more memory devices (Yang, storage and computing components allowing for the execution of the training and testing of the machine-learned models, pg. 5, Training details: “All experiments are ran over 8 V100 GPUs”), the one or more processors configured to cause the system to perform operations comprising:
processing a digital image utilizing a vision-language model comprising a vision encoder (Yang, Student visual encoder of the Masked Feature Distillation, MFD, model in FIG 1c) and a text encoder (Yang, Student text encoder of the MFD model in FIG 1c) trained to project features from a dimensionality of a model into a unified embedding space of the vision encoder and the text encoder (Yang, section 3.2.2 on pg. 3: “when the embedding sizes between the teacher and student are different, we apply a linear projection head to student embeddings to match the dimension”); and
generating, utilizing the vision-language model, a vision-language output by processing the digital image (Yang, see section 4.3 on pg. 6-7 describing output testing of the distilled smaller CLIP models).
Yang fails to teach a text encoder trained to project a lookup table of features from a dimensionality of a large language model trained on multilingual data into a unified embedding space of the vision encoder and the text encoder, wherein the lookup table of features corresponds to a mapping of tokens to corresponding latent vector representations determined by the large language model.
However, Burns teaches a multilingual vision-language model trained to project a lookup table of features from a dimensionality of a language model trained on multilingual data into a unified embedding space of multimodal model (Burns, pg. 5-6, section 3: “The intuition behind our model is to first learn a universal language embedding which represents all languages, and then learn to relate it to corresponding images using a multimodal model… we learn an efficient universal embedding with our Hybrid Embedding Model (HEM) that consists of a mix of language-agnostic and language-specific word representations”), wherein the lookup table of features corresponds to a mapping of tokens to corresponding latent vector representations determined by the language model (Burns, pg. 6, section 3.1: “The output of the HEM is the universal language embedding that is used as input to the multimodal model in Fig. 2 and is also used in the language alignment losses (Section 3.2 and Section 3.3)”; see FIGs. 1-2 attached below).
PNG
media_image4.png
558
845
media_image4.png
Greyscale
PNG
media_image5.png
549
872
media_image5.png
Greyscale
It would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to have combined the multilingual embedding method, taught by Burns, in the system of Yang in order to implement multilingual capabilities in the vision-language model (Burns, pg. 14, section 6: “We have presented a Scalable Multilingual Aligned Representation (SMALR), which addresses the trade-off between multilingual model size and downstream vision-language task performance. Our approach is modular, and thus can be used as a drop-in language representation for any vision-language method/task.”). In the combination of Yang in view of Burns, the student text encoder of the vision-language model can utilize the projected lookup table as claimed.
Additionally, Singh teaches a method wherein a large language model trained on multilingual data is utilized as a teacher for a smaller model (Singh, abstract: “apply knowledge distillation techniques to filter language-specific information from a large multilingual model into a small, fast monolingual model that can often outperform the teacher model”; see Large Language Models in the last para on pg. 3). Accordingly, it would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to have utilized the multilingual large language model to distill knowledge into a smaller model, as taught by Singh, in the method of Yang in view of Burns in order to refine the text encoder of the MFD CLIP model using the extensive language knowledge of a large language model (Singh, 1st para in section 7 on pg. 15: “The experimental results confirmed that language-distillation is viable, especially in low-resourced settings, and the resulting students were often able to outperform the teacher multilingual models while being up to four times smaller and six times faster for inference than their respective teachers”). In the combination of Yang in view of Burns and Singh, the multilingual representations learned by the LLM can be utilized in the projection method taught by Burns above.
Regarding claim 16 (dependent on claim 15), Yang in view of Burns and Singh teaches wherein the one or more processors are further configured to cause the system to generate the vision-language output (Yang, section 4.3 on pg. 6-7: “Given the pretrained teacher CLIP model, we distill several lightweight student CLIP models with various architectures.”) by using the vision-language model to determine a classification of the digital image (Yang, section 4.3 on pg. 6-7: “zero-shot ImageNet classification”).
Regarding claim 19 (dependent on claim 15), Yang in view of Burns and Singh teaches wherein the one or more processors are further configured to cause the system to generate the vision-language output (Yang, section 4.3 on pg. 6-7: “Given the pretrained teacher CLIP model, we distill several lightweight student CLIP models with various architectures.”) by using the vision-language model to retrieve, from digital image database, one or more digital images corresponding to the digital image (Yang, section 4.3 on pg. 6-7: “cross-modal retrieval…text -> image retrieval”).
Regarding claim 20 (dependent on claim 15), Yang in view of Burns and Singh teaches wherein the one or more processors are further configured to cause the system to generate an additional vision-language output (Yang, section 4.3 on pg. 6-7: “Given the pretrained teacher CLIP model, we distill several lightweight student CLIP models with various architectures.”) by using the vision-language model to generate a caption that describes relational composition of objects depicted in the digital image (Yang, section 4.3 on pg. 6-7: “cross-modal retrieval…image -> text retrieval”).
Claim 17 is rejected under 35 U.S.C. 103 as being unpatentable over Yang in view of Burns, Singh, and Dong.
Regarding claim 17 (dependent on claim 15), Yang in view of Burns and Singh fails to explicitly teach wherein the one or more processors are further configured to cause the system to generate the vision-language output by using the vision-language model to determine segmentations of objects depicted within the digital image.
Dong teaches a vision-language output by using the vision-language model to determine segmentations of objects depicted within the digital image (Dong, last para in left column on pg. 10996: “We train our MaskCLIP on a subset of a publicly available image-text pairs dataset…semantic segmentation…detection and segmentation”). Yang teaches a system comprising a vision-language model. The use of vision-language models to segment objects within a digital image is a known technique (see MPEP 2143(I)(D)). The training and use of the CLIP model of Dong to detect and segment objects from surrounding pixels could be applied to the CLIP model of Yang in view of Burns and Singh. Therefore, a person having ordinary skill in the art, before the effective filing date of the claimed invention, could have applied the known technique, as taught by Dong, in the same way to the system of Yang in view of Burns and Singh and achieved predictable results of obtaining a computer vision model for segmentation tasks (See “Semantic segmentation on ADE20K” and “Object detection and instance segmentation on MS-COCO” on pg. 11000 of Dong).
Claim 18 is rejected under 35 U.S.C. 103 as being unpatentable over Yang in view of Burns, Singh, and Goudar et al. (Goudar, R. H., Dhananjaya, G. M., Kambar, V. A., Kulkarni, A., Deshpande, S. L., & Rathod, V. (2023, November). Translingual Image-to-Text Conversion: Bridging Visual and Multilingual Semantic Representations. In 2023 IEEE North Karnataka Subsection Flagship International Conference (NKCon) (pp. 1-6). IEEE.), hereinafter Goudar.
Regarding claim 18 (dependent on claim 15), Yang in view of Burns and Singh fails to explicitly teach wherein the one or more processors are further configured to cause the system to generate the vision-language output by using the vision-language model to generate a non-English caption for the digital image.
Goudar teaches a vision-language output by using a vision-language model to generate a non-English caption for the digital image (Goudar, abstract: “translingual image-to-text conversion… the suggested method's efficiency in accurately converting images to multilingual text”; based on a multilingual embedding, section D on pg. 3: “The extracted visual features and linguistic embeddings are fused to create a unified multilingual semantic representation of the picture content”). Yang teaches a system comprising a vision-language model. The use of vision-language models to generate captions for a digital image is a known technique (see MPEP 2143(I)(D)). The training and use of the multilingual model of Goudar to generate non-English captions could be applied to the CLIP model of Yang in view of Burns and Singh. Therefore, a person having ordinary skill in the art, before the effective filing date of the claimed invention, could have applied the known technique, as taught by Goudar, in the same way to the system of Yang in view of Burns and Singh and achieved predictable results of obtaining a computer vision model for multilingual captioning.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Clinchant et al. (U.S. Patent No. 2023/0418848 A1) teaches a method for LLM distillation.
Wang et al. (Wang, X., Jiang, Y., Bach, N., Wang, T., Huang, F., & Tu, K. (2020, July). Structure-level knowledge distillation for multilingual sequence labeling. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (pp. 3317-3330).) teaches a token-level distillation method.
WO Patent No. 2024/258696 A1
Knowledge Distillation. Neural Network Distiller, 14 Jul 2023, Internet Archive [online], [retrieved on 2026-07-22]. Retrieved from the Internet <URL: https://web.archive.org/web/20230714015425/https://intellabs.github.io/distiller/knowledge_distillation.html>
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to EMMA E DRYDEN whose telephone number is (571)272-1179. The examiner can normally be reached M-F 9-5 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, ANDREW BEE can be reached at (571) 270-5183. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/EMMA E DRYDEN/Examiner, Art Unit 2677
/ANDREW W BEE/Supervisory Patent Examiner, Art Unit 2677
Prosecution Insights