DETAILED ACTION
1. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
2. This communication is in response to the Applicant’s submission filed 21 June 2024, where:
Claims 1-20 are pending.
Claims 1-20 are rejected.
Drawings
3. The drawings are objected to as failing to comply with 37 CFR 1.84(p)(5) because they include the following reference character(s) not mentioned in the description:
Fig. 6, reference 612, is not mentioned in the description.
Corrected drawing sheets in compliance with 37 CFR 1.121(d), or amendment to the specification to add the reference character(s) in the description in compliance with 37 CFR 1.121(b) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the Examiner, the Applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Claim Objections
4. Claims 2, 3, 9, 10, 16, and 17 are objected to because of the following informalities:
Claims 2, 9, and 16, each recite “PBMN-1.” The first occurrence of an acronym in a claim needs to be expanded.
Claims 3, 10, and 17, each recite “PBMN-2.” The first occurrence of an acronym in a claim needs to be expanded.
Appropriate correction is required.
Claim Rejections - 35 USC § 112
5. The following is a quotation of 35 U.S.C. § 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
6. Claims 1-20 are rejected under 35 U.S.C. § 112(b) as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor regards as the invention.
Claim 1, line 3, recites “a model,” while the claim at line 1 also recites “a model.” It is unclear whether the second occurrence of the term is intended to draw antecedence from the initial recitation of the term, or whether it is intended to be another “model” altogether. Accordingly, the claim in indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor regards as the invention.
Claim 8, line 4, recites “a model,” while the claim at line 1 also recites “a model.” It is unclear whether the second occurrence of the term is intended to draw antecedence from the initial recitation of the term, or whether it is intended to be another “model” altogether. Accordingly, the claim in indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor regards as the invention.
Claim 15, line 5, recites “a model,” while the claim at line 1 also recites “a model.” It is unclear whether the second occurrence of the term is intended to draw antecedence from the initial recitation of the term, or whether it is intended to be another “model” altogether. Accordingly, the claim in indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor regards as the invention.
Claims 2-7 depend directly or indirectly from claim 1. Claims 9-14 depend directly or indirectly from claim 1. Claims 16-20 depend directly or indirectly from claim 1. Claims 2-7, 9-14, and 16-20 are rejected as depending from a rejected claim; further, the claims fail to cure the deficiencies of claims 1, 8, and 15, respectively.
Claim Rejections - 35 U.S.C. § 101
7. 35 U.S.C. § 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
8. Claims 15-20 are rejected under 35 U.S.C. § 101 because the broadest reasonable interpretation of the terms "a computer-readable storage medium” covers both statutory and non-statutory embodiments, which are not eligible for patent protection, and therefore the claims are directed to non-statutory subject matter. See In re Nuijten, 500 F.3d 1346, 84 USPQ2d 1495 (Fed. Cir. 2007).
The plain meaning of computer-readable medium encompasses signals. The specification recites:
Without limitation, the computer readable storage medium may be an electronic storage medium, a magnetic storage medium, an optical storage medium, an electromagnetic storage medium, a semiconductor storage medium, a mechanical storage medium, or any suitable combination of the foregoing.
A computer readable storage medium, as that term is used in the present disclosure, is not to be construed as storage in the form of transitory signals per se, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through a waveguide, light pulses passing through a fiber optic cable, electrical signals communicated through a wire, and/or other transmission media.”
(Specification ¶¶ 0079-80). The specification does not rebut the presumption that the term be given its plain meaning because, though the specification provides “specific examples” of both storage and signal mediums, they are merely examples rather than a definition of two distinct species or embodiments that would rebut the presumption. Although the computer readable storage medium includes statutory embodiments, nothing in the specification specifically excludes signals per se from falling within computer readable storage media.
Examiner suggests Applicant amend the claims to exclude transitory computer usable program product and transitory computer-readable devices in order to narrow the broadest reasonable interpretation of those claims to embodiments that fall within a statutory category.
9. Claims 1-20 are rejected under 35 U.S.C. § 101 because the claimed invention is directed to an abstract idea without significantly more.
Claim 1 recites a method, which is a process, and thus one of the statutory categories of patentable subject matter. (35 U.S.C. § 101).
However, under Step 2A Prong One, the claim recites the limitations of “fine-tuning the pretrained model by learning prompts specific to each source time-series domain using remaining data from the source time-series domains,” “applying instance normalization and segmenting the time-series data into subseries-level normalized patches for the target time-series domain;” “calculating cosine similarity between the prompt of the target domain and the prompts of all source domains to identify a nearest neighbor prompt,” and “utilizing the nearest neighbor prompt for model prediction in the target time-series domain.”
These activities of “fine-tuning the pretrained model,” “applying,” “calculating,” and “utilizing . . . for model prediction” contain limitations that can practically be performed in the human mind, including, for example, observations, evaluations, judgments, and opinions, and accordingly, are a mental process, (MPEP § 2106.04(a)(2) sub III), which is one of the groupings of abstract ideas. (MPEP § 2106.04(a)(2)). Thus, claim 1 recites an abstract idea.
Under Step 2A Prong Two, the claim as a whole is not integrated into a practical application, because the additional elements recited in the claim beyond the identified judicial exception include a “model,” which is recited at a high level of generality, and accordingly, is a generic computer component used to implement the abstract idea, (MPEP § 2106.05(g)), that does not serve to integrate the abstract idea into a practical application.
The claim also recites “integrating input data from a plurality of source time-series domains to pretrain a model, the model including a set of domain-invariant representations,” “feeding the normalized patches into a transformer encoder to generate high-dimensional representations of the normalized patches,” and “utilizing a limited number of samples from the target time-series domain to learn the prompt specific to the target domain.” These activities “integrating input data,” “feeding,” and “utilizing samples . . . to learn the prompt,” are insignificant extra-solution activities of inputting data, (MPEP § 2106.05(f)), that does not serve to integrate the abstract idea into a practical application. Therefore, claim 1 is directed to the abstract idea.
Finally, under Step 2B, the additional elements, taken alone or in combination, do not represent significantly more than the abstract idea itself. the additional elements recited in the claim beyond the identified judicial exception include a “model,” which is recited at a high level of generality, and accordingly, is a generic computer component used to implement the abstract idea, (MPEP § 2106.05(g)), that does not amount to significantly more than the abstract idea.
The claim also recites “integrating input data from a plurality of source time-series domains to pretrain a model, the model including a set of domain-invariant representations,” “feeding the normalized patches into a transformer encoder to generate high-dimensional representations of the normalized patches,” and “utilizing a limited number of samples from the target time-series domain to learn the prompt specific to the target domain.” These activities “integrating input data,” “feeding,” and “utilizing samples . . . to learn the prompt,” are well-understood, routine, and conventional activities of retrieving information in memory, (MPEP § 2106.05(d) sub II.iv), that do not amount to significantly more than the abstract idea. Therefore, claim 1 is subject-matter ineligible.
Claim 8 recites a system, which is a product, and thus one of the statutory categories of patentable subject matter. (35 U.S.C. § 101).
However, under Step 2A Prong One, the claim recites the limitations of “fine-tune the pretrained model by learning prompts specific to each source time-series domain using remaining data from the source time-series domains,” “apply instance normalization and segmenting the time-series data into subseries-level normalized patches for the target time-series domain;” “calculate cosine similarity between the prompt of the target domain and the prompts of all source domains to identify a nearest neighbor prompt,” and “utilize the nearest neighbor prompt for model prediction in the target time-series domain.”
These activities of “fine-tune the pretrained model,” “apply,” “calculate,” and “utilize . . . for model prediction” contain limitations that can practically be performed in the human mind, including, for example, observations, evaluations, judgments, and opinions, and accordingly, are a mental process, (MPEP § 2106.04(a)(2) sub III), which is one of the groupings of abstract ideas. (MPEP § 2106.04(a)(2)). Thus, claim 1 recites an abstract idea.
Under Step 2A Prong Two, the claim as a whole is not integrated into a practical application, because the additional elements recited in the claim beyond the identified judicial exception include a “processor device,” a “memory storing instructions that, when executed by the processor device, cause the system to,” and a “model,” which are recited at a high level of generality, and accordingly, are generic computer component used to implement the abstract idea, (MPEP § 2106.05(g)), that does not serve to integrate the abstract idea into a practical application.
The claim also recites “integrate input data from a plurality of source time-series domains to pretrain a model, the model including a set of domain-invariant representations,” “feed the normalized patches into a transformer encoder to generate high-dimensional representations of the normalized patches,” and “utilize a limited number of samples from the target time-series domain to learn the prompt specific to the target domain.” These activities “integrate input data,” “feed,” and “utilize samples . . . to learn the prompt,” are insignificant extra-solution activities of inputting data, (MPEP § 2106.05(f)), that does not serve to integrate the abstract idea into a practical application. Therefore, claim 8 is directed to the abstract idea.
Finally, under Step 2B, the additional elements, taken alone or in combination, do not represent significantly more than the abstract idea itself. The additional elements recited in the claim beyond the identified judicial exception include a “processor device,” a “memory storing instructions that, when executed by the processor device, cause the system to,” and a “model,” which is recited at a high level of generality, and accordingly, is a generic computer component used to implement the abstract idea, (MPEP § 2106.05(g)), that does not amount to significantly more than the abstract idea.
The claim also recites “integrate input data from a plurality of source time-series domains to pretrain a model, the model including a set of domain-invariant representations,” “feed the normalized patches into a transformer encoder to generate high-dimensional representations of the normalized patches,” and “utilize a limited number of samples from the target time-series domain to learn the prompt specific to the target domain.” These activities “integrate input data,” “feed,” and “utilize samples . . . to learn the prompt,” are well-understood, routine, and conventional activities of retrieving information in memory, (MPEP § 2106.05(d) sub II.iv), that do not amount to significantly more than the abstract idea. Therefore, claim 8 is subject-matter ineligible.
Claim 15 recites a computer program product comprising a computer-readable storage medium, which is not a statutory category of patentable subject matter because the broadest reasonable interpretation of the term “computer-readable storage medium” covers both statutory and non-statutory embodiments as set out above in detail. For the purposes of examination, however, the claim is considered as complying with the statutory category requirement. (35 U.S.C. § 101).
However, under Step 2A Prong One, the claim recites the limitations of “fine-tune the pretrained model by learning prompts specific to each source time-series domain using remaining data from the source time-series domains,” “apply instance normalization and segmenting the time-series data into subseries-level normalized patches for the target time-series domain;” “calculate cosine similarity between the prompt of the target domain and the prompts of all source domains to identify a nearest neighbor prompt,” and “utilize the nearest neighbor prompt for model prediction in the target time-series domain.”
These activities of “fine-tune the pretrained model,” “apply,” “calculate,” and “utilize . . . for model prediction” contain limitations that can practically be performed in the human mind, including, for example, observations, evaluations, judgments, and opinions, and accordingly, are a mental process, (MPEP § 2106.04(a)(2) sub III), which is one of the groupings of abstract ideas. (MPEP § 2106.04(a)(2)). Thus, claim 15 recites an abstract idea.
Under Step 2A Prong Two, the claim as a whole is not integrated into a practical application, because the additional elements recited in the claim beyond the identified judicial exception include a “computer-readable storage medium having program instructions embodied therewith, the program instructions executable by a hardware processor to cause the hardware processor,” and a “model,” which are recited at a high level of generality, and accordingly, are generic computer component used to implement the abstract idea, (MPEP § 2106.05(g)), that does not serve to integrate the abstract idea into a practical application.
The claim also recites “integrate input data from a plurality of source time-series domains to pretrain a model, the model including a set of domain-invariant representations,” “feed the normalized patches into a transformer encoder to generate high-dimensional representations of the normalized patches,” and “utilize a limited number of samples from the target time-series domain to learn the prompt specific to the target domain.” These activities “integrate input data,” “feed,” and “utilize samples . . . to learn the prompt,” are insignificant extra-solution activities of inputting data, (MPEP § 2106.05(f)), that does not serve to integrate the abstract idea into a practical application. Therefore, claim 15 is directed to the abstract idea.
Finally, under Step 2B, the additional elements, taken alone or in combination, do not represent significantly more than the abstract idea itself. The additional elements recited in the claim beyond the identified judicial exception include a “computer-readable storage medium having program instructions embodied therewith, the program instructions executable by a hardware processor to cause the hardware processor,” and a “model,” which is recited at a high level of generality, and accordingly, is a generic computer component used to implement the abstract idea, (MPEP § 2106.05(g)), that does not amount to significantly more than the abstract idea.
The claim also recites “integrate input data from a plurality of source time-series domains to pretrain a model, the model including a set of domain-invariant representations,” “feed the normalized patches into a transformer encoder to generate high-dimensional representations of the normalized patches,” and “utilize a limited number of samples from the target time-series domain to learn the prompt specific to the target domain.” These activities “integrate input data,” “feed,” and “utilize samples . . . to learn the prompt,” are well-understood, routine, and conventional activities of retrieving information in memory, (MPEP § 2106.05(d) sub II.iv), that do not amount to significantly more than the abstract idea. Therefore, claim 15 is subject-matter ineligible.
Claims 2 and 3 depend from claim 1. Claims 9 and 10 depend from claim 8. Claims 16 and 17 depend from claim 15. The claims recite more details or specifics to the additional element of the “pretrained model,” (claims 2, 8, and 16: “wherein the pretrained model comprises a PBMN-1 model, and the prompts are concatenated with input time-series data;“ and claims 3, 10, and 17: “wherein the pretrained model comprises a PBMN-2 model, and the prompts are concatenated with hidden embeddings derived from time-series patches’), and accordingly, are merely more specific to the additional element. Therefore, claims 2, 3, 9, 10, 16, and 17 are subject-matter ineligible.
Claim 4 depends from claim 1. Claim 11 depends from claim 8. Claim 18 depends from claim 15. The claims recite more details or specifics to the abstract idea of “applying instance normalization and segmenting,” “wherein the instance normalization and segmenting the time-series data into patches are performed by an instance normalization operator and a patching module, respectively,” and accordingly, are merely more specific to the abstract idea. The additional elements of the claim does not serve to integrate the abstract idea into integrated into a practical application, (see MPEP § 2106.04(d)), nor do the additional elements amount to significantly more than the abstract idea, (MPEP § 2106.05 sub I; see also MPEP § 2106.05(a) – (h)), and thus, the claim recites no more than the abstract idea. Therefore, claims 4, 11, and 18 are subject-matter ineligible.
Claim 5 depends from claim 1. Claim 12 depends from claim 8. Claim 19 depends from claim 15. The claims further recites “self-attention mechanisms with the transformer encoder,” which are recited at a high-level of generality, and accordingly, are merely used to implement the abstract idea, (MPEP § 2106.05(g)), that under Step 2A Prong Two, does not serve to integrate the abstract idea into a practical application, nor under Step 2B, amounts to significantly more than the abstract idea. Also, the claim recites “generating a high-dimensional representation of the input data by applying self-attention mechanisms within the Transformer encoder,” which merely use of the generic computer components (self-attention mechanism, transformer encoder) used to implement the abstract idea, (MPEP § 2106.05(g)), that under Step 2A Prong Two, does not serve to integrate the abstract idea into a practical application, nor under Step 2B, amounts to significantly more than the abstract idea. Accordingly, claims 5, 12, and 19 are subject-matter ineligible.
Claim 6 depends from claim 1. Claim 13 depends from claim 8. Claim 20 depends from claim 15. The claims recite more details or specifics to the abstract idea of “utilize the nearest neighbor prompt for model prediction,” “wherein the model prediction in the target time-series domain comprises generating forecasts or anomaly detection based on the input data,” and accordingly, is merely more specific to the additional element. The additional elements of the claim does not serve to integrate the abstract idea into integrated into a practical application, (see MPEP § 2106.04(d)), nor do the additional elements amount to significantly more than the abstract idea, (MPEP § 2106.05 sub I; see also MPEP § 2106.05(a) – (h)), and thus, the claim recites no more than the abstract idea. Therefore, claims 6, 13, and 20 are subject-matter ineligible.
Claim 7 depends from claim 1. Claim 14 depends from claim 8. The claims recite more details or specifics to the abstract idea of “fine-tuning,” “wherein the fine-tuning of the pretrained model involves utilizing backpropagation and gradient descent techniques to optimize parameters of the model,” and accordingly, is merely more specific to the abstract idea. The additional elements of the claim does not serve to integrate the abstract idea into integrated into a practical application, (see MPEP § 2106.04(d)), nor do the additional elements amount to significantly more than the abstract idea, (MPEP § 2106.05 sub I; see also MPEP § 2106.05(a) – (h)), and thus, the claim recites no more than the abstract idea. Therefore, claims 7 and 14 are subject-matter ineligible.
Claim Rejections - 35 U.S.C. § 102
10. The following is a quotation of the appropriate paragraphs of 35 U.S.C. § 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
11. Claims 1, 4, 5, 8, 11, 12, 15, 18, and 19 are rejected under 35 U.S.C. § 102(a)(1) as being anticipated by Xing et al., “Dual Modality Prompt Tuning for Vision-Language Pre-Trained Model,” arXiv (16 Feb 2023) [hereinafter Xing].
Regarding claims 1, 8, and 15, Xing teaches [a] method for adapting a model trained from multiple source time-series domains to a target time-series domain (Xing, Abstract, teaches “Prompt tuning tries to probe the beneficial information for downstream tasks from the general knowledge stored in the pre-trained model [(that is, adapting a model trained)]. . . . [A] Class-Aware Visual Prompt Tuning (CAVPT) scheme is further proposed in our DPT, where the class-aware visual prompt is generated dynamically by performing the cross attention between text prompts features and image patch token embeddings [(that is, multiple source time-series domains)] to encode both the downstream task-related information and visual instance information [(that is, a target time-series domain)]”) of claim 1, [a] system for adapting a model trained from multiple source time-series domains to a target time-series domain, comprising: a processor device; and a memory storing instructions that, when executed by the processor device (Xing, right column of p. 2, “B. Prompt Learning,” first paragraph, teaches “In computer vision community [(that is, “computer vision” is a system that inherently includes a processor device, and a memory storing instructions)], prompt learning is explored in both the visual models [23]–[25] and the vision-language models [1], [9], [15], [18], [26], where visual prompt tuning (VPT) [23] achieved significant performance gains with only a small amount of additional parameters, i.e., prompts, in the input space [(that is, system for adapting a model trained from multiple source time-series domains to a target time-series domain)]”) of claim 8, and [a] computer program product for adapting a model trained from multiple source timeseries domains to a target time-series domain, the computer program product comprising a computer-readable storage medium having program instructions embodied therewith (Xing, right column of p. 2, “B. Prompt Learning,” first paragraph, teaches “In computer vision community [(that is, “computer vision” inherently includes a computer program product comprising a computer-readable storage medium having program instructions embodied therewith)], prompt learning is explored in both the visual models [23]–[25] and the vision-language models [1], [9], [15], [18], [26], where visual prompt tuning (VPT) [23] achieved significant performance gains with only a small amount of additional parameters, i.e., prompts, in the input space [(that is, adapting a model trained from multiple source time-series domains to a target time-series domain)]”)) of claim 15, comprising:
integrating input data from a plurality of source time-series domains to pretrain a model(Xing, right column of p. 1, “I. Introduction,” second paragraph, teaches “large scale pre-trained vision-language models have already captured most of the general knowledge (visual concepts) for the downstream tasks);
Examiner notes that the plain and ordinary meaning of “time-series” is the ordered observations in time and further is synonymous with a sequence. The broadest reasonable interpretation of the term “time-series” covers the teachings of the self-attention mechanisms of Xing, which allow models to dynamically weigh the importance of different time steps, improving accuracy in forecasting and classification by focusing on the most relevant parts of the sequence, which is not inconsistent with the Applicant’s disclosure. (MPEP § 2111; Specification ¶ 0053 (“Self-attention mechanisms can calculate the importance of each time step relative to others, allowing the model to focus on the most relevant parts of the time- series data. This step is pivotal for enhancing the model's ability to learn from complex temporal sequences and produce accurate predictions”)]),
the model including a set of domain-invariant representations (Xing, right column of p. 6, “C. Domain Generalization, Datasets,” first paragraph, teaches that by “[f]ollowing the setting in CoOp [9], we use ImageNet as the source dataset. And the target datasets are ImageNetV2 [59], ImageNet-Sketch [60], ImageNet-A [61], and ImageNet-R [62]; with regard to “CoOp,” Xing, left column of p. 3, “B. Prompt Learning,” first partial paragraph, teaches “Conditional CoOp (CoCoOp) [15] extends CoOp by learning an input-conditional token for each image to improve the cross domain generalization ability of CoOp. [27] learns the output embeddings of prompts instead of the input embeddings and employs a Gaussian distribution to model them effectively [(that is, “cross-domain generalization” is the model including a set of domain-invariant representations)]”;
[Examiner notes that the plain and ordinary meaning of the term “domain-invariant representations” are latent features that capture information common across different data domains while discarding domain-specific noises. Thus, the broadest reasonable interpretation of the term “domain-invariant representations” covers the teachings of Xing pertaining to cross domain generalization, which is not inconsistent with the Applicant’s disclosure. (MPEP § 2111; Specification ¶ 0030 (“model pretraining can be initiated by integrating a substantial portion of data from multiple source time-series domains. This step can include the aggregation of diverse time-series datasets that exhibit various temporal dependencies and patterns. The PBMN-1 or PBMN-2 model can be employed to learn domain-invariant representations.”)]);
fine-tuning the pretrained model by learning prompts specific to each source time-series domain using remaining data from the source time-series domains (Xing, Fig. 3, teaches a class-aware visual prompt tuning (CAVPT) module [Examiner annotations in dashed-line text boxes]:
PNG
media_image1.png
435
942
media_image1.png
Greyscale
Xing, right column of p. 2, “B. Prompt Learning,” first paragraph, teaches “For vision-language models, CoOp [9] brings continuous prompt optimization”; Xing, left column at p. 3, “B. Prompt Learning,” first partial paragraph, teaches “a Dual-modality Prompt Tuning paradigm, which introduces both the text prompt and visual prompt for vision-language model. Furthermore, class-aware visual prompt is proposed to enable the image feature to pay more attention to the target foreground object for downstream task”; Xing, right column of p. 3, “III. Methodology,” first paragraph, teaches “we elaborate each component of the proposed Dual-modality Prompt Tuning (DPT) paradigm, including text prompt, visual prompt and class-aware visual prompt [(that is, “text prompt, visual prompt, and class-aware visual prompt” are fine-tuning the pretrained model by learning prompts specific to each source time-series domain using remaining data from the source time-series domains)]”;
applying instance normalization and segmenting the time-series data into subseries-level normalized patches for the target time-series domain (Xing, right column of p. 4, “B. Test Prompt and Visual Prompt-Visual Prompt,” first paragraph, teaches “[f]or vision-language models, there are two encoders for visual and language modalities. Tuning text prompt alone is not enough to reduce the gap between pretrained and downstream tasks, thus leading to sub-optimal results. Motivated by the visual prompt tuning (VPT) [23] proposed for tuning vision transformers, we introduce visual prompt into the image encoder of CLIP model. The image patches . . . are firstly embedded into a d-dimensional latent space [(that is, segmenting the time-series data into subseries-level normalized patches for the target time-series domain)]”; Xing, right column of p. 5, “C. Class-Aware Visual Prompt Tuning,” second paragraph, teaches “[o]ur proposed class-aware visual prompt
P
~
l
j
∈
R
d
for the lth layer is computed as,
PNG
media_image2.png
157
760
media_image2.png
Greyscale
Where LN(•) denotes Layer normalization [(that is, applying instance normalization and segmenting the time-series data into subseries-level normalized patches for the target time-series domain )].
W
q
∈
R
d
x
d
k
,
W
k
∈
R
d
x
d
k
and
W
V
∈
R
d
x
d
denotes the parameters of cross attention.”));
feeding the normalized patches into a transformer encoder to generate high-dimensional representations of the normalized patches (Xing, left column of p. 4, “A. Contrastive Language-Image Pre-training (CLIP) Model,” first paragraph & Fig. 2 (below), teaches the “text encoder adopts a transformer [46] to encode the text information while the image encoder can either be a CNN model, like ResNet [5], or be a vision transformer such as ViT [47]. In our method, we choose ViT as the image encoder for the compatibility of visual prompt in [23] [(that is, “attention transformers” are feeding the normalized patches into a transformer encoder to generate high-dimensional representations of the normalized patches)]”);
utilizing a limited number of samples from the target time-series domain to learn the prompt specific to the target domain (Xing, left column of p. 4, “A. Contrastive Language-Image Pre-training (CLIP) Model,” first & second paragraphs, teaches in “[c]ontrastive language-image pre-training (CLIP) model aims to align image feature space and text feature space [(that is, target time-series domain)], which enables the model to have the capability of zero-shot transfer [(that is, “zero-shot” is a limited number of samples)] to downstream tasks. CLIP is composed of two encoders, one is designed for image and the other is for text. . . . [In] downstream tasks, a hand-crafted prompt is fed into the text end to synthesize a zero-shot linear classifier by embedding the class names of the target dataset [(that is, a “zero-shot linear classifier” is utilizing a limited number of samples from the target time-series domain to learn the prompt specific to the target domain)]”; Xing, right column of p. 5, “D. Training of DPT,” first paragraph, teaches “[c]onsidering that CLIP model has stored the general knowledge, we train our model to learn from Zeroshot CLIP [(that is, to learn the prompt specific to the target domain)]”);
calculating cosine similarity between the prompt of the target domain and the prompts of all source domains to identify a nearest neighbor prompt (Xing, Fig. 2, teaches a proposed Dual-Modality Prompt Tuning method having learnable components [Examiner annotations in dashed-line text boxes]:
PNG
media_image3.png
623
1065
media_image3.png
Greyscale
Xing, left column of p. 4, “A. Contrastive Language-Image Pre-training (CLIP) Model,” second paragraph, teaches “[t]aking the classification task as an example, “[CLASS]” token can be firstly extended by a template, such as “a photo of a [CLASS]”. Then the sentence is treated as a prompt and is encoded by the text encoder to derive a weight vector wi, i = {1, . . . , K}, where K is the total number of categories. At the same time, image features x are obtained by the image encoder. The prediction probability can be calculated by
PNG
media_image4.png
70
433
media_image4.png
Greyscale
where sim (•, •) represents the computation of cosine similarity [(that is, “cosine similarity” necessarily functions to identify a nearest neighbor prompt)], and τ is the temperature coefficient learned by CLIP”)
; and
utilizing the nearest neighbor prompt for model prediction in the target time-series domain (Xing, right column of p. 4, “B. Text Prompt and Visual Prompt,” first paragraph, teaches “By forwarding ti [(that is, the target time-series domain)] into the text encoder g(•), we can obtain a classification weight vector for the ith visual concepts. The corresponding prediction probability can be calculated by,
PNG
media_image5.png
57
457
media_image5.png
Greyscale
where x represents the extracted image features, and g(•) denotes the text encoder [(that is, utilizing the nearest neighbor prompt for model prediction in the target time-series domain)]”).
Examiner notes that the Applicant’s preambles do not afford patentable weight to the Applicant’s claims because the claim preamble is not “necessary to give life, meaning, and vitality” to the claim. Moreover, because the Applicant’s preamble merely states the purpose or intended use of the invention rather than any distinct definition of any of the claimed invention’s limitations, the preamble is not considered a limitation and is of no significance to claim construction.
Regarding claims 4, 11, and 18, Xing teaches all of the limitations of claims 1, 8, and 15, respectively, as described above in detail.
Xing teaches -
wherein the instance normalization and segmenting the time-series data into patches are performed by an instance normalization operator and a patching module, respectively (Xing, Fig. 3, teaches an architecture relating to normalization and patches [Examiner annotation in dashed-line text boxes]:
PNG
media_image6.png
612
767
media_image6.png
Greyscale
Xing, left column of p. 5, “B. Text Prompt and Visual Prompt,” first partial paragraph & first full paragraph, teaches “[b]y inserting visual prompts into each transformer layer, the self-attention operation for sl can be affected in two ways, as both the keys and values are actually get prepended through visual prompts: 1) The attention weights can be affected to allow sl to concentrate more on the image patch embeddings [(that is, through a patching module)] which includes target concept; 2) The visual prompts also serve as value vectors for the self-attention operation, so sl may absorb additional information that visual prompts learned. . . . Class-Aware Visual Prompt Tuning (CAVPT) to generate visual prompts by utilizing both the task-related information from the text side and instance-wise information from the visual side [(that is, “instance-wise” is an instance normalization operator)]”).
Regarding claims 5, 12, and 19, Xing teaches all of the limitations of claims 1, 8, and 15, respectively, as described above in detail.
Xing teaches -
further comprising generating a high-dimensional representation of the input data by applying self-attention mechanisms within the Transformer encoder (Xing, right column of p. 5, “B. Text Prompt and Visual Prompt,” last partial paragraph, teaches “Note that the calculation process of the image encoder, i.e., the ViT model, can be viewed as a process of global scene reasoning and sl pools the visual concepts from the image patch embeddings layer-by-layer. With the help of visual prompts, the target visual concept corresponding to the downstream task may get more highlighted in sl via the self-attention operation in each transformer layer [(that is, applying self-attention mechanisms within the Transformer encoder)]. By inserting visual prompts into each transformer layer, the self-attention operation for sl can be affected in two ways, as both the keys and values are actually get prepended through visual prompts: 1) The attention weights can be affected to allow sl to concentrate more on the image patch embeddings which includes target concept; 2) The visual prompts also serve as value vectors for the self-attention operation, so sl may absorb additional information that visual prompts learned [(that is, “additional information” is generating a high-dimensional representation of the input data by applying self-attention mechanisms within the transformer encoder)]”).
Claim Rejections – 35 U.S.C. § 103
12. The following is a quotation of 35 U.S.C. § 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
13. The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. § 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
14. This application currently names joint inventors. In considering patentability of the claims the Examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the Examiner to consider the applicability of 35 U.S.C. § 102(b)(2)(C) for any potential 35 U.S.C. § 102(a)(2) prior art against the later invention.
15. Claims 2, 9, and 16 are rejected under 35 U.S.C. § 103 as being unpatentable over Xing et al., “Dual Modality Prompt Tuning for Vision-Language Pre-Trained Model,” arXiv (16 Feb 2023) [hereinafter Xing] in view of Asai et al., “ATTEMPT: Parameter-Efficient Multi-task Tuning via Attentional Mixtures of Soft Prompts,” arXiv (2022) [hereinafter Asai].
Regarding claims 2, 9, and 16, Xing teaches all of the limitations of claims 1, 8, and 15, respectively, as described above in detail.
Xing teaches -
wherein the pretrained model comprises a PBMN-1 model (Xing, left column of p. 2, “I. Introduction,” first partial paragraph, teaches “a Dual modality Prompt Tuning (DPT) paradigm through learning text prompts and visual prompts for both the text and image encoder simultaneously, aiming at adapting the pre-trained model to downstream tasks via adjusting both the ‘classifier’ and ‘visual features’ [(that is, “adapting the pre-trained model” is a PBMN-1 model )]”;
[Examiner notes the plain and ordinary definition of the term “PBMN-1 model” is a model participating in model adaption through prompt tuning. Thus, the broadest reasonable interpretation of “PBMN-1 model” covers the teachings of Xing pertaining to adapting the pre-trained model, which is not inconsistent with the Applicant’s disclosure. (MPEP § 2111; Specification ¶ 0043 (“a system and method 400 for time-series model adaptation using a Prompt-Based Modular Network (PBMN-1) architecture including an encoder, a decoder, and dynamic routing components, is illustratively depicted in accordance with embodiments of the present invention”))]), and
* * *
Though Xing teaches prompt tuning of a pretrained model, Xing, however, does not explicitly teach –
* * *
and the prompts are concatenated with input time-series data.
But Asai teaches -
* * *
and the prompts are concatenated with input time-series data (Asai, right column of p. 4, “3.3 Multi-Task Training and Inference,” fourth paragraphs, teaches “At inference time, we load source prompts, all of the target task prompts and the shared G just once. For each instance, ATTEMPT retrieves the target task prompt and produces Pinstance as in Eq. 4, and then concatenates Pinstance to the input embedding [(that is, the prompts are concatenated with input time-series data)]. The inference process after producing instance prompt is exactly the same as in prompt tuning.”).
Xing and Asai are from the same or similar field of endeavor. Xing teaches a Dual-modality Prompt Tuning (DPT) paradigm through learning text prompts and visual prompts for both the text and image encoder simultaneously. Asai teaches ATTEMPT (ATTEntional Mixtures of Prompt Tuning), obtains source prompts as encodings of large-scale source tasks into a small number of parameters and trains an attention module to interpolate the source prompts and a newly initialized target prompt for every instance in the target task.
Thus, it would have been obvious for a person having ordinary skill as of the effective filing date of the Applicant’s claimed invention to modify Xing pertaining to text and visual prompt tuning of a pretrained model with the concatenation of prompts and input embeddings of Asai.
The motivation to do so is because “ATTEMPT significantly outperforms prompt tuning and outperforms or matches fully fine-tuned or other parameter-efficient tuning approaches that use over ten times more parameters.” (Asai, Abstract).
16. Claims 3, 10, and 17 are rejected under 35 U.S.C. § 103 as being unpatentable over Xing et al., “Dual Modality Prompt Tuning for Vision-Language Pre-Trained Model,” arXiv (16 Feb 2023) [hereinafter Xing] in view of Rong et al., “Retrieval-Enhanced Visual Prompt Learning for Few-shot Classification,” arXiv (4 June 2023) [hereinafter Rong].
Regarding claims 3, 10, and 17, Xing teaches all of the limitations of claims 1, 8, and 15, respectively, as described above in detail.
Xing teaches -
wherein the pretrained model comprises a PBMN-2 model (Xing, left column of p. 2, “I. Introduction,” first full paragraph, teaches “To make the pretrained model better transfer to the downstream task, we further introduce a Class-Aware Visual Prompt Tuning (CAVPT) mechanism into our DPT framework to help the final obtained image feature concentrate more on the target visual concept [(that is, including “CAVPT for adapting the pre-trained model” is a PBMN-2 model )]”;
[Examiner notes the plain and ordinary definition of the term “PBMN-2 model” is a model participating in model adaption through prompt tuning. Thus, the broadest reasonable interpretation of “PBMN-2 model” covers the teachings of Xing pertaining to adapting the pre-trained model, which is not inconsistent with the Applicant’s disclosure. (MPEP §2111; Specification ¶ 0043 (“a system and method 500 for time-series model adaptation using a Prompt-Based Modular Network (PBMN-2) architecture including a Patch Time Series Transformer (PatchTST) and modular network components, is illustratively depicted in accordance with embodiments of the present invention”))]), and
* * *
Though Xing teaches prompt tuning of a pretrained model, Xing, however, does not explicitly teach –
* * *
and the prompts are concatenated with hidden embeddings derived from time-series patches.
But Rong teaches –
* * *
and the prompts are concatenated with hidden embeddings derived from time-series patches (Rong, right column of p. 2, “3. VLPT: Proposed Prompt Learning Baseline,” second paragraph, teaches “[t]hese encoders encode the visual and text inputs, respectively, into a joint hidden space 𝐑d, where vision semantics and language modalities are well-aligned [54];” Rong, right column of p. 3, “3. VPT: Proposed Prompt Learning Baseline, Visual Prompt Learning,” first paragraph, teaches “In the context of image encoding with L layers, the i-th layer output li, where i = 1, 2, . . . ,L, can be expressed as:
PNG
media_image7.png
52
410
media_image7.png
Greyscale
where c ∈ 𝐑d, denotes the classification token. Zi = {z1, . . . , zS} ∈ 𝐑dxS, represents a set of input image patch tokens of the i-th layer, with length S [(that is, “joint hidden space 𝐑d” includes hidden embeddings derived from time-series patches)]. Furthermore, the learnable visual prompt PI ∈ Rd×N [(that is, the prompts)], where N is the length of visual prompt, is introduced into the input sequence of the i-th layer [(that is, being joined in sequence, which is concatenates)], which is following as:
PNG
media_image8.png
42
369
media_image8.png
Greyscale
[(that is, the prompts are concatenated with hidden embeddings derived from time-series patches)].There are two visual prompt variants, VPT-Shallow and VPT-Deep. In the VPT-Shallow approach, the class token c joined with image patch tokens and visual prompts is taken as the input of the first layer, while VPT-Deep inserts independent visual prompts into each layer.”).
Xing and Rong are from the same or similar field of endeavor. Xing teaches a Dual-modality Prompt Tuning (DPT) paradigm through learning text prompts and visual prompts for both the text and image encoder simultaneously. Rong teaches a simple dual-modal prompt tuning baseline for the few-shot image classification task, called vision-language prompt tuning (VLPT).
Thus, it would have been obvious for a person having ordinary skill as of the effective filing date of the Applicant’s claimed invention to modify Xing pertaining to text and visual prompt tuning of a pretrained model with the concatenation of prompts and hidden embeddings of Rong.
The motivation to do so is because “retrieval mechanisms to cache the knowledge representations from downstream tasks. we first construct a retrieval database from training examples, or from external examples when available. We then integrate this retrieval-enhanced mechanism into various stages of a simple prompt learning baseline. By referencing similar samples in the training set, the enhanced model is better able to adapt to new tasks with few samples.” (Rong, Abstract).
17. Claims 6, 13, and 20 are rejected under 35 U.S.C. § 103 as being unpatentable over Xing et al., “Dual Modality Prompt Tuning for Vision-Language Pre-Trained Model,” arXiv (16 Feb 2023) [hereinafter Xing] in view of Nie et al., "A Time Series Is Worth 64 Words: Long-Term Forecasting With Transformers," arXiv (March 2023) [hereinafter Nie].
Regarding claims 6, 13, and 20, Xing teaches all of the limitations of claims 1, 8, and 15, respectively, as described above in detail.
Though Xing teaches transferable representations can be adapted to a wide range of downstream tasks via prompt tuning to probe the beneficial information for downstream tasks from the general knowledge stored in the pre-trained model, Xing, however, does not explicitly teach –
wherein the model prediction in the target time-series domain comprises generating forecasts or anomaly detection based on the input data.
But Nie teaches -
wherein the model prediction in the target time-series domain comprises generating forecasts or anomaly detection based on the input data (Nie, Abstract, teaches a “channel-independent patch time series Transformer (PatchTST) can improve the long-term forecasting accuracy significantly when compared with that of [state-of-the-art] Transformer-based models [(that is, the model prediction in the target time-series domain comprises generating forecasts or anomaly detection based on the input data)]”).
Xing and Nie are from the same or similar field of endeavor. Xing teaches a Dual-modality Prompt Tuning (DPT) paradigm through learning text prompts and visual prompts for both the text and image encoder simultaneously. Nie teaches the use of transformer encoders where patches are mapped to a transformer latent space of dimension D, where multi-head attention transforms the inputs to query matrices.
Thus, it would have been obvious for a person having ordinary skill as of the effective filing date of the Applicant’s claimed invention to modify Xing pertaining to text and visual prompt tuning of a pretrained model with the forecast generation of Nie.
The motivation to do so is because “[o]ur channel-independent patch time series Transformer (PatchTST) can improve the long-term forecasting accuracy significantly when compared with that of SOTA Transformer-based models. We also apply our model to self-supervised pretraining tasks and attain excellent fine-tuning performance, which outperforms supervised training on large datasets. Transferring of masked pre-trained representation on one dataset to others also produces SOTA forecasting accuracy.” (Nie, Abstract).
18. Claims 7 and 13 are rejected under 35 U.S.C. § 103 as being unpatentable over Xing et al., “Dual Modality Prompt Tuning for Vision-Language Pre-Trained Model,” arXiv (16 Feb 2023) [hereinafter Xing] in view of US Published Application 20240428937 to Natarajan et al. [hereinafter Natarajan].
Regarding claims 7 and 14, Xing teaches all of the limitations of claims 1, 8, and 15, respectively, as described above in detail.
Thought Xing teaches the fine-tuning of pretrained models through prompt tuning, Xing, however, does not explicitly teach –
wherein the fine-tuning of the pretrained model involves utilizing backpropagation and gradient descent techniques to optimize parameters of the model.
But Natarajan teaches -
wherein the fine-tuning of the pretrained model (Natarajan ¶ 0063 teaches “Machine-learned model(s) 108 can be pre-trained on a large corpus of training data in a self-supervised manner (e.g., text, audio, or image denoising). Machine-learned model(s) 108 can be fine-tuned on various tasks. Machine-learned model(s) 108 can be fine-tuned using supervised learning or reinforcement learning or other techniques using a smaller corpus of higher quality training data relevant to different kinds of tasks [(that is, fine-tuning of the pretrained model)]”) involves utilizing backpropagation and gradient descent techniques to optimize parameters of the model (Natarajan ¶ 0361 teaches “model trainer 60 can include a pretraining pipeline for training machine-learned models using various objectives. Parameters of the image-processing model(s) can be trained, in some embodiments, using various training or learning techniques, such as, for example, backwards propagation of errors. For example, an objective or loss can be backpropagated through the pretraining pipeline(s) to update one or more parameters of the model(s) (e.g., based on a gradient of the loss function). Various determinations of loss can be used, such as mean squared error, likelihood loss, cross entropy loss, hinge loss, or various other loss functions. Gradient descent techniques can be used to iteratively update the parameters over a number of training iterations [(that is, with a “smaller corpus,” the fin-tuning of the pretrained model involves utilizing backpropagation and gradient descent techniques to optimize parameters of the model)]”).
Xing and Natarajan are from the same or similar field of endeavor. Xing teaches a Dual-modality Prompt Tuning (DPT) paradigm through learning text prompts and visual prompts for both the text and image encoder simultaneously. Natarajan teaches the present disclosure provides instruction prompt tuning techniques that can be used in lieu of or in conjunction with fine-tuning to improve output performance in a cost-effective manner.
Thus, it would have been obvious for a person having ordinary skill as of the effective filing date of the Applicant’s claimed invention to modify Xing pertaining to text and visual prompt tuning of a pretrained model with the backpropagation and gradient descent techniques of Natarajan.
The motivation to do so is because, “[a]dvantageously, example implementations of the present disclosure can provide high-quality alignment with specialized task domains in a data-efficient manner by leveraging machine-learned prompt values in conjunction with exemplar prompt values curated from example query responses from domain experts.” (Natarajan ¶ 0027).
Conclusion
19. The prior art made of record and not relied upon is considered pertinent to Applicant's disclosure:
(US Published Application 20240220722 to Khattak et al.) teaches multi-modal prompt learning of vision-language models. Encodings of image-text pairs can be combined with image prompts and text prompts before being input into an image encoder and text encoder of a vision-language model respectively. The image prompt can be generated using the text prompt using a vision-language coupling function to encourage synergy between the two prompts. The combination of encodings and prompts can be fed through the transformer layers of the encoders, and the output of each layer can be combined with a new prompt before entering the next layer, up until a specific depth. The subsequent transformer layers can process the output and generate a final representation for the image and text which can then be used for downstream tasks.
(Tan et al., “Virtual Node Tuning for Few-shot Node Classification,” arXiv (09 June 2023)) teaches utilizes a pretrained graph transformer as the encoder and injects virtual nodes as soft prompts in the embedding space, which can be optimized with few-shot labels in novel classes to modulate node embeddings for each specific FSNC task. A unique feature of virtual node tuning (VNT) is that, by incorporating a Graph-based Pseudo Prompt Evolution (GPPE) module, VNT-GPPE can handle scenarios with sparse labels in base classes.
(Bahng et al., “Exploring visual Prompts for Adapting Large-Scale Models,” arXiv (2022)) teaches We investigate the efficacy of visual prompting to adapt large-scale models in vision. Following the recent approach from prompt tuning and adversarial reprogramming, we learn a single image perturbation such that a frozen model prompted with this perturbation performs a new task. Through comprehensive experiments, we demonstrate that visual prompting is particularly effective for CLIP and robust to distribution shift, achieving performance competitive with standard linear probes. We further analyze properties of the downstream dataset, prompt design, and output transformation in regard to adaptation performance.
(Iwata et al., "Few-Shot Learning for Time-Series Forecasting," arXiv (2020)) teaches a few-shot learning method that forecasts a future value of a time-series in a target task given a few time-series in the target task. Our model is trained using time-series data in multiple training tasks that are different from target tasks. Our model uses a few time-series to build a forecasting function based on a recurrent neural network with an attention mechanism. With the attention mechanism, we can retrieve useful patterns in a small number of time-series for the current situation. Our model is trained by minimizing an expected test error of forecasting next timestep values. We demonstrate the effectiveness of the proposed method using 90 time-series datasets.
20. Any inquiry concerning this communication or earlier communications from the Examiner should be directed to KEVIN L. SMITH whose telephone number is (571) 272-5964. Normally, the Examiner is available on Monday-Thursday 0730-1730.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, Applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the Examiner by telephone are unsuccessful, the Examiner’s supervisor, KAKALI CHAKI can be reached on 571-272-3719. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/K.L.S./
Examiner, Art Unit 2122
/KAKALI CHAKI/Supervisory Patent Examiner, Art Unit 2122
Prosecution Insights