DETAILED ACTION
This office action is responsive to applicant’s amendment and arguments filed 12/16/2025.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Applicant claims the benefit of US Provisional Application No. 63/385,180, filed 11/28/2022. Claims 1-20 have been afforded the benefit of this filing date.
Response to Arguments
Regarding the provisional rejection of claims 1-20 on the ground of nonstatutory double patenting, it cannot be held in abeyance according to MPEP 804: “As filing a terminal disclaimer, or filing a showing that the claims subject to the rejection are patentably distinct from the reference application’s claims, is necessary for further consideration of the rejection of the claims, such a filing should not be held in abeyance. Only compliance with objections or requirements as to form not necessary for further consideration of the claims may be held in abeyance until allowable subject matter is indicated.” Therefore, the provisional rejection is maintained.
Applicant's arguments filed 12/16/2025 regarding the rejection of the independent claims 1, 13, and 20 have been fully considered but they are not persuasive.
Firstly, applicant argues that the amended limitations to the independent claims regarding the method of vector addition, shown on page 9, overcome the rejection under 35 U.S.C. 103. This method of vector addition, involving placing the origins of vectors at a common point (referred to as the “parallelogram law” in the specification”) is well known in the art as a standard method of calculating vector sums. Schtein does not explicitly recite this limitation, but it suggests vector addition via summing each dimension independently ([0059]-[0061]). One of ordinary skill in the art will recognize that these methods are functionally identical and will produce the same result; the method of the independent claims is simply a means of visually or geometrically representing the same summation. However, for the sake of thoroughness, an additional reference (Cuemath) has been included which explicitly teaches the claimed method of vector summation.
Secondly, in regard to the applicant's argument that the color vector summation and comparison taught by Schtein does not teach the claimed limitations because it is used for object recognition in a vending machine, a recitation of the intended use of the claimed invention must result in a structural difference between the claimed invention and the prior art in order to patentably distinguish the claimed invention from the prior art. If the prior art structure is capable of performing the intended use, then it meets the claim.
Finally, applicant argues that the combination of the references of Bahng and Schtein are improper because the simpler, vector arithmetic-based method of color vector comparison of Schtein “teaches away” from the generative adversarial network-based method of Bahng. Neural networks fundamentally operate using mathematical operations on vectors representing a relevant feature space, which is also what Schtein teaches (the 100-dimensional color space referred to in Schtein [0059]-[0061]); the two inventions do not differ in in principle as much as the applicant seems to suggest. The limitations of claims 11-12 suggests that the applicant recognizes the possibility of the methods in the independent claims being incorporated into a machine learning model. Also, the existence of a more advanced method does not preclude the use of a simpler method, not necessarily as a replacement but possibly in addition; there are reasons why one may prefer a simpler method, such as the increase in user control and feedback taught by Schtein [0098].
If the applicant is suggesting that the inventions of Bahng and Schtein may not be physically/literally compatible, the test for obviousness is not whether the features of a secondary reference may be bodily incorporated into the structure of the primary reference; nor is it that the claimed invention must be expressly suggested in any one or all of the references. Rather, the test is what the combined teachings of the references would have suggested to those of ordinary skill in the art. See In re Keller, 642 F.2d 413, 208 USPQ 871 (CCPA 1981).
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claims 1-19 provisionally rejected on the ground of nonstatutory double patenting as being unpatentable over claims 8-13 of copending Application No. 18/521,263 (reference application) in view of Cuemath. ("Parallelogram Law of Vector Addition". Retrieved from Wayback Machine (5 Jul 2022): https://web.archive.org/web/20220705180506/https://www.cuemath.com/calculus/parallelogram-law-of-vector-addition/). Although the claims at issue are not identical, they are not patentably distinct from each other because the current application claims are broader than the copending application claims and are therefore an obvious variant thereof. Please see the comparison tables below. Current application claims with equivalent, or trivial variations of, limitations are grouped together.
Current application (18/521,639)
Copending application (18/521,263)
Claim 1, 13
Claim 1, 7, 8 (and Cuemath)
Claim 5, 15
Claim 11
Claim 6
Claim 9
Claim 7, 16
Claim 10
Claim 8, 17
Claim 11
Claim 9, 18
Claim 12
Claim 10
Claim 13
Claim 11
Claim 1, 7, 8
Claim 12, 19
Claim 1, 7, 8
Claim 1 - Current application (18/521,639)
Claim 8 - Copending application (18/531,263)
A device, comprising: a processor; a memory communicatively coupled to the processor; and a logic configured to:
(Claim 1) A device, comprising: a processor; a memory communicatively coupled to the processor; and an image to palette representation logic comprising a neural network configured to…
receive a phrase;
(Claim 1) receive an input data to generate prediction data,
(Claim 7) The device of claim 1, wherein the input data is a phrase.
generate a set of vectors associated with each identified words in the phrase;
identify one or more words in the phrase; generate a set of vectors associated with each identified words;
calculate a first overall vector associated with the received phrase based on a vector summation of the generated set of vectors
calculate a second overall vector associated with the received phrase based on a vector summation of the generated set of vectors;
wherein the vector summation places the generated set of vectors so that origins of the vectors are located at a common point;
Cuemath
generate a palette for a set of colors comprising a predefined number of colors;
generate a palette for a set of colors comprising a predefined number of colors;
calculate a second overall vector associated with the palette based on a vector summation of the set of colors
calculate a third overall vector associated with the palette based on a vector summation of the set of colors;
wherein the vector summation places the set of colors so that the origins of vectors associated with the set of colors are located at a common point;
Cuemath
and in response to a first determination that a first closeness ratio associated with the generated palette is larger than a first predetermined threshold, store the generated palette,
and in response to a second determination that a second closeness ratio associated with the generated palette is larger than a second predetermined threshold, store the generated palette,
wherein the first closeness ratio is defined as an inverse of a difference between the calculated first overall vector and the calculated second overall vector.
wherein the second closeness ratio is defined as an inverse of a difference between the calculated second overall vector and the calculated third overall vector.
In addition to the claims listed in the above table, claims 2, 3, and 4 are rejected on the ground of nonstatutory double patenting due to their dependency on claim 1. Likewise, claim 14 is rejected on the ground of nonstatutory double patenting due to its dependency on claim 13.
Claim 20 provisionally rejected on the ground of nonstatutory double patenting as being unpatentable over claim 8 of copending Application No. 18/521,263 in view of Cuemath ("Parallelogram Law of Vector Addition") and Schtein et al. (US 20180096555 A1, hereinafter "Schtein"). The copending claim 8 recites each of the limitations (or a trivial variation) of the current claim 20 except for the currently-claimed a non-transitory computer-readable storage medium for storing instructions that, when executed by the one or more processors, direct the one or more processors.
Schtein teaches a non-transitory computer-readable storage medium for storing instructions that, when executed by the one or more processors, direct the one or more processors ([0126] “A device of an embodiment of this invention may comprise one or more computers, servers, cameras, hand-held electronic devices, non-transitory memory, and communication elements as needed, in any combination, to implement methods of embodiments”, [0145] “Computer—An electronic device capable of executing the algorithms described, comprising a processor, non-transitory memory, and communication ports”).
Schtein is analogous to the current and copending applications because they pertain to the issue of manipulating and comparing color vectors for the purpose of making a determination about a color feature. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have applied the teachings of Schtein to the copending claim 8. The motivation would have been to include long-term storage for a program allowing a computing device to perform the functionality of the current claims.
This is a provisional nonstatutory double patenting rejection because the patentably indistinct claims have not in fact been patented.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 2, 3, 4, 8, 9, 11, 12, 13, 14, 17, 18, 19, and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Bahng et al. (Coloring with Words: Guiding Image Colorization Through Text-based Palette Generation, hereinafter "Bahng") in view of Schtein et al. (US 20180096555 A1, hereinafter "Schtein"), Bargury et al. (US 20210243190 A1, hereinafter "Bargury"), and Cuemath ("Parallelogram Law of Vector Addition". Retrieved from Wayback Machine (5 Jul 2022): https://web.archive.org/web/20220705180506/https://www.cuemath.com/calculus/parallelogram-law-of-vector-addition/).
Regarding claim 1, Bahng teaches a device, comprising: a processor; a memory communicatively coupled to the processor; and a logic configured to:
receive a phrase (pg. 2 “our model can generate multiple plausible color palettes when given rich text input, including both single- and multi-word descriptions”);
generate a set of vectors associated with each identified words in the phrase (fig. 4 input text x = {x1, · · ·, xT }, pg. 6 ‘Objective Function’ section describes how each word in the input text is represented as a vector: “Let xi ∈ R300 be word vectors initialized by 300-dimensional pre-trained vectors from GloVe [29]”; pg. 8 ‘Networks Architecture’ section describes how the set of word vectors x = {x1, · · · , xT} is encoded as a set of hidden state vectors h = {h1, · · · , hT}, then a set of conditioning vectors ˆc = {cˆ1, · · · , cˆT} is calculated based on the hidden states; within each of these three sets of vectors, each individual vector is associated with an individual word in the input text).
calculate a first overall vector associated with the received phrase based on a vector summation of the generated set of vectors (pg. 8 ‘Networks Architecture’ section: equation 8 describes a context vector ci calculated as a sum of conditioning vectors ˆc = {cˆ1, · · · , cˆT}; each conditioning vector is calculated based on a hidden state generated from one of the original word vectors; pg. 6 ‘Objective Function’ section also teaches that the conditioning variable of the discriminator is also calculated based on a sum of the conditioning vectors);
generate a palette for a set of colors comprising a predefined number of colors (pg. 8 ‘Networks Architecture’ section: “The GRU hidden state si is given as input to a fully-connected layer f to output the i-th color of the palette ˆyi ∈ R3. The resulting five colors are combined to produce a single palette output ˆy.”);
calculate a second overall vector associated with the palette (pg. 8 ‘Networks Architecture’ section: “The resulting five colors are combined to produce a single palette output ˆy”; suggesting but not explicitly teaching that this value is computed using a sum of color vectors; ‘Discriminator’ subsection describes how the palette is concatenated with the conditioning variable which was previously described to be in vector form, indicating that palette output ˆy is also a vector); and
in response to a first determination associated with a first closeness ratio associated with the generated palette, store the generated palette (pg. 8 ‘Networks Architecture’ section: “By jointly learning features across the encoded text and palette, the discriminator classifies whether the palettes are real or fake.” In a GAN, model outputs (generated palettes, in this case) must pass the discriminator’s test; discriminator function D0 is calculated based on palette output ˆy).
Bahng also teaches that the determination to store the palette is made based on a comparison between a vector associated with the generated palette (palette output ˆy) and a vector associated with the sum of word vectors (the conditioning variable), though the exact calculation in the claimed limitation is not explicitly taught.
Bahng does not explicitly teach a device, comprising: a processor; a memory communicatively coupled to the processor; to calculate the first overall vector wherein the vector summation places the generated set of vectors so that origins of the vectors are located at a common point; or to calculate a second overall vector associated with the palette based on a vector summation of the set of colors, wherein the vector summation places the set of colors so that the origins of vectors associated with the set of colors are located at a common point; or in response to a first determination that a first closeness ratio associated with the generated palette is larger than a first predetermined threshold, store the generated palette, wherein the first closeness ratio is defined as an inverse of a difference between the calculated first overall vector and the calculated second overall vector.
Schtein teaches a device, comprising: a processor; a memory communicatively coupled to the processor ([0126] “A device of an embodiment of this invention may comprise one or more computers, servers, cameras, hand-held electronic devices, non-transitory memory, and communication elements as needed, in any combination, to implement methods of embodiments”, [0145] “Computer—An electronic device capable of executing the algorithms described, comprising a processor, non-transitory memory, and communication ports.”),
calculating a color vector based on a vector summation of the set of colors ([0061] “Applying this technique to all the pixels in an image, we obtain a vector for every pixel. These vectors may be summed to produce an “aggregate color vector,” or simply “color vector” for the image. This is a representation of all the colors in the image.”, [0104] “Another comparison is to use a new “small area color vector.” This small area color vector is computed just as for the aggregate color vectors, only now only the pixels in the small area are included in the summing. For example, if the small area is five by five pixels, there are only 25 vectors to sum, for both the reference feature and the target feature); and
storing [color information] in response to a first determination that a first closeness ratio associated with the [color information] is larger than a first predetermined threshold ([0104] “The comparison is then the Euclidean distance of the two vectors. There is a threshold in this step. If the comparison does not meet or exceed a threshold for quality then the feature is pruned from the candidate feature list. Quality, here, refers to the similarity of the target feature to the reference feature”).
Schtein does not explicitly teach the vector summation or comparison of a color palette; however, it does teach the summation of color vectors for the purpose of comparing the aggregate vectors, determining whether the comparison meets a threshold, and deciding whether to keep or discard a particular color feature; Bahng and Schtein are both analogous to the claimed invention because they pertain to these issues. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined the invention of Bahng with the teachings of Schtein. The motivation would have been to establish a more direct, simplified comparison between the text input and the generated palette that would be easier for a user to adjust.
The combination of Bahng in view of Schtein does not teach wherein the first closeness ratio is defined as an inverse of a difference between the calculated first overall vector and the calculated second overall vector.
Bargury teaches wherein the first closeness ratio is defined as an inverse of a difference between the calculated first overall vector and the calculated second overall vector ([0047] "A node embedding is a low-dimensional representation of the discrete data found in the graph as a continuous vector of real numbers. Similar nodes will have similar node embeddings.", [0053] "A similarity score is computed as the inverse of the difference between two node embeddings. The similarity score may be represented as 1/(Ni−Nj), where Ni is a node embedding for node i and Nj is a node embedding for node j.").
Bargury and the combination of Bahng in view of Schtein are both analogous to the claimed invention because they pertain to the issue of comparing vectors. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined the vector calculation and comparison of Bahng in view of Schtein with the similarity score of Bargury in order to define a quantitative, observable metric for comparing the vector based on the generated palette with the vector based on the text input, as opposed to relying on the “black-box” neural network.
The combination of Bahng in view of Schtein and Bargury does not explicitly teach calculating the vector summation of the phrase and color vectors according to the limitations: wherein the vector summation places the generated set of vectors so that origins of the vectors are located at a common point, and wherein the vector summation places the set of colors so that the origins of vectors associated with the set of colors are located at a common point, respectively.
Cuemath teaches the aforementioned limitations via discussion of the parallelogram law of vector addition.
Cuemath is analogous to the claimed invention because it pertains to the same issue of vector addition. Furthermore, the parallelogram law of vector addition is a known concept in the art, and is referred to by name in the specification. One of ordinary skill in the art will recognize that this method is functionally identical to the method suggested by Schtein and will produce the same result. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the invention of Bahng in view of Schtein and Bargury with the teachings of Cuemath to use this particular method of vector summation in order to visually or geometrically represent the summation.
Regarding claim 2, the combination of Bahng in view of Schtein, Bargury, and Cuemath teaches the device of claim 1, wherein the logic is configured to display the generated palette to a user (Bahng fig. 1, 2, 17-22, fig. 2 caption “Our model can produce a diverse selection of palettes when given a text input. Users can optionally choose which palette to be applied to the final colorization output”).
Regarding claim 3, the combination of Bahng in view of Schtein, Bargury, and Cuemath teaches the device of claim 1, wherein the phrase is received from a user (Bahng pg. 1-2 ‘Introduction’ section: “Through text input, even people without artistic backgrounds can easily create color palettes that convey high-level concepts. Since our model uses text to visualize aesthetic concepts, its range of future applications can encompass text to even speech”).
Regarding claim 4, the combination of Bahng in view of Schtein, Bargury, and Cuemath teaches the device of claim 1, wherein the logic is configured to:
access portions of a color spectrum, wherein the accessed portions include data of colors that are visible to human eye (Bahng pg. 4-5 ‘Palette-and-Text (PAT) Dataset’ section describes the dataset used to train the neural networks described by the invention, which contains pairs of text and color palettes; fig. 3 gives an example of the dataset’s color palettes, which are within the visible spectrum for humans); and
generate the palette for the set of colors comprising the predefined number of colors based on the accessed portions (Bahng pg. 4 “This dataset allows us to train our models for predicting semantically consistent color palettes with textual inputs.”).
Regarding claim 8, the combination of Bahng in view of Schtein, Bargury, and Cuemath teaches the device of claim 1, wherein the logic is configured to:
receive a set of words (Bahng pg. 2 “our model can generate multiple plausible color palettes when given rich text input, including both single- and multi-word descriptions”);
assign a weight to each of the received words (Bahng pg. 8 equation 9, a weight αij is calculated for each conditioning vector, where each conditioning vector represents an individual word of the input text), wherein the assigned weight is a number between 0 and 1 (Bahng equation 9, when calculating αij, the numerator is an exponential function, and therefore a positive number; the denominator is the sum of one or more exponential functions including the function in the numerator, and therefore a positive number greater than or equal to the numerator; the resulting quotient must be greater than 0 and less than or equal to 1);
generate a second set of vectors associated with each of the received words (Bahng fig. 4 input text x = {x1, · · , xT }, pg. 6 ‘Objective Function’ section describes how each word in the input text is represented as a vector: “Let xi ∈ R300 be word vectors initialized by 300-dimensional pre-trained vectors from GloVe [29]”; pg. 8 ‘Networks Architecture’ section describes how the set of word vectors x = {x1, · · · , xT} is encoded as a set of hidden state vectors h = {h1, · · · , hT}, then a set of conditioning vectors ˆc = {cˆ1, · · · , cˆT} is calculated based on the hidden states; within each of these three sets of vectors, each individual vector is associated with an individual word in the input text);
calculate a set of weighted vectors by applying the assigned weight to the associated received word (Bahng pg. 8 equation 8, each conditioning vector ˆcj, representing an individual word of the input text, is multiplied by weight αij); and
calculate a third overall vector associated with the received set of words based on the vector summation of the calculated set of weighted vectors (Bahng pg. 8 equation 8, context vector ci is calculated as the sum of the weighted conditioning vectors).
Regarding claim 9, the combination of Bahng in view of Schtein, Bargury, and Cuemath teaches the device of claim 8, wherein the logic is configured to:
in response to a second determination that a second closeness ratio associated with the generated palette is larger than a second predetermined threshold, store the generated palette (Schtein [0104] “The comparison is then the Euclidean distance of the two vectors. There is a threshold in this step. If the comparison does not meet or exceed a threshold for quality then the feature is pruned from the candidate feature list. Quality, here, refers to the similarity of the target feature to the reference feature.”), wherein the second closeness ratio is defined as an inverse of a difference between the calculated second overall vector and the calculated third overall vector (Bargury [0047] "A node embedding is a low-dimensional representation of the discrete data found in the graph as a continuous vector of real numbers. Similar nodes will have similar node embeddings.", [0053] "A similarity score is computed as the inverse of the difference between two node embeddings. The similarity score may be represented as 1/(Ni−Nj), where Ni is a node embedding for node i and Nj is a node embedding for node j.").
The teachings of Schtein and Bargury being combined with the invention of Bahng are the same as those previously described for the rejection of claim 1; therefore, the motivation for doing so is also the same.
Regarding claim 11, the combination of Bahng in view of Schtein, Bargury, and Cuemath teaches the device of claim 1, wherein the logic includes one or more artificial intelligence models (Bahng pg. 3 ‘Introduction’ section: “(1) We propose a novel deep neural network architecture that can generate multiple color palettes based on natural-language text input.”, pg. 6 “Text2Colors consists of two networks: Text-to-Palette Generation Networks (TPN) and Palette-based Colorization Networks (PCN). We train the first networks to generate color palettes given a multi-word text and then train the second networks to predict reasonable colorizations given a grayscale image and the generated palettes. We utilize conditional GANs (cGAN) for both networks.”), and wherein the one or more artificial intelligence models include at least one of: a convolutional neural network, a region-based convolutional neural network, and a You Only Look Once neural network (Bahng pg. 9 ‘Networks Architecture’ section: “As our discriminator D1, we use a variant of the DCGAN architecture [30]. The image and conditioning variable p are concatenated and fed into a series of conv-leaky relu layers to jointly learn features across the image and the palette”; DCGAN (Deep Convolutional Generative Adversarial Network) is a convolutional neural network architecture).
Regarding claim 12, the combination of Bahng in view of Schtein, Bargury, and Cuemath teaches the device of claim 11, wherein the one or more artificial intelligence models are configured to at least:
generate the set of vectors associated with each identified words in the phrase (Bahng fig. 4 input text x = {x1, · · , xT }, pg. 6 ‘Objective Function’ section describes how each word in the input text is represented as a vector: “Let xi ∈ R300 be word vectors initialized by 300-dimensional pre-trained vectors from GloVe [29]”; pg. 8 ‘Networks Architecture’ section describes how the set of word vectors x = {x1, · · · , xT} is encoded as a set of hidden state vectors h = {h1, · · · , hT}, then a set of conditioning vectors ˆc = {cˆ1, · · · , cˆT} is calculated based on the hidden states; within each of these three sets of vectors, each individual vector is associated with an individual word in the input text),
calculate the first overall vector (Bahng pg. 8 ‘Networks Architecture’ section: equation 8 describes a context vector ci calculated as a sum of conditioning vectors ˆc = {cˆ1, · · · , cˆT}; each conditioning vector is calculated based on a hidden state generated from one of the original word vectors) and the second overall vector (pg. 8 ‘Networks Architecture’ section: “a single palette output ˆy”, ‘Discriminator’ subsection describes how the palette is concatenated with the conditioning variable which was previously described to be in vector form, indicating that palette output ˆy is also a vector),
generate the palette for the set of colors comprising the predefined number of colors (Bahng pg. 8 ‘Networks Architecture’ section: “The GRU hidden state si is given as input to a fully-connected layer f to output the i-th color of the palette ˆyi ∈ R3. The resulting five colors are combined to produce a single palette output ˆy.”), and
determine whether the closeness ratio associated with the generated palette is larger than the predetermined threshold (Schtein [0104] “The comparison is then the Euclidean distance of the two vectors. There is a threshold in this step. If the comparison does not meet or exceed a threshold for quality then the feature is pruned from the candidate feature list. Quality, here, refers to the similarity of the target feature to the reference feature.”).
All of the teachings of Bahng described for the claim 12 rejection are implemented as part of the Text-to-Palette Generation Networks (TPN), which uses a conditional generative adversarial network architecture (Bahng pg. 6).
The teachings of Schtein being combined with the invention of Bahng are the same as those previously described for the rejection of claim 1; therefore, the motivation for doing so is also the same. The teachings of Schtein are not explicitly implemented using a neural network; however, as Bahng teaches a similar feature implemented using a neural network (discriminator D0, pg. 6 and 8), it also would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have implemented the teachings of Schtein using the neural network of Bahng.
Regarding claims 13, 14, 17, 18, and 19, they are rejected using the same references, rationales, and motivations to combine described in the rejections of claims 1, 4, 8, 9, and 11-12 respectively.
Regarding claim 20, it is rejected using the same references, rationales, and motivation to combine described in the rejection of claim 1, with the additional limitations of a system, comprising: one or more devices; one or more processors coupled to the one or more devices; and a non-transitory computer-readable storage medium for storing instructions that, when executed by the one or more processors, direct the one or more processors.
Schtein teaches a system ([0164] “Embodiments of this invention explicitly include devices and systems to implement any combination of all methods described in the claims, specification and drawings.”), comprising: one or more devices; one or more processors coupled to the one or more devices; and a non-transitory computer-readable storage medium for storing instructions that, when executed by the one or more processors, direct the one or more processors ([0126] “A device of an embodiment of this invention may comprise one or more computers, servers, cameras, hand-held electronic devices, non-transitory memory, and communication elements as needed, in any combination, to implement methods of embodiments”, [0145] “Computer—An electronic device capable of executing the algorithms described, comprising a processor, non-transitory memory, and communication ports”).
Bahng and Schtein are both analogous to the claimed invention because they pertain to the issue of manipulating and comparing color vectors for the purpose of making a determination about a color feature. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined the invention of Bahng with the teachings of Schtein. The motivation would have been to establish a physical device capable of running the palette generation and image colorization model taught by Bahng.
Claim(s) 5 and 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Bahng (Coloring with Words: Guiding Image Colorization Through Text-based Palette Generation) in view of Schtein (US 20180096555 A1), Bargury (US 20210243190 A1), and Cuemath ("Parallelogram Law of Vector Addition") as applied to claims 1 and 13 above, and further in view of Aggarwal et al. (US 20220277039 A1, hereinafter "Aggarwal").
Regarding claim 5, the combination of Bahng in view of Schtein, Bargury, and Cuemath teaches the device of claim 1, but does not explicitly teach wherein the logic is configured to parse the received phrase to identify the words.
Aggarwal teaches wherein the logic is configured to parse the received phrase to identify the words ([0090] “In some examples, the color term comprises a base color and a color modifier. If the query includes the word “and”, the user may be inquiring about images with multiple independent colors. Therefore, the color terms are tokenized, and color embeddings are determined separately and summed. The color term may be ‘yellow’, ‘fuchsia’, ‘greenish-blue’, or the like, but the present disclosure is not limited to these colors and may decipher any color term. Additionally, the color terms are not limited to the English language and may be from any natural language such as Spanish, French, Italian, or the like.”
[0091] “The system of the present disclosure may generate color palettes 615 for rarely used and complex colors, colors with misspellings, and color term in different languages. As shown in FIG. 6, the word “fuchsia” is misspelled as “fuchia”. The present disclosure can handle misspellings, as the misspelled word may be very near to the actual color in the cross-lingual word embedding space and therefore the color palette that is generated will be very similar.”).
Aggarwal and the combination of Bahng in view of Schtein, Bargury, and Cuemath are both analogous to the claimed invention because they are in the same field of generating a color palette from text input. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have combined the invention of Bahng in view of Schtein, Bargury, and Cuemath with the text parsing of Aggarwal. The motivation would have been to add the ability to pre-process the input text and correct mistakes, as taught by Aggarwal.
Regarding claim 15, it is rejected using the same references, rationales, and motivations to combine described in the rejections of claim 5.
Claim(s) 6 is/are rejected under 35 U.S.C. 103 as being unpatentable over Bahng (Coloring with Words: Guiding Image Colorization Through Text-based Palette Generation) in view of Schtein (US 20180096555 A1), Bargury (US 20210243190 A1), and Cuemath ("Parallelogram Law of Vector Addition") as applied to claim 1 above, and further in view of Dodeja et al. (US 20220180116 A1, hereinafter "Dodeja").
Regarding claim 6, the combination of Bahng in view of Schtein, Bargury, and Cuemath teaches the device of claim 1, but does not explicitly teach wherein a user selects the predefined number of colors.
Dodeja teaches wherein a user selects the predefined number of colors (fig. 3 element 310, para. [0056] “In this particular example, the color palette 114b includes an extraction constraint field 310 that enables a user to specify a maximum number n of colors to utilize to generate the color palette 114b. For instance, if the color module 110 extracts more than n colors from the source image 112b, the colors with the highest area values 210 are utilized to generate the color palette 114b up to n different colors, with remaining colors omitted from the color palette 114b”).
Dodeja and the combination of Bahng in view of Schtein, Bargury, and Cuemath are both analogous to the claimed invention because they are in the same field of generating a color palette from user input. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have modified the color palette generator of Bahng in view of Schtein, Bargury, and Cuemath with the invention of Dodeja to allow a user to specify the number of colors in the generated palette. The motivation would have been to allow a user to customize the output to their individual needs.
Claim(s) 7 and 16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Bahng (Coloring with Words: Guiding Image Colorization Through Text-based Palette Generation) in view of Schtein (US 20180096555 A1), Bargury (US 20210243190 A1), and Cuemath ("Parallelogram Law of Vector Addition") as applied to claims 1 and 13 above, and further in view of Kim et al. (KR 102248468 B1, hereinafter "Kim").
Regarding claim 7, the combination of Bahng in view of Schtein, Bargury, and Cuemath teaches the device of claim 1, wherein the logic is configured to: access a database comprising pairs of colors and corresponding adjectives (Bahng pg. 4 “This section introduces our manually curated dataset named Palette-and-Text (PAT). PAT contains 10,183 text and five-color palette pairs, where the set of five colors in a palette is associated with its corresponding text description as shown in Figs. 3(b)-(d)”, fig. 3 shows several examples of adjective and palette pairs including colors for ‘frozen’, ‘neon’, ‘colorful’, and ‘modest’).
The combination of Bahng in view of Schtein, Bargury, and Cuemath does not explicitly teach and determine an adjective for each of the set of colors of the generated palette.
Kim teaches an apparatus that generates a color palette including determine an adjective for each of the set of colors of the generated palette (fig. 5-8 color palettes correspond to fig. 9-10 adjectives, [0080] “Referring to FIGS. 9 and 10, the processor (200) can set a plurality of first adjectives corresponding to a plurality of color wheels. In the examples of FIGS. 9 and 10, the processor (200) can set pure, pretty, accentual, professional, charismatic, reasonable, sophisticated, and chic as the first adjectives in the color wheel corresponding to pl. lt, s, dk, vdk, dkg, mt, and ltg. Likewise, the processor (200) can set the corresponding first adjective for the remaining color wheels as in the examples of FIGS. 9 and 10.”).
Kim and the combination of Bahng in view of Schtein, Bargury, and Cuemath are both analogous to the claimed invention because they are in the same field of generating a color palette from user input. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have modified the color palette generator of Bahng in view of Schtein, Bargury, and Cuemath with the teachings of Kim to match adjectives to individual colors rather than full palettes. The motivation would have been to use the stored adjectives to help generate color palettes when the user input indicated an emotion aligning with a stored adjective (Kim [0076], [0082] - [0084]).
Regarding claim 16, it is rejected using the same references, rationales, and motivations to combine described in the rejections of claim 7.
Claim(s) 10 is/are rejected under 35 U.S.C. 103 as being unpatentable over Bahng (Coloring with Words: Guiding Image Colorization Through Text-based Palette Generation) in view of Schtein (US 20180096555 A1), Bargury (US 20210243190 A1), and Cuemath ("Parallelogram Law of Vector Addition") as applied to claim 1 above, and further in view of Swint et al. (US 20200410303 A1, hereinafter "Swint").
Regarding claim 10, the combination of Bahng in view of Schtein, Bargury, and Cuemath teaches the device of claim 9, but does not teach wherein a user selects the assigned weights.
Swint teaches a weighted combination of vectors wherein a user selects the assigned weights ([0062] “In Step 408, an aggregated vector is generated from the classification vectors. In one or more embodiments, the aggregated vector is generated with a combining process to which the classification vectors are input and the aggregated vector is output. As an example, the combining process may add the classification vectors and then normalize the resulting sum to form the aggregated vector. As another example, the combining process may take a weighted average of the classification vectors to form the aggregated vector with the weights determined using the feature vectors. One or more of the feature vectors may have user selected weights that are used to perform the weighted average”).
Swint and the combination of Bahng in view of Schtein, Bargury, and Cuemath are both analogous to the claimed invention because they pertain to the same issue of comparing and classifying data represented in vector format; both involve calculating a combination of multiple weighted vectors. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to have modified the invention of Bahng in view of Schtein, Bargury, and Cuemath with the teachings of Swint. The motivation would have been to allow a user to select which words in the input phrase have a greater effect on the generated palette, giving them a higher degree of control over the process.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to BENJAMIN STATZ whose telephone number is (571)272-6654. The examiner can normally be reached Mon-Fri 8am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Tammy Goddard can be reached at (571)272-7773. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/BENJAMIN TOM STATZ/Examiner, Art Unit 2611
/TAMMY PAIGE GODDARD/Supervisory Patent Examiner, Art Unit 2611