Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
DETAILED ACTION
The Action is responsive to the Amendments and Remarks filed as part of the Request for Continued Examination on 1/21/2026. Claims 1-20 are pending claims. Claims 1, 8, and 15 are written in independent form.
Priority
Applicant's claim for benefit as a continuation of previously filed non-provisional application 16/681,223, filed 11/12/2019, is acknowledged.
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claims 1-20 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1, 2, 4, 5, 8, 9, 11, 12, 15, 16, and 19 of U.S. Patent No. 12,050,639. Although the claims at issue are not identical, they are not patentably distinct from each other because every limitation in the present application claims is similar to a limitation recited in U.S. Patent No. 12,050,639.
Present Application Claims
Corresponding Claims in U.S. Patent No. 12,050,639
Claim 1
Claims 1, 5, and 7
Claim 2
Claim 2
Claim 3
Claim 1
Claim 4
Claim 1
Claim 5
Claim 5
Claim 6
Claim 5
Claim 7
Claim 4
Claim 8
Claims 8, 5, and 7
Claim 9
Claim 9
Claim 10
Claim 8
Claim 11
Claim 8
Claim 12
Claim 12
Claim 13
Claim 12
Claim 14
Claim 11
Claim 15
Claims 15, 5, and 7
Claim 16
Claim 16
Claim 17
Claim 15
Claim 18
Claim 15
Claim 19
Claim 19
Claim 20
Claim 19
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to non-patentable subject matter. The claimed invention is directed to one or more abstract ideas without significantly more. The judicial exception is not integrated into a practical application. The claims do not include additional elements that are sufficient to amount to significantly more than judicial exception. The eligibility analysis in support of these findings is provided below.
As per Claims 1, 8, and 15,
STEP 1:In accordance with Step 1 of the eligibility inquiry (as explained in MPEP 2106), the claimed method (claims 1-7), non-transitory, computer-readable medium (claims 8-14), and system (claims 15-20) are directed to one of the eligible categories of subject matter and therefore satisfies Step 1.
STEP 2A Prong One:The independent claims 1, 8, and 15 recite the following limitations directed to an abstract idea:
Generating, by a generator of a neural network, a first image corresponding to the sketch of the object;
The limitation recites a mathematical concept of a generator executing a mathematical formula by taking as input a sketch of an object and outputting a “first image”.
Concatenating, by a discriminator, the sketch and the first image to generate a concatenated image;
The limitation recites a mathematical concept of a discriminator executing a mathematical formula that takes as input a sketch and a first image, and outputs a concatenated image of the two inputs.
Encoding, by the discriminator, the concatenated image;
The limitation recites a mathematical concept of a discriminator executing a mathematical formula in the form of executing an encoding formula on the concatenated image.
Determining whether the first image is valid or invalid by assigning the encoded concatenated image with a binary value;
The limitation recites a mental process of observation, evaluation, judgement, and/or opinion capable of being performed by the human mind by observing and evaluating an encoded concatenation of a sketch and first image and based on the observation and evaluation, making a judgement and/or opinion of a binary value representing whether the first image is valid or invalid.
Based on the first image being valid, extracting a feature of the first image;
The limitation recites a mental process of observation, evaluation, judgement, and/or opinion capable of being performed by the human mind by observing and evaluating the first image and a determination that the first image is valid, and making a judgement and/or opinion on a particular feature of the first image based on the observation and evaluation.
Identifying a second image previously stored that has a feature similar to the feature of the first image; and
The limitation recites a mental process of observation, evaluation, judgement, and/or opinion capable of being performed by the human mind by observing and evaluating the feature of the first image and features of images previously stored, and making a judgement and/or opinion that a second image previously stored has a feature similar to the feature of the first image based on the observation and evaluation.
STEP 2A Prong Two:Claims 1, 8, and 15 recite that the steps are performed using “a graphical user interface” and “a generator and a discriminator of a neural network” and “a webpage”, which are a high-level recitation of generic computer components and represents mere instructions to apply on a computer as in MPEP 2106.05(f), which does not provide integration into a practical application.
Claim 8 recites that the steps are performed using “a non-transitory, computer-readable medium” and “a machine”, which is a high-level recitation of generic computer components and represents mere instructions to apply on a computer as in MPEP 2106.05(f), which does not provide integration into a practical application.
Claim 15 recites that the steps are performed using “memory” and “one or more processors”, which is a high-level recitation of generic computer components and represents mere instructions to apply on a computer as in MPEP 2106.05(f), which does not provide integration into a practical application.
The claim recites the following additional elements:
Obtaining a sketch of an object form a user;
The limitation recites an insignificant extra solution activity as retrieval of data (ie. Mere data gathering) such as ‘obtaining information’ as identified in MPEP 2106.05(g) and does not provide integration into a practical application.
Providing the second image to the user.
The limitation recites an insignificant extra solution activity as sending/providing or receiving data (ie. Mere data gathering) such as ‘obtaining information’ as identified in MPEP 2106.05(g) and does not provide integration into a practical application.
Viewing the additional limitations together and the claim as a whole, nothing provides integration into a practical application.
STEP 2B:
The conclusions for the mere implementation using a computer are carried over and does not provide significantly more.
With respect to “Obtaining a sketch of an object form a user;” identified as insignificant extra-solution activity above this is also WURC as court-identified see MPEP 2106.05(d)(II)(i).
With respect to “Providing the second image to the user.” identified as insignificant extra-solution activity above this is also WURC as court-identified see MPEP 2106.05(d)(II)(i).
Looking at the claim as a whole does not change this conclusion and the claim is ineligible.
As per Dependent Claims 2-7, 9-14, and 16-20,
STEP 1:In accordance with Step 1 of the eligibility inquiry (as explained in MPEP 2106), the claimed method (claims 1-7), non-transitory, computer-readable medium (claims 8-14), and system (claims 15-20) are directed to one of the eligible categories of subject matter and therefore satisfies Step 1.
STEP 2A Prong One:The dependent claims 2-7, 9-14, an 16-20 recite the following limitations directed to an abstract idea:
The limitation(s) of Dependent Claims 2, 9, and 16 includes the step(s) of:
Wherein the sketch of the object is drawn by the user.
The limitation recites a mental process of observation, evaluation, judgement, and/or opinion capable of being performed by the human mind with the assistance of a pen and paper by making a judgement and/or opinion of an object to draw and using the pen and paper to sketch the object based on the judgement and/or opinion.
The limitation(s) of Dependent Claims 4, 11, and 18 includes the step(s) of:
generating a dimensional code reflecting dimensions of the sketch by encoding the modified sketch; and
The limitation recites a mathematical concept of executing a mathematical formula in the form of an encoding that takes as input the modified sketch and outputs dimensional code reflecting dimensions of the sketch.
Generating the first image corresponding to the modified sketch of the object by processing the dimensional code by a plurality of decoders.
The limitation recites a mathematical concept of executing a mathematical formula in the form of inputting the dimensional code reflecting dimensions of the sketch into a plurality of decoders and outputting the first image.
The limitation(s) of Dependent Claims 7 and 14 includes the step(s) of:
Generating a feature vector based on the feature from the first image; and
The limitation recites a mental process of observation, evaluation, judgement, and/or opinion capable of being performed by the human mind by observing and evaluating the first image and the feature from the first image, and based on the observation and evaluation, generating a feature vector.
Mapping the feature vector in a feature-vector space,
The limitation recites a mathematical concept of executing a mathematical formula that takes as input the feature vector and outputs a representation of the feature vector mapped into feature-vector space.
wherein a feature vector corresponding to the second image is disposed within a threshold distance from the feature vector corresponding to the first image in the feature-vector space.
The limitation recites a mental process of observation, evaluation, judgement, and/or opinion capable of being performed by the human mind by observing and evaluating a feature vector corresponding to the second image, a threshold distance, and the feature vector corresponding to the first image in the feature-vector space, and based on the observation and evaluation, making a determination that the feature vector corresponding to the second image is disposed within the threshold distance from the feature vector corresponding to the first image in the feature-vector space.
STEP 2A Prong Two:The claim(s) recite the following additional elements:
The limitation(s) of Dependent Claims 2, 9, and 16 includes the step(s) of:
Wherein the sketch of the object is drawn in the graphical user interface by the user.
The limitation recites “the graphical user interface” for performing the sketch by the user, which is a high-level recitation of generic computer components and represents mere instructions to apply on a computer as in MPEP 2106.05(f), which does not provide integration into a practical application.
The limitation(s) of Dependent Claims 3, 10, and 17 includes the step(s) of:
Sharing, via the graphical user interface, the sketch within a social group including the user and another user; and
The limitation recites an insignificant extra solution activity as sending/sharing or receiving data (ie. Mere data gathering) such as ‘obtaining information’ as identified in MPEP 2106.05(g) and does not provide integration into a practical application.
Receiving, via the graphical user interface, a modification to the sketch from the another user.
The limitation recites an insignificant extra solution activity as sending or receiving data (ie. Mere data gathering) such as ‘obtaining information’ as identified in MPEP 2106.05(g) and does not provide integration into a practical application.
The limitation(s) of Dependent Claims 5, 12, and 19 includes the step(s) of:
Wherein the encoding the modified sketch and processing the dimensional code are via the neural network.
The limitation recites “the neural network” for performing the encoding and processing, which is a high-level recitation of generic computer components and represents mere instructions to apply on a computer as in MPEP 2106.05(f), which does not provide integration into a practical application.
The limitation(s) of Dependent Claims 6, 13, and 20 includes the step(s) of:
Wherein the neural network includes the generator and the discriminator;
The limitation recites an insignificant extra-solution activity as selecting a particular type of software components included in the neural network as identified in MPEP 2106.05(g) and does not provide integration into a practical application.
The generator comprises a first plurality of encoders and the plurality of decoders, each of which are arranged in a serial manner; and
The limitation recites an insignificant extra-solution activity as selecting a particular type of software components and their arrangement included in the generator as identified in MPEP 2106.05(g) and does not provide integration into a practical application.
The discriminator includes a concatenator, a second plurality of encoders, and a classifier.
The limitation recites an insignificant extra-solution activity as selecting a particular type of software components included in the discriminator as identified in MPEP 2106.05(g) and does not provide integration into a practical application.
Viewing the additional limitations together and the claim as a whole, nothing provides integration into a practical application.
STEP 2B:
The conclusions for the mere implementation using a computer are carried over and does not provide significantly more.
With respect to Claims 3, 10, and 17 reciting “Sharing, via the graphical user interface, the sketch within a social group including the user and another user;” identified as insignificant extra-solution activity above this is also WURC as court-identified see MPEP 2106.05(d)(II)(i).
With respect to Claims 3, 10, and 17 reciting “Receiving, via the graphical user interface, a modification to the sketch from the another user.” identified as insignificant extra-solution activity above this is also WURC as court-identified see MPEP 2106.05(d)(II)(i).
With respect to Claims 6, 13, and 20 reciting “Wherein the neural network includes the generator and the discriminator;” identified as insignificant extra-solution activity above this is also WURC when claimed in a merely generic manner as court-identified see MPEP 2106.05(d)(II)(iv).
With respect to Claims 6, 13, and 20 reciting “The generator comprises a first plurality of encoders and the plurality of decoders, each of which are arranged in a serial manner;” identified as insignificant extra-solution activity above this is also WURC when claimed in a merely generic manner as court-identified see MPEP 2106.05(d)(II)(iv).
With respect to Claims 6, 13, and 20 reciting “The discriminator includes a concatenator, a second plurality of encoders, and a classifier.” identified as insignificant extra-solution activity above this is also WURC when claimed in a merely generic manner as court-identified see MPEP 2106.05(d)(II)(iv).
Looking at the claim as a whole does not change this conclusion and the claim is ineligible.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2, 6-9, 13-16, and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Cho et al. (U.S. Pre-Grant publication No. 2021/0263963, hereinafter referred to as Cho) and further in view of Murrish (U.S. Pre-Grant Publication No. 2019/0188534).
Regarding Claim 1:
Cho teaches a method for searching content, the method comprising:
Obtaining, via a graphical user interface, a sketch of an object form a user;
Cho teaches obtaining a sketch drawn by a user (Para. [0058]) where the sketch drawn by the user is “on an external device such as a smartphone” (Para. [0059] & Fig. 3).
Cho further teaches “the user may additionally edit the image by adding a sketch or removing a part of the image with the remover UI element 35, and the content search may be performed based on the additionally edited image” (Para. [0066]) where “the way of drawing an image varies depending on users” (Para. [0190]) thereby teaching multiple users capable of drawing an image on the exemplary electronic device, and thus sharing the sketch to be modified by multiple other users.
Cho further teaches the sketch drawn by the user “on an external device such as a smartphone” (Para. [0059] & Fig. 3) thereby teaching using a graphical user interface to obtain the sketch.
Generating, by a generator of the artificial intelligence model, a first image corresponding to the sketch of the object;
Cho teaches generating “a machine-generated image based on the sketch using an artificial intelligence model trained by an artificial intelligence algorithm” (Para. [0076]) where “the artificial intelligence model may be trained by a generative adversarial network (GAN)” (Para. [0077]). It is noted that GAN models (models trained by a generative adversarial network) necessarily have a generator and a discriminator portion by definition, the generator generating new data that mimics real data, and the discriminator that takes in data generated by the generator and classifies the data as real or fake (valid or invalid).
Determining, by the discriminator of the neural network, whether the first image is valid or invalid by assigning the encoded concatenated image with a binary value;
Cho teaches displaying “a machine-generated image 820 based on a sketch 810 drawn by the user using an artificial intelligence model” (Para. [0106]) and “when the user selected the UI element 830 for search execution, at least one content corresponding to the machine generated image 820 may be searched for and provided” (Para. [0107]). Therefore, Cho teaches authenticating the image by classifying it as being a sufficiently real image to perform the search by the user selecting to use the image for searching.
It is noted that Cho teaches generating “a machine-generated image based on the sketch using an artificial intelligence model trained by an artificial intelligence algorithm” (Para. [0076]) where “the artificial intelligence model may be trained by a generative adversarial network (GAN)” (Para. [0077]). GAN models (models trained by a generative adversarial network) necessarily have a generator and a discriminator portion by definition, the generator generating new data that mimics real data, and the discriminator that takes in data generated by the generator and classifies the data as real or fake which is a binary outcome such as 1 or 0, valid or invalid, yes or no, etc..
Based on the first image being valid, extracting a feature of the first image;
Cho teaches, before performing the search, displaying the sketch and the machine-generated image on a display and then requiring a user to take an extra action to select a button, such as 830 of Fig. 8, that then causes the search to be performed (Paras. [0106]-[0107]).
Cho further teaches “for the content search, at least one of text-based image retrieval (TBIR) and content-based image retrieval (CBIR) may be used” where “the text-based image retrieval method may include, for example, a method for extracting a feature from a machine-generated image, identifying a keyword corresponding to the extracted feature, and searching for a content having a file name and metadata including the identified keyword” and “the content-based image retrieval method may include, for example, a method for digitizing and comparing visual elements such as a color, a texture, a shape, and the like of an image” (Para. [0054]).
Therefore, Cho teaches extracting one or more features from the machine-generated image upon a user authenticating that the machine generated image is classified as a sufficiently real/valid image to be used for searching.
Identifying a second image previously stored that has a feature similar to the feature of the first image; and
Cho teaches “for the content search, at least one of text-based image retrieval (TBIR) and content-based image retrieval (CBIR) may be used” where “the text-based image retrieval method ma include, for example, a method for extracting a feature from a machine-generated image, identifying a keyword corresponding to the extracted feature, and searching for a content having a file name and metadata including the identified keyword” and “the content-based image retrieval method may include, for example, a method for digitizing and comparing visual elements such as a color, a texture, a shape, and the like of an image” (Para. [0054]). Cho further teaches “The electronic device may receive a user command for selecting one of the at least one machine-generated image and search for at least one content corresponding to the image selected according to the user command (S240)” (Para. [0080])
Providing, on a webpage, the second image to the user.
Cho teaches following the search using the machine-generated image for at least one content corresponding to the machine-generated image, “the electronic device may provide the at least one searched content (S250)” to a user (Para. [0080]).
Cho further teaches “there is no limitation to contents to be searched” and “for example, the content provided from the Internet may be searched for” (Paras. [0082]-[0083]). Cho further teaches “the second element 2000 may transmit information regarding a storage address (e.g., URL address) of the machine-generated image to the first element 1000” (Para. [0148]) thereby teaching using URLs to store and access content from the web.
Cho also teaches “contents of various sources may be searched for. For example…an Internet content (for example, result searched on GOOGLE™), and the like may be searched for and provided.” (Para. [0108]) and “referring to FIG. 8, a searched result may be provided for each content source…a UI element 847 corresponding to the Internet content may be displayed” (Para. [0109]) thereby teaching providing and displaying search results on a webpage such as through a Google webpage and using the internet.
Cho explicitly teaches all of the limitations as recited above except:
Generating, by a generator of a neural network, a first image corresponding to the sketch of the object;
Generating, by a discriminator of the neural network, a concatenated image by concatenating the sketch and the first image;
Encoding, by the discriminator of the neural network, the concatenated image;
However, in the related field of endeavor of generating an image from a sketch, Murrish teaches:
Generating, by a generator of a neural network, a first image corresponding to the sketch of the object;
Murrish teaches “the computing device 104 may convert a line drawing to a rendered image using a neural network, such as a conditional Generative Adversarial Network (cGAN)” (Para. [0022]). Murrish further teaches “the cGAN may include a generator and a discriminator…[that] may include a plurality of layers including a combination of one or more convolutional layers, one or more pooling layers, and/or one or more deconvolutional layers” and “the generator may be formed using an encoder-decoder architecture…[which] may be formed as an U-Net architecture” (Para. [0026]). Therefore, Murrish teaches the encoding and processing being performed via the cGAN which is a neural network.
Generating, by a discriminator of the neural network, a concatenated image by concatenating the sketch and the first image;
Murrish teaches the discriminator comprising one or more convolutional layers and a discriminator in a GAN is itself a classifier of sorts (Paras. [0026]) thereby teaching the discriminator including a concatenator and a plurality of encoders as part of the convolutional layers and a classifier. A discriminator, in the context of Murrish, is meant to discriminate a sketch from a generated image, thereby teaching concatenating the sketch and the generated image for discrimination/comparison.
Encoding, by the discriminator of the neural network, the concatenated image;
Murrish teaches the discriminator comprising one or more convolutional layers and a discriminator in a GAN is itself a classifier of sorts (Paras. [0026]) thereby teaching the discriminator including a concatenator and a plurality of encoders as part of the convolutional layers, for processing the generated image and the sketch, and a classifier.
Thus, it would have been obvious to one of ordinary skill in the art, having the teachings of Murrish and Cho at the time that the claimed invention was effectively filed, to have modified the system and method for taking a sketch drawn by a user, producing a machine-generated image from the sketch, and performing a search for stored images similar to the machine-generated image representing the sketch, as taught by Cho, with the generator discriminator architecture, as taught by Murrish.
One would have been motivated to make such modification because while Cho teaches generating the machine generated image from the user’s sketch using a Generative Adversarial Network, Murrish teaches clarification on the GAN by teaching training the cGAN “based on different types of noise to account for irregularities in the hand-drawn image” where “this noise may allow the cGAN to account for variations, e.g., waviness, in the lines such that the cGAN does not exactly follow the lines and generates a more natural output as a result” (Para. [0022]) which would be obvious to a person having ordinary skill in the art as being an advantage to be able to detect variations in the lines for a more natural output.
Regarding Claim 2:
Murrish and Cho further teach:
Wherein the sketch of the object is drawn in the graphical user interface by the user.
Cho teaches the sketch drawn by the user “on an external device such as a smartphone” (Para. [0059] & Fig. 3).
Regarding Claim 6:
Murrish, and Cho further teach:
Wherein the neural network includes the generator and the discriminator;
Murrish teaches “the cGAN may include a generator and a discriminator” where “the generator may be formed using an encoder-decoder architecture” (Para. [0026]).
The generator comprises a first plurality of encoders and the plurality of decoders, each of which are arranged in a serial manner; and
Murrish teaches “the cGAN may include a generator and a discriminator” where “the generator may be formed using an encoder-decoder architecture” (Para. [0026]). Murrish further teaches “in some aspects, the encoder-decoder architecture may be formed as an U-Net architecture” (Para. [0026]) thereby teaching encoders and decoders arranged in a serial manner.
The discriminator includes a concatenator, a second plurality of encoders, and a classifier.
Murrish teaches the discriminator comprising one or more convolutional layers and a discriminator in a GAN is itself a classifier of sorts (Paras. [0026]) thereby teaching the discriminator including a concatenator and a plurality of encoders as part of the convolutional layers, for processing the generated image and the sketch, and a classifier.
Regarding Claim 7:
Murrish and Cho further teach the step of extracting further comprises:
Generating a feature vector based on the feature from the first image; and
Cho teaches “for the content search, at least one of text-based image retrieval (TBIR) and content-based image retrieval (CBIR) may be used” where “the text-based image retrieval method ma include, for example, a method for extracting a feature from a machine-generated image, identifying a keyword corresponding to the extracted feature, and searching for a content having a file name and metadata including the identified keyword” and “the content-based image retrieval method may include, for example, a method for digitizing and comparing visual elements such as a color, a texture, a shape, and the like of an image” (Para. [0054]) thereby teaching a feature vector comprising the extracted features from the machine-generated image for performing the search.
Mapping the feature vector in a feature-vector space, and wherein a feature vector corresponding to the second image is disposed within a threshold distance from the feature vector corresponding to the first image in the feature-vector space.
Cho teaches examples of searching for items that are “similar” to the machine-generated image from the user’s sketch using determined features of the machine-generated image (Paras. [0053], [0054], and [0082]). Cho further teaches arranging multiple output images based on their similarity score to an input image (Para. [0090]) thereby teaching a corresponding threshold distance of features of the machine-generate image used as input to a search for displaying the searched content results.
Regarding Claim 8:
Some of the limitations herein are similar to some or all of the limitations of Claim 1.
Murrish and Cho further teach:
A non-transitory, computer-readable medium having information recorded thereon for searching content items, wherein the information, when read by a machine, causes the machine to perform operations (Cho - Para. [0164]).
Regarding Claim 9:
All of the limitations herein are similar to some or all of the limitations of Claim 2.
Regarding Claim 13:
All of the limitations herein are similar to some or all of the limitations of Claim 6.
Regarding Claim 14:
All of the limitations herein are similar to some or all of the limitations of Claim 7.
Regarding Claim 15:
Some of the limitations herein are similar to some or all of the limitations of Claim 1.
Murrish and Cho further teach:
A system comprising:
Memory storing computer program instructions (Cho - Para. [0020]); and
One or more processors that, in response to executing the computer program instructions, effectuate operations (Cho - Para. [0020]).
Regarding Claim 16:
All of the limitations herein are similar to some or all of the limitations of Claim 2.
Regarding Claim 20:
All of the limitations herein are similar to some or all of the limitations of Claim 6.
Claims 3-5, 10-12, and 17-19 are rejected under 35 U.S.C. 103 as being unpatentable over Cho and Murrish, and further in view of Cao et al. (U.S. Pre-Grant Publication No. 2011/0302522, hereinafter referred to as Cao).
Regarding Claim 3:
Murrish and Cho explicitly teach all of the limitations as recited above except:
Sharing, via the graphical user interface, the sketch within a social group including the user and another user; and
Receiving, via the graphical user interface, a modification to the sketch from the another user.
However, in the related field of endeavor of using a sketch to perform a search, Cao teaches:
Sharing, via the graphical user interface, the sketch within a social group including the user and another user; and
Cho teaches obtaining a sketch drawn by a user (Para. [0058]) where the sketch drawn by the user is “on an external device such as a smartphone” (Para. [0059] & Fig. 3).
Cho further teaches “the user may additionally edit the image by adding a sketch or removing a part of the image with the remover UI element 35, and the content search may be performed based on the additionally edited image” (Para. [0066]) where “the way of drawing an image varies depending on users” (Para. [0190]) thereby teaching multiple users capable of drawing an image on the exemplary electronic device, and thus sharing the sketch to be modified by multiple other users.
Cao teaches “a number of users 101 are interacting with the application which runs on a computer 102…which includes a workspace for sketching” and “the users can use any user input device to draw a sketch and in the example shown, the users 101 may use the interactive display 103 to provide an input device to the application which defines a user-generated sketch 106” (Para. [0015]) thereby teaching multiple users in a social group of users interacting with and modifying the sketch. Cao further teaches multiple users “co-located and viewing the application via the same display (as shown in Fig. 1) or they may be collaborating remotely via a shared display space, where the application may be running on a server” (Para. [0023] & Fig. 1).
Receiving, via the graphical user interface, a modification to the sketch from the another user.
Cao teaches “a number of users 101 are interacting with the application which runs on a computer 102…which includes a workspace for sketching” and “the users can use any user input device to draw a sketch and in the example shown, the users 101 may use the interactive display 103 to provide an input device to the application which defines a user-generated sketch 106” (Para. [0015]) thereby teaching multiple users in a group of users interacting with and modifying the sketch. Cao further teaches multiple users “co-located and viewing the application via the same display (as shown in Fig. 1) or they may be collaborating remotely via a shared display space, where the application may be running on a server” (Para. [0023] & Fig. 1).
Thus, it would have been obvious to one of ordinary skill in the art, having the teachings of Cao, Murrish, and Cho at the time that the claimed invention was effectively filed, to have modified the generator discriminator architecture, as taught by Murrish, and the system and method for taking a sketch drawn by a user, producing a machine-generated image from the sketch, and performing a search for stored images similar to the machine-generated image representing the sketch, as taught by Cho, with the sketching system and method for users to locally or remotely collaborate on a sketch via a shared display space, as taught by Cao.
One would have been motivated to make such modification because Cao teaches multiple users “co-located and viewing the application via the same display (as shown in Fig. 1) or they may be collaborating remotely via a shared display space, where the application may be running on a server” (Para. [0023] & Fig. 1) and it would have been obvious to a person having ordinary skill in the art that allowing multiple users to collaborate on a sketch both locally or remotely would improve the flexibility of the users to collaborate without the limitation of location. Cao further teaches the benefits of implementing a collaboration process by teaching “ideas are commonly generated using a collaborative process known as brainstorming in which a group of people spontaneously share solutions to a problem” and “the process is intended to generate a large number of ideas which can subsequently be analyzed and refined” (Para. [0001]).
Regarding Claim 4:
Cao, Murrish, and Cho further teach:
generating a dimensional code reflecting dimensions of the sketch by encoding the modified sketch; and
Murrish teaches “the computing device 104 may convert a line drawing to a rendered image using a neural network, such as a conditional Generative Adversarial Network (cGAN)” (Para. [0022]). Murrish further teaches “the cGAN may include a generator and a discriminator…[that] may include a plurality of layers including a combination of one or more convolutional layers, one or more pooling layers, and/or one or more deconvolutional layers” and “the generator may be formed using an encoder-decoder architecture…[which] may be formed as an U-Net architecture” (Para. [0026]). The output from the encoding must generate dimensional code reflecting dimensions of the sketch because convolutional layers reduce the dimensions of the input image which are then returned to the original dimension through deconvolutional operations. Therfore, Murrish teaches encoding the line drawing (sketch) using convolutional layers to generate dimensional code reflecting dimensions of the line drawing (sketch).
It is further noted that Murrish teaches “the cGAN may learn different relationships between elements of a vehicle, such as spatial relationships between the different elements (e.g., the placement of a hood relative to a front bumper, a quarter panel, and a windshield), size relationships (e.g., the size of a tire and wheel relative to the size of a wheel well)” (Para. [0025]) which exemplifies the cGAN generating spatial dimensional code of the elements in the input line drawing (sketch) for identifying spatial relationships between elements.
Generating the first image corresponding to the modified sketch of the object by processing the dimensional code by a plurality of decoders.
Murrish teaches “the cGAN may include a generator and a discriminator…[that] may include a plurality of layers including a combination of one or more convolutional layers, one or more pooling layers, and/or one or more deconvolutional layers” and “the generator may be formed using an encoder-decoder architecture…[which] may be formed as an U-Net architecture” (Para. [0026]).
Fig. 8 and Paras. [0041]-[0042] of the present specification have been reviewed which provides support for the decoding of the encoding steps into a rendered image, and specifically states that “a deconvolution operation is sequentially performed on the generated code by the plurality of decoders 720 to obtain a rendered image 740” (Para. [0042]).
Therefore, by teaching a plurality of deconvolutional layers formed as part of a u-net architecture, Murrish is found to teach processing the dimensional code output from the encoders by a plurality of deconvolution layers, understood as reading on claimed plurality of decoders, to generate a rendered image corresponding to the input line drawing (sketch).
Regarding Claim 5:
Cao, Murrish, and Cho further teach:
Wherein the encoding the modified sketch and processing the dimensional code are via the neural network.
Murrish teaches “the computing device 104 may convert a line drawing to a rendered image using a neural network, such as a conditional Generative Adversarial Network (cGAN)” (Para. [0022]). Murrish further teaches “the cGAN may include a generator and a discriminator…[that] may include a plurality of layers including a combination of one or more convolutional layers, one or more pooling layers, and/or one or more deconvolutional layers” and “the generator may be formed using an encoder-decoder architecture…[which] may be formed as an U-Net architecture” (Para. [0026]). Therefore, Murrish teaches the encoding and processing being performed via the cGAN which is a neural network.
Regarding Claim 10:
All of the limitations herein are similar to some or all of the limitations of Claim 3.
Regarding Claim 11:
All of the limitations herein are similar to some or all of the limitations of Claim 4.
Regarding Claim 12:
All of the limitations herein are similar to some or all of the limitations of Claim 5.
Regarding Claim 17:
All of the limitations herein are similar to some or all of the limitations of Claim 3.
Regarding Claim 18:
All of the limitations herein are similar to some or all of the limitations of Claim 4.
Regarding Claim 19:
All of the limitations herein are similar to some or all of the limitations of Claim 5.
Response to Amendment
Applicant’s Amendments, filed on 1/21/2026, are acknowledged and accepted.
In light of the Amendments filed on 1/21/2026, the 112(a) rejection of claims 1-20 have been withdrawn.
Response to Arguments
On page 11 of the Remarks filed on 1/21/2026, Applicant argues with respect to the 101 rejection that “claim 1 recites ‘generating, by a generator of a neural network, a first image corresponding to the sketch of the object; generating, by a discriminator of the neural network, a concatenated image by concatenating the sketch and the first image; encoding, by the discriminator of the neural network, the concatenated image; determining, by the discriminator of the neural network, whether the first image is valid or invalid by assigning the encoded concatenated image with a binary value.’ The above-quoted claim features cannot be performed in the human mind at least because the human mind cannot perform the above-quoted steps by a neural network, and thus the claims do not fall within the "Mental processes" grouping of abstract ideas”Applicant’s argument is moot because at least some of the amended limitations being argued were never stated as being capable of being performed by the human mind. The amended limitations are addressed in full in the rejection above.
On pages 11-12 of the Remarks filed on 1/21/2026, Applicant argues with respect to the 101 rejection that “these claim features extend far beyond fundamental economic principles or practices (including hedging, insurance, mitigating risk); commercial or legal interactions (including agreements in the form of contracts; legal obligations; advertising, marketing or sales activities or behaviors; business relations); managing personal behavior or relationships or interactions between people (including social activities, teaching, and following rules or instructions), and thus do not fall within the enumerated group of Certain methods of organizing”Applicant’s argument is moot because at least some of the amended limitations being argued were never stated as being fundamental economic principles or practices, commercial or legal interactions, managing personal behavior or relationships or interactions between people. The amended limitations are addressed in full in the rejection above.
On pages 12-13 of the Remarks filed on 1/21/2026, Applicant argues with respect to the 101 rejection that “Even assuming, for the sake of argument, that claim 1 does recite an abstract idea (which the Applicant disagrees), Applicant respectfully submits that claim 1 is patent eligible under Prong Two of the Step 2A Analysis” because “The recited features of claim 1 are clearly tied to a practical application, i.e., utilizing a neural network and graphical user interface to search image content based on hand-drawn sketch and providing the search image content on a webpage. The claims provide an improvement to known technical problems caused by lack of a keyword or image for a search. Para. [0004] as filed. Applicant's claimed concept overcomes such technical problems by using a neural network and graphical user interface to provide image search results based on a hand-drawn sketch.”Applicant’s argument is not convincing because it is not the neural network that is providing any search results, but instead merely editing/modifying a sketch that is then used to find a similar image with a similar feature. It is further noted that none of the claims even recite searching as an active step, and the independent claims 1, 8, and 15 merely recite a variation of a method, medium, or system “for searching content” in the preamble.
On page 13 of the Remarks filed on 1/21/2026, Applicant argues with respect to the 101 rejection that “Claim 1’s ‘neural network’ is not a high-level recitation of generic computer component. Rather, claim 1 recites ‘neural network’ in great technical details – ‘generating, by a generator of a neural network, a first image corresponding to the sketch of the object; generating, by a discriminator of the neural network, a concatenated image by concatenating the sketch and the first image; encoding, by the discriminator of the neural network, the concatenated image; determining, by the discriminator of the neural network, whether the first image is valid or invalid by assigning the encoded concatenated image with a binary value.’”Applicant’s argument is not convincing because a generator and discriminator are merely formulas/functions included in a type of neural network (generative adversarial network) executing as mere instructions to apply on a generic computer.
On pages 14-15 of the Remarks filed on 1/21/2026, Applicant states with respect to the 101 rejection that “the Office Action appears to allege that the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. Applicant respectfully disagrees with the contentions, and further submits that claim 1 is patent eligible under Step 2B Analysis from the Guidance.” and “In addition to failing to consider Applicant's claims as an ordered combination and as a whole, the Office Action has improperly analyzed the claims without considering the "additional element(s)" in combination with the non-additional elements. As a result, the Office Action has also incorrectly and improperly identified that the alleged "additional elements" do not amount to significantly more than the alleged judicial exception.”Applicant’s statement has been considered but is not agreed nor does it overcome the analysis of the claims, considered both in ordered combination and as a whole, and the rejection presented above in relation to the amended claims.
On page 16 of the Remarks filed on 1/21/2026, Applicant argues with respect to the 103 rejection that “Neither Cho nor Murrish teaches or suggests” the amended claim language of “determining, by the discriminator of the neural network, whether the first image is valid or invalid by assigning the encoded concatenated image with a binary value” because “Cho's above teachings of machine-generated image based on a user drawn sketch and searching based on the machine-generated image do not teach or suggest assigning an image with a binary value.”Applicant’s argument is not convincing because based on further time for consideration, Cho teaches generating “a machine-generated image based on the sketch using an artificial intelligence model trained by an artificial intelligence algorithm” (Para. [0076]) where “the artificial intelligence model may be trained by a generative adversarial network (GAN)” (Para. [0077]). GAN models (models trained by a generative adversarial network) necessarily have a generator and a discriminator portion by definition, the generator generating new data that mimics real data, and the discriminator that takes in data generated by the generator and classifies the data as real or fake which is a binary outcome such as 1 or 0, valid or invalid, yes or no, etc..
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Ghosh et al, "Interactive Sketch & Fill: Multiclass Sketch-to-Image Translation", 25 Sept 2019, arXiv:1909.11081v2 (Year: 2019) teaches an interactive GAN-based sketch-to-image translation method that helps novice users create images of simple objects, starting with a user drawing a sketch of a desired object type.
Batra et al. (U.S. Pre-Grant Publication No. 2021/0064858) teaches generating a vector image from a raster image, where the raster image is, for instance, a photographed or scanned version of a hand-drawn sketch. While drawing a sketch, an artist may perform multiple strokes to draw a line, and the resultant raster image may have adjacent or partially overlapping salient and non-salient lines, where the salient lines are representative of the artist's intent, and the non-salient (or auxiliary) lines are formed due to the redundant strokes or otherwise as artefacts of the creation process. The raster image may also include other auxiliary features, such as blemishes, non-white background (e.g., reflecting the canvas on which the hand-sketch was made), and/or uneven lighting. In an example, the vector image is generated to include the salient lines, but not the non-salient lines or other auxiliary features. Thus, the generated vector image is a cleaner version of the raster image.Batra also teaches a raster-to-raster conversion module 104 in Fig. 6 that includes down-sampler, transformer, and up-sampler convolution layers with discussion of Fig. 6 beginning at Para. [0077].
Huang et al., "Multimodal Unsupervised Image-to-Image Translation", 14 Aug 2018, arXiv:1804.04732v2 (Year: 2018) teaches decomposing the image into a content code that is domain-invariant and a style code that captures domain-specific properties and recombining content code with a random style code sampled from the style pace of the target domain to translate an image to another domain.
Liu et al. (U.S. Pre-Grant Publication No. 2018/0247201) teaches image-to-image translation using variational autoencoders where the “image-to-image translation system is trained to translate sketch or hand drawn images into real images” (Para. [0033]).
Liao et al. (U.S. Pre-Grant Publication No. 2022/0044352) teaches image-to-image translation where “in the GAN structure 400, the generator of the GAN is further subdivided into an encoder portion and a decoder portion. For the input image, two encoders are used to model the content and style of the image, respectively, and extract the content-related feature representation and the appearance-related feature representation of the input image. Such separation of content and style enables application of different styles to the same content, thereby obtaining different outputs. Decoders are used to perform the reverse operations of the encoders. In such structure, the second learning network 220 for style transfer consists of the encoders in the domain X and the decoders decoding to the domain Y′” (Para. [0091]).
Chester et al. (U.S. Pre-Grant Publication No. 2017/0262479) teaches an image retrieval system using a convolutional neural network trained to identify how users draw semantic concepts and using an image search engine to search against images having a similar concept.
Wang et al. (U.S. Pre-Grant Publication No. 2012/0054177) teaches s ketch-based image search may include receiving a query curve as a sketch query input and identifying a first plurality of oriented points based on the query curve.
Mueller et al. (U.S. Patent No. 11,073,975) teaches systems, methods, and media for generating a user-created synthetic image, including receiving input from a user onto a search field, the input relating to a desired image of the user, the search field including a user interface for specifying components of the desired image for display to the user.
Zhang et al. (U.S. Pre-Grant Publication No. 2020/0073968) teaches improved sketch-based image retrieval techniques with a domain migration function configured to transform sketches into synthetic images, and the hashing function is configured to generate hash codes from synthetic images and authentic images in a manner that preserves semantic consistency across the sketch and image domains where the hash codes generated from the synthetic images can be used for accurately identifying and retrieving authentic images corresponding to sketch queries, or vice versa.
Odry et al. (U.S. Pre-Grant Publication No. 2019/0049540) teaches synthesizing protocol independent magnetic resonance images. A patient is scanned by a magnetic resonance imaging system to acquire magnetic resonance data. The magnetic resonance data is input to a machine learnt generator network trained to extract features from input magnetic resonance data and synthesize protocol independent images using the extracted features. The machine learnt generator network generates a protocol independent segmented magnetic resonance image from the input magnetic resonance data. The protocol independent magnetic resonance image is displayed.
Shen et al. (U.S. Patent No. 10,248,664) teaches improved sketch-based image retrieval (SBIR) techniques. The SBIR techniques utilize an architecture comprising three interconnected neural networks to enable zero-shot image recognition and retrieval based on free-hand sketches. Zero-shot learning may be implemented to retrieve one or more images corresponding to the sketches without prior training on all categories of the sketches. The neural network architecture may do so, at least in part, by training encoder hashing functions to mitigate heterogeneity of sketches and images, and by applying semantic knowledge that is learned during a limited training phase to unknown categories.
Spies et al. (U.S. Pre-Grant Publication No. 2021/0342496) teaches geometry-aware interactive design are described herein. In some examples, a computing device may extract geometric information from an input image of an object. The computing device may generate an output image by a generator network based on the extracted geometric information, a latent space vector of the input image and a sketch input.The reference further teaches “The generator network 106 may merge the sketch input 112 with the input image 108 based on the geometric information 110 and the latent space vector 111 to produce an output image 114. For example, the generator network 106 may explore the concatenated vector of the latent space and geometric information 110 using a histogram of oriented gradients (HOG) or pixel-wise constraints for shape and color constraints respectively. This can be done through a series of gradient descent operations that move in the direction of the constraints.” (Para. [0027]).
Son et al. (U.S. Pre-Grant Publication No. 2019/0188882) teaches a method and apparatus for processing an image interaction are provided. The apparatus extracts, using an encoder, an input feature from an input image, converts the input feature to a second feature based on an interaction for an application to the input image, and generates, using a decoder, a result image from the second feature.The reference further teaches using a discriminator to determine whether an image is “a real image” or “a fake image” (Paras. [0108]-[0109]).
Wei et al. (U.S. Pre-Grant Publication No. 2021/0165561) teaches a sketch interface to receive a sketch input from a user, an object model reservoir to store models of objects, a generator to generate additional models of objects, and a sample matching portion. The additional models generated by the generator are to be added to the object model reservoir. The sample matching portion is to select at least one matched object model from the reservoir to match to the sketch input from the user. The generator is to generate the additional models based on the matched object model.
Liu et al. (U.S. Pre-Grant Publication No. 2019/0279075) teaches a source image is processed using an encoder network to determine a content code representative of a visual aspect of the source object represented in the source image. A target class is determined, which can correspond to an entire population of objects of a particular type. The user may specify specific objects within the target class, or a sampling can be done to select objects within the target class to use for the translation. Style codes for the selected target objects are determined that are representative of the appearance of those target objects. The target style codes are provided with the source content code as input to a translation network, which can use the codes to infer a set of images including representations of the selected target objects having the visual aspect determined from the source image.The reference further teaches “An example adversarial discriminator D is trained by solving multiple adversarial classification tasks simultaneously. The discriminator in some embodiments is a patch GAN discriminator that can render an output spatial map for an input image, where each entry in the map indicates the score for the corresponding patch in the input image. Each of the tasks to be solved can be a binary classification task in some embodiments, determining whether an input image to D is a real image of a source class or a translation output coming from the generator. As there are a number of classes, the discriminator can be designed to produce a corresponding number of outputs. When updating D for a real image of a class, D can be penalized if a certain output is negative. For a translation output yielding a fake image, D can be penalized if a corresponding output is positive. D may not be penalized for not predicting negatives for images of other classes. When updating the generator G, G may only be penalized if the specified output of D is negative. The discriminator D can be designed in some embodiments based on a class-conditional discriminator that consists of several residual blocks followed by a global average pooling layer. The feature produced by the global average pooling layer is called the discriminator feature, from which classification scores can be produced using linear mappings.” (Para. [0046])
Fu et al. (U.S. Pre-Grant Publication No. 2019/0295302) teaches image generation through use of adversarial networks. An embodiment trains an image generator comprising (i) a generator implemented with a first neural network configured to generate a fake image based on a target segmentation, (ii) a discriminator implemented with a second neural network configured to distinguish a real image from a fake image and output a discrimination result as a function thereof and (iii) a segmentor implemented with a third neural network configured to generate a segmentation from the fake image. The training includes (i) operating the generator to output the fake image to the discriminator and the segmentor and (ii) iteratively operating the generator, discriminator, and segmentor during a training period, whereby the discriminator and generator train in an adversarial relationship with each other and the generator and segmentor train in a collaborative relationship with each other.The reference further teaches “An embodiment embeds an auxiliary multi-attribute classifier Dc which shares the weights with Dd in discriminator D except the output layer. Dc enables an embodiment of the SCGAN to generate attribute conditioned images. The auxiliary classifier Dc takes an image as input and classifies the image into independent probabilities of nc attribute labels. During training, the model Dc learns to classify input images into their attribute labels by optimizing the classification loss for real samples…where (x, c) are a real image with its ground-truth attribute label, Ac (⋅, ⋅) computes a multi-class binary cross-entropy loss”.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ROBERT F MAY whose telephone number is (571)272-3195. The examiner can normally be reached Monday-Friday 9:30am to 6pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Boris Gorney can be reached on 571-270-5626. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ROBERT F MAY/Examiner, Art Unit 2154 2/4/2026
/BORIS GORNEY/Supervisory Patent Examiner, Art Unit 2154