DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
The current application is examined under the PBA program.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1, 5-8, 9, 13, 14, and 18-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to a judicial exception (i.e. law of nature, natural phenomenon, or abstract idea) without significantly more.
Regarding claim 1, the claim is reproduced below with bracketed paragraph designators added for clarity and emphasis added to the claim language that recites an abstract idea:
A design system, comprising:
[(A)] one or more processors;
[(B)] a memory communicably coupled to the one or more processors and storing instructions that, when executed by the one or more processors, cause the one or more processors to:
[(C)] acquire design inputs including at least a text prompt and an active sketch;
[(D)] responsive to determining an activity type associated with the active sketch, adapt the design inputs to correspond with the activity type; and
[(E)] generate a guidance image according to the design inputs.
In determining whether a claim falls within an excluded category, the office is guided by the Court’s two-step framework, described in Mayo Collaborative Services v. Prometheus Laboratories, Inc., 566 U.S. 66 (2012), and Alice, 573 U.S. at 217–18 (citing Mayo, 566 U.S. at 75–77). In accordance with that framework, the Office first determines what concept the claim is “directed to.” See Alice, 573 U.S. at 219 (“On their face, the claims before us are drawn to the concept of intermediated settlement, i.e., the user of a third party to mitigate settlement risk.”); see also Bilski v. Kappos, 561 U.S. 593, 611 (2010) (“Claims 1 and 4 in petitioners’ application explain the basic concept of hedging, or protecting against risk.”)
Concepts determined to be abstract ideas, and thus patent ineligible, include certain methods of organizing human activity, such as fundamental economic practices (Alice, 573 U.S. at 219–20; Bilski, 561 U.S. at 611); mathematical formulas (Parker v. Flook, 437 U.S. 584, 594–95 (1978)); and mental processes (Gottschalk v. Benson, 409 U.S. 63, 67 (1972)). Concepts determined to patent eligible include physical and chemical processes, such as “molding rubber products” (Diamond v. Diehr, 450 U.S. 175, 191 (1981)); “tanning, dyeing, making water-proof cloth, vulcanizing India rubber, smelting ores” (id. At 182 n. 7 (quoting Corning v. Burden, 56 U.S. 252, 267–68 (1854))); and manufacturing flour (Benson, 409 U.S. at 69 (citing Cochrane v. Deener, 94 U.S. 780, 785 (1876))).
Step 1: The claimed subject matter falls within the four statutory categories of patentable subject matter identified by 35 U.S.C. § 101: process, machine, manufacture, or composition of matter. Accordingly, we turn to step 2A of the 2019 Guidance.
STEP 2A, PRONG 1: Under step 2A, prong 1, of the 2019 Guidance, we first look to whether the claim recites any judicial exceptions, including certain groupings of abstract ideas (i.e., mathematical concepts, certain methods of organizing human activity such as a fundamental economic practice, or mental processes). MPEP § 2106.04(a).
Limitation (D) states, “responsive to determining an activity type associated with the active sketch, adapt the design inputs to correspond with the activity type”, which is merely a mental process of looking at a sketch, making a mental judgement as to the activity and adjusting a description or terms for use, at most using a pen and paper as a tool, but also merely expressing the new terms etc. (See MPEP 2106(III) - a claim to “collecting information, analyzing it, and displaying certain results of the collection and analysis,” where the data analysis steps are recited at a high level of generality such that they could practically be performed in the human mind, Electric Power Group v. Alstom, S.A., 830 F.3d 1350, 1353-54, 119 USPQ2d 1739, 1741-42 (Fed. Cir. 2016)). Accordingly, the claim recites a judicial exception abstract
STEP 2A, PRONG 2: Under step 2A, prong 2, of the 2019 Guidance, we next analyze whether the claim recites additional elements that individually or in combination integrate the judicial exception into a practical application. 2019 Guidance, 84 Fed. Reg. at 53–55. The 2019 Guidance identifies considerations indicative of whether an additional element or combination of elements integrate the judicial exception into a practical application, such as an additional element reflecting an improvement in the functioning of a computer or an improvement to other technology or technical field. Id. at 55; MPEP § 2106.05(a).
Limitation (C) states “acquire design inputs including at least a text prompt and an active sketch” which is merely data gathering, which is insignificant pre-solution activity (iii. Adding insignificant extra-solution activity to the judicial exception, e.g., mere data gathering in conjunction with a law of nature or abstract idea such as a step of obtaining information about credit card transactions so that the information can be analyzed by an abstract mental process, as discussed in CyberSource v. Retail Decisions, Inc., 654 F.3d 1366, 1375, 99 USPQ2d 1690, 1694 (Fed. Cir. 2011) – see MPEP 2106.05).
Limitation (E) recites, “generate a guidance image according to the design inputs”, which is so generally recited that the image can be anything, including merely an output of a textual representation of the design inputs. Merely using a computer to present a result is insignificant post-solution activity (see e.g. MPEP 2106.05(g) - An example of post-solution activity is an element that is not integrated into the claim as a whole, e.g., a printer that is used to output a report of fraudulent transactions, which is recited in a claim to a computer programmed to analyze and manipulate information about credit card transactions in order to detect whether the transactions were fraudulent.)
Accordingly, the limitations are not significantly more than the abstract concept itself.
STEP 2B: Under step 2B of the 2019 Guidance, we next analyze whether the claim adds any specific limitations beyond the judicial exception that, either alone or as an ordered combination, amount to more than “well-understood, routine, conventional” activity in the field. 2019 Guidance, 84 Fed. Reg. at 56; MPEP § 2106.05(d).
The claim further discloses that the other limitations of the claim are performed by (A) one or more processors, and (B) a memory communicably coupled to the one or more processors and storing instructions that, when executed by the one or more processors, caused to operate. This is at most merely use of a generic computer to otherwise automate the recited operations, at most as an apply-it type automation using a computer merely as a tool. Without more than a general recitation of the conventional components, the limitations do not add significantly more than the abstract concepts themselves. 2019 Guidance, 84 Fed. Reg. at 52-55; MPEP § 2106.05(d). As such, the claim does not recite additional elements that, either individually or as an ordered combination , amount to significantly more than the judicial exception within the meaning of the 2019 Guidance. 2019 Guidance, 84 Fed. Reg. at 52-55; MPEP § 2106.05(d).
Regarding claim 5, the claim depends from claim 1 and further states, “wherein the instructions to determine the activity type include instructions to analyze the active sketch using an activity model that has learned to distinguish between different design phases of drawings.” Merely reciting a generic use of “an activity model” in the abstract, without any particular improvement recited as to the improvement to a technological function, is at most merely a recitation of a mathematical calculation abstract concept or abstract mental process (namely looking at gathered data, using some mental construct or “model” to analyze the data and coming up with a resulting determination). As such, the claim is directed to ineligible subject matter under 35 U.S.C. 101 for substantially the same reasons as claim 1 set forth above.
Regarding claim 6, the claim depends from claim 1 and further states, “wherein the instructions to acquire the design inputs include instructions to monitor the active sketch for a change and responsively re-generating the guidance image after adapting the design inputs” which at most is merely an iterative loop of the abstract mental processing concept of gathering data, analyzing it by observing it, and providing new data that is presented in the abstract on a display, as discussed in rejection of claim 1 above (also see MPEP 2106.05(h) – “Limiting the abstract idea of collecting information, analyzing it, and displaying certain results of the collection and analysis to data related to the electric power grid, because limiting application of the abstract idea to power-grid monitoring is simply an attempt to limit the use of the abstract idea to a particular technological environment, Electric Power Group, LLC v. Alstom S.A., 830 F.3d 1350, 1354, 119 USPQ2d 1739, 1742 (Fed. Cir. 2016).) As such, the claim is rejected for substantially the same reason as claim 1 set forth above.
Regarding claim 7, the claim depends from claim 1 and further states, “wherein the instructions further include instructions to: provide guidance image by iteratively rendering the guidance image on a display as the active sketch is updated and the guidance image is re-generated”, which at most is merely an iterative loop of the abstract mental processing concept of gathering data, analyzing it by observing it, and providing new data that is presented in the abstract on a display, as discussed in rejection of claim 1 above (also see MPEP 2106.05(h) – “Limiting the abstract idea of collecting information, analyzing it, and displaying certain results of the collection and analysis to data related to the electric power grid, because limiting application of the abstract idea to power-grid monitoring is simply an attempt to limit the use of the abstract idea to a particular technological environment, Electric Power Group, LLC v. Alstom S.A., 830 F.3d 1350, 1354, 119 USPQ2d 1739, 1742 (Fed. Cir. 2016).) As such, the claim is rejected for substantially the same reason as claim 1 set forth above.
Regarding claim 8, the claim depends from claim 1 and further states, “wherein control parameters indicate visual attributes and text- based attributes, the visual attributes including a hue, a color contrast, and a level-of-detail, the text-based attributes including a semantic similarity, and an analogical similarity.” There is no use of the data that is recited by the claim, and as such this is merely a characterization of data in the abstract. As such, the claim is at most merely the abstract concept provided in claim 1 and rejected based on the same rationale as claim 1 set forth above.
Regarding claims 9 and 13, the claims are directed to a non-transitory computer readable medium that is substantially the same as the memory of claims 1 and 5, respectively, and are rejected as being directed to ineligible subject matter under 35 U.S.C. 101 for substantially the same reasons as claims 1 and 5, respectively, set forth above.
Regarding claims 14 and 18-20, the claims are merely method implementations of the system claims 1 and 5-7, respectively, and as such claims 14 and 18-20 as directed to ineligible subject matter under 35 U.S.C. 101 for substantially the same reasons as claims 1 and 5-7, respectively, set forth above.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 3, 8, and 11 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 3 recites “wherein the instructions to adapt the design inputs include instructions to change control parameters to alter generating the guidance image by an image model by correlating the guidance image with the activity type that indicates one of: exploratory, sketching, and finishing.” Without further guidance from the claim or specification, the scope of the meaning of the terms “exploratory, sketching, and finishing” as activity types are unclear and render the claim indefinite as a measure of correlation between the guidance image and the activity type. This is further evidenced by applicant’s specification, filed 1/24/2025, which states in paragraph 25, “[k]nowing the phase of the design process allows the system to adapt the control parameters to guide image generation so that when the system generates the guidance image, the form of the guidance image fits with a current thought process of the user relative to the design stage.” This indicates that there determination is not based on any objective standard of measurement that one of ordinary skill in the art is apprised of as the intended metes and bounds of the claim. Instead, this is a subjective standard that is based on an individual’s own subjective definition of the terms. In addition, the claim language is unclear as to whether the “correlation” is related to the changing of the control parameters or the generating the guidance image. As such, the claim is indefinite. As such, the claim is rendered indefinite and fails to meet the requirements of 35 U.S.C. 112(b).
Claims 11 and 16 contain substantially the same language as claim 3 and are therefore indefinite for the same reasons as set forth above for claim 3.
Claim 8 recites the limitation " wherein control parameters indicate visual attributes and text- based attributes, the visual attributes including a hue, a color contrast, and a level-of-detail, the text-based attributes including a semantic similarity, and an analogical similarity." The claim does nothing with the recited “control parameters” and as such it is indefinite as to the purpose of the recited limitation (i.e. does the system even include the control parameters, or is it the system of claim 1, and also separate “control parameters” and if incorporated into the system itself, in what manner?) As such, the claim is rendered indefinite.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-2, 6, 8-10, 14-15 and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over:
Voynov et al. (Voynov, Andrey, et al., “Sketch-Guided Text-to-Image Diffusion Models”, Computer Vision and Pattern Recognition, arXiv:2211.13752 [cs.CV], Nov. 2022) in view of
Wei et. al. (US 2021/0165561 A1).
REGARDING CLAIM 1, Voynov discloses:
A design system, comprising: one or more processors; a memory communicably coupled to the one or more processors and storing instructions that, when executed by the one or more processors, cause the one or more processors to: (Examiner submits that the Sketch-Guided Text-to-Image Diffusion Models of Voynov requires a computing system comprising a processor and memory, further evidenced by Voynov, p. 3, ¶1 stating training of model performed on GPU)
acquire design inputs including at least a text prompt and an active sketch (Voynov, p. 3, Section 3.2. Sketch-Guided Text-to-Image Synthesis: “Given a sketch image e and a caption c, our goal is to generate a corresponding highly detailed image that follows the sketch outline.”);
generate a guidance image according to the design inputs (Voynov, p. 2, Section 3, ¶2: “The key idea of our method is to guide the inference process of a pretrained text-to-image diffusion model with an edge predictor that operates on the internal activations of the core network of the diffusion model, encouraging the edge of the synthesized image to follow a reference sketch”; p. 4, Fig. 3: “Given an encoded noisy image zt, our method extracts its deep features during the inference process of a text-to-image diffusion model, conditioned on a caption c.”; p. 4, right column, ¶1: “Once being synthesized with the guidance from the objective L, the model produces a natural image aligned with the desired sketch.”)
Voynov does not explicitly disclose responsive to determining an activity type associated with the active sketch, adapt the design inputs to correspond with the activity type.
Wei however, discloses:
responsive to determining an activity type associated with the active sketch, adapt the design inputs to correspond with the activity type (Wei, ¶10: sketch input from user using tablet, etc.; ¶11: object model reservoir stores electronic models of various objects in separate libraries corresponding to categories; ¶12: identify of select an object model from the object model reservoir as a match for sketch input, by extracting features from sketch input; ¶13: generator 140 to generate models of objects in addition to those already in the object model reservoir 120, generating the additional models based on an object selected by the sample matching portion 130 as a match for the sketch input provided by the user – i.e. Wei teaches searching for an object within a categorized store of models and using the resulting categorized object as input to a machine learning model to generate additional images – Fig. 2 and ¶15 discloses the generation of a model based on sketch input; ¶¶17-18 discloses the AAI agent generating vector representation of latent space around the object model match, where the latent vector is determined as: z−Alpha*z+(1−Alpha)*t, where Alpha is an interpolation rate, z is the current latent vector, and t is one of the anchor latent vector for a category of object (e.g., aircraft))
Both Voynov and Wei are directed to machine learning systems and techniques for utilizing a user provided sketch drawing as input to generate a related output image. It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to modify the system and technique for utilizing a user-provided sketch and text prompt to a generative image model / machine learning model to produce a related image as provided by Voynov, by incorporating the technique to identify a category of activity of the sketch image to refine the input for an image generation model as provided by Wei, using known electronic interfacing and programming techniques. The modification results in an improved sketch-to-image generation by better instructing the machine learning model to particular categories related to the user’s input to avoid hallucinations and less precise image generation results, and ensure output images better match user expectations.
REGARDING CLAIM 2, Voynov further discloses:
wherein the design inputs include the text prompt, the active sketch, and control parameters, (Voynov, p. 3, Section 3.2. Sketch-Guided Text-to-Image Synthesis: “Given a sketch image e and a caption c, our goal is to generate a corresponding highly detailed image that follows the sketch outline.”;
wherein the control parameters define a variability of attributes for controlling an image model to generate the guidance image, (Voynov, p. 3, Section 3.1, ¶¶2-4: training corpus D formed by triplets (x, e, c) of an image, edge map, and corresponding text caption, and converting edge map into 3 channel image by replicating its intensity channel, where the input tensor is the encoded image and the additive Gaussian nose, zt based on blending scalars dictated by noise scheduling of the diffusion model, and once optimized with the objective L, the model P constitutes a per-spatial location differential predictor of encoded edges for an encoded image with noise level t; also p. 6, Section 4.3 Ablations and Parameter Tuning, ¶¶1-2 discloses model using parameters such as edge guidance scale B and guidance step S value; Also p. 7, Fig. 8 – control realism vs. edge-fidelity using edge-guidance scale B)
wherein the active sketch is a free-form line drawing provided via a drawing interface, (Voynov, p. 2, ¶1: “our method can accept free-hand sketches inputs, as in Figure 1, and generate diverse results that correspond to the text-prompt and follow the spatial layout of the sketch’; ¶2: free-hand sketches Fig. 6 Input Sketch – note that receiving a “free-hand sketch” as input as taught by Voynov inherently requires a “drawing interface”, or shared boundary where humans and software/computers communicate, as the claim is exceptionally broad and not limited to any particular interface, but instead is merely some communication of a free-form line drawing – as disclosed by Voynov - input to the machine) and
wherein the text prompt is semantic description of an object to be depicted in the guidance image (Voynov, p. 5, Fig. 4: “Samples of our method applied to input sketches with different prompts (depicted above the images)”, e.g. “oil painting of a wooden house on a hill, blue sky”)
REGARDING CLAIM 6, Voynov modified by Wei further discloses:
wherein the instructions to acquire the design inputs include instructions to monitor the active sketch for a change and responsively re-generating the guidance image after adapting the design inputs (Wei, ¶14: Further, the sample matching portion 130 may update the matching based on updated sketch input. For example, a user may first sketch an airplane including the fuselage and wings only. The sample matching portion 130 may perform a comparison and identify a best match from the object model reservoir 120. When the user adds jet engines or propellers to the sketch, the sample matching portion 130 may update the match.)
Both Voynov and Wei are directed to machine learning systems and techniques for utilizing a user provided sketch drawing as input to generate a related output image. It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to modify the system and technique for utilizing a user-provided sketch and text prompt to a generative image model / machine learning model to produce a related image as provided by Voynov, by incorporating the technique to identify a category of activity of the sketch image to refine the input for an image generation model and allowing for iterative updating of the input guidance as provided by Wei, using known electronic interfacing and programming techniques. The modification results in an improved sketch-to-image generation by better instructing the machine learning model to particular categories related to the user’s input to avoid hallucinations and less precise image generation results, and ensure output images better match user expectations by allowing for additional tailorizing of the user’s input, i.e. allowing more control by a user.
REGARDING CLAIM 8, Voynov further discloses:
wherein control parameters indicate visual attributes and text-based attributes, the visual attributes including a hue, a color contrast, and a level-of-detail, the text-based attributes including a semantic similarity, and an analogical similarity (Voynov, p. 1, I. Introduction, ¶1: semantic guidance provided by the text-prompt; p. 2, section 2.2, p. 6, Section 4.3 Alations and Parameters Tuning, ¶1: edge scale B value, where for small values, more realistic image with details – i.e. level of detail; ¶2: Figure 9 shows that such a setting yields the same shape, but with variation in colors and textures, such that the stroke style affects the inner synthesis process only, which accumulates into varying colors of the output image – i.e. color which incorporates hue and contrast; Applicant’s specification states that “analogical similarity relates, in general, to the descriptive form of the text prompt (e.g. protective, sleek, etc.)”; Voynov, discloses use of text description for generating the image – see Fig. 4(a) captions including descriptive nature of caption prompts directing the look of the resulting image as opposed to the semantic similarity of the object itself, e.g. oil painting vs. photograph, etc.)
REGARDING CLAIMS 9-10, the claims are directed to a non-transitory computer readable medium that is substantially the same as the memory of claims 1-2, respectively, and as such the claims are rejected based on the same rationale as claims 1-2, respectively, set forth above.
REGARDING CLAIMS 14-15 and 19, the system of claims 1-2 and 6 perform the same method as claims 14-15 and 19, respectively, and as such the claims are rejected based on the same rationale as claims 1-2 and 6, respectively, set forth above.
Claim(s) 3, 5, 11, 13, 16 and 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over:
Voynov et al. (Voynov, Andrey, et al., “Sketch-Guided Text-to-Image Diffusion Models”, Computer Vision and Pattern Recognition, arXiv:2211.13752 [cs.CV], Nov. 2022) in view of
Wei et. al. (US 2021/0165561 A1) and in further view of
Wang et al. (CN 112182707 A)
REGARDING CLAIM 3, the limitations included from claim 1 are rejected based on the same rationale as claim 1 set forth above. Further regarding claim 3, Wang discloses:
wherein the instructions to adapt the design inputs include instructions to change control parameters to alter generating the guidance image by an image model by correlating the guidance image with the activity type that indicates one of: exploratory, sketching, and finishing (Examiner notes that the claim is indefinite – see rejection under 35 U.S.C. 112(b) above, but for sake of compact prosecution, the claim is interpreted as very broadly reciting any change control and any correlation activity type, i.e. exploratory, sketching, and finishing can be overlapping, the same, or any subjective category of determination by a machine model;
Wang, Abstract: scheme determination in the design stage is achieved through the sketch design, scheme design, scheme model comparison and selection and scheme data comparison and selection […] the established design model is reversely input into a comparison display screen, and comparison of the result with the selected scheme is performed, where the scheme design and comparison and selection process is visually displayed, scheme design data are directly used for deepening design, detail information is determined through the part design module in deepening design, the product design accuracy is ensured, the design quality is improved, and the design efficiency is improved; ¶48: In the sketch design stage, the design functions contained in the sketch are determined, and the planning and design within the area are carried out in the independent design functions to obtain a set of sketch design elements; ¶72: Step 17: Shape Plan Design and Step 17: Shape Elevation Design. The shape plan design information and shape elevation design information determine the data for the product's appearance and outline.
¶83: In the sketch design stage, the design functions contained in the sketch are determined, and the planning and design within the area are carried out in the independent design functions to obtain a set of sketch design elements; ¶88: The preset schemes in the scheme design set are extracted, and the model information of the preset schemes is compared and selected. The preset schemes include the main design model and the auxiliary design model; ¶89: The design data corresponding to the model information in the preset scheme is displayed intuitively; ¶109: The comparison display screen provides an intuitive display of the sketch design in step i, the scheme design in step ii, the scheme model comparison in step iii, and the scheme data comparison in step iv.)
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to modify the system and technique for utilizing a user-provided sketch and text prompt to a generative image model / machine learning model to produce a related image as provided by Voynov, using the technique to identify a category of activity of the sketch image to refine the input for an image generation model and allowing for iterative updating of the input guidance as provided by Wei, by further utilizing the sketch design scheme determination of the design stage of a sketch design as provided by Wang, using known electronic interfacing and programming techniques. The modification results in an improved sketch-to-image generation utilizing improved machine learning of the underlying data for improved guidance to a user (Wang, Abstract – improved design quality and efficiency).
REGARDING CLAIM 11, the claim is directed to a non-transitory computer readable medium that is substantially the same as the memory of claim 3, respectively, and as such the claim is rejected based on the same rationale as claim 3, set forth above.
REGARDING CLAIM 5, the limitations included from claim 1 are rejected based on the same rationale as claim 1 set forth above. Further regarding claim 5, Wang discloses:
wherein the instructions to determine the activity type include instructions to analyze the active sketch using an activity model that has learned to distinguish between different design phases of drawings. (Wang, Abstract: scheme determination in the design stage is achieved through the sketch design, scheme design, scheme model comparison and selection and scheme data comparison and selection […] the established design model is reversely input into a comparison display screen, and comparison of the result with the selected scheme is performed, where the scheme design and comparison and selection process is visually displayed, scheme design data are directly used for deepening design, detail information is determined through the part design module in deepening design, the product design accuracy is ensured, the design quality is improved, and the design efficiency is improved; ¶48: In the sketch design stage, the design functions contained in the sketch are determined, and the planning and design within the area are carried out in the independent design functions to obtain a set of sketch design elements; ¶72: Step 17: Shape Plan Design and Step 17: Shape Elevation Design. The shape plan design information and shape elevation design information determine the data for the product's appearance and outline.
¶83: In the sketch design stage, the design functions contained in the sketch are determined, and the planning and design within the area are carried out in the independent design functions to obtain a set of sketch design elements; ¶88: The preset schemes in the scheme design set are extracted, and the model information of the preset schemes is compared and selected. The preset schemes include the main design model and the auxiliary design model; ¶89: The design data corresponding to the model information in the preset scheme is displayed intuitively; ¶109: The comparison display screen provides an intuitive display of the sketch design in step i, the scheme design in step ii, the scheme model comparison in step iii, and the scheme data comparison in step iv.)
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to modify the system and technique for utilizing a user-provided sketch and text prompt to a generative image model / machine learning model to produce a related image as provided by Voynov, using the technique to identify a category of activity of the sketch image to refine the input for an image generation model and allowing for iterative updating of the input guidance as provided by Wei, by further utilizing the sketch design scheme determination of the design stage of a sketch design as provided by Wang, using known electronic interfacing and programming techniques. The modification results in an improved sketch-to-image generation utilizing improved machine learning of the underlying data for improved guidance to a user (Wang, Abstract – improved design quality and efficiency).
REGARDING CLAIM 13, the claim is directed to a non-transitory computer readable medium that is substantially the same as the memory of claim 5, respectively, and as such the claim is rejected based on the same rationale as claim 5, set forth above.
REGARDING CLAIM 18, the system of claim 5 performs the same method as claim 18 and as such the claim is rejected based on the same rationale as claim 5, set forth above.
REGARDING CLAIM 16, the limitations included from claim 14 are rejected based on the same rationale as claim 14 set forth above. Further regarding claim 16, Voynov further discloses:
wherein the control parameters indicate visual attributes and text-based attributes, the visual attributes including a hue, a color contrast, and a level-of-detail, the text-based attributes including a semantic similarity, and an analogical similarity. (Voynov, p. 1, I. Introduction, ¶1: semantic guidance provided by the text-prompt; p. 2, section 2.2, p. 6, Section 4.3 Alations and Parameters Tuning, ¶1: edge scale B value, where for small values, more realistic image with details – i.e. level of detail; ¶2: Figure 9 shows that such a setting yields the same shape, but with variation in colors and textures, such that the stroke style affects the inner synthesis process only, which accumulates into varying colors of the output image – i.e. color which incorporates hue and contrast; Applicant’s specification states that “analogical similarity relates, in general, to the descriptive form of the text prompt (e.g. protective, sleek, etc.)”; Voynov, discloses use of text description for generating the image – see Fig. 4(a) captions including descriptive nature of caption prompts directing the look of the resulting image as opposed to the semantic similarity of the object itself, e.g. oil painting vs. photograph, etc.)
Wang discloses:
wherein adapting the design inputs includes changing control parameters to alter generating the guidance image by an image model, including correlating the guidance image with the activity type that indicates one of: exploratory, sketching, and finishing, and (Examiner notes that the claim is indefinite – see rejection under 35 U.S.C. 112(b) above, but for sake of compact prosecution, the claim is interpreted as very broadly reciting any change control and any correlation activity type, i.e. exploratory, sketching, and finishing can be overlapping, the same, or any subjective category of determination by a machine model;
Wang, Abstract: scheme determination in the design stage is achieved through the sketch design, scheme design, scheme model comparison and selection and scheme data comparison and selection […] the established design model is reversely input into a comparison display screen, and comparison of the result with the selected scheme is performed, where the scheme design and comparison and selection process is visually displayed, scheme design data are directly used for deepening design, detail information is determined through the part design module in deepening design, the product design accuracy is ensured, the design quality is improved, and the design efficiency is improved; ¶48: In the sketch design stage, the design functions contained in the sketch are determined, and the planning and design within the area are carried out in the independent design functions to obtain a set of sketch design elements; ¶72: Step 17: Shape Plan Design and Step 17: Shape Elevation Design. The shape plan design information and shape elevation design information determine the data for the product's appearance and outline.
¶83: In the sketch design stage, the design functions contained in the sketch are determined, and the planning and design within the area are carried out in the independent design functions to obtain a set of sketch design elements; ¶88: The preset schemes in the scheme design set are extracted, and the model information of the preset schemes is compared and selected. The preset schemes include the main design model and the auxiliary design model; ¶89: The design data corresponding to the model information in the preset scheme is displayed intuitively; ¶109: The comparison display screen provides an intuitive display of the sketch design in step i, the scheme design in step ii, the scheme model comparison in step iii, and the scheme data comparison in step iv.)
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to modify the system and technique for utilizing a user-provided sketch and text prompt to a generative image model / machine learning model to produce a related image as provided by Voynov, using the technique to identify a category of activity of the sketch image to refine the input for an image generation model and allowing for iterative updating of the input guidance as provided by Wei, by further utilizing the sketch design scheme determination of the design stage of a sketch design as provided by Wang, using known electronic interfacing and programming techniques. The modification results in an improved sketch-to-image generation utilizing improved machine learning of the underlying data for improved guidance to a user (Wang, Abstract – improved design quality and efficiency).
Claim(s) 4, 12 and 17 is/are rejected under 35 U.S.C. 103 as being unpatentable over:
Voynov et al. (Voynov, Andrey, et al., “Sketch-Guided Text-to-Image Diffusion Models”, Computer Vision and Pattern Recognition, arXiv:2211.13752 [cs.CV], Nov. 2022) in view of
Wei et. al. (US 2021/0165561 A1) and in further view of
Wu et al. (Wu, et al. “DeepPortraitDrawing: Generating Human Body Images from Freehand Sketches,” Computers & Graphics, 2023, 10 pages – provided by Applicant with IDS filed 1/24/2025)
REGARDING CLAIM 4, the limitations included from claim 1 are rejected based on the same rationale as claim 1 set forth above. Further regarding claim 4, Wu discloses:
wherein the instructions to generate the guidance image include instructions to convert the active sketch from a sketch drawing to a line drawing using a conversion model and inputting the line drawing into an image model (Wu, p. 3, “A. Geometry refinement model” discloses refining freehand sketch; p. 4, Fig. 2: “Firstly, individual body parts of an input sketch are projected onto the underlying part-level manifolds and decoded into a geometrically refined sketch map and a parsing map, based on an auto-encoder architecture”; p. 4, ¶1: Next, the sketch decoder Dc S and the mask decoder Dc M for part c process the projected vector ˙vc, resulting in a refined part sketch ˙Sc and a part mask ˙ Mc, respectively. Finally, all projected part sketches { ˙Sc} and masks { ˙ Mc} are combined together to recover the global body shape, resulting in a geometry-refined sketch map ˙S and a human parsing map)
Voynov, Wei and Wu are directed to machine learning systems and techniques for utilizing a user provided sketch drawing as input to generate a related output image. It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to modify the system and technique for utilizing a user-provided sketch and text prompt to a generative image model / machine learning model to produce a related image as provided by Voynov, using the technique to identify a category of activity of the sketch image to refine the input for an image generation model and allowing for iterative updating of the input guidance as provided by Wei, by further refining the input sketch to obtain an improved drawing for use as a generative model prompt provided by Wu, using known electronic interfacing and programming techniques. The modification results in an improved sketch-to-image generation by better instructing the machine learning model using a refined image that allows for improved understanding by the system (see p. 3 of Wu, A. Geometry refinement model Section, ¶1, stating: “This has two advantages. First, locally pushing the input sketch towards the training edge maps, and second reducing the geometric errors in the input sketch.”)
REGARDING CLAIM 12, the claim is directed to a non-transitory computer readable medium that is substantially the same as the memory of claim 4, respectively, and as such the claim is rejected based on the same rationale as claim 4, set forth above.
REGARDING CLAIM 17, the system of claim 4 performs the same method as claim 17 and as such the claim is rejected based on the same rationale as claim 4, set forth above.
Claim(s) 7 and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over:
Voynov et al. (Voynov, Andrey, et al., “Sketch-Guided Text-to-Image Diffusion Models”, Computer Vision and Pattern Recognition, arXiv:2211.13752 [cs.CV], Nov. 2022) in view of
Wei et. al. (US 2021/0165561 A1) and in further view of
Yun et al. (US 2023/0215093 A1).
REGARDING CLAIM 7, the limitations included from claim 1 are rejected based on the same rationale as claim 1 set forth above. Further regarding claim 7, Yun discloses:
wherein the instructions further include instructions to: provide guidance image by iteratively rendering the guidance image on a display as the active sketch is updated and the guidance image is re-generated (Yun, ¶10: the receiving the 2D sketch data may include receiving a plurality of strokes from a user, and the method may include receiving an additional stroke from the user, inputting the additional stroke into the 3D model generation model to generate an updated 3D model of the target object, and displaying the updated 3D model on the display; further see ¶¶82-84)
It would have been obvious to one of ordinary skill in the art, before the effective filing date of the claimed invention and with a reasonable expectation of success, to modify the system and technique for utilizing a user-provided sketch and text prompt to a generative image model / machine learning model to produce a related image as provided by Voynov, using the technique to identify a category of activity of the sketch image to refine the input for an image generation model and allowing for iterative updating of the input guidance as provided by Wei, by further allowing updated sketch input form a user as provided by Yun, using known electronic interfacing and programming techniques. The modification results in an improved sketch-to-image generation by allowing user edits that results in a more flexible system, and allows for easier usability that better accounts for user preferences and choices.
REGARDING CLAIM 20, the system of claim 7 performs the same method as claim 20 and as such the claim is rejected based on the same rationale as claim 7, set forth above.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to WILLIAM A BEUTEL whose telephone number is (571)272-3132. The examiner can normally be reached Monday-Friday 9:00 AM - 5:00 PM (EST).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, DANIEL HAJNIK can be reached at 571-272-7642. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/WILLIAM A BEUTEL/Primary Examiner, Art Unit 2616
Prosecution Insights