TEXT-TO-MASK AND MASK-TO-IMAGE SYNTHESIS

§101 §103 §112

DETAILED ACTION Notice of Pre-AIA or AIA Status The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA . Compact Prosecution With respect to Claim Interpretation, the Examiner has provided some notes regarding “[BRI on the record]” throughout the Office Action, so that the record is clear about the scope of the claimed invention, and the record is also clear about the basis for the Examiner’s analyses. A clear record of the claim interpretation could expedite the examination by creating the condition to allow the examination to focus on Applicant’s inventive concept and its comparison with related prior art. If there are disagreements, Applicant may present an alternative interpretation based on MPEP 2111. The Examiner will adopt Applicant’s interpretation on the record, if Applicant’s interpretation is reasonable and/or arguments are persuasive. Applicant may amend claims relying on the Examiner’s claim interpretation provided on the record. Claim Rejections - 35 USC § 101 The rejections of Claims 1-2, 4-7, and 14-20 under 35 USC § 101 are withdrawn in view of Applicant’s arguments and amendments. Response to Arguments Some of Applicant’s arguments are moot in view of the Examiner’s new ground of rejections. However, the Examiner would like to address the following arguments/comments. (1) One premise of Applicant’s arguments is the following: PNG media_image1.png 178 644 media_image1.png Greyscale Remarks p. 25. The Examiner finds the premise invalid. Here, the claim language requires, as Applicant stated, “the second generative machine learning model generates a synthesized image based on the text prompt.” However, “based on the text prompt” does not mean that the text prompt is the input to the second generative machine learning model. For example, if an input X to the second generative machine learning model is based on the text prompt, an output of the second generative machine learning model is also based on text prompt, because the text prompt influences the output. In short, Applicant’s arguments are not based on the scope of the claim. (2) Applicant states: PNG media_image2.png 66 640 media_image2.png Greyscale Remarks p. 27. The Examiner disagrees. The term “color map” describes the appearance of a map, which Wu explicitly teaches. If the functionality of Wu’s “color map” is different from the claimed invention, Applicant has the option to specify the functionalities that differentiate the claimed “color map” from Wu’s. (3) Applicant states: PNG media_image3.png 290 648 media_image3.png Greyscale Remarks pp. 27-28. It is unclear about the relevance of the argument, which does not appear to be based on the claim language. (4) Applicant states: PNG media_image4.png 250 644 media_image4.png Greyscale Remarks p. 28. As the Examiner explained under (1), Applicant’s premise of the argument is invalid. Here, “based on the text prompt” does not require the text prompt as input to the second generative machine learning model. Claim Rejections - 35 USC § 112 The following is a quotation of the first paragraph of 35 U.S.C. 112(a): (a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention. The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112: The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention. Claim 8-9 and 12-13 are rejected under 35 U.S.C. 112(a) or 35 U.S.C. 112 (pre-AIA ), first paragraph, as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention. Based on Applicant’s explanations on pages 31-33 in the Remarks, it appears that Fig. 2 is the primary source of support for Applicant’s amendments. Claimed “segmentation model” [Wingdings font/0xE0] segmentation model in Fig. 2 220 Claimed “generative machine learning model” [Wingdings font/0xE0] data generation model in Fig. 2 210, 215 Claimed “an image” [Wingdings font/0xE0] synthetic training samples in Fig. 2 215, 220 = output “New Masks New Images” in Fig. 11 Claimed “segmentation mask” [Wingdings font/0xE0] input “Masks Images” in Fig. 11 However, the specification appears to be silent as to the segmentation model in Fig. 2 220 is trained based on both (synthetic training samples in Fig. 2 215, 220 = output “New Masks New Images” in Fig. 11) and (input “Masks Images” in Fig. 11). Claim 9 and 12-13 depend on Claim 8 and inherit the deficiency and are therefore rejected. Claim Rejections - 35 USC § 103 In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action: A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made. Claims 1, 5-7, 14, 18, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over Wu et al. (“DiffuMask: Synthesizing Images with Pixel-level Annotations for Semantic Segmentation Using Diffusion Models”) in view of Zheng et al. (CN 111507210 A). Regarding Claim 1, Wu teaches A method comprising: obtaining a text prompt describing an object within a scene ( PNG media_image5.png 686 1422 media_image5.png Greyscale Wu provides an example for a text prompt in fig. 4: “Photo of a [sub-class] car in the street,” which describes an object, mapped to “car,” within a scene, mapped to “street” scene.); generating, using a first generative machine learning model (a model comprising PNG media_image6.png 267 316 media_image6.png Greyscale ) and based on the text prompt (fig. 4: “Photo of a [sub-class] car in the street,”), a color map ( PNG media_image7.png 72 74 media_image7.png Greyscale or) corresponding to the scene (street scene with a car), wherein the color map comprises a plurality of possible colors (yellow, purple, blue, green, etc. as shown in the figure) corresponding to each pixel and indicates a region corresponding to the object (car) from the text prompt ( [BRI on the record] With respect to “color map,” the Examiner is interpretation the term to mean: a map/image that uses different color to represent respect object(s). The mapping is consistent with the specification, which states, “a color map that indicates an image region occupied by the object with a color corresponding to the object.” Spec. ¶ 36. [Mapping Analysis] PNG media_image8.png 128 74 media_image8.png Greyscale , where the color yellow is used to represent a car, which is based on the text prompt “Photo of a [sub-class] car in the street.”); generating an image segmentation mask ( PNG media_image9.png 54 38 media_image9.png Greyscale of PNG media_image8.png 128 74 media_image8.png Greyscale or PNG media_image10.png 66 40 media_image10.png Greyscale ) based on the color map ( PNG media_image8.png 128 74 media_image8.png Greyscale ) ( [BRI on the record] With respect to “segmentation mask,” the Examiner is reading the limitation to mean: a mask used for segmentation. [Mapping Analysis] “In this paper, we show that it is possible to automatically obtain accurate semantic masks of synthetic images generated by the Off-the-shelf Stable Diffusion model, which uses only text-image pairs during training.” Wu abstract. Wu Fig. 4 shows the generation process and dependence. The semantic masks could be used for segmentation, because the title recites “DiffuMask: Synthesizing Images with Pixel-level Annotations for Semantic Segmentation Using Diffusion Models”), wherein the image segmentation mask ( PNG media_image9.png 54 38 media_image9.png Greyscale of PNG media_image8.png 128 74 media_image8.png Greyscale or PNG media_image10.png 66 40 media_image10.png Greyscale ) comprises a plurality of regions (regions of a background and a car) corresponding to a plurality of image elements (a background and a car) in the scene including the region corresponding to the object (car) from the text prompt (fig. 4: “Photo of a [sub-class] car in the street”) (Wu Fig. 4); and generating, using a second generative machine learning model ( PNG media_image11.png 356 430 media_image11.png Greyscale ) different from the first generative machine learning model (the Examiner mapped to two different models with different components), a synthesized image ( PNG media_image12.png 70 160 media_image12.png Greyscale ) based on the text prompt (e.g., a text prompt in fig. 4: “Photo of a [sub-class] car in the street,”) and the image segmentation mask ( PNG media_image9.png 54 38 media_image9.png Greyscale of PNG media_image8.png 128 74 media_image8.png Greyscale or PNG media_image10.png 66 40 media_image10.png Greyscale ) ( Although the text prompt is not the direct input the second generative model the Examiner mapped, the output from the second generative model is based on the text prompt, because input(s) to the second generative model are based on the text prompt.). The Examiner’s position is that the region of the “street-based background” corresponds to a region of a mask. The car is affirmatively identified and semantically segmented. The remaining portion of the image corresponds to the “street-based background” and is lumped together. However, Applicant may argue that the remaining portion of the image should not be treated as a region of the image segmentation mask, the embodiment as taught by Wu fig. 4 does not explicitly teach the image segmentation mask comprises a plurality of regions corresponding to a plurality of image elements. Further, Wu states, “In our experiment, only a single object for an image is considered.” Wu 4.1. For the purposes of compact prosecution, the Examiner conducts obviousness analysis. In a possible different embodiment, Wu teaches the image segmentation mask comprises a plurality of regions corresponding to a plurality of image elements ( Wu teaches segmenting multiple objects from multiple classifications (person and car): PNG media_image13.png 164 672 media_image13.png Greyscale ). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Wu’s teaching of segmenting multiple objects with Wu’s fig. 4 algorithm. One of ordinary skill in the art would be motivated to process/analyze complex images that reflect real situations. However, Wu does not explicitly disclose; however, Zheng teaches wherein the color map’s possible colors for reach pixel are implemented by a plurality of channels corresponding to each pixel (“… outputting to obtain the attention map corresponding to N sample images. each pixel point in the sample image corresponding to three channels (i.e., each pixel point corresponding to three characteristic values, respectively is R, G and B), each pixel point in the attention map corresponding to a channel (i.e., the pixel point is the probability value of the pixel point corresponding to the traffic signal lamp), . . ..” Zheng p. 10.). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Zheng’s pixel channels with Wu. One of ordinary skill in the art would be motivated to embed more information in the map and/to make the map to be visually interesting and/or distinguishing. Regarding Claim 5, Wu further teaches The method of claim 1, further comprising: generating a plurality of image segmentation masks ( PNG media_image14.png 80 168 media_image14.png Greyscale ) based on the color map ( PNG media_image8.png 128 74 media_image8.png Greyscale ) (Wu fig. 4; “the final images and masks” contains three images and masks based on the mapped color map.). Regarding Claim 6, Wu further teaches The method of claim 1, further comprising: encoding the text prompt to obtain text features representing the object, wherein the color map is generated based on the text features ( Wu Fig. 4: PNG media_image15.png 98 286 media_image15.png Greyscale , and Fig. 4 shows that the color map PNG media_image8.png 128 74 media_image8.png Greyscale is generated based on the text features produced by the “Text Encoder.” Wu D provides detailed context of the “Text Encoder”: PNG media_image16.png 246 558 media_image16.png Greyscale ). Regarding Claim 7, Wu further teaches The method of claim 1, wherein: the color map includes a plurality of colors corresponding to a plurality of elements of the scene described by the text prompt ( PNG media_image13.png 164 672 media_image13.png Greyscale Here, the colors for cars and persons are different. After the combination of the embodiments of Fig 4 and Fig. 10, the feature mapped to the color map is represented by different colors for different objects. Wu 4.1: PNG media_image17.png 138 552 media_image17.png Greyscale .). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Wu’s teaching of segmenting multiple objects with Wu’s fig. 4 algorithm. One of ordinary skill in the art would be motivated to process/analyze complex images that reflect real situations. Claim 14 is substantially similar to Claim 1. The rejection analyses for Claim 1 based on Wu in view of Zheng are applied to Claim 14. In addition, Wu teaches An apparatus (“With the great development of computer graphics (e.g., generative model), an alternative way is to utilize synthetic data, which is largely available from the virtual world, and the pixel-level ground truth can be freely and automatically generated.”) comprising: at least one processor; at least one memory including instructions executable by the at least one processor (The Examiner takes an Official Notice that it would have been well known in the art that a computer may have at least one processor; at least one memory including instructions and data, wherein the instructions are executable by the at least one processor. The benefits of combining this well-known knowledge would have been for faster and accurate computation for complex algorithms. After the combination of this well-known knowledge instructions and model parameters may be stored on the memory.). Regarding Claim 18, Wu further teaches The apparatus of claim 14, wherein: the second generative machine learning model comprises a diffusion model ( Wu’s model includes a diffusion model: PNG media_image18.png 142 554 media_image18.png Greyscale Wu 4.1. Fig. 4: PNG media_image19.png 220 212 media_image19.png Greyscale ). Regarding Claim 19, Wu further teaches The apparatus of claim 14, further comprising: a text encoder configured to encode the text prompt to obtain text features representing the object, wherein the color map is generated based on the text features ( Wu Fig. 4: PNG media_image15.png 98 286 media_image15.png Greyscale , and Fig. 4 shows that the color map PNG media_image8.png 128 74 media_image8.png Greyscale is generated based on the text features produced by the “Text Encoder.” Wu D: PNG media_image16.png 246 558 media_image16.png Greyscale ). Claims 3-4 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Wu in view of Zheng as applied to Claim 1 or 14, in further view of Zhang et al. (“Adding Conditional Control to Text-to-Image Diffusion Models.”). Regarding Claim 3, Wu in view of Zheng teaches The method of claim 1. Wu in view of Zheng does not explicitly disclose; however Zhang teaches further comprising: creating a training set including the image segmentation mask and the synthesized image; and training a segmentation model using the training set ( Zhang: PNG media_image20.png 482 516 media_image20.png Greyscale The synthesized image is mapped to the input image, after the combination of Wu and Zhang, Wu’s synthetic image as shown in fig. 4 is used as the Zhang’s input image. The image segmentation mask is mapped to “input conditioning images.” Zhang 1. Introduction: PNG media_image21.png 252 510 media_image21.png Greyscale . ). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Zhang’s image generation diffusion model with Wu in view of Zheng. One of ordinary skill in the art would be motivated to give user more control to set conditions and/or to generate images with higher qualities. Zhang states, “Can we enable finer grained spatial control by letting users provide additional images that directly specify their desired image composition? In computer vision and machine learning, these additional images (e.g., edge maps, human pose skeletons, segmentation maps, depth, normals, etc.) are often treated as conditioning on the image generation process.” Abstract. Regarding Claim 4, Wu in view of Zheng and Zhang teaches The method of claim 1, further comprising: obtaining an annotated segmentation mask (Wu’s title recites “DiffuMask: Synthesizing Images with Pixel-level Annotations for Semantic Segmentation Using Diffusion Models”); and generating, using the second generative machine learning, a synthesized image based on the text prompt and the annotated segmentation mask (Zhang 1. Introduction: PNG media_image21.png 252 510 media_image21.png Greyscale . Here, the mask-to-image generation model is mapped to the text-to-image diffusion model conditioned on segmentation maps.). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Zhang’s image generation diffusion model with Wu in view of Zheng. One of ordinary skill in the art would be motivated to give user more control to set conditions and/or to generate images with higher qualities. Zhang states, “Can we enable finer grained spatial control by letting users provide additional images that directly specify their desired image composition? In computer vision and machine learning, these additional images (e.g., edge maps, human pose skeletons, segmentation maps, depth, normals, etc.) are often treated as conditioning on the image generation process.” Abstract. Regarding Claim 16, Wu in view of Zheng and Zhang teaches teaches The apparatus of claim 14, wherein: the first generative machine learning model comprises a diffusion model ( Zhang Title: “Adding Conditional Control to Text-to-Image Diffusion Models,” which corresponds to the feature of Wu’s fig. 4: PNG media_image6.png 267 316 media_image6.png Greyscale ). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Zhang’s image generation diffusion model with Wu in view of Zheng. One of ordinary skill in the art would be motivated to create visually stunning images. “With the advent of text-to-image diffusion models[54, 62, 72], we can now create visually stunning images by typing in a text prompt.” Zhang 1. Introduction. Claims 8 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Wu et al. (“DiffuMask: Synthesizing Images with Pixel-level Annotations for Semantic Segmentation Using Diffusion Models”) in view of Ahn et al. (“Learning Pixel-level Semantic Affinity with Image-level Supervision for Weakly Supervised Semantic Segmentation”) and KAUFHOLD et al. (US 20190080205 A1). Regarding Claim 8, Wu teaches A method of training a machine learning model (“In this paper, we show that it is possible to automatically obtain accurate semantic masks of synthetic images generated by the Off-the-shelf Stable Diffusion model, which uses only text-image pairs during training.” Wu Abstract. “The pre-trained Stable Diffusion [49], the text encoder of CLIP [47], AffinityNet [2] are adopted as the base components. We do not finetune the Stable Diffusion and only train AffinityNet for each category.” Wu 4.1. The training may include the pretraining, fine tuning, and/or the training of the AffinityNet.), the method comprising: trainingWu. Fig. 4 PNG media_image22.png 90 192 media_image22.png Greyscale ); generating an image using the generative machine learning model (DiffuMask) (Wu fig. 4; “Experiments demonstrate that the existing segmentation methods trained on synthetic data of DiffuMask can achieve a competitive performance over the counterpart of real data (VOC 2012, Cityscapes).” Wu Abstract. Here, “synthetic data of DiffuMask” are synthetic data generated by DiffuMask); and training a segmentation model using the image “Experiments demonstrate that the existing segmentation methods trained on synthetic data of DiffuMask can achieve a competitive performance over the counterpart of real data (VOC 2012, Cityscapes).” Wu Abstract. The image is mapped to the synthetic data generated by the DiffuMask.). Wu does not explicitly disclose; however, Ahn teaches obtaining a training set including a segmentation mask ( Ahn Fig. 1: PNG media_image23.png 304 440 media_image23.png Greyscale , stating “From the salient regions, we sample pairs of adjacent coordinates and assign binary labels to them according to their class consistency. The labeled pairs are then used to train AffinityNet (Section 3.2).” The labeled region, e.g., a bird, is mapped to the segmentation mask); training, using the training set, a generative machine learning model (Wu’s DiffuMask, which comprises Ahn’s AffinityNet) to generate images ( PNG media_image22.png 90 192 media_image22.png Greyscale ) based on the segmentation maskThe model is trained based on the segmentation mask, and output images are therefore based on the segmentation mask). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Ahn’s training of a machine learning model with Wu. One of ordinary skill in the art would be motivated to make a machine learning model intelligent and/or adapted to specific group of data. In addition, Wu explicitly discloses the use of Ahn’s AffinityNet. Wu 4.1. Wu in view of Ahn does not explicitly disclose; however, KAUFHOLD teaches training a model using a data set that comprises a first data set and a second data set that includes the segmentation mask (“The generated images 2409 are treated as training images in addition to the pre-existing relatively sparse set of training images 2413, whereby the combination of these two data sets forms a new training data set 2429.” KAUFHOLD ¶ 118. Segmentation masks PNG media_image22.png 90 192 media_image22.png Greyscale generated by Wu’s DiffuMask correspond to the first data set. Ahn’s segmentation mask corresponds to the second data set.). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine KAUFHOLD’s combining data sets with Wu in view of Ahn. One of ordinary skill in the art would be motivated to create a reasonable size of the training data and/or to diversify training data. Regarding Claim 13, Wu in view of Ahn and Kaufhold teaches The method of claim 8, further comprising: initializing the text-to-mask generation model using parameters from a text-to-image generation model ( “We show a novel insight that it is possible to automatically obtain the synthetic image and mask annotation from a text-supervised pre-trained diffusion model.” Wu 1. Introduction. “In this paper, we explore simultaneously generating images and the semantic mask described in the text prompt with the existing pre-trained diffusion model.” Wu 3. Methodology. The text-to-image generation model is mapped to the pre-trained diffusion model. Using a pretrained diffusion model is reusing parameters from the pretrained diffusion model. Wu 4.1: PNG media_image24.png 84 552 media_image24.png Greyscale PNG media_image25.png 492 538 media_image25.png Greyscale ). Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over Wu in view of Ahn and KAUFHOLD as applied to Claim 8, in further view of Rombach (“High-Resolution Image Synthesis with Latent Diffusion Models”) Regarding Claim 9, Wu in view of Ahn and Kaufhold teaches The method of claim 8. Wu in view of Ahn and Kaufhold does not explicitly disclose; however, Rombach teaches wherein training the generative machine learning model (Wu’s DiffuMask) comprises: obtaining a ground-truth color map indicating a region corresponding to an object in a scene ( The ground-truth color map is mapped to disclosed semantic maps. “We use this to train models for semantic synthesis, super-resolution (Sec. 4.4) and inpainting (Sec. 4.5). For semantic synthesis, we use images of landscapes paired with semantic maps [23, 61] and concatenate downsampled versions of the semantic maps with the latent image representation of a f = 4 model (VQ-reg., see Tab. 8).” Rombach 4.3.2 Fig. 9 (top left) provides examples for ground-truth color map, which indicates objects, like mountains, trees, lake/river in a scene. PNG media_image26.png 408 584 media_image26.png Greyscale ); computing a diffusion loss based on the ground-truth color map ( Rombach 3.3: PNG media_image27.png 252 554 media_image27.png Greyscale Here, LDM stands for latent diffusion model, as the title states, “High-Resolution Image Synthesis with Latent Diffusion Models.” Further, equation (3) is a loss function. The ground-truth color map is mapped to disclosed semantic maps. “We use this to train models for semantic synthesis, super-resolution (Sec. 4.4) and inpainting (Sec. 4.5). For semantic synthesis, we use images of landscapes paired with semantic maps [23, 61] and concatenate downsampled versions of the semantic maps with the latent image representation of a f = 4 model (VQ-reg., see Tab. 8).” Rombach 4.3.2.); and updating parameters of the generative machine learning model based on the diffusion loss ( PNG media_image28.png 346 558 media_image28.png Greyscale Rombach 3.3: PNG media_image29.png 76 432 media_image29.png Greyscale Here, learning LDM is that LDM’s parameters are updated based on the specified loss function.). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Rombach’s diffusion model training with Wu in view of Ahn and Kaufhold. One of ordinary skill in the art would be motivated to achieve high quality images. Rombach states, “By decomposing the image formation process into a se quential application of denoising autoencoders, diffusion models (DMs) achieve state-of-the-art synthesis results on image data and beyond. Additionally, their formulation al lows for a guiding mechanism to control the image gen eration process without retraining.” Rombach Abstract. Claim 12 is rejected under 35 U.S.C. 103 as being unpatentable over Wu in view of Ahn, KAUFHOLD and Rombach as applied to Claim 9, in further view of Sehgal et al. (“Generating Image Captions based on Deep Learning and Natural language Processing”). Regarding Claim 12, Wu in view of Ahn, KAUFHOLD and Rombach teaches The method of claim 8, wherein obtaining the training set comprises: obtaining an image corresponding to the ground-truth color map ( “We use this to train models for semantic synthesis, super-resolution (Sec. 4.4) and inpainting (Sec. 4.5). For semantic synthesis, we use images of landscapes paired with semantic maps [23, 61] and concatenate downsampled versions of the semantic maps with the latent image representation of a f = 4 model (VQ-reg., see Tab. 8).” Rombach 4.3.2. The ground-truth color map is mapped to the disclosed “semantic maps.” The obtained image is mapped to the disclosed “image of landscapes.”); and generating a text prompt based on the image ( Rombach 4.3: PNG media_image30.png 250 556 media_image30.png Greyscale , where the model is learned/trained. Rombach Fig. 3. The text prompt is mapped to the disclosed text conditions. Rombach: PNG media_image31.png 252 560 media_image31.png Greyscale . The text condition is to condition the obtained image, and therefore, the text prompt is based on the image.). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Rombach’s diffusion model training with Wu in view of Ahn and Kaufhold. One of ordinary skill in the art would be motivated to achieve high quality images. Rombach states, “By decomposing the image formation process into a se quential application of denoising autoencoders, diffusion models (DMs) achieve state-of-the-art synthesis results on image data and beyond. Additionally, their formulation al lows for a guiding mechanism to control the image gen eration process without retraining.” Rombach Abstract. Wu in view of Ahn, KAUFHOLD and Rombach implicitly teaches generating a text prompt based on the image. However, the teaching is not explicit. Sehgal teaches generating a text prompt based on the image ( Sehgal Title: “Generating Image Captions based on Deep Learning and Natural language Processing.” After the combination of Wu in view of Ahn, KAUFHOLD and Rombach and Sehgal, the generated caption would be used as the text prompt.). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Sehgal’s generation of text caption/description with Rombach in view of Wu in view of Ahn, KAUFHOLD and Rombach. One of ordinary skill in the art would be motivated to further automate a process and to increase efficiency of caption generation. “It is a technique in which computers learn to perform tasks that can just be performed by humans. Deep Learning algorithms are capable of giving results that can perform better than humans. There are many algorithms such as convolutional neural network, recurrent Neural Network etc that are applied in different applications like speech recognition, video recognition, NLP etc” Sehgal IV. Claim 20 is rejected under 35 U.S.C. 103 as being unpatentable over Wu in view of Zheng as applied to Claim 14, in further view of Sehgal et al. (“Generating Image Captions based on Deep Learning and Natural language Processing”). Regarding Claim 20, Wu in view of Zheng teaches The apparatus of claim 14. Wu in view of Zheng does not explicitly disclose further comprising: a captioner configured to generate an image description based on an image. Sehgal teaches further comprising: a captioner configured to generate an image description based on an image ( Sehgal Title: “Generating Image Captions based on Deep Learning and Natural language Processing.” After the combination of Wu and Sehgal, the generated caption would be used as the image description.). It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Sehgal’s generation of text caption/description with Wu in view of Zheng. One of ordinary skill in the art would be motivated to further automate a process and to increase efficiency of caption generation. “It is a technique in which computers learn to perform tasks that can just be performed by humans. Deep Learning algorithms are capable of giving results that can perform better than humans. There are many algorithms such as convolutional neural network, recurrent Neural Network etc that are applied in different applications like speech recognition, video recognition, NLP etc” Sehgal IV. Conclusion Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a). A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action. Any inquiry concerning this communication or earlier communications from the examiner should be directed to ZHENGXI LIU whose telephone number is (571)270-7509. The examiner can normally be reached M-F 9 AM - 5 PM. Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kee Tung can be reached at 571-272-7794. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. /ZHENGXI LIU/Primary Examiner, Art Unit 2611