DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 01/09/2025, 08/15/2025 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 5-7, 11-13, 15, 16 are rejected under 35 U.S.C. 103 as being unpatentable over ZHENG et al. (US 2022/0327657 A1) in view of SHEN et al. (CN114862665A) and Flir (https://www.flir.com/discover/security/thermal/5-benefits-of-thermal-imaging-cameras/ ras | Flir).
RE claim 1, Zheng is made of record as teaching a system/method of a digital image semantic layout manipulation system that generates refined digital images resembling the style of one or more input images [abstract]. Zheng teaches a method performed by at least one processor for generating a composite image, the method comprising:
(a)
receiving an input image including a background and a specific object;
Fig. 6A, Zheng teaches an input image (302) [0133]. In the example, the image shows a gallery of paintings, including four paintings on the left wall and two paintings on a back wall. The paintings can be considered said specific object and the walls can be considered said background.
(b)
extracting at least one piece of content information about the input image; and
Fig. 6A, Zheng teaches a semantic feature extraction neural network (602) that can generate the semantic layout (604) [0133]. A sematic layout assigns classification labels to each pixel in a digital image (said extracting content information). Regarding the input image (302), the pixels displaying a painting can be labeled as “art” or “painting” while other pixels in the semantic layout can be labeled as “wall,” “floor,” ceiling,” “lights,” and/or “door” (said content information) [0134].
(c)
generating a composite image of a specific domain style associated with the at least one piece of content information by using an image generation model.
Fig. 6c, Zheng teaches the semantic layout editor (606) facilitates combining semantic areas and/or semantic labels from one semantic layout to another semantic layout [0140]. For example, the semantic layout system (106) generates a first semantic layout for a first digital image that includes a landscape and a second semantic layout for a second digital image including a person. The semantic layout edit (606) may detect input adding the semantic area of the person from the second semantic layout to the first semantic layout [0140]. Then, the semantic layout system (106) follows the edited first semantic layout to generate a refined image that shows the person in the landscape (said composite image) [0140]. Fig. 7B, Zheng further teaches generated a refined image (312) (said composite image) using a semantic layout manipulation neural network (300) (said using an image generation model) [0145].
Zheng teaches generating the composite image but fails to discuss a specific domain style. Shen is made of record as teaching a system/method for generating an infrared face [0001]. Shen teaches converting a visible light face image into an infrared face image [0049]. The system of Shen uses an infrared image generation network [0052]. The second image generation unit (320) is configured to input the first batch of real visible light face images into the infrared image generation network to generate corresponding batch of infrared face images (said generating an image of a specific domain style) [0092].
It would have been obvious before the effective filing date of the claimed invention to modify Zheng’s semantic-content-based image generation technique in view of Shen to generate the output image in an infrared domain style. Shen teaches converting a visible light input image into an infrared image using an image generation network and preserving characteristics of the source image through content-based training while imparting the infrared appearance through style-based training. It would be beneficial to employ the infrared domain generation technique of Shen with the semantic content controlled generation of Zheng to obtain an infrared domain image that preserves the semantic content and structural arrangement of the objects and background of the source image, because viewing images in infrared provides allows for viewers to see what their eyes typically can’t: invisible heat radiation emitted or reflected by all objects, regardless of lighting conditions. This would benefit in the domain of security [Flir].
RE claim 5, Zheng in view of Shen and Flir teaches
(a)
wherein the content information represents structural information of the background and objects in the input image.
Fig. 6A, Zheng teaches a semantic feature extraction neural network (602) that can generate the semantic layout (604) [0133]. A sematic layout assigns classification labels to each pixel in a digital image (said content information). Regarding the input image (302), the pixels displaying a painting can be labeled as “art” or “painting” while other pixels in the semantic layout can be labeled as “wall,” “floor,” ceiling,” “lights,” and/or “door” (said content information represents structural information) [0134].
RE claim 6, the language of claim 6 recites, “wherein the at least one piece of content information comprises at least one of”, which limits the claim to needing only one of the limitations. Therefore, Zheng teaches the limitations of claim 6(a). It should be noted that since only one limitation is required, the limitations of claims 6(b)-(g) are mute.
(a)
semantic segmentation information about the input image,
Fig. 6A, Zheng teaches a semantic feature extraction neural network (602) that can generate the semantic layout (604) [0133]. A sematic layout (said semantic segmentation information) assigns classification labels to each pixel in a digital image. Regarding the input image (302), the pixels displaying a painting can be labeled as “art” or “painting” while other pixels in the semantic layout can be labeled as “wall,” “floor,” ceiling,” “lights,” and/or “door” [0134]. Furthermore, Zheng teaches the series of acts (11000) may include determining a semantic layout of the digital image utilizing a semantic segmentation neural network (said semantic segmentation) [0194].
(b)
panoptic segmentation information associated with the input image,
(c)
instance segmentation information associated with the input image,
(d)
segmentation anything model (SAM) result information associated with the input image,
(e)
bounding box information associated with the input image, edge information associated with the input image,
(f)
depth information associated with the input image, or
(g)
sketch information associated with the input image.
RE claim 7, Zheng teaches wherein the extracting the at least one piece of content information comprises:
(a)
extracting multiple different pieces of content information about the input image; and
Fig. 6A, Zheng teaches an input image (302) [0133]. In the example, the image shows a gallery of paintings, including four paintings on the left wall and two paintings on a back wall. The paintings can be considered said specific object and the walls can be considered said background. Fig. 6A, Zheng teaches a semantic feature extraction neural network (602) that can generate the semantic layout (604) [0133]. A sematic layout assigns classification labels to each pixel in a digital image (said extracting content information). Regarding the input image (302), the pixels displaying a painting can be labeled as “art” or “painting” while other pixels in the semantic layout can be labeled as “wall,” “floor,” ceiling,” “lights,” and/or “door” (said multiple different pieces of content information) [0134].
wherein the generating the composite image comprises:
(b)
based on the multiple different pieces of content information, generating, by using the image generation model, the composite image.
Fig. 6c, Zheng teaches the semantic layout editor (606) facilitates combining semantic areas and/or semantic labels from one semantic layout to another semantic layout [0140]. For example, the semantic layout system (106) generates a first semantic layout for a first digital image that includes a landscape and a second semantic layout for a second digital image including a person (said based on multiple different piece of content information). The semantic layout edit (606) may detect input adding the semantic area of the person from the second semantic layout to the first semantic layout [0140]. Then, the semantic layout system (106) follows the edited first semantic layout to generate a refined image that shows the person in the landscape (said composite image) [0140]. Fig. 7B, Zheng further teaches generated a refined image (312) (said composite image) using a semantic layout manipulation neural network (300) (said using an image generation model) [0145].
RE claim 11, Zheng in view of Shen and Flir teaches
(a)
wherein at least one of a domain style of the background and a domain style of the specific object included in the input image is different from a domain style of the composite image.
As taught in the rationale of claim 1(a), Fig. 6A, Zheng teaches an input image (302) [0133]. In the example, the image shows a gallery of paintings, including four paintings on the left wall and two paintings on a back wall. The paintings can be considered said specific object and the walls can be considered said background.
As modified by Shen, Shen teaches converting a visible light face image into an infrared face image [0049]. Therefore, the domain style of the final image (said composite image) is infrared (said different domain style).
The same motivation to combine as taught in the rationale of claim 1 is incorporated herein.
RE claim 12, Zheng in view of Shen and Flir teaches further comprising:
(a)
training the image generation model,
In further view of Shen, [0057-0066] teaches the training to establish a generation model of the infrared face image (said training the image generation model) [0066].
wherein the training the image generation model comprises:
(b)
receiving a training image of the specific domain style;
Shen teaches inputting the first batch of real visible light face images into the infrared image generation network to generate corresponding batch of initial infrared face images [0052]. The preset training image dataset includes a first preset number of real visible light face images and a second preset number of real infrared face images and real infrared face images (said training image of the specific domain style) [0051].
(c)
extracting at least one piece of content information about the training image; and
In further view of Shen, Shen teaches the first batch of real visible light face images, and the corresponding batch of initial infrared face images are respectively input into the preset auxiliary recognition network [0055]. Shen further teaches obtaining their respective feature maps to determine whether the content of the first bath of real visible light face images and the corresponding batch of initial infrared face images is the same and then obtain the content loss of the corresponding batch of initial infrared face images (said extracting piece of content information about the training image) [0055].
(d)
training the image generation model by using a pair composed of the training image and the at least one piece of content information as training data.
Shen teaches the first bath of real infrared face images, and the corresponding bath of initial infrared face images (said using a pair) are respectively input to the image discrimination network to generate discriminant loss and confrontation loss [0056]. As further discussed in the teachings of Shen, [0057-0066] teaches the training to establish a generation model of the infrared face image (said training the image generation model) [0066].
The same motivation to combine as taught in the rationale of claim 1 is incorporated herein.
RE claim 13, Zheng in view of Shen and Flir teaches
(a)
wherein the specific domain style is an infrared (IR) domain style.
In further view Shen, Shen is made of record as teaching a system/method for generating an infrared face [0001]. Shen teaches converting a visible light face image into an infrared face image [0049]. The system of Shen uses an infrared image generation network [0052]. The second image generation unit (320) is configured to input the first batch of real visible light face images into the infrared image generation network to generate corresponding batch of infrared face images (said infrared domain style) [0092].
The same motivation to combine as taught in the rationale of claim 1 is incorporated herein.
RE claim 15, claim 15 recites similar limitations as claim 1 but in manufacture form. Therefore, the same rationale used for claim 1 is applied. Furthermore, Fig. 12, Zheng teaches components (1010-1020) may include one or more instructions stored on a computer-readable storage medium and executable by processors of one or more computing devices [0177].
RE claim 16, claim 16 recites similar limitations as claim 1 but in system form. Therefore, the same rationale used for claim 1 is applied. Furthermore, Zheng teaches
(i)
a communication interface;
Fig. 12, Zheng teaches a communication interface (1210) [0206].
(ii)
a memory;
Fig. 12, Zheng teaches a computer-readable storage medium [0177]. The computing device (1200) may include memory (1204) [0206].
(iii)
and a processor connected to the memory and configured to execute at least one computer-readable program stored in the memory
Fig. 12, Zheng teaches components (1010-1020) may include one or more instructions stored on a computer-readable storage medium and executable by processors of one or more computing devices [0177]. The computing device (1200) may include one or more processor(s) (1202) and memory (1204) [0206].
Claim 2 is rejected under 35 U.S.C. 103 as being unpatentable over ZHENG et al. (US 2022/0327657 A1) in view of SHEN et al. (CN114862665A) and Flir (https://www.flir.com/discover/security/thermal/5-benefits-of-thermal-imaging-cameras/ ras | Flir) as applied to claim 1 above, and further in view of SHECHTMAN et al. (US 2020/0302251 A1).
RE claim 2, Zheng teaches Zheng teaches an input image (302) [0133]. In the example, the image shows a gallery of paintings, including four paintings on the left wall and two paintings on a back wall. The paintings can be considered said specific object and the walls can be considered said background. However, Zheng does not discuss merging images to generate the input image. Shen and Flir do not teach the limitations either.
Shechtman teaches an image composite system that employs generative adversarial network to generate adversarial network to generative realistic composite images [0025]. Schectman teaches wherein the receiving the input image comprises:
(a)
receiving a first image associated with the background;
In further view of Shechtman, Fig. 3 illustrates employing a trained generative adversarial network (300) that generates realistic composite images [0089]. The system receives a background image (306) [0090].
(b)
receiving a second image associated with the specific object; and
Shechtman teaches the system receives a foreground object (308) [0090].
(c)
generating the input image by merging the first image and the second image.
Schectman further teaches apply the warp parameters to the foreground object (308) and composite the warped foreground object with the background image (306) to generate a composite image (310) [0092].
It would have been obvious before the effective filing date of the claimed invention to employ the compositing technique of Shechtman to generate the input image of Zheng in view of Shen in order to permit a selected foreground object to be incorporated into a selected background while providing geometrically correct spatial alignment of the object with the background [Shechtman 0031], thereby producing a natural and realistic source image for subsequent content extraction and target-domain image generation. The composite image of Shechtman further provides an input image to Zheng in view of Shen based on the user’s needs, such as an image difficult to capture or customization.
Claims 3-4 are rejected under 35 U.S.C. 103 as being unpatentable over ZHENG et al. (US 2022/0327657 A1) in view of SHEN et al. (CN114862665A) and Flir (https://www.flir.com/discover/security/thermal/5-benefits-of-thermal-imaging-cameras/ ras | Flir) as applied to claim 1 above, and further in view of PARK et al. (US 202/0242774 A1).
RE claim 3, Zheng in view of Shen and Flir teaches the limitations of claim 14 except providing details about the input image. Park is made of record as teaching the art of creating an image based on semantic layout and using an image synthesis network [abstract]. In view of Park, Park teaches defining regions and assigning/associating labels to those regions [0018]. The image once created forms a type of segmentation mask, where the shape and size of each region can be thought of as a mask that enables a specified type of object to be rendered only within the respective mask region or boundaries. Because the regions are associated with labels or other designations for types of objects, this segmentation mask can also be thought of as a semantic layout, as it provides context for the types of objects in each of the different masked or bounded regions [0018]. Park teaches wherein the receiving the input image comprises:
(a)
receiving background information associated with the background;
Fig. 5, Park teaches a user can apply a label (said background information) to the background, such as to cause the image to have a “sky” label for any pixels that do not otherwise have a region associated therewith [0039].
(b)
receiving object information associated with the specific object; and
Fig. 5, Park teaches the user can provide input that can designate a boundary of a region for the image [0040]. Along with the boundary for the region, a selection of a label (said object information) for the region can be received (506), where the label is a semantic label indicating a type of object to be rendered for that region [0040].
(c)
based on the background information and the object information, generating, by using an artificial neural network model, the input image.
Fig. 5, Park teaches the semantic layout can be provided (514) as input to an image synthesis network (said artificial neural network model). A set of inferences from the network can then be used to generate (518) a photorealistic image (said generating the input image) including the types of objects indicated by the labels for the designated regions [0041].
It would have been obvious before the effective filing date of the claimed invention to modify the semantic image generation system of Zheng to utilize the semantic-layout generation technique taught by Park. The semantic-layout-based-image generation of Park provides an increased control over the type and spatial arrangement of the objects and background represented in the generated input image, thereby permitting specified objects to be generated at specified locations and within specified regions boundary while maintaining the desired overall scene structure [Park: Fig. 3].
RE claim 4, the language of claim 4 recites, “wherein the object information comprises at least one of”, which limits the claim to needing only one of the limitations. Therefore, in further view of Park, Park teaches the claim limitation of claim 4(a), 4(b), and 4(c). It should be noted that since only one limitation is required, the limitations of claim 4(d) are mute. In further view of Park, Park teaches wherein the object information comprises at least one of:
(a)
object type information associated with the specific object,
Fig. 5, Park teaches the user can provide input that can designate a boundary of a region for the image [0040]. Along with the boundary for the region, a selection of a label (said object information) for the region can be received (506), where the label is a semantic label indicating a type of object to be rendered for that region [0040].
(b)
object shape information associated with the specific object,
Fig. 5, Park teaches the user can provide input that can designate a boundary of a region for the image [0040]. The boundary indicates a shape [0041].
(c)
object location information associated with the specific object, or
Fig. 3, the GUI of Park enables the user to create the regions of the semantic layout [0022]. The regions of designated locations.
(d)
object posture information associated with the specific object.
The same motivation to combine as taught in the rationale of claim 3 is incorporated herein.
Claims 8-9 are rejected under 35 U.S.C. 103 as being unpatentable over ZHENG et al. (US 2022/0327657 A1) in view of SHEN et al. (CN114862665A) and Flir (https://www.flir.com/discover/security/thermal/5-benefits-of-thermal-imaging-cameras/ ras | Flir) as applied to claim 1, and in further view of HUANG et al. (US 2023/0045076 A1).
RE claim 8, Fig. 6A, Zheng teaches a semantic feature extraction neural network (602) that can generate the semantic layout (604) [0133]. A sematic layout assigns classification labels to each pixel in a digital image (said extracting content information). Regarding the input image (302), the pixels displaying a painting can be labeled as “art” or “painting” while other pixels in the semantic layout can be labeled as “wall,” “floor,” ceiling,” “lights,” and/or “door” (said multiple different pieces of content information) [0134]. However, Zheng in view of Shen and Flir do not discuss encoding the content to generate first, second, and third encoded data. Huang is made of record as teaching a system/method that generates image using neural networks [abstract]. Huang teaches wherein:
(a)
the multiple different pieces of content information comprise first content information, second content information, and third content information; and
As taught by Zheng, the semantic layout can label the pixels, such as “art”, “wall”, “floor”, “ceiling”, etc. [0134]. Huang teaches a user may provide input to be used to generate one or more images [0045]. A user may provide input for multiple modalities, which can be used to generate an image [0046]. Fig. 1, a user can provide multiple different types of input, i.e., segmentation map (104) (said first content information), style input (108) (said second content information), and edge representation (106) (said third content information) [0047].
the generating the composite image based on the multiple different pieces of content information comprises:
Huang further teaches a set of neural networks that can be used to generate an image. Fig. 2A, there can be separate encoder for each modality or type of conditional input [0050].
(b)
encoding the first content information to generate first encoded data;
Huang teaches segmentation encoder (206) [0050]
(c)
encoding the second content information to generate second encoded data;
Huang teaches style encoder (212) [0050]
(d)
encoding the third content information to generate third encoded data; and
Huang teaches edge encoder (210) [0050]
(e)
generating the composite image of the specific domain style by inputting the first encoded data, the second encoded data, and the third encoded data to the image generation model.
These encodings can be provided to a modality fusion module (202) that fuses or combines aspects of these various conditional inputs [0050]. Decoder (208) can utilize features from image space corresponding to this fused representation, along with shapes and semantics provided in these conditional inputs, to generate at least one output image (214) [0050].
It would have been obvious before the effective filing date of the claimed invention to modify the image generation system of Zheng in view of Shen and Flir to separately encode the different pieces of content information and combine the resulting encoded representations for use by the image generation model, as taught by Huang, in order to permit multiple different types of content information to jointly control image generation, and thereby provide the image generation model with complementary information concerning the semantic content and spatial structure of the image. Huang teaches that multiple conditional modalities can guide image synthesis and that the respective encoded representations can be fused into a combined representation used by the decoder to generate the resulting image [0050].
RE claim 9, in further view of Huang, Huang teaches wherein:
(a)
the first content information comprises semantic segmentation information;
Fig. 1, Huang teaches a user can provide multiple different types of input, i.e., segmentation map (104) (said first content information),
(b)
the second content information comprises sketch information; and
style input (108) (said second content information), and
(c)
the third content information comprises edge information.
edge representation (106) (said third content information) [0047].
The same motivation to combine as taught in the rationale of claim 8 is incorporated herein.
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over ZHENG et al. (US 2022/0327657 A1) in view of SHEN et al. (CN114862665A) and Flir (https://www.flir.com/discover/security/thermal/5-benefits-of-thermal-imaging-cameras/ ras | Flir) as applied to claim 1, and in further view of HSIAO et al. (US 2021/0142455 A1).
RE claim 10, Zhen in view of Shen and Flir teaches the limitations of claim 10 except for disclosing the background and object are different domain styles. Hsiao is made of record as teaching the art of transforming the background image or foreground image according to a selected style [abstract]. In further view of Hsiao, Hsiao teaches
(a)
wherein a domain style of the background and a domain style of the specific object included in the input image are different from each other.
Fig. 6 (62), Hsiao teaches a mask is obtained by separating an original image into a background image and a foreground image [0051]. Fig. 6 (64) the background image or foreground image can be input to the image Styler, where the image Styler will transform the background or foreground image according to the selected style [0054]. Fig. 6 (66) a stylized image is obtained according to the mask and the partial stylized image [0055]. The stylized image can be obtained by filling the mask with the partial stylized image (said input image) and at least one of the background images and the foreground image that does not subject to the transforming [0056]. Thus, the background image is styled while the foreground remains original (said domain style of background different from domain style of specific object). Fig. 10 provides an example of the final image where the foreground object is original with different stylized backgrounds [0061].
It would have been obvious before the effective filing date of the claimed invention to modify the input image of Zheng with the stylized input image of Hsiao. The teachings of Hsiao provide means for more diversified media. The system method Hsiao improves the display effect of media through style transfer [Hsiao: 0003].
Claim 14 is rejected under 35 U.S.C. 103 as being unpatentable over ZHENG et al. (US 2022/0327657 A1) in view of SHEN et al. (CN114862665A) and Flir (https://www.flir.com/discover/security/thermal/5-benefits-of-thermal-imaging-cameras/ ras | Flir) as applied to claim 1, and in further view of LIU et al. (CN113420639A)
RE claim 14, Zheng in view of Shen and Flir teaches the limitations of claim 14 except the object being associated with a defense industry. Liu is made of record as teaching a system/method for establishing a near-ground infrared target data set based on a generation countermeasure network to simulate real near-ground target infrared images [0006]. Liu teaches
(a)
wherein the specific object is an object associated with a defense industry.
Fig. 4, Liu teaches the modified generation network can generate a near-ground infrared image [0039]. The infrared data set can be four types of targets: combat personnel, tanks, armored vehicles, and civilian vehicles [0035]. The infrared data set is used to train a DCGAN [0037].
It would have been obvious before the effective filing date of the claimed invention to use the generated images of Liu as input images of Zheng in view of Shen and Flir to produce additional infrared images of military targets for augmenting training datasets used in infrared target detection and recognition, thereby increasing the diversity of available training samples and improving target-detection/recognition performance [Liu: 0003-0005].
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MICHELLE L SAMS:
direct telephone number:
(571) 272-7661
email:
michelle.sams@uspto.gov
Examiner interviews are available via telephone and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kee M. Tung can be reached on (571)272-7794. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MICHELLE L SAMS/
Primary Examiner, Art Unit 2611
17 September 2026
Prosecution Insights