Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Use of indicates a limitation is not explicitly disclosed by the reference alone.
Claim(s) 1, 7, 8, 10-11, 14-15, 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Hsiao (US2021/0142455) in view of Castillo, Targeted Style Transfer Using Instance-Aware Semantic Segmentation.
Claim 1
Examiner’s Interpretation:
Machine readable media can encompass forms of signal transmission media that falls outside of the four statutory categories of invention. MPEP 2106; citing In re Nuijten, 500 F.3d 1346, 84 USPQ2d 1495 (Fed. Cir. 2007). A claim whose BRI covers both statutory and non-statutory embodiments embraces subject matter that is not eligible for patent protection and therefore is directed to non-statutory subject matter. MPEP 2106.
Because non-transitory without additional definition excludes signal media and the like, the broadest reasonable interpretation of the claimed medium in view of Applicant’s specification covers only eligible subject matter.
Claim Mapping:
Hsiao discloses a non-transitory computer-readable medium storing executable instructions, which when executed by at least one processor, cause the at least one processor to perform operations comprising:
receiving a request to transfer a global style from a source digital image to a target digital image (Hsiao, ¶ 37: “For example, the image Styler can be activated when the camera 105 is launched to stylize an image in real-time during capturing. Alternatively, the portrait Styler can be activated for style transfer of an image album, which is stored in the memory 103 for example, upon request of a user. Similarly, the image Styler can also be used for style transfer of videos; in this mode, it stylizes the background or the foreground of videos in a video album.”);
identifying, in response to receiving the request to transfer, a foreground object within the target digital image, the foreground object comprising a style (Hsiao, ¶ 54: “The style can be selected by user or can be selected or set by the system of the terminal device by default by taking user habit or user preference into consideration according to usage history of the user. In terms of whether the background image or the foreground image will be input to the image transformation network, it can be determined or selected by the user, or can be set by the system.”); and
generating a modified digital image by transferring, utilizing a style transfer neural network (Hsiao, ¶ 39: “The image Styler provided herein can be embodied as a neural network(s).”), the global style from the source digital image to a background of the target digital image while maintaining the style of the foreground object (Hsiao, ¶ 54: “Then the image Styler will transform the background image or the foreground image input thereto according to the selected style”).
Hsiao does not explicitly reference global style. However, Castillo discloses in the same context global style (“Style transfer is an important task in computer graphics in which the style (line stokes, textures, and colors) of a source image is mapped onto that of a target image. Automated style transfer software facilitates the conversion of real-world im ages into the appropriate style to form the background in car toons, simulations, and other rendering”)
Before the effective filing date of this application, it would have been obvious to one of ordinary skill in the art to consider global style.
One of ordinary skill in the art would have motivation to allow conversion to the desired medium. One of ordinary skill in the art would have had a reasonable expectation of success because Hsiao considers a source style, but does not explicitly reference extracting the style.
Claim 7
Hsiao does not explicitly disclose, but Castillo discloses wherein transferring the global style from the source digital image to the target digital image further comprises:
extracting, utilizing an encoder neural network, a global code from the source digital image comprising features corresponding to an overall appearance of the source digital image (Castillo, Section 2: “We first map the style of the source image onto the whole target image”;
extracting, utilizing the encoder neural network, a spatial code from the target digital image corresponding to a geometric layout of the target digital image (Castillo, Section 2.2: “We use an instance-aware semantic segmentation method [11] to generate a mask for each object instance in an image. Our interface enables a user to simply click on a semantic instance, and the image style is transferred to that instance. The in stance semantic segmentation approach is built on a cascaded multi-task network using the loss function”); and
generating, utilizing a generator neural network, the modified digital image by combining the global code of the source digital image with the spatial code of the target digital image (Castillo, Section 2: “a Markov random field (MRF) based model is used to merge the extracted stylized object with the non-stylized background.”).
Before the effective filing date of this application, it would have been obvious to one of ordinary skill in the art to consider global and local code as claimed.
One of ordinary skill in the art would have motivation to limit style transfer to intended objects. One of ordinary skill in the art would have had a reasonable expectation of success because both references consider use of targeted style transfer.
Claim 8
Hsiao discloses a computer-implemented method comprising:
receiving a request to transfer a global style from a source digital image to a target digital image (Hsiao, ¶ 37: “For example, the image Styler can be activated when the camera 105 is launched to stylize an image in real-time during capturing. Alternatively, the portrait Styler can be activated for style transfer of an image album, which is stored in the memory 103 for example, upon request of a user. Similarly, the image Styler can also be used for style transfer of videos; in this mode, it stylizes the background or the foreground of videos in a video album.”);
identifying, in response to receiving the request to transfer, a foreground object within the target digital image, the foreground object comprising a style (Hsiao, ¶ 54: “The style can be selected by user or can be selected or set by the system of the terminal device by default by taking user habit or user preference into consideration according to usage history of the user. In terms of whether the background image or the foreground image will be input to the image transformation network, it can be determined or selected by the user, or can be set by the system.”); and
generating a modified digital image by transferring, utilizing a style transfer neural network (Hsiao, ¶ 39: “The image Styler provided herein can be embodied as a neural network(s).”), the global style from the source digital image to a background of the target digital image while maintaining the style of the foreground object (Hsiao, ¶ 54: “Then the image Styler will transform the background image or the foreground image input thereto according to the selected style”).
Hsiao does not explicitly reference global style. However, Castillo discloses in the same context global style (“Style transfer is an important task in computer graphics in which the style (line stokes, textures, and colors) of a source image is mapped onto that of a target image. Automated style transfer software facilitates the conversion of real-world im ages into the appropriate style to form the background in car toons, simulations, and other rendering”)
Before the effective filing date of this application, it would have been obvious to one of ordinary skill in the art to consider global style.
One of ordinary skill in the art would have motivation to allow conversion to the desired medium. One of ordinary skill in the art would have had a reasonable expectation of success because Hsiao considers a source style, but does not explicitly reference extracting the style.
Claim 10
Hsiao discloses wherein the request to transfer comprises a selection of the foreground object (Hsiao, ¶ 54: “background image or the foreground image…can be determined or selected by the user…only certain part of the image rather than the entire image is sent to be transformed, the transforming speed can be improved.”).
Claim 11
Hsiao discloses further comprising: utilizing a segmentation model to generate an object mask of the foreground object (Hsiao, ¶ 51: “image separation can be implemented via background or foreground extraction. To achieve this, a neural network…is trained”); and utilizing the object mask to avoid transferring the global style to the foreground object (Hsiao, ¶ 51: “a mask is obtained by separating an original image into a background image and a foreground image.”).
Claim 14
Hsiao does not explicitly disclose, but Castillo discloses wherein transferring the global style from the source digital image to the target digital image further comprises:
extracting, utilizing an encoder neural network, a global code from the source digital image comprising features corresponding to an overall appearance of the source digital image (Castillo, Section 2: “We first map the style of the source image onto the whole target image”;
extracting, utilizing the encoder neural network, a spatial code from the target digital image corresponding to a geometric layout of the target digital image (Castillo, Section 2.2: “We use an instance-aware semantic segmentation method [11] to generate a mask for each object instance in an image. Our interface enables a user to simply click on a semantic instance, and the image style is transferred to that instance. The in stance semantic segmentation approach is built on a cascaded multi-task network using the loss function”); and
generating, utilizing a generator neural network, the modified digital image by combining the global code of the source digital image with the spatial code of the target digital image (Castillo, Section 2: “a Markov random field (MRF) based model is used to merge the extracted stylized object with the non-stylized background.”).
Before the effective filing date of this application, it would have been obvious to one of ordinary skill in the art to consider global and local code as claimed.
One of ordinary skill in the art would have motivation to limit style transfer to intended objects. One of ordinary skill in the art would have had a reasonable expectation of success because both references consider use of targeted style transfer.
Claim 15
Hsiao discloses a system comprising: one or more memory devices; and one or more processors coupled to the one or more memory devices, the one or more processors configured to cause the system to:
receive a request to transfer a global style from a source digital image to a target digital image (Hsiao, ¶ 37: “For example, the image Styler can be activated when the camera 105 is launched to stylize an image in real-time during capturing. Alternatively, the portrait Styler can be activated for style transfer of an image album, which is stored in the memory 103 for example, upon request of a user. Similarly, the image Styler can also be used for style transfer of videos; in this mode, it stylizes the background or the foreground of videos in a video album.”);
identify, in response to receiving the request to transfer, a foreground object within the target digital image, the foreground object comprising a style (Hsiao, ¶ 54: “The style can be selected by user or can be selected or set by the system of the terminal device by default by taking user habit or user preference into consideration according to usage history of the user. In terms of whether the background image or the foreground image will be input to the image transformation network, it can be determined or selected by the user, or can be set by the system.”);
utilize a segmentation model (Hsiao, ¶ 51: “image separation can be implemented via background or foreground extraction. To achieve this, a neural network…is trained”) to generate an object mask of the foreground object (Hsiao, ¶ 51: “a mask is obtained by separating an original image into a background image and a foreground image.”); and
generate a modified digital image by transferring, utilizing a style transfer neural network and the object mask, the global style from the source digital image to an entirety of the target digital image albeit without transferring the global style to the foreground object (Hsiao, ¶ 54: “Specifically, the background image or the foreground image can be input to the image Styler, particularly, the image transformation network of the image Styler. As can be seen, here, the input of the image transformation network is the background image or the foreground image rather than the whole image. Then the image Styler will transform the background image or the foreground image input thereto according to the selected style, to obtain the partial stylized image.”).
Hsiao does not explicitly reference global style. However, Castillo discloses in the same context global style (“Style transfer is an important task in computer graphics in which the style (line stokes, textures, and colors) of a source image is mapped onto that of a target image. Automated style transfer software facilitates the conversion of real-world im ages into the appropriate style to form the background in car toons, simulations, and other rendering”)
Before the effective filing date of this application, it would have been obvious to one of ordinary skill in the art to consider global style.
One of ordinary skill in the art would have motivation to allow conversion to the desired medium. One of ordinary skill in the art would have had a reasonable expectation of success because Hsiao considers a source style, but does not explicitly reference extracting the style.
Claim 18
Hsiao does not explicitly disclose, but Castillo discloses wherein transferring the global style from the source digital image to the target digital image further comprises:
extracting, utilizing an encoder neural network, a global code from the source digital image comprising features corresponding to an overall appearance of the source digital image (Castillo, Section 2: “We first map the style of the source image onto the whole target image”;
extracting, utilizing the encoder neural network, a spatial code from the target digital image corresponding to a geometric layout of the target digital image (Castillo, Section 2.2: “We use an instance-aware semantic segmentation method [11] to generate a mask for each object instance in an image. Our interface enables a user to simply click on a semantic instance, and the image style is transferred to that instance. The in stance semantic segmentation approach is built on a cascaded multi-task network using the loss function”); and
generating, utilizing a generator neural network, the modified digital image by combining the global code of the source digital image with the spatial code of the target digital image (Castillo, Section 2: “a Markov random field (MRF) based model is used to merge the extracted stylized object with the non-stylized background.”).
Before the effective filing date of this application, it would have been obvious to one of ordinary skill in the art to consider global and local code as claimed.
One of ordinary skill in the art would have motivation to limit style transfer to intended objects. One of ordinary skill in the art would have had a reasonable expectation of success because both references consider use of targeted style transfer.
Claim(s) 2-3, 16-17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Hsiao (US2021/0142455) in view of Castillo, Targeted Style Transfer Using Instance-Aware Semantic Segmentation in view of He, Mask R-CNN
Claim 2
Hsiao does not explicitly disclose, but He discloses wherein identifying the foreground object within the target digital image comprises utilizing an object detection machine learning model to generate object labels for objects in the target digital image (He, Section 3: “Our definition of Lmask allows the network to generate masks for every class without competition among classes; we rely on the dedicated classification branch to predict the class label”).
PNG
media_image1.png
395
489
media_image1.png
Greyscale
Before the effective filing date of this application, it would have been obvious to one of ordinary skill in the art to consider object segmentation as claimed.
The use is suggested by Hsiao: “To achieve this, a neural network (see Kaiming He Georgia Gkioxari Piotr Dollar Ross Girshick, “Mask R-CNN”, ICCV 2017 for example) is trained to predict a binary mask for each ROI.” (Hsiao, ¶ 51). One of ordinary skill in the art would have had a reasonable expectation of success because of the need to determine the content of the image ROIs determined in Hsiao.
Claim 3
Hsiao discloses wherein identifying the foreground object further comprises selecting the foreground object as an object to not transfer the global style based on an object label generated for the foreground object (Hsiao, ¶ 54: “background image or the foreground image…can be determined or selected by the user…only certain part of the image rather than the entire image is sent to be transformed, the transforming speed can be improved.”)
Claim 16
Hsiao does not explicitly disclose, but He discloses wherein the one or more processors are further configured to cause the system to identify the foreground object within the target digital image by utilizing an object detection machine learning model to generate object labels for objects in the target digital image (He, Section 3: “Our definition of Lmask allows the network to generate masks for every class without competition among classes; we rely on the dedicated classification branch to predict the class label”).
PNG
media_image1.png
395
489
media_image1.png
Greyscale
Before the effective filing date of this application, it would have been obvious to one of ordinary skill in the art to consider object segmentation as claimed.
The use is suggested by Hsiao: “To achieve this, a neural network (see Kaiming He Georgia Gkioxari Piotr Dollar Ross Girshick, “Mask R-CNN”, ICCV 2017 for example) is trained to predict a binary mask for each ROI.” (Hsiao, ¶ 51). One of ordinary skill in the art would have had a reasonable expectation of success because of the need to determine the content of the image ROIs determined in Hsiao.
Claim 17
Hsiao as modified by He discloses wherein the one or more processors are further configured to cause the system to identify the foreground object by selecting the foreground object as an object to not transfer the global style based on an object label generated for the foreground object (Hsiao, ¶¶ 51-52: “To obtain the binary mask, ROI (that is, the foreground image) and other pixel points (that is, the background image) can be determined as well. Then the content of the foreground and the foreground can be extracted respectively based on the ROI determined. The resulted mask as shown in FIG. 7 will be used later to fuse a partial stylized image and partial original image, or to fuse an intermediate stylized image and partial original image.”)
Claim(s) 4-6, 9, 12-13, 19-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Hsiao (US2021/0142455) in view of Castillo, Targeted Style Transfer Using Instance-Aware Semantic Segmentation in view of Tsai, Deep Image Harmonization
Claim 4
Hsiao does not explicitly disclose, but Tsai discloses wherein generating the modified digital image further comprises harmonizing the foreground object with the background of the modified digital image having the global style of the source digital image (Tsai, Section 3: “Given a composite image and a foreground mask as the input, our model outputs a harmonized image by adjusting fore ground appearances while retaining the background region.”).
Before the effective filing date of this application, it would have been obvious to one of ordinary skill in the art to use harmonization as claimed.
One of ordinary skill in the art would have motivation to use harmonization because “the appearances of the extracted foreground region may not be consistent with the new background, making the composite image un realistic. Therefore, it is essential to adjust the appearances of the foreground region to make it compatible with the new background” (Tsai, Section 1). One of ordinary skill in the art would have had a reasonable expectation of success because Hsiao would also present this problem when creating composite images.
Claim 5
Hsiao does not explicitly disclose, but Tsai discloses wherein harmonizing the foreground object with the background of the modified digital image having the global style of the source digital image comprises utilizing a harmonization neural network to adjust one or more of color qualities, contrast, or lighting conditions of one or more of the background or the foreground object (Tsai, Section 3: “With the incorporated scene parsing model, our network can learn the color distribution of certain se mantic categories, e.g., the skin color on human or the sky like colors. In addition, the learned background semantics can help identify which region to match for better fore ground adjustment.”)
Before the effective filing date of this application, it would have been obvious to one of ordinary skill in the art to use harmonization as claimed.
One of ordinary skill in the art would have motivation to use harmonization because “the appearances of the extracted foreground region may not be consistent with the new background, making the composite image un realistic. Therefore, it is essential to adjust the appearances of the foreground region to make it compatible with the new background” (Tsai, Section 1). One of ordinary skill in the art would have had a reasonable expectation of success because Hsiao would also present this problem when creating composite images.
Claim 6
Hsiao does not explicitly disclose, but Tsai discloses wherein utilizing the harmonization neural network comprises utilizing the harmonization neural network to iteratively adjust one or more of color qualities, contrast, or lighting conditions of one or more of the background or the foreground object (Tsai, Section 3: “With the incorporated scene parsing model, our network can learn the color distribution of certain se mantic categories, e.g., the skin color on human or the sky like colors. In addition, the learned background semantics can help identify which region to match for better fore ground adjustment.”)
Before the effective filing date of this application, it would have been obvious to one of ordinary skill in the art to use harmonization as claimed.
One of ordinary skill in the art would have motivation to use harmonization because “the appearances of the extracted foreground region may not be consistent with the new background, making the composite image un realistic. Therefore, it is essential to adjust the appearances of the foreground region to make it compatible with the new background” (Tsai, Section 1). One of ordinary skill in the art would have had a reasonable expectation of success because Hsiao would also present this problem when creating composite images.
Claim 9
Hsiao does not explicitly disclose, but Tsai discloses wherein the global style comprises a landscape texture within the source digital image (Tsai, Section 3: “With the incorporated scene parsing model, our network can learn the color distribution of certain se mantic categories, e.g., the skin color on human or the sky like colors. In addition, the learned background semantics can help identify which region to match for better fore ground adjustment.”)
Before the effective filing date of this application, it would have been obvious to one of ordinary skill in the art to use harmonization as claimed.
One of ordinary skill in the art would have motivation to use harmonization because “the appearances of the extracted foreground region may not be consistent with the new background, making the composite image un realistic. Therefore, it is essential to adjust the appearances of the foreground region to make it compatible with the new background” (Tsai, Section 1). One of ordinary skill in the art would have had a reasonable expectation of success because Hsiao would also present this problem when creating composite images.
Claim 12
Hsiao does not explicitly disclose, but Tsai discloses wherein generating the modified digital image further comprises harmonizing the foreground object with the background of the modified digital image having the global style of the source digital image (Tsai, Section 3: “Given a composite image and a foreground mask as the input, our model outputs a harmonized image by adjusting foreground appearances while retaining the background region.”).
Before the effective filing date of this application, it would have been obvious to one of ordinary skill in the art to use harmonization as claimed.
One of ordinary skill in the art would have motivation to use harmonization because “the appearances of the extracted foreground region may not be consistent with the new background, making the composite image un realistic. Therefore, it is essential to adjust the appearances of the foreground region to make it compatible with the new background” (Tsai, Section 1). One of ordinary skill in the art would have had a reasonable expectation of success because Hsiao would also present this problem when creating composite images.
Claim 13
Hsiao does not explicitly disclose, but Tsai discloses wherein harmonizing the foreground object with the background of the modified digital image further comprises:
dividing the modified digital image into image patches (Tsai, Section 2: “However, these image editing pipelines may suffer from missing semantic information in the finer level during re construction, and such semantics are important cues for understanding image contents. Unlike previous methods that do not explicitly use semantics, we incorporate an additional model to predict pixel-wise scene parsing”);
generating patch embeddings from the image patches (Tsai, Section 2: “propagate this information to the harmonization model”); and
processing the patch embeddings, utilizing a transformer neural network, to adjust one or more of color qualities, contrast, or lighting conditions of one or more of the background or the foreground object (Tsai, Section 3: “The composite image is then generated by the color transfer method mentioned”).
Before the effective filing date of this application, it would have been obvious to one of ordinary skill in the art to use harmonization as claimed.
One of ordinary skill in the art would have motivation to use harmonization because “the appearances of the extracted foreground region may not be consistent with the new background, making the composite image un realistic. Therefore, it is essential to adjust the appearances of the foreground region to make it compatible with the new background” (Tsai, Section 1). One of ordinary skill in the art would have had a reasonable expectation of success because Hsiao would also present this problem when creating composite images.
Claim 19
Hsiao does not explicitly disclose, but Tsai discloses wherein the one or more processors are further configured to cause the system to generate the modified digital image by harmonizing, utilizing a harmonization neural network, the foreground object with a background of the modified digital image to which the global style has been transferred (Tsai, Section 3: “Given a composite image and a foreground mask as the input, our model outputs a harmonized image by adjusting fore ground appearances while retaining the background region.”).
Before the effective filing date of this application, it would have been obvious to one of ordinary skill in the art to use harmonization as claimed.
One of ordinary skill in the art would have motivation to use harmonization because “the appearances of the extracted foreground region may not be consistent with the new background, making the composite image un realistic. Therefore, it is essential to adjust the appearances of the foreground region to make it compatible with the new background” (Tsai, Section 1). One of ordinary skill in the art would have had a reasonable expectation of success because Hsiao would also present this problem when creating composite images.
Claim 20
Hsiao does not explicitly disclose, but Tsai discloses wherein harmonizing the foreground object with a background of the modified digital image having the global style of the source digital image comprises utilizing a harmonization neural network to adjust one or more of color qualities, contrast, or lighting conditions of one or more of the background or the foreground object (Tsai, Section 3: “With the incorporated scene parsing model, our network can learn the color distribution of certain se mantic categories, e.g., the skin color on human or the sky like colors. In addition, the learned background semantics can help identify which region to match for better fore ground adjustment.”)
Before the effective filing date of this application, it would have been obvious to one of ordinary skill in the art to use harmonization as claimed.
One of ordinary skill in the art would have motivation to use harmonization because “the appearances of the extracted foreground region may not be consistent with the new background, making the composite image un realistic. Therefore, it is essential to adjust the appearances of the foreground region to make it compatible with the new background” (Tsai, Section 1). One of ordinary skill in the art would have had a reasonable expectation of success because Hsiao would also present this problem when creating composite images.
New Prior Art
Additional prior art relevant to Applicant’s disclosure but not relied upon:
Adamson (US Patent 11,636,649) considers style transfer considering foreground and background objects:
PNG
media_image2.png
308
579
media_image2.png
Greyscale
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to RYAN M GRAY whose telephone number is (571)272-4582. The examiner can normally be reached on Monday through Friday, 9:00am-5:30pm (EST).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kee Tung can be reached on (571)272-7794. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see https://ppair-my.uspto.gov/pair/PrivatePair. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/RYAN M GRAY/Primary Examiner, Art Unit 2611
Prosecution Insights