DETAILED ACTION
The action is in response to the original filing on October 11, 2024 and the Remarks and Amendments filed on June 15, 2026. Claims 1-20 are pending and have been considered below. Claims 1, 5-8, 14-15, and 18 are amended accordingly.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-8, 10-12, 15, & 20 are rejected under 35 U.S.C. 103 as being unpatentable over Krishna Singh (Pat. Pub. GB-2628694-A, herein after “Singh”) in view of Lisa Figueroa (Pat. Pub. CN-118072309-A, herein after “Figueroa”) and Zhihong Ding (Pat. Pub. US-20210027497-A1, herein after “Ding”).
In regard to claim 1, Singh teaches [a] method comprising:
detecting, utilizing a shape detection neural network, one or more shapes depicted in a digital image “the scene-based image editing system pre-processes a digital image in anticipation of user input for manipulating one or more semantic areas of a digital image, such as user input for moving or deleting one or more objects within the digital image” (Singh, ¶ [0099]) where the semantic areas are objects, which are shapes;
identifying, from the one or more shapes depicted in the digital image “the scene-based image editing system retrieves a set of object attributes for the object (e.g., size, shape, or color) from the corresponding semantic scene graph and presents the set of object attributes for display in association with the object” (Singh, ¶ [0105]), a background region comprising background pixels depicted in the digital image “the background comprises the furthest element in the image (often a semantic area like the sky, ground, water, etc.)” (Singh, ¶ [0133]);
determining, by applying a solid fill algorithm to color data extracted from the background region “the scene-based image editing system 106 generates an object mask for each portrayed object and/or generates a content fill for filling in the background behind each portrayed object” (Singh, ¶ [0133]) where a content fill, which can be a solid fill, is generated for the background of portrayed objects, that the background region depicts solid fill pixels based on a coverage percentage of a color value within the background region satisfying a solid fill threshold “a content fill includes a set of pixels generated to fill a hole (e.g., a content void) that remains after (or if) a set of pixels (e.g., a set of pixels portraying an object) has been removed from or moved within a digital image. In some cases, a content fill corresponds to a background of a digital image. To illustrate, in some implementations, a content fill includes a set of pixels generated to blend in with a portion of a background proximate to an object that could be moved/removed” (Singh, ¶ [0171]) where the machine learning model may recognize that a background may be filled, additionally, “the scene-based image editing system 106 inpaints a hole with foreground pixels and background pixels. Indeed, in one or more embodiments, the scene-based image editing system 106 utilizes an inpainting neural network to generate foreground pixels and background pixels for the resulting hole and utilizes the generated pixels to inpaint the hole, resulting in a modified digital image (e.g., an inpainted digital image) where the object 3202 has been removed and the corresponding portion of the digital image 3204 has been filled in” (Singh, ¶ [0492]) where the artificial intelligence is used to fill in the object hole based on the background content, which can be solid or textured based on the surrounding background;
determining a prominent color for the solid fill pixels based on determining that the background region depicts solid fill pixels “the scene-based image editing system 106 generates an object mask for each portrayed object and/or generates a content fill for filling in the background behind each portrayed object” (Singh, ¶ [0133]), for example “an object includes a collection of buildings that make up a skyline. In some instances, an object more broadly includes a (portion of a) foreground or other element(s) depicted in the digital image 4002 as distinguished from a background” (Singh, ¶ [0532]) where the background is determined to be the area around an identified object, may be a solid color or a scene, and is filled based on the object mask; and
extracting, from the digital image, a background image layer corresponding to the background region, the background image layer comprising pixels with values indicated by the prominent color “a content fill corresponds to a background of a digital image. To illustrate, in some implementations, a content fill includes a set of pixels generated to blend in with a portion of a background proximate to an object that could be moved/removed” (Singh, ¶ [0171]) where a content fill includes a set of pixels that blend in, or are similar to a particular color, and can be moved, removed, or extracted as a layer.
Singh does not explicitly teach determining a prominent color for the solid fill pixels.
Figueroa teaches determining a prominent color for the solid fill pixels “when the probability output of a particular attribute satisfies a positive attribute threshold (e.g., a positive probability, a probability greater than 0.5), the scene-based image editing system 106 identifies the attribute as a positive attribute” (Figueroa, Page 46) where the positive attributes of an object are features present above a threshold, such as object color or shape.
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of identifying objects and a background using artificial intelligence taught by Singh with the method of identifying a prominent color taught by Figueroa to produce an artificial intelligence capable of identifying objects from an image, including their prominent color. The motivation to do so would be to produce an AI tool capable of identifying what prominent color an object is made of.
Singh in view of Figueroa fail to efficiently teach a coverage percentage of a color value within the background region satisfying a solid fill threshold with a background image layer corresponding to the background region, the background image layer comprising pixels with values indicated by the prominent color.
Ding teaches a coverage percentage of a color value within the background region satisfying a solid fill threshold “the color classification system 106 also generates an object mask of the detected query object 502 to further isolate the pixels of the detected query object 502 from the background pixels of the image, which commonly are of a different color” (Ding, ¶ [0103]) where the background can be masked apart from the foreground objects and “[t]he color classification system can then calculate a color-matching score for the object based on the number or percentage of pixels within the color similarity region (i.e., valid pixels) and pixels outside of the color similarity region (i.e., invalid pixels)” (Ding, ¶ [0033]) and “color classification system 106 applies a color similarity threshold to each mapped point” (Ding, ¶ [0072]) with a background image layer corresponding to the background region, the background image layer comprising pixels with values indicated by the prominent color “the series of acts 500 can include the act 532 of the color classification system 106 analyzing the percentage of valid pixels in the detected query object 502” (Ding, ¶ [0117]) where an identified object may be analyzed for pixels that correspond with the desired color, and may then be edited as desired which is read as an object comprising pixels indicated by a prominent color.
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of identifying objects and a background using artificial intelligence and identifying a prominent color taught by Singh in view of Figueroa with the method of detecting a coverage percentage based on a threshold pixel color and editing the respective detected area taught by Ding to produce an artificial intelligence capable of identifying objects from an image, including their prominent color that corresponds to an area within a set threshold. The motivation to do so would be to produce an AI tool capable of identifying what prominent color an object is made of and editing the area based on the detected region.
In regard to claim 2, Singh in view of Figueroa and Ding teach [t]he method of claim 1, wherein detecting the one or more shapes comprises detecting a partially occluded shape depicted in the digital image utilizing the shape detection neural network “the scene-based image editing system 106 detects occluders/detractors covering part of a human portrayed within a digital image and removes the occluders/detractors” (Singh, ¶ [0621]). The neural network in the editing system may detect and remove objects blocking other objects.
In regard to claim 3, Singh in view of Figueroa and Ding teach [t]he method of claim 1, wherein detecting the one or more shapes comprises utilizing the shape detection neural network to generate bounding boxes for the one or more shapes depicted in the digital image “the detection-masking neural network 300 indicates the detected objects utilizing approximate boundaries (e.g., bounding boxes 319, 321, and 323)” (Singh, ¶ [0159]).
In regard to claim 4, Singh in view of Figueroa and Ding teach [t]he method of claim 1, wherein identifying the background region comprises determining a set of pixels farthest from a viewpoint of a user viewing the digital image via a client device by removing the one or more shapes, detected objects, and detected text from the digital image “the scene-based image editing system 106 creates the completed background 714 as part of the digital image 706. For instance, in one or more embodiments, the scene-based image editing system 106 positions the content fills 712 behind their corresponding object (e.g., as a separate image layer) in the digital image 706” (Singh, ¶ [0229]) where the object and mask layers are placed on top of the background layer. In further explanation, “the background for an object can comprise part of another object and part of the furthest element in the image” (Singh, ¶ [0133]) and “upon deleting the object 208c from the digital image 206, the scene-based image editing system 106 automatically reveals background pixels that have been positioned in place of the object 208c” (Singh, ¶ [0136]) where the object is not limited from being a shape, human, or text.
In regard to claim 5, Singh in view of Figueroa and Ding teach [t]he method of claim 1, wherein determining that the background region depicts solid fill pixels comprises utilizing the solid fill algorithm to determine that at least a threshold percentage of pixels within the background region depict values within a range corresponding to a common color label “the scene-based image editing system 106 also generates a semantic boundary. In particular, the semantic boundary includes an indication of the shape or the form of pixels sharing a common semantic classification. For instance, a semantic boundary indicates an area of pixels sharing a pant semantic classification” (Singh, ¶ [0600]) where boundaries are based on areas with similar color or patterns and a semantic classification is read as a color label, additionally “a region includes an area of a digital image where all pixels are of the same color or brightness” (Singh, ¶ [0454]) which is also used to identify features based on how much they stand out, and may also be utilized as a “heatmap”.
In regard to claim 6, Singh in view of Figueroa and Ding teach [t]he method of claim 1
Singh does not explicitly teach wherein determining the prominent color for the solid fill pixels comprises using a prominent color algorithm to:
identify the prominent color as a color that satisfies a coverage threshold as part of the solid fill algorithm; and
validate the prominent color by comparing the prominent color with a converted version of the prominent color binned according to a hue value.
Figueroa teaches wherein determining the prominent color for the solid fill pixels comprises using a prominent color algorithm to:
identify the prominent color as a color that satisfies a coverage threshold as part of the solid fill algorithm “when the probability output of a particular attribute satisfies a positive attribute threshold (e.g., a positive probability, a probability greater than 0.5), the scene-based image editing system 106 identifies the attribute as a positive attribute” (Figueroa, Page 46) where the positive attributes of an object are features present above a threshold, such as object color or shape; and
validate the prominent color by comparing the prominent color with a converted version of the prominent color binned according to a hue value “The "chair") includes a positive attribute tag 1728 and a negative attribute tag 1730. In some embodiments, the scene-based image editing system 106 utilizes local image-object feature vectors … Then, as shown in FIG. 17, the classifier neural network 1732 (flock) generates a positive attribute tag 1728 (e.g., red, bright red, clean, giant, wood), and also generates a negative attribute tag 1730 (e.g., blue, fill, pattern, multicolor)” (Figueroa, Page 50) where the positive attribute tag comprises features of an object, in contrast to a negative attribute tag, which comprises features lacking in an object.
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of identifying objects and using artificial intelligence to identify their traits such as color taught by Singh with the method of identifying a positive attribute to determine if an object contains a certain feature such as color or shape taught by Figueroa to produce an artificial intelligence capable of identifying objects from an image, including their traits such as shape and color. The motivation to do so would be to produce an AI tool capable of identifying what prominent color an object is made of.
In regard to claim 7, Singh in view of Figueroa and Ding teach [t]he method of claim 1, further comprising generating, for the background image layer, a modified background region by filling the background region with the prominent color “the infill segmentation map 5908 assigns labels to the region determined to be inpainted. Accordingly, various models utilize the infill segmentation map 5908 to determine how to infill the region indicated to be inpainted” (Singh, ¶ [0680]) where the region may be an object or part of an object or mask. The object or mask may be a layer on the image, and the infill function can modify, or inpaint a segment of the image by replacing it with a color, pattern, or texture.
Figueroa additionally teaches generating, for the background image layer, a modified background region by filling the background region with the prominent color “scene-based image editing system 106 generates an object mask for each depicted object and/or generates content fill for filling the background behind each depicted object” (Figueroa, Page 11) where a modified background can be read as a filled background.
In regard to claim 8, Singh in view of Figueroa and Ding teach [a] system comprising:
a memory component “The memory 9504” (Singh, ¶ [0941])”; and
one or more processing devices coupled to the memory component “The memory 9504 may be used for storing data, metadata, and programs for execution by the processor(s)” (Singh, ¶ [0941]), the one or more processing devices to perform operations comprising:
detecting, utilizing a shape detection neural network “the scene-based image editing system utilizes a segmentation neural network to generate, for each object portrayed in a digital image, an object mask” (Singh, ¶ [0099]) where detecting objects with a segmentation neural network is taught, one or more shapes depicted in a digital image “the scene-based image editing system pre-processes a digital image in anticipation of user input for manipulating one or more semantic areas of a digital image, such as user input for moving or deleting one or more objects within the digital image” (¶ [0099]) where the semantic areas are objects, which are shapes;
identifying a shape comprising solid fill pixels from the one or more shapes depicted in the digital image by: “FIG. 46 illustrates a number of infill semantic maps option 4600 for selecting a number of infill semantic map outputs” (Singh, ¶ [0591]) where the object may be a solid pixel color from being infilled or may be read in the digital image as solid fill pixels
applying a solid fill algorithm to color data extracted from the shape “the scene-based image editing system utilizes a hole-filing model to generate. for each object (e.g., for each corresponding object mask), a content fill (e.g., an inpainting segment)” (Singh, ¶ [0099]) where eh objects, or shapes, can be inpainted, or filled, based on their pixel information; and
determining that a coverage percentage of a color value within the shape satisfies a solid fill threshold;
extracting, from the digital image, an image layer for the shape depicted in the digital image and comprising pixels with values indicated by the prominent color “In response to selection of the query selection option 5202, the scene-based image editing system 106 provides for display in the user interface a masked digital image 5208 based on the indication in the query selection option 5202. In this instance, FIG. 52A shows the masked digital image 5208 with all the apparel and the background in the digital image 5200 masked” (Singh, ¶ [0616]) where the user can select the object and view it as a separate layer extracted from the background.
Singh does not explicitly teach determining, based on determining that the shape comprises solid fill pixels, a prominent color for the solid fill pixels.
Figueroa teaches determining, based on determining that the shape comprises solid fill pixels, a prominent color for the solid fill pixels “when the probability output of a particular attribute satisfies a positive attribute threshold (e.g., a positive probability, a probability greater than 0.5), the scene-based image editing system 106 identifies the attribute as a positive attribute” (Figueroa, Page 46) where the positive attributes of an object are features present above a threshold, such as object color or shape.
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of identifying objects and a background using artificial intelligence taught by Singh with the method of identifying a prominent color taught by Figueroa to produce an artificial intelligence capable of identifying objects from an image, including their prominent color. The motivation to do so would be to produce an AI tool capable of identifying what prominent color an object is made of.
Singh in view of Figueroa fail to effectively teach determining that a coverage percentage of a color value within the shape satisfies a solid fill threshold;
Ding teaches determining that a coverage percentage of a color value within the shape satisfies a solid fill threshold “The term “color-matching threshold” refers to a condition or value where a pixel is considered a match to a color (e.g., a target color). For instance, the color-matching threshold indicates when the color value for a pixel is similar enough to a color in multidimensional color space to match the color” (Ding, ¶ [0050]) where a similarity requirement must be met for the color match to meet the threshold and “ The color classification system can then calculate a color-matching score for the object based on the number or percentage of pixels within the color similarity region” (Ding, ¶ [0033]) where the similarity region must have a percentage of pixels that are similar;
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of identifying objects and a background using artificial intelligence and identifying a prominent color taught by Singh in view of Figueroa with the method of detecting a coverage percentage based on a threshold pixel color taught by Ding to produce an artificial intelligence capable of identifying objects from an image, including their prominent color that corresponds to an area within a set threshold. The motivation to do so would be to produce an AI tool capable of identifying what prominent color an object is made of and editing the area based on the detected region.
In regard to claim 10, Singh in view of Figueroa and Ding teach [t]he system of claim 8, wherein the operations further comprise determining that the shape depicts solid fill pixels “the scene-based image editing system 106 also generates a semantic boundary. In particular, the semantic boundary includes an indication of the shape or the form of pixels sharing a common semantic classification. For instance, a semantic boundary indicates an area of pixels sharing a pant semantic classification” (¶ [0600]) where boundaries are based on areas with similar color or patterns, additionally “a region includes an area of a digital image where all pixels are of the same color or brightness” (¶ [0454]) which is also used to identify features based on how much they stand out, and may also be utilized as a “heatmap”.
Singh does not explicitly teach determining that at least a threshold percentage of pixels within the shape have values within a range corresponding to a color label.
Figueroa teaches determining that at least a threshold percentage of pixels within the shape have values within a range corresponding to a color label “when the probability output of a particular attribute satisfies a positive attribute threshold (e.g., a positive probability, a probability greater than 0.5), the scene-based image editing system 106 identifies the attribute as a positive attribute” (Figueroa, Page 46) where the positive attributes of an object are features present above a threshold, such as object color or shape.
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of identifying objects and using artificial intelligence to identify their traits such as color taught by Singh with the method of identifying a threshold to determine if an object contains a certain feature such as color or shape taught by Figueroa to produce an artificial intelligence capable of identifying objects from an image, including their traits such as shape and color. The motivation to do so would be to produce an AI tool capable of identifying what prominent color an object is made of.
In regard to claim 11, Singh in view of Figueroa and Ding teach [t]he system of claim 8, wherein detecting the one or more shapes comprises using the shape detection neural network to detect at least one occluded shape depicted in the digital image “the scene-based image editing system 106 detects occluders/detractors covering part of a human portrayed within a digital image and removes the occluders/detractors” (¶ [0621]) where the object may be a human, or another shape/feature.
In regard to claim 12, Singh in view of Figueroa and Ding teach [t]he system of claim 8, wherein the operations further comprise training the shape detection neural network using training data comprising template images depicting occluded geometric shapes “the scene-based image editing system 106 trains the heatmap network 2702 on whole images, including digital images where there are no distracting objects and digital images that portray main subject objects mid distracting objects” (Singh, ¶ [0455]) where the neural network is trained using digital images that can portray distracting objects, additionally, it is noted that the features of the disclosure may be tuned based on inputs (Singh, ¶ [0146]).
Claim 14 is rejected under 35 U.S.C. 103 as being unpatentable over Singh in view of Ding.
In regard to claim 14, Singh teaches [a] non-transitory computer readable medium storing instructions which, when executed by a processing device, cause the processing device to perform operations “upon reaching various computer system components, program code means in the form of computer-executable instructions or data structures can be transferred automatically from transmission media to non-transitory computer-readable storage media (devices) (or vice versa)” (Singh, ¶ [0933]) comprising:
generating a set of logical segments from a digital image by “the scene based image editing system 106 utilizes a segmentation neural network to identify and classify objects portrayed in a digital image (as discussed above with reference to FIG. 3)” (Singh, ¶ [0297]):
detecting object segments depicted in the digital image using an object segmentation model “object detection machine learning model 308 and the object segmentation machine learning model 310 are separate machine learning models for processing objects within target and/or source digital images. FIG. 3 illustrates the encoder 302, detection heads 304, and the masking head 306 as a single model for detecting and segmenting objects of a digital image” (Singh, ¶ [0154]) where there is a model for detecting objects and a model for segmenting objects, additionally, a model may be capable of both detecting and segmenting digital image objects;
extracting text segments depicted in the digital image using a text segmentation model “a semantic classification includes a classification for an object of the digital image 4900. The scene-based image editing system 106 can include these classifications in a variety of ways, including numbers, text, textures, or other labels. In relation to FIG. 49A, the image editing system utilizes colors to indicate different classifications of pixels” (Singh, ¶ [0598]) where the semantic classification may be text, and the process of extracting the text from the image may be performed; and
detecting shape segments depicted in the digital image using a shape segmentation model “in some embodiments, based on determining that the class of one or more of the identified objects comprises a human or person, the scene-based image editing system 106 utilizes a special human object mask neural network to generate an object mask, such as the object mask 324 shown in FIG. 3” (Singh, ¶ [0166]) where the object’s class may influence the detection or segmentation, such as if the shape is of a person;
determining, based on the set of logical segments, a prominent color for a solid fill region of the digital image by: “a region of the digital image 4900 that corresponds with the legs of a human, the scene-based image editing system 106 utilizes a semantic classification with a bright pink shade. In other instances, for a region of the digital image 4900 that corresponds with the arms of a human, the semantic classification includes a yellow shade for one arm and a blue shade for the shirt. Alternatively, the scene-based image editing system 106 can utilize various shades of grey to indicate semantic classifications” (Singh, ¶ [0598]) where a shape, text, object, etc. may be segmented from the digital image, and a shade that stands out, or a prominent color, are determined.
applying a solid fill algorithm to color data extracted from the solid fill region “the scene-based image editing system utilizes a hole-filing model to generate. for each object (e.g., for each corresponding object mask), a content fill (e.g., an inpainting segment)” (Singh, ¶ [0099]) where objects, or shapes, can be inpainted, or filled, based on their pixel information
generating a modified solid fill region by filling the solid fill region with the prominent color “a region of the digital image 4900 that corresponds with the legs of a human, the scene-based image editing system 106 utilizes a semantic classification with a bright pink shade. In other instances, for a region of the digital image 4900 that corresponds with the arms of a human, the semantic classification includes a yellow shade for one arm and a blue shade for the shirt. Alternatively, the scene-based image editing system 106 can utilize various shades of grey to indicate semantic classifications” (Singh, ¶ [0598]) where the semantic classification is able to discern a fill region and prominent color by assessing the object and its appropriate color, or prominent color; and
generating, from the modified solid fill region, a set of image layers corresponding to the set of logical segments “the scene based image editing system 106 utilizes a segmentation neural network to identify and classify objects portrayed in a digital image (as discussed above with reference to FIG. 3)” (Singh, ¶ [0297]) where segmenting objects is read as logical segments and “the scene-based image editing system 106 positions the content fills 712 behind their corresponding object (e.g., as a separate image layer) in the digital image 706” (Singh, ¶ [0229]) where each object’s content fill is a separate layer.
Singh fails to explicitly teach determining that a coverage percentage of a color value within the solid fill region satisfies a solid fill threshold;
Ding teaches a coverage percentage of a color value within the solid fill region region satisfying a solid fill threshold “[t]he color classification system can then calculate a color-matching score for the object based on the number or percentage of pixels within the color similarity region (i.e., valid pixels) and pixels outside of the color similarity region (i.e., invalid pixels)” (Ding, ¶ [0033]) where the color similarity region can be read as the fill region given the color of the region muse be similar to a given percent and “color classification system 106 applies a color similarity threshold to each mapped point” (Ding, ¶ [0072]).
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of identifying objects and a background using artificial intelligence taught by Singh with the use of a threshold for similarity and a color matching score taught by Ding to produce an artificial intelligence capable of identifying objects from an image, including their prominent color based on a percentage. The motivation to do so would be to produce an AI tool capable of identifying what prominent color an object is made of.
In regard to claim 15, Singh in view of Ding teach [t]he non-transitory computer readable medium of claim 14, wherein detecting the shape segments depicted in the digital image comprises using a shape detection neural network trained to detect occluded shapes depicted in digital images based on a custom dataset “the scene-based image editing system 106 provides the progress indicator 3012 while utilizing a distractor detection neural network to identify and classify distracting objects within the digital image 3006” (Singh, ¶ [0470]) where object distractors can cover, block, or occlude an object to be detected by the neural network.
In regard to claim 16, Singh in view of Ding teach [t]he non-transitory computer readable medium of claim 14, wherein the operations further comprise:
receiving, from a client device, a selection of an image layer corresponding to the modified solid fill region from among the set of image layers corresponding to the set of logical segments “in some embodiments, upon determining that an object has been selected for modification, the scene-based image editing system retrieves characteristics of the object from the semantic scene graph to facilitate the modification” (Singh, ¶ [0104]) where the user may select an object that has been segmented, or separated for modification; and
modifying the image layer in response to a user interaction with the client device “in some cases, the scene-based image editing system detects a user interaction selecting an object portrayed in a digital image for modification. The scene-based image editing system references the semantic scene graph previously generated for the digital image to identify a relationship between that object and one or more other objects portrayed in the digital image. Based on the identified relationships, the scene-based image editing system also targets the one or more related objects for the modification” (Singh, ¶ [0107]) where various forms of modification can be performed on the selected image segment.
In regard to claim 17, Singh in view of Ding teach [t]he non-transitory computer readable medium of claim 14, wherein the operations further comprise identifying, from the set of logical segments, a background region comprising background pixels depicted in the digital image “the scene based image editing system 106 utilizes a segmentation neural network to identify and classify objects portrayed in a digital image (as discussed above with reference to FIG. 3)” (Singh, ¶ [0297]) where segmentation is taught and “Background refers to what is behind an object in an image. Thus, when a first object is positioned in front of a second object, the second object forms at least part of the background for the first object. Alternatively, the background comprises the furthest element in the image (often a semantic area like the sky, ground, water, etc.)” (Singh, ¶ [0133]) where the background can be identified in contrast to a segmented object, or group of objects.
In regard to claim 18, Singh in view of Ding teach [t]he non-transitory computer readable medium of claim 17, wherein the operations further comprise determining that the background region comprises the solid fill region of the digital image by determining that a coverage percentage of a color value within the background region satisfies the solid fill threshold “the color classification system 106 also generates an object mask of the detected query object 502 to further isolate the pixels of the detected query object 502 from the background pixels of the image, which commonly are of a different color” (Ding, ¶ [0103]) where the background can be masked apart from the foreground objects and “[t]he color classification system can then calculate a color-matching score for the object based on the number or percentage of pixels within the color similarity region (i.e., valid pixels) and pixels outside of the color similarity region (i.e., invalid pixels)” (Ding, ¶ [0033]) and “color classification system 106 applies a color similarity threshold to each mapped point” (Ding, ¶ [0072]).
In regard to claim 20, Singh in view of Ding teach [t]he non-transitory computer readable medium of claim 14, wherein generating the set of image layers comprises:
generating a background layer for the modified solid fill region “the scene-based image editing system 106 creates the completed background 714 as part of the digital image 706. For instance, in one or more embodiments, the scene-based image editing system 106 positions the content fills 712 behind their corresponding object (e.g., as a separate image layer) in the digital image 706” (Singh, ¶ [0229]);
generating object layers for the object segments “the scene-based image editing system 106 determines that the user interaction selects a distinct semantic area (e.g., the object 208c) rather than the particular underlying pixels or image layers with which the user interacted” (Singh, ¶ [0134]) where distinct semantic areas or objects are detected and can be distinguished as layers for later editing;
generating text layers for the text segments “a semantic classification includes a classification for an object of the digital image 4900. The scene-based image editing system 106 can include these classifications in a variety of ways, including numbers, text, textures, or other labels. In relation to FIG. 49A, the image editing system utilizes colors to indicate different classifications of pixels” (Singh, ¶ [0598]) where the semantic classification may be text, and may have its own layer as an object; and
generating shape layers for the shape segments “the scene-based image editing system retrieves a set of object attributes for the object (e.g., size, shape, or color) from the corresponding semantic scene graph and presents the set of object attributes for display in association with the object” (Singh, ¶ [0105]), also “the semantic boundary includes an indication of the shape or the form of pixels sharing a common semantic classification” (Singh, ¶ [0600]) where the semantic classification may be shape, and may have its own layer as an object.
Claim 9 is rejected under 35 U.S.C. 103 as being unpatentable over Singh in view Figueroa, Ding, and further in view of Mark Kawano (Pat. Pub. US-20090204913-A1, herein after “Kawano”).
In regard to claim 9, Singh in view of Figueroa and Ding teach [t]he system of claim 8, wherein determining the prominent color comprises:
determining, based on converting a color value of the solid fill pixels from a first color space to a second color space, a color bin corresponding to a converted color value of the solid fill pixels “in one or more embodiments the scene-based image editing system 106 provides an option to indicate the color shade and texture of apparel to replace within the digital image 5200” (Singh, ¶ [0619]) where colors may be selected or automatically used to change the shade of an object;
Singh does not explicitly teach determining a color bin value associated with the color bin in the second color space.
Kawano teaches determining a color bin value associated with the color bin in the second color space “if the color values of the colors in the color bin 220 range from 120-179, then the slider 315 can be used to change the hue of portions of the image 130 that are displayed according to color values 120-179 because slider 315 in the expanded view of the control panel 205 corresponds to color bin 220 in the compact view of the control panel 205” (Kawano, ¶ [0042]) where the color bin has a value for changing an object’s hue.
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of identifying objects and using artificial intelligence to identify their traits such as color, and to modify their traits as desired taught by Singh with the method of using color bins to combine like colors and change object hues using the bins taught Kawano by to have an artificial intelligence capable of identifying objects from an image, including their traits such as shape and color, and changing their color with respect to given color groups. The motivation to do so would be to produce an AI tool capable of identifying what prominent color an object is made of, and alter it based on the provided color bins.
Claim 13 is rejected under 35 U.S.C. 103 as being unpatentable over Singh in view of Figueroa, Ding, and further in view of Joshua Abbott (Pat. Pub. US-20240281988-A1, herein after “Abbott”).
In regard to claim 13, Singh in view of Figueroa and Ding teach [t]he system of claim 12
Singh does not explicitly teach wherein the operations further comprise determining the training data by:
selecting, from an image database, template images depicting one or more of rectangles, ellipses, or triangles that are occluded by other objects depicted in the template images and;
determining ground truth shapes depicted in the template images.
Abbott teaches selecting, from an image database, template images depicting one or more of rectangles, ellipses, or triangles that are occluded by other objects depicted in the template images “Regardless of the implementation, input training data for the machine learning model(s) 408 (e.g., input data 406) may be generated from sensor data using the techniques for operating the machine learning model(s) 408 described herein. For example, a scene may be observed with one or more sensors (e.g., one or more LiDAR sensor(s), camera(s), etc.), and a projected representation of the sensor data (e.g., a projection image) may be generated with the same size and dimensionality as the input data 406 and used as input training data” (Abbott, ¶ [0066]) where the training may be performed by objects observed by a sensor, and is not limited to a list of shapes; and
determining ground truth shapes depicted in the template images “To generate ground truth data, the projected representation of the sensor data may be annotated with 2D or 3D polylines and corresponding class labels. Annotations may be generated manually and/or automatically, and used to derive corresponding ground truth outputs for the machine learning model(s) 408 (e.g., class confidence data 410 such as confidence maps or segmentation masks, shape regression data 412, etc.)” (Abbott, ¶ [0067]) where a ground truth shape is derived from the image.
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of identifying objects and training artificial intelligence with partly occluded shapes to identify their traits such as color, and to modify their traits as desired taught by Singh with the method of determining the ground truth shapes taught Kawano by to have an artificial intelligence training, using partly occluded shapes, and determining the shape within the given image. The motivation to do so would be to train the artificial intelligence model to recognize shapes based on an observable portion.
Claim 19 is rejected under 35 U.S.C. 103 as being unpatentable over Singh in view of Figueroa, Ding, and further in view of Sato Toshio (Pat. Pub. EP-0804777-B1, herein after “Toshio”).
In regard to claim 19, Singh in view of Figueroa teach The non-transitory computer readable medium of claim 14, wherein generating the modified solid fill region comprises:
validating, using a prominent color algorithm, the prominent color by:
generating a converted prominent color in an alternative color space “Apparatus 100 further includes a value/hue/chroma converter 114 coupled to an output of section 112, a color feature extractor 116 coupled to an output of converter 114, and a dictionary data storage section 118 for storing reference data coupled to an output of a dictionary data controller 120” (, Page 4) where the value may be converted with respect to HVC, or RGB;
filling the solid fill region with the prominent color based on validating the prominent color “the infill segmentation map 5908 assigns labels to the region determined to be inpainted. Accordingly, various models utilize the infill segmentation map 5908 to determine how to infill the region indicated to be inpainted” (Singh, ¶ [0680]) where the region is filled.
Singh does not explicitly teach binning the converted prominent color according to the alternative color space; and
comparing the converted
prominent color with the prominent color
Toshio teaches binning the converted prominent color according to the alternative color space “PATTERN RECOGNITION LETTERS, Vol. 11, No. 1, January 1990, Amsterdam, NL, pages 51-58, XP84831; I. ANDREADIS et al.: "Image pixel classification by chromaticity analysis" discloses an apparatus and method for pattern recognition, wherein for each pixel of image data representing a color image color fractions are computed and classified in color-bins as part of a recognition procedure in which an image can be considered as acceptable or otherwise, if its color-bin distribution matches a reference distribution within empirically derived error bounds or not” (Toshio, Page 2) where color-bins are known in the art to store color values in a given space, and conversions may be from RGB to HVC, or similar color spaces; and
comparing the converted prominent color with the prominent color “Pattern matching section 122 compares the dictionary data output from dictionary data controller 120 with the image data of the object to be identified and an identity of the target image data in the category corresponding to the data from extractor 116' is output from;identification result output section 124” (Toshio, Page 13) where the colors are compared.
It would have been prima facie obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the method of identifying, changing, and filling an object with a pertinent color taught by Singh with the method of binning colors and comparing said colors taught Kawano by to produce a method of validating object colors within the digital image. The motivation to do so would be to validate the colors used and changed when filling an object or the background.
Response to Arguments
The Examiner appreciates Attorney Brett Bunderson’s cooperation and clarity during the telephonic interview and would like to gratefully thank the Applicant for their patience and understanding. Applicant’s arguments with respect to claims 1-8, 10-12, 14, 15 and 20 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Applicant’s arguments, see Remarks pages 10-13, filed June 15, 2026, with respect to the rejections of claims 1, 8, and 14 under 35 U.S.C. § 103 have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new grounds of rejection is made in view of Zhihong Ding (Pat. Pub. US-20210027497-A1, herein after “Ding”).
Applicant argues that Singh, in view of any cited reference fails to teach the amended limitations of claim 1 determining, by applying a solid fill algorithm to color data extracted from the background region, that the background region depicts solid fill pixels based on a coverage percentage of a color value within the background region satisfying a solid fill threshold and extracting, from the digital image, a background image layer corresponding to the background region, the background image layer comprising pixels with values indicated by the prominent color. The Applicant specifies the critical untaught claimed subject matter as the solid fill algorithm being applied to color data extracted from the background, and that the background solid fill pixels are based on a coverage percentage of a color value satisfying a fill threshold. The Examiner finds that Singh teaches a solid fill algorithm to color data extracted by the background region “a content fill includes a set of pixels generated to fill a hole (e.g., a content void) that remains after (or if) a set of pixels (e.g., a set of pixels portraying an object) has been removed from or moved within a digital image. In some cases, a content fill corresponds to a background of a digital image … “ (Singh, ¶ [0171]) where the broadest reasonable interpretation of a given solid fill algorithm can be read as a content fill used to fill a hole, which is solid pixels, and the data is extracted from the background region, which is where the hole is based and the content fill is placed. Without a provided or limited scope as to how the solid fill is specifically performed, or how the extraction of background data is completed, the provided reference can be interpreted as prior art. After additional search and consideration the Examiner cites Ding to teach the absent amended features. Ding teaches detecting an area based on a coverage percentage “[t]he color classification system can then calculate a color-matching score for the object based on the number or percentage of pixels within the color similarity region (i.e., valid pixels) and pixels outside of the color similarity region (i.e., invalid pixels)” (Ding, ¶ [0033]) that satisfies a fill threshold “color classification system 106 applies a color similarity threshold to each mapped point” (Ding, ¶ [0072]). Examiner additionally notes the obviousness and motivation to combine the background detection capabilities as taught by Singh in view of Figueroa with the object detection and pixel color coverage percentage taught by Ding to produce a background detection based on a coverage percentage of similar pixels within a threshold.
The Applicant further argues the similar stance regarding the coverage percentage of a color value within the background region satisfying a solid fill threshold of claims 8 and 14 which the above response citing the additional Ding reference is incorporated herein. The Applicant additional notes Singh in view of the cited references fail to teach a coverage percentage of a color value within the shape satisfies a color threshold and a coverage percentage of a color value within the solid fill region satisfies a solid fill threshold in which the substantially similar rejection in view of Ding is made, providing the obvious motivation to switch the background region with the shape or solid fill region as the coverage percentage determination follows the same steps.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Yu-Ling Chen (Pat. Pub. US-20130155475-A1) discloses color fill methods, including background detection and enclosed region detection for the color fills (abstract).
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CAIDEN ALEXANDER USSERY whose telephone number is (571)272-1192. The examiner can normally be reached Monday - Friday* 7:30AM - 5PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Tammy Goddard can be reached at (571) 272-7773. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/C.A.U./Examiner, Art Unit 2611 /TAMMY GODDARD/Supervisory Patent Examiner, Art Unit 2611
Prosecution Insights