DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
PROSECUTION REOPENED
In view of the Appeal Brief filed on May 29, 2026, PROSECUTION IS HEREBY REOPENED. A new ground of rejection is set forth below.
To avoid abandonment of the application, appellant must exercise one of the following two options:
(1) file a reply under 37 CFR 1.111 (if this Office action is non-final) or a reply under 37 CFR 1.113 (if this Office action is final); or,
(2) initiate a new appeal by filing a notice of appeal under 37 CFR 41.31 followed by an appeal brief under 37 CFR 41.37. The previously paid notice of appeal fee and appeal brief fee can be applied to the new appeal. If, however, the appeal fees set forth in 37 CFR 41.20 have been increased since they were previously paid, then appellant must pay the difference between the increased fees and the amount previously paid.
A Supervisory Patent Examiner (SPE) has approved of reopening prosecution by signing below:
/ILANA L SPAR/Supervisory Patent Examiner, Art Unit 3622
Response to Amendment
In light of Applicant's submission filed November 21, 2025, the Examiner has maintained and updated the 35 USC § 103 rejection.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claim(s) 1-3,5-6, 9-12, 14, 17-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Frey (WO 2021/230863) in view of Lin et al. (US 2022/0366546) in further view of Somanath et al. (US 11, 423, 308)
Claims 1, 10, 18: Frey discloses a method, comprising:
receiving a plurality of images to train a first neural network;(see for example [0038], machine learning using images or video(e.g. a video is a plurality of images)
masking a portion of each of the plurality of images, wherein the masking a portion comprises: identifying products shown in each of the plurality of images, and masking portions of each of the plurality of images corresponding to the identified products; (see for example [0002], may include identifying a first object at a position within a first image, masking, based on the first image and the position of the first object, a target area to produce a masked image, generating, based on the masked image and an inpainting machine learning model, a second image different from the first image, the inpainting machine learning model being trained using a difference between the target area of training images and content of generated images at a location corresponding to the target area of the training images, generating, based on the masked image and the second image, a third image, and adding, to the third image, a new object different from the first object. (also see [0040,0044, 00660-0063])
inputting the masked images to the first neural network;(see for example [0009], discloses The techniques described in this document enable a system to provide an improved solution to image replacement by masking an area to be replaced, using an inpainting machine learning model (also referred to as an inpainting model) trained using areas surrounding particular types of objects within an image, to inpaint (e.g., predictively create or reconstruct) a proper subset of the image(also see [0040, 0044, and 0063])
receiving a plurality of frames associated with a video stream;(see for example [0023], discloses plurality of frames in a video)
identifying a first object image included in at least some of the plurality of frames;(see for example [0023] first item(e.g. object) inside the bounding box)
masking a region, in the at least some of the plurality of frames, associated with the first object image;(see for example [0023] discloses masking the item in the bounding box)
receiving information identifying at least one attribute associated with a user; (see for example [0011], discloses receiving user preference information that can be used to updated content. Also at [0023] the image can be adapted based on the users location. (e.g. attribute associated with the user))
identifying, based on the received information, a second object image to replace the first object image;(see for example [0023], The video includes an image that is to be adapted based on a viewer’s location. his example, the video is a public service announcement to be used across the United States, and the image can be a logo displayed in a comer of each frame of the video. The logo can be adapted to be the state in which a viewer is located.)
replacing pixel values in the masked region with contextually suitable pixel values associated with the second object image; (see for example [0046], determines target object, relative to pixel location and an area occupied by the target object and [0047] discloses image replacement processor generates a mask for the area of final rendering and generate a masked image. Also see [0048])
outputting the video stream with the second object image replacing the first object image in the at least some of the plurality of frames. (see for example [0023], and [0049] discloses generate a second image using the masked image that includes the content within the area of final rendering recreated by inpainting model. In other words, the second image can be a recreation of the final rendering generated by the inpainting model using the masked image) but does not explicitly disclose forward the images including the probable pixel values to a second neural network; determine, by the second neural network, whether each of the probable pixel values is contextually suitable; identify pixels, in each of the plurality of images, that are not contextually suitable; However Lin discloses forward the images including the probable pixel values to a second neural network; ([0109 and 0110], Image 702 can be input into coarse result neural network 704. Coarse result neural network 704 can be comprised of an encoder and a decoder. From image 702, coarse result neural network 704 can generate coarse completed image 706. Coarse completed image 706 can be image 702 with the hole designated by hole mask 702a filled using inpainting. This coarse completed image can have low pixel values (e.g., 64×64). In embodiments when coarse result neural network 704 is undergoing training, loss can be determined based on coarse completed image 706. For instance, L1 loss can be used to update coarse result neural network 704.
[0110] Coarse completed image 706 can be input into fine result neural network 708. Fine result neural network 708 can be comprised of an encoder and two decoders. The first decoder of the fine result neural network 708 (e.g., an image decoder) can generate inpainting image result 710. Inpainting image result 710 can have high pixel values (e.g., 256×256). The second decoder of fine result neural network 708 (e.g., a confidence decoder) can generate corresponding confidence map 712 of inpainting image result 710 (e.g., from the image decoder). During training, confidence map 712 can be binarized. In particular, confidence map 712 map can be set such that predicted pixels with a pixel value over a predefined threshold (e.g., 0.5) are set as high-confidence “known” pixels (e.g., white portion), and predicted pixels with a pixel value under the predefined threshold (e.g., 0.5) are set as low-confidence pixels (e.g., black portion) that remain as undefined pixels where information is not know (also see 0027 and 0030)) determine, by the second neural network, whether each of the probable pixel values is contextually suitable and identify pixels, in each of the plurality of images, that are not contextually suitable; [0028] In more detail, the image inpainting system can use information related to a confidence map as a feedback mechanism during iterative image inpainting. In particular, the image inpainting system can be used to predict pixels (e.g., predict pixel information) for one or more regions in an image with undefined pixels where information for those respective pixels is not known (e.g., holes). For instance, pixels can be predicted using the coarse result neural network and the fine result neural network. Further, the fine result neural network can generate a corresponding confidence map of the predicted pixels. By analyzing confidence values of the confidence map for the predicted pixels, high-confidence pixels can be identified (e.g., pixels with a confidence value over a predefined threshold). These high-confidence pixels can be used to replace the undefined pixels in the image in a subsequent iteration of inpainting (e.g., treat the high-confidence pixels as known pixels). In this way, the confidence map can be used as a feedback mechanism such that high-confidence pixels are used to iteratively replace undefined pixels, thus filling one or more holes in the image.. [0032], The high confidence “known” pixels can be used to replace corresponding undefined pixels from the initial input image such that in a subsequent iteration of image inpainting the input image can be the initial input image with undefined pixels replaced with high-confidence “known” pixels as determined in the first iteration. [0111], High confidence pixels 714 can be used to replace the undefined pixels in the image in a subsequent iteration of inpainting. For instance, image 720 can be image 702 with high confidence pixels 714 added such that hole mask 720a is smaller than hole mas)(also see [0069, 0071 and 0072]) Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention was made to modify, Frey to include generating, by the first neural network, probable pixel values for pixels located in the masked portion of each of the plurality of images; forwarding the images including the probable pixel values to a second neural network; determining, by the second neural network, whether each of the probable pixel values is contextually suitable; identifying pixels, in each of the plurality of images, that are not contextually suitable, in order to determine pixels that are deemed unnatural or inconsistent with the surrounding image context and thereby allow for intelligent filling of missing areas by generating plausible content that matches the surrounding image (see Lin, abstract).
Frey and Lin do not explicitly disclose generate, by the first neural network, probable pixel values for pixels located in the masked portion of each of the plurality of images, wherein each of the probable pixel values includes red, green and blue values.
However, Somanath discloses generate, by the first neural network, probable pixel values for pixels located in the masked portion of each of the plurality of images, wherein each of the probable pixel values includes red, green and blue values;(see for example Col. 10 lines 8-22, . With respect to using feature from pre-trained networks, a convolutional neural network (CNN) that was trained for an image-based task such as classification or segmentation could be used. Features at some layers can be extracted for both output and training/ground-truth images, and the distance between those features used as a loss. For example, L1 or L2 distances between those features may be used as the loss to be minimized. (43) To generate the content, a probabilistic classifier may be used to select color attribute values (e.g., RGB values) for each pixel of the missing part of the image. To do so, a pixel color attribute is segmented into a number of bins (e.g., value ranges) that are used as classes. The classifier determines probabilities for each of the bins of the pixel color attribute for each pixel and generates the content by selecting the bin having the highest probability. And Col 11 lines 20-29, In some implementations, the input data is a vector or statement describing attributes of an environment (e.g., “girl on yellow bus looking out window crying on a sunny day”). The input data may identify which pixels of the image are undefined, e.g. via a mask or other data structure. (48) At block 404, the method 400 generates content for the missing part using a machine-learning model, such as a classification-based neural network. ) Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention was made to modify Frey’s machine learning based inpainting process to include Somanath classification based pixel generation technique. Somanath teaches using classification based machine learning to generate perceptually plausible content for a missing portion of an image by determining probabilities associated with probable color values, including RGB values for pixels of the missing part of an image. The aforementioned modification would have involved the predictable use Somanath’s pixel generation technique with Frey’s image inpainting system to have the expected result of generating plausible content for the masked region. (see Col 4 lines 35-40, Col. 13 lines 1-22 and 34-46, Somanath) Claims 2, 11, 19: Frey disclose the method of claim 1 and system of claim 10 and non-transitory computer readable medium claim 18, further comprising:
identifying, based on the received information, items of interest associated with the user, wherein the identified items of interest include an object depicted by the second object image. (see for example [0011], the system allows for dynamic updating of content based on particular characteristics, including user preferences.)
Claims 3, 12, 20: Frey disclose the method of claim 2, system of claim 10, non-transitory computer readable medium claim 18 wherein the receiving information comprises at least one of:
receiving information from an external data source identifying characteristics or preferences for the user, or receiving information input by the user, wherein the information input by the user includes preferences for the user. (see for example [0032], information obtained from a user’s social network. Also discloses receiving information from a content server)
Claims 5,14: Frey disclose the method of claim 4 and system of claim 10, wherein the masking a portion of each of the plurality of images comprises:
masking a random portion of each of the plurality of images.(see [0040] applies a random forest algorithm to the masked area)
Claim 6: Frey disclose the method of claim 4 and system of , wherein the masking a portion of each of the plurality of images comprises:
masking a predetermined percentage of each of the plurality of images. ([0060], discloses using a maximum percentage)
Claim 9, 17: Frey discloses the method of claim 1 and system of claim 10 further comprising:
outputting, based on the received video stream, different video streams to a plurality of users, wherein the different video streams include different object images in place of the first object image.(see for example [0023], logo being adapted based on the location of the user)
Claim(s) 7 and 15 is/are rejected under 35 U.S.C. 103 as being unpatentable over Frey (WO 2021/230863) in view of Lin et al. (US 2022/0366546)) in further view of Somanath et al. (US 11, 423, 308) and in further view of Duan et al. (US 2020/0226714) Claims 7: Frey disclose the method of claim 4 and system of claim 10
the identified products.[0023, 0026, 0035] but does not explicitly disclose wherein determining, by the second neural network, whether each of the probable pixel values is contextually suitable comprises: analyzing RGB values associated with pixels that surround the pixels located in the masked portion to determine whether each of the probable pixel values is contextually suitable. However Lin disclose wherein determining, by the second neural network, whether each of the probable pixel values is contextually suitable comprises; ([0028, 0032, 0069, 0071 and 0072])
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention was made to modify, Frey to include determining, by the second neural network, whether each of the probable pixel values is contextually suitable comprises, in order to determine pixels that are deemed unnatural or inconsistent with the surrounding image context and thereby allow for intelligent filling of missing areas by generating plausible content that matches the surrounding image (see Lin, abstract).
However Duan discloses by the second neural network, whether each of the probable pixel values is contextually suitable comprises: analyzing RGB values associated with pixels that surround the pixels located in the masked portion to determine whether each of the probable pixel values is contextually suitable. ([0054, 065-0068, 0071)
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention was made to modify, Frey,Lin and Somanath to include by the second neural network, whether each of the probable pixel values is contextually suitable comprises: analyzing RGB values associated with pixels that surround the pixels located in the masked portion to determine whether each of the probable pixel values is contextually suitable, in order to produce natural looking images. (Duan [0077])
Claim 15: Frey discloses the system of claim 10, wherein the identified products comprise consumer products; .[0023, 0026, 0035] but does not explicitly disclose wherein the at least one processing device is configured to implement the first neural network and the second neural network, and wherein when determining, by the second neural network, whether each of the probable pixel values is contextually suitable, the at least one processing device is configured to: analyze RGB values associated with pixels that surround the pixels located in the masked portion to determine whether each of the probable pixel values is contextually suitable.
However Lin discloses wherein the at least one processing device is configured to implement the first neural network and the second neural network, and wherein when determining, by the second neural network ([0028, 0032, 0069, 0071 and 0072])
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention was made to modify, Frey to include determining, by the second neural network, whether each of the probable pixel values is contextually suitable comprises, in order to determine pixels that are deemed unnatural or inconsistent with the surrounding image context and thereby allow for intelligent filling of missing areas by generating plausible content that matches the surrounding image (see Lin, abstract).
Frey, Lin, S does not explicitly discloses determining, by the second neural network, whether each of the probable pixel values is contextually suitable, the at least one processing device is configured to: analyze RGB values associated with pixels that surround the pixels located in the masked portion to determine whether each of the probable pixel values is contextually suitable.
However Duan discloses determining, by the second neural network, whether each of the probable pixel values is contextually suitable, the at least one processing device is configured to: analyze RGB values associated with pixels that surround the pixels located in the masked portion to determine whether each of the probable pixel values is contextually suitable. ([0054, 065-0068, 0071)
Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention was made to modify, Frey,Lin and Somanath to include determining, by the second neural network, whether each of the probable pixel values is contextually suitable, the at least one processing device is configured to: analyze RGB values associated with pixels that surround the pixels located in the masked portion to determine whether each of the probable pixel values is contextually suitable, in order to produce natural looking images. (Duan [0077])
Claim(s) 8 and 16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Frey (WO 2021/230863) in view Lin et al. (US 2022/0366546)) in further view of Somanath et al. (US 11, 423, 308) and in further view of SEVASTOPOLSKIY et al. (US 2022/0157014)
Claims 8, 16: Frey discloses the method of claim 1 and system of claim 10, wherein the replacing pixel values comprises: but does not explicitly disclose identifying, by a neural network, pixel values associated with a received image; and outputting pixels values associated with the received image for the masked region. However SEVASTOPOLSKIY identifying, by a neural network, pixel values associated with a received image; and outputting pixels values associated with the received image for the masked region [0033] Therefore, it would have been obvious to a person of ordinary skill in the art before the effective filing date of the claimed invention was made to modify, Frey to include identifying, by a neural network, pixel values associated with a received image; and outputting pixels values associated with the received image for the masked region, in order to segmenting and filtering can improve the quality characteristics of the generated portrait by improving the accuracy of albedo, normal, environmental shadow maps, and segmentation mask predictions performed by the deep neural network. ([0034] of SEVASTOPOLSKIY)
Claim(s) 21 and 22 is/are rejected under 35 U.S.C. 103 as being unpatentable over Frey (WO 2021/230863) in view of Lin et al. (US 2022/0366546)) in further view of Somanath et al. (US 11, 423, 308) and in further view of Agarwal et al. (US 2022/0051479)
Claim 21: Frey discloses the method of claim 1, but does not explicitly disclose wherein the receiving information identifying at least one attribute associated with a user comprises: receiving purchase history information associated with the user. However, Agarwal discloses wherein the receiving information identifying at least one attribute associated with a user comprises: receiving purchase history information associated with the user. [0049] It would have been obvious to one of ordinary skill in the art, at the time of the invention, to have modified the method and system of Frey so as to have included wherein the receiving information identifying at least one attribute associated with a user comprises: receiving purchase history information associated with the user in order better target customers to provide relevant advertising. Furthermore because each individual element and its function are shown in the prior art, albeit in different references or embodiments, the difference between the claimed subject matter and the prior art rests not on any individual element or function but in the very combination itself––that is in the substitution of a purchase history information disclosed by Agarwal for information of the applicant. Thus, the simple substitution of one known element for another producing a predictable result renders the claim obvious. Claim 22: Frey discloses the system of claim 10, but wherein when receiving information identifying at least one attribute associated with a user, the at least one processing device is configured to: but does not explicitly disclose receive purchase history information associated with the user. [0049 Agarwal] It would have been obvious to one of ordinary skill in the art, at the time of the invention, to have modified the method and system of Frey so as to have included receive purchase history information associated with the user in order better target customers to provide relevant advertising. Furthermore because each individual element and its function are shown in the prior art, albeit in different references or embodiments, the difference between the claimed subject matter and the prior art rests not on any individual element or function but in the very combination itself––that is in the substitution of a purchase history information disclosed by Agarwal for information of the applicant. Thus, the simple substitution of one known element for another producing a predictable result renders the claim obvious.
Response to Arguments
Applicant's arguments filed May 29, 2026 have been fully considered but they are not persuasive.
The applicant argues the 103 rejection by stating, “do not disclose or suggest masking a portion of each of the plurality of images, wherein the masking a portion comprises identifying products shown in each of the plurality of images, as recited in” As stated in the previous final rejection October 2, 2025,the Examiner maintains this stance and as previously stated [0061 discloses [0061] The process 400 for replacing an object in an image begins with identifying a first object at a position within a first image (402). For example, image replacement processor 120 receives a final rendering 202 and identifies a target object 204 at a particular position within the first image. Object processor 122 of image replacement processor 120 can perform the identification. Object processor 122 can also identify an object type of target object 204 and determine a position of target object 204 in final rendering 202. Object processor 122 can generate a bounding box 206 surrounding target object 204. [0062] The process 400 continues with masking, based on the first image and the position of the first object, a target area to produce a masked image (404). For example, image replacement processor 120 can generate a mask to remove the area within bounding box 206 based on final rendering 202. Object processor 122 or inpainting model 124 can perform the mask generation and application process as described above with respect to FIGS. 1-3." Thus based on fig. 4 and paragraphs [0060-0062] the reference of Frey discloses identifying the object in an image. Also at [044] states “ the target object can be other types of content such as text, an open shape, a closed shape, a complex shape with transparency parameters, a simple shape, a moving shape, and/or a changing shape, among other types of objects.” The applicant’s “product” is merely a descriptor for an object and thus using broadest reasonable interpretation the reference of Frey “object is substantially equivalent to the applicant’s “product”. The applicant argues that the reference of Lin does not disclose forwarding images including probable pixel values to a second neural network. The Examiner respectfully disagrees, the reference of Lin discloses a first neural network at [0027, 0109 and 0110] (e.g. coarse neural network) and a second neural (e.g. fine result neural network). Thus Lin teaches implement ting an image-inpainting process that uses a coarse result neural network that generates a completed image containing predicted pixels and a second fine result neural network that receives the completed image generated by the first neural network.
The applicant further argues the reference of Lin does not disclose determining whether each of the probable pixel values are contextually suitable. The applicant also argues the aforementioned limitation by referencing [0026] of the applicant’s specification. However the Examiner respectfully disagrees it appears the applicant is improperly arguing a narrower requirement from the specification into the claim. The claim does not recite determining suitability by comparing each probable pixel value to surrounding pixel values. The claim simply recites determining whether each of the probable pixel values is contextually suitable. However, the Examiner respectfully disagrees the applicant has not defined “contextually suitable” and clearly limited the term. Thus, using broadest reasonable interpretation, the reference of Lin second network confidence based determination of whether predicted inpainting pixels are acceptable for use in completing the image reads on the limitation. See Lin at [0027, 0028, 0056, 0069-0072]
Applicant’s arguments with respect to the reference of Gowda and claims 7 and 15 have been considered but are moot due to the updated rejection above.
In response to applicant’s argument that there is no teaching, suggestion, or motivation to combine the references, the examiner recognizes that obviousness may be established by combining or modifying the teachings of the prior art to produce the claimed invention where there is some teaching, suggestion, or motivation to do so found either in the references themselves or in the knowledge generally available to one of ordinary skill in the art. See In re Fine, 837 F.2d 1071, 5 USPQ2d 1596 (Fed. Cir. 1988), In re Jones, 958 F.2d 347, 21 USPQ2d 1941 (Fed. Cir. 1992), and KSR International Co. v. Teleflex, Inc., 550 U.S. 398, 82 USPQ2d 1385 (2007). In this case, Frey and Lin are both in the same technical field of machine learning image inpainting of missing or masked image regions. The reference of Lin is directed to improving the same operations performed in Frey, the reconstructing of missing image content. Therefore, the reference of Lin is reasonably pertinent to the reference of Frey’s image inpainting process.
The applicant argues claims 21 and 22, however the applicant argues in regard to limitations of claim 1 and claim 10. The reference of Agarwal at [0049] user profiles may include historical purchase information for the respective user, which is equivalent to the applicant’s receiving purchase history associated with the user, thus this argument is moot.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DARNELL A POUNCIL whose telephone number is (571)270-3509. The examiner can normally be reached Monday - Friday 10:00 - 6:00.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Ilana Spar can be reached at (571) 270-7537. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/D.A.P/Examiner, Art Unit 3622
/ILANA L SPAR/Supervisory Patent Examiner, Art Unit 3622