DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-15 and 17-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Simard et al. (US 2003/0202697 A1) in view of He et al. (US 2021/0398334 A1).
Regarding claim 1, Simard teaches:
A computer-implemented method comprising:
generating a background digital image by:
determining a background digital image mask utilizing a method; ([0032], “The mask generated by the mask separator 102 can be employed to segment the document image into two layers, a foreground image and a background image. It is appreciated that alternate aspects of the invention can segment the image into more than two layers. The mask, also referred to as the mask image, is a binary image, where a pixel value determines whether that respective pixel belongs in the foreground image or the background image.”) and
generating background pixel colors for pixels indicated by the background digital image mask utilizing a method; ([0036], “The foreground background segmenter 104 receives the mask from the mask separator 102 and the document image. The foreground background segmenter 104 uses the mask to segment the document image into the foreground image and the background image. For each pixel of the document image, a corresponding pixel of the mask is referenced. The pixel is allocated to the foreground image or the background image based on the corresponding pixel of the mask. For example, if the corresponding pixel of the mask is a "1", the pixel is assigned to the foreground image. Conversely, if the corresponding pixel of the mask is a "0", the pixel is assigned to the background image.”)
generating a foreground digital image by:
determining a foreground digital image mask utilizing a method; ([0032], “The mask generated by the mask separator 102 can be employed to segment the document image into two layers, a foreground image and a background image. It is appreciated that alternate aspects of the invention can segment the image into more than two layers. The mask, also referred to as the mask image, is a binary image, where a pixel value determines whether that respective pixel belongs in the foreground image or the background image.”) and
generating foreground pixel colors for pixels indicated by the foreground digital image mask utilizing a method; ([0036], “The foreground background segmenter 104 receives the mask from the mask separator 102 and the document image. The foreground background segmenter 104 uses the mask to segment the document image into the foreground image and the background image. For each pixel of the document image, a corresponding pixel of the mask is referenced. The pixel is allocated to the foreground image or the background image based on the corresponding pixel of the mask. For example, if the corresponding pixel of the mask is a "1", the pixel is assigned to the foreground image. Conversely, if the corresponding pixel of the mask is a "0", the pixel is assigned to the background image.”) and
generating a collage digital image comprising the background digital image and the foreground digital image.([0075], “The combiner 510 combines the foreground image, the background image and the mask into a recombined document image.” [0157])
However, Simard does not teaches:
Each of the methods (generating a mask image, a foreground image and a background image) can be using a neural network, (such as a mask generator neural network, utilizing a pixel generator neural network)
On the other hand, He teaches:
a neural network can be used to generating a mask image, a foreground image and a background image. (claim 3: “wherein a network structure of each of the background image generation branch, the mask image generation branch and the foreground image generation branch is a deep neural network.”)
Simard teaches generating a mask image, a foreground image and a background image. He teaches neural networks can be used for generating a mask image, a foreground image and a background image.
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to have combined the teachings of Simard with the specific teachings of He to use neural networks to generate images to reduce the complexity of generating images.
Regarding claim 2, Simard in view of He teaches:
The computer-implemented method of claim 1, wherein generating the collage digital image comprises combining the background digital image and the foreground digital image via alpha compositing.( Simard [0103]-[0104], “Instead of utilizing a constant assumption for foreground and background, a polynomial regression can be employed to represent the foreground and background. For example, if the polynomials are planes of equation .alpha.x+.beta.y+.mu., the energy would be defined by: 18 v F = x , y F ( f ( x , y ) - F x + y + F ) 2 Eq . 23 v B = x , y B ( f ( x , y ) - B x + B y + B ) 2 Eq . 24 [0104] Where x,y index the pixel locations, and .alpha..sub.F,.beta..sub.F and .mu..sub.F are scalars that minimize .nu..sub.F and .alpha..sub.B,.beta..sub.B and .mu..sub.B are scalars that minimize .nu..sub.B. Note that .alpha..sub.F,.beta..sub.F and .mu..sub.F can be solved in constant time using the quantities .SIGMA..function.(x,y).sup.2- ,.SIGMA..function.(x,y)x,.SIGMA..function.(x,y)y, and .SIGMA..function.(x,y). This is a linear system of three unknown and three equations, and the same applies to .alpha..sub.B,.beta..sub.B and .mu..sub.B. As before, the algorithm is bottom-up and minimizes E at each merge. The foregrounds and backgrounds cannot be sorted by average, and therefore all seven combinations are tested to determine which combination minimizes E. To keep performing each test and merge in constant time, the quantities .SIGMA..function.(x,y).sup.2,.SIGMA..functi- on.(x,y)x,.SIGMA..function.(x,y)y,.SIGMA..function.(x,y) and N should be maintained for each region for the foreground and the background. The simple region optimization is still possible, but could assume a constant over the region, a polynomial regression, or both.”)
Regarding claim 3, Simard in view of He teaches:
The computer-implemented method of claim 1,
wherein: generating the background digital image comprises utilizing a digital image collaging neural network including the mask generator neural network and the pixel generator neural network to generate a combined digital image depicting the background pixel colors at unmasked pixels indicated by the background digital image mask; (Simard See FIG. 6, generated background image. He teaches the mask generation neural network and background image generation network and foreground image generation network. See claim 1, for the combination rationale.) and
generating the foreground digital image comprises utilizing the digital image collaging neural network to generate the foreground digital image mask and the foreground pixel colors from the combined digital image.( Simard FIG. 6, image 610 and 611. He teaches the mask generation neural network and background image generation network and foreground image generation network. See claim 1, for the combination rationale. )
Regarding claim 4, Simard in view of He teaches:
The computer-implemented method of claim 1, wherein: determining the background digital image mask comprises…determine, … masked regions and unmasked regions for composing the background digital image; (Simard See FIG. 6, 612) and determining the foreground digital image mask comprises … determine…, masked regions and unmasked regions for composing the foreground digital image. (Simard See FIG. 6, 611 )
He further teaches: utilizing the mask generator neural network to determine, from a noise vector, to generating background and foreground image. ([0029], “the generative adversarial network may be trained in the following manner: taking the second image in the training sample as a real sample; after the first image is input into the generator, acquiring firstly image features of the first image, the image features being deep semantic information of the image represented by a vector, then inputting the image features into the foreground image generation branch, the mask image generation branch and the background image generation branch respectively,” FIG. 2. The combination rationale of claim 1 is incorporated here.)
Regarding claim 5, Simard in view of He teaches:
The computer-implemented method of claim 4, wherein: generating the background pixel colors comprises …to determine, ….colors for pixels of the unmasked regions for composing the background digital image; (Simard See FIG. 6, 611 ) and generating the foreground pixel colors comprises … to determine, … colors for pixels of the unmasked regions for composing the foreground digital image. (Simard See FIG. 6, 611 ) He further teaches: utilizing the pixel generator neural network to determine, from a noise vector, to generating background and foreground image. ([0029], “the generative adversarial network may be trained in the following manner: taking the second image in the training sample as a real sample; after the first image is input into the generator, acquiring firstly image features of the first image, the image features being deep semantic information of the image represented by a vector, then inputting the image features into the foreground image generation branch, the mask image generation branch and the background image generation branch respectively,” FIG. 2. The combination rationale of claim 1 is incorporated here.)
Regarding claim 6, Simard in view of He teaches:
The computer-implemented method of claim 1, further comprising: generating an intermediate digital image by: determining an intermediate digital image mask utilizing the mask generator neural network; (Simard [0032], “The mask generated by the mask separator 102 can be employed to segment the document image into two layers, a foreground image and a background image. It is appreciated that alternate aspects of the invention can segment the image into more than two layers. The mask, also referred to as the mask image, is a binary image, where a pixel value determines whether that respective pixel belongs in the foreground image or the background image.” See claim 1 for the neural network teachings.) and generating intermediate pixel colors for pixels indicated by the intermediate digital image mask utilizing the pixel generator neural network; (Simard [0036], “The foreground background segmenter 104 receives the mask from the mask separator 102 and the document image. The foreground background segmenter 104 uses the mask to segment the document image into the foreground image and the background image. For each pixel of the document image, a corresponding pixel of the mask is referenced. The pixel is allocated to the foreground image or the background image based on the corresponding pixel of the mask. For example, if the corresponding pixel of the mask is a "1", the pixel is assigned to the foreground image. Conversely, if the corresponding pixel of the mask is a "0", the pixel is assigned to the background image.” See claim 1 for the neural network teachings.)
and generating an additional collage digital image comprising the background digital image, the foreground digital image, and the intermediate digital image. (Simard [0075], “The combiner 510 combines the foreground image, the background image and the mask into a recombined document image.” [0157])
Although Simard in view of He does not explicitly teach generating intermediate images, it would have been well-known in the art a method of can be reused. In this case, the method of Simard in view of He can be used again and again for different input images, e.g. can be used again by inputting the first output collage image of generated using the Simard in view of He method into the Simard in view of He method. In this case, the images generated during generating the first output collage image are intermediate images. It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to have combined the teachings of Simard in view of He with the well-known knowledge to allow users flexibility of generating an image.
Regarding claim 7, Simard in view of He teaches:
The computer-implemented method of claim 1, wherein generating the background digital image is performed iteratively by utilizing a previous collage digital image generated in a previous iteration as input when generating the background digital image. (Simard [0036], “The foreground background segmenter 104 receives the mask from the mask separator 102 and the document image. The foreground background segmenter 104 uses the mask to segment the document image into the foreground image and the background image. For each pixel of the document image, a corresponding pixel of the mask is referenced. The pixel is allocated to the foreground image or the background image based on the corresponding pixel of the mask. For example, if the corresponding pixel of the mask is a "1", the pixel is assigned to the foreground image. Conversely, if the corresponding pixel of the mask is a "0", the pixel is assigned to the background image.” Although Simard in view of He does not explicitly teach generating images iteratively, it would have been well-known in the art a method of can be reused. In this case, the method of Simard in view of He can be used again and again for different input images, e.g. can be used again by inputting the first output collage image of generated using the Simard in view of He method into the Simard in view of He method to generate another background image. It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to have combined the teachings of Simard in view of He with the well-known knowledge to allow users flexibility of generating an image.)
Regarding claim 8, Simard in view of He teaches:
A non-transitory computer readable medium storing instructions thereon that, when executed by at least one processor, cause the at least one processor to perform operations (Simard [0184], FIG. 11) the rest of claim 8 recites similar limitations of claim 1, thus are rejected accordingly.
Regarding claim 10, Simard in view of He teaches:
The non-transitory computer readable medium of claim 8, wherein generating the background digital image comprises utilizing a digital image collaging neural network to extract encoded features from a noise vector. (He [0029], “the generative adversarial network may be trained in the following manner: taking the second image in the training sample as a real sample; after the first image is input into the generator, acquiring firstly image features of the first image, the image features being deep semantic information of the image represented by a vector, then inputting the image features into the foreground image generation branch, the mask image generation branch and the background image generation branch respectively,” FIG. 2. The combination rationale of claim 8 is incorporated here.)
Regarding claim 11, Simard in view of He teaches:
The non-transitory computer readable medium of claim 8, wherein the operations further comprise: generating a revised collage digital image by: utilizing the collage digital image as input to generate a revised background digital image; ( Simard [0036], “The foreground background segmenter 104 receives the mask from the mask separator 102 and the document image. The foreground background segmenter 104 uses the mask to segment the document image into the foreground image and the background image. For each pixel of the document image, a corresponding pixel of the mask is referenced. The pixel is allocated to the foreground image or the background image based on the corresponding pixel of the mask. For example, if the corresponding pixel of the mask is a "1", the pixel is assigned to the foreground image. Conversely, if the corresponding pixel of the mask is a "0", the pixel is assigned to the background image.”) and compositing the revised background digital image with a revised foreground digital image. (Simard [0075], “The combiner 510 combines the foreground image, the background image and the mask into a recombined document image.” [0157])
Although Simard in view of He does not explicitly teach using the output of the method as an input, it would have been well-known in the art a method of can be reused. In this case, the method of Simard in view of He can be used again and again for different input images, e.g. can be used again by inputting the first output collage image of generated using the Simard in view of He method into the Simard in view of He method. The final output would be a revised output. It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to have combined the teachings of Simard in view of He with the well-known knowledge to allow users flexibility of generating an image.
Claim 9 recites similar limitations of claim 2, thus are rejected accordingly.
Claim 12-14 recites similar limitations of claim 3-5 respectively, thus are rejected accordingly.
Claim 15 recites similar limitations of claim 8, thus are rejected accordingly.
Claim 17-20 recites similar limitations of claim 7, 2, 11 and 10 respectively, thus are rejected accordingly.
Claim(s) 16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Simard in view of He and further in view of Pipher (US 2023/0077552 A1).
Regarding claim 16, Simard in view of He teaches:
The system of claim 15,
However, Simard in view of He does not, but Pipher teaches:
wherein generating the collage digital image further comprises combining the background digital image and the foreground digital image with an additional background digital image and an additional foreground digital image. (Pipher teaches filming different background and foreground images from different angles with different cameras. Then combine the different foreground and background images to generate a composite image. FIG. 1, 5)
Simard in view of He teaches combine background and foreground images. Pipher teaches the combination can include more background and foreground images.
It would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to have combined the teachings of Simard in view of He with the specific teachings of Pipher in order to generate an composite image with a wide view.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to YANNA WU whose telephone number is (571)270-0725. The examiner can normally be reached Monday-Thursday 8:00-5:30 ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Alicia Harrington can be reached at 5712722330. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/YANNA WU/Primary Examiner, Art Unit 2615
Prosecution Insights