Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) was filed on 10/24/2023. The submission is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement is being considered by the examiner.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1, 3-10, 14-23 and 25-26 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Pugh et. al., hereafter Pugh (US Publication No. 20210142497 A1).
As per claim 1, Pugh teaches “A method executed by one or more computing devices for foreground object deletion and inpainting, the method comprising:
storing contextual information corresponding to an image of a scene, the contextual information comprising depth information corresponding to a plurality of pixels in the image and a semantic map indicating semantic labels associated with the plurality of pixels in the image; (See all of paragraphs 21 and 78. Paragraph 21 talks about utilizing depth maps (depth information corresponding to a plurality of pixels in the image) and determines an object class per pixel using semantic segmentation (a semantic map indicating semantic labels associated with the plurality of pixels in the image). In paragraph 78 it shows more details of the semantic map utilized “[0078] Segmenting the scene S420 preferably functions to determine semantic probabilities for each of a set of pixels, and can optionally determine edges for each identified object. Segmenting the scene (S420) can include generating a semantic segmentation map that can be used to: refine edge depth in the dense, scaled, fused depth map; generate object masks; and/or be otherwise used. ” See also paragraph 145 “For example, semantic segmentation information can be a bitmask, a probability mask, and/or any other suitable mask (e.g., identifying one or a plurality of known labeled features, such as floor which must never occlude rendered 3D objects, like furniture or floor coverings). ” Pugh)
identifying one or more foreground objects in the scene based at least in part on the contextual information, each foreground object having a corresponding object mask; (See paragraphs 121, 123, 21, 29, 172. Paragraph 121 shows teaches that foreground objects have foreground masks, that information is used to identify objects in the scene as seen in para. 123. “[0121] S500 preferably functions to determine foreground occlusion masks and/or depths for the scene imagery (e.g., for each of a set of objects appearing in the scene),”. See also figure 5 where it shows foreground objects, and in paragraph 21 it is taught that each object has its own class and therefore its own label, therefore identified. Paragraph 78 states that it generates object masks of each object. Pugh )
identifying at least one foreground object in the one or more foreground objects for removal from the image;(See paragraph 29 and 172. “[0029] The applicant has further enabled dynamic occlusion (controllable obscuring of virtual objects by existing physical objects) and disocclusion (removal of existing foreground objects) using computer vision techniques and a standard 3D graphics engine (e.g., by developing custom shaders and transforming the visual information to a format compatible with the graphics engine).” “[0172]… In a seventh example, user input identifying selection of pixels to be replaced includes: information identifying desire to remove foreground objects, yielding an empty room. In this example, the system removes all pixels that are not included in image segments related to structural components of a room (e.g., floor, walls, ceiling, stairs) or objects that are not likely to be removed from a room (e.g., doors, cabinetry, built-ins). However, one or more objects or sets of pixels to be removed can otherwise be identified.” Pugh)
generating a removal mask corresponding to the at least one foreground object based at least in part on at least one object mask corresponding to the at least one foreground object; (Examiner interprets “removal mask” as an “occlusion mask”. Paragraph 78 teaches that it generates object masks based on each object “[0078]… Segmenting the scene (S420) can include generating a semantic segmentation map that can be used to: refine edge depth in the dense, scaled, fused depth map; generate object masks; and/or be otherwise used.”, paragraph 121 says that step S500 is used to determine foreground occlusion masks of the image “[0121] S500 preferably functions to determine foreground occlusion masks and/or depths for the scene imagery (e.g., for each of a set of objects appearing in the scene), but can additionally or alternatively perform any other suitable set of functionalities.”, para. 128 says that occlusion masks are based on data obtained from 420 (which includes object mask of each object) “In variants, the occlusion masks are determined based on one or more of: the semantic segmentation map (e.g., from S420, from S470, etc.); a subset of the semantic segmentation map (e.g., segments for a given object class); a depth map (e.g., fused depth map from S460, dense depth map, sparse depth map, etc.); and/or any other suitable data. The occlusion masks can be determined using filters (e.g., median filtering, pseudo-median filters, bilateral filters, smoothing, other non-linear digital filtering, etc.) or otherwise determined.”, finally para. 149 says that an occlusion mask is the foregrounds masks from s500 “[0149]… The occlusion mask is preferably the foreground masks from S500, but can additionally or alternatively be any other suitable masks. ”. See also paragraph 150, occlusion masks form part of occlusion information. Pugh)
determining an estimated geometry of the scene behind the at least one foreground object based at least in part on the contextual information; and (See paragraph 171, pixels of the object are identified and the geometry of the scene behind the object is reconstructed. “[0171] In a first variation, removing pixels of real object(s) in the rendered scene (S700) includes one or more of: identifying pixels to remove S710; estimating depths of replacement pixels (e.g., by using estimated scene geometries to reconstruct the 3D depths and/or shapes likely to appear behind the removed pixels)…” See also paragraph 174. Pugh)
inpainting pixels corresponding to the removal mask with a replacement texture omitting the at least one foreground object based at least in part on the estimated geometry of the scene behind the at least one foreground object. (See fig. 1I and paragraph 171 and 183, it shows that pixels are filled in (inpainting) according to the removed object which is based on the occlusion information (removal mask) used for the occluding pixels as shown in the figure. “[0183] In variants, determining the color of replacement pixels at S730 includes one or more of: performing context-aware fill (inpainting) to determine a color for one or more pixels included in the region of replacement pixels (e.g., the region of pixels replacing the pixels belonging to the removed object);… using texture synthesis techniques;…”. See also paragraph 170 “ In some implementations, removing pixels of a real object from a rendered scene includes changes to occlusion behavior such as disabling occlusion for removed pixels, removing depth information for pixels of the object from the 3D depth map of the scene, and/or replacing the depths of the removed pixels with new depth values.”. It is corresponding to the removal mask because the occlusion mask need to be updated when removing objects as seen in fig. 1I. Pugh)
Claims 25 and 26 are rejected under the same analysis as claim 1. Claim 25 only additionally mentions an apparatus, processors, memories and storing. (See figure 10 and paragraph 20. Pugh) Claim 26 only additionally mentions using a computer readable medium. (See figure 10 and paragraphs 206 and 30-32. Pugh)
As per claim 3, Pugh teaches “The method of claim 1, wherein the depth information corresponding to the plurality of pixels in the image comprises one of:
a dense depth map corresponding to the plurality of pixels;
a sparse depth map corresponding to the plurality of pixels;
a plurality of depth pixels storing both color information and depth information.
a mesh representation corresponding to the plurality of pixels;
a voxel representation corresponding to the plurality of pixels;” (See paragraphs 21, and paragraph 52 “[0052]…The depth data can be depth maps (e.g., sparse, dense, etc.), 3D meshes or models, signed-distance fields, point clouds, voxel maps, or any other suitable depth data representation. ” See also paragraph 134 and 131 “[0134] S520 can include regularizing connected component occlusion depths by combining depth maps, color images and normal maps.” ”[0131]….By more accurately aligning depth edges to color edges of objects, object occlusion is also more convincing.”. Pugh)
“depth information associated with one or more polygons corresponding to the plurality of pixels; or three-dimensional geometry information corresponding to the plurality of pixels.” (See paragraph 52 “[0052] The three-dimensional features can be: captured, extracted, calculated, estimated, or otherwise determined. The three-dimensional features can be captured concurrently, asynchronously, or otherwise captured with the images. Three-dimensional features can include depth data. The depth data can be depth maps (e.g., sparse, dense, etc.), 3D meshes or models, signed-distance fields, point clouds, voxel maps, or any other suitable depth data representation. The three-dimensional features can be determined based on the individual images from the set, multiple images from the set, or any other suitable combination of images in the set. The three-dimensional features can be extracted using photogrammetry (e.g., structure from motion (SFM), multi-view stereo (MVS), etc.), three-dimensional point projection, or any other suitable method. Three-dimensional point projection can include determining image planes for an image pair using respective camera poses and projecting three-dimensional points to both image planes using camera poses, or any other suitable method.”) (Therefore Pugh teaches at least one of the limitations presented.)
As per claim 4, Pugh teaches “The method of claim 1, wherein the semantic labels comprise one or more of: floor, wall, table, desk, window, curtain, ceiling, chair, sofa, furniture, light fixture, or lamp. (See paragraphs 172 “ Example user input approaches include: a “magic eraser brush”, lasso and/or region selector, segment selector, and/or object selector (e.g., couch, table, and/or any other suitable connected component)… In this example, the system removes all pixels that are not included in image segments related to structural components of a room (e.g., floor, walls, ceiling, stairs) or objects that are not likely to be removed from a room (e.g., doors, cabinetry, built-ins). ” See also paragraph 46 “… semantics (e.g., correlating low level features such as colors; gradient orientation; with the content of the scene imagery such as wall, window, table, carpet, mirror, etc.); ” See also paragraph 169 ” [0169] In variants, S700 functions to remove real objects, or portions of real objects, from the rendered scene (e.g. removing a sofa from a scene so you can replace with another). ” See also paragraph 181 “The training dataset can be generated by creating depth maps of 3D CAD models of representative scenes with room structure and furniture models. Selection regions can be generated by choosing scene objects (e.g., furniture, art, etc.) to remove.”. Therefore Pugh teaches at least one of the limitations presented. Pugh)
As per claim 5, Pugh teaches “The method of claim 1, wherein identifying one or more foreground objects in the scene based at least in part on the contextual information comprises: identifying a plurality of object pixels corresponding to an object in the scene; (See paragraph 166 “[0166] In some variants, S640 includes rendering virtual objects using occlusion information generated at S630, using the generated occlusion information to perform occlusion processing for virtual objects that overlap real objects in the rendered scene. For each virtual object pixel of the virtual object, the depth for the virtual object pixel is identified. ”) identifying one or more semantic labels corresponding to the plurality of object pixels; (See paragraph 172 and figure 1I “In some implementations, the user interface includes user input elements that receive user input that is used by the system to identify pixels to be replaced. Example user input approaches include: a “magic eraser brush”, lasso and/or region selector, segment selector, and/or object selector (e.g., couch, table, and/or any other suitable connected component)… In a fourth example, user input identifying selection of pixels to be replaced includes: information identifying a segment of the image (e.g., generated as output of a semantic segmentation process performed on the image).”) and classifying the object as either a foreground object or a background object based at least in part on the identified one or more semantic labels. (See both paragraphs 128 and 139. Paragraph 139 shows that images are stored with an eight-bit telling if they are in a foreground or background mask, and in paragraph 128 it shows that the masks are based on semantic segmentation (which includes semantic labels), therefore the objects are determined whether they are in the foreground or background since all objects have a mask as seen in paragraph 123. See also paragraph 29 and figure 5 which shows that foreground objects are identified. Pugh )
As per claim 6, Pugh teaches “The method of claim 1, wherein identifying at least one foreground object in the one or more foreground objects for removal from the image comprises: receiving a selection from a user of the at least one foreground object in the one or more foreground objects for removal from the image.” (See paragraph 172 “[0172] In a seventh example, user input identifying selection of pixels to be replaced includes: information identifying desire to remove foreground objects, yielding an empty room. In this example, the system removes all pixels that are not included in image segments related to structural components of a room (e.g., floor, walls, ceiling, stairs) or objects that are not likely to be removed from a room (e.g., doors, cabinetry, built-ins).” Pugh)
As per claim 7, Pugh teaches “The method of claim 1, wherein identifying at least one foreground object in the one or more foreground objects for removal from the image comprises: identifying all foreground objects in the one or more foreground objects for removal from the image. (See paragraph 29, 121 and 172. Paragraph 29 shows that all foreground objects can be removed, paragraph 121 shows that the processing is for all objects appearing a scene “[0121] S500 preferably functions to determine foreground occlusion masks and/or depths for the scene imagery (e.g., for each of a set of objects appearing in the scene),”, paragraph 172 shows that all pixels to be removed are identified, which include the foreground objects to be removed. “[0172] Identifying pixels to remove from the rendered scene S710… In a seventh example, user input identifying selection of pixels to be replaced includes: information identifying desire to remove foreground objects,… However, one or more objects or sets of pixels to be removed can otherwise be identified. ” Pugh)
As per claim 8, Pugh teaches “The method of claim 7, wherein generating a removal mask corresponding to the at least one foreground object based at least in part on the at least one object mask corresponding to the at least one foreground object comprises: generating a furniture mask corresponding to all foreground objects in the one or more foreground objects by combining one or more object masks corresponding to the one or more foreground objects.” (See paragraph 121 which shows that there is a foreground mask for every set of objects appearing in the image, and paragraph 149 shows that the final occlusion mask is all of the foreground masks seen in paragraph 121, therefore a combination of all object masks has happened, and therefore one or more objects have been combined to create a “furniture mask”. Examiner interprets “furniture mask” as any mask having one or more objects, “furniture” is just a label. “[0149]… The occlusion mask is preferably the foreground masks from S500, but can additionally or alternatively be any other suitable masks.” Pugh. See also paragraph 81)
As per claim 9, Pugh teaches The method of claim 1, wherein the one or more foreground objects comprise a plurality of foreground objects and wherein identifying at least one foreground object in the one or more foreground objects for removal from the image comprises:
combining two or more foreground objects in the plurality of foreground objects into a compound foreground object based at least in part on one or more of: proximity between pixels corresponding to the two or more foreground objects, overlap between pixels corresponding to the two or more foreground objects, or semantic labels corresponding to the two or more foreground objects, wherein the compound foreground object comprises a compound object mask; and identifying the compound foreground object for removal from the image. (See paragraphs 149,172, 121 and 81. See paragraph 121 which shows that there is a foreground mask for every set of objects appearing in the image, and paragraph 149 shows that the final occlusion mask is all of the foreground masks seen in paragraph 121, therefore a combination of all object masks has happened. “[0149]… The occlusion mask is preferably the foreground masks from S500, but can additionally or alternatively be any other suitable masks”. Paragraph 172 shows that a user can select which one or more objects to remove, they can also select a region (proximity between pixels) and also which pixels to be removed. A user can also utilize a lasso to delimit specific objects to remove. A user can also select objects based on their semantic labels (object pixel classes), it also shows an object selector (which can choose more two or more objects). Additionally paragraph 172 shows that one or more objects can be identified. See also paragraph 81 where it shows that pixel object classes (which includes semantic labels) can be combined to obtain object masks. Pugh)
As per claim 10, Pugh teaches “The method of claim 9, wherein generating a removal mask corresponding to the at least one foreground object based at least in part on the at least one object mask corresponding to the at least one foreground object comprises: combining two or more object masks corresponding to the two or more foreground objects into a compound object mask.” (See paragraph 121 which shows that there is a foreground mask for every set of objects appearing in the image, and paragraph 149 shows that the final occlusion mask is all of the foreground masks seen in paragraph 121, therefore a combination of all object masks has happened. “[0149]… The occlusion mask is preferably the foreground masks from S500, but can additionally or alternatively be any other suitable masks.” Pugh. See also paragraph 81.)
As per claim 14, Pugh teaches “The method of claim 1, wherein generating a removal mask corresponding to the at least one foreground object based at least in part on at least one object mask corresponding to the at least one foreground object comprises: modifying the at least one object mask corresponding to the at least one foreground object based at least in part on the contextual information. (See paragraph 205 which shows that refinement of the depth can be performed (depth is contextual information), and in paragraph 206 the masks are updated, which include the object masks of paragraph 123. See also paragraph 172, it shows that foreground objects (which correspond to the object mask) can be removed (also a modification). Pugh)
As per claim 15, Pugh teaches “The method of claim 1, wherein determining an estimated geometry of the scene behind the at least one foreground object based at least in part on the contextual information comprises:
identifying a plurality of planes corresponding to a plurality of background objects in the scene based at least in part on the depth information and the semantic map; (See paragraphs 21, 41, 93, and 100-104 “[0021]… determining the floor plane(s) (e.g., using a cascade of 3D depthmap(s), surface normals, gravity, AR-detected planes, and semantic segmentation, etc.);”, “[0103] S450 preferably identifies horizontal planes (e.g., floors), but can additionally or alternatively identify vertical planes (e.g., walls) and/or any other suitable plane. S450 can optionally determine heights, surface normal, orientation, and/or any other suitable plane information…”)
storing a plurality of plane equations corresponding to the plurality of planes and a plurality of plane masks corresponding to the plurality of planes, wherein each plane mask indicates the presence or absence of a particular plane at a plurality of pixel locations; (See paragraphs 103-106. Paragraph 104 shows the use of plane equations and a plurality of planes “determining horizontal planes based on fitting planes to point clouds with a surface normal parallel to the gravity vector (e.g., using histogram search, RANSAC, search, and/or any other suitable model fit); determining floor planes by filtering point clouds for points labeled as semantic floor classes, before horizontal plane fitting; determining horizontal and/or floor planes using a trained neural network that determines plane regions and plane equations; determining architectural boundaries (e.g., floor, wall, ceiling, etc.) based on floor/wall/ceiling points near wall seams, near chair legs, near sofa boundaries, and/or based on any other suitable set of points;”. By determining the planes with the semantic class, plane masks are produced. Pugh also teaches storing all information such as planes, see paragraphs 139 and figure 10. Pugh)
determining one or more planes behind the at least one foreground object based at least in part on the plurality of plane masks and a location of the at least one foreground object; and (See paragraph 185 “ In particular, the ghosting pattern can be evocative of the room geometry that lies behind the removed pixels. Pixels with floor behind them can be filled with a ghost pattern that looks like a tile floor. Pixels with wall behind them can be filled with a ghost pattern that looks like a simple wallpaper pattern. These ghosting color patterns can optionally be perspective warped by the replaced depth (e.g., wall, floor depths) for added realism.” The plane behind the removed pixels (which correspond to the removed foreground object as seen in paragraphs 169-180) is determined, which is based on planes seen in paragraphs 103-106. Pugh )
determining an estimated geometry of the one or more planes behind the at least one foreground object based at least in part on one or more plane equations corresponding to the one or more planes. (See paragraphs 97, 185 and 103-106. Paragraph 112 shows that the geometry is always determined and regularized when processing segmentation, the planes are segmented based on the foreground object and have equations as seen in paragraphs 103-106. Pugh )
As per claim 16, Pugh teaches “The method of claim 1, wherein the estimated geometry comprises one or more planes and wherein inpainting pixels corresponding to the removal mask with a replacement texture omitting the at least one foreground object based at least in part on the estimated geometry of the scene behind the at least one foreground object comprises, for each plane in the one or more planes:
inpainting a set of pixels corresponding to at least a portion of the removal mask that overlaps the plane; and (See paragraphs 183 and 194. Paragraph 194 shows that inpainting occurs to the region belonging to the removed object, which corresponds to the occlusion mask, it overlaps the plane determined in paragraphs 100-104. “[0194]… Assigning replacement colors to replacement pixels can include one or more of cloning, CNN inpainting, propagating, or patch-matching colors of related regions (e.g., wall regions, floor regions, instances, classes) to the region of replacement pixels.”. See also figures 5-8. Pugh)
adjusting depth information for at least a portion of the inpainted pixels based at least in part on depth information corresponding to the plane. (See paragraph 174 “[0174] In a first variant, S720 includes: for each replacement pixel, setting the depth to a depth related to a known depth or a predicted depth for a key geometric surface (or surfaces) behind (or predicted to be behind) the location of the pixel being replaced. In some implementations, the key geometric surface is an architectural geometric surface of a room (e.g., a wall, a floor, etc.). In variants, the new depth for a replacement pixel is interpolated based on known or estimated depths for pixels of the partially-occluded surface (or surfaces) that surround the replacement pixel…”)
As per claim 17, Pugh teaches “The method of claim 16, further comprising, for each plane in the one or more planes:
updating semantic labels associated with the inpainted pixels based at least in part on semantic labels associated with the plane. (See paragraphs 174-175 “[0174]… For example, in removing an object that is positioned in front of a wall in an image of a room, the new depths for the pixels at the location (in the image's coordinate space) of the removed object are determined based on known depths for the wall. ” “[0175] The system identifies the most likely key surface (or surfaces) that is behind the pixel being replaced, and uses the known depths of the identified surface to interpolate (or estimate) the new depth of the replacement pixel.”. By identifying the key surface behind the removed inpainted pixels, the semantic labels associated are updated. Examiner interprets surface as a plane. Pugh )
As per claim 18, Pugh teaches The method of claim 16, wherein inpainting pixels corresponding to the removal mask with a replacement texture omitting the at least one foreground object based at least in part on the estimated geometry of the scene behind the at least one foreground object further comprises, for each plane in the one or more planes:
performing a homography to warp pixels of the plane from an original viewpoint into a fronto-parallel plane prior to inpainting the set of pixels; and (See paragraphs 63-65 and 194. They teach homography to warp pixels and it happens before inpainting, and paragraph 65 shows a fronto-parallel plane “[0065] In a first example, rectifying the image includes: adjusting the pitch angle of camera to make vertical lines (which appear to slant in 2D due to converging perspective) closer to parallel (e.g., in the image and/or in the 3D model).” Pugh)
performing a reverse homography to warp the pixels of the plane back to the original viewpoint subsequent to inpainting the set of pixels. (See paragraph 194, it shows reverse homography “Further, the texture synthesis can be performed on a rectified version of the image of that plane and then returned to the image via perspective warping (homography). Many texture synthesis algorithms produce better results on such an image of a rectified plane.” Pugh)
As per claim 19, Pugh teaches The method of claim 1, wherein inpainting pixels corresponding to the removal mask with a replacement texture omitting the at least one foreground object based at least in part on the estimated geometry of the scene behind the at least one foreground object comprises:
performing a transformation to warp pixels corresponding to the estimated geometry from an original viewpoint into a frontal viewpoint prior to inpainting the set of pixels; ( Examiner interprets “estimated geometry” as anything geometry related. See paragraphs 63-66 and 194. It teaches that a transformation is done to warp pixels to a different viewpoint, since the viewpoint can be changed however, a frontal viewpoint is achieved such as said in the examples. “[0065] In a first example, rectifying the image includes: adjusting the pitch angle of camera to make vertical lines (which appear to slant in 2D due to converging perspective) closer to parallel (e.g., in the image and/or in the 3D model). In a second example, rectifying the image includes adjusting the roll angle of the camera to make the scene horizon line (or other arbitrary horizontal line) level. In a third example, rectifying the image includes adjusting angles or cropping to optimize field of view. In a fourth example, rectifying the image includes moving the horizontal & vertical components of the principal point of the image.” Pugh)
performing a reverse transformation to warp the pixels of the estimated geometry back to the original viewpoint subsequent to inpainting the set of pixels. (Examiner interprets “estimated geometry” as anything geometry related. See paragraphs 63-66 and 194. Paragraph 194 shows that a reverse transformation can be applied after inpainting. Pugh)
As per claim 20, Pugh teaches The method of claim 16, wherein inpainting a set of pixels corresponding to at least a portion of the removal mask that overlaps the plane comprises:
extracting one or more texture regions from the plane based at least in part on a plane mask corresponding to the plane, the plane mask indicating the presence or absence of that plane at a plurality of pixel locations; and (See paragraphs 174-176 and 194. Paragraph 174 [0194] In an eleventh variant, determining a color for a replacement pixel includes performing texture synthesis to identify a texture of pixels surrounding the replacement pixels… Assigning replacement colors to replacement pixels can include one or more of cloning, CNN inpainting, propagating, or patch-matching colors of related regions (e.g., wall regions, floor regions, instances, classes) to the region of replacement pixels. However, texture synthesis can otherwise be performed. If the replacement pixels have had their depth replaced by depths that agree with or were drawn from an architectural plane (e.g., wall, floor) then the texture synthesis can be automatically sourced from elsewhere nearby on that plane. Further, the texture synthesis can be performed on a rectified version of the image of that plane and then returned to the image via perspective warping (homography)... Pugh)
inpainting the set of pixels based at least in part on the one or more texture regions. (See paragraph 194, examiner interprets “inpainting” as assigning replacement colors. “Assigning replacement colors to replacement pixels can include one or more of cloning, CNN inpainting, propagating, or patch-matching colors of related regions (e.g., wall regions, floor regions, instances, classes) to the region of replacement pixels. However, texture synthesis can otherwise be performed. If the replacement pixels have had their depth replaced by depths that agree with or were drawn from an architectural plane (e.g., wall, floor) then the texture synthesis can be automatically sourced from elsewhere nearby on that plane. Further, the texture synthesis can be performed on a rectified version of the image of that plane and then returned to the image via perspective warping (homography).” Pugh )
As per claim 21, Pugh teaches The method of claim 16, wherein inpainting a set of pixels corresponding to at least a portion of the removal mask that overlaps the plane comprises: inpainting the set of pixels with a pattern. (See paragraphs 174-176 and 194-195. Examiner interprets “inpainting” as assigning replacement colors. “[0194] In an eleventh variant, determining a color for a replacement pixel includes performing texture synthesis to identify a texture of pixels surrounding the replacement pixels. In some implementations, the identified texture is represented by a pattern of pixels having a specific assignment of colors… Assigning replacement colors to replacement pixels can include one or more of cloning, CNN inpainting, propagating, or patch-matching colors of related regions (e.g., wall regions, floor regions, instances, classes) to the region of replacement pixels. However, texture synthesis can otherwise be performed. ” Pugh)
As per claim 22, Pugh already teaches “the method of claim 16, wherein inpainting a set of pixels corresponding to at least a portion of the removal mask that overlaps the plane comprises:” Pugh also teaches “inpainting the set of pixels with a texture retrieved from an electronic texture bank.” (Examiner interprets “electronic texture bank” as any electronic component that stores texture. Paragraphs 183 and 194 already show texture inpainting, and paragraphs 145-146 and 148 shows the inclusion of a texture memory where the information for textures is stored. “[0146] In a second variation of S600, depth occlusion information and semantic segmentation information can be stored in texture memory (e.g., 601 shown in FIG. 10). In some implementations, the depth occlusion information (e.g., 602 shown in FIG. 10) and semantic segmentation information (e.g., 603 shown in FIG. 10) can be stored in the texture memory (e.g., 601) as components of a packed 3 or 4 component texture and used as a depth value and a write mask in a shader. ”)
As per claim 23, Pugh teaches “The method of claim 16, wherein inpainting a set of pixels corresponding to at least a portion of the removal mask that overlaps the plane comprises: inpainting the set of pixels with a neural-network produced texture that is generated based at least in part on one or more textures corresponding to one or more background objects in the image. (See paragraph 181 and 183. “[0181] In some implementations, S720 can include training a neural network (e.g., convolutional neural network, fully-connected neural network, generative neural network, feed forward neural network, etc.) such as a deep neural network (e.g., generative image inpainting with contextual attention) on dense depth maps with regions marked for removal supervised by dense depth maps with the correct replacement depth in these regions…” “[1083]… using texture synthesis techniques; using neural network inpainting techniques;”. See also paragraph 194 “[0194] In an eleventh variant, determining a color for a replacement pixel includes performing texture synthesis to identify a texture of pixels surrounding the replacement pixels… Assigning replacement colors to replacement pixels can include one or more of cloning, CNN inpainting…” Pugh )
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 2 and 11, are rejected under 35 U.S.C. 103 as being unpatentable over Pugh in view of Yoneda Keigo, hereafter Yoneda (US Publication No. 20210358204 A1).
As per claim 2, Pugh teaches “The method of claim 1, wherein the contextual information further comprises one or more of:
a gravity vector corresponding to the scene; (See paragraph 26 “…In some implementations, the base of the 3D location of the placed object is computed using relative pointer motion, the scene surface mesh, and the gravity vector sliding the object along the surface contour using physically representative mechanics and collisions…”)
an edge map corresponding to a plurality of edges in the scene; (See paragraph 28 “…In examples, improving the object boundary depths is accomplished by: identifying the edges within a dense (reasonably accurate) depth map (e.g., based on depth gradients, based on an edge map extracted from the same input image(s), based on a semantic segmentation map determined from the same input image(s), etc.); determining the object that the edges belong to (e.g., based on the semantic segmentation map); and correcting the edge depths based on the depth of the object that the edges belong to…”)
“a normal map corresponding to a plurality of normals in the scene;” (0112] In variants, regularizing geometries and segmentation S470 functions to jointly improve geometry (including planar surfaces) and segmentation. In many cases, segmentation can be used to improve geometry, and geometry can be used to improve segmentation. S470 can regularize the geometry using: a segmentation map (e.g., by regularizing geometries within the same segment), normal maps,…. See also paragraphs 104 and 114. Pugh )
an instance map indicating a plurality of instance labels associated with the plurality of pixels in the image; or a plurality of object masks corresponding to a plurality of objects in the scene.” (See paragraph 194 and see also paragraph 123 where it shows that object masks correspond to real objects “[0123] The data assets generated at S500 can be used to identify real objects in the rendered scene, and depths for each identified real object. For example, an object mask can identify pixels associated with each real object in the rendered scene. By generating object masks with clean depth edges, the object boundaries can more easily be identified.” Pugh) (Therefore, Pugh teaches at least one of the limitations presented.)
However Pugh does not teach “a shadow mask corresponding to a plurality of shadows in the scene;”
Yoneda teaches “a shadow mask corresponding to a plurality of shadows in the scene;” (See paragraph 6 and 40-42. “0006] According to the technique of the present disclosure, an image processing apparatus obtains foreground images and background images based on images obtained by a plurality of imaging apparatuses, generates shadow mask images by extracting shadow areas from the obtained foreground images,”)
It would have been obvious to one of ordinary skill in the art before the effective filing
date of the claimed invention to combine the teachings of Pugh with the teachings of Yoneda to include the use of a shadow mask. The modification would have been motivated by the desire to remove shadow areas from a foreground image and generate another viewpoint of the image, therefore it is an improvement, as suggested by Yoneda (See paragraphs 6 and 84 “[0084] In S907, the virtual viewpoint image generating unit 2022 generates a virtual viewpoint image in which a shadow is rendered using the shadow mask images received from the communication control unit 2021. Like the first embodiment, the virtual viewpoint image is generated by generating a virtual viewpoint foreground image and a virtual viewpoint background image, generating a shadow-added virtual viewpoint background image using the shadow mask images, and combining the virtual viewpoint foreground image with the shadow-added virtual viewpoint background image.”)
As per claim 11, Pugh already teaches “the method of claim 1, wherein the contextual information comprises… and wherein generating a removal mask corresponding to the at least one foreground object based at least in part on at least one object mask corresponding to the at least one foreground object comprises: combining at least a portion… with the at least one object mask corresponding to the at least one foreground object”, however Pugh does not completely teach “a shadow mask” and “combining at least a portion of the shadow mask with the at least one object mask corresponding to the at least one foreground object.”
Yoneda teaches “a shadow mask” and “combining at least a portion of the shadow mask with the at least one object mask corresponding to the at least one foreground object.” (See abstract, paragraphs 6, 84, 40-41 and fig. 9. Abstract “A virtual viewpoint foreground image generating unit generates a virtual viewpoint foreground image, which is an image of a foreground object seen from a virtual viewpoint without a shadow, based on received multi-viewpoint images and a received virtual viewpoint parameter. A virtual viewpoint background image generating unit generates a virtual viewpoint background image, which is an image of a background object seen from the virtual viewpoint, based on the received multi-viewpoint images and virtual viewpoint parameter. A shadow mask image generating unit generates shadow mask images from the received multi-viewpoint images. A shadow-added virtual viewpoint background image generating unit renders a shadow in the virtual viewpoint background image based on the received virtual viewpoint background image, shadow mask images, and virtual viewpoint parameter. A combined image generating unit generates a virtual viewpoint image by combining the virtual viewpoint foreground image with the shadow-added virtual viewpoint background image.”)
It would have been obvious to one of ordinary skill in the art before the effective filing
date of the claimed invention to combine the teachings of Pugh with the teachings of Yoneda to include the use of a shadow mask as contextual information and utilize the shadow mask to combine it with the foreground object mask. The modification would have been motivated by the desire to remove shadow areas from a foreground image and generate another viewpoint of the image, therefore it is an improvement, as suggested by Yoneda (See paragraphs 6 and 84 “[0084] In S907, the virtual viewpoint image generating unit 2022 generates a virtual viewpoint image in which a shadow is rendered using the shadow mask images received from the communication control unit 2021. Like the first embodiment, the virtual viewpoint image is generated by generating a virtual viewpoint foreground image and a virtual viewpoint background image, generating a shadow-added virtual viewpoint background image using the shadow mask images, and combining the virtual viewpoint foreground image with the shadow-added virtual viewpoint background image.”)
Claim 12 is rejected under 35 U.S.C. 103 as being unpatentable over Pugh in view of Chai et. al. , hereafter Chai (WO Publication No. 2022103431 A1).
As per claim 12, Pugh teaches “The method of claim 1, wherein generating a removal mask corresponding to the at least one foreground object based at least in part on at least one object mask corresponding to the at least one foreground object comprises:”, however Pugh does not teach “dilating the at least one object mask corresponding to the at least one foreground object by a predetermined quantity of pixels to thereby inflate the at least one object mask.”
Chai teaches “dilating the at least one object mask corresponding to the at least one foreground object by a predetermined quantity of pixels to thereby inflate the at least one object mask.” (See paragraph 4 “The operations include generating a foreground object mask for a foreground object in a first image; dilating the foreground object mask to obtain a dilated mask;”. See also paragraph 38 where it shows a predetermined size of pixels to expand. “[0038] At block 208, the process 200 involves dilating the foreground object masks. The image processing application 104 can dilate an object mask using a structuring element having a size N-by-N. In some examples, the size N of the structuring element is determined based on the disparities of the foreground object.”. See also paragraph 35-36 which show that the disparities are pixels. See also fig.3 which shows a dilated object mask. Pugh)
It would have been obvious to one of ordinary skill in the art before the effective filing
date of the claimed invention to combine the teachings of Pugh with the teachings of Chai to dilate an object mask with predetermined pixels. The modification would have been motivated by the desire to have less ghosting and have better quality, therefore it is an improvement, as suggested by Chai (See paragraphs 4 and 3 “[0004] Certain embodiments involve de-ghosting and see-through prevention for image fusion. In one example, a computer-implemented method includes receiving a first image and a second image; generating a foreground object mask for a foreground object in the first image; dilating the foreground object mask to obtain a dilated mask;…”. “[003] Image fusion is a process of combining the information from different sources of images into one compact form of an image. For example, a color image and a near-infrared (NIR) image can be fused to increase details of the color image due to extra information provided by the NIR image while preserving the color and brightness of the color image. However, when fusing multiple images, there may be ghosting artifacts in the combined image where multiple instances of the same object (i.e., ghosts) appear in the fused image. The ghosting artifacts are caused by, for example, a typical image misalignment associated with multi-image registration…”)
Claim 13 is rejected under 35 U.S.C. 103 as being unpatentable over Pugh in view of Chai and further in view of Yoneda.
As per claim 13, Pugh in view of Chai already teaches “the method of claim 12, wherein the contextual information comprises… and wherein generating a removal mask corresponding to the at least one foreground object based at least in part on at least one object mask corresponding to the at least one foreground object further comprises:... with the dilated at least one object mask corresponding to the at least one foreground object.
Pugh in view of Chai does not teach “a shadow mask” and “combining at least a portion of the shadow mask with the… at least one object mask corresponding to the at least one foreground object”
Yoneda teaches “a shadow mask” and “combining at least a portion of the shadow mask with the… at least one object mask corresponding to the at least one foreground object” (See abstract, paragraphs 6, 84, 40-41 and fig. 9. Abstract “A virtual viewpoint foreground image generating unit generates a virtual viewpoint foreground image, which is an image of a foreground object seen from a virtual viewpoint without a shadow, based on received multi-viewpoint images and a received virtual viewpoint parameter. A virtual viewpoint background image generating unit generates a virtual viewpoint background image, which is an image of a background object seen from the virtual viewpoint, based on the received multi-viewpoint images and virtual viewpoint parameter. A shadow mask image generating unit generates shadow mask images from the received multi-viewpoint images. A shadow-added virtual viewpoint background image generating unit renders a shadow in the virtual viewpoint background image based on the received virtual viewpoint background image, shadow mask images, and virtual viewpoint parameter. A combined image generating unit generates a virtual viewpoint image by combining the virtual viewpoint foreground image with the shadow-added virtual viewpoint background image.”)
It would have been obvious to one of ordinary skill in the art before the effective filing
date of the claimed invention to combine the teachings of Pugh and Chai with the teachings of Yoneda to include the use of a shadow mask as contextual information and utilize the shadow mask to combine it with the dilated foreground object mask. The modification would have been motivated by the desire to remove shadow areas from a foreground image and generate another viewpoint of the image, therefore it is an improvement, as suggested by Yoneda (See paragraphs 6 and 84 “[0084] In S907, the virtual viewpoint image generating unit 2022 generates a virtual viewpoint image in which a shadow is rendered using the shadow mask images received from the communication control unit 2021. Like the first embodiment, the virtual viewpoint image is generated by generating a virtual viewpoint foreground image and a virtual viewpoint background image, generating a shadow-added virtual viewpoint background image using the shadow mask images, and combining the virtual viewpoint foreground im