DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1-3, 7, and 8 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Sheng (“Controllable Shadow Generation Using Pixel Height Maps”).
As to claim 1, Sheng discloses a method comprising:
generating, utilizing a height prediction neural network (p. 7-8, section 3.2; a neural network is used for estimating/predicting pixel heights), an object height map comprising pixel heights for pixels of a digital object portrayed in a digital image (p. 7-8, section 3.2; an object such as a human has its pixel height mapped);
generating, utilizing the height prediction neural network, a background height map comprising additional pixel heights for pixels of a background portrayed in the digital image (fig. 11; p. 4-5, section 3; p. 5-7, section 3.1; p. 7-8, section 3.2; variables, including pixel height, are estimated; the variables of a background can either be input or estimated by the network; the output of such an estimation would be a pixel height map for the background object);
and modifying the digital image to include an object shadow for the digital object using the object height map and the background height map (fig. 11; p. 4-5, section 3; a shadow is added to the image using variables, including pixel height maps as noted above, of the object and background).
As to claim 2, Sheng discloses determining a set of ground points corresponding to the pixels of the background utilizing the additional pixel heights for the pixels of the background from the background height map (p. 1-2, section 1; p. 5-6, section 3.1; p. 7-8, section 3.2; a set of footpoints, which read on ground points corresponding to the pixel height, are part of the variables determined for each element, which would include the background and the object, each with its associated map), wherein modifying the digital image to include the object shadow for the digital object using the background height map comprises generating the object shadow for the digital object utilizing the set of ground points (p. 1-2, section 1; p. 5-6, section 3.1; p. 7-8, section 3.2; the footpoints/ground points are part of the formula for shadow points).
As to claim 3, Sheng discloses determining an additional set of ground points corresponding to the pixels of the digital object utilizing the pixel heights for the pixels of the digital object from the object height map (p. 1-2, section 1; p. 5-6, section 3.1; p. 7-8, section 3.2; a set of footpoints, which read on ground points corresponding to the pixel height, are part of the variables determined for each element, which would include the background and the object, each with its associated map), wherein modifying the digital image to include the object shadow for the digital object using the background height map comprises generating the object shadow for the digital object utilizing the additional set of ground points (p. 1-2, section 1; p. 5-6, section 3.1; p. 7-8, section 3.2; the footpoints/ground points are part of the formula for shadow points).
As to claim 7, Sheng discloses wherein modifying the digital image to include the object shadow for the digital object using the object height map and the background height map comprises generating a hard object shadow for the digital object using the object height map and the background height map (p. 4-5, section 3; p. 5-7, section 3.1; p. 7-8, section 3.2; the shadow rendered is a hard shadow).
As to claim 8, Sheng discloses wherein modifying the digital image to include the object shadow for the digital object comprises generating, utilizing a shadow stylization neural network, a soft object shadow for the digital object based on the hard object shadow (p. 8-9, section 3.3; the hard shadow is input to a soft shadow neural network to stylize the shadow to a particular softness).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
Claims 4 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Sheng in view of Snyder (U.S. Publication 2009/0309877).
As to claim 4, Sheng does not expressly disclose, but Snyder discloses determining that a pixel of the digital object is blocking light from reaching a pixel of the background within a three-dimensional space corresponding to the digital image utilizing the object height map and the background height map, wherein modifying the digital image to include an object shadow for the digital object using the object height map and the background height map comprises modifying the pixel of the background to include a shadow pixel based on determining that the pixel of the digital object is blocking the light from reaching the pixel of the background (fig. 1a; p. 2, sections 0019-0025; based on height fields/maps of the mountain peaks in the figure, one of which can read on object and another on background, shadows are cast to background objects based on how the object blocks light to pixels in the background). The motivation for this is to provide a user with a more realistic virtual experience while placing fewer demands on computational resources (p. 1, section 0004). It would have been obvious to one skilled in the art before the effective filing date of the claimed invention to modify Sheng to determine that a pixel of the digital object is blocking light from reaching a pixel of the background within a three-dimensional space corresponding to the digital image utilizing the object height map and the background height map, wherein modifying the digital image to include an object shadow for the digital object using the object height map and the background height map comprises modifying the pixel of the background to include a shadow pixel based on determining that the pixel of the digital object is blocking the light from reaching the pixel of the background in order to provide a user with a more realistic virtual experience while placing fewer demands on computational resources as taught by Snyder.
As to claim 13, Sheng does not disclose, but Snyder discloses wherein determining the set of pixels from the pixels of the background that correspond to the shadow projection of the digital object within the digital image, for a pixel of the background, comprises: determining a line between the pixel of the background and a light source of the digital image; identifying one or more pixels of the digital object that are positioned on the line; and determining whether at least one pixel from the one or more pixels of the digital object is blocking light from reaching the pixel of the background (fig. 1a; p. 2, sections 0019-0025; based on height fields/maps of the mountain peaks in the figure, one of which can read on object and another on background, shadows are cast to background objects based on how the object blocks light to pixels in the background; a line is plotted between a background object and a point in the sky where light is originating; the pixel at the top of the blocking object is identified on the line, and that pixel and pixels below it are identified as blocking light coming from particular angles). Motivation for the combination is given in the rejection to claim 4.
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over Sheng in view of Snyder and further in view of Trojahner (U.S. Publication 2023/0056589).
As to claim 5, Sheng discloses wherein determining that the pixel of the digital object is blocking the light from reaching the pixel of the background within the three-dimensional space corresponding to the digital image utilizing the object height map and the background height map (fig. 11; p. 1-2, section 1; p. 4-5, section 3; in a 3D model, a shadow is added to the image using variables, including pixel height maps of the object and background; shadowing is a determination of blocked light) comprises: determining a ground point corresponding to the pixel of the digital object within the three-dimensional space utilizing a pixel height for the pixel of the digital object from the object height map; determining a ground point corresponding to the pixel of the background within the three-dimensional space utilizing a pixel height for the pixel of the background from the background height map (p. 1-2, section 1; p. 5-6, section 3.1; p. 7-8, section 3.2; a set of footpoints, which read on ground points corresponding to the pixel height, are part of the variables determined for each element, which would include the background and the object, each with its associated map; the footpoints/ground points are part of the formula for shadow points);
Sheng does not expressly disclose, but Trojahner discloses determining that the pixel of the digital object is blocking the light from reaching the pixel of the background utilizing the ground point corresponding to the pixel of the digital object and the ground point corresponding to the pixel of the background (p. 4, sections 0043-0045; based on ground position and height at the ground position, an object pixel and a background pixel are analyzed to see if a light ray can reach the further back object or if it is blocked). The motivation for this is to determine which areas are prevented from data gathering (p. 1, section 0015). It would have been obvious to one skilled in the art before the effective filing date of the claimed invention to modify Sheng and Snyder to determine that the pixel of the digital object is blocking the light from reaching the pixel of the background utilizing the ground point corresponding to the pixel of the digital object and the ground point corresponding to the pixel of the background in order to determine which areas are prevented from data gathering as taught by Trojahner.
Claims 9, 12, and 17-20 are rejected under 35 U.S.C. 103 as being unpatentable over Sheng.
As to claim 9, see the rejection to claim 1. Further, while Sheng discloses a GPU-based rendering using a CUDA platform (p. 5-7, section 3.1; p. 8-9, section 3.3), Shen does not explicitly disclose that instructions executed by the processing device are stored on a non-transitory medium. However, Official Notice has been taken that use of a non-transitory computer-readable medium storing executable instructions to cause a processing device to perform operations is well-known in the art (see MPEP 2144.03), with the motivation being that using a non-transitory medium can store the instructions for repeated use.
As to claim 12, Sheng discloses wherein: the operations further comprise determining a set of pixels from the pixels of the background that correspond to a shadow projection of the digital object within the digital image using the object height map and the background height map; and modifying the digital image to include the object shadow for the digital object using the object height map and the background height map comprises modifying the set of pixels from the pixels of the background to include shadow pixels based on determining that the set of pixels correspond to the shadow projection of the digital object (fig. 4; fig. 11; p. 5-7, section 3.1; using the pixel height maps, a projection is made from the height of the digital object to the height of the background shadow receiver, and pixels that are in shadow are modified according to their visibility).
As to claim 17, see the rejections to claims 1-3 and 9. Further, Sheng discloses that the background is non-planar (fig. 11; p. 5-7, section 3.1).
As to claim 18, Sheng discloses wherein the one or more processing devices further perform operations comprising: determining a vertical movement of the digital object within the digital image; and adding a shift value to the pixel heights for the pixels of the digital object within the object height map in accordance with the vertical movement of the digital object (fig. 11; an object is determined to be floating, i.e. moved vertically, above the ground, and this is dealt with by adding a shift value to the pixel height map for the object at the appropriate pixels).
As to claim 19, Sheng discloses wherein the one or more processing devices further perform operations comprising modifying the object shadow across the non-planar background within the digital image utilizing the background height map and the object height map with the shift value (fig. 11; as can be seen from the figure, the shadow is raised and extended on the non-planar background after the shift value is added)
As to claim 20, see the rejections to claims 1-3, 9, and 17.
Claims 10 and 11 are rejected under 35 U.S.C. 103 as being unpatentable over Sheng in view of Srivastava (U.S. Patent 11,030,763).
As to claim 10, Sheng discloses generating, utilizing a height prediction neural network, the background height map (fig. 11; p. 4-5, section 3; p. 5-7, section 3.1; p. 7-8, section 3.2; variables, including pixel height, are estimated; the variables of a background can either be input or estimated by the network; the output of such an estimation would be a pixel height map for the background object). Sheng does not disclose, but Srivastava discloses extracting, from the digital image, a background mask corresponding to the background portrayed in the digital image (col. 9, lines 39-56; a background region mask is extracted using foreground/background segmentation); and determining the background height map comprises generating, the background height map based on the background mask (col. 9, lines 4-38; each individual region, which would include the background, has a generated height map based on a region mask applied to a geometric representation). The motivation for this is that the use of height mapping and segmentation/masking together can increase speed and accuracy (col. 2, lines 16-43). It would have been obvious to one skilled in the art before the effective filing date of the claimed invention to modify Sheng to extract, from the digital image, a background mask corresponding to the background portrayed in the digital image and determine the background height map comprising generating, the background height map based on the background mask in order to increase speed and accuracy as taught by Srivastava.
As to claim 11, Sheng discloses generating, utilizing a height prediction neural network, the background height map (fig. 11; p. 4-5, section 3; p. 5-7, section 3.1; p. 7-8, section 3.2; variables, including pixel height, are estimated; the variables of a background can either be input or estimated by the network; the output of such an estimation would be a pixel height map for the background object). Sheng does not disclose, but Srivastava discloses determining at least one of a background cutout for the background portrayed in the digital image or a coordinate scheme for the digital image (col. 9, lines 4-56; a background region mask is extracted using foreground/background segmentation; the mask is defined by a bounding box, which reads on a cutout, or coordinate boundaries, which would read on a coordinate scheme); and generating, the background height map comprises generating the background height map based on the background cutout or the coordinate scheme (col. 9, lines 4-38; each individual region, which would include the background, has a generated height map based on a region mask applied to a geometric representation; the region mask is defined by a bounding box reading on a cutout or a coordinate scheme, as discussed above). Motivation for the combination is given in the rejection to claim 10.
Conclusion
Claims 6 and 14-16 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
As to claim 6, Sheng discloses, determining that the pixel of the digital object is blocking the light from reaching the pixel of the background utilizing the ground point corresponding to the pixel of the digital object and the ground point corresponding to the pixel of the background (fig. 11; p. 1-2, section 1; p. 4-5, section 3; p. 5-7, section 3.1; in a 3D model, a shadow is added to the image using variables, including pixel height maps of the object and background; shadowing is a determination of blocked light; the footpoints/ground points are part of the formula for shadow points). Sheng does not disclose that this comprises: determining a line between the ground point corresponding to the pixel of the background and a ground point corresponding to a light source of the digital image; and determining that the ground point corresponding to the pixel of the digital object is positioned on the line. In Sheng, specifically fig. 4, a line between ground points of the light source and foreground object is plotted, and then this line is projected towards a shadow point. There is no determination of whether the foreground object is positioned on a line between ground point of light source and ground point of background. Similar analysis applies to claim 14.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to AARON M RICHER whose telephone number is (571)272-7790. The examiner can normally be reached 9AM-5PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, King Poon can be reached at (571)272-7440. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/AARON M RICHER/Primary Examiner, Art Unit 2617
Prosecution Insights