CTNF 18/725,698 CTNF 91949 DETAILED ACTION Notice of Pre-AIA or AIA Status 07-03-aia AIA 15-10-aia The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA. 07-06 AIA 15-10-15 In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status. Claim Rejections - 35 USC § 103 07-20-aia AIA The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action: A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made. 07-23-aia AIA The factual inquiries set forth in Graham v. John Deere Co. , 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows: 1. Determining the scope and contents of the prior art. 2. Ascertaining the differences between the prior art and the claims at issue. 3. Resolving the level of ordinary skill in the pertinent art. 4. Considering objective evidence present in the application indicating obviousness or nonobviousness. 07-21-aia AIA Claims 1, 2, 3, 5 – 8, 10, 11, 13, 14, and 16 – 22 are rej ected under 35 U.S.C. 103 as being unpatentable over Iwa o (Publication: US 2018/0241714 A1) in view of Sharma et al. (Publication: US 10664722 B1). Reg arding claim 1, see rejection on claim 10. Regarding claim 2, Iwao in view of Sharma disclose all the limitation of claim 1. Iwao discloses sequentially obtaining at least one video frame to be processed from a target video ( [0040], [0041], Fig. 4 – Acquired image in S401, video, group of images. ); determining gradient information, in a first direction and a second direction, of at least one pixel of the video frame to be processed to obtain normal information of the at least one pixel ( [0041] – determined the three types of gradient illumination patters; the luminances of multiple illumination (a plurality of point light sources 205) which are discretely arranged and of which the luminances can be continuously changed as in the light source apparatus 202 are smoothly changed in x-, y-, and z-axis directions orthogonal to one another, “ a first direction and a second direction ”, thereby performing illumination in three types of gradient illumination patterns. That is, one type of pattern is created by smoothly changing the luminances in the x-axis direction. Then, one type of pattern is created by smoothly changing the luminances in the y-axis direction. ) ; and obtaining the target normal map based on the normal information of the at least one pixel ( [0052], [0053] - Fig. 4 – S403 Calculate Normal with respect to each pixel and A condition for determining the necessity of repetitive processing in S404. [0056] In step S405, the height map calculation unit 306 calculates a height map from the normal output ). Regarding claim 3, Iwao in view of Sharma disclose all the limitation of claim 1. Iwao discloses obtaining the video frame to be processed ( [0040], [0041], Fig. 4 – Acquired image in S401, video, group of images. ), a target segmentation region corresponding to the video frame to be processed ( [0040], [0064] - In step S401, the image acquisition unit 301 acquires an image of the object 203 as a processing target from the image capturing apparatus 201, where the head (the face) of a person seated in a chair is a normal calculation target object. . ); and determining gradient information, in a first direction and a second direction, of at least one pixel in the target segmentation region to obtain normal information of the at least one pixel, and determining, based on the normal information, the target normal map of the video frame to be processed ( [0041] – determined the three types of gradient illumination patters; the luminances of multiple illumination (a plurality of point light sources 205) which are discretely arranged and of which the luminances can be continuously changed as in the light source apparatus 202 are smoothly changed in x-, y-, and z-axis directions orthogonal to one another, “ a first direction and a second direction ”, thereby performing illumination in three types of gradient illumination patterns. That is, one type of pattern is created by smoothly changing the luminances in the x-axis direction. Then, one type of pattern is created by smoothly changing the luminances in the y-axis direction. [0052], [0053] - Fig. 4 – S403 Calculate Normal with respect to each pixel and A condition for determining the necessity of repetitive processing in S404. [0056] In step S405, the height map calculation unit 306 calculates a height map from the normal output ). Sharma discloses determining, based on a pre-trained image segmentation model ( Column 16 lines 31 - After such re-training, the process can be repeated. A different set of real-world images harvested from Flickr is presented to the network. Again, each is ranked by its top-neuron score. The top-ranked images are used to again re-train the network—either defining an entirely new Class 0 training set, or replacing respective ones of those earlier Class 0 training images having lower scores. This process can repeat for several cycles. Again, network accuracy—particularly false positive behavior—is further improved. ). Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to modify Iwao in view Sharma with determining, based on a pre-trained image segmentation model as taught by Sharma. The motivation for doing is to mitigate recognition difficulties. Regarding claim 5, Iwao in view of Sharma disclose all the limitation of claim 1. Iwao discloses for each of the at least one pixel: determining target normal information corresponding to a current pixel in the target normal map ( [0052], [0053] - Fig. 4 – S403 Calculate Normal with respect to each pixel and A condition for determining the necessity of repetitive processing in S404. [0056] In step S405, the height map calculation unit 306 calculates a height map from the normal output ), and determining target lighting intensity information of the current pixel based on the target normal information, the attribute information of the light source , and shooting angle information of a video frame to which the current pixel belongs ( [0045] - a point light source 205 emitting specularly reflected light to be regularly reflected from the object 203 and incident on the image capturing apparatus 201, with respect to each pixel. If a specular reflection coefficient at each pixel (x′, y′) is ρ(x′, y′), a luminance (a reflection intensity) I observed using the image capturing apparatus 201 can be represented with respect to each pixel by formula (3) in the case of gradient illumination, and can be represented by formula (4) in the case of uniform illumination. PNG media_image1.png 164 358 media_image1.png Greyscale ). Regarding claim 6, Iwao in view of Sharma disclose all the limitation of claim 5. Iwao discloses determining light direction information of the current pixel based on the position information of the light source ( [0041], [0042] - The image (the normal calculation image) acquired in step S401 is a group of images obtained by imaging the object 203 illuminated by the light source apparatus 202 lit up according to a predetermined rule (predetermined illumination patterns) by the control unit 302. . A radiance L.sub.1 of a point light source 205 present in a direction at an angle θ.sub.1 when viewed from the center O of the light source apparatus 202 can be represented by formula (1), using a unit vector ω, which indicates the direction of a ray incident on the center O of the light source apparatus 202, and a constant c. In formula (1), i represents either of x, y, and z. Further, θ.sub.i represents the angle between a straight line connecting the center O of the light source apparatus 202 and the point light source 205, and the i-axis. The center O of the light source apparatus 202 is also the center of the object 203. ); determining a diffuse lighting value of the current pixel based on the light direction information, the target normal information, and a preset diffuse coefficient ( [0045] – S402 calculate position of point light source with respect to each pixel, based on the group of normal calculation images, the light source position calculation unit 303 calculates the position (hereinafter referred to as a “light source position”) of, among the plurality of point light sources 205, a point light source 205 emitting specularly reflected light to be regularly reflected from the object 203 and incident on the image capturing apparatus 201, with respect to each pixel. If a specular reflection coefficient at each pixel (x′, y′) is ρ(x′, y′), a luminance (a reflection intensity) I observed using the image capturing apparatus 201 can be represented with respect to each pixel by formula (3) in the case of gradient illumination, and can be represented by formula (4) in the case of uniform illumination. PNG media_image2.png 156 352 media_image2.png Greyscale [0002] - the reflected light from the normal calculation target object is made incident on an image capturing apparatus. Then, diffusely reflected light is excluded from the reflected light, and only specularly reflected light is extracted from the reflected light. ); determining a target reflection angle based on the light direction information and the target normal information, and determining a reflection intensity value of the current pixel based on the target reflection angle, the shooting angle information and a preset reflection coefficient ( [0042] A radiance L.sub.1 of a point light source 205 present in a direction at an angle θ.sub.1 when viewed from the center O of the light source apparatus 202 can be represented by formula (1), using a unit vector ω, which indicates the direction of a ray incident on the center O of the light source apparatus 202, and a constant c. In formula (1), i represents either of x, y, and z. Further, θ.sub.i represents the angle between a straight line connecting the center O of the light source apparatus 202 and the point light source 205, and the i-axis. The center O of the light source apparatus 202 is also the center of the object 203, “ determining a target reflection angle ”. [0045] – S402 calculate position of point light source with respect to each pixel, based on the group of normal calculation images, the light source position calculation unit 303 calculates the position (hereinafter referred to as a “light source position”) of, among the plurality of point light sources 205, a point light source 205 emitting specularly reflected light to be regularly reflected from the object 203 and incident on the image capturing apparatus 201, with respect to each pixel. If a specular reflection coefficient at each pixel (x′, y′) is ρ(x′, y′), a luminance (a reflection intensity) I observed using the image capturing apparatus 201 can be represented with respect to each pixel by formula (3) in the case of gradient illumination, and can be represented by formula (4) in the case of uniform illumination. PNG media_image2.png 156 352 media_image2.png Greyscale ); and determining the target lighting intensity information based on the diffuse lighting value, the reflection intensity value, and an ambient lighting intensity value corresponding to the attribute information of the light source ( [0045] – S402 calculate position of point light source with respect to each pixel, based on the group of normal calculation images, the light source position calculation unit 303 calculates the position (hereinafter referred to as a “light source position”) of, among the plurality of point light sources 205, a point light source 205 emitting specularly reflected light to be regularly reflected from the object 203 and incident on the image capturing apparatus 201, with respect to each pixel. If a specular reflection coefficient at each pixel (x′, y′) is ρ(x′, y′), a luminance (a reflection intensity) I observed using the image capturing apparatus 201 can be represented with respect to each pixel by formula (3) in the case of gradient illumination, and can be represented by formula (4) in the case of uniform illumination. PNG media_image2.png 156 352 media_image2.png Greyscale [0002] - the reflected light from the normal calculation target object is made incident on an image capturing apparatus. Then, diffusely reflected light is excluded from the reflected light, and only specularly reflected light is extracted from the reflected light. ) Regarding claim 7, Iwao in view of Sharma disclose all the limitation of claim 6. Iwao discloses determining the target lighting intensity information as a value among the diffuse lighting value, the reflection intensity value, and the ambient lighting intensity value ( [0045] – S402 calculate position of point light source with respect to each pixel, based on the group of normal calculation images, the light source position calculation unit 303 calculates the position (hereinafter referred to as a “light source position”) of, among the plurality of point light sources 205, a point light source 205 emitting specularly reflected light to be regularly reflected from the object 203 and incident on the image capturing apparatus 201, with respect to each pixel. If a specular reflection coefficient at each pixel (x′, y′) is ρ(x′, y′), a luminance (a reflection intensity) I observed using the image capturing apparatus 201 can be represented with respect to each pixel by formula (3) in the case of gradient illumination, and can be represented by formula (4) in the case of uniform illumination. PNG media_image2.png 156 352 media_image2.png Greyscale [0002] - the reflected light from the normal calculation target object is made incident on an image capturing apparatus. Then, diffusely reflected light is excluded from the reflected light, and only specularly reflected light is extracted from the reflected light. ). Sharma discloses determining the target lighting intensity information as a maximum value ( column 7 line 5 to go to the maximum possible value, causing incident light to the reflected onto pixel of the camera sensor at an intensity that saturates the photosensor. ). Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to modify Iwao in view Sharma with determining the target lighting intensity information as a maximum value as taught by Sharma. The motivation for doing is to mitigate recognition difficulties. Regarding claim 8, Iwao in view of Sharma disclose all the limitation of claim 1. Iwao discloses wherein determining the display information of the at least one pixel based on the target lighting intensity information of the at least one pixel comprises: Iwao discloses updating the display information of the at least one pixel based on the target lighting intensity information and pixel value information of the at least one pixel ( [0036], [0048] In step S403, the normal calculation unit 304 calculates a normal with respect to each pixel in the image from the depth of the object 203, the position of the image capturing apparatus 201, and the light source position output from the depth calculation unit 307. The depth of the object 203 can be used as information indicating the normal acquisition position 206, “ based on the target lighting intensity information and pixel value information ”. [0002] - based on the intensity of the specularly reflected light, the position of a light source emitting light to be regularly reflected from the object and incident on the image capturing apparatus is calculated, “ target lighting intensity information ”. Fig. 1 - The output data from the output interface to display. ). 9. (Canceled). Regarding claim 10, Iwao discloses an electronic device, comprising: at least one processor; and a memory storing at least one program; wherein the at least one processor, when executing the at least one program, is caused to implement a method comprising ( [0015], [0016] - FIG. 1 is a diagram illustrating an example of the configuration of an image processing apparatus 100. The image processing apparatus 100 includes a central processing unit (CPU) 101, a random-access memory (RAM) 102, a read-only memory (ROM), memory stores software processed by the CPU to perform the following methods: ): obtaining a video frame to be processed and determining a target normal map of the video frame to be processed ( [0040], [0041], Fig. 4 – Acquired image in S401, video, group of images. S401, acquire Normal Calculatin Image . The image acquisition unit 301 acquires an image of the object 203 as a processing target from the image capturing apparatus 201. That is, the image acquisition unit 301 acquires a normal calculation image from the image capturing apparatus 201. Then, the image acquisition unit 301 outputs the normal calculation image to the light source position calculation unit 303. ); determining, based on the target normal map and a light source ( [0045] - in S402, calculate position of point light source with respect to each pixel are based on previous S401 Acquire normal calculation image and extract specular reflect light component. ), target lighting intensity information of at least one pixel of the video frame to be processed ( [0045] - a point light source 205 emitting specularly reflected light to be regularly reflected from the object 203 and incident on the image capturing apparatus 201, with respect to each pixel. If a specular reflection coefficient at each pixel (x′, y′) is ρ(x′, y′), a luminance (a reflection intensity) I observed using the image capturing apparatus 201 can be represented with respect to each pixel by formula (3) in the case of gradient illumination, and can be represented by formula (4) in the case of uniform illumination. PNG media_image1.png 164 358 media_image1.png Greyscale ); and determining display information of the at least one pixel based on the target lighting intensity information of the at least one pixel, to determine, based on the display information, a target video frame corresponding to the video frame to be processed ( Fig. 1 - The output data from the output interface to display. [0002], [0036], [0048] In step S403, the normal calculation unit 304 calculates a normal with respect to each pixel in the image from the depth of the object 203, the position of the image capturing apparatus 201, and the light source position output from the depth calculation unit 307. based on the intensity of the specularly reflected light, the position of a light source emitting light to be regularly reflected from the object and incident on the image capturing apparatus is calculated, “ based on the target lighting intensity information of the at least one pixel”. As shown in Fig. 4 – the S404 threshold for display is based on the S403 “ calculate normal with respect to each pixel image ”, S405 “ Acquired Height Map of Object ” and S406 . Fig. 1 - The output data from the output interface to display. ). Iwao does not Sharma discloses determined based on preset attribute information of a light source ( column 35 line 46 - Object can be confounding for machine-recognition due to the myriad difference appearances such region can take – depending on viewing angle, lighting, and objects behinds Such region are detected and to mitigate recognition difficulties. ). Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to modify Iwao with determined based on preset attribute information of a light source as taught by Sharma. The motivation for doing is to mitigate recognition difficulties. Regarding claim 11, see rejection on claim 10. Regarding claim 13, see rejection on claim 2. Regarding claim 14, see rejection on claim 3. Regarding claim 16, see rejection on claim 5. Regarding claim 17, see rejection on claim 6. Regarding claim 18, see rejection on claim 7. Regarding claim 19, see rejection on claim 8. Regarding claim 20, see rejection on claim 2. Regarding claim 21, see rejection on claim 3. Regarding claim 22, see rejection on claim 4 . 07-21-aia AIA Claim s 4 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Iwao (Publication: US 2018/0241914 A1) in view of Sharma et al. (Publication: US 10664722 B1), and Schwarta et al. (US 2014/0146139 A1) . Regarding claim 4, Iwao in view of Sharma disclose all the limitation of claim 2. Iwao discloses gradient information, in the first direction and the second direction, of each pixel in the video frame to be used, and determining the normal information of the at least one pixel ( [41] – performing illumination in three types of gradient illumination patterns. That is, one type of pattern is created by smoothly changing the luminances in the x-axis direction. Then, one type of pattern is created by smoothly changing the luminances in the y-axis direction. Then, one type of pattern is created by smoothly changing the luminances in the z-axis direction. Then, the object 203 is irradiated with light. [0045] In step S402, based on the group of normal calculation images, the light source position calculation unit 303 calculates the position (hereinafter referred to as a “light source position”) of, among the plurality of point light sources 205, a point light source 205 emitting specularly reflected light to be regularly reflected from the object 203 and incident on the image capturing apparatus 201, with respect to each pixel. If a specular reflection coefficient at each pixel (x′, y′) is ρ(x′, y′), a luminance (a reflection intensity) I observed using the image capturing apparatus 201 can be represented with respect to each pixel by formula (3) in the case of gradient illumination, and can be represented by formula (4) in the case of uniform illumination. PNG media_image3.png 26 324 media_image3.png Greyscale ). Iwao in view of Sharma do not however Schwartz discloses filtering, via joint bilateral filtering, the video frame to be processed to obtain a video frame to be used ( [0032], [0037] - FIG. 5 shows results for depth map upscaling a factor 8 for the Middlebury "Venus" set. (a) shows the original depth, (b) a raw nearest neighbor upscaling. (c) is upscaled using JBU and (d) upscaled with the herein suggested technology (EWOC). Upscaling of video depth. ); and determining, using a Sobel operator ( [0043] - The full resolution texture image edge map E.sub.I can further be improved by adding the results of horizontal and vertical Sobel filtering of the full resolution texture image I. where C stands for the edge detection results on the different color channels. S.sub.x and S.sub.y stand for the results of the horizontal and vertical Sobel operator respectively. ) . Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to modify Iwao in view Sharma with filtering, via joint bilateral filtering, the video frame to be processed to obtain a video frame to be used; and determining, using a Sobel operator as taught by Schwartz. The motivation for doing is to enable enjoy the quality of 3D experience. Regarding claim 15, see rejection on claim 4 . Conclusion 07-96 AIA The prior art made of record and not relied upon is considered pertinent to applicant's disclosure . Any inquiry concerning this communication or earlier communications from the examiner should be directed to MING WU whose telephone number is (571)270-0724. The examiner can normally be reached on Monday - Thursday and alternate Fridays: 9:30am - 6:00pm EST . Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice. If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Devona Faulk can be reached on 571-272-7515. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300. Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000. /MING WU/ Primary Examiner, Art Unit 2618 Application/Control Number: 18/725,698 Page 2 Art Unit: 2618
Prosecution Insights