DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
1. The information disclosure statement (IDS) submitted on 12/20/2024 is in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement has been considered by the examiner.
Drawings
2. The drawings are objected to as failing to comply with 37 CFR 1.84(p)(5) because they do not include the following reference sign(s) mentioned in the description: 400 of [0077], line 3. Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
3. The drawings are objected to as failing to comply with 37 CFR 1.84(p)(5) because they include the following reference character(s) not mentioned in the description: 500 in Fig. 5. Corrected drawing sheets in compliance with 37 CFR 1.121(d), or amendment to the specification to add the reference character(s) in the description in compliance with 37 CFR 1.121(b) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
4. The drawings are objected to as failing to comply with 37 CFR 1.84(p)(4) because reference character “606” has been used to designate both "m and "operating system 606. Corrected drawing sheets in compliance with 37 CFR 1.121(d) are required in reply to the Office action to avoid abandonment of the application. Any amended replacement drawing sheet should include all of the figures appearing on the immediate prior version of the sheet, even if only one figure is being amended. Each drawing sheet submitted after the filing date of an application must be labeled in the top margin as either “Replacement Sheet” or “New Sheet” pursuant to 37 CFR 1.121(d). If the changes are not accepted by the examiner, the applicant will be notified and informed of any required corrective action in the next Office action. The objection to the drawings will not be held in abeyance.
Specification
5. The disclosure is objected to because of the following informalities:
In [0038], line 3, "values of the the target point cloud is calculated" should read "values of the target point cloud are calculated"
In [0042], line 1, "pixels of the initial color image is processed" should read "pixels of the initial color image are processed"
In [0046], line 2, "there are a hole" should read "there is a hole"
Appropriate correction is required.
Claim Objections
6. Claims 2, 8 and 14 are objected to because of the following informalities:
In claim 2, line 2, "target depth image" in the preamble appears not to be correct as the relative displacement appears (and is assumed for examination) to correspond to an original image or color image instead of a depth image. Appropriate revision is advised.
In claim 2, line 9, "according to initial" should read "according to the initial"
In claim 8, line 4 and claim 14, line 4, "corresponding to the target original image" should read "corresponding to a target original image"
Appropriate correction is required.
Claim Rejections - 35 USC § 103
7. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
8. Claims 1, 6-7, 12-13, and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Chen et al. (US-2014/0063206-A1, hereinafter "Chen") in view of Neill (US-2014/0125650-A1), and further in view of Schmalstieg et al. (US-2019/0066365-A1, hereinafter "Schmalstieg").
9. As per claim 1, Chen discloses: A 3D image generating method, comprising: (Chen, [0009], “In view of the foregoing, it is an object of the embodiment of the present invention to provide a system and method of viewer centric depth adjustment for stereoscopic images to adjust both object size and disparity map, therefore providing better viewing experience by minimizing the distortion mentioned above.” and [0022], “According to the stereoscopic image adjustment scheme discussed above, absolute disparity remapping for specific viewing distance is first adopted, and then image resizing is applied to deal with inconsistency of monocular cues and binocular cues, therefore improving 3D viewing experience.”)
obtaining an initial color image and an initial depth image [[from a back-end cache,]] the initial color image and the initial depth image being associated with each other; (Chen, [0010], “According to one embodiment, a system of viewer centric depth adjustment includes a sensor and a depth remapping unit. The sensor is configured to measure viewing distance of a viewer from a screen. The depth remapping unit is configured to receive a color image, a depth map and the viewing distance, and accordingly to remap depth values of the depth map such that the viewer perceives same depth at different viewing distances.”)
remapping the initial depth image to obtain a target depth image; (Chen, [0015], “Based on the color image, the depth map and the viewer distance, the depth remapping unit 21 remaps (or shifts) depth values of the depth map such that the viewer perceives the same depth at different viewing distances.” and [0017]-[0018], “The depth remapping unit 21 also includes a disparity shift unit 212 that is configured to shift image disparity without changing viewer's perceived depth. ... According to the shifted image disparity do, a new perceived depth Zo may be obtained.”)
determining a relative displacement corresponding to the initial color image; (Chen, [0019]-[0020], “Referring to FIG. 2, the system 200 further includes an image warping unit 23 utilized to prevent size distortion while applying depth shifting, compared with a conventional system that seldom takes image warping into account and thus normally causes size distortion. To deal with this artifact, the embodiment proposes image resizing algorithm to reconstruct the original scale. Geometry assumption is used to get the scaling ratio. When a viewer moves close to the objects, the viewer feels the nearer the object, the bigger it grows in the real world viewing experience. Besides, for the scene far from the viewer, it almost remains the same size. Using the geometry relation we can solve the resizing ratio for each object. In the embodiment, the scaling ratio may be decided by equations as following:
r
a
t
i
o
=
Z
i
Z
×
D
+
V
D
Z
o
where ratio is the scaling ratio, xi is original horizontal position, xo is virtual position, and width is extent of a screen from side to side. In other words, the image warping unit 23 first calculates the original horizontal position xi and then calculates the virtual position xo according to the viewer's movement.”)
rendering the initial color image based on the relative displacement and the target depth image to obtain a target color image; and (Chen, [0020]-[0022], “In other words, the image warping unit 23 first calculates the original horizontal position xi and then calculates the virtual position xo according to the viewer's movement. Finally, the color image and the depth map processed by the depth remapping unit 21 and the image warping unit 23 are then forwarded to a depth image-based rendering (DIBR) unit 24. … According to the stereoscopic image adjustment scheme discussed above, absolute disparity remapping for specific viewing distance is first adopted, and then image resizing is applied to deal with inconsistency of monocular cues and binocular cues, therefore improving 3D viewing experience.”)
generating a 3D image based on the target color image [[and the initial color image.]] (Chen, [0021]-[0022], “Finally, the color image and the depth map processed by the depth remapping unit 21 and the image warping unit 23 are then forwarded to a depth image-based rendering (DIBR) unit 24. ... According to the stereoscopic image adjustment scheme discussed above, absolute disparity remapping for specific viewing distance is first adopted, and then image resizing is applied to deal with inconsistency of monocular cues and binocular cues, therefore improving 3D viewing experience.” and [0003], “The present invention generally relates to stereoscopic images, and more particularly to a system and method of viewer centric depth adjustment for stereoscopic images.”)
10. Chen doesn't explicitly disclose but Neill discloses: [[generating a 3D image based on the target color image]] and the initial color image. (Neill, [0004], “Stereoscopic imagery is created by generating separate image frames for the left eye and the right eye, creating a three-dimensional viewing experience. In order to reduce the rendering and transmission time needed to generate two separate image streams, stereoscopic imagery may be accomplished by rendering images and depth information for one eye. Images for the second eye are then generated using a post-rendering process called depth-image-based rendering (DIBR).” and [0040], “The method 400 begins at step 402, where the processing unit receives an image frame corresponding to the left eye. At step 404, the processing unit receives a depth frame for the left eye, where the depth frame corresponds to the image frame received in step 402. At step 406, the processing unit generates an image frame for the right eye, based on the image frame and the depth frame for the left eye. The image frame for the right eye is generated using re-projection of the left eye image frame based on the depth information in the corresponding depth frame.” and [0041], “At step 412, the processing unit receives an image frame corresponding to the right eye. At step 414, the processing unit receives a depth frame for the right eye, where the depth frame corresponds to the image frame received in step 412. At step 416, the processing unit generates an image frame for the left eye, based on the image frame and the depth frame for the right eye. The image frame for the left eye is generated using re-projection of the right eye image frame based on the depth information in the corresponding depth frame.” and [0043], “In sum, a processing unit generates stereoscopic images by receiving an image frame with an associated depth frame for a first eye and reprojecting the image frame to the perspective of the second eye.”)
11. Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to modify the method of Chen to include the disclosure of using an initial color image, along with a target color image, to generate a 3D image, of Neill. The motivation for this modification could have been to more efficiently utilize computing resources of a 3D image generation method device as only one image would need to be generated as a pair image to the initial color image. The 3D image generation method device could then be available on hardware that might not otherwise be able to computationally generate two stereo images but is capable of producing an image as a match to an input image for a stereo pair.
12. Chen in view of Neill doesn't explicitly disclose but Schmalstieg discloses: [[obtaining an initial color image and an initial depth image]] from a back-end cache, [[the initial color image and the initial depth image being associated with each other;]] (Schmalstieg, [0024], “In accordance with the techniques of this disclosure, GPU 110 may perform only the geometry stage and the rasterization stage of a graphics processing pipeline. Performance of these stages may result in intermediate graphics data, which GPU 110 may store to a graphics buffer (G-buffer), e.g., of memory 106. The intermediate graphics data may include texture data for graphics objects defined by the graphics primitives and/or the graphics primitives themselves and depth data for the graphics objects.” and Schmalstieg, [0059], “GPU 190 receives the intermediate graphics data and buffers the intermediate graphics data in the G-buffer of graphics memory 192. Pixel processing pipeline 194 and texture engine 196 then perform a second portion of the image rendering process to render one or more images from the intermediate graphics data.”)
13. Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to modify the method of Chen in view of Neill to include the disclosure of utilizing a back-end cache to obtain an initial color image and an initial depth image that are associated with each other, of Schmalstieg. The motivation for this modification could have been to utilize a back-end cache as a means to not only store but to quickly send stereo images to a 3D image display device. By storing the stereo images in a cache, this makes the image data available for additional quick processing, such as to generate a new viewpoint image. Also, by quickly being able to display the images, this can help to increase the refresh rate of the stereo images to reduce frame rate flickering and to increase the comfort of the viewing experience.
14. As per claim 6, Chen in view of Neill, and further in view of Schmalstieg discloses: The method of claim 1, wherein the generating the 3D image according to the target color image and the initial color image comprises: (See rejection for claim 1.)
scaling the initial color image and the target color image to obtain a left color image and a right color image correspondingly; (Chen, [0009], “In view of the foregoing, it is an object of the embodiment of the present invention to provide a system and method of viewer centric depth adjustment for stereoscopic images to adjust both object size and disparity map, therefore providing better viewing experience by minimizing the distortion mentioned above.” and Chen, [0019], “Referring to FIG. 2, the system 200 further includes an image warping unit 23 utilized to prevent size distortion while applying depth shifting, compared with a conventional system that seldom takes image warping into account and thus normally causes size distortion. To deal with this artifact, the embodiment proposes image resizing algorithm to reconstruct the original scale.” and Chen, [0022], “According to the stereoscopic image adjustment scheme discussed above, absolute disparity remapping for specific viewing distance is first adopted, and then image resizing is applied to deal with inconsistency of monocular cues and binocular cues, therefore improving 3D viewing experience.”)
writing the left color image and the right color image into the back-end cache (Neill, [0024], “The parallel processing subsystem 112 may also store the left eye image frame N 210 in a memory buffer.” and Neill, [0028], “The parallel processing subsystem 112 may also store the right eye image frame N+1 250 in a memory buffer.” and Neill, [0041], “This pixel information may be stored in a memory buffer. For example, the pixel information may be from a prior image frame for the left eye may be retrieved from the storage buffer described above in conjunction with step 410. At step 420, the processing unit stores the right eye image frame received in step 412 in a memory buffer.” and Neill, [0018], “In some embodiments, some or all of the PPUs in parallel processing subsystem 112 are graphics processors with rendering pipelines that can be configured to perform various operations related to generating pixel data from graphics data supplied by CPU 102 and/or system memory 104 via memory bridge 105 and the second communication path 113, interacting with local parallel processing memory (which can be used as graphics memory including, e.g., a conventional frame buffer) to store and update pixel data, delivering pixel data to display device 110, and the like.”) in association; and (Schmalstieg, [0027], “For example, GPU 126 may warp texture information of the intermediate graphics data, using depth information of the intermediate graphics data, to produce a stereoscopic image pair. … In some examples, GPU 126 may store the rendered (or warped) images to respective ones of framebuffer memories 130A, 130B. For example, GPU 126 may store the left-eye view image to framebuffer memory 130A and the right-eye view image to framebuffer memory 130B.”)
upon receiving a display instruction, interleaving the left color image and the right color image from the back-end cache to generate the 3D image. (Neill, [0031], “Over time, the parallel processing subsystem 112 transmits a series of images to the display device 110, whereby images received by the parallel processing subsystem 112 are interleaved with images reprojected by the parallel processing subsystem 112 on each of the left and right channel.” and Neill, [0034], “The left eye image frame 310 illustrates a typical image frame received by the parallel processing subsystem 112. The parallel processing unit 112 transmits the left eye image frame 310 to the left channel of the display device 110.” and Neill, [0024], “The parallel processing subsystem 112 may also store the left eye image frame N 210 in a memory buffer.” and Neill, [0028], “The parallel processing subsystem 112 may also store the right eye image frame N+1 250 in a memory buffer.” and Neill, [0041], “This pixel information may be stored in a memory buffer. For example, the pixel information may be from a prior image frame for the left eye may be retrieved from the storage buffer described above in conjunction with step 410. At step 420, the processing unit stores the right eye image frame received in step 412 in a memory buffer.” and Schmalstieg, [0024]-[0025], “In accordance with the techniques of this disclosure, GPU 110 may perform only the geometry stage and the rasterization stage of a graphics processing pipeline. Performance of these stages may result in intermediate graphics data, which GPU 110 may store to a graphics buffer (G-buffer), e.g., of memory 106. The intermediate graphics data may include texture data for graphics objects defined by the graphics primitives and/or the graphics primitives themselves and depth data for the graphics objects. … Output interface 112 may then send the intermediate graphics data (i.e., the G-buffer data) to VR headset device 120 via communication medium 114 via output interface 112.”)
15. Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to modify the method of claim 1 of Chen to include the disclosure of writing the left color image and the right color image into the back-end cache and upon receiving a display instruction, interleaving the left color image and the right color image from the back-end cache to generate the 3D image, of Neill, and that the writing process to the back-end cache of the left color image and the right color image are in association with each other, of Schmalstieg. The motivation for the modifications of Neill and Schmalstieg could have been to allow for quick storage of image data that has been processed for stereo displays. Storing the image data, and in particular interleaving left and right imaging data together to be stored, allows for both the left and right images to quickly be sent together and displayed on a 3D imaging device. By quickly being able to display the images, this can help to increase the refresh rate of the stereo images to reduce frame rate flickering and to increase the comfort of the viewing experience.
16. Claim 7 is similar in scope to claim 1 except for different limitations that Chen in view of Neill, and further in view of Schmalstieg discloses: A 3D image generating apparatus, comprising: (Chen, [0009], “In view of the foregoing, it is an object of the embodiment of the present invention to provide a system and method of viewer centric depth adjustment for stereoscopic images to adjust both object size and disparity map, therefore providing better viewing experience by minimizing the distortion mentioned above.” and [0022], “According to the stereoscopic image adjustment scheme discussed above, absolute disparity remapping for specific viewing distance is first adopted, and then image resizing is applied to deal with inconsistency of monocular cues and binocular cues, therefore improving 3D viewing experience.” and Neill, [0008]-[0009], “One embodiment of the present invention sets forth a method for generating stereoscopic images. ... Other embodiments include, without limitation, a subsystem that includes a processing unit configured to implement one or more aspects of the disclosed methods as well as a system configured to implement one or more aspects of the disclosed methods.”)
a memory configured for storing program code; and (Neill, [0008]-[0009], “One embodiment of the present invention sets forth a method for generating stereoscopic images. ... Other embodiments include, without limitation, a computer-readable medium that includes instructions that enable a processing unit to implement one or more aspects of the disclosed methods.” and Neill, [0018], “In one embodiment, the parallel processing subsystem 112 includes one or more parallel processing units (PPUs), each of which is coupled to a local parallel processing (PP) memory.”)
a processor configured for calling the program codes in the memory to execute a 3D image generating method comprising: (Chen, [0021]-[0022], “Finally, the color image and the depth map processed by the depth remapping unit 21 and the image warping unit 23 are then forwarded to a depth image-based rendering (DIBR) unit 24. ... According to the stereoscopic image adjustment scheme discussed above, absolute disparity remapping for specific viewing distance is first adopted, and then image resizing is applied to deal with inconsistency of monocular cues and binocular cues, therefore improving 3D viewing experience.” and Neill, [0008]-[0009], “One embodiment of the present invention sets forth a method for generating stereoscopic images. ... Other embodiments include, without limitation, a subsystem that includes a processing unit configured to implement one or more aspects of the disclosed methods as well as a system configured to implement one or more aspects of the disclosed methods.”)
reading an initial color image and an initial depth image associated with the initial color image (Chen, [0015], “Based on the color image, the depth map and the viewer distance, the depth remapping unit 21 remaps (or shifts) depth values of the depth map such that the viewer perceives the same depth at different viewing distances.” and Chen, [0021], “Finally, the color image and the depth map processed by the depth remapping unit 21 and the image warping unit 23 are then forwarded to a depth image-based rendering (DIBR) unit 24.” and Neill, [0008], “One embodiment of the present invention sets forth a method for generating stereoscopic images. The method includes receiving a first image frame associated with a first eye, and receiving a first depth frame associated with the first eye.”) from a back-end cache; (Schmalstieg, [0024], “In accordance with the techniques of this disclosure, GPU 110 may perform only the geometry stage and the rasterization stage of a graphics processing pipeline. Performance of these stages may result in intermediate graphics data, which GPU 110 may store to a graphics buffer (G-buffer), e.g., of memory 106. The intermediate graphics data may include texture data for graphics objects defined by the graphics primitives and/or the graphics primitives themselves and depth data for the graphics objects.” and Schmalstieg, [0059], “GPU 190 receives the intermediate graphics data and buffers the intermediate graphics data in the G-buffer of graphics memory 192. Pixel processing pipeline 194 and texture engine 196 then perform a second portion of the image rendering process to render one or more images from the intermediate graphics data.”)
17. Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to modify the method of Chen to include the disclosure of a memory configured for storing program code, of Neill, and utilizing a back-end cache to read an initial color image and an initial depth image, of Schmalstieg. The motivation for the modification of Neill could have been to utilize the 3D image generation method on a processing device with a memory. This could allow for the method to be available to many different types of devices and systems. The motivation for the modification of Schmalstieg could have been to utilize a back-end cache as a means to not only store but to quickly send stereo images to a 3D image display device. By storing the stereo images in a cache, this makes the image data available for additional quick processing, such as to generate a new viewpoint image. Also, by quickly being able to display the images, this can help to increase the refresh rate of the stereo images to reduce frame rate flickering and to increase the comfort of the viewing experience.
18. Claim 12, which is similar in scope to dependent claim 6 and independent claim 7, is thus
rejected under the same rationale as described above. The motivation for this modification is the same as claim 6.
19. Claim 13 is similar in scope to claims 1 and 7 except for different limitations that Chen in view of Neill, and further in view of Schmalstieg discloses: A computer device, comprising:
a processor, a memory and a transceiver (Neill, [0031], “The DIBR process continues in this fashion, where the parallel processing subsystem 112 receives image and depth frames for one eye … Over time, the parallel processing subsystem 112 transmits a series of images to the display device 110, whereby images received by the parallel processing subsystem 112 are interleaved with images reprojected by the parallel processing subsystem 112 on each of the left and right channel.” And Neill, [0018], “In one embodiment, the parallel processing subsystem 112 includes one or more parallel processing units (PPUs), each of which is coupled to a local parallel processing (PP) memory.”) connected therewith, wherein the memory is configured to store program codes, and the processor is configured to call the program codes in the memory to execute a 3D image generating method comprising: (Chen, [0021]-[0022], “Finally, the color image and the depth map processed by the depth remapping unit 21 and the image warping unit 23 are then forwarded to a depth image-based rendering (DIBR) unit 24. ... According to the stereoscopic image adjustment scheme discussed above, absolute disparity remapping for specific viewing distance is first adopted, and then image resizing is applied to deal with inconsistency of monocular cues and binocular cues, therefore improving 3D viewing experience.” and Neill, [0008]-[0009], “One embodiment of the present invention sets forth a method for generating stereoscopic images. ... Other embodiments include, without limitation, a subsystem that includes a processing unit configured to implement one or more aspects of the disclosed methods as well as a system configured to implement one or more aspects of the disclosed methods.”)
20. Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to modify the method of Chen in view of Schmalstieg to include the disclosure of a transceiver, of Neill. The motivation for this modification could have been to utilize a processing component that is capable of both receiving and transmitting data. Such a processing device would be able to receive data, such as imaging data, perform some processing on the data, such as generating a 3D image stereo pair, and then transmit that data to a display to be shown.
21. Claim 18, which is similar in scope to dependent claim 6 and independent claim 13, is thus
rejected under the same rationale as described above. The motivation for this modification is the same as claim 6.
22. Claims 2, 8, and 14 are rejected under 35 U.S.C. 103 as being unpatentable over Chen et al. (US-2014/0063206-A1, hereinafter "Chen") in view of Neill (US-2014/0125650-A1), further in view of Schmalstieg et al. (US-2019/0066365-A1, hereinafter "Schmalstieg"), and further in view of Sawhney et al. (US-2002/0061131-A1, hereinafter "Sawhney").
23. As per claim 2, Chen in view of Neill, and further in view of Schmalstieg discloses: [[The method of claim 1, wherein the determining the relative displacement corresponding to the target depth image comprises:]] (See rejection for claim 1.)
24. Chen in view of Neill, and further in view of Schmalstieg doesn't explicitly disclose but Sawhney discloses: determining a y-axis rotation angle and an x-axis rotation angle corresponding to a target original image, (Sawhney, [0069], “Accordingly, the 3D coordinates of the point, P in an image matrix M are P=M-1[x,y,1]TZ. Using rotation and translation matrices R and T, respectively, its 3D coordinates in the new view are P′=RM-1[x,y,1]TZ+T and the new depth is then derived.” and [0067], “Because the relative pose between the reference view and the new view is known, the depth transformation can be easily derived.”; Examiner’s note: The rotation matrix R disclosed by Sawhney in [0069] can include both y-axis and x-axis rotation angles.) the target original image being an original image corresponding to the initial color image and the initial depth image; (Sawhney, [0067], “Once the depth maps are known, warp parameters to warp the real images to the new viewpoint can be generated at step 105 using a depth based warping algorithm. Before the depth information derived in the reference image can be used it is desirably converted into the new view coordinate system for rendering purposes. Because the relative pose between the reference view and the new view is known, the depth transformation can be easily derived.” and [0070], “By using the depth information, the image warping process 106 tends to produce the correct visibility (which part of the scene should be visible).”)
determining rendering position information according to the y-axis rotation angle and the x-axis rotation angle; and (Sawhney, [0069]-[0070], “Accordingly, the 3D coordinates of the point, P in an image matrix M are P=M-1[x,y,1]TZ. Using rotation and translation matrices R and T, respectively, its 3D coordinates in the new view are P′=RM-1[x,y,1]TZ+T and the new depth is then derived. … By using the depth information, the image warping process 106 tends to produce the correct visibility (which part of the scene should be visible).”)
determining the relative displacement according to initial position information corresponding to the target original image and the rendering position information. (Sawhney, [0067]-[0070], “Once the depth maps are known, warp parameters to warp the real images to the new viewpoint can be generated at step 105 using a depth based warping algorithm. Before the depth information derived in the reference image can be used it is desirably converted into the new view coordinate system for rendering purposes. Because the relative pose between the reference view and the new view is known, the depth transformation can be easily derived. The inventors have determined that the depth, Z, of a pixel in the reference view may be expressed by equation (1).
Z
=
1
1
d
π
+
k
=
d
π
1
+
k
d
π
(1)
In equation (1), k is the parallax and dπ is the distance from the frontal plane. Accordingly, the 3D coordinates of the point, P in an image matrix M are P=M-1[x,y,1]TZ. Using rotation and translation matrices R and T, respectively, its 3D coordinates in the new view are P′=RM-1[x,y,1]TZ+T and the new depth is then derived. This depth information is useful, for example, when multiple warped images are blended using z-buffering. By using the depth information, the image warping process 106 tends to produce the correct visibility (which part of the scene should be visible).”)
25. Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to modify the method of claim 1 of Chen in view of Neill, and further in view of Schmalstieg to include the disclosure of determining a y-axis rotation angle and an x-axis rotation angle corresponding to a target original image, the target original image being an original image corresponding to the initial color image and the initial depth image, determining rendering position information according to the y-axis rotation angle and the x-axis rotation angle, and determining the relative displacement according to initial position information corresponding to the target original image and the rendering position information, of Sawhney. The motivation for this modification could have been to utilize the y-axis and x-axis rotation information as a means of determining a projection point for a new 3D view. This information will help provide an existing image with the proper displacement in generating a new view.
26. Claim 8, which is similar in scope to dependent claim 2 and independent claim 7, is thus
rejected under the same rationale as described above. The motivation for this modification is the same as claim 2.
27. Claim 14, which is similar in scope to dependent claim 2 and independent claim 13, is thus
rejected under the same rationale as described above. The motivation for this modification is the same as claim 2.
28. Claims 3, 9, and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Chen et al. (US-2014/0063206-A1, hereinafter "Chen") in view of Neill (US-2014/0125650-A1), further in view of Schmalstieg et al. (US-2019/0066365-A1, hereinafter "Schmalstieg"), further in view of Sawhney et al. (US-2002/0061131-A1, hereinafter "Sawhney"), and further in view of Park et al. (CN-106898048-A, hereinafter "Park").
29. As per claim 3, Chen in view of Neill, further in view of Schmalstieg, and further in view of Sawhney discloses: The method of claim 2, wherein, the rendering position information [[is determined according to a formula:]] (See rejection for claim 2.)
30. Chen in view of Neill, further in view of Schmalstieg, and further in view of Sawhney doesn't explicitly disclose but Park discloses: [[The method of claim 2, wherein, the rendering position information]] is determined according to a formula:
Dx = tan a * Rd, and (See Park page 9, [0061]-[0065] below.)
Dy = tan b * Rd, (See Park page 9, [0061]-[0065] below.)
wherein Dx is a coordinate value of the rendering position information on the x-axis, Dy is a coordinate value of the rendering position information on the y-axis, a is the y-axis rotation angle, b is the x-axis rotation angle, and Rd is a preset constant. (Park, page 9, [0061]-[0065], “According to the principle of projection geometry, sub-image can be expressed as:
Iθ (x, y) = I (x + Vqx, y + Vqy) (9)
Where I (x, y) is the pixel value of the color image at the (x, y) point, Vqx and Vqy are the pixel offset values Δq along the x- and y-axis directions, Δq depends on the depth information and the projection angle, Can be calculated according to equation (10):
Δq = (z (x, y) -dc) × tan θ (10)
Where z (x, y) is the depth value of the point (x, y) in the restored depth image, dc is the distance between the center depth plane and the zero plane (z = 0), and θ is the projection angle ...”; Examiner’s note: As disclosed on page 9, [0061]-[0065] of Park, the equation for Δq is analogous to the Dx and Dy equations of the claim (which is of the format tan (angle) × (constant)). Specifically, Δq correlates to Vqx and Vqy pixel offset values in the x-axis and y-axis direction. In addition, the equation Iθ provides the final locations for these points based on the offset. The y-axis and x-axis rotation angle is represented by θ and the constant is represented by (z (x,y) -dc) where “z (x, y) is the depth value of the point (x, y)” and “dc is the distance between the center depth plane and the zero plane (z = 0).”
31. Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to modify the method of claim 2 of Chen in view of Neill, further in view of Schmalstieg, and further in view of Sawhney to include the disclosure to utilize formulas Dx and Dy to determine a rendering position where Dx is the rendering position for the x-axis and Dy is the rendering position for the y-axis, of Park. The motivation for this modification could have been to utilize an equation to determine appropriate image offset coordinates using y-axis and x-axis rotation angles to create a new 3D view. This information will help provide an existing image with the proper displacement coordinates in generating a new view.
32. Claim 9, which is similar in scope to dependent claim 3 and independent claim 7, is thus
rejected under the same rationale as described above. The motivation for this modification is the same as claim 3.
33. Claim 15, which is similar in scope to dependent claim 3 and independent claim 13, is thus
rejected under the same rationale as described above. The motivation for this modification is the same as claim 3.
34. Claims 4, 10, and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Chen et al. (US-2014/0063206-A1, hereinafter "Chen") in view of Neill (US-2014/0125650-A1), further in view of Schmalstieg et al. (US-2019/0066365-A1, hereinafter "Schmalstieg"), further in view of Sawhney et al. (US-2002/0061131-A1, hereinafter "Sawhney"), and further in view of Liu et al. (US-2015/0339844-A1, hereinafter "Liu").
35. As per claim 4, Chen in view of Neill, further in view of Schmalstieg, and further in view of Sawhney discloses: The method of claim 2, wherein the y-axis rotation angle [[is determined according to a formula:]] (See rejection for claim 2.)
the x-axis rotation angle [[is determined according to a formula:]] (See rejection for claim 2.)
36. Chen in view of Neill, further in view of Schmalstieg, and further in view of Sawhney doesn't explicitly disclose but Liu discloses: [[The method of claim 2, wherein the y-axis rotation angle]] is determined according to a formula:
a = arctan (H x tan α / (H + J)), (Liu, Fig. 2-3; [0033], “… an angle by which the scene is rotated about the Y-axis is determined to be
a
=
a
r
c
t
a
n
L
·
t
a
n
α
L
+
Z
… according to the angles α and β, a distance L from the human eye to the screen and a distance Z from the center of the scene to the screen.”)
wherein a is the y-axis rotation angle, H is a distance between a user's eye and a display screen, α is an angle between a first target projection and a positive z-axis of a coordinate system, and J is a distance between a center of scene corresponding to the target original image and the display screen, the display screen is a screen for displaying the 3D image, the first target projection is a projection of a target connection line on an XOZ plane, and the target connection line is a line from the user's eye to a center of the display screen; and (Liu, Fig. 2-3; [0032]-[0033], “... in a 3D space rectangular coordinate system O-XYZ prior to the rotation, the center of the screen is located at an origin of the coordinate system O-XYZ, a projection of a connecting line from the human eye to the center of the screen on the XOZ plane includes an angle α with the positive Z-axis direction, a projection of the connecting line from the human eye to the center of the screen on the YOZ plane includes an angle β with the positive Z-axis direction, the X-axis direction points from a midpoint of a left edge of the screen towards a midpoint of a right edge of the screen, and the Y-axis direction points from a midpoint of a top edge of the screen towards a midpoint of a bottom edge of the screen, and an angle by which the scene is rotated about the Y-axis is determined to be
a
=
a
r
c
t
a
n
L
·
t
a
n
α
L
+
Z
and an angle by which the scene is rotated about the X-axis is determined to be
b
=
a
r
c
t
a
n
L
·
t
a
n
β
L
+
Z
according to the angles α and β, a distance L from the human eye to the screen and a distance Z from the center of the scene to the screen.” and [0051], “An angle a by which the scene is rotated about the Y-axis is represented by
a
=
a
r
c
t
a
n
L
·
t
a
n
α
L
+
Z
and, similarly, an angle by which the scene is rotated about the X-axis is represented by
b
=
a
r
c
t
a
n
L
·
t
a
n
β
L
+
Z
.
The virtual holographic 3D view matrix is obtained by rotating the original virtual scene about the Y-axis by the angle a and then about the X-axis by the angle b.”)
[[the x-axis rotation angle]] is determined according to a formula:
b = arctan (H x tan β / (H + J)), (Liu, Fig. 2-3; [0033], “… and an angle by which the scene is rotated about the X-axis is determined to be
b
=
a
r
c
t
a
n
L
·
t
a
n
β
L
+
Z
according to the angles α and β, a distance L from the human eye to the screen and a distance Z from the center of the scene to the screen.”)
wherein b is the x-axis rotation angle, β is an angle between a second target projection and the positive z-axis of the coordinate system, O is an origin of the coordinate system, and the second target projection is a projection of the target connection line on an YOZ plane. (Liu, Fig. 2-3; [0032]-[0033], “... in a 3D space rectangular coordinate system O-XYZ prior to the rotation, the center of the screen is located at an origin of the coordinate system O-XYZ, a projection of a connecting line from the human eye to the center of the screen on the XOZ plane includes an angle α with the positive Z-axis direction, a projection of the connecting line from the human eye to the center of the screen on the YOZ plane includes an angle β with the positive Z-axis direction, the X-axis direction points from a midpoint of a left edge of the screen towards a midpoint of a right edge of the screen, and the Y-axis direction points from a midpoint of a top edge of the screen towards a midpoint of a bottom edge of the screen, and an angle by which the scene is rotated about the Y-axis is determined to be
a
=
a
r
c
t
a
n
L
·
t
a
n
α
L
+
Z
and an angle by which the scene is rotated about the X-axis is determined to be
b
=
a
r
c
t
a
n
L
·
t
a
n
β
L
+
Z
according to the angles α and β, a distance L from the human eye to the screen and a distance Z from the center of the scene to the screen.” and [0051], “An angle a by which the scene is rotated about the Y-axis is represented by
a
=
a
r
c
t
a
n
L
·
t
a
n
α
L
+
Z
and, similarly, an angle by which the scene is rotated about the X-axis is represented by
b
=
a
r
c
t
a
n
L
·
t
a
n
β
L
+
Z
.
The virtual holographic 3D view matrix is obtained by rotating the original virtual scene about the Y-axis by the angle a and then about the X-axis by the angle b.”)
37. Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to modify the method of claim 2 Chen in view of Neill, further in view of Schmalstieg, and further in view of Sawhney to include the disclosure to utilize formulas to determine the y-axis and x-axis rotation angles a and b respectively, of Liu. The motivation for this modification could have been to utilize an equation to determine the y-axis and a-axis rotation angles in order to determine a proper displacement factor for a new image view. The rotation angles will help determine an appropriate image offset coordinates to create a new 3D view from an existing image.
38. Claim 10, which is similar in scope to dependent claim 4 and independent claim 7, is thus
rejected under the same rationale as described above. The motivation for this modification is the same as claim 4.
39. Claim 16, which is similar in scope to dependent claim 4 and independent claim 13, is thus
rejected under the same rationale as described above. The motivation for this modification is the same as claim 4.
40. Claims 5, 11, and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Chen et al. (US-2014/0063206-A1, hereinafter "Chen") in view of Neill (US-2014/0125650-A1), further in view of Schmalstieg et al. (US-2019/0066365-A1, hereinafter "Schmalstieg"), and further in view of Um Gi Mun et al. (KR-20200107217-A, hereinafter "Um Gi Mun").
41. As per claim 5, Chen in view of Neill, and further in view of Schmalstieg discloses: The method of claim 1, wherein the rendering the initial color image based on the relative displacement and the target depth image to obtain the target color image comprises: (See rejection for claim 1.)
42. Chen in view of Neill, and further in view of Schmalstieg doesn't explicitly disclose but Um Gi Mun discloses: determining an initial point cloud corresponding to the target depth image; (Um Gi Mun, [0007], “An image processing method according to an embodiment of the present invention includes obtaining a multiview depth map of a plurality of viewpoint images and determining depth reliability of each point of the depth map, mapping each viewpoint image to a 3D point cloud on a reference coordinate system” and [0011], “The mapping to the 3D point cloud may include mapping the multi-view depth map to a 3D point cloud on the reference coordinate system based on camera information.” and [0015], “The generating of the virtual viewpoint image includes projecting each 3D point of the 3D point cloud to the virtual viewpoint for each depth cluster while proceeding in a direction in which the depth value of the depth cluster decreases.”)
adjusting coordinate values of each point in the initial point cloud according to the relative displacement to obtain a target point cloud; (Um Gi Mun, [0077]-[0078], “Returning to FIG. 3 again, in step 350, the virtual viewpoint image generator 550 includes projecting each 3D point of the 3D point cloud to a virtual viewpoint for each depth cluster generated in step 345. In step 350, the virtual viewpoint image generator 550 projects each 3D point of the 3D point cloud to a virtual viewpoint for each depth cluster while proceeding in a direction in which the depth value of the depth cluster generated in step 345 decreases. . That is, after the depth clustering in step 345 is completed, the virtual viewpoint image generator 550 moves each 3D point of the 3D point cloud in the virtual viewpoint direction in the virtual viewpoint direction from the cluster to the front and each cluster. Projected sequentially to the virtual viewpoint. Even within the same cluster, the depth is projected to the virtual viewpoint while coming forward from the 3D point behind. The virtual viewpoint image is generated in the order of the background, the distant object, and the near object by projecting to a virtual viewpoint for each cluster while coming forward from the cluster with the depth behind.”)
processing coordinate values of each point in the target point cloud to obtain a reference image; and (Um Gi Mun, [0077]-[0078], “Returning to FIG. 3 again, in step 350, the virtual viewpoint image generator 550 includes projecting each 3D point of the 3D point cloud to a virtual viewpoint for each depth cluster generated in step 345. In step 350, the virtual viewpoint image generator 550 projects each 3D point of the 3D point cloud to a virtual viewpoint for each depth cluster while proceeding in a direction in which the depth value of the depth cluster generated in step 345 decreases. That is, after the depth clustering in step 345 is completed, the virtual viewpoint image generator 550 moves each 3D point of the 3D point cloud in the virtual viewpoint direction in the virtual viewpoint direction from the cluster to the front and each cluster. Projected sequentially to the virtual viewpoint. Even within the same cluster, the depth is projected to the virtual viewpoint while coming forward from the 3D point behind. The virtual viewpoint image is generated in the order of the background, the distant object, and the near object by projecting to a virtual viewpoint for each cluster while coming forward from the cluster with the depth behind.” and [0080], “In step 360, the virtual viewpoint image generator 550 determines that the 3D points that are not projected from the virtual viewpoint images generated through steps 350 and 355 are the virtual viewpoint among 3D points already projected on the virtual viewpoint image.” and [0079], “When a plurality of 3D points are projected onto the same XY position of the virtual viewpoint image, the virtual viewpoint image generator 550 selects a 3D point whose depth reliability is greater than or equal to the reference reliability Th1 among the plurality of 3D points.”; Examiner’s note: In Step 350, Um Gi Mun discloses in [0077], discloses the process of “projecting each 3D point of the 3D point cloud to a virtual viewpoint for each depth cluster generated” and in [0078], “the depth is projected to the virtual viewpoint while coming forward from the 3D point behind. The virtual viewpoint image is generated in the order of the background, the distant object, and the near object by projecting to a virtual viewpoint for each cluster while coming forward from the cluster with the depth behind.” The projection process uses the depth as a reference depth image that is used to create the virtual viewpoint image. Also, it is disclosed in [0080] that “virtual viewpoint images [are] generated through steps 350 and 355.” Step 350 discloses the process of generating a virtual viewpoint image through projecting the depth values while step 355 discloses the process of determining the color of a projected point in the virtual viewpoint.)
processing pixels in the initial color image according to depth values of pixels of the reference image to obtain the target color image. (Um Gi Mun, [0079], “In step 355, the virtual viewpoint image generator 550 determines the color of the 3D point projected to the virtual viewpoint in step 350. When a plurality of 3D points are projected onto the same XY position of the virtual viewpoint image, the virtual viewpoint image generator 550 selects a 3D point whose depth reliability is greater than or equal to the reference reliability Th1 among the plurality of 3D points. The virtual viewpoint image generator 550 determines two 3D points in the order of the smallest depth value among the selected 3D points, and when the depth difference between the determined two 3D points is greater than or equal to the reference depth difference Th3, the two Among the 3D points, the color of the 3D point located at the foremost in the direction of the virtual viewpoint is determined as the color of the corresponding XY position. On the other hand, when the depth difference between the two 3D points is less than the reference depth difference (Th3), the color obtained by blending the colors of the two 3D points is determined as the color of the corresponding XY position by using the depth reliability of the two 3D points as a weight.”)
43. Before the effective filing date of the claimed invention, it would have been obvious to a person having ordinary skill in the art to modify the method of claim 1 of Chen in view of Neill, and further in view of Schmalstieg to include the disclosure of determining an initial point cloud corresponding to the target depth image, adjusting coordinate values of each point in the initial point cloud according to the relative displacement to obtain a target point cloud, processing coordinate values of each point in the target point cloud to obtain a reference image, and processing pixels in the initial color image according to depth values of pixels of the reference image to obtain the target color image, of Um Gi Mun. The motivation for this modification could have been to utilize a point cloud generated from a depth image so as to make sure the 2D depth image is properly translated into 3D space. Each point in the cloud is given a specific depth according to the depth image. The point cloud can then be adjusted in 3D space and used to generate a new depth reference image. The adjustment may be easier to perform due to the depth points preserving their relationship to one another. This process can then generate a new 3D color view image based on the point cloud adjustments.
44. Claim 11, which is similar in scope to dependent claim 5 and independent claim 7, is thus
rejected under the same rationale as described above. The motivation for this modification is the same as claim 5.
45. Claim 17, which is similar in scope to dependent claim 5 and independent claim 13, is thus
rejected under the same rationale as described above. The motivation for this modification is the same as claim 5.
Conclusion
46. Any inquiry concerning this communication or earlier communications from the examiner should be directed to MATTHEW CLOTHIER whose telephone number is (571)272-4667. The examiner can normally be reached Mon-Fri 8:00am-4:00pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kent Chang can be reached at (571)272-7667. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MATTHEW CLOTHIER/Examiner, Art Unit 2614
/KENT W CHANG/Supervisory Patent Examiner, Art Unit 2614
Prosecution Insights