DETAILED ACTION
Notice of Pre-AIA or AIA Status
1. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
2. In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Information Disclosure Statement
3. The Information Disclosure Statement filed 2 April 2024 has been fully considered by Examiner. An annotated copy is included herewith.
Claim Rejections - 35 USC § 101
4. 35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
5. Claim 18 is rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter. The claim does not fall within at least one of the four categories of patent eligible subject matter because a program per se does not fall within one of the four statutory categories of invention. See MPEP § 2106.03(I).
Claim Rejections - 35 USC § 102
6. The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
7. Claims 1, 9, 12-14, 17 and 18 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Cappello (GB-2576548-A).
Regarding claim 1: Cappello discloses an information processing device (fig 3 and page 13, lines 11-12 of Cappello) configured to: acquire a color image (fig 3(301) and page 13, lines 14-17) at a first viewpoint (fig 1(106A-106C); fig 2(S201); and page 5, lines 21-28 of Cappello) and a depth image (fig 3(302) and page 13, lines 24-27 of Cappello) at a second viewpoint (fig 1(106A-106C); fig 2(S202); and page 5, line 30 to page 6, line 5 of Cappello – colour and depth images each captured at multiple viewpoints); and generate an output color image at a virtual viewpoint different from the first viewpoint (fig 2(S204-S205) and page 7, lines 24-33 of Cappello) on a basis of a result of separation processing of separating the depth image into a foreground depth image and a background depth image (fig 1(106A,106B); page 4, lines 9-16; page 6, lines 32-34; and page 7, lines 13-22 of Cappello – face-on view (foreground) and behind-view (background) images are separated for reconstructing an image at an alternate viewpoint).
Regarding claim 9: Cappello discloses the information processing device according to claim 1 (as rejected above), wherein, in the separation processing, the past depth information is multiplied in multiple depth images including a plurality of depth images and is retained, and the multiple depth images projected onto the virtual viewpoint are combined to generate a combined depth image (fig 1(106A-106C) and page 7, line 24 to page 8, line 19 of Cappello – past depth information of multiple depth images used to form the point cloud, which is used to produce the projected 2D image at the alternative viewpoint).
Regarding claim 12: Cappello discloses the information processing device according to claim 1 (as rejected above), wherein the virtual viewpoint is a viewpoint corresponding to a display included in a head mount display (page 15, lines 9-16 of Cappello).
Regarding claim 13: Cappello discloses the information processing device according to claim 1 (as rejected above), wherein the virtual viewpoint is a viewpoint corresponding to an eye of a user wearing a head mount display (page 15, lines 9-16 of Cappello).
Regarding claim 14: Cappello discloses the information processing device according to claim 1 (as rejected above), wherein the first viewpoint is a viewpoint of a color camera that images the color image (page 5, lines 21-25 of Cappello).
Regarding claim 17: Cappello discloses an information processing method (fig 2 and page 2, lines 14-15 of Cappello) comprising: acquiring a color image (fig 3(301) and page 13, lines 14-17) at a first viewpoint (fig 1(106A-106C); fig 2(S201); and page 5, lines 21-28 of Cappello) and a depth image (fig 3(302) and page 13, lines 24-27 of Cappello) at a second viewpoint (fig 1(106A-106C); fig 2(S202); and page 5, line 30 to page 6, line 5 of Cappello – colour and depth images each captured at multiple viewpoints); and generating an output color image at a virtual viewpoint different from the first viewpoint (fig 2(S204-S205) and page 7, lines 24-33 of Cappello) on a basis of a result of separation processing of separating the depth image into a foreground depth image and a background depth image (fig 1(106A,106B); page 4, lines 9-16; page 6, lines 32-34; and page 7, lines 13-22 of Cappello – face-on view (foreground) and behind-view (background) images are separated for reconstructing an image at an alternate viewpoint).
Regarding claim 18: Cappello discloses a program causing a computer to execute an information processing method (fig 3 and page 13, lines 11-12 of Cappello) of: acquiring a color image (fig 3(301) and page 13, lines 14-17) at a first viewpoint (fig 1(106A-106C); fig 2(S201); and page 5, lines 21-28 of Cappello) and a depth image (fig 3(302) and page 13, lines 24-27 of Cappello) at a second viewpoint (fig 1(106A-106C); fig 2(S202); and page 5, line 30 to page 6, line 5 of Cappello – colour and depth images each captured at multiple viewpoints); and generating an output color image at a virtual viewpoint different from the first viewpoint (fig 2(S204-S205) and page 7, lines 24-33 of Cappello) on a basis of a result of separation processing of separating the depth image into a foreground depth image and a background depth image (fig 1(106A,106B); page 4, lines 9-16; page 6, lines 32-34; and page 7, lines 13-22 of Cappello – face-on view (foreground) and behind-view (background) images are separated for reconstructing an image at an alternate viewpoint).
Claim Rejections - 35 USC § 103
8. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
9. Claims 2-7 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Cappello (GB-2576548-A) in view of Sun (US-2023/0162338).
Regarding claim 2: Cappello discloses the information processing device according to claim 1 (as rejected above). Cappello does not disclose wherein first processing is performed on the foreground depth image, and second processing is performed on the background depth image.
Sun discloses wherein first processing is performed on the foreground depth image, and second processing is performed on the background depth image ([0059], [0076], and [0088]-[0089] of Sun).
Cappello and Sun are analogous art because they are from the same field of endeavor, namely 3D image data processing based on virtual viewpoints in a virtual reality system. Before the effective filing date of the invention, it would have been obvious to one of ordinary skill in the art to perform first processing on the foreground depth and perform second processing on the background depth, as taught by Sun. The motivation for doing so would have been to more effectively and accurately correct image data in the different image data regions, thus producing an improved result. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to modify Cappello according to the relied-upon teachings of Sun to obtain the invention as specified in claim 2.
Regarding claim 3: Cappello in view of Sun discloses the information processing device according to claim 2 (as rejected above), wherein, in the second processing, a current background depth image projected onto the virtual viewpoint and a past background depth image projected onto the virtual viewpoint are combined to generate a combined background depth image at the virtual viewpoint ([0095]-[0098], and [0100] of Cappello – original (past) background depth image combined with background depth image hole filled in (current)). Cappello and Sun are combined for the reasons set forth above with respect to claim 2.
Regarding claim 4: Cappello discloses the information processing device according to claim 1 (as rejected above). Cappello does not disclose wherein a background color image obtained by removing a region including pixels in which depth values are present on the foreground depth image from the color image is generated.
Sun discloses wherein a background color image obtained by removing a region including pixels in which depth values are present on the foreground depth image from the color image is generated ([0080], and [0082]-[0084] of Sun – foreground pixels are removed according to depth values and region edges to separate foreground from background).
Cappello and Sun are analogous art because they are from the same field of endeavor, namely 3D image data processing based on virtual viewpoints in a virtual reality system. Before the effective filing date of the invention, it would have been obvious to one of ordinary skill in the art to generate a background color image obtained by removing a region including pixels in which depth values are present on the foreground depth image from the color image, as taught by Sun. The motivation for doing so would have been to more effectively and accurately correct image data in the different image data regions, thus producing an improved result. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to modify Cappello according to the relied-upon teachings of Sun to obtain the invention as specified in claim 4.
Regarding claim 5: Cappello in view of Sun discloses the information processing device according to claim 4 (as rejected above), wherein the background color image projected onto the virtual viewpoint and the past background color image projected onto the virtual viewpoint are combined to generate a combined background color image at the virtual viewpoint ([0095]-[0098], and [0100] of Cappello – original (past) background depth image combined with background depth image hole filled in (current)). Cappello and Sun are combined for the reasons set forth above with respect to claim 4.
Regarding claim 6: Cappello in view of Sun discloses the information processing device according to claim 5 (as rejected above), wherein the output color image is generated by combining a foreground color image obtained by projecting the color image onto the virtual viewpoint with the combined background color image (figs 6-7 and [0089]-[0095] of Sun – background and foreground are both repaired separated and combined for an output color image). Cappello and Sun are combined for the reasons set forth above with respect to claim 4.
Regarding claim 7: Cappello discloses the information processing device according to claim 1 (as rejected above). Cappello does not disclose wherein, in the separation processing, a fixed threshold for a depth value is set, and the input depth image is separated into the foreground depth image and the background depth image on a basis of a comparison result of the depth value with the threshold.
Sun discloses wherein, in the separation processing, a fixed threshold for a depth value is set, and the input depth image is separated into the foreground depth image and the background depth image on a basis of a comparison result of the depth value with the threshold ([0076]-[0077], and [0089] of Sun).
Cappello and Sun are analogous art because they are from the same field of endeavor, namely 3D image data processing based on virtual viewpoints in a virtual reality system. Before the effective filing date of the invention, it would have been obvious to one of ordinary skill in the art to set a fixed threshold for a depth value in the separation processing, and separate the input depth image into the foreground depth image and the background depth image on a basis of a comparison result of the depth value with the threshold, as taught by Sun. The motivation for doing so would have been to more effectively and accurately correct image data in the different image data regions, thus producing an improved result. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to modify Cappello according to the relied-upon teachings of Sun to obtain the invention as specified in claim 7.
Regarding claim 15: Cappello in view of Sun discloses the information processing device according to claim 3 (as rejected above), wherein smoothing filter processing is performed on the combined background depth image (page 9, line 32 to page 10, line 2 of Sun). Cappello and Sun are combined for the reasons set forth above with respect to claim 2.
10. Claims 8 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Cappello (GB-2576548-A) in view of Sun (US-2023/0162338), and in further view of obvious engineering design choice.
Regarding claim 8: Cappello discloses the information processing device according to claim 1 (as rejected above). Cappello does not disclose wherein, in the separation processing, a fixed threshold for a depth value is set, and the input depth image is separated into the foreground depth image and the background depth image on a basis of a comparison result of the depth value with the threshold.
Sun discloses wherein, in the separation processing, a dynamic threshold for a depth value is set, and the input depth image is separated into the foreground depth image and the background depth image on a basis of a comparison result of the depth value with the threshold ([0076]-[0077], and [0089] of Sun).
Cappello and Sun are analogous art because they are from the same field of endeavor, namely 3D image data processing based on virtual viewpoints in a virtual reality system. Before the effective filing date of the invention, it would have been obvious to one of ordinary skill in the art to set a fixed threshold for a depth value in the separation processing, and separate the input depth image into the foreground depth image and the background depth image on a basis of a comparison result of the depth value with the threshold, as taught by Sun. The motivation for doing so would have been to more effectively and accurately correct image data in the different image data regions, thus producing an improved result. Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the invention to modify Cappello according to the relied-upon teachings of Sun.
Cappello in view of Sun does not disclose the threshold is a dynamic threshold. However, it would have been an obvious engineering design choice to instead use a dynamic threshold, since doing so would allow for finer adjustments and better control of the resultant output image.
Regarding claim 16: Cappello in view of Sun discloses the information processing device according to claim 2 (as rejected above). Cappello in view of Sun does not expressly disclose wherein the second processing has more processing steps than the first processing. However, it would have been an obvious engineering design choice for the second processing to have more steps than the first processing, since how steps are defined depends on how the steps and their corresponding sub-steps are defined by the designer.
Allowable Subject Matter
11. Claims 10 and 11 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter: Examiner has not discovered prior art which fully teaches either claim 10 or 11. The closest prior art discovered is the combination of Cappello (GB-2576548-A), Sun (US-2023/0162338), Kang (US-2009/0201384), and Takeda (US-2019/0325611). However, claims 10 and 11 each distinguish over the cited prior art, and any other prior art discovered by Examiner.
Contact Information
Any inquiry concerning this communication or earlier communications from the examiner should be directed to James A Thompson whose telephone number is (571)272-7441. The examiner can normally be reached M-F 8am-6pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Alicia Harrington can be reached at 571-272-2330. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JAMES A THOMPSON/Primary Examiner, Art Unit 2615
Prosecution Insights