DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Claims 2 and 14 are objected to because of the following informalities: claim 2 and 14 respectively recite “A method in accordance with claim 1” and “A system in accordance with claim 13”. The Examiner believes the preambles of these two claims should read “The method in accordance with claim 1” and “The system in accordance with claim 13”. Appropriate correction is required.
Claim Rejections - 35 USC § 112
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-24 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 1 partially recites “for a current active screen view, synthesizing a virtual camera view by motion compensated interpolation between the one or more reference views and the current active screen view”. The Examiner is unclear on whether the limitation is directed to the determination of a current active screen view by synthesizing a virtual camera view based on reference view(s) and the current active screen view (if this is the case, the Examiner notes the active screen view is already determined since it is needed in order to obtain the virtual camera view) or if the limitation is directed to obtaining a virtual camera view by performing a motion compensated interpolation between a reference view(s) and a current active screen view.
Claim 13 presents the same issue as claim 1.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-10 and 13-22 is/are rejected under 35 U.S.C. 103 as being unpatentable over Hillis et al. (US 2010/0283830) in view of Winter et al. (US 2022/0109822).
Regarding claim 1 Hillis discloses a method for motion compensated single camera viewpoint shifting, comprising:
capturing one or more reference views of a user looking at one of a camera and a screen (images 116 and 118 of participant 10 which is looking at display 30);
for a current active screen view, synthesizing a virtual camera view by interpolation between the one or more reference views and the current active screen view (image 146 is generated by combining interpolated images 136, 138 which are obtained using captured images 116 and 118);
wherein the virtual camera view comprises a virtual image of the user looking at the screen (image 146 in Figure 1A).
However, fails to explicitly disclose synthesizing a virtual camera view by motion compensation.
In his disclosure Winter teaches synthesizing a virtual camera view by motion compensation (obtaining a virtual camera view by synthesizing a subset of camera views neighboring the position of the desired virtual camera; the virtual view may be generated using homography based on motion vectors or feature correspondences between two or more cameras; the virtual view may be generated using any other suitable view interpolation method, including, without limitation, (1) depth-from-stereo view interpolation, (2) sparse or dense motion vectors between two or more cameras, (3) synthetic aperture blending (image-based techniques), and/or (4) Deep-learning based view interpolation – [0096]).
It would have been obvious to a person with ordinary skill in the art, before the effective filing date of the claimed invention, to synthesize a virtual camera view by motion compensation as taught by Winter into the teachings of Hillis because such incorporation improves the quality of the image operations (par. 97).
Examiner’s Note: the limitation “capturing one or more reference views of a user looking at one of a camera and a screen” is written in alternative form. The Examiner has chosen the alternative “capturing one or more reference view of a user looking at a screen”.
Regarding claim 2 Hillis discloses a method in accordance with claim 1, wherein the one or more reference views comprises one or more training camera views of a user looking into the camera, the synthesizing of the virtual camera view further comprising: determining the one or more training camera views that best matches the current active screen view; estimating motion vectors between the matched training camera view and the current active screen view, and interpolating the current active screen view using the motion vectors to synthesize the virtual camera view (images 116 and 118 of participant 10 which is looking at display 30).
Examiner’s Note: Claim 2 is further limiting an alternative form claim 1 that was not chosen by the Examiner.
Regarding claim 3 Hillis discloses the method in accordance with claim 1, wherein the one or more reference views comprise at least one of: one or more training camera views of the user looking into the camera, each of the one or more training camera views captured at a different orientation of the user relative to the camera; and one or more training screen views of the user looking at the screen, each of the one or more training screen views captured at a different orientation of the user relative to the screen (images 116 and 118 of participant 10 which is looking at display 30; note images 116 and 118 are captured at two different orientations of the user relative to display 30).
Examiner’s Note: the limitation “the one or more reference views comprise at least one of: one or more training camera views of the user looking into the camera, each of the one or more training camera views captured at a different orientation of the user relative to the camera; and one or more training screen views of the user looking at the screen, each of the one or more training screen views captured at a different orientation of the user relative to the screen” is written in alternative form. The Examiner has chosen the alternative “the one or more reference views comprise at least one of: one or more training screen views of the user looking at the screen, each of the one or more training screen views captured at a different orientation of the user relative to the screen”.
Regarding claim 4 Hillis discloses the method in accordance with claim 3, wherein the synthesizing of the virtual camera view further comprises:
determining the one or more training screen views that best matches the current active screen view; estimating first motion vectors between the matched training screen view and the current active screen view; estimating second motion vectors between the matched training screen view and a corresponding one of the one or more training camera views; mapping the second motion vectors onto the current active screen view using the first motion vectors to derive third motion vectors; interpolating the current active screen view using the third motion vectors to synthesize the virtual camera view (stereo matching – [0008]; calculating dense correspondence between images 116 and 118 – [0071, 0099-0100]; calculating a dense correspondence to determine a displacement in at least a first dimension for each of the pixels in the first image digital version which would move each of the pixels to a most nearly corresponding pixel in the image digital versions of at least one other member of the imaging device collection – [0107]; generating an interpolated image for at least two of the imaging device collection members from the at least one dense correspondence of the at least two images – [0100]).
Regarding claim 5 Hillis discloses the method in accordance with claim 4, wherein each of the one or more training camera views is paired with a corresponding training screen view of the one or more training screen views (calculating a dense correspondence to determine a displacement in at least a first dimension for each of the pixels in the first image digital version 116 to move each of the pixels to a most nearly corresponding pixel in the image digital versions of at least one other member of the imaging device collection 118 – [0071]).
Regarding claim 6 Hillis discloses the method in accordance with claim 4, wherein the reference views further comprise one or more of: updated training camera views obtained during a video conference; and updated training screen views obtained during the video conference (from Fig. 1A, video delivery system 80 presents 82 second participant 90 motion video stream 72 generated by motion video portal 70 conveying eye contact based upon the succession composite images 146 – [0057]).
Regarding claim 7 Hillis discloses the method in accordance with claim 4, wherein a series of the virtual camera views are produced as the video conference progresses (means 100 for generating composite image 146 is communicatively coupled 142 to motion video portal 70, providing a succession of composite images 146, each based upon at least synchronized image collection 116 and 118 to 72 video delivery system 80 – [0056]).
Regarding claim 8 Hillis discloses the method in accordance with claim 4. However, fails to explicitly disclose wherein at least one of the x-component or the y-component of one or more of the first motion vector, the second motion vector, or the third motion vector may be one of: set to zero; multiplied by a number; and provided with a fixed bias.
In his disclosure Winter teaches at least one of the x-component or the y-component of one or more of the first motion vector, the second motion vector, or the third motion vector may be one of: set to zero; multiplied by a number; and provided with a fixed bias (obtaining a virtual camera view by synthesizing a subset of camera views neighboring the position of the desired virtual camera; the virtual view may be generated using homography based on motion vectors or feature correspondences between two or more cameras; the virtual view may be generated using any other suitable view interpolation method, including, without limitation, (1) depth-from-stereo view interpolation, (2) sparse or dense motion vectors between two or more cameras, (3) synthetic aperture blending (image-based techniques), and/or (4) Deep-learning based view interpolation – [0096]).
It would have been obvious to a person with ordinary skill in the art, before the effective filing date of the claimed invention, to synthesize a virtual camera view by motion compensation as taught by Winter into the teachings of Hillis because such incorporation improves the quality of the image operations (par. 97).
Regarding claim 9 Hillis discloses the method in accordance with claim 4. However, fails to explicitly disclose each of the third motion vectors comprising a horizontal x-component and a vertical y-component, the third motion vectors being defined as Mac(n)[dx, dy]; the x-component of the third motion vectors are set to zero to obtain modified third motion vectors defined as Mac(n)[d0, dy]; the modified third motion vectors are utilized in the interpolating step.
In his disclosure Winter teaches each of the third motion vectors comprising a horizontal x-component and a vertical y-component, the third motion vectors being defined as Mac(n)[dx, dy]; the x-component of the third motion vectors are set to zero to obtain modified third motion vectors defined as Mac(n)[d0, dy]; the modified third motion vectors are utilized in the interpolating step (obtaining a virtual camera view by synthesizing a subset of camera views neighboring the position of the desired virtual camera; the virtual view may be generated using homography based on motion vectors or feature correspondences between two or more cameras; the virtual view may be generated using any other suitable view interpolation method, including, without limitation, (1) depth-from-stereo view interpolation, (2) sparse or dense motion vectors between two or more cameras, (3) synthetic aperture blending (image-based techniques), and/or (4) Deep-learning based view interpolation – [0096]).
It would have been obvious to a person with ordinary skill in the art, before the effective filing date of the claimed invention, to synthesize a virtual camera view by motion compensation as taught by Winter into the teachings of Hillis because such incorporation improves the quality of the image operations (par. 97).
Regarding claim 10 Hillis discloses the method in accordance with claim 1, wherein the camera is mounted on an outside of a viewing area of the screen (Fig. 1A shows cameras 41 and 42 outside of the viewing area of display 30).
Claim 13 corresponds to the system that performs the method of claim 1. Therefore, claim 13 is being rejected on the same basis as claim 1.
Claim 14 corresponds to the system that performs the method of claim 2. Therefore, claim 14 is being rejected on the same basis as claim 2.
Claim 15 corresponds to the system that performs the method of claim 3. Therefore, claim 15 is being rejected on the same basis as claim 3.
Claim 16 corresponds to the system that performs the method of claim 4. Therefore, claim 16 is being rejected on the same basis as claim 4.
Claim 17 corresponds to the system that performs the method of claim 5. Therefore, claim 17 is being rejected on the same basis as claim 5.
Claim 18 corresponds to the system that performs the method of claim 6. Therefore, claim 18 is being rejected on the same basis as claim 6.
Claim 19 corresponds to the system that performs the method of claim 7. Therefore, claim 19 is being rejected on the same basis as claim 7.
Claim 20 corresponds to the system that performs the method of claim 8. Therefore, claim 20 is being rejected on the same basis as claim 8.
Claim 21 corresponds to the system that performs the method of claim 9. Therefore, claim 21 is being rejected on the same basis as claim 9.
Claim 22 corresponds to the system that performs the method of claim 10. Therefore, claim 22 is being rejected on the same basis as claim 10.
Claim(s) 11-12 and 23-24 is/are rejected under 35 U.S.C. 103 as being unpatentable over Hillis et al. (US 2010/0283830) in view of Winter et al. (US 2022/0109822) further in view of Chang (US 2004/0085451).
Regarding claim 11 Hillis discloses the method in accordance with claim 1. However, fails to explicitly disclose a location of the virtual camera view is selectable by the user.
In his disclosure Chang teaches a location of the virtual camera view is selectable by the user (the user may select a virtual viewpoint located between the two cameras in order to view a synthesized image corresponding to the virtual viewpoint selected – [0014]).
It would have been obvious for a person with ordinary skill in the art, before the effective filing date of the claimed invention, to incorporate the teachings of Chang into the teachings of Hillis because such incorporation would enhance the experience of the user.
Regarding claim 12 Hillis discloses the method in accordance with claim 11, wherein a relative location between the camera and the virtual camera view remains constant regardless of a position or angle of a user's face relative to the screen (Fig. 1A shows the virtual camera view being on the front of user 10 and the location between cameras 41 and 42 and the virtual camera being constant).
Claim 23 corresponds to the system that performs the method of claim 11. Therefore, claim 23 is being rejected on the same basis as claim 11.
Claim 24 corresponds to the system that performs the method of claim 12. Therefore, claim 24 is being rejected on the same basis as claim 12.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MARIA E VAZQUEZ COLON whose telephone number is (571)270-1103. The examiner can normally be reached M-F 7:30 AM-3:30 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, CHRISTOPHER S KELLEY can be reached at (571)272-7331. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MARIA E VAZQUEZ COLON/ Examiner, Art Unit 2482
Prosecution Insights