DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement(s) (IDS) submitted on 11/16/2023 and 7/19/2024 is/are in compliance with the provisions of 37 CFR 1.97. Accordingly, the information disclosure statement(s) has/have been considered by the examiner.
Specification
The title of the invention is not descriptive. A new title is required that is clearly indicative of the invention to which the claims are directed. The following title is suggested: An image processing apparatus that estimates camera work for a reference video and generates a video mimicking the estimated camerawork of a 3d model of a subject.
The lengthy specification has not been checked to the extent necessary to determine the presence of all possible minor errors. Applicant’s cooperation is requested in correcting any errors of which applicant may become aware in the specification.
Examiner’s Note
The claim language of claim 3 is consistent with Superguide Corp. v. DirecTV Enterprises, Inc., 358 F.3d 870, 875, 69 USPQ2d 1865, 1868 (Fed. Cir. 2004) decision regarding the claim interpretation of “at least one of x, y, and z.” at pages 15-16 set forth the rational for determining that the term “and” is conjunctive requiring at least one instance of each and every item listed. Claim 3 recites such limitations. If Applicant intends for an interpretation of only one of these items being required for claim interpretation, Applicant can amend the claim language to, instead recite, “at least one of … or …”. In SuperGuide, the Federal Circuit held that the plain meaning of “at least one of A, B, and C” means: at least one A, at least one of B, and at least one of C. The Court held that if the applicant intended “at least one of A, B, and C” to mean A, B or C, they should have used “OR.” Therefore the plain and ordinary meaning of the current claim language “at least one of a change in a viewpoint position, a change in a line-of-sight direction, and a change in a viewing angle when the first video is acquired on a basis of a result of the tracking by the tracking unit” in light of the specification is interpreted to be “at least one of a change in a viewpoint position, at least one of a change in a line-of-sight direction, and at least one of a change in a viewing angle when the first video is acquired on a basis of a result of the tracking by the tracking”.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitation(s) is/are:
Claims 1, 9 and 10
tracking unit… tracking of a region structure found in ¶70 with hardware structure found Fig. 3 and ¶34-37
first estimation unit… estimate camerawork, structure found in ¶72
Claim 2
extraction unit… extract first subject, structure found in ¶69
Claim 5
acquisition unit… acquire multi-viewpoint image, structure found in ¶73
3D model generation unit.. generate 3D model of the second subject, structure found in ¶75
Second estimation unit… estimates arrangement and posture of first subject structure found in ¶71
Setting unit… set virtual viewpoint structure found in ¶76
Video generation unit… generate second video, structure found in ¶77
Claim 6
correlation estimation unit… estimate a corresponding relationship between the arrangement and the posture of the first subject estimated by the second estimation unit and the arrangement and the posture of the second subject in the multi-viewpoint image obtained by imaging the second subject from a plurality of viewpoints, structure found in ¶74
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 USC § 101
Claim 10 is rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter as follows. The claim is a “program causing a computer to function as” wherein the claim is explicitly directed to a computer program per se with no embodying non-transitory computer readable medium. Since a computer program does not fall within the statutory categories of 35 U.S.C. 101 (i.e., process, machine, manufacture, or composition of matter), the claim is ineligible under 35 U.S.C. 101. The Examiner suggests “A program” in line 1 of claim 10 to be -- A non-transitory computer readable medium that stores a program --.
Appropriate correction is required.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claim(s) 1-3, and 9-10 is/are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Ma (Ma S, Wang W. Effective camera motion analysis approach. In2010 International Conference on Networking, Sensing and Control (ICNSC) 2010 Apr 10 (pp. 111-116). IEEE.).
Regarding claim 1, Ma discloses An information processing apparatus comprising: a tracking unit configured to perform tracking of a region having a similar pixel pattern among a plurality of frame images included in a first video in which a first subject is captured; and (Ma Section II. Our CMA Approach – A. Tracking Keypoints ¶1 – found on p. 112 lefthand column and Tracking keypoints boxed process found on p. 113 righthand column; motion vectors are created for keypoints, which are tracked through frames. Section I. Introduction ¶3 found on p. 112 top of lefthand column; reducing foreground motion is discussed (i.e. the frame has a subject).) a first estimation unit configured to estimate camerawork for the first video on a basis of a result of the tracking by the tracking unit. (Ma Section II. Our CMA Approach – B. Analyzing Camera Motion Pattern ¶1; motion vectors are utilized to categorize camera motion.)
Regarding claim 2, Ma discloses the claim limitations with regards to claim 1, Ma further discloses further comprising an extraction unit configured to extract the first subject from frame images included in the first video, (Ma Section II. Our CMA Approach – B. Analyzing Camera Motion Pattern – Step 1 ¶1-2 found on lefthand column p. 114 and Step 2 ¶3-4 found on lefthand column p. 115; foreground keypoints are defined (subject is extracted).) wherein the tracking unit performs the tracking on a region excluding the first subject in the frame images among the plurality of frame images included in the first video. (Ma Section II. Our CMA Approach – C. Computing Parametric Model ¶1; the background key points are used to estimate camera motion and not foreground.)
Regarding claim 3, Ma discloses the claim limitations with regards to claim 1 as described above. Ma further discloses wherein the first estimation unit estimates at least one of a change in a viewpoint position, a change in a line-of-sight direction, and a change in a viewing angle when the first video is acquired on a basis of a result of the tracking by the tracking unit. (Ma Section II. Our CMA Approach – B. Analyzing Camera Motion Pattern ¶1 – found on p. 114 lefthand column; camera motion categories of static, translation (change in viewpoint position and change in viewing angle), zoom and rotation (change in line-of-sight and change in viewing angle) are disclosed.)
Regarding claims 9 and 10 they are the corresponding method and program claims to claim 1 and are rejected for similar reasons.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 4 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ma (Ma S, Wang W. Effective camera motion analysis approach. In2010 International Conference on Networking, Sensing and Control (ICNSC) 2010 Apr 10 (pp. 111-116). IEEE.) in view of Murahashi (Pub. No . US20140240602A1).
Regarding claim 4, Ma discloses the claim limitations with regards to claim 1 as described above.
Ma further discloses handling cases of flashing or occlusion but not explicitly scene changes (Ma Section II. Our CMA Approach – B. Analyzing Camera Motion Pattern – Step 2 ¶2 – found p. 114 righthand column to p. 115 righthand column)
Murahashi, however, discloses wherein when the tracking unit cannot track a region having the pixel pattern, the first estimation unit determines that a scene change occurs in the first video. (Murahashi ¶114-115 scene change is when interframe difference is greater than a threshold value.)
It would have been obvious, before the effective filing date of the claimed invention, to one of ordinary skill in the art to modify apparatus of Ma with the teachings of Murahashi by including detection of scene changes in order to be able to process camerawork of full films.
Claim(s) 5 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ma (Ma S, Wang W. Effective camera motion analysis approach. In2010 International Conference on Networking, Sensing and Control (ICNSC) 2010 Apr 10 (pp. 111-116). IEEE.) in view of Ashigahara (Pub. No. JP2002232783A see attached translation for paragraph numbers).
Regarding claim 5, Ma discloses the claim limitations with regards to claim 1 as described above.
Ma does not explicitly disclose further comprising: an acquisition unit configured to acquire a multi- viewpoint image obtained by imaging a second subject from a plurality of viewpoints; a 3D model generation unit configured to generate a 3D model of the second subject on a basis of the multi- viewpoint image acquired by the acquisition unit; a second estimation unit configured to estimate an arrangement and a posture of the first subject from frame images included in the first video; a setting unit configured to set a virtual viewpoint that moves along the camerawork estimated by the first estimation unit such that the second subject approaches the arrangement and the posture of the first subject estimated by the second estimation unit; and a video generation unit configured to generate a second video obtained by observing the 3D model from the virtual viewpoint set by the setting unit.
Ashigahara, however, discloses further comprising: an acquisition unit configured to acquire a multi- viewpoint image obtained by imaging a second subject from a plurality of viewpoints; (Ashigahara ¶9; an image pick up unit with a plurality of cameras is disclosed. ¶39 and Fig.2 an example configuration is disclosed) a 3D model generation unit configured to generate a 3D model of the second subject on a basis of the multi- viewpoint image acquired by the acquisition unit; (Ashigahara ¶3-5, 15, three-dimensional model of the users face (second subject) is disclosed) a second estimation unit configured to estimate an arrangement and a posture of the first subject from frame images included in the first video; (Ashigahara ¶44-44 and 47-48; the face of a specific character (first subject) in the material video (first video) is found.) a setting unit configured to set a virtual viewpoint that moves along the camerawork estimated by the first estimation unit such that the second subject approaches the arrangement and the posture of the first subject estimated by the second estimation unit; and (Ashigahara ¶49-51; matching the position, orientation, and size of the face is disclosed. ¶22 and 52 explains it can be a virtual view point.) a video generation unit configured to generate a second video obtained by observing the 3D model from the virtual viewpoint set by the setting unit. (Ashigahara ¶52; image combining unit places the face of the user (second subject) into the material video (first video) creating a new video.)
It would have been obvious, before the effective filing date of the claimed invention, to one of ordinary skill in the art to modify the apparatus of Ma with the teachings of Ashigahara by including filming a 3D model of a subject in order to be able to create scenes where a user can interact in place of a character in an already filmed scene (Ashigahara ¶2).
Claim(s) 6-7 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ma (Ma S, Wang W. Effective camera motion analysis approach. In2010 International Conference on Networking, Sensing and Control (ICNSC) 2010 Apr 10 (pp. 111-116). IEEE.) in view of Ashigahara (Pub. No. JP2002232783A see attached translation for paragraph numbers) and Germann (Pub. No . US20110267344A1).
Regarding claim 6, the combination of Ma and Ashigahara disclose the claim limitations with regards to claim 5, as described above. They further disclose further comprising a unit and (Ashigahara ¶43 and 87 choosing a frame where facial expressions are close to material video is disclosed) the arrangement and the posture of the second subject in the multi-viewpoint image obtained by imaging the second subject from a plurality of viewpoints, wherein the setting unit sets the virtual viewpoint such that the second subject and the first subject estimated to be associated with the second subject by the correlation estimation unit are captured at the same position, in the same direction, and in the same size. (Ashigahara ¶49 the same position, orientation, and size of face to be input (second subject) as the character’s face (first subject) is disclosed.)
The combination of Ma and Ashigahara disclose choosing a frame where the facial expressions match(Ma ¶43 and 87) but not explicitly a correlation estimation unit.
Germann, however discloses a correlation estimation unit (Germann ¶48-53 and ¶76-79; correlation between a 3d model and reference silhouette is calculated.)
It would have been obvious, before the effective filing date of the claimed invention, to one of ordinary skill in the art to modify the apparatus of the combination of Ma and Ashigahara with the teachings of Germann by including a correlation pose estimate in order to create the most realistic virtual image (Germann ¶53).
Regarding claim 7, the combination of Ma, Ashigahara and Germann disclose the claim limitations of claim 6, as described above. They further disclose wherein the correlation estimation unit estimates a correspondence relationship between the first subject and the second subject on a basis of a skeleton of the first subject and a skeleton of the second subject. (Germann ¶48-53 and ¶76-79; correlation between a 3d model (second subject) and reference silhouette(first subject).)
Claim(s) 8 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ma (Ma S, Wang W. Effective camera motion analysis approach. In2010 International Conference on Networking, Sensing and Control (ICNSC) 2010 Apr 10 (pp. 111-116). IEEE.) in view of Ashigahara (Pub. No. JP2002232783A see attached translation for paragraph numbers) and Isogawa (Pub. No. JP2015231114A see attached translation for paragraph numbers).
Regarding claim 8, the combination of Ma and Ashigahara disclose the claim limitations with regards to claim 5, as described above.
The combination of Ma and Ashigahara does not explicitly disclose wherein in a case where an imaging position of the second subject is in a vicinity of the virtual viewpoint set by the setting unit, the setting unit sets a virtual viewpoint that passes through the imaging position.
Isogawa, however, discloses wherein in a case where an imaging position of the second subject is in a vicinity of the virtual viewpoint set by the setting unit, the setting unit sets a virtual viewpoint that passes through the imaging position. (Isogawa ¶23-31; transitioning between a user selected view and an actually captured view is discussed.)
It would have been obvious, before the effective filing date of the claimed invention, to one of ordinary skill in the art to modify the apparatus of the combination of Ma and Ashigahara with the teachings of Isogawa by including virtual viewpoints that pass through view points that were actually acquired in order to provide the highest resolution view to the user, while still providing a view close to their selection (Isogawa ¶2-3).
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to MEREDITH TAYLOR whose telephone number is (571)270-5805. The examiner can normally be reached M-Th 7:30-5.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Vincent Rudolph can be reached at (571)272-8243. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MEREDITH TAYLOR/Examiner, Art Unit 2671
/VINCENT RUDOLPH/Supervisory Patent Examiner, Art Unit 2671
Prosecution Insights