Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Arguments
Applicant’s arguments with respect to claim(s) 1-11, 13-20 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Specifically, claims 1-11, 13-18 have been amended and newly presented subject matter may be found in sections [0185-0188, 0200,0207-0210] of the original specifications applicant states.
In regards to the U.S.C 103 rejection of claims 1 and 18, applicant argues that the prior art of “Kobayashi merely describes that a barycenter of an object is projected onto a depth image for a current frame and a previous frame, and a translation amount is calculated in order to determine a viewpoint of a virtual camera that cancels movement of the object. In particular, Kobayashi determines the viewpoint of the virtual camera so as to stabilize the object by compensating for motion between frames. However, Kobayashi is not shown to teach or suggest an imaging system including a camera that comprises a mode selection unit configured to control whether an origin position moves in conjunction with movement of the first camera and whether the origin position is correctable by a user (page 10).” This is not persuasive since applicant seems to take the prior art in insolation, it is the combination of the teachings of Nakazato in view of Kobayashi that reads upon the newly amended claims as currently stated using the broadest reasonable interpretation since the claim language still reads broad.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-11, 13-16, 18-20 are rejected under 35 U.S.C. 103 as being unpatentable over Nakazato (U.S. Patent No.: 10,785,469 B2) in view of Kobayashi et al. (U.S. Publication No.: 2019/0266786 A1).
In regards to claim 1, Nakazato teaches of an image processing apparatus comprising:
a 2D video generation unit configured to:
acquire, as virtual viewpoint information, camera positional information of a first camera (Fig. 1, item 1001, column 2, lines 47-50) that executes an imagine operation at a first place (column 2, lines 47-50).
acquire a 3D model, of a person, generated based on an imagine operation executed at a second place different from the first place (Fig. 2, column 2, lines 59-63), and
wherein an imaging system including the first camera includes a mode selection unit configured to control whether an origin position moves in conjunction with movement of the first camera and whether the origin position is correctable by a user (column 4, lines 48-60).
See Fig. 4 (generation apparatus 100, virtual viewpoint acquisition unit 103).
See §§10–11, 18–21, 27–29 (system acquires camera parameters, generates 2D video from virtual viewpoint using 3D model and images from different locations).
See §29: “The rendering unit 105 generates (renders) a virtual viewpoint image, based on the three-dimensional model, the correspondence information, the captured image…and the virtual viewpoint information…”
Nakazato fails to explicitly teach the concept of generates a 2D video obtained based on view of the 3D model of the person from a viewpoint corresponding to the camera positional information.
Kobayashi teaches of the concept of generates a 2D video obtained based on view of the 3D model of the person from a viewpoint corresponding to the camera positional information (Fig. 6, section [0061]).
It would have been obvious to one with ordinary skills in the art to incorporate the 2D video as taught by Kobayashi with the 2D model of the person as taught by Nakazato in order to improve the accuracy of a 3D model depth image (Kobayashi, section [0004]).
In regards to claim 2, Nakazato in view of Kobayashi teaches that the image processing apparatus according to claim 1, wherein the 2D video generation unit is further configured to generate a 2D video and a depth video of the person viewed from the viewpoint of the first camera.
Nakazato See §§20, 22, 29 (the system uses 3D models and can generate both texture-mapped 2D images and, by implication, depth information from the model; see also discussion of mesh models and correspondence mapping).
Nakazato See §38: “The three-dimensional model may be any type of information as long as the information represents a three-dimensional shape.” (implies depth data available).
In regards to claim 3, Nakazato in view of Kobayashi teaches that the image processing apparatus according to claim 1, wherein a frame number is included in the virtual viewpoint information, and
the 2D video generation unit is further configured to:
assign the frame number to the generated 2D video, and output the 2D video.
Nakazato See §§27–29, 34–35 (processing is performed “frame by frame,” with virtual viewpoint information and images acquired and rendered per frame).
Nakazato See Fig. 5 (flowchart showing per-frame processing: S1040, S1050, S1060).
In regards to claim 4, Nakazato in view of Kobayashi teaches that the image processing apparatus according to claim 1, further comprising:
a video synthesizing unit configured to generate a synthesized image obtained based on synthesis of the the 2D video generated by the 2D video generation unit and a 2D video generated by the first camera.
Nakazato See §§53–57 (foreground/background separation and synthesis; rendering unit synthesizes images from generated viewpoint and captured camera images).
Nakazato See §57: “…the rendered foreground image is then superimposed on the background image.”
In regards to claim 5, Nakazato in view of Kobayashi teaches that the image processing apparatus according to claim 4, wherein the video synthesizing unit is further configured to:
select a closer subject from the 2D video generated by the 2D video generation unit and the 2D video generated by the first camera, and
generate the synthesized image.
Nakazato See §64: “When a virtual viewpoint image representing a scene from a virtual viewpoint is rendered, the foreground image may be synthesized on the background image (frontward as viewed from the virtual camera), or synthesis may be performed after a depth is determined based on depth information of the foreground image and the background image.”
Nakazato (System selects and composites closer subjects based on depth.)
In regards to claim 6, Nakazato in view of Kobayashi teaches that the image processing apparatus according to claim 4, wherein the video synthesizing unit is further configured to generate the synthesized image based on the synthesis of the 2D video generated by the 2D video generation unit and the 2D video generated by the first camera, which have a same frame number.
Nakazato See §§34–35, Fig. 5 (frame-by-frame processing; synthesis performed per frame).
Nakazato See §61: “In step S2060, the rendering unit 205 renders (generates) a virtual viewpoint image based on the virtual viewpoint information acquired in step S2040. In addition to the foreground image and the background image…”
In regards to claim 7, Nakazato in view of Kobayashi teaches that the image processing apparatus according to claim 1, further comprising:
a plurality of the 2D video generation units, wherein
the plurality of the 2D video generation unit includes the 2D video generation unit and
the plurality of the 2D video generation units is configured to:
acquire, as the virtual viewpoint information, pieces of camera positional information from a plurality of cameras that include the first camera, and (Kobayashi, Fig. 6, section [0061]).
generate 2D videos.
Nakazato See §§11, 18–21, 27 (system supports multiple cameras and can generate multiple virtual viewpoint images using different camera parameters).
In regards to claim 8, Nakazato in view of Kobayashi teaches that the image processing apparatus according to claim 1, further comprising:
a plurality of 2D video generation units, wherein
the plurality of 2D video generation units includes the 2D video generation unit, and (Kobayashi, Fig. 6, section [0061]).
plurality of 2D video generation units are configured to:
acquire as the virtual viewpoint information, the camera positional information from the first camera, and
generate 2D videos.
Nakazato See §§11, 18–21, 27 (system supports multiple cameras and can generate multiple virtual viewpoint images using different camera parameters).
In regards to claim 9, Nakazato in view of Kobayashi teaches that the image processing apparatus according to claim 1, further comprising:
a plurality of video synthesizing units configured to synthesize 2D videos generated by a plurality of the 2D video generation units and 2D videos imaged by a plurality of cameras, wherein
the plurality of 2D video generation units include the 2D video generation unit, and
the plurality of cameras includes the first camera.
Nakazato See §§11, 18–21, 27 (system supports multiple cameras and can generate multiple virtual viewpoint images using different camera parameters).
In regards to claim 10, Nakazato in view of Kobayashi teaches that the image processing apparatus according to claim 9, further comprising:
a selection unit configured to:
select one of a plurality of synthesized images generated by the plurality of video synthesizing units, and
output the selected one of the plurality of synthesized image.
Nakazato See §§11, 18–21, 27 (system supports multiple cameras and can generate multiple virtual viewpoint images using different camera parameters).
In regards to claim 11, Nakazato in view of Kobayashi teaches that the image processing apparatus according to claim 1, wherein the first camera is a camera that outputs a 2D video and a depth video.
Nakazato See §49: “The camera 1001 may be a color camera or a gray camera, or may be an RGB-D camera or a three-dimensional measurement apparatus.”
Nakazato (RGB-D or 3D camera outputs both 2D and depth data.)
In regards to claim 15, Nakazato in view of Kobayashi teaches that the image processing apparatus according to claim 1, wherein the 2D video generation unit is further configured to:
acquire illumination information of the first place, and
output the acquired illumination information to an illumination control device that controls an illumination device at the second place.
Nakazato See §51: “The rendering unit 105 may set a light source based on the time and weather at the time of imaging, when performing rendering…”
Nakazato (Implied acquisition and use of illumination information.)
In regards to claim 18, Nakazato in view of Kobayashi teaches of an image processing system comprising:
a 2D video generation device configured to:
that acquire, as virtual viewpoint information, camera positional information of a camera that executes an imaging operation at a first place,
acquire a 3D model, of a person, generated based on an imaging operation executed at a second place different from the first place, and
generate a 2D video obtained by based on view of the 3D model of the person from a viewpoint corresponding to the camera positional information,
wherein an imaging system including the camera includes a mode selection unit configured to control whether an origin position moves in conjunction with movement of the camera and whether the origin position is correctable by a user; and (Kobayashi, section [0004]).
a video synthesizing device configured to generate a synthesized image obtained based on synthesis of a 2D video generated by the 2D video generation device and a 2D video generated by the camera.
Nakazato See §§18–19, 27–29, 53–57 (system includes a generation apparatus with units for acquiring virtual viewpoint information, generating 2D video from a 3D model, and synthesizing with captured camera video).
In regards to claim 19, Nakazato in view of Kobayashi teaches that the image processing apparatus of claim 1, wherein the mode selection unit is further configured to switch among a mode in which the origin position moves in conjunction with the movement of the first camera, a mode in which the origin position is fixed, and a mode in which the origin position is able to be corrected (Kobayashi, section [0004]).
In regards to claim 20, Nakazato in view of Kobayashi teaches that the image processing system of claim 18, wherein the mode selection unit is further configured to switch among a mode in which the origin position moves in conjunction with the movement of the camera, a mode in which the origin position is fixed, and a mode in which the origin position is able to be corrected (Kobayashi, section [0004]).
In regards to claim 13, Nakazato fails to explicitly teach that the image processing apparatus according to claim 1, wherein the first camera includes a camera of a smartphone.
Kobayashi teaches that the camera includes a camera of a smartphone (see paragraph [0215]].
It would have been obvious to one with ordinary skills in the art to take the phone camera of Kobayashi to be used with the image process apparatus of Nakazoto in order to help improve the 3D model depth image process (Kobayashi, section [0004]).
In regards to claim 14, Nakazato fails to explicitly teach that the image processing apparatus according to claim 1, wherein the first camera includes a camera of a drone.
Kobayashi teaches that the camera includes a camera of a drone (see section [0199]).
It would have been obvious to one with ordinary skills in the art to incorporate the drone of Kobayashi to be used with the image process apparatus of Nakazoto in order to help improve the 3D model depth image process (Kobayashi, section [0004]).
In regards to claim 16, Nakazato fails to explicitly teach that the image processing apparatus according to claim 1, wherein the imaging system including the first camera includes a second camera that images a periphery of the first camera, and a video of the second camera is configured to be displayed on a display of the second place.
Kobayashi teaches of an imaging system including the camera includes a second camera that images a periphery of the camera, and a video of the second camera is configured to be displayed on a display of the second place (see section [0056]).
It would have been obvious to one with ordinary skills in the art to incorporate the concept of having a second camera as taught by Kobayashi to be used with the image process apparatus of Nakazoto in order to help improve the 3D model depth image process (Kobayashi, section [0004]).
Claim 17 is rejected under 35 U.S.C. 103 as being unpatentable over Nakazato (U.S. Patent No.: 10,785,469 B2).
In regards to claim 17, Nakazato fails to explicitly teach that the image processing apparatus according to claim 1, wherein the 2D video generation unit is further configured to acquire the virtual viewpoint information by using a FreeD protocol.
However, applicant fails to specify why the exact method of using the FreeD protocol is needed since hence examiner uses design choice to reject claim 17.
Nakazato teaches that the image processing apparatus according to claim 1, wherein the 2D video generation unit acquires the virtual viewpoint information by using a protocol (Fig. 6).
Because the end result of both Nakazato and applicant are the same and both methods seem equally capable of generating a virtual viewpoint it would have been obvious to one with ordinary skills in the art to use the method as taught by Nakazato with the teachings of the applicant since both yield substantially similar results.
Conclusion
THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to TAMMY GODDARD whose telephone number is (571)272-7773. The examiner can normally be reached 9am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/TAMMY GODDARD/Supervisory Patent Examiner, Art Unit 2611
Prosecution Insights