DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on nonstatutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The filing of a terminal disclaimer by itself is not a complete reply to a nonstatutory double patenting (NSDP) rejection. A complete reply requires that the terminal disclaimer be accompanied by a reply requesting reconsideration of the prior Office action. Even where the NSDP rejection is provisional the reply must be complete. See MPEP § 804, subsection I.B.1. For a reply to a non-final Office action, see 37 CFR 1.111(a). For a reply to final Office action, see 37 CFR 1.113(c). A request for reconsideration while not provided for in 37 CFR 1.113(c) may be filed after final for consideration. See MPEP §§ 706.07(e) and 714.13.
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The actual filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/apply/applying-online/eterminal-disclaimer.
Claims 1-3 and 5-14 are rejected on the ground of nonstatutory double patenting as being unpatentable over claims 1-3 and 5-14 of U.S. Patent No. 12,112,425 in view of Aizawa U.S. Pub. No. 2019/0349531. Although the claims at issue are not identical, they are not patentably distinct from each other because the claims of the Instant Application are obvious over the claims of the U.S. Patent in view of Aizawa U.S. Pub. No. 2019/0349531.
Instant Application 18/824,919
U.S. Patent No. 12,112,425
1. An information processing apparatus comprising: a processor; and a memory that is connected to or incorporated in the processor, wherein the processor acquires reference imaging device information corresponding to a position, an imaging direction, and an angle of view of a reference imaging device, and on a condition that an instruction to start generating a virtual viewpoint image based on a plurality of images obtained by imaging an imaging region with a plurality of imaging devices is given, generates the virtual viewpoint image by using the reference imaging device information as a reference,
acquires image quality information indicating an image quality of a first reference image obtained by imaging the imaging region with the reference imaging device,
determines an image quality of the virtual viewpoint image on the basis of the image quality information on a condition that the instruction is given,
generates the virtual view point image by using the reference imaging device information and the image quality of the virtual view point image,
and outputs a superimposed image in which the virtual viewpoint image is superimposed on a second reference image obtained by imaging the imaging region with the reference imaging device, on a condition that the instruction is given.
1. An information processing apparatus comprising: a processor; and a memory that is connected to or incorporated in the processor, wherein the processor acquires reference imaging device information corresponding to a position, an imaging direction, and an angle of view of a reference imaging device, and on a condition that an instruction to start generating a virtual viewpoint image based on a plurality of images obtained by imaging an imaging region with a plurality of imaging devices is given, generates the virtual viewpoint image by using the reference imaging device information as a reference,
wherein a position, an imaging direction, and an angle of view of the virtual viewpoint image match the position, the imaging direction, and the angle of view of the reference imaging device.
4. The information processing apparatus according to claim 1, wherein the processor is configured to acquire image quality information indicating an image quality of a first reference image obtained by imaging the imaging region with the reference imaging device,
and determine an image quality of the virtual viewpoint image on the basis of the image quality information on a condition that the instruction is given.
5. The information processing apparatus according to claim 1, wherein the processor is configured to output a superimposed image in which the virtual viewpoint image is superimposed on a second reference image obtained by imaging the imaging region with the reference imaging device, on a condition that the instruction is given.
Claim 1 of the instant Application is obvious over claims 1, 4 and 5 of the U.S. Patent. Claim 1 of the Instant Application has an additional limitation that is taught by Aizawa. Aizawa teaches generates the virtual view point image by using the reference imaging device information and the image quality of the virtual view point image. (“An image generator 303 obtains the photographed images (captured images) that are photographed by the physical cameras and obtains the various kinds of information about the virtual camera at the virtual viewpoint from the virtual-information-obtaining unit 302. The image generator 303 has a image-generation unit function. The image generator 303 configured to generate the virtual viewpoint image based on the images and the virtual viewpoint. ”; Aizawa, [0040]). The image generator generates the virtual viewpoint image (generates the virtual viewpoint image) based on the images captured by the physical camera (by using the reference imaging device information).
(“The image generator 303 generates the virtual viewpoint image that is seen from the viewpoint (virtual viewpoint) of the virtual camera on the basis of the photographed images (captured images) from the physical cameras and the information about the virtual camera... Examples of the information about the virtual camera include a position, a posture, an angle of view, and the number of pixels as in the physical camera.”; Aizawa, [0040], [0039]). The image generator generates the virtual viewpoint image (generates the virtual viewpoint image) on the basis of images captured by the physical camera (by using the reference imaging device information) and information about the virtual camera, such as, a position, a posture, an angle of view and the number of pixels in a physical camera (the image quality of the virtual view point image).
(“Since the virtual viewpoint image is generated on the basis of the images that are photographed by the physical cameras as described above, the generated virtual viewpoint image when the virtual viewpoint image is near a location at which the physical cameras are densely arranged can have an image quality higher than that when the virtual viewpoint image is near a location at which the physical cameras are sparsely arranged.”; Aizawa, [0049]). Since the virtual viewpoint image is generated based on images captured by the physical cameras, the image quality of the virtual viewpoint image is higher (image quality of the virtual viewpoint image) when the virtual viewpoint image is near a location where the physical cameras are densely arranged and the image quality of the virtual viewpoint image is lower (image quality of the virtual viewpoint image) when the virtual viewpoint image is near a location where the physical cameras are sparsely arranged. Therefore, it would have been obvious to one of ordinary skill in the art at the time of the effective filing date, to modify the U.S. Patent by adding the feature of generates the virtual view point image by using the reference imaging device information and the image quality of the virtual view point image, in order to allow the operator to know a range in which a virtual viewpoint image that has a higher image quality can be generated, as taught by Aizawa ([0049]).
2. The information processing apparatus according to claim 1, wherein in a case where the reference imaging device information is continuously transmitted by a transmission device, the processor generates the virtual viewpoint image by using, as a reference, the reference imaging device information transmitted from the transmission device, on the condition that the instruction is given.
2. The information processing apparatus according to claim 1, wherein in a case where the reference imaging device information is continuously transmitted by a transmission device, the processor is configured to generate the virtual viewpoint image by using, as a reference, the reference imaging device information transmitted from the transmission device, on a condition that the instruction is given.
3. The information processing apparatus according to claim 1, wherein the processor updates the reference imaging device information each time the processor acquires the reference imaging device information.
3. The information processing apparatus according to claim 1, wherein the processor is configured to update the reference imaging device information each time the processor acquires the reference imaging device information.
5. The information processing apparatus according to claim 1,
wherein the processor gradually changes a ratio between the second reference image and the virtual viewpoint image in the superimposed image.
5. The information processing apparatus according to claim 1, wherein the processor is configured to output a superimposed image in which the virtual viewpoint image is superimposed on a second reference image obtained by imaging the imaging region with the reference imaging device, on a condition that the instruction is given.
6. The information processing apparatus according to claim 5, wherein the processor is configured to gradually change a ratio between the second reference image and the virtual viewpoint image in the superimposed image.
Claim 6 of the U.S. Patent depends from claim 5. Claim 5 of the Instant Application depends from claim 1. The limitation of claim 5 of the U.S. Patent is recited in claim 1 of the Instant Application.
6. The information processing apparatus according to claim 1, wherein the processor outputs the virtual viewpoint image to a display,
and receives a change signal for continuously changing at least one of a viewpoint position, a line-of-sight direction, or an angle of view in the output virtual viewpoint image.
7. The information processing apparatus according to claim 1, wherein the processor is configured to output the virtual viewpoint image to a display,
and receive a change signal for continuously changing at least one of a viewpoint position, a line-of-sight direction, or an angle of view in the output virtual viewpoint image.
7. The information processing apparatus according to claim 1, wherein the reference imaging device is an imaging device capable of changing at least one of the position, the imaging direction, or the angle of view.
8. The information processing apparatus according to claim 1, wherein the reference imaging device is an imaging device capable of changing at least one of the position, the imaging direction, or the angle of view.
8. The information processing apparatus according to claim 1, wherein the processor acquires the reference imaging device information on the basis of a third reference image obtained by imaging the imaging region with the reference imaging device.
9. The information processing apparatus according to claim 1, wherein the processor is configured to acquire the reference imaging device information on the basis of a third reference image obtained by imaging the imaging region with the reference imaging device.
9. The information processing apparatus according to claim 1, wherein the reference imaging device information is the position, the imaging direction, and the angle of view of the reference imaging device.
10. The information processing apparatus according to claim 1, wherein the reference imaging device information is the position, the imaging direction, and the angle of view of the reference imaging device.
10. The information processing apparatus according to claim 1, wherein the reference imaging device is one of the plurality of imaging devices.
11. The information processing apparatus according to claim 1, wherein the reference imaging device is one of the plurality of imaging devices.
11. The information processing apparatus according to claim 10, wherein the reference imaging device is capable of being switched between the plurality of imaging devices.
12. The information processing apparatus according to claim 11, wherein the reference imaging device is capable of being switched between the plurality of imaging devices.
12. The information processing apparatus according to claim 1, wherein the processor outputs a fourth reference image obtained by imaging the imaging region with the reference imaging device,
acquires the reference imaging device information, and generates a virtual viewpoint image corresponding to the fourth reference image by using the reference imaging device information as a reference, on the condition that the instruction is given.
13. The information processing apparatus according to claim 1, wherein the processor is configured to output a fourth reference image obtained by imaging the imaging region with the reference imaging device,
acquire the reference imaging device information, and generate a virtual viewpoint image corresponding to the fourth reference image by using the reference imaging device information as a reference, on a condition that the instruction is given.
13. A method of operating an information processing apparatus including a processor and a memory connected to or incorporated in the processor, the method comprising: acquiring reference imaging device information corresponding to a position, an imaging direction, and an angle of view of a reference imaging device; on a condition that an instruction to start generating a virtual viewpoint image based on a plurality of images obtained by imaging an imaging region with a plurality of imaging devices is given, generating the virtual viewpoint image by using the reference imaging device information as a reference;
acquiring image quality information indicating an image quality of a first reference image obtained by imaging the imaging region with the reference imaging device;
determines an image quality of the virtual viewpoint image on the basis of the image quality information on a condition that the instruction is given,
generates the virtual view point image by using the reference imaging device information and the image quality of the virtual view point image
and outputting a superimposed image in which the virtual viewpoint image is superimposed on a second reference image obtained by imaging the imaging region with the reference imaging device, on a condition that the instruction is given.
1. An information processing apparatus comprising: a processor; and a memory that is connected to or incorporated in the processor, wherein the processor acquires reference imaging device information corresponding to a position, an imaging direction, and an angle of view of a reference imaging device, and on a condition that an instruction to start generating a virtual viewpoint image based on a plurality of images obtained by imaging an imaging region with a plurality of imaging devices is given, generates the virtual viewpoint image by using the reference imaging device information as a reference,
wherein a position, an imaging direction, and an angle of view of the virtual viewpoint image match the position, the imaging direction, and the angle of view of the reference imaging device.
4. The information processing apparatus according to claim 1, wherein the processor is configured to acquire image quality information indicating an image quality of a first reference image obtained by imaging the imaging region with the reference imaging device,
and determine an image quality of the virtual viewpoint image on the basis of the image quality information on a condition that the instruction is given.
5. The information processing apparatus according to claim 1, wherein the processor is configured to output a superimposed image in which the virtual viewpoint image is superimposed on a second reference image obtained by imaging the imaging region with the reference imaging device, on a condition that the instruction is given.
Claim 13 of the instant Application is obvious over claims 1, 4 and 5 of the U.S. Patent, where claim 13 is a method and claims 1, 4 and 5 are an apparatus. Claim 13 of the Instant Application has an additional limitation that is taught by Aizawa. Aizawa teaches generating the virtual view point image by using the reference imaging device information and the image quality of the virtual view point image. (“An image generator 303 obtains the photographed images (captured images) that are photographed by the physical cameras and obtains the various kinds of information about the virtual camera at the virtual viewpoint from the virtual-information-obtaining unit 302. The image generator 303 has a image-generation unit function. The image generator 303 configured to generate the virtual viewpoint image based on the images and the virtual viewpoint.”; Aizawa, [0040]). The image generator generates the virtual viewpoint image (generating the virtual viewpoint image) based on the images captured by the physical camera (by using the reference imaging device information).
(“The image generator 303 generates the virtual viewpoint image that is seen from the viewpoint (virtual viewpoint) of the virtual camera on the basis of the photographed images (captured images) from the physical cameras and the information about the virtual camera... Examples of the information about the virtual camera include a position, a posture, an angle of view, and the number of pixels as in the physical camera.”; Aizawa, [0040], [0039]). The image generator generates the virtual viewpoint image (generating the virtual viewpoint image) on the basis of images captured by the physical camera (by using the reference imaging device information) and information about the virtual camera, such as, a position, a posture, an angle of view and the number of pixels in a physical camera (the image quality of the virtual view point image).
(“Since the virtual viewpoint image is generated on the basis of the images that are photographed by the physical cameras as described above, the generated virtual viewpoint image when the virtual viewpoint image is near a location at which the physical cameras are densely arranged can have an image quality higher than that when the virtual viewpoint image is near a location at which the physical cameras are sparsely arranged.”; Aizawa, [0049]). Since the virtual viewpoint image is generated based on images captured by the physical cameras, the image quality of the virtual viewpoint image is higher (image quality of the virtual viewpoint image) when the virtual viewpoint image is near a location where the physical cameras are densely arranged and the image quality of the virtual viewpoint image is lower (image quality of the virtual viewpoint image) when the virtual viewpoint image is near a location where the physical cameras are sparsely arranged. Therefore, it would have been obvious to one of ordinary skill in the art at the time of the effective filing date, to modify the U.S. Patent by adding the feature of generating the virtual view point image by using the reference imaging device information and the image quality of the virtual view point image, in order to allow the operator to know a range in which a virtual viewpoint image that has a higher image quality can be generated, as taught by Aizawa ([0049]).
14. A non-transitory computer-readable storage medium storing a program executable by a computer that is applied to an information processing apparatus including a processor and a memory connected to or incorporated in the processor to perform a process comprising: acquiring reference imaging device information corresponding to a position, an imaging direction, and an angle of view of a reference imaging device; on a condition that an instruction to start generating a virtual viewpoint image based on a plurality of images obtained by imaging an imaging region with a plurality of imaging devices is given, generating the virtual viewpoint image by using the reference imaging device information as a reference;
acquiring image quality information indicating an image quality of a first reference image obtained by imaging the imaging region with the reference imaging device;
determines an image quality of the virtual viewpoint image on the basis of the image quality information on a condition that the instruction is given,
generates the virtual view point image by using the reference imaging device information and the image quality of the virtual view point image
and outputting a superimposed image in which the virtual viewpoint image is superimposed on a second reference image obtained by imaging the imaging region with the reference imaging device, on a condition that the instruction is given.
1. An information processing apparatus comprising: a processor; and a memory that is connected to or incorporated in the processor,
wherein the processor acquires reference imaging device information corresponding to a position, an imaging direction, and an angle of view of a reference imaging device, and on a condition that an instruction to start generating a virtual viewpoint image based on a plurality of images obtained by imaging an imaging region with a plurality of imaging devices is given, generates the virtual viewpoint image by using the reference imaging device information as a reference,
wherein a position, an imaging direction, and an angle of view of the virtual viewpoint image match the position, the imaging direction, and the angle of view of the reference imaging device.
4. The information processing apparatus according to claim 1, wherein the processor is configured to acquire image quality information indicating an image quality of a first reference image obtained by imaging the imaging region with the reference imaging device,
and determine an image quality of the virtual viewpoint image on the basis of the image quality information on a condition that the instruction is given.
5. The information processing apparatus according to claim 1, wherein the processor is configured to output a superimposed image in which the virtual viewpoint image is superimposed on a second reference image obtained by imaging the imaging region with the reference imaging device, on a condition that the instruction is given.
Claim 14 of the instant Application is obvious over claims 1, 4 and 5 of the U.S. Patent, where claim 14 is medium and claims 1, 4, and 5 are an apparatus. Claim 14 of the Instant Application has an additional limitation that is taught by Aizawa. Aizawa teaches generating the virtual view point image by using the reference imaging device information and the image quality of the virtual view point image. (“An image generator 303 obtains the photographed images (captured images) that are photographed by the physical cameras and obtains the various kinds of information about the virtual camera at the virtual viewpoint from the virtual-information-obtaining unit 302. The image generator 303 has a image-generation unit function. The image generator 303 configured to generate the virtual viewpoint image based on the images and the virtual viewpoint.”; Aizawa, [0040]). The image generator generates the virtual viewpoint image (generating the virtual viewpoint image) based on the images captured by the physical camera (by using the reference imaging device information).
(“The image generator 303 generates the virtual viewpoint image that is seen from the viewpoint (virtual viewpoint) of the virtual camera on the basis of the photographed images (captured images) from the physical cameras and the information about the virtual camera... Examples of the information about the virtual camera include a position, a posture, an angle of view, and the number of pixels as in the physical camera.”; Aizawa, [0040], [0039]). The image generator generates the virtual viewpoint image (generating the virtual viewpoint image) on the basis of images captured by the physical camera (by using the reference imaging device information) and information about the virtual camera, such as, a position, a posture, an angle of view and the number of pixels in a physical camera (the image quality of the virtual view point image).
(“Since the virtual viewpoint image is generated on the basis of the images that are photographed by the physical cameras as described above, the generated virtual viewpoint image when the virtual viewpoint image is near a location at which the physical cameras are densely arranged can have an image quality higher than that when the virtual viewpoint image is near a location at which the physical cameras are sparsely arranged.”; Aizawa, [0049]). Since the virtual viewpoint image is generated based on images captured by the physical cameras, the image quality of the virtual viewpoint image is higher (image quality of the virtual viewpoint image) when the virtual viewpoint image is near a location where the physical cameras are densely arranged and the image quality of the virtual viewpoint image is lower (image quality of the virtual viewpoint image) when the virtual viewpoint image is near a location where the physical cameras are sparsely arranged. Therefore, it would have been obvious to one of ordinary skill in the art at the time of the effective filing date, to modify the U.S. Patent by adding the feature of determines an image quality of the virtual viewpoint image on the basis of the image quality information on a condition that the instruction is given, generates the virtual view point image by using the reference imaging device information and the image quality of the virtual view point image, in order to allow the operator to know a range in which a virtual viewpoint image that has a higher image quality can be generated, as taught by Aizawa ([0049]).
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1, 13, 14; 4, 5, 7, 8, 9, 10, 11 and 12 is/are rejected under 35 U.S.C. 103 as being unpatentable over Umemura U.S. Pub. No. 2020/0066028 in view of Uemura U.S. Pub. No. 2021/0160473, Hanamoto U.S. Patent No. 10,970,915, Aizawa U.S. Pub. No. 2019/0349531 and Ogata U.S. Pub. No. 2019/0043245.
Re: claims 1, 13 and 14, Umemura teaches
1. (Currently Amended) An information processing apparatus comprising: a processor; and a memory that is connected to or incorporated in the processor, (“Within each of the camera systems 110a to 110j, image capturing units 111a to 111j each including a lens, an imaging sensor, and the like and camera adaptors 112a to 112j each performing control of the image capturing unit and predetermined image processing in accordance with instructions of the control device 130 are included. The camera adaptor includes a calculation processing device (CPU or ASIC) and memories (RAM or ROM) necessary for control and image processing.”; Umemura, [0037], Fig. 3)
The camera system includes a camera adaptor that includes CPU (processor) and a RAM (memory that is connected to or incorporated in the processor).
and on a condition that an instruction to start generating a virtual viewpoint image based on a plurality of images obtained by imaging an imaging region with a plurality of imaging devices is given, (“At step 906, based on user instructions, the control device 130 transmits instructions to generate a virtual viewpoint image (generation start command) to the server 140 along with the information relating to the virtual viewpoint and the gaze point.”; Umemura, [0051])
The virtual viewpoint image is generated based on the control device transmitting instructions, such as a generation start command (on a condition that an instruction to start generating a virtual viewpoint image), to generate a virtual viewpoint image, to the server.
(“’Fig. 9 is a sequence diagram showing a generation process of a virtual viewpoint image… In accordance with the start of a soccer game, or the like, at step 901, the control device 130 sends instructions to capture a multi-viewpoint image (image capturing start command), which is the source of a virtual viewpoint image, to the server 140… Then, at step 903, each of the camera systems 110a to 110j starts image capturing of the target three-dimensional space (here, three-dimensional space on the field 200). Due to this, for example, in the camera system 110i,an image during a soccer game as shown in Fig. 10A is obtained. Then, image capturing of the image such as this, whose viewpoint is different from one another, is performed in each camera system. ”; Umemura, [0048], [0049], Figs. 9 and 10A)
The control device sends instructions to capture a multi-viewpoint image, which is the source of a virtual viewpoint image, to the server. Fig. 10 A illustrates that each of the camera systems 110a-110j starts capturing images of the field 200 (based on a plurality of images obtained by imaging a region with a plurality of imaging devices).
Umemura and Uemura teach wherein the processor acquires reference imaging device information corresponding to a position, an imaging direction, and an angle of view of a reference imaging device, (“The camera parameters refer to information on the installation position and orientation (line-of-sight direction) of each of the image capturing units 111a to 111j, the focal distance of the lens, and the like... the server 140 generates a three-dimensional model of the structure (here, the soccer goal 202) configured by voxels described previously based on the image data of the structure received from each camera system and the camera parameters of each camera system.”; Umemura, [0040], [0046])
The server (processor) receives camera parameters, such as installation position (reference imaging device information corresponding to a position), line-of sight direction (imaging direction and angle of view of a reference imaging device). Umemura is silent regarding an angle of view, however, Uemura teaches this limitation.
(“The virtual viewpoint video imaging group 20 is an imaging apparatus group including multiple imaging apparatuses installed to surround a game field or the like… The virtual viewpoint video imaging apparatus 20 outputs images captured by the respective imaging apparatuses to the image processing apparatus 30… The image processing apparatus 30 outputs the generated virtual viewpoint video to the video switching apparatus 10 or the storage apparatus 50… The video switching apparatus 10 includes… a virtual viewpoint control unit 104… The virtual viewpoint control unit 103 sets virtual viewpoint parameters (setting information of a virtual viewpoint such as the position of the virtual viewpoint, the direction of view from the virtual viewpoint, and the angle of view) at the switch time N based on the obtained imaging apparatus parameters at the switch time N… Then the virtual viewpoint control unit 103 outputs the virtual viewpoint parameters corresponding to all of times included int eh obtained time information to the image processing apparatus 30 as control information.”; Uemura, [0017], [0018], [0021], [0024], Fig. 1)
Fig. 1 illustrates the viewpoint control unit 103, which sets virtual viewpoint parameters such as, position of the virtual viewpoint (position), direction of view from the virtual viewpoint (imaging direction) and the angle of view, based on the obtained imaging apparatus parameters. The viewpoint control unit then outputs the virtual viewpoint parameters, such as position, direction of view and angle of view to the image processing apparatus (processor acquires reference imaging device information corresponding to a position, an imaging direction, and an angle of view of a reference imaging device). Uemura is combined with Umemura such that the angle of view is included in the camera parameters of Umemura. Therefore, it would have been obvious to one of ordinary skill in the art at the time of the effective filing date, to modify the method of Umemura by adding the feature of the processor acquires reference imaging device information corresponding to… an angle of view of a reference imaging device, in
Umemura and Uemura are silent regarding acquires image quality information indicating an image quality of a first reference image obtained by imaging the imaging region with the reference imaging device, however Hanamoto teaches
acquires image quality information indicating an image quality of a first reference image obtained by imaging the imaging region with the reference imaging device, (“In the telephoto camera group’s image capturing area 601, it is possible to generate a virtual viewpoint video image using multi-viewpoint video images captured by the telephoto camera group 109. In the telephoto camera group’s image capturing area 601, the degree of resolution of an object is relatively high, and thus, it is possible to maintain the quality of the virtual viewpoint video image even when the virtual camera moves closer to the object (even if the height of the virtual camera is lowered). ”; Hanamoto, col. 5, lines 58-67, Fig. 6)
Fig. 6 illustrates that the multi-viewpoint video images captured by the telephoto camera group are high resolution images (acquires image quality information indicating an image quality of a first reference image obtained by the imaging region with the reference imaging device). Therefore, it would have been obvious to one of ordinary skill in the art at the time of the effective filing date, to modify the method of Umemura by adding the feature of acquires image quality information indicating an image quality of a first reference image obtained by imaging the imaging region with the reference imaging device, in order to maintain the quality of virtual viewpoint image even when the virtual camera moves close to the object, as taught by Hanamoto. (col. 5, lies 62-67)
Umemura, Uemura and Hanamoto are silent regarding determines an image quality of the virtual viewpoint image on the basis of the image quality information on a condition that the instruction is given, generates the virtual view point image by using the reference imaging device information and the image quality of the virtual view point image, however, Aizawa teaches
determines an image quality of the virtual viewpoint image on the basis of the image quality information on a condition that the instruction is given, (“The backend server 104 configured to generate information indicating a relationship between the virtual viewpoint and the image quality of the virtual viewpoint image. The backend server 104 generates various kinds of indicator information about the image quality of the virtual viewpoint image on the basis of the obtained information. The information about the position and posture of the physical camera represents the position and posture of the physical camera that is actually disposed. The information about the angle of view and number of pixels of the physical camera represents the angle of view and the number of pixels that are actually set in the physical camera. The backend server 104 outputs the generated various kinds of indicator information to the virtual-viewpoint-specifying device 105.”; Aizawa, [0028])
The backend server generates information indicating a relationship between the virtual viewpoint and the image quality of the virtual viewpoint image. The backend server generates information about the image quality of the virtual viewpoint image (determines an image quality of the virtual viewpoint image) such as, position and posture of the physical camera (on the basis of image quality information), angle of view and number of pixels of the physical camera (on the bases of image quality information on a condition that the instruction is given).
(“The virtual-viewpoint-specifying device 105 configured to cause a display unit to display information indicating a relationship between at least one of the position and the direction of the virtual viewpoint and an image quality of the virtual viewpoint image together with the virtual viewpoint image”; Aizawa, [0029])
The virtual viewpoint specifying device causes a display to display information indicating a relationship between the position and/or direction of the virtual viewpoint image and the quality of the virtual viewpoint image (determine an image quality of the virtual viewpoint image on the basis of quality information) together with the virtual viewpoint image.
(“A virtual-information-obtaining unit 302 obtains various kinds of information about the virtual camera at the virtual viewpoint from the virtual viewpoint-specifying device 105. Examples of the information about the virtual camera include a position, a posture, an angle of view, and the number of pixels as in the physical camera. Since the virtual camera does not actually exist, the virtual-viewpoint-specifying device 105 generates information about the position, posture, angle of view, and number of pixels of the virtual camera at the virtual viewpoint on the basis of a specification from an operator, and the virtual-information-obtaining unit 302 obtains the generated information.”; Aizawa, [0039])
A virtual information obtaining unit obtains information about the virtual camera at the virtual viewpoint, such as, position, posture, angle of view, and number of pixels in the physical camera and position, posture, angle of view and number pixels of the virtual camera at the virtual viewpoint.
generates the virtual view point image by using the reference imaging device information and the image quality of the virtual view point image,
(“An image generator 303 obtains the photographed images (captured images) that are photographed by the physical cameras and obtains the various kinds of information about the virtual camera at the virtual viewpoint from the virtual-information-obtaining unit 302. The image generator 303 has a image-generation unit function. The image generator 303 configured to generate the virtual viewpoint image based on the images and the virtual viewpoint. ”; Aizawa, [0040])
The image generator generates the virtual viewpoint image (generates the virtual viewpoint image) based on the images captured by the physical camera (by using the reference imaging device information).
(“The image generator 303 generates the virtual viewpoint image that is seen from the viewpoint (virtual viewpoint) of the virtual camera on the basis of the photographed images (captured images) from the physical cameras and the information about the virtual camera... Examples of the information about the virtual camera include a position, a posture, an angle of view, and the number of pixels as in the physical camera.”; Aizawa, [0040], [0039])
The image generator generates the virtual viewpoint image (generates the virtual viewpoint image) on the basis of images captured by the physical camera (by using the reference imaging device information) and information about the virtual camera, such as, a position, a posture, an angle of view and the number of pixels in a physical camera (the image quality of the virtual view point image).
(“Since the virtual viewpoint image is generated on the basis of the images that are photographed by the physical cameras as described above, the generated virtual viewpoint image when the virtual viewpoint image is near a location at which the physical cameras are densely arranged can have an image quality higher than that when the virtual viewpoint image is near a location at which the physical cameras are sparsely arranged.”; Aizawa, [0049])
Since the virtual viewpoint image is generated based on images captured by the physical cameras, the image quality of the virtual viewpoint image is higher (image quality of the virtual viewpoint image) when the virtual viewpoint image is near a location where the physical cameras are densely arranged and the image quality of the virtual viewpoint image is lower (image quality of the virtual viewpoint image) when the virtual viewpoint image is near a location where the physical cameras are sparsely arranged. Therefore, it would have been obvious to one of ordinary skill in the art at the time of the effective filing date, to modify the method of Umemura by adding the feature of determines an image quality of the virtual viewpoint image on the basis of the image quality information on a condition that the instruction is given, generates the virtual view point image by using the reference imaging device information and the image quality of the virtual view point image, in order to allow the operator to know a range in which a virtual viewpoint image that has a higher image quality can be generated, as taught by Aizawa ([0049]).
Umemura and Uemura and Hanamoto are silent regarding outputs a superimposed image in which the virtual viewpoint image is superimposed on a second reference image obtained by imaging the imaging region with the reference imaging device, on a condition that the instruction is given, however, Ogata teaches
and outputs a superimposed image in which the virtual viewpoint image is superimposed on a second reference image obtained by imaging the imaging region with the reference imaging device, on a condition that the instruction is given. (“The information processing apparatus according to the second exemplary embodiment further separates background images other than the users (A, B, and C) from the images photographed by the L viewpoint camera 121 and the R viewpoint camera 122, further performs filling correction of an image region (occlusion region) that is not output by being overshadowed by the users (A, B, and C), and uses the corrected background image and the virtual viewpoint image including the users A to C or the like to generate a combined image to be displayed on the display unit at the second location.”; Ogata, [0416], Figs. 22-23)
Figs. 22-23 illustrate that an operation is performed on the reference image such that the background image is separated from the foreground image, which includes users A, B and C. Filling correction is then performed on the occlusion region of the background image. Then, virtual viewpoint image is combined with the corrected background image.
(“The information processing apparatus at the first location superimposes the virtual viewpoint image including the user image on the background image generated in this manner to generate a final combined image, that is, an image to be displayed on the display unit of the second location... ”; Ogata, [0426])
The virtual viewpoint image is combined with the background by superimposing the virtual viewpoint image over the background for display. Therefore, it would have been obvious to one of ordinary skill in the art at the time of the effective filing date, to modify the method of Umemura by adding the feature of outputs a superimposed image in which the virtual viewpoint image is superimposed on a second reference image obtained by imaging the imaging region with the reference imaging device, on a condition that the instruction is given, in order to reduce the artificiality to give a strange feeling about the viewpoint of the user displayed on the display unit not matching with the actual viewpoint, as taught by Ogata ([0050]).
Claim 14 is a medium analogous to the apparatus of claim 1, is similar in scope and is rejected under the same rationale. Claim 14 has an additional limitation. Re: claim 14, Umemura teaches,
14. (Currently Amended) A non-transitory computer-readable storage medium storing a program executable by a computer that is applied to an information processing apparatus including a processor and a memory connected to or incorporated in the processor to perform a process comprising: (“It is also possible to implement the present invention by processing to supply a program that implements one or more functions of the above-described embodiments to a system or an apparatus via a network or a storage medium and to cause one or more processors in a computer of the system or the apparatus to read and execute the program.”; Umemura, [0075])
A storage medium stores a program, executable by the processor, which is connected to the memory, to implement the functions of the embodiments.
Re: claim 5, Umemura, Uemura, Hanamoto, Aizawa and Ogata teach
5. (Original) The information processing apparatus according to claim 1, wherein the processor gradually changes a ratio between the second reference image and the virtual viewpoint image in the superimposed image. (“... in the case where the switching between the virtual viewpoint video and the captured video is to be performed, a blending is performed for multiple frames in the switching and the switching between the videos is performed stepwise by gradually changing a ratio of blend. ”; Uemura, [0065])
When switching between the virtual viewpoint video and the captured video, blending is performed for the multiple frames, in a stepwise manner to gradually change a ratio of a blend (gradually changes a ratio) between a virtual viewpoint frame and a captured video frame. Umemura, Uemura and Hanamoto are silent regarding the superimposed image, however, Ogata teaches this limitation.
(“The information processing apparatus at the first location superimposes the virtual viewpoint image including the user image on the background image generated in this manner to generate a final combined image, that is, an image to be displayed on the display unit of the second location... ”; Ogata, [0426])
The virtual viewpoint image is combined with the background by superimposing the virtual viewpoint image over the background for display. Uemura and Ogata are combined with Umemura and Hanamoto such that the superimposing of Ogata includes the gradual changing of the ratio of Uemura. Therefore, it would have been obvious to one of ordinary skill in the art at the time of the effective filing date, to modify the method of Umemura by adding the feature of the processor gradually changes a ratio between the second reference image and the virtual viewpoint image in the superimposed image, in order to reduce the differences between the frames in the output video and thus reduce the strangeness felt by the viewer, as taught by Uemura ([0065]) and in order to reduce the artificiality to give a strange feeling about the viewpoint of the user displayed on the display unit not matching with the actual viewpoint, as taught by Ogata ([0050]).
Re: claim 7, Umemura, Uemura, Hanamoto, Aizawa and Ogata teach
7. (Original) The information processing apparatus according to claim 1, wherein the reference imaging device is an imaging device capable of changing at least one of the position, the imaging direction, or the angle of view.
(“… for example, the virtual viewpoint image in a case where the point to which the viewpoint from the image capturing unit 111i of the camera system 110i is moved in the height direction (+z direction) is taken to be the virtual viewpoint will be the image as shown in Fig. 10C.”; Umemura, [0053], Figs. 10A and 10C)
The viewpoint of the image capturing unit 111i (reference imaging device) is moved in the height (+z) direction (changing at least one of the position). Figs. 10A and 10C illustrate, that the viewpoint position of the image capturing unit is changed in the +z direction.
Re: claim 8, Umemura, Uemura, Hanamoto, Aizawa and Ogata teach
8. (Original) The information processing apparatus according to claim 1, wherein the processor acquires the reference imaging device information on the basis of a third reference image obtained by imaging the imaging region with the reference imaging device. (“The imaging apparatus 40 is an imaging apparatus of a broadcast station arranged beside the game field or the like, an imaging apparatus suspended from an upper level of a stadium via wires… or the like and an imaging apparatus operator controls the position, the direction (orientation), and the angle of view of the imaging apparatus. The imaging apparatus 40 outputs the captured video and a time (time code) to the storage apparatus 50…. Various setting values of the imaging apparatus (imaging apparatus parameters) including the position, direction, and angle of view of the imaging apparatus are outputted to the storage apparatus 50 as setting information of the imaging apparatus.”; Uemura, [0019])
Fig. 1 illustrates imaging apparatus 40 (reference imaging device) that outputs the captured video (third reference image) and includes various setting values such as, position, direction and angle of view (reference imaging device information). Imaging apparatus 4 outputs the captured video (imaging the imaging region with the reference imaging device).
(“The storage apparatus 50 outputs the captured video and the imaging apparatus parameters to the video switching apparatus 10 depending on control information outputted from the video switching apparatus 10… The virtual viewpoint control unit 103 obtains the switch time N from the time information received from the time setting unit 101 and obtains the imaging apparatus parameters at the switch time N from the storage apparatus 50.”; Uemura, [0020], [0024])
Fig. 1 illustrates that the virtual viewpoint control unit (processor) receives (acquires) the imaging apparatus parameters, of imaging apparatus 40, that are stored in storage apparatus 50 (processor acquires the reference imaging device information on the basis of the third reference image). Therefore, it would have been obvious to one of ordinary skill in the art at the time of the effective filing date, to modify the method of Umemura by adding the feature of the processor acquires the reference imaging device information on the basis of a third reference image obtained by imaging the imaging region with the reference imaging device, in order to provide a video experience with a greater feeling of presence to the viewer, as taught by Uemura. ([0003])
Re: claim 9 Umemura, Uemura, Hanamoto, Aizawa and Ogata teach
9. (Original) The information processing apparatus according to claim 1, wherein the reference imaging device information is the position, the imaging direction, and the angle of view of the reference imaging device. (“The camera parameters refer to information on the installation position and orientation (line-of-sight direction) of each of the image capturing units 111a to 111j, the focal distance of the lens, and the like... the server 140 generates a three-dimensional model of the structure (here, the soccer goal 202) configured by voxels described previously based on the image data of the structure received from each camera system and the camera parameters of each camera system.”; Umemura, [0040], [0046])
The camera parameters (reference image device information) includes the installation position (position), line-of sight direction (imaging direction and angle of view of a reference imaging device). Umemura is silent regarding an angle of view, however Uemura teaches this limitation.
(“Various setting values of the imaging apparatus (imaging apparatus parameters) including the position, direction, and angle of view of the imaging apparatus are outputted to the storage apparatus 50 as setting information of the imaging apparatus.”; Uemura, [0019])
Fig. 1 illustrates imaging apparatus 40 (reference imaging device) that outputs the captured video (third reference image) and includes various setting values such as, position, direction and angle of view (reference imaging device information). Imaging apparatus 4 outputs the captured video (imaging the imaging region with the reference imaging device). Uemura is combined with Umemura such that the angle of view is included in the camera parameters of Umemura. Therefore, it would have been obvious to one of ordinary skill in the art at the time of the effective filing date, to modify the method of Umemura by adding the feature of the reference imaging device information is… the angle of view of the reference imaging device, in order to provide a video experience with a greater feeling of presence to the viewer, as taught by Uemura. ([0003])
Re: claim 10, Umemura, Uemura, Hanamoto, Aizawa and Ogata teach
10. (Original) The information processing apparatus according to claim 1, wherein the reference imaging device is one of the plurality of imaging devices. (“Fig. 2 is a diagram showing arrangement of a total of ten camera systems 110a to 110j configuring a virtual viewpoint image generation system in a bird’s eye diagram in a case where a field 200 is viewed from directly above… Fig. 7A shows an image obtained by the image capturing unit 111i of the camera system 110i capturing the field 200 in the state without the soccer goal 202.”; Umemura, [0035], [0042], Figs. 2, and 7A)
Fig. 2 illustrates that there are 10 camera systems 110a-110j. Fig 7A illustrates an image captured by the image capturing unit 111i of camera system 110j (reference imaging device is one of the plurality of imaging devices).
Re: claim 11, Umemura, Uemura, Hanamoto, Aizawa and Ogata teach
11. (Original) The information processing apparatus according to claim 10, wherein the reference imaging device is capable of being switched between the plurality of imaging devices. (“The virtual viewpoint video imaging apparatus group 20 is an imaging apparatus group including multiple imaging apparatuses installed to surround a game field or the like… The imaging apparatus 40 is an imaging apparatus of a broadcast station arranged beside the game field or the like… The imaging apparatus outputs a captured video and a time (time code) to the storage apparatus 50… The switching unit 104 obtains the switch time N form the time information received from the time setting unit 101, obtains start video selection information that is set by the user and that specifies one of the captured video and the virtual viewpoint video to be used as a start video, and performs switching between the virtual viewpoint video and the captured video… Fig. 3 illustrates an outline processing preformed in the case where the virtual viewpoint video is set as the start video and the captured video is set as the video after switching”; Uemura, [0017], [0019], [0025], [0034], Figs. 1 and 3)
Fig. 1 illustrates a virtual viewpoint video imaging apparatus group 20 that includes multiple imaging apparatuses installed to surround a game field and an imaging apparatus 40 that is arranged beside the game field. The video switching apparatus 10 includes a switching unit 104 that performs switching between the virtual viewpoint video of the viewpoint video imaging apparatus group 20 and the captured video of the imaging apparatus 20 (reference imaging device is capable of being switched between the plurality of imaging devices). Fig. 3 illustrates switching between the virtual viewpoint video imaging apparatus group 20 and the imaging apparatus 40. Therefore, it would have been obvious to one of ordinary skill in the art at the time of the effective filing date, to modify the method of Umemura by adding the feature of the reference imaging device is capable of being switched between the plurality of imaging devices, in order to provide a video experience with a greater feeling of presence to the viewer, as taught by Uemura. ([0003])
Re: claim 12, Umemura, Uemura, Hanamoto, Aizawa and Ogata teach
12. (Currently Amended) The information processing apparatus according to claim 1, wherein the processor outputs a fourth reference image obtained by imaging the imaging region with the reference imaging device, (“The multiple imaging apparatuses are imaging apparatuses that are different from the imaging apparatus 40 to be described later and that capture multi-viewpoint images used to generate a virtual viewpoint video. The virtual viewpoint video imaging apparatus group 20 outputs images captured by the respective imaging apparatuses to the image processing apparatus 30.”; Uemura, [0017])
The virtual viewpoint imaging apparatus outputs images captured by the respective imaging apparatuses (output reference images that include the fourth reference obtained by imaging the region with the reference imaging device) the image processing apparatus.
acquires the reference imaging device information, and generates a virtual viewpoint image corresponding to the fourth reference image by using the reference imaging device information as a reference, on the condition that the instruction is given. (“The imaging apparatus 40 is an imaging apparatus of a broadcast station arranged beside the game field or the like, an imaging apparatus suspended from an upper level of a stadium via wires… or the like and an imaging apparatus operator controls the position, the direction (orientation), and the angle of view of the imaging apparatus. The imaging apparatus 40 outputs the captured video and a time (time code) to the storage apparatus 50…. Various setting values of the imaging apparatus (imaging apparatus parameters) including the position, direction, and angle of view of the imaging apparatus are outputted to the storage apparatus 50 as setting information of the imaging apparatus.”; Uemura, [0019])
Fig. 1 illustrates imaging apparatus 40 (reference imaging device) that outputs the captured video (fourth reference image) and includes various setting values such as, position, direction and angle of view (reference imaging device information). Imaging apparatus 4 outputs the captured video (imaging the imaging region with the reference imaging device).
(“The storage apparatus 50 outputs the captured video and the imaging apparatus parameters to the video switching apparatus 10 depending on control information outputted from the video switching apparatus 10… The virtual viewpoint control unit 103 obtains the switch time N from the time information received from the time setting unit 101 and obtains the imaging apparatus parameters at the switch time N from the storage apparatus 50.”; Uemura, [0020], [0024])
Fig. 1 illustrates that the virtual viewpoint control unit (processor) receives (acquires) the imaging apparatus parameters, of imaging apparatus 40, that are stored in storage apparatus 50. Therefore, it would have been obvious to one of ordinary skill in the art at the time of the effective filing date, to modify the method of Umemura by adding the feature of the processor outputs a fourth reference image obtained by imaging the imaging region with the reference imaging device, acquires the reference imaging device information, and generates a virtual viewpoint image corresponding to the fourth reference image by using the reference imaging device information as a reference, on the condition that the instruction is given, in order to provide a video experience with a greater feeling of presence to the viewer, as taught by Uemura. ([0003])
Claim(s) 2, 3 and 6 is/are rejected under 35 U.S.C. 103 as being unpatentable over Umemura in view of Uemura, Aizawa, Hanamoto and Ogata as applied to claim 1 above, and further in view of Nakao et al. U.S. Pub. No. 2020/0234495.
Re: claim 2, Umemura, Uemura, Hanamoto, Aizawa and Ogata are silent regarding the reference imaging device information is continuously transmitted by a transmission device, the processor generates the virtual viewpoint image by using, as a reference, the reference imaging device information transmitted from the transmission device, on a condition that the instruction is given, however, Nakao teaches
2. (Currently Amended) The information processing apparatus according to claim 1, wherein in a case where the reference imaging device information is continuously transmitted by a transmission device, the processor generates the virtual viewpoint image by using, as a reference, the reference imaging device information transmitted from the transmission device, on the condition that the instruction is given. (“The image processing system 100 includes… a plurality of imaging apparatuses 2… and generates a virtual viewpoint image… that corresponds to an observation image from an arbitrary viewpoint in a three-dimensional space on the basis of imaging information acquired from the plurality of imaging apparatuses 2… In the present example, each imaging apparatus 2 acquires a captured image according to a moving image… Each imaging apparatus 2 images a manner in which, in this example, a ball game such as basketball or soccer is being carried out…”; Nakao, [0090], [0091], [0092], Fig. 1)
Fig. 1 illustrates an image processing system that includes plural imaging apparatuses 2 that capture images of a ball game, such as basketball or soccer.
(“… the calibration apparatus 5 extracts camera parameters representative of the position and imaging direction (gaze direction) of each imaging apparatus 2 on the basis of the acquired camera internal parameters. The extracted camera parameters are transferred in a state in which they allow identification regarding with which imaging apparatus 2 they are associated to the image processing apparatus 1.”; Nakao, [0098], Fig. 1)
The calibration apparatus extracts camera parameters, such as position and imaging direction, from each imaging apparatus. The extracted camera parameters are transferred (continuously transmitted by a transmission device) to the image processing apparatus, such that the particular imaging device is identified.
(“… the virtual viewpoint image generation section 13… determines an imaging object, which moves, for example, as a player, as the target and changes (sets) the position of a viewpoint of a virtual viewpoint image following the movement of the imaging object… the virtual viewpoint image generation section 13 changes the position of the viewpoint of the virtual viewpoint image on the basis of operation information from the inputting apparatus 6 while keeping such following of the movement of the imaging object as described above…”; Nakao, [0113], [0014], Fig. 1)
Fig. 1 illustrates the virtual viewpoint image generation section that determines a moving object, such as a player, and changes the position of the viewpoint based on the player’s movement.
(“Here, by setting the viewpoint position for a virtual viewpoint image, for example, for each fixed time interval (frame rate) on the basis of three-dimensional information of the imaging object that changes together with the movement of the imaging object and analysis information of the imaging object, the viewpoint position set at a certain point in time differs from the viewpoint position set immediately before then (at a different point of time). In particular, since the viewpoint position changes following the movement of the imaging object, also the virtual viewpoint image generated on the basis of the viewpoint position results in following the movement of the imaging object (resulting in an image that continues to display the imaging object within a picture frame).”; Nakao, [0015], Fig. 1)
The viewpoint position for the virtual viewpoint image is set for each fixed time interval (frame rate) based on 3D information of the imaging object (which includes extracted camera parameters from the calibration apparatus 5) and the movement of the imaging object, such as a player. The viewpoint position changes as it follows the movement of the player. This results in a virtual viewpoint images that follow the movement of the player (the processor generates the virtual viewpoint image by using, as a reference, the reference imaging device information transmitted from the transmission device). Therefore, it would have been obvious to one of ordinary skill in the art at the time of the effective filing date, to modify the method of Umemura by adding the feature of in a case where the reference imaging device information is continuously transmitted by a transmission device, the processor generates the virtual viewpoint image by using, as a reference, the reference imaging device information transmitted from the transmission device, on a condition that the instruction is given, in order to follow the movement of the imaging object resulting in an object that continues to display within a picture frame, as taught by Nakao. ([0015])
Re: claim 3, Umemura, Uemura, Hanamoto, Aizawa and Ogata are silent regarding the processor updates the reference imaging device information each time the processor acquires the reference imaging device information, however, Nakao teaches
3. (Original) The information processing apparatus according to claim 1, wherein the processor updates the reference imaging device information each time the processor acquires the reference imaging device information. (“… the calibration apparatus 5 extracts camera parameters representative of the position and imaging direction (gaze direction) of each imaging apparatus 2 on the basis of the acquired camera internal parameters. The extracted camera parameters are transferred in a state in which they allow identification regarding with which imaging apparatus 2 they are associated to the image processing apparatus 1.”; Nakao, [0098], Fig. 1)
The calibration apparatus extracts camera parameters, such as position and imaging direction, from each imaging apparatus. The extracted camera parameters are transferred to the image processing apparatus, such that the particular imaging device is identified.
(“Here, by setting the viewpoint position for a virtual viewpoint image, for example, for each fixed time interval (frame rate) on the basis of three-dimensional information of the imaging object that changes together with the movement of the imaging object and analysis information of the imaging object, the viewpoint position set at a certain point in time differs from the viewpoint position set immediately before then (at a different point of time). In particular, since the viewpoint position changes following the movement of the imaging object, also the virtual viewpoint image generated on the basis of the viewpoint position results in following the movement of the imaging object (resulting in an image that continues to display the imaging object within a picture frame).”; Nakao, [0015], Fig. 1)
The viewpoint position for the virtual viewpoint image is determined (updates the reference image device information) for each fixed time interval (frame rate), based on the 3D information from the 3D model generation section 11 (which includes the extracted camera parameters from the calibration apparatus 5) and the movement of the imaging object, such as a player (updates the reference imaging device information each time the processor acquires the reference imaging device information). Therefore, it would have been obvious to one of ordinary skill in the art at the time of the effective filing date, to modify the method of Umemura by adding the feature of the processor updates the reference imaging device information each time the processor acquires the reference imaging device information, in order to follow the movement of the imaging object resulting in an object that continues to display within a picture frame, as taught by Nakao. ([0015])
Re: claim 6, Umemura, Uemura, Hanamoto, Aizawa and Ogata are silent regarding, the processor outputs the virtual viewpoint image to a display, and receives a change signal for continuously changing at least one of a viewpoint position, a line-of-sight direction, or an angle of view in the output virtual viewpoint image, however, Nakao teaches
6. (Original) The information processing apparatus according to claim 1, wherein the processor outputs the virtual viewpoint image to a display, and receives a change signal for continuously changing at least one of a viewpoint position, a line-of-sight direction, or an angle of view in the output virtual viewpoint image. (“… the virtual viewpoint image generation section 13… determines an imaging object, which moves, for example, as a player, as the target and changes (sets) the position of a viewpoint of a virtual viewpoint image following the movement of the imaging object… the virtual viewpoint image generation section 13 changes the position of the viewpoint of the virtual viewpoint image on the basis of operation information from the inputting apparatus 6 while keeping such following of the movement of the imaging object as described above…”; Nakao, [0113], [0014], [0015], Fig. 1)
Fig. 1 illustrates the virtual viewpoint image generation section that determines a moving object, such as a player, and changes the position of the viewpoint based on the player’s movement.
(“Here, by setting the viewpoint position for a virtual viewpoint image, for example, for each fixed time interval (frame rate) on the basis of three-dimensional information of the imaging object that changes together with the movement of the imaging object and analysis information of the imaging object, the viewpoint position set at a certain point in time differs from the viewpoint position set immediately before then (at a different point of time). In particular, since the viewpoint position changes following the movement of the imaging object, also the virtual viewpoint image generated on the basis of the viewpoint position results in following the movement of the imaging object (resulting in an image that continues to display the imaging object within a picture frame).”; Nakao, [0015], Fig. 1)
The viewpoint position for the virtual viewpoint image is set for each fixed time interval (frame rate) based on the movement of the imaging object, such as a player (receives a change signal for continuously changing… a viewpoint position in the output virtual viewpoint image). Thus, the viewpoint position changes as it follows the movement of the player and results in a virtual viewpoint images that follow the movement of the player (processor outputs the virtual viewpoint image to a display). Therefore, it would have been obvious to one of ordinary skill in the art at the time of the effective filing date, to modify the method of Umemura by adding the feature of the processor outputs the virtual viewpoint image to a display, and receives a change signal for continuously changing at least one of a viewpoint position, a line-of-sight direction, or an angle of view in the output virtual viewpoint image, in order to follow the movement of the imaging object resulting in an object that continues to display within a picture frame, as taught by Nakao. ([0015])
Response to Arguments
Applicant’s arguments with respect to claim(s) 1, 13 and 14 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument. Applicant argues:
“The amended claim states "determines an image quality of the virtual viewpoint image..." and also states "generates the virtual view point image by using the reference imaging device information and the image quality of the virtual view point image". However neither Umemura nor Hanamoto nor Ogata disclose as such. Hanamoto merely teaches that the use of high- quality images can "maintain" quality but does not disclose the limitation determines an image quality of the virtual viewpoint image. In plain words, Hanamoto does not disclose a processor actively acquires "image quality information" and, based on this information, "determines" the quality of the output image (i.e. on the basis of the image quality information on a condition that the instruction is given). The ground of rejection as stated in the Office Action is predicated upon an inference regarding inherent properties, rather than upon an explicit teaching. Further, the Umemura reference and the Ogata reference both do not contain relevant disclosure in this matter. Overall, the prior art references at best describes that if the resolution of the image of a camera group is high, the resolution can be maintained. The prior art does not teach determining the resolution of both the reference image and the virtual view point image so that the virtual view point image can be generated based on the image quality information. Instead, the prior art relies on the inherency of the disclosure from the cited reference. Therefore, in view of the amendments to claims 1, 13, and 14 and the reasons provided, claims 1, 13, and 14 overcome the current ground of rejection. Claims 2-3 and 5-12 also overcome the current ground of rejection due to claim dependencies on claim 1.”
Applicant’s arguments are persuasive. A new grounds of rejection is necessitated by the amendment. Aizawa teaches this amended limitation. Aizawa teaches that the backend server generates information indicating a relationship between the virtual viewpoint and the image quality of the virtual viewpoint image. The backend server generates information about the image quality of the virtual viewpoint image (determines an image quality of the virtual viewpoint image) such as, position and posture of the physical camera (on the basis of image quality information), angle of view and number of pixels of the physical camera (on the bases of image quality information on a condition that the instruction is given). (Aizawa, [0028]). The virtual viewpoint specifying device causes a display to display information indicating a relationship between the position and/or direction of the virtual viewpoint image and the quality of the virtual viewpoint image (determine an image quality of the virtual viewpoint image on the basis of quality information) together with the virtual viewpoint image. (Aizawa, [0029]). A virtual information obtaining unit obtains information about the virtual camera at the virtual viewpoint, such as, position, posture, angle of view, and number of pixels in the physical camera and position, posture, angle of view and number pixels of the virtual camera at the virtual viewpoint. (Aizawa, [0039]). The image generator generates the virtual viewpoint image (generates the virtual viewpoint image) based on the images captured by the physical camera (by using the reference imaging device information). (Aizawa, [0040]). The image generator generates the virtual viewpoint image (generates the virtual viewpoint image) on the basis of images captured by the physical camera (by using the reference imaging device information) and information about the virtual camera, such as, a position, a posture, an angle of view and the number of pixels in a physical camera (the image quality of the virtual view point image). (Aizawa, [0040], [0039]). Since the virtual viewpoint image is generated based on images captured by the physical cameras, the image quality of the virtual viewpoint image is higher (image quality of the virtual viewpoint image) when the virtual viewpoint image is near a location where the physical cameras are densely arranged and the image quality of the virtual viewpoint image is lower (image quality of the virtual viewpoint image) when the virtual viewpoint image is near a location where the physical cameras are sparsely arranged. (Aizawa, [0049]).
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DONNA J RICKS whose telephone number is (571)270-7532. The examiner can normally be reached on M-F 7:30am-5pm EST (alternate Fridays off).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Devona Faulk can be reached on 571-272-7776. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see https://ppair-my.uspto.gov/pair/PrivatePair. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Donna J. Ricks/Examiner, Art Unit 2618
/DEVONA E FAULK/Supervisory Patent Examiner, Art Unit 2618
Prosecution Insights