DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Yoshikawa in view of Sheng
Claims 1-7, 9, 10, and 17-20 are rejected under 35 U.S.C. 103 as being unpatentable over Yoshikawa et al.(USPubN 2019/0174109; hereinafter Yoshikawa) in view of Sheng(USPubN 2020/0288112).
As per claim 1, Yoshikawa teaches an information processing apparatus that generates a virtual viewpoint video based on a virtual viewpoint by using a motion picture obtained by imaging an imaging region with a plurality of imaging devices, (“The free-viewpoint video here refers to a video of a target scene viewed from a virtual camera moving between any spatial positions. To generate a free-viewpoint video, it is necessary to determine a camerawork that represents temporal changes in position and orientation of the virtual camera” in Para.[0024]) the information processing apparatus comprising:
a processor; and a memory built in or connected to the processor, wherein the processor displays, on a display(“Each component may be achieved by a program execution unit such as a central processing unit (CPU) or a processor that reads and executes a software program stored in a storage medium such as a hard disk or a semiconductor memory, instead” in Para.[0116], “each apparatus included in the video distribution system or the video display device includes processing circuitry and a storage (can be accessed from the processing circuitry) electrically connected to the processing circuitry.” in Para.[0117]),
a standard image corresponding to the imaging region, a plurality of virtual viewpoint paths disposed in the standard image and representing a trajectory of movement of the virtual viewpoint(“camerawork generation unit 123 disposes a virtual camera in a three-dimensional space automatically and generates camerawork 154 used to generate a free-viewpoint video (S104)” in Para.[0053], “video processing device 102 acquires multi-view video 151 (S111). Next, three-dimensional model generation unit 121 uses multi-view video 151 to generate three-dimensional model 152 of a target scene that is a capturing space (S112).” in Para.[0057], “In camerawork display field 201, three-dimensional model 152 and a camera path that is a path of camerawork 154 are displayed. There may be one camera path displayed or a plurality of candidates for a camera path displayed.“ in Para.[0073], The image presented in the camerawork display field can be interpreted as the standard image. Camera path can be interpreted as viewpoint paths.),
an indicator indicating a reproduction position of the virtual viewpoint video, and a reference image based on a virtual viewpoint image viewed from the virtual viewpoint corresponding to the reproduction position of the virtual viewpoint path among a plurality of virtual viewpoint images configuring the virtual viewpoint video(“Slide bar 204 is used to operate a time point in camerawork check video 155. when an editor operates slide bar 204, a virtual camera moves on a camera path, and camerawork check video 155 corresponding to an operated time point is displayed in camerawork check video display field 203” in Para.[0076], Slide bar can be interpreted as an indicator and the image represented in camerawork check video display field can be interpreted as the reference image.),
wherein a plurality of reference images at the same reproduction position is displayed along the virtual viewpoint path in the standard image(“a virtual camera moves on a camera path, and camerawork check video 155 corresponding to an operated time point is displayed in camerawork check video display field 203. Alternatively, a playback may be started automatically as a camera path is selected, and slide bar 204 may be moved” in Para.[0076], The image in camerawork check video display field is display along a camera path in the camerawork display field. A camera path is selected a plurality of images automatically are displayed.).
Yoshikawa is silent about wherein a plurality of reference images are displayed simultaneously along the plurality of virtual viewpoint paths in the standard image.
Sheng teaches wherein a plurality of reference images are displayed simultaneously along the plurality of virtual viewpoint paths in the standard image(“the user may slide the screen to switch the virtual viewpoint. In an example embodiment of the present disclosure, referring to FIG. 5, when the user slides the screen with his/her finger to the right, the virtual viewpoint for viewing may be switched. Still referring to FIG. 2, the position of the virtual viewpoint before sliding may be VP.sub.1. The position of the virtual viewpoint may be VP.sub.2 after the virtual viewpoint is switched by sliding the screen. Referring to FIG. 6, after sliding the screen, the reconstructed image displayed on the screen may be as shown in FIG. 6. The reconstructed image may be obtained by performing image reconstruction based on multi-angle free-perspective data generated from data captured by multiple capturing devices in an actual capturing scenario” in Para.[0122], “Referring to FIG. 27, equally splitting the stitched image up and down is still taken as an example for further description. The upper half of the stitched image in FIG. 27 is the image area, which is split into eight image sub-areas, which store the pixel data of the synchronized eight images respectively. Each image has a different shooting angle, i.e., a different perspective. The lower half of the stitched image is the depth map area, which is split into 8 depth map sub-areas, which store the depth maps of the 8 images respectively” in Para.[0206], Fig. 27).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings Yoshikawa with the above teachings of Sheng in order to improve user experience for viewing different virtual path images within same display.
As per claim 2, Yoshikawa and Sheng teach all of limitation of claim 1.
Yoshikawa teaches wherein the processor disposes at least one of the plurality of virtual viewpoint paths at an indicated position in the standard image(“In camerawork display field 201, three-dimensional model 152 and a camera path that is a path of camerawork 154 are displayed. There may be one camera path displayed or a plurality of candidates for a camera path displayed. In addition, colors and line types may be applied to the plurality of camera paths to represent information such as recommendation levels of the respective camera paths. Here, the recommendation levels each indicate a degree of match between the recommendation level and a preference of a user or a degree of viewing frequency.” in Para.[0073], “Slide bar 204 is used to operate a time point in camerawork check video 155. For example, when an editor operates slide bar 204, a virtual camera moves on a camera path, and camerawork check video 155 corresponding to an operated time point is displayed in camerawork check video display field 203” in Para.[0076]).
As per claim 3, Yoshikawa and Sheng teach all of limitation of claim 1.
Yoshikawa teaches wherein the processor changes the reproduction position indicated by the indicator in response to a given instruction(“Slide bar 204 is used to operate a time point in camerawork check video 155. For example, when an editor operates slide bar 204, a virtual camera moves on a camera path, and camerawork check video 155 corresponding to an operated time point is displayed in camerawork check video display field 203” in Para.[0076]).
As per claim 4, Yoshikawa and Sheng teach all of limitation of claim 1.
Yoshikawa teaches wherein, in a case where the reproduction position is changed, the processor displays the reference image corresponding to the reproduction position after being changed on the display(“Slide bar 204 is used to operate a time point in camerawork check video 155. For example, when an editor operates slide bar 204, a virtual camera moves on a camera path, and camerawork check video 155 corresponding to an operated time point is displayed in camerawork check video display field 203” in Para.[0076]).
As per claim 5, Yoshikawa and Sheng teach all of limitation of claim 4.
Yoshikawa teaches wherein, in a case where the reproduction position is changed, the processor displays the reference image corresponding to the reproduction position after being changed at a position different from a position of the reference image corresponding to the reproduction position before being changed(“display 125 displays an interface screen for checking three-dimensional model 152, camerawork 154, and camerawork check video 155 (S116). FIG. 5 is a diagram illustrating an example of the interface screen. The interface screen includes camerawork display field 201, camera information display field 202, camerawork check video display field 203, slide bar 204, evaluation display field 205, and upload button 206” in Para.[0072], “In camerawork display field 201, three-dimensional model 152 and a camera path that is a path of camerawork 154 are displayed. There may be one camera path displayed or a plurality of candidates for a camera path displayed. In addition, colors and line types may be applied to the plurality of camera paths to represent information such as recommendation levels of the respective camera paths. Here, the recommendation levels each indicate a degree of match between the recommendation level and a preference of a user or a degree of viewing frequency. Displaying the plurality of camera paths can provide choices to a user (viewer) or an editor” in Para.[0073], “In camerawork check video display field 203, camerawork check video 155 corresponding to a camerawork is displayed for checking” in Para.[0075], “Slide bar 204 is used to operate a time point in camerawork check video 155. For example, when an editor operates slide bar 204, a virtual camera moves on a camera path, and camerawork check video 155 corresponding to an operated time point is displayed in camerawork check video display field 203. Alternatively, a playback may be started automatically as a camera path is selected, and slide bar 204 may be moved. In addition, operating buttons or an operation menu to play back a camerawork may be provided” in Para.[0076]).
As per claim 6, Yoshikawa and Sheng teach all of limitation of claim 1.
Yoshikawa teaches wherein the processor displays only one reference image for one virtual viewpoint path(“In camerawork check video display field 203, camerawork check video 155 corresponding to a camerawork is displayed for checking” in Para.[0075], “Slide bar 204 is used to operate a time point in camerawork check video 155. For example, when an editor operates slide bar 204, a virtual camera moves on a camera path, and camerawork check video 155 corresponding to an operated time point is displayed in camerawork check video display field 203. Alternatively, a playback may be started automatically as a camera path is selected, and slide bar 204 may be moved. In addition, operating buttons or an operation menu to play back a camerawork may be provided” in Para.[0076]).
As per claim 7, Yoshikawa and Sheng teach all of limitation of claim 1.
Yoshikawa teaches wherein the processor changes the reproduction position in response to an operation of moving the reference image on the standard image(“In camerawork check video display field 203, camerawork check video 155 corresponding to a camerawork is displayed for checking” in Para.[0075], “Slide bar 204 is used to operate a time point in camerawork check video 155. For example, when an editor operates slide bar 204, a virtual camera moves on a camera path, and camerawork check video 155 corresponding to an operated time point is displayed in camerawork check video display field 203. Alternatively, a playback may be started automatically as a camera path is selected, and slide bar 204 may be moved. In addition, operating buttons or an operation menu to play back a camerawork may be provided” in Para.[0076]).
As per claim 9, Yoshikawa and Sheng teach all of limitation of claim 1.
Yoshikawa teaches wherein the processor changes at least one of a position, a length, or a shape of the virtual viewpoint path in response to a given instruction(“In camerawork display field 201, three-dimensional model 152 and a camera path that is a path of camerawork 154 are displayed. There may be one camera path displayed or a plurality of candidates for a camera path displayed. In addition, colors and line types may be applied to the plurality of camera paths to represent information such as recommendation levels of the respective camera paths. Here, the recommendation levels each indicate a degree of match between the recommendation level and a preference of a user or a degree of viewing frequency. Displaying the plurality of camera paths can provide choices to a user (viewer) or an editor” in Para.[0073], “the camera paths for one target object, the camera paths may be displayed for each of a plurality of target objects. The plurality of camera paths may be selected in order, and corresponding camerawork check videos 155 may be played back consecutively in order of the selection” in Para.[0074])
As per claim 10, Yoshikawa and Sheng teach all of limitation of claim 1.
Yoshikawa teaches wherein the processor displays at least one of the indicator or the virtual viewpoint path in different aspects before and after the reproduction position(“In camerawork display field 201, three-dimensional model 152 and a camera path that is a path of camerawork 154 are displayed. There may be one camera path displayed or a plurality of candidates for a camera path displayed. In addition, colors and line types may be applied to the plurality of camera paths to represent information such as recommendation levels of the respective camera paths. Here, the recommendation levels each indicate a degree of match between the recommendation level and a preference of a user or a degree of viewing frequency. Displaying the plurality of camera paths can provide choices to a user (viewer) or an editor” in Para.[0073]).
As per claim 17, Yoshikawa and Sheng teach all of limitation of claim 1.
Yoshikawa teaches wherein the processor displays the reference image corresponding to the virtual viewpoint along the virtual viewpoint path for the plurality of virtual viewpoint paths(Para.[0072], [0073], [0075], [0076]).
As per claim 18, Yoshikawa and Sheng teach all of limitation of claim 1.
Yoshikawa teaches wherein the processor performs a selection combining process of cutting out parts of the plurality of virtual viewpoint paths and combining the cut-out parts in response to a given instruction, and generates the virtual viewpoint video on the basis of a path obtained by combining the cut-out parts in the selection combining process(“The position and orientation of the camera and the camera parameters can be edited by an editor. This enables the editor to modify camerawork 154 generated by camerawork generation unit 123 according to a preference of the editor. In place of the camera parameters, an interface on which to select a type of a camera may be provided. When an editor selects a type of a camera, camera parameters and the like of the selected camera are set automatically.” in Para.[0077], “Video processing device 102 may have a function of giving a feedback on results of the selection, details of the modification, and proposition that are made by an editor.” in Para.[0078], “For example, in a live performance, a plurality of cameraworks 154 (or camerawork check videos 155) is displayed to an editor. The editor selects some of a plurality of cameraworks 154, and evaluation values of some cameraworks 154 are then calculated according to a result of the selection or the like. These evaluation values are then used to create a non-realtime replay video such as a broadcasting material.” in Para.[0080], “In such a manner, by a selection or an edition made by an editor, camerawork 154 used for free-viewpoint video 156 to be distributed is determined (S117). For example, when upload button 206 is operated by an editor, edited camerawork 154 is transmitted to rendering unit 124. Rendering unit 124 generates free-viewpoint video 156 corresponding to this camerawork 154 (S118).” In Para.[0082]).
As per claim 19, the limitations in the claim 19 has been discussed in the rejection claim 1 and rejected under the same rationale.
As per claim 20, Yoshikawa teaches a non-transitory computer-readable storage medium storing a program executable by a computer(“the software program or a non-transitory computer-readable storage medium storing the software program.” in Para.[0118]) and the other limitations in the claim 20 has been discussed in the rejection claim 1 and rejected under the same rationale.
Yoshikawa in view of Sheng and Ito
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Yoshikawa et al.(USPubN 2019/0174109; hereinafter Yoshikawa) in view of Sheng(USPubN 2020/0288112) further in view of Ito et al.(USPubN 2016/0105660; hereinafter Ito).
As per claim 8, Yoshikawa and Sheng teach all of limitation of claim 1.
Yoshikawa and Sheng are silent about wherein the processor displays two reference images for two adjacent virtual viewpoint paths according to relative positions of the two virtual viewpoint paths.
Ito teaches wherein the processor displays two reference images for two adjacent virtual viewpoint paths according to relative positions of the two virtual viewpoint paths (“The detecting unit 21 generates the parallax images of two hierarchical levels of the reference viewpoint by setting a parallax image formed with the disparity value of the reference viewpoint as the parallax image (viewpoint parallax image) of a first hierarchical level, and setting a parallax image formed with the disparity value of the occlusion region of the reference viewpoint as the parallax image (occlusion parallax image) of a second hierarchical level. The detecting unit 21 then supplies the parallax images of the two hierarchical levels of the reference viewpoint to the virtual-viewpoint image generating unit 22” in Para.[0046], “The virtual-viewpoint image generating unit 22 stores the parallax images of the two hierarchical levels of the reference viewpoint supplied from the detecting unit 21. Using the stored parallax images of the two hierarchical levels of the reference viewpoint, the virtual-viewpoint image generating unit 22 generates parallax images of viewpoints other than the reference viewpoint” in Para.[0047]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings Yoshikawa and Sheng with the above teachings of Ito in order to improve user experience for parallax images.
Yoshikawa in view of Sheng and Arai
Claims 11 and 12 are rejected under 35 U.S.C. 103 as being unpatentable over Yoshikawa et al.(USPubN 2019/0174109; hereinafter Yoshikawa) in view of Sheng(USPubN 2020/0288112) further in view of Arai(USPubN 2020/0082603).
As per claim 11, Yoshikawa and Sheng teach all of limitation of claim 1.
Yoshikawa and Sheng are silent about wherein the processor generates gaze point information representing a gaze point at which the virtual viewpoint included in the virtual viewpoint path gazes in response to a given instruction.
Arai teaches wherein the processor generates gaze point information representing a gaze point at which the virtual viewpoint included in the virtual viewpoint path gazes in response to a given instruction (“The information processing apparatus 100 selects one gaze point from one or a plurality of gaze points indicated by the gaze point data acquired from the image generation apparatus 200. Further, the information processing apparatus 100 acquires user terminal information from the user terminal 300. Then, the information processing apparatus 100 determines the movement path of the virtual viewpoint relating to the virtual viewpoint image to be generated and the line-of-sight direction of the virtual viewpoint moving on the movement path based on the selected gaze point and the user terminal information acquired from the user terminal 300. In the user terminal information, for example, terminal operation information indicating the operation contents, position information indicating the terminal position, and the like are included. Further, in the terminal operation information, for example, information indicating operations, such as specification of a virtual viewpoint and playback of a virtual viewpoint image, and the like are included” in Para.[0030]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings Yoshikawa and Sheng with the above teachings of Arai in order to improve user experience.
As per claim 12, Yoshikawa, Sheng and Arai teach all of limitation of claim 11.
Yoshikawa and Sheng are silent about wherein the processor generates gaze point information representing a gaze point at which the virtual viewpoint included in the virtual viewpoint path gazes in response to a given instruction.
Arai teaches wherein the processor generates gaze point information representing a gaze point at which the virtual viewpoint included in the virtual viewpoint path gazes in response to a given instruction (“The information processing apparatus 100 selects one gaze point from one or a plurality of gaze points indicated by the gaze point data acquired from the image generation apparatus 200. Further, the information processing apparatus 100 acquires user terminal information from the user terminal 300. Then, the information processing apparatus 100 determines the movement path of the virtual viewpoint relating to the virtual viewpoint image to be generated and the line-of-sight direction of the virtual viewpoint moving on the movement path based on the selected gaze point and the user terminal information acquired from the user terminal 300. In the user terminal information, for example, terminal operation information indicating the operation contents, position information indicating the terminal position, and the like are included. Further, in the terminal operation information, for example, information indicating operations, such as specification of a virtual viewpoint and playback of a virtual viewpoint image, and the like are included” in Para.[0030]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings Yoshikawa and Sheng with the above teachings of Arai in order to improve user experience.
Yoshikawa in view of Sheng and Yoshimura
Claims 15 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Yoshikawa et al.(USPubN 2019/0174109; hereinafter Yoshikawa) in view of Sheng(USPubN 2020/0288112) further in view of Yoshimura et al.(USPubN 2018/0184072; Yoshimura).
As per claim 15, Yoshikawa and Sheng teach all of limitation of claim 1.
Yoshikawa and Sheng are silent about wherein the processor displays the plurality of virtual viewpoint paths and the reference image in the single standard image.
Yoshimura teaches wherein the processor displays the plurality of virtual viewpoint paths and the reference image in the single standard image (“The limit focal length derivation unit 306 drives a range in which a zoom is possible (limit focal length) at each position on the path of the set virtual viewpoint. This limit focal length is derived as follows. First, based on the position coordinates of the virtual viewpoint, the position coordinates of the object, the position coordinates of the virtual gaze point, and the image capturing parameters, a reference image used for generation of an image viewed from a position on the path of the virtual viewpoint, on which attention is focused (hereinafter, virtual viewpoint position of interest), is determined. Specifically, by expression (6) below, an optical axis vector C at the virtual viewpoint position is calculated, an inner product E of the calculated optical axis vector C and an optical axis vector Ln is found for all the cameras by expression (7) below, and the captured image of the camera n whose inner product E is the largest is selected as a reference image.” in Para.[0043], “In the present embodiment, based on the set virtual viewpoint, by substituting the distance d′ from each position on the path of the virtual viewpoint to the virtual gaze point in expression (8) described above as a fixed value, the limit focal length f′ is found. However, it may also be possible to find the limit distance d′ of the virtual viewpoint that can be approached by determining the focal length of the virtual viewpoint in advance. Alternatively, it may also be possible to find the value of f′/d′ in order to obtain an appropriate combination of the focal length and the distance from the virtual viewpoint position to the virtual gaze point. Further, in the present embodiment, a single reference image is selected, but it may also be possible to select a plurality of reference images. For example, the configuration may be one in which a weight of each captured image is found based on the value of the inner product E and captured images of a plurality of cameras are selected as reference images, and then, image combination is performed by using a plurality of reference images selected by the virtual viewpoint video image generation unit 309, to be described later. As the limit focal length in this case, it may be possible to use the smallest value of a plurality of limit focal lengths found based on each selected captured image, a value obtained by performing weighted addition of the plurality of limit focal lengths, and so on” in Para.[0047]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings Yoshikawa and Sheng with the above teachings of Yoshimura in order to improve user experience.
As per claim 16, Yoshikawa and Sheng teach all of limitation of claim 1.
Yoshikawa and Sheng are silent about wherein the processor displays the virtual viewpoint path and the reference image in a plurality of the standard images representing at least a part of the imaging region.
Yoshimura teaches wherein the processor displays the virtual viewpoint path and the reference image in a plurality of the standard images representing at least a part of the imaging region (“The limit focal length derivation unit 306 drives a range in which a zoom is possible (limit focal length) at each position on the path of the set virtual viewpoint. This limit focal length is derived as follows. First, based on the position coordinates of the virtual viewpoint, the position coordinates of the object, the position coordinates of the virtual gaze point, and the image capturing parameters, a reference image used for generation of an image viewed from a position on the path of the virtual viewpoint, on which attention is focused (hereinafter, virtual viewpoint position of interest), is determined. Specifically, by expression (6) below, an optical axis vector C at the virtual viewpoint position is calculated, an inner product E of the calculated optical axis vector C and an optical axis vector Ln is found for all the cameras by expression (7) below, and the captured image of the camera n whose inner product E is the largest is selected as a reference image.” in Para.[0043], “In the present embodiment, based on the set virtual viewpoint, by substituting the distance d′ from each position on the path of the virtual viewpoint to the virtual gaze point in expression (8) described above as a fixed value, the limit focal length f′ is found. However, it may also be possible to find the limit distance d′ of the virtual viewpoint that can be approached by determining the focal length of the virtual viewpoint in advance. Alternatively, it may also be possible to find the value of f′/d′ in order to obtain an appropriate combination of the focal length and the distance from the virtual viewpoint position to the virtual gaze point. Further, in the present embodiment, a single reference image is selected, but it may also be possible to select a plurality of reference images. For example, the configuration may be one in which a weight of each captured image is found based on the value of the inner product E and captured images of a plurality of cameras are selected as reference images, and then, image combination is performed by using a plurality of reference images selected by the virtual viewpoint video image generation unit 309, to be described later. As the limit focal length in this case, it may be possible to use the smallest value of a plurality of limit focal lengths found based on each selected captured image, a value obtained by performing weighted addition of the plurality of limit focal lengths, and so on” in Para.[0047]).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify the teachings Yoshikawa and Sheng with the above teachings of Yoshimura in order to improve user experience.
Double Patenting
The nonstatutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A nonstatutory double patenting rejection is appropriate where the claims at issue are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); and In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on a nonstatutory double patenting ground provided the reference application or patent either is shown to be commonly owned with this application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The USPTO internet Web site contains terminal disclaimer forms which may be used. Please visit http://www.uspto.gov/forms/. The filing date of the application will determine what form should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to http://www.uspto.gov/patents/process/file/efs/guidance/eTD-info-I.jsp.
2. Claims 1-20 are rejected on the ground of nonstatutory obviousness-type double patenting as being unpatentable over claims 1-18 of U.S. Patent No. 12,549,697. Although the conflicting claims at issue are not identical, they are not patentably distinct from each other. See the reasons sets forth below:
Instance Application No. 19/177623
U.S. Patent No. 12,549,697
1. An information processing apparatus that generates a virtual viewpoint video based on a virtual viewpoint by using a motion picture obtained by imaging an imaging region with a plurality of imaging devices, the information processing apparatus comprising: a processor; and a memory built in or connected to the processor, wherein the processor displays, on a display, a standard image corresponding to the imaging region, a plurality of virtual viewpoint paths disposed in the standard image and representing a trajectory of movement of the virtual viewpoint, an indicator indicating a reproduction position of the virtual viewpoint video, and a reference image based on a virtual viewpoint image viewed from the virtual viewpoint corresponding to the reproduction position of the virtual viewpoint path among a plurality of virtual viewpoint images configuring the virtual viewpoint video, wherein a plurality of reference images at the same reproduction position are displayed simultaneously along the plurality of virtual viewpoint paths in the standard image.
2. The information processing apparatus according to claim 1, wherein the processor disposes at least one of the plurality of virtual viewpoint paths at an indicated position in the standard image.
3. The information processing apparatus according to claim 1, wherein the processor changes the reproduction position indicated by the indicator in response to a given instruction.
4. The information processing apparatus according to claim 1, wherein, in a case where the reproduction position is changed, the processor displays the reference image corresponding to the reproduction position after being changed on the display.
5. The information processing apparatus according to claim 4, wherein, in a case where the reproduction position is changed, the processor displays the reference image corresponding to the reproduction position after being changed at a position different from a position of the reference image corresponding to the reproduction position before being changed.
6. The information processing apparatus according to claim 1, wherein the processor displays only one reference image for one virtual viewpoint path.
7. The information processing apparatus according to claim 1, wherein the processor changes the reproduction position in response to an operation of moving the reference image on the standard image.
8. The information processing apparatus according to claim 1, wherein the processor displays two reference images for two adjacent virtual viewpoint paths according to relative positions of the two virtual viewpoint paths.
9. The information processing apparatus according to claim 1, wherein the processor changes at least one of a position, a length, or a shape of the virtual viewpoint path in response to a given instruction.
10. The information processing apparatus according to claim 1, wherein the processor displays at least one of the indicator or the virtual viewpoint path in different aspects before and after the reproduction position.
11. The information processing apparatus according to claim 1, wherein the processor generates gaze point information representing a gaze point at which the virtual viewpoint included in the virtual viewpoint path gazes in response to a given instruction.
12. The information processing apparatus according to claim 11, wherein the gaze point information is a gaze point path representing a trajectory of movement of the gaze point, and the processor disposes a plurality of the gaze point paths in the standard image in response to a given instruction.
13. The information processing apparatus according to claim 11, wherein the processor selects one reference image from among a plurality of the reference images corresponding to the same reproduction position in response to a given instruction, sets the gaze point corresponding to the one selected reference image as a standard gaze point, and generates, as other reference images that are not selected, an image based on a virtual viewpoint image showing an aspect of a case of gazing at the standard gaze point from the virtual viewpoint path corresponding to the non-selected other reference images.
14. The information processing apparatus according to claim 13, wherein the processor changes a display aspect of a section including the virtual viewpoint in which a virtual viewpoint image obtained by gazing at the standard gaze point is not generatable in the virtual viewpoint path.
15. The information processing apparatus according to claim 1, wherein the processor displays the plurality of virtual viewpoint paths and the reference image in the single standard image.
16. The information processing apparatus according to claim 1, wherein the processor displays the virtual viewpoint path and the reference image in a plurality of the standard images representing at least a part of the imaging region.
17. The information processing apparatus according to claim 1, wherein the processor displays the reference image corresponding to the virtual viewpoint along the virtual viewpoint path for the plurality of virtual viewpoint paths.
18. The information processing apparatus according to claim 1, wherein the processor performs a selection combining process of cutting out parts of the plurality of virtual viewpoint paths and combining the cut-out parts in response to a given instruction, and generates the virtual viewpoint video on the basis of a path obtained by combining the cut-out parts in the selection combining process.
19. An information processing method of generating a virtual viewpoint video based on a virtual viewpoint by using a motion picture obtained by imaging an imaging region with a plurality of imaging devices, the information processing method comprising: displaying, on a display, a standard image corresponding to the imaging region, a plurality of virtual viewpoint paths disposed in the standard image and representing a trajectory of movement of the virtual viewpoint, an indicator indicating a reproduction position of the virtual viewpoint video, and a reference image based on a virtual viewpoint image viewed from the virtual viewpoint corresponding to the reproduction position of the virtual viewpoint path among a plurality of virtual viewpoint images configuring the virtual viewpoint video, wherein a plurality of reference images at the same reproduction position are displayed simultaneously along the plurality of virtual viewpoint paths in the standard image.
20. A non-transitory computer-readable storage medium storing a program executable by a computer to perform information processing to generate a virtual viewpoint video based on a virtual viewpoint by using a motion picture obtained by imaging an imaging region with a plurality of imaging devices, the information processing comprising: displaying, on a display, a standard image showing the imaging region, a plurality of virtual viewpoint paths disposed in the standard image and representing a trajectory of movement of the virtual viewpoint, an indicator indicating a reproduction position of the virtual viewpoint video, and a reference image based on a virtual viewpoint image viewed from the virtual viewpoint corresponding to the reproduction position of the virtual viewpoint path among a plurality of virtual viewpoint images configuring the virtual viewpoint video, wherein a plurality of reference images at the same reproduction position are displayed simultaneously along the plurality of virtual viewpoint paths in the standard image.
1. An information processing apparatus that generates a virtual viewpoint video based on a virtual viewpoint by using a motion picture obtained by imaging an imaging region with a plurality of imaging devices, the information processing apparatus comprising: a processor; and a memory built in or connected to the processor, wherein the processor displays, on a display, a standard image corresponding to the imaging region, a plurality of virtual viewpoint paths disposed in the standard image and representing a trajectory of movement of the virtual viewpoint, an indicator indicating a reproduction position of the virtual viewpoint video, and a reference image based on a virtual viewpoint image viewed from the virtual viewpoint corresponding to the reproduction position of the virtual viewpoint path among a plurality of virtual viewpoint images configuring the virtual viewpoint video, wherein the processor generates gaze point information representing a gaze point at which the virtual viewpoint included in the virtual viewpoint path gazes in response to a given instruction, wherein the processor selects one reference image from among a plurality of the reference images corresponding to the same reproduction position in response to a given instruction, sets the gaze point corresponding to the one selected reference image as a standard gaze point, and generates, as other reference images that are not selected, an image based on a virtual viewpoint image showing an aspect of a case of gazing at the standard gaze point from the virtual viewpoint path corresponding to the non-selected other reference images.
2. The information processing apparatus according to claim 1, wherein the processor disposes at least one of the plurality of virtual viewpoint paths at an indicated position in the standard image.
3. The information processing apparatus according to claim 1, wherein the processor changes the reproduction position indicated by the indicator in response to a given instruction.
4. The information processing apparatus according to claim 1, wherein, in a case where the reproduction position is changed, the processor displays the reference image corresponding to the reproduction position after being changed on the display.
5. The information processing apparatus according to claim 4, wherein, in a case where the reproduction position is changed, the processor displays the reference image corresponding to the reproduction position after being changed at a position different from a position of the reference image corresponding to the reproduction position before being changed.
6. The information processing apparatus according to claim 1, wherein the processor displays only one reference image for one virtual viewpoint path.
7. The information processing apparatus according to claim 1, wherein the processor changes the reproduction position in response to an operation of moving the reference image on the standard image.
8. The information processing apparatus according to claim 1, wherein the processor displays two reference images for two adjacent virtual viewpoint paths according to relative positions of the two virtual viewpoint paths.
9. The information processing apparatus according to claim 1, wherein the processor changes at least one of a position, a length, or a shape of the virtual viewpoint path in response to a given instruction.
10. The information processing apparatus according to claim 1, wherein the processor displays at least one of the indicator or the virtual viewpoint path in different aspects before and after the reproduction position.
11. The information processing apparatus according to claim 1, wherein the gaze point information is a gaze point path representing a trajectory of movement of the gaze point, and the processor disposes a plurality of the gaze point paths in the standard image in response to a given instruction.
12. The information processing apparatus according to claim 1, wherein the processor changes a display aspect of a section including the virtual viewpoint in which a virtual viewpoint image obtained by gazing at the standard gaze point is not generatable in the virtual viewpoint path.
13. The information processing apparatus according to claim 1, wherein the processor displays the plurality of virtual viewpoint paths and the reference image in the single standard image.
14. The information processing apparatus according to claim 1, wherein the processor displays the virtual viewpoint path and the reference image in a plurality of the standard images representing at least a part of the imaging region.
15. The information processing apparatus according to claim 1, wherein the processor displays the reference image corresponding to the virtual viewpoint along the virtual viewpoint path for the plurality of virtual viewpoint paths.
16. The information processing apparatus according to claim 1, wherein the processor performs a selection combining process of cutting out parts of the plurality of virtual viewpoint paths and combining the cut-out parts in response to a given instruction, and generates the virtual viewpoint video on the basis of a path obtained by combining the cut-out parts in the selection combining process.
17. An information processing method of generating a virtual viewpoint video based on a virtual viewpoint by using a motion picture obtained by imaging an imaging region with a plurality of imaging devices, the information processing method comprising: displaying, on a display, a standard image corresponding to the imaging region, a plurality of virtual viewpoint paths disposed in the standard image and representing a trajectory of movement of the virtual viewpoint, an indicator indicating a reproduction position of the virtual viewpoint video, and a reference image based on a virtual viewpoint image viewed from the virtual viewpoint corresponding to the reproduction position of the virtual viewpoint path among a plurality of virtual viewpoint images configuring the virtual viewpoint video; generating gaze point information representing a gaze point at which the virtual viewpoint included in the virtual viewpoint path gazes in response to a given instruction; selecting one reference image from among a plurality of the reference images corresponding to the same reproduction position in response to a given instruction; setting the gaze point corresponding to the one selected reference image as a standard gaze point; and generating, as other reference images that are not selected, an image based on a virtual viewpoint image showing an aspect of a case of gazing at the standard gaze point from the virtual viewpoint path corresponding to the non-selected other reference images.
18. A non-transitory computer-readable storage medium storing a program executable by a computer to perform information processing to generate a virtual viewpoint video based on a virtual viewpoint by using a motion picture obtained by imaging an imaging region with a plurality of imaging devices, the information processing comprising: displaying, on a display, a standard image showing the imaging region, a plurality of virtual viewpoint paths disposed in the standard image and representing a trajectory of movement of the virtual viewpoint, an indicator indicating a reproduction position of the virtual viewpoint video, and a reference image based on a virtual viewpoint image viewed from the virtual viewpoint corresponding to the reproduction position of the virtual viewpoint path among a plurality of virtual viewpoint images configuring the virtual viewpoint video; generating gaze point information representing a gaze point at which the virtual viewpoint included in the virtual viewpoint path gazes in response to a given instruction; selecting one reference image from among a plurality of the reference images corresponding to the same reproduction position in response to a given instruction; setting the gaze point corresponding to the one selected reference image as a standard gaze point; and generating, as other reference images that are not selected, an image based on a virtual viewpoint image showing an aspect of a case of gazing at the standard gaze point from the virtual viewpoint path corresponding to the non-selected other reference images.
Claims 1-20 are anticipated by U.S. Patent No. 12,549,697 claims 1-18 as show in the table above.
Allowable Subject Matter
Claims 13 and 14 would be allowable if rewritten to overcome the rejection(s) under nonstatutory obviousness-type double patenting set forth in this office action and to include all of the limitations of the base claim and any intervening claims.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to SUNGHYOUN PARK whose telephone number is (571)270-1333. The examiner can normally be reached M - Thur 6:00 am - 4 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, THAI Q TRAN can be reached at (571)272-7382. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/SUNGHYOUN PARK/Examiner, Art Unit 2484
Prosecution Insights