Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
Applicant’s amendments and remarks submitted 04/03/2026 have been entered and considered, Claims 1-7, 9-10, 13-15 are amended. Claims 8, 11-12 are cancelled. Claims 13-17 are new. This action is made final.
Response to Arguments
Applicant’s arguments filed on 04/03/2026 6 have been fully considered but are not persuasive.
Applicant argues “In particular, YONEDA does not teach or suggest at least "obtain a user operation; set, based on the obtained user operation, positions and orientations of a plurality of virtual cameras for generating the virtual viewpoint image; and generate a camera path from the positions and the orientations of the plurality of virtual cameras; wherein the one or more processors generate the camera path according to a first procedure in a case of generating the camera path using the set positions and orientations of the plurality of virtual cameras, and generate the camera path according to a second procedure different from the first procedure in a case of generating the camera path using a current position and a current orientation of the virtual camera, and the stored preset position and the stored preset orientation" as presently claimed.”.
However, applicant should submit an argument under the heading “Remarks” pointing out disagreements with the examiner’s contentions. Applicant must also discuss the references applied against the claims, explaining how the claims avoid the references or distinguish from them. Simply stating the prior art does not teach the cited limitation without further reasoning is not persuasive. Please see detail rejection below.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-7, 9-10, 13-14, 17 are rejected under 35 U.S.C. 103 as being unpatentable over Yoneda et al (US20200275083).
Regarding Claim 1. Yoneda teaches An information processing apparatus comprising:
one or more memories storing instructions and storing, in advance, a preset position and a preset orientation of a virtual camera for generating a virtual viewpoint image generated from a plurality of images captured by a plurality of image capturing apparatuses (Yoneda, [0084] Note that the marks indicating state information and a camera path of virtual viewpoints may be of any shape as long as the state information and the camera path of the virtual viewpoints can be recognized, and are not limited to the marks shown in FIG. 4B. Furthermore, the functions that can be operated by an operator using the UI screen 402 are not limited to those. For example, a switching button may also be displayed for switching modes between automatically determining and manually determining the interpolation method managed by the interpolation method management unit 314 or interpolation of camera parameters. Alternatively, a slide bar with which it is possible to change a parameter according to the condition for determining an interpolation method may also be displayed.
[0107] In step S701, the interpolation method determination unit 315 acquires, from the key frame management unit 313, camera parameters of two target key frames. Then, the interpolation method determination unit 315 calculates the sight vector C1 based on the camera parameter indicating the position and orientation of the virtual viewpoint of one (first key frame) of the key frames. The interpolation method determination unit 315 calculates the sight vector C2 based on the camera parameter indicating the position and orientation of the virtual viewpoint of the other one (second key frame) of the key frames. Then, the interpolation method determination unit 315 calculates the distance (shortest distance) at which the sight vector C1 and the sight vector C2 are the closest to each other.
Therefore, in automatic camera path interpolation method, the system sets the camera path based on existing (pre-stored/preset) parameters/data regarding camera positions and orientations.); and
one or more processors executing the instructions to (Yoneda, abstract, the invention teaches an information processing apparatus which sets a plurality of key frames that each indicate a position of a virtual viewpoint that corresponds to a virtual viewpoint image, the plurality of key frames corresponding to different times, the virtual viewpoint image being generated based on images of an image capture region captured by a plurality of image capturing apparatuses from different directions. The information processing apparatus determines, based on a position of a virtual viewpoint indicated by a set first key frame and a position of a virtual viewpoint indicated by a set second key frame, a moving path of the virtual viewpoints during a time period between time that corresponds to the first key frame and time that corresponds to the second key frame, and outputs viewpoint information that indicates the determined moving path of the virtual viewpoints.
[0213] Embodiment(s) of the present invention can also be realized by a computer of a system or apparatus that reads out and executes computer executable instructions (e.g., one or more programs) recorded on a storage medium (which may also be referred to more fully as a 'non-transitory computer-readable storage medium') to perform the functions of one or more of the above-described embodiment(s)… The computer may comprise one or more processors (e.g., central processing unit (CPU), micro processing unit (MPU)) and may include a network of separate computers or separate processors to read out and execute the computer executable instructions.):
obtain a user operation;
set, based on the obtained user operation, positions and orientations of a plurality of virtual cameras for generating the virtual viewpoint image (Yoneda, [0079] FIGS. 4A and 4B are diagrams illustrating examples of a virtual viewpoint image and a UI screen that are displayed on the monitor (display unit 420) by the input/output processing unit 311. With reference to FIGS. 4A and 4B, an example of the virtual viewpoint image obtained by capturing an image of a soccer ground on which players are playing will be described. FIG. 4A shows a virtual viewpoint image 401 that was generated by the image generating unit 302 and is displayed on the monitor (image display unit 420a) via the input/output processing unit 311.
[0080] FIG. 4B shows an example of a UI screen 402 for editing, using the input/output processing unit 311, a camera path that is displayed on the monitor (UI display unit 420b).
[0190] In step S1602, the position/orientation designation unit 1304 designates the position/orientation as the state of the virtual camera of the key frame at the starting point in accordance with the operation performed by the user. For example, using the edit controller 407, the user moves the virtual camera frontward, rearward, leftward, rightward, upward, or downward, or rotates the virtual camera in a pan, tilt, or roll direction, to designate the position/orientation of the virtual camera.); and
generate a camera path from the positions and the orientations of the plurality of virtual cameras (Yoneda, [0194] In step S1609, the key frame addition unit 1303 performs processing for adding the key frame at the middle point, in accordance with an operation performed by user using the edit controller 407. Details of the adding processing will be described later with reference to the flowchart shown in FIG. 17. In step S1610, the allocation unit 1310 allocates the camera path 1505 obtained through inter keyframe interpolation to the position/orientation of the virtual camera of the middle frame. In step S1611, the camera path output unit 1311 outputs the camera path constituted by the key frames and the intermediate frame.);
wherein the one or more processors generate the camera path according to a first procedure in a case of generating the camera path using the set positions and orientations of the plurality of virtual cameras, and generate the camera path according to a second procedure different from the first procedure in a case of generating the camera path using a current position and a current orientation of the virtual camera, and the stored preset position and the stored preset orientation (Yoneda, [0084] Note that the marks indicating state information and a camera path of virtual viewpoints may be of any shape as long as the state information and the camera path of the virtual viewpoints can be recognized, and are not limited to the marks shown in FIG. 4B. Furthermore, the functions that can be operated by an operator using the UI screen 402 are not limited to those. For example, a switching button may also be displayed for switching modes between automatically determining and manually determining the interpolation method managed by the interpolation method management unit 314 or interpolation of camera parameters. Alternatively, a slide bar with which it is possible to change a parameter according to the condition for determining an interpolation method may also be displayed.
Although Yoneda didn’t explicitly indicate differentiating between these two camera path generation procedures, Yoneda teaches user is allowed to manually modify the virtual camera’s position and orientation. It is obvious to a person with ordinary skill in the art that when the user didn’t manually modify the virtual camera’s position, the system automatically calculates the camera path based on the preset virtual camera position and orientations. Therefore, the automatic calculation is the default procedure (first procedure), while the user’s modification is the customized procedure (second procedure).).
Regarding Claim 2. Yoneda further teaches The information processing apparatus according to Claim 1, wherein the camera path generated according to the second procedure is shorter than the camera path generated according to the first procedure (Yoneda, [0101] The following will describe in detail the processing for determining an inter-keyframe interpolation method that is performed by the interpolation method determination unit 315. Similar to FIG. 1, FIGS. 6A and 6B are diagrams showing, as the virtual camera 104 and the virtual camera 105, the camera parameters designated as key frames. Furthermore, in FIGS. 6A and 6B, a sight vector C1, which indicates the line of sight of the virtual camera 104, and a sight vector C2, which indicates the line of sight of the virtual camera 105, are superimposed on the diagrams.
[0104] A middle point between the point Pl and the point P2 is assumed to be a point O. The distance from the position of the virtual camera 104 (position of the virtual viewpoint) to the point O is assumed to be a D1. The distance from the position of the virtual camera 105 (position of the virtual viewpoint) to the point O is assumed to be a distance D2. In the present embodiment, the interpolation method determination unit 315 determines an interpolation method based on the distance D1 and the distance D2.
[0107] In step S701, the interpolation method determination unit 315 acquires, from the key frame management unit 313, camera parameters of two target key frames. Then, the interpolation method determination unit 315 calculates the sight vector C1 based on the camera parameter indicating the position and orientation of the virtual viewpoint of one (first key frame) of the key frames. The interpolation method determination unit 315 calculates the sight vector C2 based on the camera parameter indicating the position and orientation of the virtual viewpoint of the other one (second key frame) of the key frames. Then, the interpolation method determination unit 315 calculates the distance (shortest distance) at which the sight vector C1 and the sight vector C2 are the closest to each other.
[0114] The determination method in which the difference between the distance D1 and the distance D2 is compared with the threshold Th is used, but the present invention is not limited to this. Additionally, the interpolation method determination unit 315 may also determine, based on the ratio of the distance D1 to the distance D2, whether to use circular interpolation or spline interpolation. In this case, for example, a configuration is also possible in which, if the ratio of the distance D1 to the distance D2 is within a predetermined value, it is determined to use circular interpolation, and otherwise, it is determined to use spline interpolation.
Therefore, as shown in Fig 6A&6B, when the user move the virtual camera 104 closer to object 102 so that D1 becomes shorter, the corresponding camera path interpolation will generate a shorter camera path comparing to the automatically generated camera path generated based on default/original camera 104 position.).
Regarding Claim 3. Yoneda further teaches The information processing apparatus according to Claim 1, wherein the one or more processors further execute the instructions to hold information on the set positions and the set orientations of the virtual cameras, IDs, and time information (Yoneda, [0097] The order in which the camera path generated by the camera parameter interpolation unit 316 passes the key frames may be the same as the order of the key frames stored in the key frame management unit 313, or as the order of the above-described IDs of the key frames. Alternatively, if the camera parameters of the key frames include time parameters, the chronological order thereof may also be used.
[0107] In step S701, the interpolation method determination unit 315 acquires, from the key frame management unit 313, camera parameters of two target key frames. Then, the interpolation method determination unit 315 calculates the sight vector C1 based on the camera parameter indicating the position and orientation of the virtual viewpoint of one (first key frame) of the key frames.).
Regarding Claim 4. Yoneda further teaches The information processing apparatus according to Claim 3, wherein the one or more processors execute the instructions to generate the camera path according to the first procedure by interpolating the set positions and the set orientations of the virtual cameras in order of the IDs, and to generate the camera path according to the second procedure by interpolating between the current position and the current orientation of the virtual camera and the stored preset position and the stored preset orientation of the virtual camera (Yoneda, [0097] The order in which the camera path generated by the camera parameter interpolation unit 316 passes the key frames may be the same as the order of the key frames stored in the key frame management unit 313, or as the order of the above-described IDs of the key frames. Alternatively, if the camera parameters of the key frames include time parameters, the chronological order thereof may also be used.
[0101] The following will describe in detail the processing for determining an inter-keyframe interpolation method that is performed by the interpolation method determination unit 315. Similar to FIG. 1, FIGS. 6A and 6B are diagrams showing, as the virtual camera 104 and the virtual camera 105, the camera parameters designated as key frames. Furthermore, in FIGS. 6A and 6B, a sight vector C1, which indicates the line of sight of the virtual camera 104, and a sight vector C2, which indicates the line of sight of the virtual camera 105, are superimposed on the diagrams.
[0104] A middle point between the point Pl and the point P2 is assumed to be a point O. The distance from the position of the virtual camera 104 (position of the virtual viewpoint) to the point O is assumed to be a D1. The distance from the position of the virtual camera 105 (position of the virtual viewpoint) to the point O is assumed to be a distance D2. In the present embodiment, the interpolation method determination unit 315 determines an interpolation method based on the distance D1 and the distance D2.
Therefore, as shown in Fig 6A&6B, when the user move the virtual camera 104 closer to object 102 so that D1 becomes shorter and virtual camera 105 stays the same, the corresponding camera path interpolation will generate a camera path between current position and orientation of camera 104 and preset position and orientation of camera 105.).
Regarding Claim 5. Yoneda further teaches The information processing apparatus according to Claim 3, wherein the one or more processors executes the instructions to:
further store movement time for generating the camera path according to the second procedure, the movement time being different from the time information;
generate the camera path according to the first procedure further based on the time information, and generate according to the second procedure the camera path further based on the movement time (Yoneda, [0158] FIG. 14 shows an example (user I/F unit 400) of a user interface (hereinafter, referred to as "UI") for editing a camera path according to the present embodiment. The image display unit 420a displays a virtual viewpoint image.
[0159] FIG. 15 shows a display example of the UI display unit 420b. A timeline 1501 indicates frames on a single time axis. That is to say, the timeline 1501 schematically indicates a frame line of virtual viewpoint images or computer graphic images, and the positions of key frames in the frame line. In FIG. 15, three key frames, namely, a key frame 1502, a key frame 1503, and a key frame 1504 are shown. … The key frame 1504 is a key frame that corresponds to the key frame at the middle point. On the other hand, the camera path 1505 indicates movement (transition) of the positions/orientations of the virtual cameras that correspond to the respective frames…. Also, a mode display unit 1509 displays a camera path change mode that is currently being executed. The present embodiment uses, as the camera path change mode, a time change mode and a frame number change mode, which will be described later.
[0161] … The key frames 1502, 1503, and 1504 added by the user are displayed on the timeline 1501 shown in FIG. 15. Note that, by designating a desired position on the timeline 1501 using a pointing device such as a mouse, the user can also add a key frame to the position.
Therefore, the recorded time of keyframes are displayed on the timeline axis such as 1501. User can further modify the recorded time for each added keyframe and set a new movement time for related keyframe. The camera path is generated based on the recorded time or the designated movement time of each individual keyframes. Further see [0166-0170].).
Regarding Claim 6. Yoneda further teaches The information processing apparatus according to Claim 1, wherein the one or more processors further executes the instructions to, when generating the camera path according to the second procedure, correct the current orientation of the virtual cameras based on the stored preset orientation (Yoneda, [0191] The user views the virtual viewpoint image displayed on the image display unit 420a, and checks whether or not the position/orientation of the camera of the key frame at the starting point is correct. If it is determined that it is correct, the user notifies, using the edit controller 407, the camera path edit unit 301 of the fact that it is correct. Upon being notified of the fact that it is correct, the camera path edit unit 301 determines that it is correct (Yes in step S1604), the procedure moves to step S1605. If it is determined that the position/orientation of the camera of the key frame at the starting point is not correct, the user designates again, using the edit controller 407, the position/orientation of the virtual camera. If, using the edit controller 407, the position/orientation of the virtual camera is designated again, the camera path edit unit 301 determines that it is not correct (No in step S1604), the procedure returns to step S1602.).
Regarding Claim 7. Yoneda further teaches The information processing apparatus according to Claim 1, wherein the one or more processors further executes the instructions to switch, based on the user operation, between a first control mode for generating the camera path according to the first procedure and a second control mode for generating the camera path according to the second procedure (Yoneda, [0084] Note that the marks indicating state information and a camera path of virtual viewpoints may be of any shape as long as the state information and the camera path of the virtual viewpoints can be recognized, and are not limited to the marks shown in FIG. 4B. Furthermore, the functions that can be operated by an operator using the UI screen 402 are not limited to those. For example, a switching button may also be displayed for switching modes between automatically determining and manually determining the interpolation method managed by the interpolation method management unit 314 or interpolation of camera parameters. Alternatively, a slide bar with which it is possible to change a parameter according to the condition for determining an interpolation method may also be displayed.).
Claim 9 is similar in scope as Claim 1, and thus is rejected under same rationale.
Claim 10 is similar in scope as Claim 1, and thus is rejected under same rationale.
Regarding Claim 13. Yoneda further teaches The information processing apparatus according to claim 1, wherein, in the second procedure, the one or more processors decrement a remaining movement time by one frame for each frame of the virtual viewpoint image playback, and terminate the second procedure when the remaining movement time becomes zero (Yoneda, [0175] Frame Number Change Mode
[0176] The frame number change unit 1309 changes the number of intermediate frames between key frames so that the difference in the per-frame change amount in the state (position/orientation) of the virtual camera between the key frames is reduced. Specifically, in the present embodiment, the frame number change unit 1309 changes the number of intermediate frames so that the per-frame change amounts in the state (position/orientation) of the virtual camera before and after the key frame at the middle point are equal to each other. Note that the frame number change unit 1309 operates in the frame number change mode for changing the number of intermediate frames between key frames. Note that the state (virtual viewpoint state) of the virtual camera is as described above.
Therefore, when adding more intermediate frames between keyframes, the movement time for each time is decreased. The total movement time cannot exceed the time between keyframes. So the movement time will become zero when close to the keyframe.).
Regarding Claim 14. Yoneda further teaches The information processing apparatus according to claim 1, wherein the one or more processors, upon completion of the second procedure, transmit a signal to switch a control mode to a manual control mode (Yoneda, [0084] Note that the marks indicating state information and a camera path of virtual viewpoints may be of any shape as long as the state information and the camera path of the virtual viewpoints can be recognized, and are not limited to the marks shown in FIG. 4B. Furthermore, the functions that can be operated by an operator using the UI screen 402 are not limited to those. For example, a switching button may also be displayed for switching modes between automatically determining and manually determining the interpolation method managed by the interpolation method management unit 314 or interpolation of camera parameters. Alternatively, a slide bar with which it is possible to change a parameter according to the condition for determining an interpolation method may also be displayed.
Therefore, it is obvious to a person with ordinary skill in the art that the system will stays at the manual mode after the user manually editing the camera path unless user click on the switch mode button to change it to auto mode.).
Regarding Claim 17. Yoneda further teaches The information processing apparatus according to claim 1, wherein the set positions and the set orientations are different from the preset position and the preset orientation respectively (Yoneda, [0035] Moreover, when setting a camera path indicating movement of virtual viewpoints relating to a virtual viewpoint image, a user can designate time and virtual viewpoints (positions and orientations) of key frames to easily set the camera path. In this case, the moving speed of the viewpoints on the camera path depends on the time interval between the designated plurality of key frames and the amount of movement of the viewpoints. However, depending on the relationship of the plurality of key frames designated by the user, the moving speed of the viewpoints largely varies between before and after the key frames, and thus a virtual viewpoint image obtained based on the camera path is unnatural. Embodiment 4 discloses a configuration in which it is possible to easily set the movement of virtual viewpoints for use in generating a virtual viewpoint image with reduced unnaturalness and reduced feeling of strangeness.
[0161] … The key frames 1502, 1503, and 1504 added by the user are displayed on the timeline 1501 shown in FIG. 15. Note that, by designating a desired position on the timeline 1501 using a pointing device such as a mouse, the user can also add a key frame to the position.).
Claim 16 is rejected under 35 U.S.C. 103 as being unpatentable over Yoneda et al (US20200275083) in view of Matsubayashi (US20230028531).
Regarding Claim 16. Yoneda fails to explicitly teach, however, Matsubayashi teaches The information processing apparatus according to claim 1, wherein the one or more processors further execute the instructions to hold a setting indicating whether parameter correction is ON or OFF, and apply the setting when generating the camera path in the second procedure (Matsubayashi, abstract, the invention describes an information processing apparatus that outputs viewpoint information for generation of a virtual viewpoint image based on image data obtained by performing image capturing from directions different from one another by a plurality of image capturing apparatuses and comprises: an acquisition unit configured to acquire viewpoint information having a plurality of virtual viewpoint parameter sets respectively indicating positions and orientations of a virtual viewpoint at a plurality of points in time; a change unit configured to change a virtual viewpoint parameter set included in the viewpoint information based on a user operation during playback of a virtual viewpoint image in accordance with viewpoint information acquired by the acquisition unit; and an output unit configured to output viewpoint information having a virtual viewpoint parameter set changed by the change unit.
[0067] FIG. 6A and FIG. 6B are each a diagram in which a part of the pilot window in a case where the camera) control tab in FIG. 5 is selected is excerpted. In FIG. 6A and FIG. 6B, an operation axis name 511 indicates the item name of each operation axis. A camera parameter value 512 indicates the current value of each camera parameter determined in accordance with the operation for each operation axis.
[0068] An On switch 514 is a switch for making effective the editing operation of each operation axis (for making a parameter corresponding to the operation axis a change target) and an Off switch 515 is a switch for making ineffective the editing operation of each operation axis (for excluding a parameter corresponding to the operation axis from the change target). Specifically, in a case where the On switch 514 is clicked, a state (On state) where the editing operation is possible with the operation axis is brought about…. Further, in a case where the Off switch 515 is clicked, a state (Off state) where the editing operation is not possible with the operation axis is brought about.).
Yoneda and Matsubayashi are analogous art because they both teach method of generating virtual viewpoint images corresponding to virtual camera position/orientation changes made by user(s). Matsubayashi further teaches GUI with ON/OFF button enable/disable the editing of camera parameter values. Therefore, it would have been obvious to a person with ordinary skill in the art before the effective filing date of the claimed invention, to modify the virtual camera path generation method (taught in Yoneda), to further use the GUI for turning ON/OFF the virtual camera parameter value modification (taught in Matsubayashi), so as to accurately adjust the virtual camera parameter value and generate desired virtual viewpoint image (Matsubayashi, [0004-0006]).
Allowable Subject Matter
Claim 15 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Regarding Claim 15, it recites “The information processing apparatus according to claim 1, wherein, in the second procedure, the one or more processors perform calculation of the camera path in consideration of drop frames when a frame rate is a non-integer frame rate” in the context of Claim 15.
The prior arts of record either alone or in combination fails to teach or suggest the above quoted limitation of Claim 15. Therefore, Claim 15 is allowable over prior art.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to XIN SHENG whose telephone number is (571)272-5734. The examiner can normally be reached M-F 9:30AM-3:30PM 6:00PM-8:30PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jason Chan can be reached at 5712723022. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Xin Sheng/Primary Examiner, Art Unit 2619
Prosecution Insights