DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
Applicant’s response, filed 25 June 2025, to the last office action has been entered and made of record.
In response to the amendments to the claims, they are acknowledged, supported by the original disclosure, and no new matter is added.
Amendments to the independent claims 1, 19, and 20 have necessitated an updated ground of rejection over the applied prior art. Please see below for the updated interpretations and rejections.
Response to Arguments
Applicant's arguments filed 25 June 2025 have been fully considered but they are not persuasive.
Examiner notes the claims are treated with their broadest reasonable interpretations consistent with the specification. See MPEP 2111. Although the claims are interpreted in light of the specification, limitations from the specification are not read into the claims. See In re Van Geuns, 988 F.2d 1181, 26 USPQ2d 1057 (Fed. Cir. 1993). Furthermore, the test for obviousness is what the combined teachings of the references would have suggested to those of ordinary skill in the art. See In re Keller, 642 F.2d 413, 208 USPQ871 (CCPA 1981).
In response to Applicant’s arguments p. 11-14 of Applicant’s reply, that the combined teachings of Kondo, Yee, Bhuruth (i.e. US 2020/0168252, also referred herein as Bhuruth’20), and Yoneda fail to teach or suggest, disposing a virtual screen in the three-dimensional virtual space, and displaying at least one icon in spatial relation to the camera trajectory, the at least one icon indicating a respective imaging position along the camera trajectory of each respective key frame of the generated camera viewpoint video, the Examiner respectfully disagrees.
Kondo is relied upon to teach in a head mounted display (HMD) system, where a virtual camera 2 is arranged in and configured take an image / video in a virtual space, where the position of the virtual camera 2 can be arranged in any position, and that the virtual camera 2 takes a second virtual camera space image, including a ball and a character playing a soccer game in the virtual space, and that a display object, being an object for display in the virtual space, is rendered and arranged at a position in the virtual space, in which the second virtual camera space image taken by the virtual camera 2 is rendered (see Kondo [0033]-[0034] and [0045]).
Bhuruth’20 is relied upon to teach a known technique for determining a virtual camera path, which reproduces video footage of a scene on a display and allows a user to configure an action path through the virtual environment by a graphical user interface associated with the display of the video footage, in which a selected camera path template comprising a virtual camera path and focus path lines are inserted into the virtual environment and transformed via the user interface and used to synthesize video frames capturing the scene from the transformed virtual camera path to reproduce captured video (see Bhuruth’20 [0069]-[0070]; Fig. 3 and [0074]-[0079]; Fig. 4-6 and [0085]-[0088]).
Yoneda is further relied upon to teach a known technique of controlling a virtual camera position and orientation with a fixed camera path and displaying and indicating key frames by a black circle that is along a fixed camera path (see Yoneda [0019]-[0020], [0028], [0033], and Fig. 4).
From the combined teachings of the cited prior art, notably Kondo, Bhuruth’20, and Yoneda, one of ordinary skill in the art would have found it obvious that display objects of Kondo can be rendered and arranged at a position in the virtual space in which render and display images/videos taken by a virtual camera, and that by applying Bhuruth’20’s teachings would allow for the suggested system of Kondo and Yee to further insert and display a virtual camera path in the virtual space / environment as a line to allow a user to configure and define virtual camera path for determining the virtual camera parameters, for improved visualizing and controlling the positioning of the virtual camera in the virtual space, and that applying Yoneda’s teachings would allow for the display and indication of key frames along the virtual camera path by a marker, further improving the control for positioning the virtual camera in the virtual space with defining and indicating the positions of key frames along the virtual camera path.
Thus, the combined teachings of the cited prior art provides for the broadest reasonable interpretation for disposing a virtual screen in the three-dimensional virtual space, and displaying at least one icon in spatial relation to the camera trajectory, the at least one icon indicating a respective imaging position along the camera trajectory of each respective key frame of the generated camera viewpoint video.
In response to Applicant’s arguments p. 19-20 of Applicant’s reply, that the further teachings of Bhuruth’302 (i.e. US 2019/0287302, and also referred herein as Bhuruth’19), combined with the teachings of Kondo, Yee, Bhuruth’20, and Yoneda, fail to teach or suggest the virtual screen displaying generated viewpoint video from at least one position along the camera trajectory, and the generated camera viewpoint video configured to be displayed on the virtual screen, the Examiner respectfully disagrees.
The combined teachings of Kondo, Yee, Bhuruth’20, and Yoneda are relied upon to suggest a HMD system where a virtual camera 2 is arranged in and configured to take an image or video in a virtual space, where the position of the virtual camera 2 can be arranged according to a user defined virtual camera path, which includes set key frames, and the image in virtual space is rendered on a display object arranged in the virtual space (see Kondo [0033]-[0034], [0045], and [0048]; see Bhuruth’20 Fig. 3-6, [0069]-[0070], [0074]-[0079], and [0085]-[0088]; and see Yoneda [0019]-[0020], [0028], [0033], and Fig. 4).
Bhuruth’19 further teaches a known technique for controlling a virtual camera, where a user can select and edit a control point on a camera path to generate virtual camera footage in accordance to the edited control point, where the virtual camera video is generated by rendering virtual camera viewpoints according to the virtual camera poses and modified camera path and timeline corresponding to the virtual camera poses, and that a viewpoint for a particular virtual camera pose and moment in time in the scene can be synthesized (see Bhuruth’19 Fig. Fig. 9, Fig. 10, Fig. 14 and [0111]-[0116]).
From the combined teachings of the cited prior art, and further in view of Bhuruth’19 teachings, one of ordinary skill in the art would have found it obvious the application of Bhuruth’19’s techniques, would allow a user to select and edit a particular control point corresponding to a key frame of the virtual camera path, and that a corresponding viewpoint for the particular virtual camera pose and moment of the selected control point can be synthesized and rendered on the virtual display object, providing for improved capturing of image or videos of virtual objects with the virtual camera in the virtual space by allowing a user to select and edit control points corresponding to key frames of the virtual camera path.
Thus, the combined teachings of the cited prior art provides for the broadest reasonable interpretation for the virtual screen displaying generated viewpoint video from at least one position along the camera trajectory, and the generated camera viewpoint video configured to be displayed on the virtual screen.
Applicant’s remaining arguments with respect to claims 1, 19, and 20 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
Claims 1-7 and 17-21 are rejected under 35 U.S.C. 103 as being unpatentable over Kondo (US 2017/0153700), in view of Yee (US 2019/0083885), Bhuruth (US 2020/0168252, effectively filed 27 November 2018), herein Bhuruth’20, Yoneda (US 2020/0145635, effectively filed 6 November 2018), and Bhuruth (US 2019/0287302, effectively filed 13 March 2018), herein Bhuruth’19.
Regarding claim 1, Kondo discloses a video generation apparatus comprising:
processing circuitry (see Kondo [0022]-[0027], where disclosed memory stores programs to be processed by processing circuits, including various processing circuits such as a central processing unit, to perform the disclosed teachings) configured to
generate a camera viewpoint video in which a three-dimensional object in three-dimensional virtual space is shot using a virtual camera by moving the virtual camera in the three- dimensional virtual space (see Kondo [0033]-[0034] and [0045], where a virtual camera 2 is arranged in and configured take an image/video in a virtual space, where the position of the virtual camera 2 can be arranged in any position, and that the virtual camera 2 takes a second virtual camera space image including a ball and a character playing a soccer game in the virtual space; and see Kondo [0037]-[0038], where a control circuit unit is configured to control the output image to the display and includes an object control unit which includes a second virtual camera image generating unit); and
dispose a virtual screen in the three-dimensional virtual space, the virtual screen displaying the generated camera viewpoint video (see Kondo [0034] and [0048], where a display object in which the second virtual camera space image is rendered on is arranged, the display object is an object for display in the virtual space in which the image taken by the virtual camera 2 is rendered and arranged at a position in the virtual space; and see Kondo [0037]-[0038], where a control circuit unit is configured to control the output image to the display and includes an object control unit which includes a second virtual camera image generating unit).
Kondo does not explicitly disclose that the processing circuitry is configured to receive user operation, that moving the virtual camera in the three- dimensional virtual space is based on the received user operation, and control display of a camera trajectory of the virtual camera in the three- dimensional virtual space.
Yee teaches in a related and pertinent system and method for controlling a virtual camera (see Yee Abstract), where a virtual camera position input may be generated by a human virtual camera operator and be based on input from a user interface device such as a joystick, mouse or similar controller, and the processor receives controlling input that specifies the position, orientation, zoom, and other simulated camera features of the virtual camera (see Yee [0068] and [0070]), and that a path for the virtual camera may be defined by the user input by several methods (see Yee [0101]-[0102] and [0119]-[0123]), where the path for the virtual camera can be displayed (see Yee Fig. 4A and [0119]).
At the time of filing, one of ordinary skill in the art would have found it obvious to apply the teachings of Yee to the teachings of Kondo, such that a user interface device can be used to input the virtual camera position, which can be defined as a path, and that the corresponding virtual camera path can be displayed, which when combined with Kondo’s head mounted display (HMD) system, the virtual camera path would be suggested to be displayed in the virtual space.
This modification is rationalized as an application of a known technique to a known apparatus ready for improvement to yield predictable results.
In this instance, Kondo discloses a base HMD system where a virtual camera 2 is arranged in and configured take an image in a virtual space, where the position of the virtual camera 2 can be arranged in any position, and that the virtual camera 2 takes a second virtual camera space image including a ball and a character playing a soccer game in the virtual space and a display object in which the second virtual camera space image is rendered on is arranged in the virtual space.
Yee teaches a known technique of providing virtual camera position input by a human virtual camera operator and be based on input from a user interface device such as a joystick, mouse or similar controller, and that a path for the virtual camera may be defined by the user input by several methods, where the path for the virtual camera can be displayed.
One of ordinary skill in the art would have recognized that by applying Yee’s technique would allow for the system of Kondo to use a user interface device to input the virtual camera position, which can be defined as a path, where the corresponding virtual camera path can be displayed, suggesting that the virtual camera path would be displayed in the virtual space of Kondo’s head mounted display (HMD) system, leading to an improved control for positioning the virtual camera in the virtual space.
Kondo and Yee do not explicitly disclose while the camera trajectory of the virtual camera is displayed as a line in the three-dimensional virtual space.
Bhuruth’20 teaches in a related and pertinent system and method determining a virtual camera path (see Bhuruth’20 Abstract), where for a method in determining a virtual camera path, which reproduces video footage of a scene on a display and allows a user to configure an action path through the virtual environment by a graphical user interface associated with the display of the video footage (see Bhuruth’20 [0069]-[0070]), in which a selected camera path template comprising a virtual camera path and focus path lines are inserted into the virtual environment and transformed via the user interface and used to synthesize video frames capturing the scene from the transformed virtual camera path to reproduce captured video (see Bhuruth’20 Fig. 3 and [0074]-[0079]; see also Bhuruth’20 Fig. 4-6 and [0085]-[0088]).
At the time of filing, one of ordinary skill in the art would have found it obvious to apply the teachings of Bhuruth’20 to the teachings of Kondo and Yee, such that in defining a virtual camera path for determining the virtual camera position, the virtual camera path is inserted and displayed in the virtual environment as a line.
This modification is rationalized as an application of a known technique to a known apparatus ready for improvement to yield predictable results.
In this instance, Kondo and Yee discloses a base HMD system where a virtual camera 2 is arranged in and configured take an image in a virtual space, where the position of the virtual camera 2 can be arranged in any position, and that the virtual camera 2 takes a second virtual camera space image including a ball and a character playing a soccer game in the virtual space and a display object in which the second virtual camera space image is rendered on is arranged in the virtual space, in which a user interface device is used to input the virtual camera position, which can be defined as a path, where the corresponding virtual camera path can be displayed.
Bhuruth’20 teaches a known technique for determining a virtual camera path, which reproduces video footage of a scene on a display and allows a user to configure an action path through the virtual environment by a graphical user interface associated with the display of the video footage, in which a selected camera path template comprising a virtual camera path and focus path lines are inserted into the virtual environment and transformed via the user interface and used to synthesize video frames capturing the scene from the transformed virtual camera path to reproduce captured video.
One of ordinary skill in the art would have recognized that by applying Bhuruth’20’s technique would allow for the system of Kondo and Yee to insert and display the virtual camera path in the virtual environment as a line and allow a user to configure and define virtual camera path for determining the virtual camera parameters, leading to an improved method for visualizing and controlling for positioning of the virtual camera in the virtual space.
Kondo, Yee, and Bhuruth’20 do not explicitly disclose control display of at least one icon in spatial relation to the camera trajectory, the at least one icon indicating a respective imaging position along the camera trajectory of each respective key frame of the generated camera viewpoint video.
Yoneda teaches in a related and pertinent information processing apparatus for controlling the position of a virtual viewpoint (see Yoneda Abstract), where a virtual viewpoint image viewed from a virtual camera is generated, where the viewpoint of the virtual camera is represented by parameters specifying the viewpoint of the virtual camera and the position and orientation parameters of the virtual camera can be controlled (see Yoneda [0019]-[0020]), and that the virtual camera can move along a path specified by a fixed camera path (see Yoneda [0028]), where key frames are displayed and indicated by a black circle that is along a fixed camera path (see Yoneda Fig. 4 and [0033]).
At the time of filing, one of ordinary skill in the art would have found it obvious to apply the teachings of Yoneda to the teachings of Kondo, Yee, and Bhuruth’20, such that key frames are displayed and indicated along the virtual camera path by a marker.
This modification is rationalized as an application of a known technique to a known apparatus ready for improvement to yield predictable results.
In this instance, Kondo, Yee, and Bhuruth’20 disclose a base HMD system where a virtual camera 2 is arranged in and configured take an image in a virtual space, where the position of the virtual camera 2 can be arranged according to a user defined virtual camera path.
Yoneda teaches a known technique of controlling a virtual camera position and orientation with a fixed camera path and displaying and indicating key frames by a black circle that is along a fixed camera path.
One of ordinary skill in the art would have recognized that by applying Yoneda’s technique would allow for the system of Kondo, Yee, and Bhuruth’20, to similarly display and indicate key frames along the virtual camera path by a marker, leading to an improved control for positioning the virtual camera in the virtual space with defining and indicating the positions of key frames along the virtual camera path.
While Kondo teaches the virtual camera 2 is positioned in the virtual space and captures an image/video of an application performed in the virtual space, and a display object is rendered and arranged at a position in the virtual space, in which the image taken by the virtual camera 2 is rendered (see Kondo [0033]-[0034], [0045], and [0048]); Kondo, Yee, Bhuruth’20, and Yoneda do not explicitly disclose that the generated camera viewpoint video from at least one imaging position included along the camera trajectory, and the generated camera viewpoint video configured to be displayed on the virtual screen.
Bhuruth’19 teaches in a related and pertinent system and method of controlling a virtual camera (see Bhuruth’19 Abstract), where a user can select and edit a control point on a camera path to generate virtual camera footage in accordance to the edited control point, where the virtual camera video is generated by rendering virtual camera viewpoints according to the virtual camera poses and modified camera path and timeline corresponding to the virtual camera poses, and that a viewpoint for a particular virtual camera pose and moment in time in the scene can be synthesized (see Bhuruth’19 Fig. Fig. 9, Fig. 10, Fig. 14 and [0111]-[0116]).
At the time of filing, one of ordinary skill in the art would have found it obvious to apply the teachings of Bhuruth’19 to the teachings of Kondo, Yee, Bhuruth’20, and Yoneda such that a particular control point corresponding to a key frame of the virtual camera path can be selected to be edited and that a corresponding viewpoint for the particular virtual camera pose and moment of the selected control point can be synthesized and rendered on the virtual display object.
This modification is rationalized as an application of a known technique to a known apparatus ready for improvement to yield predictable results.
In this instance, Kondo, Yee, Bhuruth’20, and Yoneda disclose a base HMD system where a virtual camera 2 is arranged in and configured to take an image or video in a virtual space, where the position of the virtual camera 2 can be arranged according to a user defined virtual camera path, which includes set key frames, and the image in virtual space is rendered on a display object arranged in the virtual space.
Bhuruth’19 teaches a known technique for controlling a virtual camera, where a user can select and edit a control point on a camera path to generate virtual camera footage in accordance to the edited control point, where the virtual camera video is generated by rendering virtual camera viewpoints according to the virtual camera poses and modified camera path and timeline corresponding to the virtual camera poses, and that a viewpoint for a particular virtual camera pose and moment in time in the scene can be synthesized.
One of ordinary skill in the art would have recognized that by applying Bhuruth’19’s technique would allow for the system of Kondo, Yee, Bhuruth’20, and Yoneda to similarly select and edit a particular control point corresponding to a key frame of the virtual camera path, and that a corresponding viewpoint for the particular virtual camera pose and moment of the selected control point can be synthesized and rendered on the virtual display object, leading to an improved system for capturing image or videos of virtual objects with the virtual camera in the virtual space by allowing a user to select and edit control points corresponding to key frames of the virtual camera path.
Regarding claim 2, please see the above rejection of claim 1. Kondo, Yee, Bhuruth’20, Yoneda, and Bhuruth’19 disclose the video generation apparatus according to claim 1, wherein the processing circuitry further configured (see Kondo [0022]-[0027], where disclosed memory stores programs to be processed by processing circuits to perform the disclosed teachings) to
generate a user viewpoint video of viewing an inside of the three- dimensional virtual space from a user viewpoint set in the three-dimensional virtual space (see Kondo [0033]-[0034], where a first virtual camera 1 is associated with a first person perspective of the user in the virtual space), and
display the camera trajectory in the user viewpoint video (see Yee Fig. 4A and [0119], where the path for the virtual camera can be displayed; where the combined teaching suggest to display the virtual camera path in the virtual space, which the user views the virtual space with a first person perspective).
Regarding claim 3, please see the above rejection of claim 1. Kondo, Yee, Bhuruth’20, Yoneda, and Bhuruth’19 disclose the video generation apparatus according to claim 1, wherein the processing circuitry is further configured to display, on the virtual screen, a seek bar indicating a playback position of the camera viewpoint video (see Yee Fig. 5 and [0135]-[0138], where a time slider on a timeline is displayed for controlling the timing of playback of the video captured using the virtual camera path),
receive the user operation on the displayed seek bar (see Yee Fig. 5 and [0135]-[0138], where the time slider is used to move along the previously generated path by increasing or decreasing the scene time and by manipulating the time slider, the user can preview the resultant video, or use additional methods of controlling the timing of playback), and
change a playback position of the three-dimensional object and the playback position of the camera viewpoint video in accordance with the received user operation on the displayed seek bar (see Yee Fig. 5 and [0135]-[0138], where a time slider on a timeline is displayed for controlling the timing of playback of the video captured using the virtual camera path).
Regarding claim 4, please see the above rejection of claim 1. Kondo, Yee, Bhuruth’20, Yoneda, and Bhuruth’19 disclose the video generation apparatus according to claim 1, wherein
the processing circuitry receives movement operation of moving a controller in real space as the user operation (see Yee [0070], where a virtual camera position input may be generated by a human virtual camera operator and be based on input from a user interface device such as a joystick, mouse or similar controller), and
wherein the processing circuitry generates the camera viewpoint video obtained by moving the virtual camera in accordance with a trajectory of the controller (see Yee [0112], in response to detecting user input, an updated virtual camera image is synthesized to correspond to the virtual camera’s position, orientation and zoom; see also Yee [0114]).
Regarding claim 5, please see the above rejection of claim 1. Kondo, Yee, Bhuruth’20, Yoneda, and Bhuruth’19 disclose the video generation apparatus according to claim 1, wherein
the processing circuitry generates the camera viewpoint video obtained by moving the virtual camera in accordance with the user operation while fixing a height of the virtual camera in a fixed mode in which the height of the virtual camera is fixed (see Yee [0121], where the height of the virtual camera above the scene may be constrained using the path information).
Regarding claim 6, please see the above rejection of claim 1. Kondo, Yee, Bhuruth’20, Yoneda, and Bhuruth’19 the video generation apparatus according to claim 1, wherein
the processing circuitry receives, as the user operation, setting of a point of interest with respect to the three- dimensional object (see Yee [0114], where the virtual camera can be changed to a front-on view so that objects can be seen closer and user inputs may be detected in accordance with the configured virtual camera controls), and
wherein the processing circuitry generates the camera viewpoint video obtained by moving the virtual camera in accordance with the user operation while controlling a posture of the virtual camera such that a center of an angle of view of the virtual camera is directed to the point of interest (see Yee [0114], where the virtual camera can be changed to a front-on view of the objects).
Regarding claim 7, please see the above rejection of claim 1. Kondo, Yee, Bhuruth’20, Yoneda, and Bhuruth’19 disclose the video generation apparatus according to claim 1, wherein the processing circuitry is further configured to
display a menu screen in the three-dimensional virtual space (see Kondo [0020], where the virtual space includes various objects that can be operated by the user, menu images, and the like), and
receive the user operation on the menu screen (see Kondo [0020], where the virtual space includes various objects that can be operated by the user, menu images, and the like; suggesting that a user operates with menu images in the virtual space; see also Bhuruth’20 [0053], where other forms of user input devices may be used to navigate about menus).
Regarding claim 17, please see the above rejection of claim 2. Kondo, Yee, Bhuruth’20, Yoneda, and Bhuruth’19 disclose the video generation apparatus according to claim 2, wherein
the processing circuitry is further configured to receive user operation related to a position and a gaze direction of the user viewpoint in the three-dimensional virtual space (see Kondo [0030]-[0038], where the virtual camera corresponding to the first person perspective of the user is controlled by the movement sensor and eye gaze sensor), and
wherein the processing circuitry is further configured to control the position and the gaze direction of the user viewpoint in the three- dimensional virtual space based on the received user operation (see Kondo [0030]-[0038], where the virtual camera corresponding to the first person perspective of the user is controlled by the movement sensor and eye gaze sensor; see also Kondo [0039]-[0043] and [0056], where the field of view image is generated according to the field of view region as determined from the movement sensor and eye gaze sensor).
Regarding claim 18, please see the above rejection of claim 2. Kondo, Yee, Bhuruth’20, Yoneda, and Bhuruth’19 disclose the video generation apparatus according to claim 2, wherein
the processing circuitry displays the user viewpoint video obtained by moving the user viewpoint along the camera trajectory (see Yee [0101]-[0102] and [0119]-[0123], where a path for the virtual camera may be defined by the user input; see Kondo [0033]-[0034], where a first virtual camera 1 is associated with a first person perspective of the user in the virtual space; where the combined teachings suggests that the first virtual camera may travel along the user defined virtual camera path).
Regarding claim 19, it recites a method performing the apparatus functions of claim 1. Kondo, Yee, Bhuruth’20, Yoneda, and Bhuruth’19 teach the method performing the apparatus functions of claim 1. Please see above for detailed claim analysis.
Please see the above rejection for claim 1, as the rationale to combine the teachings of Kondo, Yee, Bhuruth’20, Yoneda, and Bhuruth’19 are similar, mutatis mutandis.
Regarding claim 20, it recites a non-transitory computer readable medium having embodied thereon a program, which when executed by a computer causes the computer to perform the apparatus functions of claim 1. Kondo, Yee, Bhuruth’20, Yoneda, and Bhuruth’19 teach a non-transitory computer readable medium having embodied thereon a program causing a computer to perform the apparatus functions of claim 1 (see Kondo [0022]-[0027], where disclosed memory stores programs to be processed by processing circuits to perform the disclosed teachings). Please see above for detailed claim analysis.
Please see the above rejection for claim 1, as the rationale to combine the teachings of Kondo, Yee, Bhuruth’20, Yoneda, and Bhuruth’19 are similar, mutatis mutandis.
Regarding claim 21, please see the above rejection of claim 1. Kondo, Yee, Bhuruth’20, Yoneda, and Bhuruth’19 disclose the video generation apparatus according to claim 1, wherein, when the user selects a respective icon, the processing circuitry controls the virtual screen to display the respective key frame of the generated camera viewpoint video corresponding to the respective icon selected by the user (see Bhuruth’19 Fig. Fig. 9, Fig. 10, Fig. 14 and [0111]-[0116], where a user can select and edit a control point on a camera path to generate virtual camera footage in accordance to the edited control point, where the virtual camera video is generated by rendering virtual camera viewpoints according to the virtual camera poses and modified camera path and timeline corresponding to the virtual camera poses, and that a viewpoint for a particular virtual camera pose and moment in time in the scene can be synthesized; where the combined teachings suggest that a particular control point corresponding to a key frame of the virtual camera path can be selected to be edited and that a corresponding viewpoint for the particular virtual camera pose and moment of the selected control point can be synthesized and rendered on the virtual display object).
Claims 8-14 are rejected under 35 U.S.C. 103 as being unpatentable over Kondo, Yee, Bhuruth’20, Yoneda, and Bhuruth’19 as applied to claim 1 above, and further in view of Kato (US 2021/0144358, effectively filed 20 July 2018).
Regarding claim 8, please see the above rejection of claim 1. Kondo, Yee, Bhuruth’20, Yoneda, and Bhuruth’19 do not explicitly disclose the video generation apparatus according to claim 1, wherein
the processing circuitry receives setting of each key frame in the camera viewpoint video.
Kato teaches in a related and pertinent information processing apparatus generates camera path information for a virtual viewpoint (see Kato Abstract), where an operation unit receives operation data in response to a user operation related to start of generation of a camera path, where the camera path information includes a key frame number, and corresponds to information that represents the chronological order of key frames on the camera path (see Kato [0027]-[0030]).
At the time of filing, one of ordinary skill in the art would have found it obvious to apply the teachings of Kato to the teachings of Kondo, Yee, Bhuruth’20, Yoneda, and Bhuruth’19, such that a user can provide operation data which includes information to set key frames in the virtual camera path.
This modification is rationalized as an application of a known technique to a known apparatus ready for improvement to yield predictable results.
In this instance, Kondo, Yee, Bhuruth’20, Yoneda, and Bhuruth’19 discloses a base HMD system where a virtual camera 2 is arranged in and configured take an image in a virtual space, where the position of the virtual camera 2 can be arranged according to a user defined virtual camera path with marked key frames.
Kato teaches a known technique of generating camera path information for a virtual viewpoint, where an operation unit receives operation data in response to a user operation related to start of generation of a camera path, where the camera path information includes a key frame number, and corresponds to information that represents the chronological order of key frames on the camera path.
One of ordinary skill in the art would have recognized that by applying Kato’s technique would allow for the system of Kondo, Yee, Bhuruth’20, Yoneda, and Bhuruth’19 to allow a user to provide operation data which includes information to set key frames in the virtual camera path, leading to an improved control for positioning the virtual camera in the virtual space with defining key frames along the virtual camera path.
Regarding claim 9, please see the above rejection of claim 8. Kondo, Yee, Bhuruth’20, Yoneda, Bhuruth’19, and Kato disclose the video generation apparatus according to claim 8, wherein the processing circuitry is further configured (see Kondo [0022]-[0027], where disclosed memory stores programs to be processed by processing circuits to perform the disclosed teachings) to
correct the camera trajectory based on each key frame set by the user operation (see Bhuruth’20 [0070]-[0072], where the user can configure an action path through the virtual environment by the selection of three dimensional points or locations and the each point defined by the user specifies a spatial and temporal reference point in the video data of the scene, and that a line of best fit between multiple user specified points is generated and is used to become the action path), and
regenerate, at time of editing the camera viewpoint video, the camera viewpoint video obtained by such a manner that the virtual camera is moved along the corrected camera trajectory (see Bhuruth’20 [0079], where video frames capturing the scene from the transformed virtual camera path are generated and reproduce the captured video on the display; and see Kato [0033], where camera path manager corrects the key frame number to generate the camera path information; where the combined teachings would suggest that a virtual camera video would be moved along the corrected virtual camera path).
Regarding claim 10, please see the above rejection of claim 9. Kondo, Yee, Bhuruth’20, Yoneda, Bhuruth’19, and Kato disclose the video generation apparatus according to claim 9, wherein
the processing circuitry is further configured to display, in the three- dimensional virtual space, a guide line that assists the camera trajectory of the virtual camera (see Bhuruth’20 [0072], where a line of best fit between multiple user specified points is generated and is used to become the action path; and see Yoneda Fig. 4 and [0033]-[0034], where a virtual camera may switch from a free camera path to a fixed camera path).
Regarding claim 11, please see the above rejection of claim 10. Kondo, Yee, Bhuruth’20, Yoneda, Bhuruth’19, and Kato disclose the video generation apparatus according to claim 10, wherein
the processing circuitry corrects at least one of the respective imaging position or a respective imaging time of at least one key frame such that the at least on corrected key frame is set on the guide line (see Bhuruth’20 [0082], the position and orientation of the virtual camera can be adjusted to align with the multiple user defined points; and see Kato [0033], where camera path manager corrects the key frame number to generate the camera path information), and
wherein the processing circuitry corrects the camera trajectory in accordance with the corrected key frame (see Bhuruth’20 [0072], where a line of best fit between multiple user specified points is generated and is used to become the action path; see Bhuruth’20 [0082], where the position and orientation of the virtual camera can be adjusted to align with the multiple user defined points; and see Yoneda [0034], where the camera parameters that fill in between a free camera path position and a key frame position are acquired by interpolation processing to smoothly connect the virtual camera to the fixed camera path).
Regarding claim 12, please see the above rejection of claim 9. Kondo, Yee, Bhuruth’20, Yoneda, Bhuruth’19, and Kato disclose the video generation apparatus according to claim 9, wherein the processing circuitry corrects the respective imaging position of at least one key frame such that two or more consecutive time intervals between a plurality of the frames are equal to teach other or two or more consecutive distance intervals between the imaging positions of a plurality of the key frame are equal to each other (see Kato [0037], where a time control unit may identify two key frames before and after the camera path time of the frame that is being played back at the time and may acquire the moving image time by linear interpolation of the moving images of the two key frames, enabling the difference between the frame rate of the termina that generates the camera path information and frame rate of the terminal that uses the camera path information to be canceled and equal to each other).
Regarding claim 13, please see the above rejection of claim 9. Kondo, Yee, Bhuruth’20, Yoneda, Bhuruth’19, and Kato disclose the video generation apparatus according to claim 9, wherein
when at least one key frame is added or deleted by the user operation, the processing circuitry corrects the camera trajectory on a basis of the at least one added or deleted key frame (see Kato [0033], where camera path manager identifies a keyframe number related to the camera path information and corrects the key frame number to generate the camera path information; where the combined teachings would suggest that a virtual camera video would be moved along the corrected virtual camera path).
Regarding claim 14, please see the above rejection of claim 9. Kondo, Yee, Bhuruth’20, Yoneda, Bhuruth’19, and Kato disclose the video generation apparatus according to claim 9, wherein
the processing circuitry corrects the camera trajectory on a basis of a trajectory drawn by the user operation (see Yee [0130], where a user interaction creating a path which is also associated with a time line control, where end points of the path can be associated with a particular time marker and inferred through the action of inputting the path and the locations determined from the path the speed at which the movement of the virtual camera along the path will occur may be inferred; suggesting that the trajectory of the virtual camera can be corrected according to the user defined path).
Claim 15 is rejected under 35 U.S.C. 103 as being unpatentable over Kondo, Yee, Bhuruth’20, Yoneda, Bhuruth’19, and Kato as applied to claim 9 above, and further in view of Liu (US 2021/0201591, effectively filed 5 December 2018).
Regarding claim 15, please see the above rejection of claim 9. Kondo, Yee, Bhuruth’20, Yoneda, Bhuruth’19, and Kato do not explicitly disclose the video generation apparatus according to claim 9, wherein
the processing circuitry is further configured to change, in accordance with an illumination mode for the three-dimensional object set by the user operation, the illumination mode for the three- dimensional object.
Liu teaches in a related and pertinent method for observing a virtual environment (see Liu Abstract), where when an object is moved from a first scene to a second scene, where the first scene may be a bright scene and the second scene is a dim scene, the first observation manner is adjusted to a second observation manner, e.g. observing with night vision equipment turned on (see Liu Fig. 4 and [0068]-[0080]).
At the time of filing, one of ordinary skill in the art would have found it obvious to apply the teachings of Liu to the teachings of Kondo, Yee, Bhuruth’20, Yoneda, Bhuruth’19, and Kato such that a corresponding illumination correction is made when the illumination for a scene of an imaged virtual character changes, as suggested by Liu which observes a virtual object with night vision when the a first bright scene changes to a second dim scene.
This modification is rationalized as an application of a known technique to a known apparatus ready for improvement to yield predictable results.
In this instance, Kondo, Yee, Bhuruth’20, Yoneda, Bhuruth’19, and Kato disclose a base HMD system where a virtual camera 2 is arranged in and configured take an image in a virtual space, where the position of the virtual camera 2 can be arranged according to a user defined virtual camera path, which includes set key frames, where the virtual camera can be used to take an image of a virtual character.
Liu teaches a known technique of adjusting the observation manner when a virtual object is moved from a first scene to a second scene, where the first scene may be a bright scene and the second scene is a dim scene, the first observation manner is adjusted to a second observation manner, e.g. observing with night vision equipment turned on.
One of ordinary skill in the art would have recognized that by applying Liu’s technique would allow for the system of Kondo, Yee, Bhuruth’20, Yoneda, Bhuruth’19, and Kato to similarly make a corresponding illumination correction when the illumination for a scene of an imaged virtual character change, leading to an improved control for imaging virtual objects with the virtual camera in the virtual space by making illumination corrections.
Claim 16 is rejected under 35 U.S.C. 103 as being unpatentable over Kondo, Yee, Bhuruth’20, Yoneda, and Bhuruth’19, as applied to claim 1 above, and further in view of Kanatsu et al. (US 2018/0204381), herein Kanatsu.
Regarding claim 16, please see the above rejection of claim 1. Kondo, Yee, Bhuruth’20, Yoneda, and Bhuruth’19 do not explicitly disclose the video generation apparatus according to claim 1, wherein
the processing circuitry generates the camera viewpoint video in which a screen size of the three- dimensional object is associated as meta information with each of a plurality of frames in the camera viewpoint video.
Kanatsu teaches in a related and pertinent image processing apparatus includes a generation unit configured to generate a virtual viewpoint image corresponding to a virtual viewpoint (see Kanatsu Abstract), wherefor successive three-dimensional model information generation, a camera adapter acquires a captured image from a camera connected to the camera adapter, where the camera adapter can perform appending the meta-information when transferring the foreground image and the background image to a subsequent camera adapter, e.g. the position of the foreground image in a frame, a data size, a frame number, and image capturing time are appended as the meta information. (see Kanatsu [0145]-[0150]).
At the time of filing, one of ordinary skill in the art would have found it obvious to apply the teachings of Kanatsu to the teachings of Kondo, Yee, Bhuruth’20, Yoneda, and Bhuruth’19 such that images captured of a virtual character in the virtual space includes meta information corresponding to the position of the image in a frame, a data size, a frame number, and image capturing time of the captured image is appended.
This modification is rationalized as an application of a known technique to a known apparatus ready for improvement to yield predictable results.
In this instance, Kondo, Yee, Bhuruth’20, Yoneda, and Bhuruth’19 disclose a base HMD system where a virtual camera 2 is arranged in and configured take an image in a virtual space, where the position of the virtual camera 2 can be arranged according to a user defined virtual camera path, which includes set key frames, where the virtual camera can be used to take an image of a virtual character.
Kanatsu teaches a known technique where the camera adapter can perform appending meta-information when transferring the foreground image and the background image to a subsequent camera adapter, e.g. the position of the foreground image in a frame, a data size, a frame number, and image capturing time are appended as the meta information.
One of ordinary skill in the art would have recognized that by applying Kanatsu’s technique would allow for the system of Kondo, Yee, Bhuruth’20, Yoneda, and Bhuruth’19 to similarly append to the captured images of a virtual character in the virtual space meta information corresponding to the position of the image in a frame, a data size, a frame number, and image capturing time of the captured image, leading to an improved system for capturing images of virtual objects with the virtual camera in the virtual space by appending additional meta information.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to TIMOTHY WING HO CHOI whose telephone number is (571)270-3814. The examiner can normally be reached 9:00 AM to 5:00 PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, VINCENT RUDOLPH can be reached at (571) 272-8243. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/TIMOTHY CHOI/Examiner, Art Unit 2671
/VINCENT RUDOLPH/Supervisory Patent Examiner, Art Unit 2671