Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
The Amendment filed November 19th, 2025 has been entered. Claims 1 and 6-20 have been amended. Claims 1-20 remain rejected in the application. Applicant’s amendments to the specifications and claims have overcome each and every objection previously set forth in the Non-Final Office Action mailed August 19th, 2025 and have therefore been withdrawn. In addition, the amendments made to claim 20 have overcome the previous 35 U.S.C. 101 rejection and has therefore also been withdrawn.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-7, 11-13, 15-17, 19 and 20 are rejected under 35 U.S.C. 103 as being unpatentable over Masuda et al. (Pub. No.: US 2018/0302610 A1), hereinafter Masuda, in view of Hanamoto (Pub. No.: US 2020/0344456 A1).
Regarding claim 1, Masuda discloses an information processing device (Paragraph 28 teaches that with reference to the accompanying drawings, the following describes a viewpoint selection assistance program, a viewpoint selection assistance method, and a viewpoint selection assistance apparatus) comprising:
circuitry (Paragraph 58 teaches that the control part 15 is a processor that performs overall control of the client terminal 10) configured to
select data to be used to generate a free viewpoint image according to at least one of an event or a viewpoint within one or more frames of the free viewpoint image (Paragraph 41 teaches that implemented in the client terminal 10 according to the present embodiment is a user interface (UI) equipped with a viewpoint selection assistance function which assists in selecting a scene-suited viewpoint, paragraph 60 teaches that the control part 15 virtually implements the processors described below by loading a viewpoint selection assistance program for implementing the viewpoint selection assistance function into a work area in a RAM such as a dynamic random-access memory (DRAM) or a static random-access memory (SRAM) and paragraph 35 teaches that in this case, the plurality of cameras 20 are placed around the field F, facing toward the inside of the field F. The cameras 20 are so placed that the entire area of the field F is captured within the combined image capturing fields of the respective cameras 20. In order to enable analysis of the 3D structure of a subject on the field F based on pixels common between images captured by two or more cameras 20, each camera 20 is placed so that its image capturing range may overlap that of another one or more cameras 20. When the cameras 20 thus arranged capture images in synchronization with one another on a frame-by-frame basis, a plurality of images captured simultaneously from different viewpoints, or multiview images, are obtained per frame.),
wherein the data is selected by setting either a plurality of pieces of captured image data obtained by imaging a subject included in the event from the plurality of viewpoints (Paragraph 35 teaches that in this case, the plurality of cameras 20 are placed around the field F, facing toward the inside of the field F. The cameras 20 are so placed that the entire area of the field F is captured within the combined image capturing fields of the respective cameras 20)
or processing data obtained by executing processing related to three-dimensional information generation of the subject on the captured image data as selection target data (Paragraph 85 teaches that the video generator 15f is a processor that generates free viewpoint video. For example, the video generator 15f generates a free viewpoint image by generating a 3D model based on the 3D information in the video generation information 13c, performing texture mapping on the 3D model with multiview images based on the position of the viewpoint selected via the input part 11a, and rendering the texture mapped 3D model. Free viewpoint video is reproduced when free viewpoint images thus generated are displayed on the display part 11b one after another). However, Masuda fails to disclose select data to be used to generate a free viewpoint image according to an importance related to at least one of an event or a viewpoint within one or more frames of the free viewpoint image.
Hanamoto discloses select data to be used to generate a free viewpoint image according to an importance related to at least one of an event or a viewpoint within one or more frames of the free viewpoint image (Paragraph 54 teaches that as described above, with the image processing apparatus of the first embodiment, three-dimensional object data for each object is generated with multiple classes, and the importance levels of the objects is decided based on the characteristics of the contents, such as the analysis results of a game category or an actual game. Also, since the class of the model data constituting the model data for transmission is set according to the importance level, the model data for the virtual viewpoint image corresponding to the interactive operation can be generated suitably and transmitted efficiently. Additionally, FIG. 9A and paragraph 46 teach that note that in order to display the content selection window 901, the user terminal 130 acquires a list of pieces of content to be selected from the image processing apparatus 100. Each piece of content in the list corresponds to a group of multi-viewpoint frames that are temporally continuous. For example, based on the play content (analysis result of the analysis apparatus 140), one piece of content may be generated using a series of multi-viewpoint frames including time codes close to that at which the play occurred. For example, as described in step S411, the model data associated with each piece of play information may be used as one piece of content. Alternatively, for example, separate pieces of content may be generated for the multi-viewpoint frames of the first half of a match and the multi-viewpoint frames of the last half of the match.). Since Masuda teaches the initial processing device steps related to a selective assistance function that can assist a user in receiving and selecting multi-frame data related to a free viewpoint image according to a subject or event and Hanamoto teaches a processing device that provides importance data related to objects/subjects and a selection window that allows for the selection of specific important events related to important objects/subjects and can then display those important events to the user within one or more frames of the selected free viewpoint image/important event, it would have been obvious to a person having ordinary skill in the art to combine the functions together so that the selective assistance function could also assist the user in displaying events that are important to them and allow the user to view that important event via a free image viewpoint that they could then view through multiple frames within one another in relation to the selected event.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Masuda to incorporate the teachings of Hanamoto, so that the combined functions together would allow for more accurate and specific free image viewpoints by providing the user with the ability to select from events/subjects important to them and can then view and scroll through multiple frames of image data related to those specific important events.
Furthermore, Masuda in view of Hanamoto disclose and control generation of the free viewpoint image based on the captured image data or the processing data set as the selection target data for each frame of the free viewpoint image (FIGs 9A & 9B and paragraph 50 of Hanamoto teach that in step S815, the user terminal 130 (CPU 131) receives the model data. In step S816, the CPU 131 uses the received model data to generate a virtual viewpoint image and regenerates the virtual viewpoint image on the display unit 135. Upon receiving the model data from the image processing apparatus 100, the CPU 131 executing the application in the user terminal 130 transitions to the virtual viewpoint window 903 shown in FIG. 9B. In the virtual viewpoint window 903, player model data 904 and background model data 905 are displayed, and it is possible to display an image at any camera position/direction/angle of view according to a touch operation on the screen or the like. Also, it is possible to move to an image of any time code using a time code slider bar 906.).
Regarding claim 2, Masuda in view of Hanamoto disclose everything claimed as applied above (see claim 1), in addition, Masuda in view of Hanamoto disclose wherein the importance includes an importance of a scene included in the event (Paragraph 98 teaches that as described above, the client terminal 10 according to the present embodiment narrows the players to one or ones that are involved with the ball based on the moving path of the ball identified by reference to tracking information on a specified scene, and sets, as an area of interest, an area containing the moving paths of the ball and the player(s) thus narrowed down to. By providing such an area of interest as viewpoint selection assistance information, the client terminal 10 may assist in selecting one of free viewpoints that captures within its field of view a feature that draws viewers' interest in sports spectating. Thus, the client terminal 10 of the present embodiment may assist in selecting a scene-suited viewpoint).
Regarding claim 3, Masuda in view of Hanamoto disclose everything claimed as applied above (see claim 1), in addition, Masuda in view of Hanamoto disclose wherein the importance includes an importance for each viewpoint (Paragraph 66 teaches that although the example in FIG. 3 illustrates a case where a scene is defined by the scene tag, the start time ts, and the end time te, any other items may also be included. As an example of such an item, the scene information 13a may also include game identification information for identifying a soccer game. In this case, for example, scenes from the same game may be extracted from the scenes included in the scene information 13a. Further, if the scene information 13a also includes the date of a soccer game as an item, for example, scenes of games held on the same date may be extracted from the scenes included in the scene information 13a. For example, if the scene information 13a also includes, as an item, identification information on an opposing team, scenes from the games played by a given team may be extracted from the scenes included in the scene information 13a).
Regarding claim 4, Masuda in view of Hanamoto disclose everything claimed as applied above (see claim 3), in addition, Masuda in view of Hanamoto disclose wherein the importance for each viewpoint includes an importance based on an application of a camera arranged for each viewpoint (Paragraph 108 teaches that FIG. 16 illustrates, as an example, a case where an angle of 45° is formed by the horizontal plane of the field F and a line connecting the position of the viewpoint VA and the coordinates of the ball in the goal. With the position of the viewpoint VA being thus selected, the direction of the viewpoint is allowed to follow the ball coordinates from the start time to the end time of the scene. In other words, a virtual camera at the viewpoint position VA may be panned toward the ball coordinates at each time point in the scene. In addition, if the 3D coordinates of the ball are obtainable, the virtual camera may be tilted according to the height of the ball).
Regarding claim 5, Masuda in view of Hanamoto disclose everything claimed as applied above (see claim 3), in addition, Masuda in view of Hanamoto disclose wherein the importance for each viewpoint includes an importance based on an imaging target from the viewpoint (Paragraph 43 teaches that the client terminal 10 according to the present embodiment narrows the players down to ones involved with the ball based on the moving path of the ball specified by reference to the tracking information on a given scene, and sets an area of interest to an area containing the movement paths of the ball and the players thus narrowed down to. Provision of such an area of interest as viewpoint selection assistance information helps select, from free viewpoints, a viewpoint whose field of view captures the features of interest in sport spectating. Thus, the client terminal 10 according to the present embodiment helps assist in selecting a scene-suited viewpoint. Additionally, paragraph 75 teaches that the second identifier 15d is a processor that identifies one or more players who are within a predetermined distance from the position of the ball for a predetermined duration of time or longer in a predetermined period of time).
Regarding claim 6, Masuda in view of Hanamoto disclose everything claimed as applied above (see claim 1), in addition, Masuda in view of Hanamoto disclose wherein the circuitry selects the data according to the importance, based on image analysis on the captured image data (Paragraph 38 teaches that for example, the server apparatus 30 analyzes 3D information on a subject based on multiview images captured by the cameras 20. This enables generation of free viewpoint video for soccer spectating. Then, the server apparatus 30 provides the client terminal 10 with information used for generation of free viewpoint video, such as the 3D information and a time series of the multiview images).
Regarding claim 7, Masuda in view of Hanamoto disclose everything claimed as applied above (see claim 1), in addition, Masuda in view of Hanamoto disclose wherein the circuitry determines the importance based on a user operation input (Paragraph 70 teaches that based on the video generation information 13c thus acquired, free viewpoint video is generated for the position of the viewpoint selected by the viewer via the input part 11a and paragraph 63 teaches that in one aspect, the acquirer 15a downloads from the server apparatus 30 the scene information 13a defining scenes in soccer-game spectating to prompt a viewer (a user) to specify a scene to view on the browser of the client terminal 10).
Regarding claim 11, Masuda in view of Hanamoto disclose everything claimed as applied above (see claim 1), in addition, Masuda in view of Hanamoto disclose wherein the circuitry selects the captured image data or the processing data for each frame of the free viewpoint image, as the selection target data to be used to generate the free viewpoint image, according to the importance (FIG. 13 and paragraph 115 teach that the viewpoint selection assistance process 180a reads various pieces of data from the HDD 170, loads them into a memory area in the RAM 180 allocated to the viewpoint selection assistance process 180a, executes various kinds of processing using the various pieces of data thus loaded. An example of processing executed by the viewpoint selection assistance process 180a includes the processing illustrated in FIG. 13. Additionally, paragraph 86 teaches that in one embodiment, when the viewpoint is preset to the initial position V0, the video generator 15f uses the video generation information 13c stored in the storage part 13 to generate, as a preview, a free viewpoint image at the start time of the scene from the initial position V0 and direction of the viewpoint. When instructed to reproduce free viewpoint video via the input part 11a, the video generator 15f generates a free viewpoint image for a new frame every time a certain period of time elapses from the start time of the scene. If the viewpoint is moved on the spherical plane, the video generator 15f generates a free viewpoint image that corresponds to the moved viewpoint. If the viewpoint is not moved, the video generator 15f generates a free viewpoint image that corresponds to the viewpoint being selected. Such reproduction of the free viewpoint video is repeated till the end of the scene.).
Regarding claim 12, Masuda in view of Hanamoto disclose everything claimed as applied above (see claim 1), in addition, Masuda in view of Hanamoto disclose wherein the importance includes an importance related to each viewpoint (Paragraph 64 teaches that for example, in the scene information 13a, each scene is defined by items such as a scene tag, a start time ts, and an end time te. The “scene tag” here indicates a tag that identifies a scene),
and wherein the circuitry selects the captured image data of selected viewpoints as data to be used to generate a free viewpoint image, according to the importance related to each viewpoint (Paragraph 38 teaches that for example, the server apparatus 30 may provide the video generation information for the entire soccer game from the start to the end of the game, or only for a particular scene from the game, such as a goal-scoring scene or a scene of a phenomenal play. In addition to providing such free viewpoint video for a game or a scene from a game, the server apparatus 30 may also provide the client terminal 10 with tracking information, in which the movements of a ball, players, and referees are tracked using the 3D information).
Regarding claim 13, Masuda in view of Hanamoto disclose everything claimed as applied above (see claim 12), in addition, Masuda in view of Hanamoto disclose wherein the circuitry selects the processing data as data to be used to generate the free viewpoint image, for other viewpoints other than the selected viewpoints (Paragraph 101 teaches that in the example given in Embodiment 1 above, a player with the coordinates within a predetermined distance from a point of change is extracted. In addition to this, a player other than such a player may also be extracted. For a goal-scoring scene, for example, the client terminal 10 may refer to the tracking information to extract a defender and/or a goal keeper located within a predetermined distance from the goal line or within the penalty area, and set a video output range which further contains the path(s) of the player(s) thus extracted).
Regarding claim 15, Masuda in view of Hanamoto disclose everything claimed as applied above (see claim 1), in addition, Masuda in view of Hanamoto disclose wherein the circuitry selects the captured image data or the processing data, as the selection target data to be used to generate the free viewpoint image, based on the importance related to the event (Paragraph 74 teaches that in one embodiment, when the specification receiver 15b receives specification of a scene, the first identifier 15c identifies, in the tracking information 13b acquired by the acquirer 15a, a time series of ball coordinates (or the path of the ball) for a period corresponding to the specified scene. FIG. 5 is a diagram illustrating an example of the path of a ball. In FIG. 5, a time series of ball coordinates from the tracking information 13b in FIG. 4 is plotted on the field F. In other words, coordinates Pts of a ball 15 minutes and 00 second after the start of the game, coordinates Pt1 of the ball 15 minutes and 01 second after the start of the game, coordinates Pt2 of the ball 15 minutes and 02 seconds after the start of the game, coordinates Pt3 of the ball 15 minutes and 03 seconds after the start of the game, and coordinates Pte of the ball 15 minutes and 04 seconds after the start of the game are plotted with black circular marks. It is seen in FIG. 5 that, in the scene “Team A Scoring Goal”, the ball moves along a path which offensively progresses on the right side of Team A, is thrown to the center near the right goalpost of Team B, and then is sent into the goal),
and wherein, in a case where the captured image data is selected as the selection target data to be used to generate the free viewpoint image, the circuitry selects the captured image data of which one of the viewpoints as the selection target data to be used to generate the free viewpoint image, based on the importance related to the viewpoint (Paragraph 70 teaches that the video generation information 13c is used for generation of free viewpoint video for the scene specified via the specification receiver 15b. The video generation information 13c includes, for example, 3D information and a time series of multiview images of a subject on the field F. For example, when the scene tag “Team A Scoring Goal” included in the scene information 13a illustrated in FIG. 3 is specified, time series for a period from the start time “15:00” to the end time “15:04” of the scene are acquired. Based on the video generation information 13c thus acquired, free viewpoint video is generated for the position of the viewpoint selected by the viewer via the input part 11a).
Regarding claim 16, Masuda in view of Hanamoto disclose everything claimed as applied above (see claim 1), in addition, Masuda in view of Hanamoto disclose wherein the circuitry is further configured to generate management information so that data selected according to the importance from the selection target data recorded in a single or a plurality of recording media is held in the single or the plurality of recording media in a recorded state (Paragraph 72 teaches that in one embodiment, the specification receiver 15b displays on the display part 11b a scene-list screen containing a list of scenes included in the scene information 13a stored in the storage part 13, and receives specification of a scene made on the scene list screen via the input part 11a. For example, such a list of scenes may include scenes from games held on the same date or scenes from games played by a team selected through the input part 11a or the like. Additionally, paragraph 56 teaches that in one embodiment, the storage part 13 is implemented as an auxiliary storage device of the client terminal 10. For example, the auxiliary storage device includes a hard disk drive (HDD), an optical disk, and a solid-state drive (SSD)).
Regarding claim 17, Masuda in view of Hanamoto disclose everything claimed as applied above (see claim 1), in addition, Masuda in view of Hanamoto disclose wherein the circuitry executes processing of outputting data selected from the selection target data recorded in a single or a plurality of recording media according to the importance, to another or a plurality of other recording media (Paragraph 114 teaches that as illustrated in FIG. 17, the HDD 170 stores therein a viewpoint selection assistance program 170a that implements the same functions as the acquirer 15a, the specification receiver 15b, the first identifier 15c, the second identifier 15d, the setter 15e, and the video generator 15f described in Embodiment 1. Additionally, paragraph 116 teaches to note that the viewpoint selection assistance program 170a does not have to be stored in the HDD 170 and the ROM 160 from the beginning. For example, the viewpoint selection assistance program 170a may be stored in a “portable physical medium” insertable to the computer 100, such as a flexible disk (FD), a CD-ROM, a DVD, a magneto-optical disk, or an IC card. Then, the computer 100 may acquire the viewpoint selection assistance program 170a from the portable physical medium and execute it. Alternatively, the viewpoint selection assistance program 170a may be stored in a different computer or a server apparatus connected to the computer 100 via a public telephone circuit, the Internet, a LAN, a WAN, or the like, and the computer 100 may acquire the viewpoint selection assistance program 170a from the different computer or the server apparatus and execute it).
Regarding claim 19, the method steps correspond to and are rejected similarly to the processing device steps of claim 1 (see claim 1 above).
Regarding claim 20, the program steps correspond to and are rejected similarly to the processing device steps of claim 1 (see claim 1 above). Additionally, Masuda teaches a non-transitory computer readable storage medium having embodied thereon a program, which when executed by a computer device, causes the computer device to execute a method (Paragraph 2 teaches that the embodiments discussed herein are related to a method, an apparatus, and a non-transitory computer-readable storage medium for viewpoint selection assistance in free viewpoint video generation. Additionally, FIG. 17 and paragraph 112 teach that each processing described in the above embodiments may be implemented when a computer such as a personal computer or a work station executes a prepared program. The following describes, using FIG. 17, an example of a computer that executes a viewpoint selection assistance program having the same functions as the above embodiments and paragraph 114 teaches that as illustrated in FIG. 17, the HDD 170 stores therein a viewpoint selection assistance program 170a that implements the same functions as the acquirer 15a, the specification receiver 15b, the first identifier 15c, the second identifier 15d, the setter 15e, and the video generator 15f described in Embodiment 1.).
Claims 8-10, 14 and 18 are rejected under 35 U.S.C. 103 as being unpatentable over Masuda in view of Hanamoto and further in view of Itakura (Pub. No.: US 2021/0136349 A1).
Regarding claim 8, Masuda in view of Hanamoto discloses everything claimed as applied above (see claim 1), however, Masuda in view of Hanamoto fail to disclose wherein the processing data includes silhouette image data of a subject.
Itakura discloses wherein the processing data includes silhouette image data of a subject (Paragraph 31 teaches that FIG. 2B is a diagram explaining the captured image that is obtained in a case where the camera 15 located at the position facing the camera 11 has a viewing angle narrower than that of the camera 11. In this case, by the camera 15, a captured image 28 in which the object is enlarged (that is, the resolution of texture is high) and a silhouette image 29 thereof are obtained. As described above, by setting different viewing angles to the cameras installed so as to face each other with the gaze point being sandwiched in between, it is possible to acquire a texture image whose resolution of the object is high without reducing the accuracy of the three-dimensional shape data on the object to be generated). Since Masuda in view of Hanamoto teach the initial processing device steps related to processing obtained data related to three-dimensional information generation of a subject and Itakura teaches obtaining and including silhouette images to be used in the captured image data of a subject, it would have been obvious to a person having ordinary skill in the art to combine the teachings together so that when obtaining three-dimensional information generation of a subject for target data, a targets silhouette information could also be captured in the process.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Masuda in view of Hanamoto to incorporate the teachings of Itakura, so that the combined features together would allow for additional captured data to be utilized, including target silhouette information, which would improve the accuracy of determining the selection target data.
Regarding claim 9, Masuda in view of Hanamoto disclose everything claimed as applied above (see claim 1), however, Masuda in view of Hanamoto fail to disclose wherein the processing data includes three-dimensional data of a subject generated by visual hull from captured image data of the plurality of viewpoints.
Itakura discloses wherein the processing data includes three-dimensional data of a subject generated by visual hull from captured image data of the plurality of viewpoints (Paragraph 63 teaches that at S510, the virtual viewpoint image generation unit 404 instructs the server 270 to create three-dimensional shape data (shape estimation) of each object included in the multi-viewpoint image acquired at S509. Upon receipt of the instructions, the server 270 cuts out the object area from each captured image and estimates the three-dimensional shape of each object. For the estimation of a three-dimensional shape based on the multi-viewpoint image, it may be possible to use a publicly known method, for example, such as the visual hull method and the stereo matching method). Since Masuda in view of Hanamoto teach the initial processing device steps related to processing obtained data related to three-dimensional information generation of a subject and Itakura teaches utilizing a visual hull method for estimation of three-dimensional data of an object included in multiple images/viewpoints, it would have been obvious to a person having ordinary skill in the art to combine the teachings together so that a known method in the art, such as the visual hull method, could be implemented to improve the overall processing of any of the captured data.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Masuda in view of Hanamoto to incorporate the teachings of Itakura, so that the combined features together would allow for better capturing, tracking and analyzing of target object/human movement, when incorporating a visual hull method for generating the subject from the captured image data of the different viewpoints.
Regarding claim 10, Masuda in view of Hanamoto disclose everything claimed as applied above (see claim 1), however, Masuda in view of Hanamoto fail to disclose wherein the processing data includes polygon mesh data of a subject generated from the captured image data of the plurality of viewpoints.
Itakura discloses wherein the processing data includes polygon mesh data of a subject generated from the captured image data of the plurality of viewpoints (Paragraph 28 teaches that in the present embodiment, three-dimensional shape data is data representing a three-dimensional shape of an object and represented by, for example, a point cloud having positional information on x, y, and z in a three-dimensional space in a world coordinate space uniquely indicating an image capturing-target image capturing space. Three-dimensional shape data is not limited to that represented by a point cloud and may be represented by another data format and may be represented by, for example, polygon mesh data configured by faces of a simple convex polygon, such as a triangle and a rectangle, or voxels). Since Masuda in view of Hanamoto teach the initial processing device steps related to processing obtained data related to three-dimensional information generation of a subject and Itakura teaches incorporating different three-dimensional shape data generation, including the use of polygon mesh data of a subject, it would have been obvious to a person having ordinary skill in the art to combine the teachings together so that additional three-dimensional subject generation data could be used, including those that involve the implementation of polygon mesh data.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Masuda in view of Hanamoto to incorporate the teachings of Itakura, so that the combined features together would allow for polygon mesh data to be used for subject generation which could help improve efficiency and performance when generating the related captured image data.
Regarding claim 14, Masuda in view of Hanamoto discloses everything claimed as applied above (see claim 1), however, Masuda in view of Hanamoto fail to disclose wherein the processing data includes polygon mesh data of a subject generated from the captured image data of the plurality of viewpoints.
Itakura discloses wherein the processing data includes polygon mesh data of a subject generated from the captured image data of the plurality of viewpoints (Paragraph 28 teaches that in the present embodiment, three-dimensional shape data is data representing a three-dimensional shape of an object and represented by, for example, a point cloud having positional information on x, y, and z in a three-dimensional space in a world coordinate space uniquely indicating an image capturing-target image capturing space. Three-dimensional shape data is not limited to that represented by a point cloud and may be represented by another data format and may be represented by, for example, polygon mesh data configured by faces of a simple convex polygon, such as a triangle and a rectangle, or voxels). Since Masuda in view of Hanamoto teach the initial processing device steps related to processing obtained data related to three-dimensional information generation of a subject and Itakura teaches incorporating different three-dimensional shape data generation, including the use of polygon mesh data of a subject, it would have been obvious to a person having ordinary skill in the art to combine the teachings together so that additional three-dimensional subject generation data could be used, including those that involve the implementation of polygon mesh data.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Masuda in view of Hanamoto to incorporate the teachings of Itakura, so that the combined features together would allow for polygon mesh data to be used for subject generation which could help improve efficiency and performance when generating the related captured image data.
Furthermore, Masuda in view of Hanamoto and Itakura disclose and the selection processing unit selects the polygon mesh data as data to be used to generate a free viewpoint image, according to the importance (Paragraph 65 teaches that at S512, the virtual viewpoint image generation unit 404 instructs the server 270 to generate a virtual viewpoint image in accordance with the virtual viewpoint that is set at S511. Upon receipt of the instructions, the server 270 generates a virtual viewpoint image by a method, such as publicly known viewpoint-dependent rendering, by using the multi-viewpoint image acquired at S509 and the three-dimensional shape data created at S510. The generated virtual viewpoint image is output to the monitor 308 and it is possible for a user to browse the virtual viewpoint image from an arbitrary viewpoint that is set by the user him/herself).
Regarding claim 18, Masuda in view of Hanamoto disclose everything claimed as applied above (see claim 1), however, Masuda in view of Hanamoto fail to disclose wherein the selection processing unit converts a resolution or a frame rate according to the importance, for at least one of the captured image data or the processing data.
Itakura discloses wherein the selection processing unit converts a resolution or a frame rate according to the importance, for at least one of the captured image data or the processing data (Paragraph 41 teaches that the image capturing modules 110a to 110g are connected in a daisy chain. By this connection form, there is an effect that it is possible to reduce the number of connection cables and save the wiring work in a case where the amount of image data increases accompanying an increase in resolution of a captured image to 4K or 8K and an increase in the frame rate). Since Masuda in view of Hanamoto teach the initial processing device steps related to selecting and processing captured image data of multiple viewpoints and subjects and Itakura teaches the ability to increase and change the resolution or frame rate of the captured data from multiple viewpoints, it would have been obvious to a person having ordinary skill in the art to combine the teachings together so that any of the captured data related to the multiple different viewpoints, could then be viewed through different resolutions or frame rates, by making adjustments to them, if necessary.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Masuda in view of Hanamoto to incorporate the teachings of Itakura, so that the combined features together would allow for more efficient generation of the multiple viewpoints being able to change and optimize the frame rate or resolution of the outputted captured images/data.
Response to Arguments
Applicant’s arguments, see Remarks pages 1-15, filed November 19th, 2025, with respect to the rejections of claims 1-7, 11-13, 15-17, 19 and 20 under 35 U.S.C. 102(a)(1) have been fully considered and are persuasive. Therefore, the rejection has been withdrawn. However, upon further consideration, a new ground(s) of rejection is made in view of newly applied prior art reference of Hanamoto and therefore, the newly amended claim language is rejected under 35 U.S.C. 103 (see respectively, claims 1, 19 and 20 above for reasoning).
In regards to the additional arguments regarding any of the dependent claims 2-18, for the virtue of their dependency are moot because the independent claims are not allowable.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Morisawa et al. (Pub. No.: US 2020/0336719 A1) teaches an information processing apparatus for a system that generates a virtual viewpoint image based on image data obtained by performing imaging from a plurality of directions using a plurality of cameras.
Maeda (Pub. No.: US 2020/0068188 A1) teaches an image processing system having a plurality of image capturing units, which aims to generate a wide-area image from a high-quality virtual viewpoint.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to George Renze whose telephone number is (703)756-5811. The examiner can normally be reached Monday-Friday 9:00am - 6:00pm EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Xiao Wu can be reached at (571) 272-7761. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/G.R./Examiner, Art Unit 2613
/XIAO M WU/Supervisory Patent Examiner, Art Unit 2613