DETAILED ACTION
Notice of Pre-AIA or AIA Status
1. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
2. Receipt is acknowledged of certified copies of papers required by 37 CFR 1.55.
Response to Amendment
3. The amendment filed March 20, 2026 has been entered. Claims 1-11, 13, 15-18, and 20-22 remain pending in the application.
Response to Arguments
4. Applicant's arguments filed March 20, 2026 have been fully considered but they are not persuasive.
5. Applicant argues that Arai (U.S. Patent Application Publication No. 2019/0289275 A1), Lyren (U.S. Patent Application Publication No. 2020035546 A1), and Kanzaki et al. (U.S. Patent Application Publication No. 2021/0072940 A1), hereinafter referred to as Kanzaki, do not teach "obtaining the event information corresponding to an event that occurs in a predetermined time period after a reproducing time in order to operate the virtual viewpoint". The Applicant further argues that Arai's event information is obtained to generate a virtual viewpoint image and not for operating the virtual viewpoint.
Examiner replies that Arai teaches in Paragraph 62, Figure 9 step S903 determining and collecting event information that occurs within a predetermined time period from the first time.
Arai also teaches in Paragraph 39-41 that the event information includes position information which is used to determine a gazing point using the event information and its position information. Arai in Paragraph 72 then teaches that the virtual viewpoint is determined based on the determined gazing point and Paragraph 74 teaches the apparatus may “generate and output a virtual viewpoint image based on the determined gazing point and virtual viewpoint”. Thus, the event information is used to generate a virtual viewpoint image of an event occurring in a predetermined time period after a first time based on the position acquired in the event information.
Arai Paragraph 51 also teaches generating multiple virtual viewpoint images which assigns a virtual viewpoint direction depending on the gazing point data generated. This teaches changing a position of the virtual viewpoint. For each virtual viewpoint image generated the virtual viewpoint position can change depending on the gazing point data which depends on the event information. Thus, Arai teaches that the event information has information for changing the position of the virtual viewpoint to generate and operate the virtual viewpoint image.
In addition, in response to applicant's argument that the references fail to show certain features of the invention, it is noted that the features upon which applicant relies (i.e., a predetermined time period after a reproducing time) are not recited in the rejected claim(s). The claim language states “a predetermined period of time after a first time” instead. Although the claims are interpreted in light of the specification, limitations from the specification are not read into the claims. See In re Van Geuns, 988 F.2d 1181, 26 USPQ2d 1057 (Fed. Cir. 1993).
6. Conclusion: The rejections set in the previous Office Action are shown to have been proper, and the claims are rejected below. New citations and parenthetical remarks can be considered new grounds of rejection and such new grounds of rejection are necessitated by the Applicant’s amendments to the claims. Therefore, the present Office Action is made final.
Claim Objections
7. Claim 1 objected to because of the following informalities: Line 9 and 14 "and/or" should be only "and" or "or".
Claim 18 objected to because of the following informalities: Line
Claim 20 objected to because of the following informalities: Line
Claim 21 objected to because of the following informalities: Line
Claim 22 objected to because of the following informalities: Line 9 and 14 "and/or" should be
Appropriate correction is required.
Claim Rejections - 35 USC § 103
8. In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
9. The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
10. Claim(s) 1-4, 6-11, 15-16, 18, and 20-22 is/are rejected under 35 U.S.C. 103 as being unpatentable over Arai (U.S. Patent Application Publication No. 2019/0289275 A1) in view of Lyren (U.S. Patent Application Publication No. 2020035546 A1) and Kanzaki et al. (U.S. Patent Application Publication No. 2021/0072940 A1), hereinafter referred to as Kanzaki.
11. Regarding claim 1, Arai teaches an image processing apparatus comprising: one or more memories storing instructions; and one or more processors executing the instructions to (Paragraph 78 teaches an apparatus that has a CPU to execute instructions stored on a storage medium or memory): obtain a position of a virtual viewpoint corresponding to a virtual viewpoint image to be generated based on images captured by a plurality of imaging apparatuses (Paragraph 5 teaches viewpoint information that contains the position and direction of the virtual viewpoint is acquired. Then, a virtual viewpoint image is generated based off the virtual viewpoint information and images captured by one or more sets of cameras or imaging apparatuses),
a line-of-sight direction from the virtual viewpoint (Paragraph 5 teaches the viewpoint information has the direction of the virtual viewpoint which is the line-of-sight direction),
and event information about an event occurring at a second time included within a predetermined period of time after a first time (Paragraph 62, Figure 9 step S903 teaches determining and collecting event information that occurs within a predetermined time period from the first time; Paragraph 39, Figure 5 teaches the event information includes the position information of the event),
wherein the event information indicates an occurrence position of the event (Paragraph 39, Figure 5 teach event information is obtained that has information on the position of an event and when the event has occurred) and is obtained for changing the position of the virtual viewpoint and/or the line-of-sight direction from the virtual viewpoint (Paragraph 39 teaches the event information includes position information. Paragraph 41 then teaches determining a gazing point using the event information which has position information; Paragraph 72 teaches the virtual viewpoint is determined based on the gazing point and Paragraph 74 teaches the apparatus may “generate and output a virtual viewpoint image based on the determined gazing point and virtual viewpoint”. Thus, the event information is used to generate a virtual viewpoint image based on the position acquired in the event information; Paragraph 51 teaches generating multiple virtual viewpoint images which assigns a virtual viewpoint direction depending on the gazing point data generated. This teaches changing a position of the virtual viewpoint. For each virtual viewpoint image generated the virtual viewpoint position can change depending on the gazing point data which depends on the event information. Thus, the event information has information for changing the position of the virtual viewpoint to generate the virtual viewpoint images);
However, Arai fails to teach generating the virtual viewpoint image corresponding to the first time, and superimpose and display an image indicating the event information; and display, based on the event information, operation assistance information indicating how to change the position of the virtual viewpoint and/or the line-of-sight direction from the virtual viewpoint such that the occurrence position of the event is included within an angle of view of the virtual viewpoint.
Lyren teaches generating the virtual viewpoint image corresponding to the first time, and superimpose and display an image indicating the event information; and display, based on the event information, operation assistance information indicating how to change the position of the virtual viewpoint and/or the line-of-sight direction from the virtual viewpoint (Paragraph 247, Figure 14A teaches a superimposed image 1438 displayed on the virtual viewpoint image 1432. The superimposed image 1438 has event information including action information, occurrence position of the event, and time information. The time information is represented through the countdown which is a difference between the current time, or first time, of the frame and the second time, or time when the event occurs. The event information also has a path 1436 superimposed on the virtual viewpoint telling the user where to move to. The path can be considered operation assistance information pointing towards the occurrence position of the event which will indicate how to change the position of the virtual viewpoint. The user moving will adjust the virtual viewpoint. This can be considered to guidance for adjustment of the virtual viewpoint because it indicates the occurrence position of the event and the countdown to the event; Paragraphs 251-253 also teach the display or virtual viewpoint can be altered in a way such as sound, color, highlights, arrows, and more to show users where or when to perform an action or the future event. The path 1436 and countdown along with the colors and arrows are operation assistance information that provides guidance on the adjustment of the virtual viewpoint based on the occurrence position of the event).
Arai is analogous to the claimed invention because it teaches an apparatus to generate virtual viewpoints for event viewing. Lyren is analogous to the claimed invention because it teaches analyzing information in a viewpoint and displaying an image with event information to a user. Thus, it would have been obvious to a person holding ordinary skill in the art before the effective filing date to modify the image processing apparatus to generate a virtual viewpoint image taught by Arai with the superimposed event information image displayed to user as taught by Lyren in order to assist a user with seeing and determining the movement or activity of a target in the area (Lyren Paragraph 37).
However, Arai and Lyren are not relied upon for the below claim language: operation assistance information indicating how to change the position of the virtual viewpoint and/or the line-of-sight direction from the virtual viewpoint such that the occurrence position of the event is included within an angle of view of the virtual viewpoint.
Kanzaki teaches operation assistance information indicating how to change the position of the virtual viewpoint and/or the line-of-sight direction from the virtual viewpoint such that the occurrenceposition of the event is included within an angle of view of the virtual viewpoint (Paragraph 69 and 71, Figure 10 step S109 teaches that an arrow icon is used when the event is located outside the angle of view of the virtual viewpoint. The arrow icon is an image indicating operation assistance; Paragraph 20, Figure 9 teaches the notification or event icon has an arrow attached pointing to the direction in which the position of the event is which indicates operation assistance information indicating how to change the position of the virtual viewpoint. The arrow also represents the occurrence position of the event and is present in the angle of view of the virtual viewpoint or user to let them know where it is).
Arai is analogous to the claimed invention because it teaches an apparatus to generate virtual viewpoints for event viewing. Lyren and Kanzaki are analogous to the claimed invention because they teach analyzing information in a viewpoint and displaying an image with event information to a user. Thus, it would have been obvious to a person holding ordinary skill in the art before the effective filing date to modify the image processing apparatus that generates virtual viewpoint images as taught by Arai and Lyren with the operation assistance information indicating how to change the virtual viewpoint to include the event within an angle of view of the virtual viewpoint taught by Kanzaki in order to provide guidance on where the position of the target or event is to the user (Kanzaki Paragraph 28).
12. Regarding claim 2, Arai in view of Lyren and Kanzaki teaches the limitations of claim 1. Arai further teaches the image processing apparatus wherein the event information includes information indicating the second time (Paragraph 39, Figure 5 teaches the obtained event information has information on when the event occurs. The time when the event occurs can be considered the second time).
13. Regarding claim 3, Arai in view of Lyren and Kanzaki teaches the limitations of claim 2. However, Arai fails to teach the image processing apparatus wherein the event information includes information indicating a time period before the event occurs, the time period being a difference between the first time and the second time.
Lyren teaches the image processing apparatus wherein the event information includes information indicating a time period before the event occurs, the time period being a difference between the first time and the second time (Paragraph 247, Figure 14A teaches a superimposed image 1438 displayed on the virtual viewpoint image 1432. The superimposed event information image 1438 time information included. The time information is represented by the countdown which is a difference between the current time, or first time, of the frame and the second time, or time when the event occurs).
Arai is analogous to the claimed invention because it teaches an apparatus to generate virtual viewpoints for event viewing. Lyren and Kanzaki are analogous to the claimed invention because it teaches analyzing information in a viewpoint and displaying an image with event information to a user. Thus, it would have been obvious to a person holding ordinary skill in the art before the effective filing date to modify the image processing apparatus to generate a virtual viewpoint image taught by Arai in view of Kanzaki with the superimposed event information image with time information taught by Lyren in order to assist a user with seeing and determining the movement or activity of a target in the area (Lyren Paragraph 37).
14. Regarding claim 4, Arai in view of Lyren and Kanzaki teaches the limitations of claim 3. However, Arai fails to teach the image processing apparatus wherein the image indicating the event information includes a number indicating the time period before the event occurs, and a three-dimensional virtual object or an icon indicating the occurrence position of the event.
Lyren teaches the image processing apparatus wherein the image indicating the event information includes a number indicating the time period before the event occurs, and a three-dimensional virtual object or an icon indicating the occurrence position of the event (Paragraph 247, Figure 14A teaches a superimposed image 1438 displayed on the virtual viewpoint image 1432. The superimposed image 1438 has event information including a countdown number which indicates the time period before the event will occur; Paragraph 54-55 mentions the recommendation can be displayed as a visual image, text, audio, and other mediums as seen in Figure 14A, marker 1438. The visual image can be considered an icon).
Arai is analogous to the claimed invention because it teaches an apparatus to generate virtual viewpoints for event viewing. Lyren and Kanzaki are analogous to the claimed invention because it teaches analyzing information in a viewpoint and displaying an image with event information to a user. Thus, it would have been obvious to a person holding ordinary skill in the art before the effective filing date to modify the image processing apparatus to generate a virtual viewpoint image taught by Arai in view of Kanzaki with the superimposed event information image with time information taught by Lyren in order to assist a user with seeing and determining the movement or activity of a target in the area (Lyren Paragraph 37).
15. Regarding claim 6, Arai in view of Lyren and Kanzaki teaches the limitations of claim 1. However, Arai fails to teach the image processing apparatus wherein the virtual viewpoint image is associated with a first time code, wherein the event information is associated with a second time code, wherein the first time code is a code indicating the first time, and wherein the second time code is a code indicating the second time.
Lyren teaches the image processing apparatus wherein the virtual viewpoint image is associated with a first time code, wherein the event information is associated with a second time code, wherein the first time code is a code indicating the first time, and wherein the second time code is a code indicating the second time (Paragraph 247, Figure 14A teaches a superimposed image 1438 displayed on the virtual viewpoint image 1432. The superimposed image 1438 has event information including a countdown number which indicates the time period before the event will occur and inherently includes two times, first time code and second time code, to countdown and calculate the difference to. The first time code is the first time which is the current frame. The second time code is the second time which is when the event will occur and is what the countdown is counting down to).
Arai is analogous to the claimed invention because it teaches an apparatus to generate virtual viewpoints for event viewing. Lyren and Kanzaki are analogous to the claimed invention because it teaches analyzing information in a viewpoint and displaying an image with event information to a user. Thus, it would have been obvious to a person holding ordinary skill in the art before the effective filing date to modify the image processing apparatus to generate a virtual viewpoint image taught by Arai in view of Kanzaki with the superimposed event information image with time information taught by Lyren in order to assist a user with seeing and determining the movement or activity of a target in the area (Lyren Paragraph 37).
16. Regarding claim 7, Arai in view of Lyren and Kanzaki teaches the limitations of claim 1. Arai further teaches the image processing apparatus wherein the event information includes information indicating a name of the event (Paragraph 39, Figure 5 teaches the event information is stored as a data structure and has an event name in text attached to each event),
However, Arai fails to teach the image processing apparatus wherein the image indicating the event information includes text indicating the name of the event.
Lyren teaches the image processing apparatus wherein the image indicating the event information includes text indicating the name of the event (Paragraph 247, Figure 14A teaches a superimposed image 1438 displayed on the virtual viewpoint image 1432. The superimposed image 1438 has event information like the name of the event written in text in the image, shown in Figure 14A marker 1438).
Arai is analogous to the claimed invention because it teaches an apparatus to generate virtual viewpoints for event viewing. Lyren and Kanzaki are analogous to the claimed invention because it teaches analyzing information in a viewpoint and displaying an image with event information to a user. Thus, it would have been obvious to a person holding ordinary skill in the art before the effective filing date to modify the image processing apparatus to generate a virtual viewpoint image taught by Arai in view of Kanzaki with the superimposed event information image with name information taught by Lyren in order to assist a user with seeing and determining the movement or activity of a target in the area (Lyren Paragraph 37).
17. Regarding claim 8, Arai in view of Lyren and Kanzaki teaches the limitations of claim 1. Arai teaches the image processing apparatus wherein the event information includes information indicating a subject related to the event (Paragraph 39, Figure 5 teaches the event information is stored as a data structure and has an event name in text attached to each event),
However, Arai fails to teach the image processing apparatus wherein the image indicating the event information includes at least one of text, an icon, and an effect for identifying the subject related to the event.
Lyren teaches the image processing apparatus wherein the image indicating the event information includes at least one of text, an icon, and an effect for identifying the subject related to the event (Paragraph 247, Figure 14A teaches a superimposed image 1438 displayed on the virtual viewpoint image 1432. The superimposed image 1438 has event information like the name of the event written in text in the image, shown in Figure 14A marker 1438. The name of the event can be considered to be the subject of the vent as well).
Arai is analogous to the claimed invention because it teaches an apparatus to generate virtual viewpoints for event viewing. Lyren and Kanzaki are analogous to the claimed invention because it teaches analyzing information in a viewpoint and displaying an image with event information to a user. Thus, it would have been obvious to a person holding ordinary skill in the art before the effective filing date to modify the image processing apparatus to generate a virtual viewpoint image taught by Arai in view of Kanzaki with the superimposed event information image with name information taught by Lyren in order to assist a user with seeing and determining the movement or activity of a target in the area (Lyren Paragraph 37).
18. Regarding claim 9, Arai in view of Lyren and Kanzaki teaches the limitations of claim 8. Arai further teaches wherein the subject related to the event includes a plurality of subjects related to the event (Paragraph 39 teaches a plurality of objects like a person or ball are involved with the event. The event name can be considered the subject, for example foul, and the person and ball can be considered the plurality of subjects related to the event).
19. Regarding claim 10, Arai in view of Lyren and Kanzaki teaches the limitations of claim 1. However, Arai and Lyren fail to teach the image processing apparatus wherein the one or more processors further execute the instructions to change, based on whether the occurrence position of the event is within an angle of view of the virtual viewpoint, the image indicating the event information to be superimposed and displayed on the virtual viewpoint image.
Kanzaki teaches the image processing apparatus wherein the one or more processors further execute the instructions to change, based on whether the occurrence position of the event is within an angle of view of the virtual viewpoint, the image indicating the event information to be superimposed and displayed on the virtual viewpoint image (Paragraph 69 and 71, Figure 10 step S109 teach that an arrow icon is used to indicate the direction where the event is located when it is outside the field of view. Paragraph 70, Figure 10 step S108 teaches that when the event is in view, only the notification details at the position of the event is shown. Thus, the image indicating event information changes depending on whether the position of the event is in the angle of view of the virtual viewpoint or out of it; Paragraph 20, Figure 9 shows the arrow icon used for the event information when the position is outside the angle of view and Paragraph 18, Figure 7 shows the event information displayed when the event is in the angle of view).
Arai is analogous to the claimed invention because it teaches an apparatus to generate virtual viewpoints for event viewing. Lyren and Kanzaki are analogous to the claimed invention because they teach analyzing information in a viewpoint and displaying an image with event information to a user. Thus, it would have been obvious to a person holding ordinary skill in the art before the effective filing date to modify the image processing apparatus that generates virtual viewpoint images as taught by Arai and Lyren with the changing the event information image depending on the angle of view taught by Kanzaki in order to provide guidance on where the position of the target or event is to the user (Kanzaki Paragraph 28).
20. Regarding claim 11, Arai in view of Lyren and Kanzaki teaches the limitations of claim 1. However, Arai and Lyren fail to teach the image processing apparatus wherein, in a case where the occurrence position of the event is within the angle of view of the virtual viewpoint, the virtual viewpoint image includes a three-dimensional virtual object or an icon at the occurrence position of the event, and wherein, in a case where the occurrence position of the event is not within the angle of view of the virtual viewpoint, the virtual viewpoint image includes a three-dimensional virtual object or an icon indicating a direction in which the occurrence position of the event is located.
Kanzaki teaches the image processing apparatus wherein, in a case where the occurrence position of the event is within the angle of view of the virtual viewpoint, the virtual viewpoint image includes a three-dimensional virtual object or an icon at the occurrence position of the event (Paragraph 70, Figure 10 step S108 teaches that when the event is within the angle of view of the virtual viewpoint, the notification or event details at the position of the event is shown; Paragraph 18, Figure 7 shows the event information displayed as an icon at the location of the event), and wherein, in a case where the occurrence position of the event is not within the angle of view of the virtual viewpoint, the virtual viewpoint image includes a three-dimensional virtual object or an icon indicating a direction in which the occurrence position of the event is located (Paragraph 69 and 71, Figure 10 step S109 teaches that an arrow icon is used when the event is located outside the angle of view of the virtual viewpoint; Paragraph 20, Figure 9 shows the notification or event icon has an arrow attached pointing to the direction in which the position of the event is).
Arai is analogous to the claimed invention because it teaches an apparatus to generate virtual viewpoints for event viewing. Lyren and Kanzaki are analogous to the claimed invention because they teach analyzing information in a viewpoint and displaying an image with event information to a user. Thus, it would have been obvious to a person holding ordinary skill in the art before the effective filing date to modify the image processing apparatus that generates virtual viewpoint images as taught by Arai and Lyren with the changing the event information image depending on the angle of view taught by Kanzaki in order to provide guidance on where the position of the target or event is to the user (Kanzaki Paragraph 28).
21. Regarding claim 15, Arai in view of Lyren and Kanzaki teaches the limitations of claim 1. Arai further teaches the image processing apparatus wherein the event is an event related to a sport or a concert (Paragraph 39 teaches the event information is related to an event caused by a player in a sport like soccer).
22. Regarding claim 16, Arai in view of Lyren and Kanzaki teaches the limitations of claim 1. Arai further teaches wherein the event information is obtained from the images captured by the plurality of imaging apparatuses, a global positioning system (GPS) sensor, or an audio recording apparatus (Paragraph 40 teaches the information processing apparatus uses a plurality of cameras, which are a plurality of imaging apparatuses, to acquire an image and generate event information).
23. Regarding claim 18, Arai teaches an image processing apparatus comprising: one or more memories storing instructions; and one or more processors executing the instructions to (Paragraph 78 teaches an apparatus that has a CPU to execute instructions stored on a storage medium or memory): input viewpoint information indicating a position of a virtual viewpoint corresponding to a virtual viewpoint image to be generated based on images captured by a plurality of imaging apparatuses and a line-of-sight direction from the virtual viewpoint (Paragraph 5 teaches viewpoint information that contains the position and direction of the virtual viewpoint is acquired. Then, a virtual viewpoint image is generated based off the virtual viewpoint information and images captured by one or more sets of cameras or imaging apparatuses);
and is obtained for changing the position of the virtual viewpoint and/or the line-of-sight direction from the virtual viewpoint (Paragraph 39 teaches the event information includes position information. Paragraph 41 then teaches determining a gazing point using the event information which has position information; Paragraph 72 teaches the virtual viewpoint is determined based on the gazing point and Paragraph 74 teaches the apparatus may “generate and output a virtual viewpoint image based on the determined gazing point and virtual viewpoint”. Thus, the event information is used to generate a virtual viewpoint image based on the position acquired in the event information; Paragraph 51 teaches generating multiple virtual viewpoint images which assigns a virtual viewpoint direction depending on the gazing point data generated. This teaches changing a position of the virtual viewpoint. For each virtual viewpoint image generated the virtual viewpoint position can change depending on the gazing point data which depends on the event information. Thus, the event information has information for changing the position of the virtual viewpoint to generate the virtual viewpoint images);
However, Arai fails to teach displaying event information indicating an occurrence position of an event occurring at a time included within a predetermined period of time after a time corresponding to the virtual viewpoint image, and indicating time information about a time period before the event occurs; generating the virtual viewpoint image corresponding to the first time, and superimpose and display an image indicating the event information; and displaying, based on the event information, operation assistance information indicating how to change the position of the virtual viewpoint and/or the line-of-sight direction from the virtual viewpoint such that the occurrence position of the event is included within an angle of view of the virtual viewpoint.
Lyren teaches displaying event information indicating an occurrence position of an event occurring at a time included within a predetermined period of time after a time corresponding to the virtual viewpoint image, and indicating time information about a time period before the event occurs (Paragraph 247, Figure 14A teaches a superimposed image 1438 displayed on the virtual viewpoint image 1432. The superimposed image 1438 has event information including action information, occurrence position of the event, and time information. The time information has a countdown which indicates the remaining time period before the event occurs which indicates a predetermined time period where the event occurs at a time later than the time corresponding to the current virtual viewpoint image); generating the virtual viewpoint image corresponding to the first time, and superimpose and display an image indicating the event information; and displaying, based on the event information, operation assistance information indicating how to change the position of the virtual viewpoint and/or the line-of-sight direction from the virtual viewpoint (Paragraph 247, Figure 14A teaches a superimposed image 1438 displayed on the virtual viewpoint image 1432. The superimposed image 1438 has event information including action information, occurrence position of the event, and time information. The time information is represented through the countdown which is a difference between the current time, or first time, of the frame and the second time, or time when the event occurs. The event information also has a path 1436 superimposed on the virtual viewpoint telling the user where to move to. The path can be considered operation assistance information pointing towards the occurrence position of the event which will indicate how to change the position of the virtual viewpoint. The user moving will adjust the virtual viewpoint. This can be considered to guidance for adjustment of the virtual viewpoint because it indicates the occurrence position of the event and the countdown to the event; Paragraphs 251-253 also teach the display or virtual viewpoint can be altered in a way such as sound, color, highlights, arrows, and more to show users where or when to perform an action or the future event. The path 1436 and countdown along with the colors and arrows are operation assistance information that provides guidance on the adjustment of the virtual viewpoint based on the occurrence position of the event).
Arai is analogous to the claimed invention because it teaches an apparatus to generate virtual viewpoints for event viewing. Lyren is analogous to the claimed invention because it teaches analyzing information in a viewpoint and displaying an image with event information to a user. Thus, it would have been obvious to a person holding ordinary skill in the art before the effective filing date to modify the image processing apparatus to generate a virtual viewpoint image taught by Arai with the superimposed event information image displayed to user as taught by Lyren in order to assist a user with seeing and determining the movement or activity of a target in the area (Lyren Paragraph 37).
However, Arai and Lyren are not relied upon for the below claim language: operation assistance information indicating how to change the position of the virtual viewpoint and/or the line-of-sight direction from the virtual viewpoint such that the occurrence position of the event is included within an angle of view of the virtual viewpoint.
Kanzaki teaches operation assistance information indicating how to change the position of the virtual viewpoint and/or the line-of-sight direction from the virtual viewpoint such that the occurrence (Paragraph 69 and 71, Figure 10 step S109 teaches that an arrow icon is used when the event is located outside the angle of view of the virtual viewpoint. The arrow icon is an image indicating operation assistance; Paragraph 20, Figure 9 teaches the notification or event icon has an arrow attached pointing to the direction in which the position of the event is which indicates operation assistance information indicating how to change the position of the virtual viewpoint. The arrow also represents the occurrence position of the event and is present in the angle of view of the virtual viewpoint or user to let them know where it is).
Arai is analogous to the claimed invention because it teaches an apparatus to generate virtual viewpoints for event viewing. Lyren and Kanzaki are analogous to the claimed invention because they teach analyzing information in a viewpoint and displaying an image with event information to a user. Thus, it would have been obvious to a person holding ordinary skill in the art before the effective filing date to modify the image processing apparatus that generates virtual viewpoint images as taught by Arai and Lyren with the operation assistance information indicating how to change the virtual viewpoint to include the event within an angle of view of the virtual viewpoint taught by Kanzaki in order to provide guidance on where the position of the target or event is to the user (Kanzaki Paragraph 28).
24. Regarding claim 20, Arai teaches an image processing system comprising: one or more memories storing instructions; and one or more processors executing the instructions to (Paragraph 78 teaches an apparatus that has a CPU to execute instructions stored on a storage medium or memory):
transmit viewpoint information indicating a position of a virtual viewpoint corresponding to a virtual viewpoint image to be generated based on images captured by a plurality of imaging apparatuses and a line-of-sight direction from the virtual viewpoint (Paragraph 5 teaches viewpoint information that contains the position and direction of the virtual viewpoint is acquired. Then, a virtual viewpoint image is generated based off the virtual viewpoint information and images captured by one or more sets of cameras or imaging apparatuses; Paragraph 26 teaches transmitting information indicating the virtual viewpoint information to the information processing apparatus. Then the information processing apparatus and image generation apparatus generate the virtual viewpoint image corresponding to the virtual viewpoint information transmitted and using the plurality of camera photographs; Paragraph 28 teaches transmitting viewpoint information which includes the position and direction of the viewpoint. The direction is the line-of-sight direction from the viewpoint);
and receive event information indicating an occurrence position of an event (Paragraph 38-39, Figure 5 teaches acquiring event information which includes an occurrence position of an event.) (Paragraph 62, Figure 9 step S903 teaches determining and collecting event information that occurs within a predetermined time period from the first time; Paragraph 39, Figure 5 teaches the event information includes the position information of the event),
(Paragraph 39 teaches the event information includes position information. Paragraph 41 then teaches determining a gazing point using the event information which has position information; Paragraph 72 teaches the virtual viewpoint is determined based on the gazing point and Paragraph 74 teaches the apparatus may “generate and output a virtual viewpoint image based on the determined gazing point and virtual viewpoint”. Thus, the event information is used to generate a virtual viewpoint image based on the position acquired in the event information; Paragraph 51 teaches generating multiple virtual viewpoint images which assigns a virtual viewpoint direction depending on the gazing point data generated. This teaches changing a position of the virtual viewpoint. For each virtual viewpoint image generated the virtual viewpoint position can change depending on the gazing point data which depends on the event information. Thus, the event information has information for changing the position of the virtual viewpoint to generate the virtual viewpoint images);
However, Arai fails to teach indicating time information about a time period before the event occurs, and generating the virtual viewpoint image corresponding to the first time, and superimpose and display an image indicating the event information, and displaying, based on the event information, operation assistance information indicating how to change the position of the virtual viewpoint and/or the line-of-sight direction from the virtual viewpoint such that the occurrence position of the event is included within an angle of view of the virtual viewpoint.
Lyren teaches receiving event information indicating an occurrence position of an event occurring at a time included within a predetermined period of time after a time corresponding to the virtual viewpoint image, and indicating time information about a time period before the event occurs (Paragraph 247, Figure 14A teaches a superimposed image 1438 displayed on the virtual viewpoint image 1432. The superimposed image 1438 has event information including the position of the event and a countdown which indicates the time information for a time period before an event occurs. The countdown which counts down to the time the event occurs at inherently means the event occurs at a predetermined time period later than the current frame or current virtual viewpoint image),
and generating the virtual viewpoint image corresponding to the first time, and superimpose and display an image indicating the event information, and displaying, based on the event information, operation assistance information indicating how to change the position of the virtual viewpoint and/or the line-of-sight direction from the virtual viewpoint (Paragraph 247, Figure 14A teaches a superimposed image 1438 displayed on the virtual viewpoint image 1432. The superimposed image 1438 has event information including action information, occurrence position of the event, and time information. The time information is represented through the countdown which is a difference between the current time, or first time, of the frame and the second time, or time when the event occurs. The event information also has a path 1436 superimposed on the virtual viewpoint telling the user where to move to. The path can be considered a directional indicator pointing towards the occurrence position of the event. The user moving will adjust the virtual viewpoint. This can be considered to guidance for adjustment of the virtual viewpoint because it indicates the occurrence position of the event and the countdown to the event; Paragraphs 251-253 also teach the display or virtual viewpoint can be altered in a way such as sound, color, highlights, arrows, and more to show users where or when to perform an action or the future event. The path 1436 and countdown along with the colors and arrows which are directional indicators can be considered to provide guidance on the adjustment of the virtual viewpoint based on the occurrence position of the event).
Arai is analogous to the claimed invention because it teaches an apparatus to generate virtual viewpoints for event viewing. Lyren is analogous to the claimed invention because it teaches analyzing information in a viewpoint and displaying an image with event information to a user. Thus, it would have been obvious to a person holding ordinary skill in the art before the effective filing date to modify the image processing apparatus to generate a virtual viewpoint image taught by Arai with the superimposed event information image displayed to user as taught by Lyren in order to assist a user with seeing and determining the movement or activity of a target in the area (Lyren Paragraph 37).
However, Arai and Lyren are not relied upon for the below claim language: operation assistance information indicating how to change the position of the virtual viewpoint and/or the line-of-sight direction from the virtual viewpoint such that the occurrence position of the event is included within an angle of view of the virtual viewpoint.
Kanzaki teaches operation assistance information indicating how to change the position of the virtual viewpoint and/or the line-of-sight direction from the virtual viewpoint such that the occurrence (Paragraph 69 and 71, Figure 10 step S109 teaches that an arrow icon is used when the event is located outside the angle of view of the virtual viewpoint. The arrow icon is an image indicating operation assistance; Paragraph 20, Figure 9 teaches the notification or event icon has an arrow attached pointing to the direction in which the position of the event is which indicates operation assistance information indicating how to change the position of the virtual viewpoint. The arrow also represents the occurrence position of the event and is present in the angle of view of the virtual viewpoint or user to let them know where it is).
Arai is analogous to the claimed invention because it teaches an apparatus to generate virtual viewpoints for event viewing. Lyren and Kanzaki are analogous to the claimed invention because they teach analyzing information in a viewpoint and displaying an image with event information to a user. Thus, it would have been obvious to a person holding ordinary skill in the art before the effective filing date to modify the image processing system that generates virtual viewpoint images as taught by Arai and Lyren with the operation assistance information indicating how to change the virtual viewpoint to include the event within an angle of view of the virtual viewpoint taught by Kanzaki in order to provide guidance on where the position of the target or event is to the user (Kanzaki Paragraph 28).
25. Regarding claim 21, claim 21 is the method claim of apparatus claim 1 and is accordingly rejected using substantially similar rationale as to that which is set for with respect to claim 1.
26. Regarding claim 22, claim 22 is the non-transitory computer-readable storage medium storing a computer program (Arai Paragraph 78 teaches an apparatus that has a CPU to execute instructions stored on a storage medium or memory) claim of apparatus claim 1 and is accordingly rejected using substantially similar rationale as to that which is set for with respect to claim 1.
27. Claim(s) 5 is/are rejected under 35 U.S.C. 103 as being unpatentable over Arai (U.S. Patent Application Publication No. 2019/0289275 A1) in view of Lyren (U.S. Patent Application Publication No. 2020035546 A1) and Kanzaki et al. (U.S. Patent Application Publication No. 2021/0072940 A1), hereinafter referred to as Kanzaki, as applied to claim 3 above, and further in view of Ueno (U.S. Patent Application Publication No. 2020/0162677 A1).
Regarding claim 5, Arai in view of Lyren and Kanzaki teaches the limitations of claim 3. However, Arai and Lyren fail to teach the image processing apparatus wherein, in the image indicating the event information displayed on the virtual viewpoint image, at least one of a color, a size, and a blinking period of the event information is changed depending on the time period before the event occur.
Ueno teaches the image processing apparatus wherein, in the image indicating the event information displayed on the virtual viewpoint image, at least one of a color, a size, and a blinking period of the event information is changed depending on the time period before the event occur (Paragraph 148 teaches the countdown items change color as the counting progresses. The countdown is an item in the event information image taught by Lyren).
Arai is analogous to the claimed invention because it teaches an apparatus to generate virtual viewpoints for event viewing. Lyren and Kanzaki are analogous to the claimed invention because it teaches analyzing information in a viewpoint and displaying an image with event information to a user. Ueno is analogous to the claimed invention because it teaches a time period before an event happens. Thus, it would have been obvious to a person holding ordinary skill in the art before the effective filing date to modify the image processing apparatus of generating a virtual viewpoint image taught by Arai in view of Lyren and Kanzaki with the information changing color according to a time period taught by Ueno in order to let the person know when the countdown is low and an event is about to occur.
28. Claim(s) 13 is/are rejected under 35 U.S.C. 103 as being unpatentable over Arai (U.S. Patent Application Publication No. 2019/0289275 A1) in view of Lyren (U.S. Patent Application Publication No. 2020035546 A1) and Kanzaki et al. (U.S. Patent Application Publication No. 2021/0072940 A1), hereinafter referred to as Kanzaki, as applied to claim 1 above, and further in view of Makita et al. (U.S. Patent Application Publication No. 2020/0334481 A1), hereinafter referred to as Makita.
Regarding claim 13, Arai in view of Lyren and Kanzaki teaches the limitations of claim 1. However, Arai and Lyren fail to teach the image processing apparatus wherein the one or more processors further execute the instructions to change the predetermined time period based on a priority of the event.
Makita teaches the image processing apparatus wherein the one or more processors further execute the instructions to change the predetermined time period based on a priority of the event (Paragraph 36 teaches using a longer predetermined time period to detect a second event and a shorter predetermined time period to detect a first event because the second event is deemed higher priority that the first event. The priority is decided based on the quality of the image for the events).
Arai is analogous to the claimed invention because it teaches an apparatus to generate virtual viewpoints for event viewing. Lyren and Kanzaki are analogous to the claimed invention because it teaches analyzing information in a viewpoint and displaying an image with event information to a user. Makita is analogous to the claimed invention because it teaches detecting event information. Thus, it would have been obvious to a person holding ordinary skill in the art before the effective filing date to modify the image processing apparatus that generates virtual viewpoint images as taught by Arai in view of Lyren and Kanzaki with the predetermined time dependency on a priority of an event taught by Makita in order to save and allocate resources to other image capturing apparatuses (Makita Paragraph 47).
29. Claim(s) 17 is/are rejected under 35 U.S.C. 103 as being unpatentable over Arai (U.S. Patent Application Publication No. 2019/0289275 A1) in view of Lyren (U.S. Patent Application Publication No. 2020035546 A1) and Kanzaki et al. (U.S. Patent Application Publication No. 2021/0072940 A1), hereinafter referred to as Kanzaki, as applied to claim 1 above, and further in view of Williams et al. (U.S. Patent Application Publication No. 2016/0379682 A1), hereinafter referred to as Williams.
Regarding claim 17, Arai in view of Lyren and Kanzaki teaches the limitations of claim 1. However, Arai and Lyren fail to teach wherein the one or more processors further execute the instructions to generate the virtual viewpoint image including a captured image corresponding to a time later than the first time, or an overhead view image.
Williams teaches wherein the one or more processors further execute the instructions to generate the virtual viewpoint image including a captured image corresponding to a time later than the first time, or an overhead view image (Paragraph 43-44, Figures 3A-3C and 4A teaches an overhead view image 305 included as part of the virtual viewpoint image).
Arai is analogous to the claimed invention because it teaches an apparatus to generate virtual viewpoints for event viewing. Lyren and Kanzaki are analogous to the claimed invention because it teaches analyzing information in a viewpoint and displaying an image with event information to a user. Williams is analogous to the claimed invention because it teaches analyzing and generating event information for a sporting event. Thus, it would have been obvious to a person holding ordinary skill in the art before the effective filing date to modify the image processing apparatus to generate a virtual viewpoint image taught by Arai in view of Lyren and Kanzaki with the overhead view image taught by Williams in order to let users know where on the entire field the current view is showing (Williams Paragraph 44).
Conclusion
30. Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
31. Any inquiry concerning this communication or earlier communications from the examiner should be directed to CHRISTINE Y AHN whose telephone number is (571)272-0672. The examiner can normally be reached M-F 9-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Alicia Harrington can be reached at (571)272-2330. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CHRISTINE YERA AHN/Examiner, Art Unit 2615
/ALICIA M HARRINGTON/Supervisory Patent Examiner, Art Unit 2615
Prosecution Insights