DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Receipt is acknowledged of certified copies of papers submitted under 35 U.S.C. 119(a)-(d), which papers have been placed of record in the file.
Response to Amendment
Applicant’s remarks, filed 07/15/2026, regarding the objections made to the claims submitted in the non-final office action dated 04/16/2026 are withdrawn due to the amendments made to the claims.
Response to Arguments
Applicant’s arguments see remarks, filed 07/15/2026, with respect to the claims 1-8 and 11-21 have been fully considered but are moot because the arguments do not apply to the current combinations of references being used in the current rejection.
Additionally, Applicant’s arguments, filed 07/15/2026, regarding the 112(f) claim interpretation submitted in the non-final office action dated 04/16/2026 have been considered but are maintained as the amendments to the claims do not recite sufficient structure/material.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that use the word “means” or “step” but are nonetheless not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph because the claim limitation(s) recite(s) sufficient structure, materials, or acts to entirely perform the recited function.
Claim 1 recites limitations that use words like “means” (or “step”) or similar terms with functional language and does invoke 35 U.S.C. 112(f):
Claim 1; recites the limitation, “captured by a plurality of image capturing apparatuses……,” [Line 5].
Because this/these claim limitation(s) is/are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, it/they is/are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
After a careful analysis, as disclosed above, and a careful review of the specification the following limitations in claim 1;
(i) “image capturing apparatuses” (Paragraph [0021]-It is assumed that each of the image capturing apparatuses according to the present exemplary embodiment is a camera having an independent body, capable of capturing images with an individual viewpoint. However, the image capturing apparatuses are not limited thereto, and two or more image capturing apparatuses may be arranged in one body. For example, a single camera having a plurality of lens groups and sensors, capable of capturing images from a plurality of viewpoints, may be installed as the plurality of image capturing apparatuses. Thus, the image capturing apparatuses have sufficient structure or material wherein are cameras.).
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Double Patenting
The non-statutory double patenting rejection is based on a judicially created doctrine grounded in public policy (a policy reflected in the statute) so as to prevent the unjustified or improper timewise extension of the “right to exclude” granted by a patent and to prevent possible harassment by multiple assignees. A non-statutory double patenting rejection is appropriate where the conflicting claims are not identical, but at least one examined application claim is not patentably distinct from the reference claim(s) because the examined application claim is either anticipated by, or would have been obvious over, the reference claim(s). See, e.g., In re Berg, 140 F.3d 1428, 46 USPQ2d 1226 (Fed. Cir. 1998); In re Goodman, 11 F.3d 1046, 29 USPQ2d 2010 (Fed. Cir. 1993); In re Longi, 759 F.2d 887, 225 USPQ 645 (Fed. Cir. 1985); In re Van Ornum, 686 F.2d 937, 214 USPQ 761 (CCPA 1982); In re Vogel, 422 F.2d 438, 164 USPQ 619 (CCPA 1970); In re Thorington, 418 F.2d 528, 163 USPQ 644 (CCPA 1969).
A timely filed terminal disclaimer in compliance with 37 CFR 1.321(c) or 1.321(d) may be used to overcome an actual or provisional rejection based on non-statutory double patenting provided the reference application or patent either is shown to be commonly owned with the examined application, or claims an invention made as a result of activities undertaken within the scope of a joint research agreement. See MPEP § 717.02 for applications subject to examination under the first inventor to file provisions of the AIA as explained in MPEP § 2159. See MPEP § 2146 et seq. for applications not subject to examination under the first inventor to file provisions of the AIA . A terminal disclaimer must be signed in compliance with 37 CFR 1.321(b).
The USPTO Internet website contains terminal disclaimer forms which may be used. Please visit www.uspto.gov/patent/patents-forms. The filing date of the application in which the form is filed determines what form (e.g., PTO/SB/25, PTO/SB/26, PTO/AIA /25, or PTO/AIA /26) should be used. A web-based eTerminal Disclaimer may be filled out completely online using web-screens. An eTerminal Disclaimer that meets all requirements is auto-processed and approved immediately upon submission. For more information about eTerminal Disclaimers, refer to www.uspto.gov/patents/process/file/efs/guidance/eTD-info-I.jsp.
Claims 1, 7-8, 11-12, and 15-16 are rejected on the ground of non-statutory double patenting as being unpatentable over claims 1-2, 6, 8-9, and 11-12 of Co-Pending Application No. 18/783,041 in view of MITSUMOTO (US 20170171570 A1) and further in view of YAMAMOTO et al. (US 20190147604 A1).
Claims 2-3 are rejected on the ground of non-statutory double patenting as being unpatentable over claim 1 of Co-Pending Application No. 18/783,041 in view of MITSUMOTO (US 20170171570 A1) and further in view of YAMAMOTO et al. (US 20190147604 A1), and further in view of GREEN et al. (US 20240062407 A1).
Claims 4-5 are rejected on the ground of non-statutory double patenting as being unpatentable over claim 1 of Co-Pending Application No. 18/783,041 in view of MITSUMOTO (US 20170171570 A1) and further in view of YAMAMOTO et al. (US 20190147604 A1) and further in view of GREEN et al. (US 20240062407 A1) and further in view of FERLATTE et al. (US 20130148861 A1).
Claim 6 is rejected on the ground of non-statutory double patenting as being unpatentable over claim 1 of Co-Pending Application No. 18/783,041 in view of MITSUMOTO (US 20170171570 A1) and further in view of YAMAMOTO et al. (US 20190147604 A1), and further in view of NOMURA (US 20150091715 A1).
Claim 13 is rejected on the ground of non-statutory double patenting as being unpatentable over claims 1 and 2 of Co-Pending Application No. 18/783,041 in view of MITSUMOTO (US 20170171570 A1) and further in view of YAMAMOTO et al. (US 20190147604 A1), and further in view of ONUMA (US 20220383586 A1).
Claim 14 and 19 are rejected on the ground of non-statutory double patenting as being unpatentable over claim 1 of Co-Pending Application No. 18/783,041 in view of MITSUMOTO (US 20170171570 A1) and further in view of YAMAMOTO et al. (US 20190147604 A1), and further in view of MARUYAMA (US 20180246631 A1).
Although the claims 1-8 and 11-17, and 19 of this Application No. 18/749,402 and claims at issue are not identical, they are not patentably distinct from each other because the instant application and the conflicting Patent are claiming common subject matter, as follows:
This Application No. 18/749,402
Co-Pending Application No. 18/783,041
Claim 1: An information processing apparatus comprising:
one or more memories storing instructions; and
one or more processors executing the instructions to;
acquire position information indicating a position of an object included in an image capturing area captured by a plurality of image capturing apparatuses;
determine a position of a virtual viewpoint and a line-of-sight direction of the virtual viewpoint,
acquire position information indicating a plurality of positions of interest,
wherein each of the plurality of positions of interest corresponds to one of a plurality of stationary objects;
specify one position of interest from among a plurality of positions of interest, based on a moving direction of the object; and
such that a stationary object corresponding to the specified position of interest and the object are included in a virtual viewpoint image.
Claim 15: An information processing method comprising:
acquiring position information indicating a position of an object included in an image capturing area captured by a plurality of image capturing apparatuses;
determining a position of a virtual viewpoint and a line-of-sight direction of the virtual viewpoint,
acquiring position information indicating a plurality of positions of interest, wherein each of the plurality of positions of interest corresponds to one of a plurality of stationary objects;
specifying one position of interest from among a plurality of positions of interest, based on a moving direction of the object; and
such that a stationary object corresponding to the specified position of interest and the object are included in a virtual viewpoint image.
Claim 16: A non-transitory computer-readable storage medium storing a computer program for causing a computer to execute an image processing method comprising:
acquiring position information indicating a position of an object included in an image capturing area captured by a plurality of image capturing apparatuses;
determining a position of a virtual viewpoint and a line-of-sight direction of the virtual viewpoint,
acquiring position information indicating a plurality of positions of interest, wherein each of the plurality of positions of interest corresponds to one of a plurality of stationary objects;
specifying one position of interest from among a plurality of positions of interest, based on a moving direction of the object; and
such that a stationary object corresponding to the specified position of interest and the object are included in a virtual viewpoint image.
Claim 1: An information processing system comprising:
one or more memories storing instructions; and
one or more processors executing the instructions to:
acquire subject information indicating a position of a subject, images of which are captured by a plurality of imaging apparatuses, in a virtual space that corresponds to a position in a real space, region information indicating a plurality of regions on a two-dimensional plane corresponding to a ground surface or a floor surface included in an image capturing region in the virtual space, and interest position information indicating a plurality of interest positions in a case where the subject exists within a first region among the plurality of regions,
determine a position of a virtual viewpoint corresponding to a virtual viewpoint image and a line-of-sight direction from the virtual viewpoint based on the subject information and a first interest position among the plurality of interest positions; and
display a virtual viewpoint image generated based on the position of the virtual viewpoint and the line-of-sight direction from the virtual viewpoint,
wherein, in the virtual viewpoint image, an image region corresponding to the first region is displayed below an image region corresponding to a second region in which the subject does not exist among the plurality of regions, and
wherein, in the virtual viewpoint image, the subject and the first interest position are displayed such that image coordinates corresponding to the position of the subject and image coordinates corresponding to the first interest position have a positional relationship corresponding to a combination of the first region and the first interest position.
Claim 11: An information processing method comprising:
acquiring subject information indicating a position of a subject, images of which are captured by a plurality of imaging apparatuses, in a virtual space that corresponds to a position in a real space, region information indicating a plurality of regions on a two-dimensional plane corresponding to a ground surface or a floor surface included in an image capturing region in a three-dimensional space, and interest position information indicating a plurality of interest positions;
in a case where the subject exists within a first region among the plurality of regions, determining a position of a virtual viewpoint corresponding to a virtual viewpoint image and a line-of-sight direction from the virtual viewpoint based on the subject information and a first interest position among the plurality of interest positions; and
displaying a virtual viewpoint image generated based on the position of the virtual viewpoint and the line-of-sight direction from the virtual viewpoint, wherein, in the virtual viewpoint image, an image region corresponding to the first region is displayed below an image region corresponding to a second region in which the subject does not exist among the plurality of regions, and wherein, in the virtual viewpoint image, the subject and the first interest position are displayed such that image coordinates corresponding to the position of the subject and image coordinates corresponding to the first interest position have a positional relationship corresponding to a combination of the first region and the first interest position.
Claim 12: A non-transitory computer readable storage medium storing a program for causing a computer to execute an information processing method comprising:
acquiring subject information indicating a position of a subject, images of which are captured by a plurality of imaging apparatuses, in a virtual space that corresponds to a position in a real space, region information indicating a plurality of regions on a two-dimensional plane corresponding to a ground surface or a floor surface included in an image capturing region in a three-dimensional space, and interest position information indicating a plurality of interest positions;
in a case where the subject exists within a first region among the plurality of regions, determining a position of a virtual viewpoint corresponding to a virtual viewpoint image and a line-of-sight direction from the virtual viewpoint based on the subject information and a first interest position among the plurality of interest positions; and
displaying a virtual viewpoint image generated based on the position of the virtual viewpoint and the line-of-sight direction from the virtual viewpoint, wherein, in the virtual viewpoint image, an image region corresponding to the first region is displayed below an image region corresponding to a second region in which the subject does not exist among the plurality of regions, and wherein, in the virtual viewpoint image, the subject and the first interest position are displayed such that image coordinates corresponding to the position of the subject and image coordinates corresponding to the first interest position have a positional relationship corresponding to a combination of the first region and the first interest position.
Although Co-Pending Application No. 18/783,041 claim 1 teaches An information processing apparatus comprising: one or more memories storing instructions; and one or more processors executing the instructions to; acquire position information indicating a position of an object included in an image capturing area captured by a plurality of image capturing apparatuses; determine a position of a virtual viewpoint and a line-of-sight direction of the virtual viewpoint, Co-Pending Application No. 18/783,041 claim 1 as stated in the table above with respect to claim 1, fails to clearly disclose acquire position information indicating a plurality of positions of interest, wherein each of the plurality of positions of interest corresponds to one of a plurality of stationary objects; such that a stationary object corresponding to the specified position of interest and the object are included in a virtual viewpoint image.
However, MITSUMOTO (US 20170171570 A1) explicitly teaches acquire position information indicating a plurality of positions of interest (Fig. 12A-12B. Paragraph [0048]-MITSUMOTO discloses positions of the goal 105 and the goal 106 in the field coordinate system (certain positions in the areas of the respective goals) are determined in advance. Further in paragraph [0072]-MITSUMOTO discloses since a position of each goal is a certain position in the area of the goal, the center of the width of the goal may be the position of the goal as illustrated in FIG. 12A, or a position of an end of the width of the goal may be the position of the goal as illustrated in FIG. 12B (wherein the goals are positions of interest).),
wherein each of the plurality of positions of interest corresponds to one of a plurality of stationary objects (Fig. 12A-12B. Paragraph [0048]-MITSUMOTO discloses positions of the goal 105 and the goal 106 in the field coordinate system (certain positions in the areas of the respective goals) are determined in advance. Further in paragraph [0072]-MITSUMOTO discloses since a position of each goal is a certain position in the area of the goal, the center of the width of the goal may be the position of the goal as illustrated in FIG. 12A, or a position of an end of the width of the goal may be the position of the goal as illustrated in FIG. 12B (wherein goals are stationary objects).);
such that a stationary object corresponding to the specified position of interest and the object are included in a virtual viewpoint image (Fig. 11A-11C, illustrate a position of interest and the object included in a virtual viewpoint image (wherein the position of interest is the goal and the object is the ball). Paragraph [0101]-MITSUMOTO discloses a first information processing apparatus 201 decides on a camera (or a virtual viewpoint) on the basis of a positional relationship between a first predetermined object (for example, a ball) and a second predetermined object (for example, a goal) and outputs the decision to a second information processing apparatus 201.).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of Co-Pending Application No. 18/783,041 claim 1 of an information processing apparatus comprising: one or more memories storing instructions; and one or more processors executing the instructions to; acquire position information indicating a position of an object included in an image capturing area captured by a plurality of image capturing apparatuses; determine a position of a virtual viewpoint and a line-of-sight direction of the virtual viewpoint with the teachings of MITSUMOTO (US 20170171570 A1) of acquire position information indicating a plurality of positions of interest, wherein each of the plurality of positions of interest corresponds to one of a plurality of stationary objects; such that a stationary object corresponding to the specified position of interest and the object are included in a virtual viewpoint image.
Wherein having Co-Pending Application No. 18/783,041 claim 1 of acquire position information indicating a plurality of positions of interest, wherein each of the plurality of positions of interest corresponds to one of a plurality of stationary objects; such that a stationary object corresponding to the specified position of interest and the object are included in a virtual viewpoint image.
The motivation behind the modification would have been to obtain an apparatus for generating a virtual viewpoint and virtual viewpoint image that enhances the accuracy of identifying and tracking subjects/objects of interest.
Co-Pending Application No. 18/783,041 claim 1 in view of MITSUMOTO (US 20170171570 A1) fail to explicitly teach specify one position of interest from among a plurality of positions of interest, based on a moving direction of the object; and
However, YAMAMOTO et al. (US 20190147604 A1) explicitly teaches specify one position of interest (Fig. 15. Paragraph [0079]-YAMAMOTO discloses the ball detection unit 150b registers information on the ball candidate 20d in the ball management table 140c. That is, the ball detection unit 150b registers candidate identification information and coordinates of the ball candidate 20d in association with the time “t” in the ball management table 140c (wherein ball candidate 20d is one position of interest).) from among a plurality of positions of interest (Fig. 15. Paragraph [0077]-YAMAMOTO discloses it is assumed that a ball candidate 18 is a ball candidate detected at a time t−2. It is assumed that a ball candidate 19 is a ball candidate detected at a time t−1. It is assumed that ball candidates 20a, 20b, and 20c are ball candidates detected at a time “t” (wherein ball candidates 20a-c are a plurality of positions of interest).), based on a moving direction of the object (Fig. 15. Paragraph [0078-0079]-YAMAMOTO discloses by taking the ball candidate 19 as a reference, the ball detection unit 150b predicts a region 21 of a movement destination of the ball candidate at the time “t,” based on the shape and the speed corresponding to the event identification information defined in the movement definition information 140e. Here, as an example, it is assumed that the event identification information is preset in the ball detection unit 150b. Step S11 of FIG. 15 will be described. When no ball candidate exists in the region 21 at the time “t,” the ball detection unit 150b adds a ball candidate 20d to the region 21. The ball detection unit 150b registers information on the ball candidate 20d in the ball management table 140c. That is, the ball detection unit 150b registers candidate identification information and coordinates of the ball candidate 20d in association with the time “t” in the ball management table 140c (wherein the ball is the object moving).); and
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of Co-Pending Application No. 18/783,041 claim 1 in view of MITSUMOTO (US 20170171570 A1) of an information processing apparatus comprising: one or more memories storing instructions; and one or more processors executing the instructions to; acquire position information indicating a position of an object included in an image capturing area captured by a plurality of image capturing apparatuses; determine a position of a virtual viewpoint and a line-of-sight direction of the virtual viewpoint with the teachings of YAMAMOTO et al. (US 20190147604 A1) of specify one position of interest from among a plurality of positions of interest, based on a moving direction of the object; and.
Wherein having Co-Pending Application No. 18/783,041 claim 1 of specify one position of interest from among a plurality of positions of interest, based on a moving direction of the object; and.
The motivation behind the modification would have been to obtain an apparatus for generating a virtual viewpoint and virtual viewpoint image that enhances the accuracy of identifying and tracking subjects/objects of interest.
Although Co-Pending Application No. 18/783,041 claim 11 teaches an information processing method comprising: acquiring position information indicating a position of an object included in an image capturing area captured by a plurality of image capturing apparatuses; determining a position of a virtual viewpoint and a line-of-sight direction of the virtual viewpoint, Co-Pending Application No. 18/783,041 claim 11 as stated in the table above with respect to claim 15, fails to clearly disclose acquiring position information indicating a plurality of positions of interest, wherein each of the plurality of positions of interest corresponds to one of a plurality of stationary objects; and such that a stationary object corresponding to the specified position of interest and the object are included in a virtual viewpoint image.
However, MITSUMOTO explicitly teaches acquiring position information indicating a plurality of positions of interest (Fig. 12A-12B. Paragraph [0048]-MITSUMOTO discloses positions of the goal 105 and the goal 106 in the field coordinate system (certain positions in the areas of the respective goals) are determined in advance. Further in paragraph [0072]-MITSUMOTO discloses since a position of each goal is a certain position in the area of the goal, the center of the width of the goal may be the position of the goal as illustrated in FIG. 12A, or a position of an end of the width of the goal may be the position of the goal as illustrated in FIG. 12B (wherein the goals are positions of interest).),
wherein each of the plurality of positions of interest corresponds to one of a plurality of stationary objects (Fig. 12A-12B. Paragraph [0048]-MITSUMOTO discloses positions of the goal 105 and the goal 106 in the field coordinate system (certain positions in the areas of the respective goals) are determined in advance. Further in paragraph [0072]-MITSUMOTO discloses since a position of each goal is a certain position in the area of the goal, the center of the width of the goal may be the position of the goal as illustrated in FIG. 12A, or a position of an end of the width of the goal may be the position of the goal as illustrated in FIG. 12B (wherein goals are stationary objects).);
such that a stationary object corresponding to the specified position of interest and the object are included in a virtual viewpoint image (Fig. 11A-11C, illustrate a position of interest and the object included in a virtual viewpoint image (wherein the position of interest is the goal and the object is the ball). Paragraph [0101]-MITSUMOTO discloses a first information processing apparatus 201 decides on a camera (or a virtual viewpoint) on the basis of a positional relationship between a first predetermined object (for example, a ball) and a second predetermined object (for example, a goal) and outputs the decision to a second information processing apparatus 201.).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of Co-Pending Application No. 18/783,041 claim 11 of an information processing method comprising: acquiring position information indicating a position of an object included in an image capturing area captured by a plurality of image capturing apparatuses; determining a position of a virtual viewpoint and a line-of-sight direction of the virtual viewpoint with the teachings of MITSUMOTO (US 20170171570 A1) of acquiring position information indicating a plurality of positions of interest, wherein each of the plurality of positions of interest corresponds to one of a plurality of stationary objects; and such that a stationary object corresponding to the specified position of interest and the object are included in a virtual viewpoint image.
Wherein having Co-Pending Application No. 18/783,041 claim 1 of acquiring position information indicating a plurality of positions of interest, wherein each of the plurality of positions of interest corresponds to one of a plurality of stationary objects; and such that a stationary object corresponding to the specified position of interest and the object are included in a virtual viewpoint image.
The motivation behind the modification would have been to obtain an apparatus for generating a virtual viewpoint and virtual viewpoint image that enhances the accuracy of identifying and tracking subjects/objects of interest.
Co-Pending Application No. 18/783,041 claim 11 in view of MITSUMOTO (US 20170171570 A1) fail to explicitly teach specifying one position of interest from among a plurality of positions of interest, based on a moving direction of the object;
However, YAMAMOTO et al. (US 20190147604 A1) explicitly teaches specifying one position of interest (Fig. 15. Paragraph [0079]-YAMAMOTO discloses the ball detection unit 150b registers information on the ball candidate 20d in the ball management table 140c. That is, the ball detection unit 150b registers candidate identification information and coordinates of the ball candidate 20d in association with the time “t” in the ball management table 140c (wherein ball candidate 20d is one position of interest).) from among a plurality of positions of interest (Fig. 15. Paragraph [0077]-YAMAMOTO discloses it is assumed that a ball candidate 18 is a ball candidate detected at a time t−2. It is assumed that a ball candidate 19 is a ball candidate detected at a time t−1. It is assumed that ball candidates 20a, 20b, and 20c are ball candidates detected at a time “t” (wherein ball candidates 20a-c are a plurality of positions of interest).), based on a moving direction of the object (Fig. 15. Paragraph [0078-0079]-YAMAMOTO discloses by taking the ball candidate 19 as a reference, the ball detection unit 150b predicts a region 21 of a movement destination of the ball candidate at the time “t,” based on the shape and the speed corresponding to the event identification information defined in the movement definition information 140e. Here, as an example, it is assumed that the event identification information is preset in the ball detection unit 150b. Step S11 of FIG. 15 will be described. When no ball candidate exists in the region 21 at the time “t,” the ball detection unit 150b adds a ball candidate 20d to the region 21. The ball detection unit 150b registers information on the ball candidate 20d in the ball management table 140c. That is, the ball detection unit 150b registers candidate identification information and coordinates of the ball candidate 20d in association with the time “t” in the ball management table 140c (wherein the ball is the object moving).); and
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of Co-Pending Application No. 18/783,041 claim 11 in view of MITSUMOTO (US 20170171570 A1) of an information processing method comprising: acquiring position information indicating a position of an object included in an image capturing area captured by a plurality of image capturing apparatuses; determining a position of a virtual viewpoint and a line-of-sight direction of the virtual viewpoint with the teachings of YAMAMOTO et al. (US 20190147604 A1) of specifying one position of interest from among a plurality of positions of interest, based on a moving direction of the object; and
Wherein having Co-Pending Application No. 18/783,041 claim 11 of specifying one position of interest from among a plurality of positions of interest, based on a moving direction of the object; and
The motivation behind the modification would have been to obtain a method for generating a virtual viewpoint and virtual viewpoint image that enhances the accuracy of identifying and tracking subjects/objects of interest.
Although Co-Pending Application No. 18/783,041 claim 12 teaches a non-transitory computer-readable storage medium storing a computer program for causing a computer to execute an image processing method comprising: acquiring position information indicating a position of an object included in an image capturing area captured by a plurality of image capturing apparatuses; determining a position of a virtual viewpoint and a line-of-sight direction of the virtual viewpoint, Co-Pending Application No. 18/783,041 claim 12 as stated in the table above with respect to claim 16, fails to clearly disclose acquiring position information indicating a plurality of positions of interest, wherein each of the plurality of positions of interest corresponds to one of a plurality of stationary objects; and such that a stationary object corresponding to the specified position of interest and the object are included in a virtual viewpoint image.
However, MITSUMOTO explicitly teaches acquiring position information indicating a plurality of positions of interest (Fig. 12A-12B. Paragraph [0048]-MITSUMOTO discloses positions of the goal 105 and the goal 106 in the field coordinate system (certain positions in the areas of the respective goals) are determined in advance. Further in paragraph [0072]-MITSUMOTO discloses since a position of each goal is a certain position in the area of the goal, the center of the width of the goal may be the position of the goal as illustrated in FIG. 12A, or a position of an end of the width of the goal may be the position of the goal as illustrated in FIG. 12B (wherein the goals are positions of interest).),
wherein each of the plurality of positions of interest corresponds to one of a plurality of stationary objects (Fig. 12A-12B. Paragraph [0048]-MITSUMOTO discloses positions of the goal 105 and the goal 106 in the field coordinate system (certain positions in the areas of the respective goals) are determined in advance. Further in paragraph [0072]-MITSUMOTO discloses since a position of each goal is a certain position in the area of the goal, the center of the width of the goal may be the position of the goal as illustrated in FIG. 12A, or a position of an end of the width of the goal may be the position of the goal as illustrated in FIG. 12B (wherein goals are stationary objects).);
such that a stationary object corresponding to the specified position of interest and the object are included in a virtual viewpoint image (Fig. 11A-11C, illustrate a position of interest and the object included in a virtual viewpoint image (wherein the position of interest is the goal and the object is the ball). Paragraph [0101]-MITSUMOTO discloses a first information processing apparatus 201 decides on a camera (or a virtual viewpoint) on the basis of a positional relationship between a first predetermined object (for example, a ball) and a second predetermined object (for example, a goal) and outputs the decision to a second information processing apparatus 201.).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of Co-Pending Application No. 18/783,041 claim 12 of a non-transitory computer-readable storage medium storing a computer program for causing a computer to execute an image processing method comprising: acquiring position information indicating a position of an object included in an image capturing area captured by a plurality of image capturing apparatuses; determining a position of a virtual viewpoint and a line-of-sight direction of the virtual viewpoint with the teachings of MITSUMOTO (US 20170171570 A1) of acquiring position information indicating a plurality of positions of interest, wherein each of the plurality of positions of interest corresponds to one of a plurality of stationary objects; and such that a stationary object corresponding to the specified position of interest and the object are included in a virtual viewpoint image.
Wherein having Co-Pending Application No. 18/783,041 claim 1 of acquiring position information indicating a plurality of positions of interest, wherein each of the plurality of positions of interest corresponds to one of a plurality of stationary objects; and such that a stationary object corresponding to the specified position of interest and the object are included in a virtual viewpoint image.
The motivation behind the modification would have been to obtain an apparatus for generating a virtual viewpoint and virtual viewpoint image that enhances the accuracy of identifying and tracking subjects/objects of interest.
Co-Pending Application No. 18/783,041 claim 12 in view of MITSUMOTO (US 20170171570 A1) fail to explicitly teach specifying one position of interest from among a plurality of positions of interest, based on a moving direction of the object;
However, YAMAMOTO et al. (US 20190147604 A1) explicitly teaches specifying one position of interest (Fig. 15. Paragraph [0079]-YAMAMOTO discloses the ball detection unit 150b registers information on the ball candidate 20d in the ball management table 140c. That is, the ball detection unit 150b registers candidate identification information and coordinates of the ball candidate 20d in association with the time “t” in the ball management table 140c (wherein ball candidate 20d is one position of interest).) from among a plurality of positions of interest (Fig. 15. Paragraph [0077]-YAMAMOTO discloses it is assumed that a ball candidate 18 is a ball candidate detected at a time t−2. It is assumed that a ball candidate 19 is a ball candidate detected at a time t−1. It is assumed that ball candidates 20a, 20b, and 20c are ball candidates detected at a time “t” (wherein ball candidates 20a-c are a plurality of positions of interest).), based on a moving direction of the object (Fig. 15. Paragraph [0078-0079]-YAMAMOTO discloses by taking the ball candidate 19 as a reference, the ball detection unit 150b predicts a region 21 of a movement destination of the ball candidate at the time “t,” based on the shape and the speed corresponding to the event identification information defined in the movement definition information 140e. Here, as an example, it is assumed that the event identification information is preset in the ball detection unit 150b. Step S11 of FIG. 15 will be described. When no ball candidate exists in the region 21 at the time “t,” the ball detection unit 150b adds a ball candidate 20d to the region 21. The ball detection unit 150b registers information on the ball candidate 20d in the ball management table 140c. That is, the ball detection unit 150b registers candidate identification information and coordinates of the ball candidate 20d in association with the time “t” in the ball management table 140c (wherein the ball is the object moving).); and
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of Co-Pending Application No. 18/783,041 claim 12 in view of MITSUMOTO (US 20170171570 A1) of a non-transitory computer-readable storage medium storing a computer program for causing a computer to execute an image processing method comprising: acquiring position information indicating a position of an object included in an image capturing area captured by a plurality of image capturing apparatuses; determining a position of a virtual viewpoint and a line-of-sight direction of the virtual viewpoint with the teachings of YAMAMOTO et al. (US 20190147604 A1) of specifying one position of interest from among a plurality of positions of interest, based on a moving direction of the object; and
Wherein having Co-Pending Application No. 18/783,041 claim 12 of specifying one position of interest from among a plurality of positions of interest, based on a moving direction of the object; and
The motivation behind the modification would have been to obtain a method for generating a virtual viewpoint and virtual viewpoint image that enhances the accuracy of identifying and tracking subjects/objects of interest.
The further limitations of the dependent claims are similar as indicated below:
This Application No. 18/749,402
Co-Pending Application No. 18/783,041
Claim 8: wherein the specified position of interest is set based on a user operation. (Line 1-2)
Claim 11: wherein the stationary object is a basket goal. (Line 1-2)
Claim 12: wherein the position of the virtual viewpoint and the line-of-sight direction of the virtual viewpoint are determined based on a half line extending from the position of the object to the specified position of interest. (Line 1-4)
Claim 6: wherein the interest position is set based on setting a user operation. (Line 1-2)
Claim 8: the background model is a stationary object (Line 1-2).
Claim 9: wherein the background model is a basketball goal. (Line 1-2)
Claim 2: wherein a position of the virtual viewpoint and a line-of-sight direction from the virtual viewpoint are determined based on a straight line passing through a position of the subject in the virtual space and the interest position (wherein a straight line is a half line). (Line 1-4)
Claims 8 and 11-12 contain the same limitations as Co-Pending Application No. 18/783,041 claims 6, 8-9, and 2 respectively. Therefore, given that claims 8 and 11-12 depend from claim 1, and claims 6, 8-9, and 2 depend from claim 1. Claims 8 and 11-12 are rejected for the same reasons set forth in the rejection of the independent claim above.
Claims 2-3 are rejected on the ground of non-statutory double patenting as being unpatentable over claim 1 of Co-Pending Application No. 18/783,041 in view of MITSUMOTO (US 20170171570 A1) and further in view of YAMAMOTO et al. (US 20190147604 A1), and further in view of GREEN et al. (US 20240062407 A1).
Regarding claim 2, Co-Pending Application No. 18/783,041 claim 1 in view of MITSUMOTO (US 20170171570 A1) and further in view of YAMAMOTO et al. (US 20190147604 A1) explicitly teaches the information processing apparatus according to claim 1, Co-Pending Application No. 18/783,041 claim 1 in view of MITSUMOTO (US 20170171570 A1) and further in view of YAMAMOTO et al. (US 20190147604 A1) fail to explicitly teach wherein the position information indicates positions of a plurality of objects, and wherein the specified position of interest is specified based on moving directions of the plurality of objects.
However, GREEN et al. (US 20240062407 A1) explicitly teaches wherein the position information indicates positions of a plurality of objects (Fig. 6, illustrates position information for a plurality of objects (i.e. players and balls). Paragraph [0029]-GREEN discloses the relative positions of objects with respect to each other, and with respect to certain areas of the pitch (e.g. a goal area) may then be calculated.), and
wherein the specified position of interest is specified based on moving directions of the plurality of objects (Fig. 3. Paragraph [0031]-GREEN discloses in past sporting events there will be a higher number of successful goals in instances where, at a point in time when a player has just kicked the ball, velocity information (that is, speed and direction information) of the ball indicates a predicted ball trajectory that enters (or remains inside) the 3D space 301 when the ball reaches the edge of the pitch defining the goal line in order for a goal to be scored (wherein edge 302 of the goal line is a specified position based on the moving directions of the player and a ball (i.e. the player kicking the ball).). Further see Fig. 9 where a player is also kicking a ball, but the goal has not been specified as the position of interest.).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of Co-Pending Application No. 18/783,041 claim 1 in view of MITSUMOTO (US 20170171570 A1) and further in view of YAMAMOTO et al. (US 20190147604 A1) of an information processing apparatus, with the teachings of GREEN et al. (US 20240062407 A1) of wherein the position information indicates positions of a plurality of objects, and wherein the specified position of interest is specified based on moving directions of the plurality of objects.
Wherein having Co-Pending Application No. 18/783,041 claim 1’s an information processing apparatus wherein the position information indicates positions of a plurality of objects, and wherein the specified position of interest is specified based on moving directions of the plurality of objects.
The motivation behind the modification would have been to obtain information processing apparatus that enhances the accuracy of identifying and tracking subjects/objects of interest.
Regarding claim 3, Co-Pending Application No. 18/783,041 claim 1 in view of MITSUMOTO (US 20170171570 A1) and further in view of YAMAMOTO et al. (US 20190147604 A1) and further in view of GREEN et al. (US 20240062407 A1) explicitly teach the information processing apparatus according to claim 2,
Co-Pending Application No. 18/783,041 claim 1 in view of MITSUMOTO (US 20170171570 A1) and further in view of YAMAMOTO et al. (US 20190147604 A1) fail to explicitly teach wherein the specified position of interest is specified based on the moving directions of the plurality of objects and moving velocities of the plurality of objects.
However, GREEN et al. (US 20240062407 A1) explicitly teaches wherein the specified position of interest is specified based on the moving directions of the plurality of objects and moving velocities of the plurality of objects (Fig. 3. Paragraph [0031]-GREEN discloses in past sporting events there will be a higher number of successful goals in instances where, at a point in time when a player has just kicked the ball, velocity information (that is, speed and direction information) of the ball indicates a predicted ball trajectory that enters (or remains inside) the 3D space 301 when the ball reaches the edge of the pitch defining the goal line in order for a goal to be scored (wherein edge 302 of the goal line is a specified position based on the moving directions of the player and a ball (i.e. the player kicking the ball towards the goal).). Further in Paragraph [0036]-GREEN discloses at a point in time when the player 201 is about to kick the ball 202 (such as depicted in FIG. 4), the velocity of the player's striking foot and the backlift b may be used to predict the trajectory of the ball 202 (wherein the velocity of the player’s striking foot used to predicted the trajectory of the ball is a moving velocity of an object that can be used to further specify a position of interest as shown in paragraph [0031]).). Further see Fig. 9 where a player is also kicking a ball, but the goal has not been specified as the position of interest.).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of Co-Pending Application No. 18/783,041 claim 1 in view of MITSUMOTO (US 20170171570 A1) and further in view of YAMAMOTO et al. (US 20190147604 A1) of an information processing apparatus, with the teachings of GREEN et al. (US 20240062407 A1) of wherein the specified position of interest is specified based on the moving directions of the plurality of objects and moving velocities of the plurality of objects.
Wherein having Co-Pending Application No. 18/783,041 claim 1’s an information processing apparatus wherein the specified position of interest is specified based on the moving directions of the plurality of objects and moving velocities of the plurality of objects.
The motivation behind the modification would have been to obtain information processing apparatus that enhances the accuracy of identifying and tracking subjects/objects of interest.
Claims 4-5 are rejected on the ground of non-statutory double patenting as being unpatentable over claim 1 of Co-Pending Application No. 18/783,041 in view of MITSUMOTO (US 20170171570 A1) and further in view of YAMAMOTO et al. (US 20190147604 A1) and further in view of GREEN et al. (US 20240062407 A1) and further in view of FERLATTE et al. (US 20130148861 A1).
Regarding claim 4, Co-Pending Application No. 18/783,041 claim 1 in view of MITSUMOTO (US 20170171570 A1) and further in view of YAMAMOTO et al. (US 20190147604 A1) and further in view of GREEN et al. (US 20240062407 A1) explicitly teach the information processing apparatus according to claim 2,
Co-Pending Application No. 18/783,041 claim 1 in view of MITSUMOTO (US 20170171570 A1) and further in view of YAMAMOTO et al. (US 20190147604 A1) fail to explicitly teach wherein the plurality of objects is classified into a plurality of categories including a first category and a second category, and.
However, GREEN et al. (US 20240062407 A1) explicitly teaches wherein the plurality of objects (Fig. 5, illustrates a plurality of objects, i.e. players 501-503. Paragraph [0040]-GREEN discloses three players, 501-503, are positioned between the ball 202 and the goal 300 such that they partially obscure the goal 300 from the ball 202.) is classified into a plurality of categories including a first category (Fig. 5. Paragraph [0042]-GREEN discloses the goalkeeper may be identified by using any suitable method, for example using an image processing technique that identifies their jersey number and colour, using position information indicating their position relative to the pitch, or the like (wherein the goalkeeper is the first category and wherein the players on field, as illustrated in Fig. 5, are the plurality of objects).) and a second category (Fig. 13, illustrates a picture of a referee (i.e. the second category). Paragraph [0082]-GREEN discloses FIG. 13 depicts a person 1301 who is a match official of the football match (for example, a referee or assistant referee) and who has a pose indicative that a foul has occurred. The person 1301 is determined to be a match official using any suitable method, for example using an image processing technique that identifies their clothing to match the clothing of a match official, using position information indicating their position relative to the pitch, or the like (wherein the match official/referee is the second category).), and
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of Co-Pending Application No. 18/783,041 claim 1 in view of MITSUMOTO (US 20170171570 A1) and further in view of YAMAMOTO et al. (US 20190147604 A1) of an information processing apparatus, with the teachings of GREEN et al. (US 20240062407 A1) of wherein the plurality of objects is classified into a plurality of categories including a first category and a second category, and.
Wherein having Co-Pending Application No. 18/783,041 claim 1’s an information processing apparatus wherein the plurality of objects is classified into a plurality of categories including a first category and a second category, and.
The motivation behind the modification would have been to obtain information processing apparatus that enhances the accuracy of identifying and tracking subjects/objects of interest.
Co-Pending Application No. 18/783,041 claim 1 in view of MITSUMOTO (US 20170171570 A1) and further in view of YAMAMOTO et al. (US 20190147604 A1) and further in view of GREEN et al. (US 20240062407 A1) fail to explicitly teach wherein the specified position of interest is specified based on the moving directions of the plurality of objects corresponding to the first category.
However, FERLATTE et al. (US 20130148861 A1) explicitly teaches wherein the specified position of interest (Fig. 3a. Paragraph [0079]-FERLATTE discloses the template may indicate that a bench area is not of interest and an area by the goal is of greater interest (wherein the goal is the specified position of interest).) is specified based on the moving directions of the plurality of objects corresponding to the first category (Fig. 3a. Paragraph [0078]-FERLATTE discloses a player quickly moving towards the goal may have a higher interest rating then a player slowly moving towards the bench. The interest-rating may be determined based on a variety of interest-rating factors such as position or location within the playing area 90, speed or velocity, location within a predefined area, direction of movement, time lapsed in the sporting event, proximity to other objects, the stage of game play, the identity/personality of a player 110 and so on (wherein the plurality of objects are the players moving towards or away from the goal). Further in paragraph [0079]-FERLATTE discloses the template may indicate that a bench area is not of interest and an area by the goal is of greater interest to distinguish between a situation where a group of players gathers by the bench and another group of players gathers by the goal area, where the latter is determined to be of greater interest.).).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of Co-Pending Application No. 18/783,041 claim 1 in view of MITSUMOTO (US 20170171570 A1) and further in view of YAMAMOTO et al. (US 20190147604 A1) and further in view of GREEN et al. (US 20240062407 A1) of an information processing apparatus, with the teachings of FERLATTE et al. (US 20130148861 A1) of wherein the specified position of interest is specified based on the moving directions of the plurality of objects corresponding to the first category.
Wherein having Co-Pending Application No. 18/783,041 claim 1’s an information processing apparatus wherein the specified position of interest is specified based on the moving directions of the plurality of objects corresponding to the first category.
The motivation behind the modification would have been to obtain information processing apparatus that enhances the accuracy of identifying and tracking subjects/objects of interest.
Regarding claim 5, Co-Pending Application No. 18/783,041 claim 1 in view of MITSUMOTO (US 20170171570 A1) and further in view of YAMAMOTO et al. (US 20190147604 A1) and further in view of GREEN et al. (US 20240062407 A1) and further in view of FERLATTE et al. (US 20130148861 A1) explicitly teach the information processing apparatus according to claim 4,
Co-Pending Application No. 18/783,041 claim 1 fails to explicitly teach wherein the plurality of image capturing apparatuses captures an athletic competition.
However, YAMAMOTO et al. (US 20190147604 A1) explicitly teaches wherein the plurality of image capturing apparatuses captures an athletic competition. (Fig. 1, illustrates an example of an athletic competition with image capturing apparatuses surrounding the competition. Paragraph [0042]-YAMAMOTO discloses FIG. 1 is a view illustrating an example of a system according to Embodiment 1. As illustrated in FIG. 1, the system includes cameras 10a, 10b, 10c, and 10d and an information processing apparatus 100. Although not illustrated, it is assumed that multiple players are playing a game using a ball in a game area 5. For example, the game that the players are playing includes, for example, a soccer, basketball, handball, hockey, rugby, and lacrosse. The cameras 10a, 10b, 10c, and 10d and the information processing apparatus 100 are connected to each other.),
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of Co-Pending Application No. 18/783,041 claim 1 of an information processing apparatus with the teachings of YAMAMOTO et al. (US 20190147604 A1) of wherein the plurality of image capturing apparatuses captures an athletic competition.
Wherein having Co-Pending Application No. 18/783,041 claim 1 of wherein the plurality of image capturing apparatuses captures an athletic competition.
The motivation behind the modification would have been to obtain information processing apparatus that enhances the accuracy of identifying and tracking subjects/objects of interest.
Co-Pending Application No. 18/783,041 claim 1 in view of MITSUMOTO (US 20170171570 A1) and further in view of YAMAMOTO et al. (US 20190147604 A1) fail to explicitly teach wherein the first category is a player, and wherein the second category is a referee.
However, GREEN et al. (US 20240062407 A1) explicitly teaches wherein the first category is a player (Fig. 5. Paragraph [0042]-GREEN discloses the goalkeeper may be identified by using any suitable method, for example using an image processing technique that identifies their jersey number and colour, using position information indicating their position relative to the pitch, or the like (wherein the goalkeeper is a player and is thus the first category).), and
wherein the second category is a referee (Fig. 13, illustrates a picture of a referee, the second category. Paragraph [0082]-GREEN discloses FIG. 13 depicts a person 1301 who is a match official of the football match (for example, a referee or assistant referee) and who has a pose indicative that a foul has occurred. The person 1301 is determined to be a match official using any suitable method, for example using an image processing technique that identifies their clothing to match the clothing of a match official, using position information indicating their position relative to the pitch, or the like (wherein the match official/referee is the second category).).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of Co-Pending Application No. 18/783,041 claim 1 in view of MITSUMOTO (US 20170171570 A1) and further in view of YAMAMOTO et al. (US 20190147604 A1) of an information processing apparatus, with the teachings of GREEN et al. (US 20240062407 A1) of wherein the first category is a player, and wherein the second category is a referee.
Wherein having Co-Pending Application No. 18/783,041 claim 1’s an information processing apparatus wherein the first category is a player, and wherein the second category is a referee.
The motivation behind the modification would have been to obtain information processing apparatus that enhances the accuracy of identifying and tracking subjects/objects of interest.
Claim 6 is rejected on the ground of non-statutory double patenting as being unpatentable over claim 1 of Co-Pending Application No. 18/783,041 in view of MITSUMOTO (US 20170171570 A1) and further in view of YAMAMOTO et al. (US 20190147604 A1), and further in view of NOMURA (US 20150091715 A1).
Regarding claim 6, Co-Pending Application No. 18/783,041 claim 1 in view of MITSUMOTO (US 20170171570 A1) and further in view of YAMAMOTO et al. (US 20190147604 A1) explicitly teach the information processing apparatus according to claim 1,
YAMAMOTO et al. (US 20190147604 A1) explicitly teach wherein the one or more processors further execute the instructions to (Fig. 2. Paragraph [0176]-YAMAMOTO discloses a computer 500 includes a CPU 501 that executes various arithmetic processes, an input device 502 that receives an input of data from a user, and a display 503. Further, the computer 500 includes a reading device 504 that reads out, for example, a program from a storage medium, and an interface device 505 that exchanges data with, for example, a recording device via a wired or wireless network. Further in paragraph [0177]-YAMAMOTO discloses the hard disk device 507 includes a detection program 507a, a generation program 507b, a trajectory determination program 507c, and a statistical information generation program 507d. The CPU 501 reads out the programs 507a to 507d and develops the programs in the RAM 506 (wherein the programs are instructions).)
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of Co-Pending Application No. 18/783,041 claim 1 of an information processing apparatus, with the teachings of YAMAMOTO et al. (US 20190147604 A1) of wherein the one or more processors further execute the instructions to.
Wherein having Co-Pending Application No. 18/783,041 claim 1’s an information processing apparatus wherein the one or more processors further execute the instructions to.
The motivation behind the modification would have been to obtain information processing apparatus that enhances the accuracy of identifying and tracking subjects/objects of interest.
Co-Pending Application No. 18/783,041 claim 1 in view of MITSUMOTO (US 20170171570 A1) and further in view of YAMAMOTO et al. (US 20190147604 A1) fail to explicitly teach acquire condition information indicating the plurality of positions of interest corresponding to a predetermined area in a three-dimensional space, and wherein, in a case where the object is present in the predetermined area indicated by the condition information, the specified position of interest is specified from the plurality of positions of interest corresponding to the predetermined area.
However, NOMURA (US 20150091715 A1) explicitly teaches acquire condition information indicating the plurality of positions of interest corresponding to a predetermined area in a three-dimensional space (Fig. 2. Paragraph [0045]-NOMURA discloses the object identifying module 164 groups the subject parts (pixels and blocks) within the detection area 214 by using the 3D positional information based on the distance image 212 to identify an object Specifically, the object identifying module 164 selects any one of subject parts as a reference point and groups the selected subject part with other subject parts of which differences in the horizontal distance, differences in the height, and differences in the depth distance are within a predetermined range to make these subject parts to be one unitary object (wherein the plurality of positions of interest are the subject parts and wherein the predetermined area is the detection area).), and
wherein, in a case where the object is present in the predetermined area indicated by the condition information (Fig. 2. Paragraph [0045]-NOMURA discloses the object identifying module 164 groups the subject parts (pixels and blocks) within the detection area 214 by using the 3D positional information based on the distance image 212 to identify an object (wherein the subject parts comprise an object, which is within the detection area (i.e. the predetermined area)).), the specified position of interest is specified from the plurality of positions of interest corresponding to the predetermined area (Fig. 2. Paragraph [0045]-NOMURA discloses the object identifying module 164 groups the subject parts (pixels and blocks) within the detection area 214 by using the 3D positional information based on the distance image 212 to identify an object Specifically, the object identifying module 164 selects any one of subject parts as a reference point and groups the selected subject part with other subject parts of which differences in the horizontal distance, differences in the height, and differences in the depth distance are within a predetermined range to make these subject parts to be one unitary object (wherein the specified position of interest is the reference point and wherein the plurality of positions corresponding to the predetermined area are the subject parts located in the detection area).).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of Co-Pending Application No. 18/783,041 claim 1 in view of MITSUMOTO (US 20170171570 A1) and further in view of YAMAMOTO et al. (US 20190147604 A1) of an information processing apparatus, with the teachings of NOMURA (US 20150091715 A1) of acquire condition information indicating the plurality of positions of interest corresponding to a predetermined area in a three-dimensional space, and wherein, in a case where the object is present in the predetermined area indicated by the condition information, the specified position of interest is specified from the plurality of positions of interest corresponding to the predetermined area.
Wherein having Co-Pending Application No. 18/783,041 claim 1’s an information processing apparatus acquire condition information indicating the plurality of positions of interest corresponding to a predetermined area in a three-dimensional space, and wherein, in a case where the object is present in the predetermined area indicated by the condition information, the specified position of interest is specified from the plurality of positions of interest corresponding to the predetermined area.
The motivation behind the modification would have been to obtain information processing apparatus that enhances the accuracy of identifying and tracking subjects/objects of interest.
Claim 7 is rejected on the ground of non-statutory double patenting as being unpatentable over claim 1 of Co-Pending Application No. 18/783,041 in view of MITSUMOTO (US 20170171570 A1) and further in view of YAMAMOTO et al. (US 20190147604 A1).
Regarding claim 7, Co-Pending Application No. 18/783,041 claim 1 in view of MITSUMOTO (US 20170171570 A1) and further in view of YAMAMOTO et al. (US 20190147604 A1) explicitly teach the information processing apparatus according to claim 1,
Co-Pending Application No. 18/783,041 claim 1 fails to explicitly teach wherein the moving direction of the object is specified based on change of a position of the object at predetermined time.
However, YAMAMOTO explicitly teaches wherein the moving direction of the object (Fig. 15. Paragraph [0078-0079]-YAMAMOTO discloses by taking the ball candidate 19 as a reference, the ball detection unit 150b predicts a region 21 of a movement destination of the ball candidate at the time “t,” based on the shape and the speed corresponding to the event identification information defined in the movement definition information 140e. Here, as an example, it is assumed that the event identification information is preset in the ball detection unit 150b. Step S11 of FIG. 15 will be described. When no ball candidate exists in the region 21 at the time “t,” the ball detection unit 150b adds a ball candidate 20d to the region 21. The ball detection unit 150b registers information on the ball candidate 20d in the ball management table 140c. That is, the ball detection unit 150b registers candidate identification information and coordinates of the ball candidate 20d in association with the time “t” in the ball management table 140c (wherein the ball is the object moving).) is specified based on change of a position of the object at predetermined time (Fig. 15. Paragraph [0077]-YAMAMOTO discloses it is assumed that a ball candidate 18 is a ball candidate detected at a time t−2. It is assumed that a ball candidate 19 is a ball candidate detected at a time t−1. It is assumed that ball candidates 20a, 20b, and 20c are ball candidates detected at a time “t” (wherein time t is the predetermined time that the object has changed positions).).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of Co-Pending Application No. 18/783,041 claim 1 of an information processing apparatus, with the teachings of YAMAMOTO et al. (US 20190147604 A1) of wherein the moving direction of the object is specified based on change of a position of the object at predetermined time.
Wherein having Co-Pending Application No. 18/783,041 claim 1’s an information processing apparatus wherein the moving direction of the object is specified based on change of a position of the object at predetermined time.
The motivation behind the modification would have been to obtain information processing apparatus that enhances the accuracy of identifying and tracking subjects/objects of interest.
Claim 13 is rejected on the ground of non-statutory double patenting as being unpatentable over claims 1 and 2 of Co-Pending Application No. 18/783,041 in view of MITSUMOTO (US 20170171570 A1) and further in view of YAMAMOTO et al. (US 20190147604 A1), and further in view of ONUMA (US 20220383586 A1).
Regarding claim 13, claim 2 of Co-Pending Application No. 18/783,041 in view of MITSUMOTO (US 20170171570 A1) and further in view of YAMAMOTO et al. (US 20190147604 A1) explicitly teach explicitly teach the information processing apparatus according to claim 12,
Co-Pending Application No. 18/783,041 claim 2 in view of MITSUMOTO (US 20170171570 A1) and further in view of YAMAMOTO et al. (US 20190147604 A1) fail to explicitly teach wherein the position of the virtual viewpoint and the line-of-sight direction of the virtual viewpoint are determined operable to an angle formed by the half line extending from the position of the object to the specified position of interest and the line-of-sight direction of the virtual viewpoint becomes a predetermined angle.
However, ONUMA (US 20220383586 A1) explicitly teaches wherein the position of the virtual viewpoint and the line-of-sight direction of the virtual viewpoint are determined operable to an angle (Figs. 6A-6C. Paragraph [0052]-ONUMA discloses the direction and the angle of view of the virtual viewpoint may be set so as to include a direction based on two pieces of sight information after having set a single virtual viewpoint at a point (sight designation position) w % here a plurality of pieces of sight information intersect (wherein the angle of view is an angle).) formed by the half line extending from the position of the object to the specified position of interest (Figs. 6A-6C., illustrates an angle (i.e. angle of view of virtual viewpoint) formed by the half line extending from the position of the object to the position of interest (i.e. sight information, the line extending from the first object to the sight destination position). Paragraph [0052]-ONUMA discloses the direction and the angle of view of the virtual viewpoint may be set so as to include a direction based on two pieces of sight information after having set a single virtual viewpoint at a point (sight designation position) w % here a plurality of pieces of sight information intersect.) and the line-of-sight direction of the virtual viewpoint becomes a predetermined angle (Figs. 6A-6C, illustrate the line of sight direction (shown in 6A) becoming a predetermined angle (shown in 6C). Paragraph [0052]-ONUMA discloses the direction and the angle of view of the virtual viewpoint may be set so as to include a direction based on two pieces of sight information after having set a single virtual viewpoint at a point (sight designation position) w % here a plurality of pieces of sight information intersect (wherein the angle of view is the predetermined angle).).).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of Co-Pending Application No. 18/783,041 claim 2 in view of MITSUMOTO (US 20170171570 A1) and further in view of YAMAMOTO et al. (US 20190147604 A1) of an information processing apparatus, with the teachings of ONUMA (US 20220383586 A1) of wherein the position of the virtual viewpoint and the line-of-sight direction of the virtual viewpoint are determined operable to an angle formed by the half line extending from the position of the object to the specified position of interest and the line-of-sight direction of the virtual viewpoint becomes a predetermined angle.
Wherein having Co-Pending Application No. 18/783,041 claim 1’s an information processing apparatus wherein the position of the virtual viewpoint and the line-of-sight direction of the virtual viewpoint are determined operable to an angle formed by the half line extending from the position of the object to the specified position of interest and the line-of-sight direction of the virtual viewpoint becomes a predetermined angle.
The motivation behind the modification would have been to obtain information processing apparatus that enhances the accuracy of identifying and tracking subjects/objects of interest.
Claim 14 and 19 are rejected on the ground of non-statutory double patenting as being unpatentable over claim 1 of Co-Pending Application No. 18/783,041 in view of MITSUMOTO (US 20170171570 A1) and further in view of YAMAMOTO et al. (US 20190147604 A1), and further in view of MARUYAMA (US 20180246631 A1).
Regarding claim 14, Co-Pending Application No. 18/783,041 claim 1 in view of MITSUMOTO (US 20170171570 A1) and further in view of YAMAMOTO et al. (US 20190147604 A1) explicitly teach the information processing apparatus according to claim 1,
Co-Pending Application No. 18/783,041 claim 1 in view of MITSUMOTO (US 20170171570 A1) and further in view of YAMAMOTO et al. (US 20190147604 A1) explicitly teach fail to wherein the position of the virtual viewpoint and the line-of-sight direction of the virtual viewpoint are determined operable to the object and the specified position of interest are included in the virtual viewpoint image.
MARUYAMA (US 20180246631 A1) explicitly teaches wherein the position of the virtual viewpoint (Fig. 3. Paragraph [0032]-MARUYAMA discloses a movement path 7 of a virtual viewpoint is set to follow an object of interest from rear (wherein the movement path is a position).) and the line-of-sight direction of the virtual viewpoint (Fig. 3. Paragraph [0032]-MARUYAMA discloses the path generation unit 302 determines a viewpoint position and a viewing direction on a movement path of the virtual viewpoint.) are determined operable to the object (Fig. 3. Paragraph [0032]-MARUYAMA a movement path 7 of a virtual viewpoint is set to follow an object of interest from rear. In this example, the object of interest is a ball moving along a trajectory 8 toward a goal.) and the specified position of interest are included in the virtual viewpoint image (Figs. 9A-9C, illustrate virtual viewpoint images with the position of interest included. Paragraph [0060]-MARUYAMA discloses FIGS. 9A, 9B, and 9C illustrate the display examples corresponding to the virtual viewpoints 6a, 6c, and 6f, respectively (wherein the position of interest is the goal, which is included in Figs. 9A-9C).).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of Co-Pending Application No. 18/783,041 claim 1 in view of MITSUMOTO (US 20170171570 A1) and further in view of YAMAMOTO et al. (US 20190147604 A1) of an information processing apparatus, with the teachings of MARUYAMA (US 20180246631 A1) of wherein the position of the virtual viewpoint and the line-of-sight direction of the virtual viewpoint are determined operable to the object and the specified position of interest are included in the virtual viewpoint image.
Wherein having Co-Pending Application No. 18/783,041 claim 1’s an information processing apparatus wherein the position of the virtual viewpoint and the line-of-sight direction of the virtual viewpoint are determined operable to the object and the specified position of interest are included in the virtual viewpoint image.
The motivation behind the modification would have been to obtain information processing apparatus that enhances the accuracy of identifying and tracking subjects/objects of interest.
Regarding claim 19, Co-Pending Application No. 18/783,041 claim 1 in view of MITSUMOTO (US 20170171570 A1) and further in view of YAMAMOTO et al. (US 20190147604 A1) explicitly teach the information processing apparatus according to claim 1,
Co-Pending Application No. 18/783,041 claim 1 in view of MITSUMOTO (US 20170171570 A1) and further in view of YAMAMOTO et al. (US 20190147604 A1) explicitly teach fail to wherein the one or more processors identify a first position of interest among the plurality of positions of interest at a first time, and identify a second position of interest different from the first position of interest among the plurality of positions of interest at a second time after the first time, and determine the position of the virtual viewpoint that transitions, over a predetermined period of time, from a position of the virtual viewpoint determined based on the first position of interest to a position of the virtual viewpoint determined based on the second position of interest.
However, MARUYAMA (US 20180246631 A1) explicitly teaches wherein the one or more processors (Fig. 2, #211 called CPU. Paragraph [0036]) identify a first position of interest among the plurality of positions of interest at a first time (Fig. 3, #8 illustrates multiple positions of interest over the course of time, including a first and second position of interest (wherein the 6f, the first virtual viewpoint, is at a first time). Paragraph [0032]-MARUYAMA discloses in FIG. 3, a movement path 7 of a virtual viewpoint is set to follow an object of interest from rear. In this example, the object of interest is a ball moving along a trajectory 8 toward a goal. Further in paragraph [0062]-MARUYAMA discloses the position of the virtual object at the first point in time (when the virtual viewpoint 6 is located at the virtual viewpoint 6f) in the three-dimensional space is determined, based on the virtual viewpoint at the first point in time (virtual viewpoint 6f) and the virtual viewpoint at the second point in time (such as the virtual viewpoint 6a). Please see annotated Fig. 3 below.), and
identify a second position of interest different from the first position of interest among the plurality of positions of interest at a second time after the first time (Fig. 3, #8 illustrates multiple positions of interest over the course of time, including a first and second position of interest (wherein the 6a, a second virtual viewpoint, is at a second time). Paragraph [0032]-MARUYAMA discloses in FIG. 3, a movement path 7 of a virtual viewpoint is set to follow an object of interest from rear. In this example, the object of interest is a ball moving along a trajectory 8 toward a goal. Further in paragraph [0062]-MARUYAMA discloses the position of the virtual object at the first point in time (when the virtual viewpoint 6 is located at the virtual viewpoint 6f) in the three-dimensional space is determined, based on the virtual viewpoint at the first point in time (virtual viewpoint 6f) and the virtual viewpoint at the second point in time (such as the virtual viewpoint 6a). Please see annotated Fig. 3 below.), and
determine the position of the virtual viewpoint that transitions (Fig. 3, #7 called movement path. Paragraph [0031]-MARUYAMA discloses the path generation unit 302 determines a viewpoint position and a viewing direction on a movement path of the virtual viewpoint, based on the instruction. The path generation unit 302 then generates the virtual camera path, which is formed of a plurality of virtual camera parameters that is temporally consecutive. Each of the virtual camera parameters is a parameter indicating the position and the direction of the virtual viewpoint.), over a predetermined period of time, from a position of the virtual viewpoint determined based on the first position of interest to a position of the virtual viewpoint determined based on the second position of interest (Fig. 3. Paragraph [0032]-MARUYAMA discloses in FIG. 3, a movement path 7 of a virtual viewpoint is set to follow an object of interest from rear. In this example, the object of interest is a ball moving along a trajectory 8 toward a goal. The virtual camera path generated by the path generation unit 302 is, for example, information for identifying the movement path 7. The virtual camera path includes the virtual camera parameter corresponding to each of virtual viewpoints 6a to 6f on the movement path 7. Thus, in this example, the virtual camera path for identifying the virtual viewpoint for six frames is generated. Please see annotated Fig. 3 below.).
PNG
media_image1.png
595
477
media_image1.png
Greyscale
Annotated diagram of MARUYAMA’s Fig. 3 illustrating a different positions of an object over time with a transitioning virtual viewpoint on a movement path.
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of Co-Pending Application No. 18/783,041 claim 1 in view of MITSUMOTO (US 20170171570 A1) and further in view of YAMAMOTO et al. (US 20190147604 A1) of an information processing apparatus, with the teachings of MARUYAMA (US 20180246631 A1) of wherein the one or more processors identify a first position of interest among the plurality of positions of interest at a first time, and identify a second position of interest different from the first position of interest among the plurality of positions of interest at a second time after the first time, and determine the position of the virtual viewpoint that transitions, over a predetermined period of time, from a position of the virtual viewpoint determined based on the first position of interest to a position of the virtual viewpoint determined based on the second position of interest.
Wherein having Co-Pending Application No. 18/783,041 claim 1’s an information processing apparatus wherein the one or more processors identify a first position of interest among the plurality of positions of interest at a first time, and identify a second position of interest different from the first position of interest among the plurality of positions of interest at a second time after the first time, and determine the position of the virtual viewpoint that transitions, over a predetermined period of time, from a position of the virtual viewpoint determined based on the first position of interest to a position of the virtual viewpoint determined based on the second position of interest.
The motivation behind the modification would have been to obtain information processing apparatus that enhances the accuracy of identifying and tracking subjects/objects of interest.
Claim 17 is rejected on the ground of non-statutory double patenting as being unpatentable over claim 1 of Co-Pending Application No. 18/783,041 claim 1 in view of MITSUMOTO (US 20170171570 A1) and further in view of YAMAMOTO et al. (US 20190147604 A1) and further in view of GREEN et al. (US 20240062407 A1) and further in view of YEE et al. (US 20180167553 A1).
Regarding claim 17, Co-Pending Application No. 18/783,041 claim 1 in view of MITSUMOTO (US 20170171570 A1) and further in view of YAMAMOTO et al. (US 20190147604 A1) and further in view of GREEN et al. (US 20240062407 A1) explicitly teach the information processing apparatus according to claim 3,
Co-Pending Application No. 18/783,041 claim 1 in view of MITSUMOTO (US 20170171570 A1) and further in view of YAMAMOTO et al. (US 20190147604 A1) and further in view of GREEN et al. (US 20240062407 A1) fail to explicitly teach wherein the one or more processors calculate an average velocity vector of velocity vectors of the plurality of objects, and specify the one position of interest based on the average velocity vector.
However, YEE et al. (US 20180167553 A1) explicitly teaches wherein the one or more processors (Fig. 8A-8B. Paragraph [0067]-YEE discloses the processor 805 is given a set of instructions which are executed therein. The processor 805 waits for a subsequent input, to which the processor 805 reacts to by executing another set of instructions.) calculate an average velocity vector (Fig. 6. Paragraph [0083]-YEE discloses a process may be used at step 320 to determine a virtual camera position for which group velocity (i.e., the average velocity of the subjects including the ball 240, and the players 220 and 230) is directed both towards the virtual camera and across the field of view of the virtual camera from left to right or right to left.) of velocity vectors of the plurality of objects (Fig. 6. Paragraph [0082]-YEE discloses the locations of the players 220 and 230 may be determined, for example, by detecting a first object representing the player 220 and a second object representing the player 230 in the scene, where each object has at least one motion attribute including, for example, velocity and trajectory as described below.), and
specify the one position of interest based on the average velocity vector (Fig. 3 and 6. Paragraph [0083]-YEE discloses a process may be used at step 320 to determine a virtual camera position for which group velocity (i.e., the average velocity of the subjects including the ball 240, and the players 220 and 230) is directed both towards the virtual camera and across the field of view of the virtual camera from left to right or right to left (wherein the virtual camera position is the position of interest).).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of Co-Pending Application No. 18/783,041 claim 1 in view of MITSUMOTO (US 20170171570 A1) and further in view of YAMAMOTO et al. (US 20190147604 A1) of an information processing apparatus, with the teachings of YEE et al. (US 20180167553 A1) of wherein the one or more processors calculate an average velocity vector of velocity vectors of the plurality of objects, and specify the one position of interest based on the average velocity vector.
Wherein having Co-Pending Application No. 18/783,041 claim 1’s an information processing apparatus wherein the one or more processors calculate an average velocity vector of velocity vectors of the plurality of objects, and specify the one position of interest based on the average velocity vector.
The motivation behind the modification would have been to obtain information processing apparatus that enhances the accuracy of identifying and tracking subjects/objects of interest.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 7-8, 11, 14-16, and 19 are rejected under 35 U.S.C. 103 as being unpatentable over MARUYAMA (US 20180246631 A1), hereinafter referenced as MARUYAMA, in view of YAMAMOTO et al. (US 20190147604 A1), hereinafter referenced as YAMAMOTO, and further in view of MITSUMOTO (US 20170171570 A1), hereinafter referenced as MITSUMOTO.
Regarding claim 1, MARUYAMA explicitly teaches an information processing apparatus comprising (Fig. 2, #2 called image processing apparatus. Paragraph [0022]-MARUYAMA discloses the image processing apparatus 2 includes a virtual viewpoint image generation unit 201 (hereinafter referred to as image generation unit 201), a virtual object placement determination unit 202 (hereinafter referred to as placement determination unit 202), and a virtual object modeling unit 203 (hereinafter referred to as modeling unit 203).):
one or more memories (Fig. 2, illustrates a read only memory (ROM) 212, a random access memory (RAM) 213, an auxiliary storage device 214. Paragraph [0036]) storing instructions (Fig. 2. Paragraph [0036]-MARUYAMA discloses the CPU 211 controls the entire image processing apparatus 2, by using a computer program and data stored in the ROM 212 or the RAM 213 (wherein a computer program is instructions).); and
one or more processors (Fig. 2, #211 called CPU. Paragraph [0036]) executing the instructions to (Fig. 2. Paragraph [0036]-MARUYAMA discloses the CPU 211 controls the entire image processing apparatus 2, by using a computer program and data stored in the ROM 212 or the RAM 213 (wherein a computer program is instructions). Further in paragraph [0083]-MARUYAMA discloses the computer may comprise one or more processors (e.g., central processing unit (CPU), micro processing unit (MPU)) and may include a network of separate computers or separate processors to read out and execute the computer executable instructions.);
determine a position of a virtual viewpoint (Fig. 3 and Fig. 9A-9C. Paragraph [0032]-MARUYAMA discloses a movement path 7 of a virtual viewpoint is set to follow an object of interest from rear (wherein the movement path is a position).) and a line-of-sight direction of the virtual viewpoint (Fig. 3. Paragraph [0032]-MARUYAMA discloses the path generation unit 302 determines a viewpoint position and a viewing direction on a movement path of the virtual viewpoint.),
MARUYAMA fails to explicitly teach acquire position information indicating a position of an object included in an image capturing area captured by a plurality of image capturing apparatuses; specify one position of interest from among a plurality of positions of interest, based on a moving direction of the object; and.
However, YAMAMOTO explicitly teaches acquire position information (Fig. 8. Paragraph [0060]-YAMAMOTO the table 143a of the ball management table 140c holds information on coordinates of a ball candidate detected at each time. In the table 143a, time, the number of candidates, candidate identification information, and coordinates are associated with each other (wherein coordinates of a ball candidate is position information).) indicating a position of an object included in an image capturing area (Fig. 8. Paragraph [0060]-YAMAMOTO discloses the number of candidates indicates the number of ball candidates included in each of image frames captured by the cameras 10a to 10d at the same time. The candidate identification information is information for uniquely identifying a ball candidate on an image frame. The coordinates are information indicating coordinates (global coordinates) of a ball. When multiple ball candidates are detected at the same time, multiple sets each including the candidate identification information and the coordinates are registered in the same record (wherein an image frame is an image capturing area).) captured by a plurality of image capturing apparatuses (Fig. 1, illustrates the plurality of image capturing apparatuses, called camera 10a-10c. Paragraph [0060]-YAMAMOTO discloses the number of candidates indicates the number of ball candidates included in each of image frames captured by the cameras 10a to 10d at the same time (wherein a camera is an image capturing apparatus).);
specify one position of interest (Fig. 15. Paragraph [0079]-YAMAMOTO discloses the ball detection unit 150b registers information on the ball candidate 20d in the ball management table 140c. That is, the ball detection unit 150b registers candidate identification information and coordinates of the ball candidate 20d in association with the time “t” in the ball management table 140c (wherein ball candidate 20d is one position of interest).) from among a plurality of positions of interest (Fig. 15. Paragraph [0077]-YAMAMOTO discloses it is assumed that a ball candidate 18 is a ball candidate detected at a time t−2. It is assumed that a ball candidate 19 is a ball candidate detected at a time t−1. It is assumed that ball candidates 20a, 20b, and 20c are ball candidates detected at a time “t” (wherein ball candidates 20a-c are a plurality of positions of interest).), based on a moving direction of the object (Fig. 15. Paragraph [0078-0079]-YAMAMOTO discloses by taking the ball candidate 19 as a reference, the ball detection unit 150b predicts a region 21 of a movement destination of the ball candidate at the time “t,” based on the shape and the speed corresponding to the event identification information defined in the movement definition information 140e. Here, as an example, it is assumed that the event identification information is preset in the ball detection unit 150b. Step S11 of FIG. 15 will be described. When no ball candidate exists in the region 21 at the time “t,” the ball detection unit 150b adds a ball candidate 20d to the region 21. The ball detection unit 150b registers information on the ball candidate 20d in the ball management table 140c. That is, the ball detection unit 150b registers candidate identification information and coordinates of the ball candidate 20d in association with the time “t” in the ball management table 140c (wherein the ball is the object moving).); and
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of an information processing apparatus comprising: one or more memories storing instructions; and one or more processors executing the instructions to; determine a position of a virtual viewpoint and a line-of-sight direction of the virtual viewpoint with the teachings of YAMAMOTO of acquire position information indicating a position of an object included in an image capturing area captured by a plurality of image capturing apparatuses; specify one position of interest from among a plurality of positions of interest, based on a moving direction of the object; and.
Wherein having MARUYAMA’s apparatus for generating a virtual viewpoint and virtual viewpoint image acquire position information indicating a position of an object included in an image capturing area captured by a plurality of image capturing apparatuses; specify one position of interest from among a plurality of positions of interest, based on a moving direction of the object; and.
The motivation behind the modification would have been to obtain an apparatus for generating a virtual viewpoint and virtual viewpoint image that enhances the accuracy of identifying and tracking subjects/objects of interest. Since both MARUYAMA and YAMAMOTO relate to tracking subjects/objects on a sports/athletics field or court, wherein MARUYAMA can reduce a possibility that the viewer may have difficulty viewing a subject of interest in the virtual viewpoint image, while YAMAMOTO the continuity of the tracking may be improved. Please see MARUYAMA (US 20180246631 A1), Paragraph [0079], and YAMAMOTO et al. (US 20190147604 A1), Paragraph [0174].
MARUYAMA in view of YAMAMOTO fail to explicitly teach acquire position information indicating a plurality of positions of interest, wherein each of the plurality of positions of interest corresponds to one of a plurality of stationary objects; such that a stationary object corresponding to the specified position of interest and the object are included in a virtual viewpoint image.
However, MITSUMOTO explicitly teaches acquire position information indicating a plurality of positions of interest (Fig. 12A-12B. Paragraph [0048]-MITSUMOTO discloses positions of the goal 105 and the goal 106 in the field coordinate system (certain positions in the areas of the respective goals) are determined in advance. Further in paragraph [0072]-MITSUMOTO discloses since a position of each goal is a certain position in the area of the goal, the center of the width of the goal may be the position of the goal as illustrated in FIG. 12A, or a position of an end of the width of the goal may be the position of the goal as illustrated in FIG. 12B (wherein the goals are positions of interest).),
wherein each of the plurality of positions of interest corresponds to one of a plurality of stationary objects (Fig. 12A-12B. Paragraph [0048]-MITSUMOTO discloses positions of the goal 105 and the goal 106 in the field coordinate system (certain positions in the areas of the respective goals) are determined in advance. Further in paragraph [0072]-MITSUMOTO discloses since a position of each goal is a certain position in the area of the goal, the center of the width of the goal may be the position of the goal as illustrated in FIG. 12A, or a position of an end of the width of the goal may be the position of the goal as illustrated in FIG. 12B (wherein goals are stationary objects).);
such that a stationary object corresponding to the specified position of interest and the object are included in a virtual viewpoint image (Fig. 11A-11C, illustrate a position of interest and the object included in a virtual viewpoint image (wherein the position of interest is the goal and the object is the ball). Paragraph [0101]-MITSUMOTO discloses a first information processing apparatus 201 decides on a camera (or a virtual viewpoint) on the basis of a positional relationship between a first predetermined object (for example, a ball) and a second predetermined object (for example, a goal) and outputs the decision to a second information processing apparatus 201.).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of MARUYAMA in view of YAMAMOTO of an information processing apparatus comprising: one or more memories storing instructions; and one or more processors executing the instructions to; determine a position of a virtual viewpoint and a line-of-sight direction of the virtual viewpoint with the teachings of MITSUMOTO of acquire position information indicating a plurality of positions of interest, wherein each of the plurality of positions of interest corresponds to one of a plurality of stationary objects; such that a stationary object corresponding to the specified position of interest and the object are included in a virtual viewpoint image.
Wherein having MARUYAMA’s apparatus for generating a virtual viewpoint and virtual viewpoint image acquire position information indicating a plurality of positions of interest, wherein each of the plurality of positions of interest corresponds to one of a plurality of stationary objects; such that a stationary object corresponding to the specified position of interest and the object are included in a virtual viewpoint image.
The motivation behind the modification would have been to obtain an apparatus for generating a virtual viewpoint and virtual viewpoint image that enhances the accuracy of identifying and tracking subjects/objects of interest. Since both MARUYAMA and MITSUMOTO relate to tracking subjects/objects on a sports/athletics field or court and determining virtual viewpoints, wherein MARUYAMA can reduce a possibility that the viewer may have difficulty viewing a subject of interest in the virtual viewpoint image, while MITSUMOTO the lag in switching is reduced so that a video image is not distorted. Please see MARUYAMA (US 20180246631 A1), Paragraph [0079], and MITSUMOTO (US 20170171570 A1), Paragraph [0054].
Regarding claim 7, MARUYAMA in view of YAMAMOTO and further in view of MITSUMOTO explicitly teach the information processing apparatus according to claim 1,
MARUYAMA fails to explicitly teach wherein the moving direction of the object is specified based on change of a position of the object at predetermined time.
However, YAMAMOTO explicitly teaches wherein the moving direction of the object (Fig. 15. Paragraph [0078-0079]-YAMAMOTO discloses by taking the ball candidate 19 as a reference, the ball detection unit 150b predicts a region 21 of a movement destination of the ball candidate at the time “t,” based on the shape and the speed corresponding to the event identification information defined in the movement definition information 140e. Here, as an example, it is assumed that the event identification information is preset in the ball detection unit 150b. Step S11 of FIG. 15 will be described. When no ball candidate exists in the region 21 at the time “t,” the ball detection unit 150b adds a ball candidate 20d to the region 21. The ball detection unit 150b registers information on the ball candidate 20d in the ball management table 140c. That is, the ball detection unit 150b registers candidate identification information and coordinates of the ball candidate 20d in association with the time “t” in the ball management table 140c (wherein the ball is the object moving).) is specified based on change of a position of the object at predetermined time (Fig. 15. Paragraph [0077]-YAMAMOTO discloses it is assumed that a ball candidate 18 is a ball candidate detected at a time t−2. It is assumed that a ball candidate 19 is a ball candidate detected at a time t−1. It is assumed that ball candidates 20a, 20b, and 20c are ball candidates detected at a time “t” (wherein time t is the predetermined time that the object has changed positions).).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of MARUYAMA of an information processing apparatus comprising: one or more memories storing instructions; and one or more processors executing the instructions to; determine a position of a virtual viewpoint and a line-of-sight direction of the virtual viewpoint with the teachings of YAMAMOTO of wherein the moving direction of the object is specified based on change of a position of the object at predetermined time.
Wherein having MARUYAMA’s apparatus for generating a virtual viewpoint and virtual viewpoint image wherein the moving direction of the object is specified based on change of a position of the object at predetermined time.
The motivation behind the modification would have been to obtain an apparatus for generating a virtual viewpoint and virtual viewpoint image that enhances the accuracy of identifying and tracking subjects/objects of interest. Since both MARUYAMA and YAMAMOTO relate to tracking subjects/objects on a sports/athletics field or court, wherein MARUYAMA can reduce a possibility that the viewer may have difficulty viewing a subject of interest in the virtual viewpoint image, while YAMAMOTO the continuity of the tracking may be improved. Please see MARUYAMA (US 20180246631 A1), Paragraph [0079], and YAMAMOTO et al. (US 20190147604 A1), Paragraph [0174].
Regarding claim 8, MARUYAMA in view of YAMAMOTO and further in view of MITSUMOTO explicitly teach the information processing apparatus according to claim 1,
MARUYAMA further explicitly teaches wherein the specified position of interest is set based on a user operation (Fig. 3, illustrates #6a, 6d, 6f are positions specified by the user. Paragraph [0032]-MARUYAMA discloses when specifying the movement path 7 via the editing unit 301, the user may specify a line indicating the movement path 7, or may specify some virtual viewpoints (e.g., virtual viewpoints 6a, 6d, and 6f) on a movement path, as points (wherein a point on a movement path is a position of interest).).
Regarding claim 11, MARUYAMA in view of YAMAMOTO and further in view of MITSUMOTO explicitly teach the information processing apparatus according to claim 1,
MARUYAMA is silent to the term basket.
However, YAMAMOTO explicitly teaches wherein the stationary object is a basket goal (Fig. 1, illustrates a basketball court (called game area #5) with basket goals on the baselines. Paragraph [0043]-YAMAMOTO discloses the game area 5 is divided and images of the divided areas are captured using the cameras 10a to 10d. Please see annotated Fig. 1 below.).
PNG
media_image2.png
312
459
media_image2.png
Greyscale
Annotated diagram of YAMAMOTO’s Fig. 1 illustrating a basketball court with basketball hoops (i.e. basket goals) as background models
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of MARUYAMA of an information processing apparatus comprising: one or more memories storing instructions; and one or more processors executing the instructions to; determine a position of a virtual viewpoint and a line-of-sight direction of the virtual viewpoint with the teachings of YAMAMOTO of wherein the stationary object is a basket goal.
Wherein having MARUYAMA’s apparatus for generating a virtual viewpoint and virtual viewpoint image wherein the stationary object is a basket goal.
The motivation behind the modification would have been to obtain an apparatus for generating a virtual viewpoint and virtual viewpoint image that enhances the accuracy of identifying and tracking subjects/objects of interest. Since both MARUYAMA and YAMAMOTO relate to tracking subjects/objects on a sports/athletics field or court, wherein MARUYAMA can reduce a possibility that the viewer may have difficulty viewing a subject of interest in the virtual viewpoint image, while YAMAMOTO the continuity of the tracking may be improved. Please see MARUYAMA (US 20180246631 A1), Paragraph [0079], and YAMAMOTO et al. (US 20190147604 A1), Paragraph [0174].
Regarding claim 14, MARUYAMA in view of YAMAMOTO and further in view of MITSUMOTO explicitly teach the information processing apparatus according to claim 1,
MARUYAMA further explicitly teaches wherein the position of the virtual viewpoint (Fig. 3. Paragraph [0032]-MARUYAMA discloses a movement path 7 of a virtual viewpoint is set to follow an object of interest from rear (wherein the movement path is a position).) and the line-of-sight direction of the virtual viewpoint (Fig. 3. Paragraph [0032]-MARUYAMA discloses the path generation unit 302 determines a viewpoint position and a viewing direction on a movement path of the virtual viewpoint.) are determined operable to the object (Fig. 3. Paragraph [0032]-MARUYAMA a movement path 7 of a virtual viewpoint is set to follow an object of interest from rear. In this example, the object of interest is a ball moving along a trajectory 8 toward a goal.) and the specified position of interest are included in the virtual viewpoint image (Figs. 9A-9C, illustrate virtual viewpoint images with the position of interest included. Paragraph [0060]-MARUYAMA discloses FIGS. 9A, 9B, and 9C illustrate the display examples corresponding to the virtual viewpoints 6a, 6c, and 6f, respectively (wherein the position of interest is the goal, which is included in Figs. 9A-9C).).
Regarding claim 15, MARUYAMA explicitly teaches an information processing method comprising (Fig. 1. Paragraph [0032]-MARUYAMA discloses a method for generating the virtual viewpoint image by the image processing apparatus 2):
determining a position of a virtual viewpoint (Fig. 3 and Fig. 9A-9C. Paragraph [0032]-MARUYAMA discloses a movement path 7 of a virtual viewpoint is set to follow an object of interest from rear (wherein the movement path is a position).) and a line-of-sight direction of the virtual viewpoint (Fig. 3. Paragraph [0032]-MARUYAMA discloses the path generation unit 302 determines a viewpoint position and a viewing direction on a movement path of the virtual viewpoint.),
MARUYAMA fails to explicitly teach acquiring position information indicating a position of an object included in an image capturing area captured by a plurality of image capturing apparatuses; specifying one position of interest from among a plurality of positions of interest, based on a moving direction of the object; and.
However, YAMAMOTO explicitly teaches acquiring position information (Fig. 8. Paragraph [0060]-YAMAMOTO the table 143a of the ball management table 140c holds information on coordinates of a ball candidate detected at each time. In the table 143a, time, the number of candidates, candidate identification information, and coordinates are associated with each other (wherein coordinates of a ball candidate is position information).) indicating a position of an object included in an image capturing area (Fig. 8. Paragraph [0060]-YAMAMOTO discloses the number of candidates indicates the number of ball candidates included in each of image frames captured by the cameras 10a to 10d at the same time. The candidate identification information is information for uniquely identifying a ball candidate on an image frame. The coordinates are information indicating coordinates (global coordinates) of a ball. When multiple ball candidates are detected at the same time, multiple sets each including the candidate identification information and the coordinates are registered in the same record (wherein an image frame is an image capturing area).) captured by a plurality of image capturing apparatuses (Fig. 1, illustrates the plurality of image capturing apparatuses, called camera 10a-10c. Paragraph [0060]-YAMAMOTO discloses the number of candidates indicates the number of ball candidates included in each of image frames captured by the cameras 10a to 10d at the same time (wherein a camera is an image capturing apparatus).);
specifying one position of interest (Fig. 15. Paragraph [0079]-YAMAMOTO discloses the ball detection unit 150b registers information on the ball candidate 20d in the ball management table 140c. That is, the ball detection unit 150b registers candidate identification information and coordinates of the ball candidate 20d in association with the time “t” in the ball management table 140c (wherein ball candidate 20d is one position of interest).) from among a plurality of positions of interest (Fig. 15. Paragraph [0077]-YAMAMOTO discloses it is assumed that a ball candidate 18 is a ball candidate detected at a time t−2. It is assumed that a ball candidate 19 is a ball candidate detected at a time t−1. It is assumed that ball candidates 20a, 20b, and 20c are ball candidates detected at a time “t” (wherein ball candidates 20a-c are a plurality of positions of interest).), based on a moving direction of the object (Fig. 15. Paragraph [0078-0079]-YAMAMOTO discloses by taking the ball candidate 19 as a reference, the ball detection unit 150b predicts a region 21 of a movement destination of the ball candidate at the time “t,” based on the shape and the speed corresponding to the event identification information defined in the movement definition information 140e. Here, as an example, it is assumed that the event identification information is preset in the ball detection unit 150b. Step S11 of FIG. 15 will be described. When no ball candidate exists in the region 21 at the time “t,” the ball detection unit 150b adds a ball candidate 20d to the region 21. The ball detection unit 150b registers information on the ball candidate 20d in the ball management table 140c. That is, the ball detection unit 150b registers candidate identification information and coordinates of the ball candidate 20d in association with the time “t” in the ball management table 140c (wherein the ball is the object moving).); and
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of MARUYAMA of an information processing method comprising: determining a position of a virtual viewpoint and a line-of-sight direction of the virtual viewpoint with the teachings of YAMAMOTO of acquiring position information indicating a position of an object included in an image capturing area captured by a plurality of image capturing apparatuses; specifying one position of interest from among a plurality of positions of interest, based on a moving direction of the object; and.
Wherein having MARUYAMA’s method for generating a virtual viewpoint and virtual viewpoint image acquiring position information indicating a position of an object included in an image capturing area captured by a plurality of image capturing apparatuses; specifying one position of interest from among a plurality of positions of interest, based on a moving direction of the object; and.
The motivation behind the modification would have been to obtain a method for generating a virtual viewpoint and virtual viewpoint image that enhances the accuracy of identifying and tracking subjects/objects of interest. Since both MARUYAMA and YAMAMOTO relate to tracking subjects/objects on a sports/athletics field or court, wherein MARUYAMA can reduce a possibility that the viewer may have difficulty viewing a subject of interest in the virtual viewpoint image, while YAMAMOTO the continuity of the tracking may be improved. Please see MARUYAMA (US 20180246631 A1), Paragraph [0079], and YAMAMOTO et al. (US 20190147604 A1), Paragraph [0174].
MARUYAMA in view of YAMAMOTO fail to explicitly teach acquiring position information indicating a plurality of positions of interest, wherein each of the plurality of positions of interest corresponds to one of a plurality of stationary objects; such that a stationary object corresponding to the specified position of interest and the object are included in a virtual viewpoint image.
However, MITSUMOTO explicitly teaches acquiring position information indicating a plurality of positions of interest (Fig. 12A-12B. Paragraph [0048]-MITSUMOTO discloses positions of the goal 105 and the goal 106 in the field coordinate system (certain positions in the areas of the respective goals) are determined in advance. Further in paragraph [0072]-MITSUMOTO discloses since a position of each goal is a certain position in the area of the goal, the center of the width of the goal may be the position of the goal as illustrated in FIG. 12A, or a position of an end of the width of the goal may be the position of the goal as illustrated in FIG. 12B (wherein the goals are positions of interest).),
wherein each of the plurality of positions of interest corresponds to one of a plurality of stationary objects (Fig. 12A-12B. Paragraph [0048]-MITSUMOTO discloses positions of the goal 105 and the goal 106 in the field coordinate system (certain positions in the areas of the respective goals) are determined in advance. Further in paragraph [0072]-MITSUMOTO discloses since a position of each goal is a certain position in the area of the goal, the center of the width of the goal may be the position of the goal as illustrated in FIG. 12A, or a position of an end of the width of the goal may be the position of the goal as illustrated in FIG. 12B (wherein goals are stationary objects).);
such that a stationary object corresponding to the specified position of interest and the object are included in a virtual viewpoint image (Fig. 11A-11C, illustrate a position of interest and the object included in a virtual viewpoint image (wherein the position of interest is the goal and the object is the ball). Paragraph [0101]-MITSUMOTO discloses a first information processing apparatus 201 decides on a camera (or a virtual viewpoint) on the basis of a positional relationship between a first predetermined object (for example, a ball) and a second predetermined object (for example, a goal) and outputs the decision to a second information processing apparatus 201.).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of MARUYAMA in view of YAMAMOTO of an information processing method comprising: determining a position of a virtual viewpoint and a line-of-sight direction of the virtual viewpoint with the teachings of MITSUMOTO of acquiring position information indicating a plurality of positions of interest, wherein each of the plurality of positions of interest corresponds to one of a plurality of stationary objects; such that a stationary object corresponding to the specified position of interest and the object are included in a virtual viewpoint image.
Wherein having MARUYAMA’s apparatus for generating a virtual viewpoint and virtual viewpoint image acquiring position information indicating a plurality of positions of interest, wherein each of the plurality of positions of interest corresponds to one of a plurality of stationary objects; such that a stationary object corresponding to the specified position of interest and the object are included in a virtual viewpoint image.
The motivation behind the modification would have been to obtain an apparatus for generating a virtual viewpoint and virtual viewpoint image that enhances the accuracy of identifying and tracking subjects/objects of interest. Since both MARUYAMA and MITSUMOTO relate to tracking subjects/objects on a sports/athletics field or court and determining virtual viewpoints, wherein MARUYAMA can reduce a possibility that the viewer may have difficulty viewing a subject of interest in the virtual viewpoint image, while MITSUMOTO the lag in switching is reduced so that a video image is not distorted. Please see MARUYAMA (US 20180246631 A1), Paragraph [0079], and MITSUMOTO (US 20170171570 A1), Paragraph [0054].
Regarding claim 16, MARUYAMA explicitly teaches a non-transitory computer-readable storage medium (Fig. 2. Paragraph [0083]-MARUYAMA discloses the storage medium may include, for example, one or more of a hard disk, a random-access memory (RAM), a read only memory (ROM), a storage of distributed computing systems, an optical disk (such as a compact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™), a flash memory device, a memory card, and the like.) storing a computer program for causing a computer to execute an image processing method comprising (Fig. 2. Paragraph [0083]-MARUYAMA discloses the computer may comprise one or more processors (e.g., central processing unit (CPU), micro processing unit (MPU)) and may include a network of separate computers or separate processors to read out and execute the computer executable instructions. The computer executable instructions may be provided to the computer, for example, from a network or the storage medium.):
determining a position of a virtual viewpoint (Fig. 3 and Fig. 9A-9C. Paragraph [0032]-MARUYAMA discloses a movement path 7 of a virtual viewpoint is set to follow an object of interest from rear (wherein the movement path is a position).) and a line-of-sight direction of the virtual viewpoint (Fig. 3. Paragraph [0032]-MARUYAMA discloses the path generation unit 302 determines a viewpoint position and a viewing direction on a movement path of the virtual viewpoint.),
MARUYAMA fails to explicitly teach acquiring position information indicating a position of an object included in an image capturing area captured by a plurality of image capturing apparatuses; specifying one position of interest from among a plurality of positions of interest, based on a moving direction of the object; and
However, YAMAMOTO explicitly teaches acquire position information (Fig. 8. Paragraph [0060]-YAMAMOTO the table 143a of the ball management table 140c holds information on coordinates of a ball candidate detected at each time. In the table 143a, time, the number of candidates, candidate identification information, and coordinates are associated with each other (wherein coordinates of a ball candidate is position information).) indicating a position of an object included in an image capturing area (Fig. 8. Paragraph [0060]-YAMAMOTO discloses the number of candidates indicates the number of ball candidates included in each of image frames captured by the cameras 10a to 10d at the same time. The candidate identification information is information for uniquely identifying a ball candidate on an image frame. The coordinates are information indicating coordinates (global coordinates) of a ball. When multiple ball candidates are detected at the same time, multiple sets each including the candidate identification information and the coordinates are registered in the same record (wherein an image frame is an image capturing area).) captured by a plurality of image capturing apparatuses (Fig. 1, illustrates the plurality of image capturing apparatuses, called camera 10a-10c. Paragraph [0060]-YAMAMOTO discloses the number of candidates indicates the number of ball candidates included in each of image frames captured by the cameras 10a to 10d at the same time (wherein a camera is an image capturing apparatus).);
specifying one position of interest (Fig. 15. Paragraph [0079]-YAMAMOTO discloses the ball detection unit 150b registers information on the ball candidate 20d in the ball management table 140c. That is, the ball detection unit 150b registers candidate identification information and coordinates of the ball candidate 20d in association with the time “t” in the ball management table 140c (wherein ball candidate 20d is one position of interest).) from among a plurality of positions of interest (Fig. 15. Paragraph [0077]-YAMAMOTO discloses it is assumed that a ball candidate 18 is a ball candidate detected at a time t−2. It is assumed that a ball candidate 19 is a ball candidate detected at a time t−1. It is assumed that ball candidates 20a, 20b, and 20c are ball candidates detected at a time “t” (wherein ball candidates 20a-c are a plurality of positions of interest).), based on a moving direction of the object (Fig. 15. Paragraph [0078-0079]-YAMAMOTO discloses by taking the ball candidate 19 as a reference, the ball detection unit 150b predicts a region 21 of a movement destination of the ball candidate at the time “t,” based on the shape and the speed corresponding to the event identification information defined in the movement definition information 140e. Here, as an example, it is assumed that the event identification information is preset in the ball detection unit 150b. Step S11 of FIG. 15 will be described. When no ball candidate exists in the region 21 at the time “t,” the ball detection unit 150b adds a ball candidate 20d to the region 21. The ball detection unit 150b registers information on the ball candidate 20d in the ball management table 140c. That is, the ball detection unit 150b registers candidate identification information and coordinates of the ball candidate 20d in association with the time “t” in the ball management table 140c (wherein the ball is the object moving).); and
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of MARUYAMA of a non-transitory computer-readable storage medium storing a computer program for causing a computer to execute an image processing method comprising: determining a position of a virtual viewpoint and a line-of-sight direction of the virtual viewpoint with the teachings of YAMAMOTO of acquiring position information indicating a position of an object included in an image capturing area captured by a plurality of image capturing apparatuses; specifying one position of interest from among a plurality of positions of interest, based on a moving direction of the object; and.
Wherein having MARUYAMA’s apparatus for generating a virtual viewpoint and virtual viewpoint image acquiring position information indicating a position of an object included in an image capturing area captured by a plurality of image capturing apparatuses; specifying one position of interest from among a plurality of positions of interest, based on a moving direction of the object; and.
The motivation behind the modification would have been to obtain an apparatus for generating a virtual viewpoint and virtual viewpoint image that enhances the accuracy of identifying and tracking subjects/objects of interest. Since both MARUYAMA and YAMAMOTO relate to tracking subjects/objects on a sports/athletics field or court, wherein MARUYAMA can reduce a possibility that the viewer may have difficulty viewing a subject of interest in the virtual viewpoint image, while YAMAMOTO the continuity of the tracking may be improved. Please see MARUYAMA (US 20180246631 A1), Paragraph [0079], and YAMAMOTO et al. (US 20190147604 A1), Paragraph [0174].
MARUYAMA in view of YAMAMOTO fail to explicitly teach acquiring position information indicating a plurality of positions of interest, wherein each of the plurality of positions of interest corresponds to one of a plurality of stationary objects; specifying one position of interest from among a plurality of positions of interest, based on a moving direction of the object; and such that a stationary object corresponding to the specified position of interest and the object are included in a virtual viewpoint image.
However, MITSUMOTO explicitly teaches acquiring position information indicating a plurality of positions of interest (Fig. 12A-12B. Paragraph [0048]-MITSUMOTO discloses positions of the goal 105 and the goal 106 in the field coordinate system (certain positions in the areas of the respective goals) are determined in advance. Further in paragraph [0072]-MITSUMOTO discloses since a position of each goal is a certain position in the area of the goal, the center of the width of the goal may be the position of the goal as illustrated in FIG. 12A, or a position of an end of the width of the goal may be the position of the goal as illustrated in FIG. 12B (wherein the goals are positions of interest).),
wherein each of the plurality of positions of interest corresponds to one of a plurality of stationary objects (Fig. 12A-12B. Paragraph [0048]-MITSUMOTO discloses positions of the goal 105 and the goal 106 in the field coordinate system (certain positions in the areas of the respective goals) are determined in advance. Further in paragraph [0072]-MITSUMOTO discloses since a position of each goal is a certain position in the area of the goal, the center of the width of the goal may be the position of the goal as illustrated in FIG. 12A, or a position of an end of the width of the goal may be the position of the goal as illustrated in FIG. 12B (wherein goals are stationary objects).); ;
such that a stationary object corresponding to the specified position of interest and the object are included in a virtual viewpoint image (Fig. 11A-11C, illustrate a position of interest and the object included in a virtual viewpoint image (wherein the position of interest is the goal and the object is the ball). Paragraph [0101]-MITSUMOTO discloses a first information processing apparatus 201 decides on a camera (or a virtual viewpoint) on the basis of a positional relationship between a first predetermined object (for example, a ball) and a second predetermined object (for example, a goal) and outputs the decision to a second information processing apparatus 201.).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of MARUYAMA in view of YAMAMOTO of a non-transitory computer-readable storage medium storing a computer program for causing a computer to execute an image processing method comprising: determining a position of a virtual viewpoint and a line-of-sight direction of the virtual viewpoint with the teachings of MITSUMOTO of acquiring position information indicating a plurality of positions of interest, wherein each of the plurality of positions of interest corresponds to one of a plurality of stationary objects; specifying one position of interest from among a plurality of positions of interest, based on a moving direction of the object; and such that a stationary object corresponding to the specified position of interest and the object are included in a virtual viewpoint image.
Wherein having MARUYAMA’s apparatus for generating a virtual viewpoint and virtual viewpoint image acquiring position information indicating a plurality of positions of interest, wherein each of the plurality of positions of interest corresponds to one of a plurality of stationary objects; specifying one position of interest from among a plurality of positions of interest, based on a moving direction of the object; and such that a stationary object corresponding to the specified position of interest and the object are included in a virtual viewpoint image.
The motivation behind the modification would have been to obtain an apparatus for generating a virtual viewpoint and virtual viewpoint image that enhances the accuracy of identifying and tracking subjects/objects of interest. Since both MARUYAMA and MITSUMOTO relate to tracking subjects/objects on a sports/athletics field or court and determining virtual viewpoints, wherein MARUYAMA can reduce a possibility that the viewer may have difficulty viewing a subject of interest in the virtual viewpoint image, while MITSUMOTO the lag in switching is reduced so that a video image is not distorted. Please see MARUYAMA (US 20180246631 A1), Paragraph [0079], and MITSUMOTO (US 20170171570 A1), Paragraph [0054].
Regarding claim 19, MARUYAMA in view of YAMAMOTO and further in view of MITSUMOTO explicitly teach the information processing apparatus according to claim 1,
MARUYAMA further explicitly teaches wherein the one or more processors (Fig. 2. Paragraph [0083]-MARUYAMA discloses the computer may comprise one or more processors (e.g., central processing unit (CPU), micro processing unit (MPU)) and may include a network of separate computers or separate processors to read out and execute the computer executable instructions. The computer executable instructions may be provided to the computer, for example, from a network or the storage medium.) identify a first position of interest among the plurality of positions of interest at a first time (Fig. 3, #8 illustrates multiple positions of interest over the course of time, including a first and second position of interest (wherein the 6f, the first virtual viewpoint, is at a first time). Paragraph [0032]-MARUYAMA discloses in FIG. 3, a movement path 7 of a virtual viewpoint is set to follow an object of interest from rear. In this example, the object of interest is a ball moving along a trajectory 8 toward a goal. Further in paragraph [0062]-MARUYAMA discloses the position of the virtual object at the first point in time (when the virtual viewpoint 6 is located at the virtual viewpoint 6f) in the three-dimensional space is determined, based on the virtual viewpoint at the first point in time (virtual viewpoint 6f) and the virtual viewpoint at the second point in time (such as the virtual viewpoint 6a). Please see annotated Fig. 3 below.), and
identify a second position of interest different from the first position of interest among the plurality of positions of interest at a second time after the first time (Fig. 3, #8 illustrates multiple positions of interest over the course of time, including a first and second position of interest (wherein the 6a, a second virtual viewpoint, is at a second time). Paragraph [0032]-MARUYAMA discloses in FIG. 3, a movement path 7 of a virtual viewpoint is set to follow an object of interest from rear. In this example, the object of interest is a ball moving along a trajectory 8 toward a goal. Further in paragraph [0062]-MARUYAMA discloses the position of the virtual object at the first point in time (when the virtual viewpoint 6 is located at the virtual viewpoint 6f) in the three-dimensional space is determined, based on the virtual viewpoint at the first point in time (virtual viewpoint 6f) and the virtual viewpoint at the second point in time (such as the virtual viewpoint 6a). Please see annotated Fig. 3 below.), and
determine the position of the virtual viewpoint that transitions (Fig. 3, #7 called movement path. Paragraph [0031]-MARUYAMA discloses the path generation unit 302 determines a viewpoint position and a viewing direction on a movement path of the virtual viewpoint, based on the instruction. The path generation unit 302 then generates the virtual camera path, which is formed of a plurality of virtual camera parameters that is temporally consecutive. Each of the virtual camera parameters is a parameter indicating the position and the direction of the virtual viewpoint.), over a predetermined period of time, from a position of the virtual viewpoint determined based on the first position of interest to a position of the virtual viewpoint determined based on the second position of interest (Fig. 3. Paragraph [0032]-MARUYAMA discloses in FIG. 3, a movement path 7 of a virtual viewpoint is set to follow an object of interest from rear. In this example, the object of interest is a ball moving along a trajectory 8 toward a goal. The virtual camera path generated by the path generation unit 302 is, for example, information for identifying the movement path 7. The virtual camera path includes the virtual camera parameter corresponding to each of virtual viewpoints 6a to 6f on the movement path 7. Thus, in this example, the virtual camera path for identifying the virtual viewpoint for six frames is generated. Please see annotated Fig. 3 below.).
PNG
media_image1.png
595
477
media_image1.png
Greyscale
Annotated diagram of MARUYAMA’s Fig. 3 illustrating a different positions of an object over time with a transitioning virtual viewpoint on a movement path.
Claims 2-3 are rejected under 35 U.S.C. 103 as being unpatentable over MARUYAMA (US 20180246631 A1), hereinafter referenced as MARUYAMA, in view of YAMAMOTO et al. (US 20190147604 A1), hereinafter referenced as YAMAMOTO, and further in view of MITSUMOTO (US 20170171570 A1), hereinafter referenced as MITSUMOTO, and further in view of GREEN et al. (US 20240062407 A1), hereinafter referenced as GREEN.
Regarding claim 2, MARUYAMA in view of YAMAMOTO and further in view of MITSUMOTO explicitly teach the information processing apparatus according to claim 1,
MARUYAMA in view of YAMAMOTO fail to explicitly teach wherein the position information indicates positions of a plurality of objects.
However, MITSUMOTO explicitly teaches wherein the position information indicates positions of a plurality of objects (Fig. 12A-12B. Paragraph [0048]-MITSUMOTO discloses positions of the goal 105 and the goal 106 in the field coordinate system (certain positions in the areas of the respective goals) are determined in advance. Further in paragraph [0072]-MITSUMOTO discloses since a position of each goal is a certain position in the area of the goal, the center of the width of the goal may be the position of the goal as illustrated in FIG. 12A, or a position of an end of the width of the goal may be the position of the goal as illustrated in FIG. 12B (wherein the goals are objects).), and
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of MARUYAMA in view of YAMAMOTO of an information processing apparatus comprising: one or more memories storing instructions; and one or more processors executing the instructions to; determine a position of a virtual viewpoint and a line-of-sight direction of the virtual viewpoint with the teachings of MITSUMOTO of wherein the position information indicates positions of a plurality of objects.
Wherein having MARUYAMA’s apparatus for generating a virtual viewpoint and virtual viewpoint image having wherein the position information indicates positions of a plurality of objects.
The motivation behind the modification would have been to obtain an apparatus for generating a virtual viewpoint and virtual viewpoint image that enhances the accuracy of identifying and tracking subjects/objects of interest. Since both MARUYAMA and MITSUMOTO relate to tracking subjects/objects on a sports/athletics field or court and determining virtual viewpoints, wherein MARUYAMA can reduce a possibility that the viewer may have difficulty viewing a subject of interest in the virtual viewpoint image, while MITSUMOTO the lag in switching is reduced so that a video image is not distorted. Please see MARUYAMA (US 20180246631 A1), Paragraph [0079], and MITSUMOTO (US 20170171570 A1), Paragraph [0054].
MARUYAMA in view of YAMAMOTO and further in view of MITSUMOTO fail explicitly teach wherein the specified position of interest is specified based on moving directions of the plurality of objects.
However, GREEN explicitly teaches wherein the specified position of interest is specified based on moving directions of the plurality of objects (Fig. 3. Paragraph [0031]-GREEN discloses in past sporting events there will be a higher number of successful goals in instances where, at a point in time when a player has just kicked the ball, velocity information (that is, speed and direction information) of the ball indicates a predicted ball trajectory that enters (or remains inside) the 3D space 301 when the ball reaches the edge of the pitch defining the goal line in order for a goal to be scored (wherein edge 302 of the goal line is a specified position based on the moving directions of the player and a ball (i.e. the player kicking the ball).). Further see Fig. 9, where a player is also kicking a ball, but the goal has not been specified as the position of interest.).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of MARUYAMA in view of YAMAMOTO and further in view of MITSUMOTO of an information processing apparatus comprising: one or more memories storing instructions; and one or more processors executing the instructions to; determine a position of a virtual viewpoint and a line-of-sight direction of the virtual viewpoint with the teachings of GREEN of wherein the specified position of interest is specified based on moving directions of the plurality of objects.
Wherein having MARUYAMA’s apparatus for generating a virtual viewpoint and virtual viewpoint image having wherein the specified position of interest is specified based on moving directions of the plurality of objects.
The motivation behind the modification would have been to obtain an apparatus for generating a virtual viewpoint and virtual viewpoint image that enhances the accuracy of identifying and tracking subjects/objects of interest. Since both MARUYAMA and GREEN relate to tracking subjects/objects on a sports/athletics field or court, wherein MARUYAMA can reduce a possibility that the viewer may have difficulty viewing a subject of interest in the virtual viewpoint image, while GREEN allows information to be collected that indicates player movement, for example, which can then be used for various purposes such as live commentary, post-game analysis of player technique, or the like. Please see MARUYAMA (US 20180246631 A1), Paragraph [0079], and GREEN et al. (US 20240062407 A1), Paragraph [0004].
Regarding claim 3, MARUYAMA in view of YAMAMOTO and further in view of MITSUMOTO and further in view of GREEN explicitly teach the information processing apparatus according to claim 2,
Although MARUYAMA in view of YAMAMOTO and further in view of MITSUMOTO explicitly teach wherein the specified position of interest is specified based moving velocities of the plurality of objects, MARUYAMA in view of YAMAMOTO and further in view of MITSUMOTO fail to explicitly teach wherein the specified position of interest is specified based on the moving directions of the plurality of objects and moving velocities of the plurality of objects.
However, GREEN explicitly teaches wherein the specified position of interest is specified based on the moving directions of the plurality of objects and moving velocities of the plurality of objects (Fig. 3. Paragraph [0031]-GREEN discloses in past sporting events there will be a higher number of successful goals in instances where, at a point in time when a player has just kicked the ball, velocity information (that is, speed and direction information) of the ball indicates a predicted ball trajectory that enters (or remains inside) the 3D space 301 when the ball reaches the edge of the pitch defining the goal line in order for a goal to be scored (wherein edge 302 of the goal line is a specified position based on the moving directions of the player and a ball (i.e. the player kicking the ball towards the goal).). Further in Paragraph [0036]-GREEN discloses at a point in time when the player 201 is about to kick the ball 202 (such as depicted in FIG. 4), the velocity of the player's striking foot and the backlift b may be used to predict the trajectory of the ball 202 (wherein the velocity of the player’s striking foot used to predicted the trajectory of the ball is a moving velocity of an object that can be used to further specify a position of interest as shown in paragraph [0031]).). Further see Fig. 9, where a player is also kicking a ball, but the goal has not been specified as the position of interest.).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of MARUYAMA in view of YAMAMOTO and further in view of MITSUMOTO of an information processing apparatus comprising: one or more memories storing instructions; and one or more processors executing the instructions to; determine a position of a virtual viewpoint and a line-of-sight direction of the virtual viewpoint with the teachings of GREEN of wherein the specified position of interest is specified based on the moving directions of the plurality of objects and moving velocities of the plurality of objects.
Wherein having MARUYAMA’s apparatus for generating a virtual viewpoint and virtual viewpoint image wherein the specified position of interest is specified based on the moving directions of the plurality of objects and moving velocities of the plurality of objects.
The motivation behind the modification would have been to obtain an apparatus for generating a virtual viewpoint and virtual viewpoint image that enhances the accuracy of identifying and tracking subjects/objects of interest. Since both MARUYAMA and GREEN relate to tracking subjects/objects on a sports/athletics field or court, wherein MARUYAMA can reduce a possibility that the viewer may have difficulty viewing a subject of interest in the virtual viewpoint image, while GREEN allows information to be collected that indicates player movement, for example, which can then be used for various purposes such as live commentary, post-game analysis of player technique, or the like. Please see MARUYAMA (US 20180246631 A1), Paragraph [0079], and GREEN et al. (US 20240062407 A1), Paragraph [0004].
Claims 4-5 are rejected under 35 U.S.C. 103 as being unpatentable over MARUYAMA (US 20180246631 A1), hereinafter referenced as MARUYAMA, in view of YAMAMOTO et al. (US 20190147604 A1), hereinafter referenced as YAMAMOTO, and further in view of MITSUMOTO (US 20170171570 A1), hereinafter referenced as MITSUMOTO, and further in view of GREEN et al. (US 20240062407 A1), hereinafter referenced as GREEN, and further in view of FERLATTE et al. (US 20130148861 A1), hereinafter referenced as FERLATTE.
Regarding claim 4, MARUYAMA in view of YAMAMOTO and further in view of MITSUMOTO and further in view of GREEN explicitly teach the information processing apparatus according to claim 2,
MARUYAMA in view of YAMAMOTO and further in view of MITSUMOTO fail to explicitly teach wherein the plurality of objects is classified into a plurality of categories including a first category and a second category, and
However, GREEN explicitly teaches wherein the plurality of objects (Fig. 5, illustrates a plurality of objects, i.e. players 501-503. Paragraph [0040]-GREEN discloses three players, 501-503, are positioned between the ball 202 and the goal 300 such that they partially obscure the goal 300 from the ball 202.) is classified into a plurality of categories including a first category (Fig. 5. Paragraph [0042]-GREEN discloses the goalkeeper may be identified by using any suitable method, for example using an image processing technique that identifies their jersey number and colour, using position information indicating their position relative to the pitch, or the like (wherein the goalkeeper is the first category and wherein the players on field, as illustrated in Fig. 5, are the plurality of objects).) and a second category (Fig. 13, illustrates a picture of a referee (i.e. the second category). Paragraph [0082]-GREEN discloses FIG. 13 depicts a person 1301 who is a match official of the football match (for example, a referee or assistant referee) and who has a pose indicative that a foul has occurred. The person 1301 is determined to be a match official using any suitable method, for example using an image processing technique that identifies their clothing to match the clothing of a match official, using position information indicating their position relative to the pitch, or the like (wherein the match official/referee is the second category).), and
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of MARUYAMA in view of YAMAMOTO and further in view of MITSUMOTO of an information processing apparatus comprising: one or more memories storing instructions; and one or more processors executing the instructions to; determine a position of a virtual viewpoint and a line-of-sight direction of the virtual viewpoint with the teachings of GREEN of wherein the plurality of objects is classified into a plurality of categories including a first category and a second category, and.
Wherein having MARUYAMA’s apparatus for generating a virtual viewpoint and virtual viewpoint image wherein the plurality of objects is classified into a plurality of categories including a first category and a second category, and.
The motivation behind the modification would have been to obtain an apparatus for generating a virtual viewpoint and virtual viewpoint image that enhances the accuracy of identifying and tracking subjects/objects of interest. Since both MARUYAMA and GREEN relate to tracking subjects/objects on a sports/athletics field or court, wherein MARUYAMA can reduce a possibility that the viewer may have difficulty viewing a subject of interest in the virtual viewpoint image, while GREEN allows information to be collected that indicates player movement, for example, which can then be used for various purposes such as live commentary, post-game analysis of player technique, or the like. Please see MARUYAMA (US 20180246631 A1), Paragraph [0079], and GREEN et al. (US 20240062407 A1), Paragraph [0004].
MARUYAMA in view of YAMAMOTO and further in view of MITSUMOTO and further in view of GREEN fail to explicitly teach wherein the specified position of interest is specified based on the moving directions of the plurality of objects corresponding to the first category.
However, FERLATTE explicitly teaches wherein the specified position of interest (Fig. 3a. Paragraph [0079]-FERLATTE discloses the template may indicate that a bench area is not of interest and an area by the goal is of greater interest (wherein the goal is the specified position of interest).) is specified based on the moving directions of the plurality of objects corresponding to the first category (Fig. 3a. Paragraph [0078]-FERLATTE discloses a player quickly moving towards the goal may have a higher interest rating then a player slowly moving towards the bench. The interest-rating may be determined based on a variety of interest-rating factors such as position or location within the playing area 90, speed or velocity, location within a predefined area, direction of movement, time lapsed in the sporting event, proximity to other objects, the stage of game play, the identity/personality of a player 110 and so on (wherein the plurality of objects are the players moving towards or away from the goal). Further in paragraph [0079]-FERLATTE discloses the template may indicate that a bench area is not of interest and an area by the goal is of greater interest to distinguish between a situation where a group of players gathers by the bench and another group of players gathers by the goal area, where the latter is determined to be of greater interest.).).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of MARUYAMA in view of YAMAMOTO and further in view of MITSUMOTO and further in view of GREEN of an information processing apparatus comprising: one or more memories storing instructions; and one or more processors executing the instructions to; determine a position of a virtual viewpoint and a line-of-sight direction of the virtual viewpoint with the teachings of FERLATTE of wherein the specified position of interest is specified based on the moving directions of the plurality of objects corresponding to the first category.
Wherein having MARUYAMA’s apparatus for generating a virtual viewpoint and virtual viewpoint image having wherein the specified position of interest is specified based on the moving directions of the plurality of objects corresponding to the first category.
The motivation behind the modification would have been to obtain an apparatus for generating a virtual viewpoint and virtual viewpoint image that enhances the accuracy of identifying and tracking subjects/objects of interest. Since both MARUYAMA and FERLATTE relate to tracking subjects/objects on a sports/athletics field or court, wherein MARUYAMA can reduce a possibility that the viewer may have difficulty viewing a subject of interest in the virtual viewpoint image, while FERLATTE is to improve accuracy of play tracking. Please see MARUYAMA (US 20180246631 A1), Paragraph [0079], and FERLATTE et al. (US 20130148861 A1), Paragraph [0074].
Regarding claim 5, MARUYAMA in view of YAMAMOTO and further in view of MITSUMOTO and further in view of GREEN and further in view of FERLATTE explicitly teach the information processing apparatus according to claim 4,
MARUYAMA further explicitly teaches wherein the plurality of image capturing apparatuses captures an athletic competition (Fig. 3, illustrates an example of an athletic competition. Paragraph [0021]-MARUYAMA discloses the holding unit 1 holds a plurality of captured images (a plurality of viewpoint images) obtained by imaging a subject by each of a plurality of cameras from different directions. The holding unit 1 transmits the captured images to the image processing apparatus 2. The plurality of cameras is installed to surround, for example, a sports stadium field or a live stage, and synchronously performs imaging.),
MARUYAMA in view of YAMAMOTO and further in view of MITSUMOTO fail to explicitly teach wherein the first category is a player, and wherein the second category is a referee.
However, GREEN explicitly teaches wherein the first category is a player (Fig. 5. Paragraph [0042]-GREEN discloses the goalkeeper may be identified by using any suitable method, for example using an image processing technique that identifies their jersey number and colour, using position information indicating their position relative to the pitch, or the like (wherein the goalkeeper is a player and is thus the first category).), and
wherein the second category is a referee (Fig. 13, illustrates a picture of a referee, the second category. Paragraph [0082]-GREEN discloses FIG. 13 depicts a person 1301 who is a match official of the football match (for example, a referee or assistant referee) and who has a pose indicative that a foul has occurred. The person 1301 is determined to be a match official using any suitable method, for example using an image processing technique that identifies their clothing to match the clothing of a match official, using position information indicating their position relative to the pitch, or the like (wherein the match official/referee is the second category).).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of MARUYAMA in view of YAMAMOTO and further in view of MITSUMOTO of an information processing apparatus comprising: one or more memories storing instructions; and one or more processors executing the instructions to; determine a position of a virtual viewpoint and a line-of-sight direction of the virtual viewpoint with the teachings of GREEN of wherein the first category is a player, and wherein the second category is a referee.
Wherein having MARUYAMA’s apparatus for generating a virtual viewpoint and virtual viewpoint image wherein the first category is a player, and wherein the second category is a referee.
The motivation behind the modification would have been to obtain an apparatus for generating a virtual viewpoint and virtual viewpoint image that enhances the accuracy of identifying and tracking subjects/objects of interest. Since both MARUYAMA and GREEN relate to tracking subjects/objects on a sports/athletics field or court, wherein MARUYAMA can reduce a possibility that the viewer may have difficulty viewing a subject of interest in the virtual viewpoint image, while GREEN allows information to be collected that indicates player movement, for example, which can then be used for various purposes such as live commentary, post-game analysis of player technique, or the like. Please see MARUYAMA (US 20180246631 A1), Paragraph [0079], and GREEN et al. (US 20240062407 A1), Paragraph [0004].
Claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over MARUYAMA (US 20180246631 A1), hereinafter referenced as MARUYAMA, in view of YAMAMOTO et al. (US 20190147604 A1), hereinafter referenced as YAMAMOTO, and further in view of MITSUMOTO (US 20170171570 A1), hereinafter referenced as MITSUMOTO, and further in view of NOMURA (US 20150091715 A1), hereinafter referenced as NOMURA.
Regarding claim 6, MARUYAMA in view of YAMAMOTO and further in view of MITSUMOTO explicitly teach the information processing apparatus according to claim 1,
MARUYAMA further explicitly teaches wherein the one or more processors further execute the instructions to (Fig. 2. Paragraph [0036]-MARUYAMA discloses the CPU 211 controls the entire image processing apparatus 2, by using a computer program and data stored in the ROM 212 or the RAM 213 (wherein a computer program is instructions). Further in paragraph [0083]-MARUYAMA discloses the computer may comprise one or more processors (e.g., central processing unit (CPU), micro processing unit (MPU)) and may include a network of separate computers or separate processors to read out and execute the computer executable instructions.)
MARUYAMA in view of YAMAMOTO and further in view of MITSUMOTO fail to explicitly teach acquire condition information indicating the plurality of positions of interest corresponding to a predetermined area in a three-dimensional space, and wherein, in a case where the object is present in the predetermined area indicated by the condition information, the specified position of interest is specified from the plurality of positions of interest corresponding to the predetermined area.
However, NOMURA explicitly teaches acquire condition information indicating the plurality of positions of interest corresponding to a predetermined area in a three-dimensional space (Fig. 2. Paragraph [0045]-NOMURA discloses the object identifying module 164 groups the subject parts (pixels and blocks) within the detection area 214 by using the 3D positional information based on the distance image 212 to identify an object Specifically, the object identifying module 164 selects any one of subject parts as a reference point and groups the selected subject part with other subject parts of which differences in the horizontal distance, differences in the height, and differences in the depth distance are within a predetermined range to make these subject parts to be one unitary object (wherein the plurality of positions of interest are the subject parts and wherein the predetermined area is the detection area).), and
wherein, in a case where the object is present in the predetermined area indicated by the condition information (Fig. 2. Paragraph [0045]-NOMURA discloses the object identifying module 164 groups the subject parts (pixels and blocks) within the detection area 214 by using the 3D positional information based on the distance image 212 to identify an object (wherein the subject parts comprise an object, which is within the detection area (i.e. the predetermined area)).), the specified position of interest is specified from the plurality of positions of interest corresponding to the predetermined area (Fig. 2. Paragraph [0045]-NOMURA discloses the object identifying module 164 groups the subject parts (pixels and blocks) within the detection area 214 by using the 3D positional information based on the distance image 212 to identify an object Specifically, the object identifying module 164 selects any one of subject parts as a reference point and groups the selected subject part with other subject parts of which differences in the horizontal distance, differences in the height, and differences in the depth distance are within a predetermined range to make these subject parts to be one unitary object (wherein the specified position of interest is the reference point and wherein the plurality of positions corresponding to the predetermined area are the subject parts located in the detection area).).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of MARUYAMA in view of YAMAMOTO and further in view of MITSUMOTO of an information processing apparatus comprising: one or more memories storing instructions; and one or more processors executing the instructions to; determine a position of a virtual viewpoint corresponding to a virtual viewpoint image related to the object and a line-of-sight direction of the virtual viewpoint, based on the specified position of interest and the position of the object with the teachings of NOMURA of wherein the first category is a player, and wherein the second category is a referee.
Wherein having MARUYAMA’s apparatus for generating a virtual viewpoint and virtual viewpoint image wherein the first category is a player, and wherein the second category is a referee.
The motivation behind the modification would have been to obtain an apparatus for generating a virtual viewpoint and virtual viewpoint image that enhances the accuracy of identifying and tracking subjects/objects of interest. Since both MARUYAMA and NOMURA relate to tracking subjects/objects through image analysis, wherein MARUYAMA can reduce a possibility that the viewer may have difficulty viewing a subject of interest in the virtual viewpoint image, while NOMURA the determination accuracy of pedestrians can be improved. Please see MARUYAMA (US 20180246631 A1), Paragraph [0079], and NOMURA (US 20150091715 A1), Paragraph [0072].
Claims 12-13 are rejected under 35 U.S.C. 103 as being unpatentable over MARUYAMA (US 20180246631 A1), hereinafter referenced as MARUYAMA, in view of YAMAMOTO et al. (US 20190147604 A1), hereinafter referenced as YAMAMOTO et al. (US 20190147604 A1), and further in view of MITSUMOTO (US 20170171570 A1), hereinafter referenced as MITSUMOTO, and further in view of ONUMA (US 20220383586 A1), hereinafter referenced as ONUMA.
Regarding claim 12, MARUYAMA in view of YAMAMOTO and further in view of MITSUMOTO explicitly teach the information processing apparatus according to claim 1,
MARUYAMA in view of YAMAMOTO fail to explicitly teach wherein the position of the virtual viewpoint and the line-of-sight direction of the virtual viewpoint are determined based on a half line extending from the position of the object to the specified position of interest.
However, ONUMA explicitly teaches wherein the position of the virtual viewpoint and the line-of-sight direction of the virtual viewpoint are determined based on a half line extending from the position of the object to the specified position of interest (Fig. 2, illustrates a half line (i.e. sight information) extending from a position of an object to the position of interest (i.e. sight designation position). Paragraph [0033]-ONUMA discloses the sight information at this time is stored as information of a straight line (line segment) connecting the position of the object with the sight designation position as illustrated in FIG. 2 (wherein a half line is sight information stored as information of a straight line and wherein line-of-sight direction is sight information). Paragraph [0041]-ONUMA discloses the virtual viewpoint setting unit 6 generates virtual viewpoint information in accordance with the sight information accumulated in the storage unit 4.)..
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of MARUYAMA in view of YAMAMOTO and further in view of MITSUMOTO of an information processing apparatus comprising: one or more memories storing instructions; and one or more processors executing the instructions to; determine a position of a virtual viewpoint and a line-of-sight direction of the virtual viewpoint of ONUMA of wherein the position of the virtual viewpoint and the line-of-sight direction of the virtual viewpoint are determined based on a half line extending from the position of the object to the specified position of interest.
Wherein having MARUYAMA’s apparatus for generating a virtual viewpoint and virtual viewpoint image wherein the position of the virtual viewpoint and the line-of-sight direction of the virtual viewpoint are determined based on a half line extending from the position of the object to the specified position of interest.
The motivation behind the modification would have been to obtain an apparatus for generating a virtual viewpoint and virtual viewpoint image that enhances the accuracy of identifying and tracking subjects/objects of interest. Since both MARUYAMA and ONUMA relate to generating virtual viewpoint images, wherein MARUYAMA can reduce a possibility that the viewer may have difficulty viewing a subject of interest in the virtual viewpoint image, while ONUMA allows for appropriately designating the sight to the object being in the shooting space of the virtual viewpoint image. Please see MARUYAMA (US 20180246631 A1), Paragraph [0079], and ONUMA (US 20220383586 A1), Paragraph [0045].
Regarding claim 13, MARUYAMA in view of YAMAMOTO and further in view of MITSUMOTO and further in view of ONUMA explicitly teach the information processing apparatus according to claim 12,
MARUYAMA in view of YAMAMOTO and further in view of MITSUMOTO fail to explicitly teach wherein the position of the virtual viewpoint and the line-of-sight direction of the virtual viewpoint are determined operable to an angle formed by the half line extending from the position of the object to the specified position of interest and the line-of-sight direction of the virtual viewpoint becomes a predetermined angle.
However, ONUMA explicitly teaches wherein the position of the virtual viewpoint and the line-of-sight direction of the virtual viewpoint are determined operable to an angle (Figs 6A-6C. Paragraph [0052]-ONUMA discloses the direction and the angle of view of the virtual viewpoint may be set so as to include a direction based on two pieces of sight information after having set a single virtual viewpoint at a point (sight designation position) w % here a plurality of pieces of sight information intersect (wherein the angle of view is an angle).) formed by the half line extending from the position of the object to the specified position of interest (Figs 6A-6C., illustrates an angle (i.e. angle of view of virtual viewpoint) formed by the half line extending from the position of the object to the position of interest (i.e. sight information, the line extending from the first object to the sight destination position). Paragraph [0052]-ONUMA discloses the direction and the angle of view of the virtual viewpoint may be set so as to include a direction based on two pieces of sight information after having set a single virtual viewpoint at a point (sight designation position) w % here a plurality of pieces of sight information intersect.) and the line-of-sight direction of the virtual viewpoint becomes a predetermined angle (Figs. 6A-6C, illustrate the line of sight direction (shown in 6A) becoming a predetermined angle (shown in 6C). Paragraph [0052]-ONUMA discloses the direction and the angle of view of the virtual viewpoint may be set so as to include a direction based on two pieces of sight information after having set a single virtual viewpoint at a point (sight designation position) w % here a plurality of pieces of sight information intersect (wherein the angle of view is the predetermined angle).).).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of MARUYAMA in view of YAMAMOTO and further in view of MITSUMOTO of an information processing apparatus comprising: one or more memories storing instructions; and one or more processors executing the instructions to; determine a position of a virtual viewpoint and a line-of-sight direction of the virtual viewpoint with the teachings of ONUMA of wherein the position of the virtual viewpoint and the line-of-sight direction of the virtual viewpoint are determined operable to an angle formed by the half line extending from the position of the object to the specified position of interest and the line-of-sight direction of the virtual viewpoint becomes a predetermined angle.
Wherein having MARUYAMA’s apparatus for generating a virtual viewpoint and virtual viewpoint image wherein the position of the virtual viewpoint and the line-of-sight direction of the virtual viewpoint are determined operable to an angle formed by the half line extending from the position of the object to the specified position of interest and the line-of-sight direction of the virtual viewpoint becomes a predetermined angle.
The motivation behind the modification would have been to obtain an apparatus for generating a virtual viewpoint and virtual viewpoint image that enhances the accuracy of identifying and tracking subjects/objects of interest. Since both MARUYAMA and ONUMA relate to generating virtual viewpoint images, wherein MARUYAMA can reduce a possibility that the viewer may have difficulty viewing a subject of interest in the virtual viewpoint image, while ONUMA allows for appropriately designating the sight to the object being in the shooting space of the virtual viewpoint image. Please see MARUYAMA (US 20180246631 A1), Paragraph [0079], and ONUMA (US 20220383586 A1), Paragraph [0045].
Claim 17 is rejected under 35 U.S.C. 103 as being unpatentable over MARUYAMA (US 20180246631 A1), hereinafter referenced as MARUYAMA, in view of YAMAMOTO et al. (US 20190147604 A1), hereinafter referenced as YAMAMOTO, and further in view of MITSUMOTO (US 20170171570 A1), hereinafter referenced as MITSUMOTO, and further in view of GREEN et al. (US 20240062407 A1), hereinafter referenced as GREEN, and further in view of YEE et al. (US 20180167553 A1), hereinafter referenced as YEE.
Regarding claim 17, MARUYAMA in view of YAMAMOTO and further in view of MITSUMOTO and further in view of GREEN explicitly teach the information processing apparatus according to claim 3,
MARUYAMA in view of YAMAMOTO and further in view of MITSUMOTO and further in view of GREEN fail to explicitly teach wherein the one or more processors calculate an average velocity vector of velocity vectors of the plurality of objects, and specify the one position of interest based on the average velocity vector.
However, YEE explicitly teaches wherein the one or more processors (Fig. 8A-8B. Paragraph [0067]-YEE discloses the processor 805 is given a set of instructions which are executed therein. The processor 805 waits for a subsequent input, to which the processor 805 reacts to by executing another set of instructions.) calculate an average velocity vector (Fig. 6. Paragraph [0083]-YEE discloses a process may be used at step 320 to determine a virtual camera position for which group velocity (i.e., the average velocity of the subjects including the ball 240, and the players 220 and 230) is directed both towards the virtual camera and across the field of view of the virtual camera from left to right or right to left.) of velocity vectors of the plurality of objects (Fig. 6. Paragraph [0082]-YEE discloses the locations of the players 220 and 230 may be determined, for example, by detecting a first object representing the player 220 and a second object representing the player 230 in the scene, where each object has at least one motion attribute including, for example, velocity and trajectory as described below.), and
specify the one position of interest based on the average velocity vector (Fig. 3 and 6. Paragraph [0083]-YEE discloses a process may be used at step 320 to determine a virtual camera position for which group velocity (i.e., the average velocity of the subjects including the ball 240, and the players 220 and 230) is directed both towards the virtual camera and across the field of view of the virtual camera from left to right or right to left (wherein the virtual camera position is the position of interest).).
Therefore, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention was made to combine the teachings of MARUYAMA in view of YAMAMOTO and further in view of MITSUMOTO and further in view of GREEN of an information processing apparatus comprising: one or more memories storing instructions; and one or more processors executing the instructions to; determine a position of a virtual viewpoint and a line-of-sight direction of the virtual viewpoint with the teachings of YEE of wherein the one or more processors calculate an average velocity vector of velocity vectors of the plurality of objects, and specify the one position of interest based on the average velocity vector.
Wherein having MARUYAMA’s apparatus for generating a virtual viewpoint and virtual viewpoint image acquire position information wherein the one or more processors calculate an average velocity vector of velocity vectors of the plurality of objects, and specify the one position of interest based on the average velocity vector.
The motivation behind the modification would have been to obtain an apparatus for generating a virtual viewpoint and virtual viewpoint image that enhances the accuracy of identifying and tracking subjects/objects of interest. Since both MARUYAMA and YEE relate to tracking subjects/objects on a sports/athletics field or court and determining virtual viewpoints, wherein MARUYAMA can reduce a possibility that the viewer may have difficulty viewing a subject of interest in the virtual viewpoint image, while YEE allows the generation of framed content that is engaging and comfortable to watch by a typical passive viewer. Please see MARUYAMA (US 20180246631 A1), Paragraph [0079], and YEE et al. (US 20180167553 A1), Paragraph [0007].
Allowable Subject Matter
Claims 18 and 20, along with their dependent claims, 21 are therefrom objected to as being dependent upon rejected base claim, claim 1, and 1-3, 17, respectively but would be allowable if rewritten in independent form including all of the limitations of the base claims and any intervening claims, once the double patenting rejections are overcome.
The following is a statement of reasons for the indication of allowable subject matter:
Regarding claim 18, the prior arts fail to explicitly teach wherein the one or more processors obtain a projected vector by projecting the average velocity vector onto a direction connecting two positions of interest among the plurality of positions of interest, and specify the one position of interest based on the projected vector, as claimed in claim 18.
Regarding claim 20, the prior arts fail to explicitly teach wherein each of the plurality of positions of interest is a three-dimensional position different from a barycentric position of a background model of a corresponding stationary object among the plurality of stationary objects, as claimed in claim 20.
Conclusion
Listed below are the prior arts made of record and not relied upon but are considered pertinent to applicant’s disclosure.
Ishikawa et al. (US 11468258 B2) – An information processing apparatus acquires viewpoint information indicating a position and a direction of a virtual viewpoint designated for generating a virtual viewpoint image in which a predetermined object is set as an object to be targeted, and object information indicating a position of an object. The information processing apparatus further identifies an object to be targeted that corresponds to the viewpoint information, based on the acquired viewpoint information and the acquired object information, and outputs information regarding the identified object…Abstract, Fig. 5A-5E.
UEDA et al. (US 20170124388 A1) – A moving body tracking method and a moving body tracking device detect a position of a moving body in each of a plurality of frame images which configure a video, detect a trajectory of the moving body based on the image which is obtained by using the plurality of frame images, determine a final position of the moving body in each of the plurality of frame images based on the detected position and the position of the moving body which is obtained from the detected trajectory, and output the determined position…Abstract, Fig. 1.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to ETHAN N WOLFSON whose telephone number is (571)272-1898. The examiner can normally be reached Monday - Friday 8:00 am - 5:00 pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Chineyere Wills-Burns can be reached at (571) 272-9752. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/ETHAN N WOLFSON/Examiner, Art Unit 2673
/CHINEYERE WILLS-BURNS/Supervisory Patent Examiner, Art Unit 2673
Prosecution Insights