DETAILED ACTION
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application aft final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 08/28/2026 has been entered.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claims 1, 6, 8, 9, and 10 - 14 are rejected under 35 U.S.C. 103 as being unpatentable over DOHNO (Publication: US 2019/0129612 A1) in view of Takemoto (Publication: US 2020/0133388 A1).
Regarding claim 1, DOHNO discloses an image processing apparatus comprising: one or more memories storing instructions ([0102] FIG. 1, a computer 1 that executes an information processing program (hereinafter referred to as “program”). The computer 1 is an information processing apparatus such as a smartphone, a tablet terminal, a notebook computer, a personal computer, or a digital audio player. In FIG. 1, the computer 1 includes a display apparatus 2, an input part 3, a storage part 4, a communication part 5, a CPU (Central Processing Unit) 6, ROM (Read Only Memory) 7, and RAM (Random Access Memory) 8. [0103] the program that the computer 1 executes is a game program stored in RAM to perform the following: ); and
one or more processors executing the instructions to ([0102], [0103] the program that the computer 1 executes is a game program stored in RAM and executed by a CPU to perform the following:):
detect a position of a subject ([0125] As the game progresses, the position of a character and the like being the manipulation target change . When there exists a plurality of objects each being the manipulation target as the position changes, detection, the object being the manipulation target at the current time point changes. The detection of the manipulation target attribute stores, as a manipulation target attribute 16a, position information 16b that defines the position of the manipulation target, speed information 16c that defines the shift speed of the manipulation target, direction information 16d that defines the shift direction of the manipulation target, and target information 16e that defines which object is the manipulation target.);
generating a virtual viewpoint image using three-dimensional shape data of the subject ([0124] As the game progresses, the viewpoint in the virtual space changes, generated. Accordingly, the viewpoint attribute storage part 17 stores, as a viewpoint attribute 17a, the position of a viewpoint camera in three-dimensional virtual space.
[0116] - the virtual screen is generated by three-dimensional space, displaying the fourth image allows the user to easily recognize on what viewpoint the view is displayed .
[0015], [0017], [0116] - The first image on display may represent a shift speed of the manipulation target, and the second image display may represent a shift direction of the manipulation target.);
and display subject information related to movement of the subject on the viewpoint image ([0015], [0017], [0116] - The first image on display may represent a shift speed of the manipulation target, and the second image display may represent a shift direction of the manipulation target. [0112] - The first image displayed indicating may be expressed in the viewpoint, as part of the functional state display part 10e, by a meter value, a count value, a numerical value, color, size, a shape, or a rotation speed in the viewpoint.).
DOHNO does not however Takemoto discloses
obtain three-dimensional shape data of a plurality of subjects that are different from one another (
[0035] - A moving object model acquisition unit 1400 acquires a moving object model based on the result of measuring the three-dimensional shape of the moving object 110 included in the captured image.
[0067] - previous frame from the polygon centroid position in the current frame, thereby acquiring a motion vector, plurality of frames includes polygon centroid, object to calculate the motion vector thus “plurality of subjects”);
detect respective position of the plurality of subjects based on the obtained three-dimensional shape data ([0067] In S720, the moving object model estimation unit 1460 reads out the polygon centroid position in the previous frame from the memory, and subtracts the polygon centroid position in the previous frame from the polygon centroid position in the current frame , thereby acquiring a motion vector 1720 in the space, different frames “respective position of the plurality of subjects of subjects”.); and
generate an image using the obtained three-dimensional shape data based on information on the detected positions of the plurality of subjects ([0035] - A moving object model acquisition unit 1400 acquires a moving object model based on the result of measuring the three-dimensional shape of the moving object 110 included in the captured image.
[0067] In S720, the moving object model estimation unit 1460 reads out the polygon centroid position in the previous frame from the memory, and subtracts the polygon centroid position in the previous frame from the polygon centroid position in the current frame , thereby acquiring a motion vector 1720 in the space, different frames “plurality of subjects”.
FIG. 1E illustrates an arrow is generated, a moving object depicted at a position shifted in a horizontal direction by a length indicated by an arrow, “image”.)
the virtual viewpoint image displaying subject information related to a movement of each of the plurality of subject ([0007], [0024] - FIGS. 1E, and 1F indicates a field of view, virtual viewpoint in a virtual reality space, in which the image 1000 is visible, and also indicates that the image to be displayed.
[0024], [0067] - FIG. 1E illustrates a moving object depicted at a position shifted in a horizontal direction by a length indicated by an arrow based on the moving object model estimation unit 1460 reads out the polygon centroid position in the previous frame from the memory, and subtracts the polygon centroid position in the previous frame from the polygon centroid position in the current frame , thereby acquiring a motion vector 1720 in the space [0067] .);
each subject of the plurality of subjects ([0067] In S720, the moving object model estimation unit 1460 reads out the polygon centroid position in the previous frame from the memory, and subtracts the polygon centroid position in the previous frame from the polygon centroid position in the current frame , thereby acquiring a motion vector 1720 in the space, subject is in each of the frames thus “plurality of subjects”) .
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to modify DOHNO with obtain three-dimensional shape data of a plurality of subjects; detect respective position of the plurality of subjects based on the obtained three-dimensional shape data; and generate an image using the obtained three-dimensional shape data based on information on the detected positions of the plurality of subjects, the virtual viewpoint image displaying subject information related to a movement of each of the plurality of subject; each subject of the plurality of subjects as taught by Takemoto. The motivation for doing is to improve user experience.
Regarding claim 6, DOHNO in view of Takemoto disclose all the limitations of claim 1.
DOHNO discloses in a case where the plurality of subjects is seen in the virtual viewpoint image, information based on the position information of each subject is displayed ([0165] The display image generating part 13 displays, at the position defined by the display position information 15i on the display part 10, the functional state display part 10e and the horizontal/directional display part 10f of the virtual controller image 10a determined in S203 and S204 (S208). , when the display position information 15i defines that the virtual controller image 10a is displayed above the manipulation target, the display image generating part 13 determines the display position of the virtual controller image 10a based on the position of the manipulation target defined by the position information 16b. Further, the display image generating part 13 displays, on the display part 10, the virtual controller image 10a as seen from the viewpoint defined by the viewpoint attribute 17a Alternatively, when the display position information 15i defines any fixed position ( , upper left or lower left on the screen) as the display position of the virtual controller image 10a, the display image generating part 13 displays the virtual controller image 10a at that fixed position.
[0125] As the game progresses, the position, the speed, and the direction of a character and the like being the manipulation target change. Further, when there exists a plurality of objects each being the manipulation target, the object being the manipulation target at the current time point changes. Accordingly, the manipulation target attribute storage part 16 stores, as a manipulation target attribute 16a, position information 16b that defines the position of the manipulation target, speed information 16c that defines the shift speed of the manipulation target, direction information 16d that defines the shift direction of the manipulation target, and target information 16e that defines which object is the manipulation target.
).
Regarding claim 8, see rejection on claim 1.
Regarding claim 9, see rejection on claim 1.
Regarding claim 10, DOHNO in view of Takemoto disclose all the limitations of claim 1.
Takemoto discloses acquire pieces of position information of each of the plurality of subjects at a plurality of times, calculate at least one of a moving speed and a moving direction of each of the plurality of subjects based on the pieces of position information acquired at the plurality of times ([0067] In S720, the moving object model estimation unit 1460 reads out the polygon centroid position in the previous frame from the memory, and subtracts the polygon centroid position in the previous frame from the polygon centroid position in the current frame, thereby acquiring a motion vector 1720 in the space. FIG. 10 is a schematic diagram illustrating a positional relationship between a polygon centroid position 1740 in the current frame and a polygon centroid position 1710 in the previous frame. Further, an elapsed time T between frames is calculated based on the difference between the image capturing time of the current frame and the image capturing time of the previous frame, and the motion vector 1720 is divided by the elapsed time T, thereby obtaining a velocity vector E per unit time.
It is known in the computer graphics art that the “motion vector” represents the displacement of the closest matching frame in the reference frame with respect to the current frame location in different time.), and
generate the subject information related to the movement of each of the plurality of subjects based on the calculated at least one of the moving speed and the moving direction ( [0024] - FIG. 1E illustrates a moving object depicted at a position shifted in a horizontal direction by a length indicated by an arrow 150 from an ideal display position of the moving object with no delay as illustrated in FIG. 1D. This may provide the HMD wearer with a feeling of strangeness in the video image of the virtual space in proportion to the length of the arrow 150. A dotted line passing through FIGS. 1D, 1E, and 1F indicates a field of view in which the image 1000 is visible, and also indicates that the image to be displayed is shifted by ΔX based on the motion of the HMD. On the other hand, FIG. 1F illustrates an image obtained by shifting the video image generated at the time “t” in the horizontal direction of the image by the amount ΔX in which the movement amount ΔΘ of the HMD is added. First, as for the shift amount ΔX, an amount of change between an orientation at a time close to an image capturing time and a latest orientation is obtained, “moving direction”.
Moving Direction:
PNG
media_image1.png
306
294
media_image1.png
Greyscale
A black area in an image 1030 indicates a state where an area in which no image is captured is displayed as the black area because the image is shifted. The position of the still object 120 illustrated in FIG. 1F is displayed at the position on a screen of the HMD the same as the position of the still object 120 illustrated in FIG. 1D. However, the moving object deviates from the position (ideal display position of the moving object illustrated in FIG. 1D) of the moving object at the actual time (time “t+Δt”) by the amount corresponding to the length of an arrow 160.).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to modify DOHNO in view Takemoto with acquire pieces of position information of each of the plurality of subjects at a plurality of times, calculate at least one of a moving speed and a moving direction of each of the plurality of subjects based on the pieces of position information acquired at the plurality of times, and generate the subject information related to the movement of each of the plurality of subjects based on the calculated at least one of the moving speed and the moving direction as taught by Takemoto. The motivation for doing is to improve user experience.
Regarding claim 11, DOHNO in view of Takemoto disclose all the limitations of claim 1.
Takemoto discloses to acquire, as the pieces of position information acquired at the plurality of times, pieces of position information of several to several tens of frames before and after a time corresponding to the virtual viewpoint image ( [0065] FIG. 7 is a flowchart illustrating details of the processing of S525 Fig. 7, predict the position information of the subjects in Fig. 5, “movement of each of the plurality of subjects” .
[0007], [0024], [0029] - A head-mounted display (HMD), virtual reality, incorporates a display. When a user wears the HMD on the user's head, the user can watch moving images to be displayed from the viewpoint of the display on HMD. A method in which a moving object 110 is observed and arranged at a predicted position to thereby reduce a delay. Fig. 1C-F illustrates a position relationship between the viewpoint of the display on HMD and the objects associated with different time, “corresponding to the virtual viewpoint image”
[0067] In S720, the moving object model estimation unit 1460 reads out the polygon centroid position in the previous frame from the memory, and subtracts the polygon centroid position in the previous frame from the polygon centroid position in the current frame, thereby acquiring a motion vector 1720 in the space. FIG. 10 is a schematic diagram illustrating a positional relationship between a polygon centroid position 1740 in the current frame and a polygon centroid position 1710 in the previous frame. Further, an elapsed time T between frames is calculated based on the difference between the image capturing time of the current frame and the image capturing time of the previous frame, and the motion vector 1720 is divided by the elapsed time T, thereby obtaining a velocity vector E per unit time, “frames before and after a time”.
[0110] In S1503, the CG image generation unit 2700 determines whether to start the generation of the CG image at a predetermined interval of outputting the image to the HMD 200. For example, a display interval of a general display is 60 Hz. Accordingly, an instruction to start the generation of the CG image 60 times a second is made, “several to several tens of frames”.).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to modify DOHNO in view Takemoto with to acquire, as the pieces of position information acquired at the plurality of times, pieces of position information of several to several tens of frames before and after a time corresponding to the virtual viewpoint image as taught by Takemoto. The motivation for doing is to improve user experience.
Regarding claim 12, DOHNO in view of Takemoto disclose all the limitations of claim 1.
Takemoto discloses acquire position information of each of the plurality of subjects at a time corresponding to the virtual viewpoint image and pieces of position information of each of the plurality of subjects at one or more times different from the time corresponding to the virtual viewpoint image, and determine, as the subject information related to the movement of each of the plurality of subjects(Fig. 7, predict the position information of the subjects in Fig. 5, “movement of each of the plurality of subjects” .
[0007], [0024], [0029] - A head-mounted display (HMD), virtual reality, incorporates a display. When a user wears the HMD on the user's head, the user can watch moving images to be displayed from the viewpoint of the display on HMD. A method in which a moving object 110 is observed and arranged at a predicted position to thereby reduce a delay. Fig. 1C-F illustrates a position relationship between the viewpoint of the display on HMD and the objects associated with different time, “corresponding to the virtual viewpoint image and pieces of position information of each of the plurality of subjects at one or more”
[0067] In S720, the moving object model estimation unit 1460 reads out the polygon centroid position in the previous frame from the memory, and subtracts the polygon centroid position in the previous frame from the polygon centroid position in the current frame, thereby acquiring a motion vector 1720 in the space. FIG. 10 is a schematic diagram illustrating a positional relationship between a polygon centroid position 1740 in the current frame and a polygon centroid position 1710 in the previous frame. Further, an elapsed time T between frames is calculated based on the difference between the image capturing time of the current frame and the image capturing time of the previous frame, and the motion vector 1720 is divided by the elapsed time T, thereby obtaining a velocity vector E per unit time.),
a positional deviation per unit time based on a difference between the pieces of position information at the different times ([0067] In S720, the moving object model estimation unit 1460 reads out the polygon centroid position in the previous frame from the memory, and subtracts the polygon centroid position in the previous frame from the polygon centroid position in the current frame, thereby acquiring a motion vector 1720 in the space. FIG. 10 is a schematic diagram illustrating a positional relationship between a polygon centroid position 1740 in the current frame and a polygon centroid position 1710 in the previous frame. Further, an elapsed time T between frames is calculated based on the difference between the image capturing time of the current frame and the image capturing time of the previous frame, and the motion vector 1720 is divided by the elapsed time T, thereby obtaining a velocity vector E per unit time.
It is known in the computer graphics art that the “motion vector” represents the displacement of the closest matching frame in the reference frame with respect to the current frame location in different time.) .
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to modify DOHNO in view Takemoto with acquire position information of each of the plurality of subjects at a time corresponding to the virtual viewpoint image and pieces of position information of each of the plurality of subjects at one or more times different from the time corresponding to the virtual viewpoint image, and determine, a positional deviation per unit time based on a difference between the pieces of position information at the different times as the subject information related to the movement of each of the plurality of subjects, as taught by Takemoto. The motivation for doing is to improve user experience.
Regarding claim 13, DOHNO in view of Takemoto disclose all the limitations of claim 12.
Takemoto discloses to acquire the position information at a past time that is one second earlier than the time corresponding to the virtual viewpoint image (Fig. 7, predict the position information of the subjects in Fig. 5 .
[0067] In S720, the moving object model estimation unit 1460 reads out the polygon centroid position in the previous frame from the memory, and subtracts the polygon centroid position in the previous frame from the polygon centroid position in the current frame, thereby acquiring a motion vector 1720 in the space. FIG. 10 is a schematic diagram illustrating a positional relationship between a polygon centroid position 1740 in the current frame and a polygon centroid position 1710 in the previous frame.
Fig. 1A, 1B, movement amount ΔΘ in Δt seconds ).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to modify DOHNO in view Takemoto with to acquire the position information at a past time that is one second earlier than the time corresponding to the virtual viewpoint image as taught by Takemoto. The motivation for doing is to improve user experience.
Claims 2, 3, 4 are rejected under 35 U.S.C. 103 as being unpatentable over DOHNO (Publication: US 2019/0129612 A1) in view of Takemoto (Publication: US 2020/0133388 A1), Maizels et al. (Publication: US 2012/0202569 A1)
Regarding claim 2, DOHNO in view of Takemoto disclose all the limitations of claim 1 including each subject of the plurality of subjects.
DOHNO discloses wherein the one or more processors execute the instructions to display the moving speed and the moving direction as the subject information ([0150] the speed information 16c and the direction information 16d have been used as the manipulation target attribute 16a. The shape, the color or the like of the manipulation target changes according to the speed or the direction of the manipulation target. When the shape, the color and the like of the manipulation target do not change according to the speed or the direction, the display image generating part 13 can generate the manipulation target image 10b for the next frame just by rewriting the position information 16b.).
DOHNO in view of Takemoto do not however Maizels discloses
to acquire a moving speed and a moving direction of the subject based on the position information of the subject at a plurality of times ([0054] - the computer finds the direction and speed of motion based on the location history of each interaction stain.
The speed of motion may be calculated using timing information from a time stamp attached to each frame.).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to modify DOHNO in view of Takemoto with as taught by Maizels. The motivation for doing is to improve user interaction.
Regarding claim 3, DOHNO in view of Takemoto, Maizel disclose all the limitations of claim 2.
DOHNO discloses to indicate the moving direction of the subject using an arrow-shaped or triangular icon near the subject position, as the subject information ([0113] The image indicating the direction is referred to as a second image. The second image indicating the direction may be expressed by an arrow, any pointing shape, or color in a directional display part 10f-2 being part of the horizontal/directional display part 10f. As used herein, the direction not only refers to the shift direction of the manipulation target, but also the attack direction, the defense direction and the like.
[0125] As the game progresses, the position, the speed, and the direction of a character and the like being the manipulation target change. Further, when there exists a plurality of objects each being the manipulation target, the object being the manipulation target at the current time point changes. Accordingly, the manipulation target attribute storage part 16 stores, as a manipulation target attribute 16a, position information 16b that defines the position of the manipulation target, speed information 16c that defines the shift speed of the manipulation target, direction information 16d that defines the shift direction of the manipulation target, and target information 16e that defines which object is the manipulation target.
PNG
media_image2.png
724
472
media_image2.png
Greyscale
).
Regarding claim 4, DOHNO in view of Takemoto, Maizel disclose all the limitations of claim 3.
DOHNO discloses to change at least one of size, length, and color of the icon indicating the moving direction, based on the acquired moving speed of the subject, as the subject information ([0113] The image indicating the direction is referred to as a second image. The second image indicating the direction may be expressed by an arrow, any pointing shape, or color in a directional display part 10f-2 being part of the horizontal/directional display part 10f. As used herein, the direction not only refers to the shift direction of the manipulation target, but also the attack direction, the defense direction and the like. Note that, the direction is not limited to the foregoing exemplary directions.
PNG
media_image3.png
330
516
media_image3.png
Greyscale
).
Claim 5 is rejected under 35 U.S.C. 103 as being unpatentable over DOHNO (Publication: US 2019/0129612 A1) in view of Takemoto (Publication: US 2020/0133388 A1), Maizels et al. (Publication: US 2012/0202569 A1), Chen (Publication: US 2020/0311429 A1)
Regarding claim 5, DOHNO in view of Takemoto, Maizel disclose all the limitations of claim 3.
DOHNO discloses display an icon with a circular shape or a shape, based on the position information of the subject, as the subject information (
[0113] The image indicating the direction is referred to as a second image. The second image indicating the direction may be expressed by an arrow, any pointing shape, or color in a directional display part 10f-2 being part of the horizontal/directional display part 10f. As used herein, the direction not only refers to the shift direction of the manipulation target, but also the attack direction, the defense direction and the like.
[0125] As the game progresses, the position, the speed, and the direction of a character and the like being the manipulation target change. Further, when there exists a plurality of objects each being the manipulation target, the object being the manipulation target at the current time point changes. Accordingly, the manipulation target attribute storage part 16 stores, as a manipulation target attribute 16a, position information 16b that defines the position of the manipulation target, speed information 16c that defines the shift speed of the manipulation target, direction information 16d that defines the shift direction of the manipulation target, and target information 16e that defines which object is the manipulation target. ).
DOHNO in view of Takemoto do not however Chen discloses
display an icon with a circular shape or a shape surrounding the subject near feet of the subject ([0058] - As shown in FIG. 4, as the five colored circles (with colors changing) next to the head, left hand, right hand, left foot, and right foot of the human-shaped contour are displayed on the screen of the smartphone 16 and serve as on-screen guidance indicators 20 of compliance.).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to modify DOHNO in view of Takemoto with display an icon with a circular shape or a shape surrounding the subject near feet of the subject as taught by Chen. The motivation for doing is to provide accurate guidance.
Claim 7 is rejected under 35 U.S.C. 103 as being unpatentable over DOHNO (Publication: US 2019/0129612 A1) in view of Takemoto (Publication: US 2020/0133388 A1), Chen (Publication: US 2020/0311429 A1)
Regarding claim 7, DOHNO in view of Takemoto disclose all the limitations of claim 6 including the plurality of subjects.
DOHNO in view of Takemoto do not however Chen discloses
not to display the subject information on a subject that is included in the data of the subject but is not specified in advance ([0144] - the controller 150 may terminate the output of the event information, because there is no event due to the object 605 any longer. Further, in a case in which the object 605 has disappeared (e.g., the stopped vehicle has moved) before the event information is output to a corresponding absolute coordinate on the display unit 110, the event information may not be output, even if the corresponding absolute coordinate is within an FOV of the head-up display after the current viewpoint.
“No specified in advance” reads on because the event information may not be output, even if the corresponding absolute coordinate is within an FOV of the head-up display after the current viewpoint.) .
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to modify DOHNO in view of Takemoto with not to display the subject information on a subject that is included in the data of the subject but is not specified in advance as taught by Chen. The motivation for doing is to minimize computation amount of data processing.
Claim 14 is rejected under 35 U.S.C. 103 as being unpatentable over DOHNO (Publication: US 2019/0129612 A1) in view of Takemoto (Publication: US 2020/0133388 A1), Deng (Publication: US 2009/0295930 A1).
Regarding claim 14, DOHNO in view of Takemoto disclose all the limitations of claim 1.
Takemoto discloses to acquire pieces of position information of each of the plurality of subjects for a plurality of frames and generate the subject information related to the movement of each of the plurality of subjects based on [[a change in the averaged position information over time]] (Fig. 7, predict the position information of the subjects in Fig. 5, “movement of each of the plurality of subjects” .
[0067] In S720, the moving object model estimation unit 1460 reads out the polygon centroid position in the previous frame from the memory, and subtracts the polygon centroid position in the previous frame from the polygon centroid position in the current frame, thereby acquiring a motion vector 1720 in the space. FIG. 10 is a schematic diagram illustrating a positional relationship between a polygon centroid position 1740 in the current frame and a polygon centroid position 1710 in the previous frame. Further, an elapsed time T between frames is calculated based on the difference between the image capturing time of the current frame and the image capturing time of the previous frame, and the motion vector 1720 is divided by the elapsed time T, thereby obtaining a velocity vector E per unit time.).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to modify DOHNO in view Takemoto with to acquire pieces of position information of each of the plurality of subjects for a plurality of frames and generate the subject information related to the movement of each of the plurality of subjects based on [[a change in the averaged position information over time]] taught by Takemoto. The motivation for doing is to improve user experience.
DOHNO in view of Takemoto do not however Deng discloses
generate averaged position information by averaging the pieces of position information for the plurality of frames ([0033] - The determined positions are averaged to provide an average position of the at least one pixel at frame 404. A first compensation value is calculated based on a difference between a first determined position of the at least one pixel in the current frame and the average position at frame 406, over time.);
the movement of each of the plurality of subjects based on a change in the averaged position information over time ([0033] - The determined positions are averaged to provide an average position of the at least one pixel at frame 404. A first compensation value is calculated based on a difference between a first determined position of the at least one pixel in the current frame and the average position at frame 406, over time.).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to modify DOHNO in view Takemoto with generate averaged position information by averaging the pieces of position information for the plurality of frames; the movement of each of the plurality of subjects based on a change in the averaged position information over time taught by Deng. The motivation for doing is to have low cost for the operation.
Claim 15 is rejected under 35 U.S.C. 103 as being unpatentable over DOHNO (Publication: US 2019/0129612 A1) in view of Takemoto (Publication: US 2020/0133388 A1), Takada (Publication: US 2011/0002550 A1).
.
Dohno in view of Takemoto do not disclose; however Takada discloses
in a case where a present-time extracted shape overlaps respective positions of two or more of the plurality of subjects detected at a previous time, refrain from updating the respective positions of the two or more subjects and maintain the respective positions detected at the previous time ( [0008] - calculates a difference absolute value for each pixel between a current frame and a previous frame, the previous frame being a frame preceding the current frame, and detects, as an update region which is a region whose position changes from the previous frame to the current frame, a region including a shape defined by pixels where the difference absolute value is equal to or greater than a first threshold and including a pixel where the difference absolute value is equal to or greater than a second threshold, the second threshold being a value greater than the first threshold.
An update region extraction unit (602) extracts, an update region, a group including a pixel where a difference greater than the second threshold has been detected, from among a group of pixels where a difference greater than the first threshold has been detected. Difference less than the threshold thus “refrain from updating” No differences is “overlap”. ); and
resume updating the respective positions of the two or more subjects in response to a cancellation of the overlap ( [0124], Fig. 3 - compares a difference between values of pixels at the same position in each of a reference frame and a current frame, motion vector detection unit (Fig. 5) are repeat thus “resume, cancellation of the overlap”.
[0008] - calculates a difference absolute value for each pixel between a current frame and a previous frame, the previous frame being a frame preceding the current frame, and detects, as an update region which is a region whose position changes from the previous frame to the current frame, a region including a shape defined by pixels where the difference absolute value is equal to or greater than a first threshold and including a pixel where the difference absolute value is equal to or greater than a second threshold, the second threshold being a value greater than the first threshold.
An update region extraction unit (602) extracts, an update region, a group including a pixel where a difference greater than the second threshold has been detected, from among a group of pixels where a difference greater than the first threshold has been detected, “cancellation of the overlap”.
As shown in Fig. 3, two objects are updated.
PNG
media_image4.png
678
544
media_image4.png
Greyscale
).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to modify DOHNO in view Takemoto with in a case where a present-time extracted shape overlaps respective positions of two or more of the plurality of subjects detected at a previous time, refrain from updating the respective positions of the two or more subjects and maintain the respective positions detected at the previous time; and resume updating the respective positions of the two or more subjects in response to a cancellation of the overlap as taught by Takada. The motivation for doing is to reduce the amount of data associated with the transfer of plot data on the computer screen and alleviating the load imposed on the server device.
Claim 16 is rejected under 35 U.S.C. 103 as being unpatentable over DOHNO (Publication: US 2019/0129612 A1) in view of Takemoto (Publication: US 2020/0133388 A1), Yokohata (Publication: US 2011/0043639 A1).
.
Dohno in view of Takemoto do not disclose; however Yokohata discloses
wherein the subject information includes a movement-direction mark for each of the plurality of subjects, and wherein the one or more processors further execute the instructions to display each movement direction mark at a position spaced apart from the detected position of the corresponding subject and displaced around the detected position in accordance with a movement direction of the corresponding subject (
[0121] the first frame image FI.sub.1 is displayed first on the display unit 27, and in this state of the display, a user's operation of setting the tracking target is received. For instance, as illustrated in FIG. 15, the frame image FI.sub.1 is displayed, and an arrow type icon 510 is displayed on a display screen 27a of the display unit 27. The user can change the display position of the arrow type icon 510 by a predetermined operation to the operating unit 26. Then, using the operating unit 26, a predetermined determination operation is performed in the state where a display position of the arrow type icon 510 is set to a display position of the noted object (noted subject) on the display screen 27a, so that the user can set the noted object to the tracking target. As the example illustrated in FIG. 15, when the determination operation is performed in the state where a display position of the arrow type icon 510 is set to a display position of the person, the tracking process unit 61 can extract a contour of the object displayed at the display position of the arrow type icon 510 by utilizing a known contour extraction process and face detection process, so as to set the object as the tracking target from an extraction result and to set the image region in which image data of the object exists as the tracking target region on the frame image
[0153 plurality of target frame images (or a plurality of select images described later) have a function as so-called frame advance images (top forwarding images) noting the tracking target. Also in the case where a plurality of target frame images (or a plurality of select images described later) are noted, the action and the effect of adjusting the position interval between tracking targets at different time points to a desired one is realized.
[0090] the sequential photography possibility decision unit 55 (see FIG. 5) included in the sequential photography control unit 53 decides sequential photography possibility of the p target images prior to the sequential photography of the p target images. For concrete description, it is supposed that p is five, and the decision method will be described with reference to FIGS. 9 and 10. An image I.sub.n' illustrated in FIG. 9 is a virtual image of the first target image I.sub.n (hereinafter referred to as a virtual target image). The virtual target image I.sub.n' is not an image that is obtained by an actual photography but an image that is estimated from the tracking result information. A position 350 is a center position of the tracking target region on the virtual target image I.sub.n'. An arrow 360 indicates a movement direction of the tracking target on the image space. The movement direction 360 agrees with the direction from the above-mentioned center position (x.sub.A,y.sub.A) to the center position (x.sub.B,y.sub.B) (see FIGS. 7 and 8) or the direction from the center position (x.sub.C,y.sub.C) to the center position (x.sub.B,y.sub.B).
See Fig. 12 A, B, C. ).
Before the effective filing date of the claimed invention, it would have been obvious to one of ordinary skill in the art to modify DOHNO in view Takemoto with wherein the subject information includes a movement-direction mark for each of the plurality of subjects, and wherein the one or more processors further execute the instructions to display each movement direction mark at a position spaced apart from the detected position of the corresponding subject and displaced around the detected position in accordance with a movement direction of the corresponding subject as taught by Yokohata. The motivation for doing is to generate generating a so-called stroboscopic image for user experience .
Response to Arguments
Claim Rejection Under 35 U.S.C. 103
Applicant asserts “Claim 1: Solely in an effort to expedite prosecution and not acquiesce to the Office's rejection, Claim 1 has been amended to recite, in part, obtain three-dimensional shape data of a plurality of subjects that are different from one another; detect respective positions of the plurality of subjects based on the obtained threedimensional shape data; and generate a virtual viewpoint image using the obtained three-dimensional shape data based on information on the detected positions of the plurality of subjects, the virtual viewpoint image displaying subject information related to a movement of each of the plurality of subjects. Applicant submits that independent Claims 8 and 9 include similar amendments or features. As such, the below analysis will also apply to independent Claims 8 and 9. Applicant submits that DOHNO in view of TAKEMOTO, MAIZELS, CHEN, or DENG, whether considered individually or in proper combination, still do not teach or suggest the aforementioned features recited in independent Claim 1. That is to say, TAKEMOTO, MAIZELS, CHEN, or DENG does not remediate the deficiencies of DOHNO, and therefore, the aforementioned features of the present invention recited in independent Claim 1, still does not result. DOHNO describes that a user changes attributes of a virtual controller image (such as direction and power) through touch input, and that the virtual controller image and an operation target image are generated and displayed so that the position, speed, and direction of an operation target (e.g., a character) in a virtual space are changed and displayed in association with those attributes. DOHNO relates to display control for games and the like, and discloses a configuration in which position information, speed information, and directio information of an operation target are held (e.g., a manipulation target attribute storage), as well as a configuration for generating display images in accordance with a viewpoint (viewpoint attributes). However, in DOHNO, position detection is merely information that is "held" as an attribute of the operation target, and DOHNO does not disclose or suggest an aspect in which three-dimensional shape data of a subject in the real space is "acquired" and a position is "detected" based on such data. TAKEMOTO describes tracking a single "moving object llO" over a plurality of frames and predicting its future position. MAIZELS describes acquiring depth maps in a time series, analyzing movements of human body parts and the like to detect a movement direction and speed, and displaying visual feedback (e.g., icons) corresponding to positions of the user's body parts in order to control software applications such as games. MAIZELS captures the position and motion of a user's body through 3D sensing. However, MAIZELS does not disclose a configuration in which a virtual viewpoint image is generated using "three-dimensional shape data" of a subject, nor does it disclose a framework in which positions of a plurality of subjects are respectively detected and, based on such detections, a virtual viewpoint image is generated and information is displayed. CHEN describes detecting the positions and orientations of human body parts for AR guidance and the like, and displaying guidance information on a screen, such as a circular indicator. Chen's disclosure is directed to guiding a 3D scan, and is different from virtual viewpoint image generation (i.e., free-viewpoint video in which the viewpoint can be freely changed) and from detecting and visualizing positions based on three-dimensional shape data of a plurality of subjects. DENG describes discloses determining positions of one or more pixels in a current frame and in past frames, averaging those positions, and calculating a corrected position based on a difference between the current position and the averaged position. However, a review ofDOHNO in view of TAKEMOTO, MAIZELS, CHEN, or DENG, whether considered individually or in proper combination, indicates DOHNO in view of TAKEMOTO, MAIZELS, CHEN, or DENG do not teach or suggest the aforementioned features recited in the presently amended claims. In particular, TAKEMOTO does not disclose or suggest the amended limitation. Paragraphs [0066] and [0067] of TAKEMOTO describe obtaining a polygon centroid position of the same moving object in a current frame and a previous frame and calculating a motion vector based on the difference between those temporally separated positions. The moving object in the current frame and the moving object in the previous frame are not subjects that are different from one another. Rather, they are temporally different representations of the same moving object 110. The Examiner alleged to treat the presence of the same subject in different frames as a "plurality of subjects." Such an interpretation is inconsistent with the express language of the presently amended claims, which now requires a plurality of subjects that are different from one another. Different temporal instances of a single subject do not constitute subjects that are different from one another. DOHNO does not cure this deficiency. Although DOHNO states that a plurality of objects may be candidates for a manipulation target, DOHNO stores predetermined position, speed, and direction attributes of a current manipulation target in a virtual space. DOHNO does not obtain three-dimensional shape data of a plurality of different subjects or detect respective positions of those different subjects based on the obtained three-dimensional shape data. Further, arrows 150 and 160 shown in TAKEMOTO's figures are explanatory annotations illustrating a positional deviation caused by image-processing delay. TAKEMOTO does not disclose that those arrows are generated or displayed by the information processing apparatus as subject information related to movement. Accordingly, the explanatory arrows in TAKEMOTO cannot satisfy the claimed display of subject information related to movement. Thus, even when DOHNO and TAKEMOTO are considered in combination, the cited references do not teach or suggest obtaining three-dimensional shape data of a plurality of subjects that are different from one another, detecting respective positions of those subjects based on the obtained three-dimensional shape data, and enerating the claimed virtual viewpoint image displaying movement-related subject information for each of those subjects. As such, DOHNO in view of TAKEMOTO, MAIZELS, CHEN, or DENG, whether considered individually or in proper combination, does not teach or suggest at least "obtain three-dimensional shape data of a plurality of subjects that are different from one another; detect respective positions of the plurality of subjects based on the obtained three-dimensional shape data; and generate a virtual viewpoint image using the obtained three-dimensional shape data based on information on the detected positions of the plurality of subjects, the virtual viewpoint image displaying subject information related to a movement of each of the plurality of subjects" as presently claimed.”
Examiner disagrees.
DOHNO discloses [0102] FIG. 1, a computer 1 that executes an information processing program (hereinafter referred to as “program”). The computer 1 is an information processing apparatus such as a smartphone, a tablet terminal, a notebook computer, a personal computer, or a digital audio player. In FIG. 1, the computer 1 includes a display apparatus 2, an input part 3, a storage part 4, a communication part 5, a CPU (Central Processing Unit) 6, ROM (Read Only Memory) 7, and RAM (Random Access Memory) 8. [0103] the program that the computer 1 executes is a game program stored in RAM to perform the following:
[0102], [0103] the program that the computer 1 executes is a game program stored in RAM and executed by a CPU to perform the following:
[0125] As the game progresses, the position of a character and the like being the manipulation target change . When there exists a plurality of objects each being the manipulation target as the position changes, detection, the object being the manipulation target at the current time point changes. The detection of the manipulation target attribute stores, as a manipulation target attribute 16a, position information 16b that defines the position of the manipulation target, speed information 16c that defines the shift speed of the manipulation target, direction information 16d that defines the shift direction of the manipulation target, and target information 16e that defines which object is the manipulation target.
[0124] As the game progresses, the viewpoint in the virtual space changes, generated. Accordingly, the viewpoint attribute storage part 17 stores, as a viewpoint attribute 17a, the position of a viewpoint camera in three-dimensional virtual space.
[0116] - the virtual screen is generated by three-dimensional space, displaying the fourth image allows the user to easily recognize on what viewpoint the view is displayed .
[0015], [0017], [0116] - The first image on display may represent a shift speed of the manipulation target, and the second image display may represent a shift direction of the manipulation target.
[0015], [0017], [0116] - The first image on display may represent a shift speed of the manipulation target, and the second image display may represent a shift direction of the manipulation target. [0112] - The first image displayed indicating may be expressed in the viewpoint, as part of the functional state display part 10e, by a meter value, a count value, a numerical value, color, size, a shape, or a rotation speed in the viewpoint.).
Takemoto discloses [0035] - A moving object model acquisition unit 1400 acquires a moving object model based on the result of measuring the three-dimensional shape of the moving object 110 included in the captured image. [0067] - previous frame from the polygon centroid position in the current frame, thereby acquiring a motion vector, plurality of frames includes polygon centroid, object to calculate the motion vector thus “plurality of subjects” ([0067] In S720, the moving object model estimation unit 1460 reads out the polygon centroid position in the previous frame from the memory, and subtracts the polygon centroid position in the previous frame from the polygon centroid position in the current frame , thereby acquiring a motion vector 1720 in the space, different frames “respective position of the plurality of subjects of subjects”.
[0035] - A moving object model acquisition unit 1400 acquires a moving object model based on the result of measuring the three-dimensional shape of the moving object 110 included in the captured image. [0067] In S720, the moving object model estimation unit 1460 reads out the polygon centroid position in the previous frame from the memory, and subtracts the polygon centroid position in the previous frame from the polygon centroid position in the current frame , thereby acquiring a motion vector 1720 in the space, different frames “plurality of subjects”. FIG. 1E illustrates an arrow is generated, a moving object depicted at a position shifted in a horizontal direction by a length indicated by an arrow, “image”.
[0007], [0024] - FIGS. 1E, and 1F indicates a field of view, virtual viewpoint in a virtual reality space, in which the image 1000 is visible, and also indicates that the image to be displayed. [0024], [0067] - FIG. 1E illustrates a moving object depicted at a position shifted in a horizontal direction by a length indicated by an arrow based on the moving object model estimation unit 1460 reads out the polygon centroid position in the previous frame from the memory, and subtracts the polygon centroid position in the previous frame from the polygon centroid position in the current frame , thereby acquiring a motion vector 1720 in the space [0067] .
[0067] In S720, the moving object model estimation unit 1460 reads out the polygon centroid position in the previous frame from the memory, and subtracts the polygon centroid position in the previous frame from the polygon centroid position in the current frame , thereby acquiring a motion vector 1720 in the space, subject is in each of the frames thus “plurality of subjects”
Regarding dependent claim 2 – 14 , the Applicant asserts that they are not obvious over based on their dependency from independent claim 1. The examiner cannot concur with the Applicant respectfully from same reason noted in the examiner’s response to argument asserted from claim 1.
Conclusion
The prior art made of record and not relied upon is considered pertinent to Applicant's disclosure.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Ming Wu whose telephone number is (571)270-0724. The examiner can normally be reached on Monday - Friday: 9:30am - 6:00pm EST .
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Devona Faulk can be reached on 571-272-7515. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/MING WU/
Primary Examiner, Art Unit 2618
Prosecution Insights