DETAILED ACTION
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This is in reply to an Amendment filed on June 10, 2026 regarding Application No. 19/291,837. Applicants amended claims 1, 8-9, 14-15, and 20. Claims 1-20 are pending.
Priority
The instant application is a continuation of Internation Application No. PCT/JP2024/008009, filed on March 4, 2024.
Acknowledgment is made of Applicants’ claim for foreign priority under 35 U.S.C. 119(a)-(d). A certified copy of the JP 2023-037114 application filed in Japan on March 10, 2023 has been filed.
Response to Arguments
Applicants’ amendments to claims 8, 14, and 20 and remarks (Remarks, p. 8) regarding claim objections are acknowledged. In view of the amendments, the objections are moot.
Applicants’ arguments have been fully considered and without conceding arguments regarding the prior art of record, they are not persuasive and/or are moot in view of new grounds of rejection.
In response to the arguments regarding Fujii (Remarks, pp. 9-10), the Office respectfully disagrees and submits that Fujii and/or as modified teaches and/or suggests all features recited in newly amended independent claims 1, 9, and 15. More specifically, figure 16 and paragraph [0071] of Fujii teach: extracting, from a performance video representing a performance of a first keyboard instrument by a performer, a first reference portion that is a video constituting a part of the performance video and including a hand of the performer; and superimposing the first reference portion on a keyboard portion of a second keyboard instrument, thereby generating a composite video, as recited in newly amended independent claims 1, 9, and 15.
Because all features of newly amended independent claims 1, 9, and 15 are taught and/or suggested by Fujii and/or as modified, as discussed above and in the rejections, these claims are not allowable. In addition, all claims depending from newly amended independent claims 1, 9, and 15 are not allowable by virtue of their individual corresponding dependencies, and as discussed in the rejections.
For the reasons discussed above and in the rejections, pending claims 1-20 are not allowable.
Claim Rejections - 35 USC § 102
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale or otherwise available to the public before the effective filing date of the claimed invention.
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1, 6-7, 9, and 12-13 are rejected under 35 U.S.C. 102(a)(1) and/or (a)(2) as being anticipated by Fujii in JP 2017-138992 A (hereinafter Fujii; August 6, 2025 IDS reference – an original copy and translation was provided with the IDS).
Regarding claim 1, Fujii teaches:
A video processing method realized by at least one processor (110 in FIG. 20) of a computer system, the method comprising (Fujii: FIGs. 16 and 20, Abstract (“... The present invention provides a program, being executed by a computer, of a display process for causing a display part to display, by optical transmission or superposing on sight of a field image of a user that is taken, a virtual image containing at least an articulation motion image obtained by motion capture of player's performance, and a body change image which is an image of body change at the time of performer's performance.”), “[0071]... [A]s shown in FIG. 16, when the field of view image (actual image) seen while wearing the HMD is field of view image A (multiple people are wearing the HMD and watching), a virtual image B is generated which includes a fingering image seen from that position, the back view of the performer, and part of a body change image (effect image), and the field of view image A and virtual image B are superimposed and image C is displayed on the display unit. In addition, virtual image B shown in Figure 16 displays not only fingering images, the performer’s back view, and part of the body change image (effect image), but also a video image of the performer, the musical score of the song being played, and a graph showing changes in body change information over time.”, and “[0102] The memory 111 [of the performance display device 100 in FIG. 20] is a main memory, and stores a program for performing processing corresponding to the display control unit 5 described above. The memory 112 is a hard disk, flash memory, or the like, and stores joint movement data, body change data, and body change image data. Then, the processor 110 executes the program stored in the memory 101 and processes the joint movement data, body change data, and body change image data, thereby realizing functions similar to those of the above-described embodiments.”, see also claim 1, [0021]-[0022], [0025], [0043], and “[0069]... [T]he fingering Image is not just an image of the fingers and arms, but also an image of the entire performer....”)):
extracting, from a performance video representing a performance of a first keyboard instrument by a performer, a first reference portion that is a video constituting a part of the performance video and including a hand of the performer (Fujii: extracting corresponding to superimposing the first reference portion on a keyboard portion of a second keyboard instrument; FIG. 16 and “[0071]... [A]s shown in FIG. 16, when the field of view image (actual image) seen while wearing the HMD is field of view image A (multiple people are wearing the HMD and watching), a virtual image B is generated which includes a fingering image seen from that position, the back view of the performer, and part of a body change image (effect image), and the field of view image A and virtual image B are superimposed and image C is displayed on the display unit. In addition, virtual image B shown in Figure 16 displays not only fingering images, the performer’s back view, and part of the body change image (effect image), but also a video image of the performer, the musical score of the song being played, and a graph showing changes in body change information over time.”, see also [0025], [0043] and “[0069]... [T]he fingering Image is not just an image of the fingers and arms, but also an image of the entire performer....”); and
superimposing the first reference portion on a keyboard portion of a second keyboard instrument, thereby generating a composite video (Fujii: FIG. 16, “[0021] The display unit 1 is, for example, a display unit of a head mounted display (HMD). Head-mounted displays include optically transparent head-mounted displays in which the image of the field of view is optically transmitted and displayed, and video transparent head-mounted displays in which the image of the field of view is captured and displayed, but either type is acceptable. The field of view image is a concept that includes not only an image of an instrument such as a piano that the user is looking at, but also an external scene, etc.”, and “[0071]... [A]s shown in FIG. 16, when the field of view image (actual image) seen while wearing the HMD is field of view image A (multiple people are wearing the HMD and watching), a virtual image B is generated which includes a fingering image seen from that position, the back view of the performer, and part of a body change image (effect image), and the field of view image A and virtual image B are superimposed and image C is displayed on the display unit. In addition, virtual image B shown in Figure 16 displays not only fingering images, the performer’s back view, and part of the body change image (effect image), but also a video image of the performer, the musical score of the song being played, and a graph showing changes in body change information over time.”, see also FIGs. 1, 3-4, 11-12, and 17, [0022], [0025], [0030]-[0031], [0043]-[0046], [0052], [0069], [0078], and [0098]).
Regarding claim 6, Fujii teaches:
The video processing method according to claim 1, further comprising
acquiring recorded video including the second keyboard instrument in real space from an imaging device (6 in FIGs. 3-4 of Fujii) (Fujii: FIGs. 1, 3-4, and 16, “[0021] The display unit 1 [in FIG. 1] is… a display unit of a head mounted display (HMD). Head-mounted displays include optically transparent head-mounted displays in which the image of the field of view is optically transmitted and displayed, and video transparent head-mounted displays in which the image of the field of view is captured and displayed, but either type is acceptable. The field of view image… includes not only an image of an instrument such as a piano that the user is looking at, but also an external scene, etc.”, “[0030]… [A] space recognition sensor 6 (including a visible light camera or an infrared camera, etc.). A typical example of the performance information display device 3 is a head-mounted display (HMD) that... allows the user to view the real world (field of view image) as well as computer graphics and the like.”, “[0045]… [T]he space recognition sensor 6 recognizes an object connected to the floor and estimates its shape and distance. It estimates whether the recognized object matches a stored piano model….”, and [0071], see also FIGs. 2 and 17, [0022], ”[0025]… [A] field of view image including a piano….”, and [0046]),
wherein the first reference portion is superimposed on the keyboard portion of the second keyboard instrument in the recorded video (Fujii: FIG. 16 and [0071], see also FIG. 2, “[0021] The display unit 1 [in FIG. 1] is… a display unit of a head mounted display (HMD). Head-mounted displays include optically transparent head-mounted displays in which the image of the field of view is optically transmitted and displayed, and video transparent head-mounted displays in which the image of the field of view is captured and displayed, but either type is acceptable. The field of view image… includes… an image of an instrument such as a piano that the user is looking at….”, “[0025]… FIG. 2 shows an example of a performance in which a piano performance is taken as an example, and an example of an image in which the joint movement image is a fingering image showing the fingering pattern of the plano player. Specifically, a fingering image… [is] superimposed on a field of view image including a piano….”, see also FIG. 17 and [0031]).
Regarding claim 7, Fujii teaches:
The video processing method according to claim 6, further comprising displaying the composite video on a display device (4 in FIGs. 3-4 of Fujii) (Fujii: FIGs. 2-5 and 16, “[0025]… [In FIG. 2,] a fingering image… [is] superimposed on a field of view image including a piano….”, “[0031] FIG. 4 is a diagram showing an example of an HMD which is the performance information display device 3…. The user wears the HMD and practices playing… while viewing images of the musical instruments and performances displayed thereon.”, “[0044]… [T]he image processing unit 13 analyzes, based on the signal from the spatial recognition sensor 6, what kind of Image is displayed on the display unit 4 of the performance information display device 3. specifically, which part of the piano the user is looking at (for example, which key on the plano).”, and [0071], see also FIGs. 1 and 12, “[0021] The display unit 1 is, for example, a display unit of a head mounted display (HMD). Head-mounted displays include optically transparent head-mounted displays in which the image of the field of view is optically transmitted and displayed, and video transparent head-mounted displays in which the image of the field of view is captured and displayed, but either type is acceptable. The field of view image is a concept that includes not only an image of an instrument such as a piano that the user is looking at, but also an external scene, etc.”, and “[0022] The display control unit 2 is… built into a head-mounted display. The display control unit 2… a virtual image including… a joint movement image obtained by motion capturing the performer's performance… on the field of view video and displays it on the display unit 1.”),
wherein the display device and the imaging device are mounted on a head of a user (Fujii: FIGs. 3-4 and 16, “[0030] The performance information display device 3 includes a display unit 4… and a space recognition sensor 6 (including a visible light camera or an infrared camera, etc.). A typical example of the performance information display device 3 is a head-mounted display (HMD) that is worn on the head….”, and [0071]).
Regarding claim 9, this claim is rejected under similar rationale as claim 1 above.
However, it is noted that claim 9 differs from claim 1 above in that the following are recited:
A video processing system comprising:
a controller including memory storing instructions and at least one processor that implements the instructions, the instructions comprising.
Fujii teaches:
A video processing system comprising (Fujii: FIGs. 16 and 20, Abstract (“... The present invention provides a program, being executed by a computer, of a display process for causing a display part to display, by optical transmission or superposing on sight of a field image of a user that is taken, a virtual image containing at least an articulation motion image obtained by motion capture of player's performance, and a body change image which is an image of body change at the time of performer's performance.”), “[0071]... [A]s shown in FIG. 16, when the field of view image (actual image) seen while wearing the HMD is field of view image A (multiple people are wearing the HMD and watching), a virtual image B is generated which includes a fingering image seen from that position, the back view of the performer, and part of a body change image (effect image), and the field of view image A and virtual image B are superimposed and image C is displayed on the display unit. In addition, virtual image B shown in Figure 16 displays not only fingering images, the performer’s back view, and part of the body change image (effect image), but also a video image of the performer, the musical score of the song being played, and a graph showing changes in body change information over time.”, and “[0102]... [T]he processor 110 executes the program stored in the memory 101 and processes joint motion data, body change data, and body change image data, thereby realizing the same function as the above-described embodiment.”, see also claim 1, [0021]-[0022], [0025], [0043], and “[0069]... [T]he fingering Image is not just an image of the fingers and arms, but also an image of the entire performer....”)):
a controller (5 in FIG. 3 and 111 in FIG. 20) including memory (111) storing instructions and at least one processor (110) that implements the instructions, the instructions comprising (Fujii: FIGs. 3, 5, and 20, “[0101]... [T]he performance display device 100 [in FIG. 20] includes a processor 110 , 8 memory 111 and a memory 112 ....”, and “[0102] The memory 111 is a main memory, and stores a program for performing processing corresponding to the display control unit 5 described above. The memory 112 is a hard disk, flash memory, or the like, and stores joint movement data, body change data, and body change image data. Then, the processor 110 executes the program stored in the memory 101 and processes the joint movement data, body change data, and body change image data, thereby realizing functions similar to those of the above-described embodiments.”).
Regarding claim 12, this claim is rejected under similar rationale as claim 6 above.
Regarding claim 13, this claim is rejected under similar rationale as claim 7 above.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries set forth in Graham v. John Deere Co., 383 U.S. 1, 148 USPQ 459 (1966), that are applied for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or non-obviousness.
Claims 2-5, 8, 10-11, and 14-20 are rejected under 35 U.S.C. 103 as being unpatentable over Fujii.
Regarding claim 2, Fujii teaches:
The video processing method according to claim 1.
However, it is noted that Fujii (FIG. 16) does not teach:
wherein
the first reference portion further includes a keyboard of the first keyboard instrument.
Fujii also teaches:
wherein
a first reference portion further includes a keyboard of a first keyboard instrument (Fujii: FIG. 17 and “[0078] FIG. 17 shows an example in which operator images of piano operators (keyboard) are displayed above an actual piano, and a performance image is displayed above this virtual operator image.”, see also “[0036] Fingering data is obtained by having a performer actually play an electronic musical instrument or the like, and acquiring MIDI data (note number (key position), duration (sound length), velocity (keystroke strength), etc.). At the same time, in order to obtain fingering data (data necessary to generate joint movement images), the performer's fingering (which can capture hand and finger movements in three dimensions, providing information on up/down, left/right, and forward/backward movements) is also photographed (motion captured) using an infrared sensor…. Based on the data thus obtained, three-dimensional coordinate data of the fingers relative to the instrument such as the plano is obtained, and this data is arranged in time series in accordance with the tempo information of the MIDI data.”).
Before the effective filing date of the claimed invention, it would have been obvious to include: the features also taught by Fujii, such that Fujii as modified teaches: the claimed features, to provide a user with a more realistic experience by showing key actuation.
Regarding claim 3, Fujii teaches:
The video processing method according to claim 1, further comprising superimposing a second reference portion (musical score of a song portion in FIGs. 16B-C in Fujii) representing a content (musical score of the song) of a performance by the performer (see FIGs. 16B-C) near the second keyboard instrument (in FIGs. 16A and C) (Fujii: FIGs. 16A-C and “[0071]… [A]s shown in Figure 16, when the field of view image (actual image) seen while wearing the HMD is field of view image A…, a virtual image B is generated which includes a fingering image seen from that position, the back view of the performer… and field of view image A and virtual image B are superimposed and image C is displayed on the display unit. In addition, virtual image B shown in Figure 16 displays not only fingering images… [and] the performer's back view…, but also a video image of the performer… [and] the musical score of the song being played….”, see also FIG. 2, “[0025]… [In FIG. 2,] a fingering image and a body change image are superimposed on a field of view image including a piano. The fingering image shows the fingers that should operate the plano keys (operators) superimposed on the keys….”, [0030], and [0045]).
However, it is noted that Fujii does not teach:
said superimposing a second reference portion on the second keyboard instrument,
but which would have been obvious to include, such that Fujii as modified teaches: the claimed features, since it would have been within the general skill of one of ordinary skill in the art to select features on the basis of their suitability for the intended use to provide a user with a performance experience corresponding to that in the real world by including music sheets of music being played by a keyboard player on a music rack
Regarding claim 4, Fujii as modified teaches:
The video processing method according to claim 3, further comprising extracting the second reference portion from the performance video (Fujii: extracting corresponding to superimposing the second reference portion on the second keyboard instrument; FIGs. 16A-C, “[0030]… [A] space recognition sensor 6 (including a visible light camera or an infrared camera, etc.)….”, “[0045]… [T]he space recognition sensor 6 recognizes an object connected to the floor and estimates its shape and distance. It estimates whether the recognized object matches a stored piano model. If the recognized object matches a known piano model, the angle and distance to the actual piano are estimated from the positional relationship between the shooting reference position and this object. Then, the positional relationship between the object and the keys of the known piano model is assumed to be known, and the position of the keys is estimated based on the obtained angle and distance from the piano.”, and “[0071]… [A]s shown in Figure 16,… field of view image A and virtual image B are superimposed and image C is displayed on the display unit…. [V]irtual image B… displays not only fingering images… [and] the performer's back view…, but also a video image of the performer… [and] the musical score of the song being played….”, see also FIG. 2, [0025], [0036], and [0069]; also, it would have been obvious to include the claimed features since it would have been within the general skill of one of ordinary skill in the art to select features on the basis of their suitability for the intended use so that the content and performer performance correlate).
Regarding claim 5, Fujii teaches:
The video processing method according to claim 1, further comprising generating depth information representing a depth of a surface of the hand of the performer (Fujii: “[0043]… [T]he fingering data is obtained in time series as three-dimensional coordinate data of the performer's fingers by motion capture.”, see also [0046]),
wherein the depth of the surface of the hand of the performer in the first reference portion is controlled in accordance with the depth information (Fujii: FIG. 16 and [0071], see also FIG. 2, “[0025] FIG. 2 shows an example of a performance in which a piano performance is taken as an example, and an example of an image in which the joint movement image is a fingering image showing the fingering pattern of the plano player. Specifically, a fingering image… [is] superimposed on a field of view image including a piano. The fingering image shows the fingers that should operate the plano keys (operators) superimposed on the keys….”, “[0043] By analyzing the fingering data and the MIDI data, It is possible to determine the timing at which a certain key on the keyboard should be struck.… [T]he fingering data is obtained in time series as three-dimensional coordinate data of the performer's fingers by motion capture.”, see also [0046]).
However, it is noted that Fujii does not teach:
said generating information from the performance video,
but which would have been obvious to include, such that Fujii as modified teaches: the claimed features, since it would have been within the general skill of one of ordinary skill in the art to select features on the basis of their suitability for the intended use to provide a three-dimensional keyboard performance image.
Regarding claim 8, Fujii teaches:
The video processing method according to claim 1.
However, it is noted that Fujii (FIG. 16) does not teach:
wherein
the first reference portion is superimposed on the second keyboard instrument placed in a virtual space, and the second keyboard instrument is a virtual keyboard instrument.
Fujii also teaches:
wherein
a first reference portion is superimposed on a second keyboard instrument placed in a virtual space, and the second keyboard instrument is a virtual keyboard instrument (Fujii: FIGs. 2, “[0021] The display unit 1 is, for example, a display unit of a head mounted display (HMD). Head-mounted displays include optically transparent head-mounted displays in which the image of the field of view is optically transmitted and displayed, and video transparent head-mounted displays in which the image of the field of view is captured and displayed, but either type is acceptable. The field of view image is a concept that includes not only an image of an instrument such as a piano that the user is looking at, but also an external scene, etc.”, and “[0025]… [In FIG. 2,] a fingering image… [is] superimposed on a field of view image including a piano….”, see also FIG. 17, “[0098]… [E]ven if there is no actual piano…, it is possible to generate images of a virtual piano… and display the images of the virtual piano… [and] fingering images… in any position of the user's choosing.”, and [0071]).
Before the effective filing date of the claimed invention, it would have been obvious to include: the features also taught by Fujii, such that Fujii as modified teaches: the claimed features, to provide a user with a virtual musical experience performed by a virtual performer playing a virtual keyboard where there is no actual keyboard.
Regarding claim 10, this claim is rejected under similar rationale as claim 3 above.
Regarding claim 11, this claim is rejected under similar rationale as claim 5 above.
Regarding claim 14, this claim is rejected under similar rationale as claim 8 above.
Regarding claim 15, Fujii teaches:
A computer-readable storage medium (111 in FIG. 20) storing a program executable by at least one processor (110) of a computer system to perform a video processing method, the video processing method comprising (Fujii: FIGs. 16 and 20, Abstract (“... The present invention provides a program, being executed by a computer, of a display process for causing a display part to display, by optical transmission or superposing on sight of a field image of a user that is taken, a virtual image containing at least an articulation motion image obtained by motion capture of player's performance, and a body change image which is an image of body change at the time of performer's performance.”), “[0071]... [A]s shown in FIG. 16, when the field of view image (actual image) seen while wearing the HMD is field of view image A (multiple people are wearing the HMD and watching), a virtual image B is generated which includes a fingering image seen from that position, the back view of the performer, and part of a body change image (effect image), and the field of view image A and virtual image B are superimposed and image C is displayed on the display unit. In addition, virtual image B shown in Figure 16 displays not only fingering images, the performer’s back view, and part of the body change image (effect image), but also a video image of the performer, the musical score of the song being played, and a graph showing changes in body change information over time.”, and “[0102] The memory 111 is a main memory, and stores a program for performing processing corresponding to the display control unit 5 described above. The memory 112 is a hard disk, flash memory, or the like, and stores joint movement data, body change data, and body change image data. Then, the processor 110 executes the program stored in the memory 101 and processes the joint movement data, body change data, and body change image data, thereby realizing functions similar to those of the above-described embodiments.”, see also claim 1, [0021]-[0022], [0025], [0043], and “[0069]... [T]he fingering Image is not just an image of the fingers and arms, but also an image of the entire performer....”).
extracting, from a performance video representing a performance of a first keyboard instrument by a performer, a first reference portion that is a video constituting a part of the performance video and including a hand of the performer (Fujii: extracting corresponding to superimposing the first reference portion on a keyboard portion of a second keyboard instrument; FIG. 16 and “[0071]... [A]s shown in FIG. 16, when the field of view image (actual image) seen while wearing the HMD is field of view image A (multiple people are wearing the HMD and watching), a virtual image B is generated which includes a fingering image seen from that position, the back view of the performer, and part of a body change image (effect image), and the field of view image A and virtual image B are superimposed and image C is displayed on the display unit. In addition, virtual image B shown in Figure 16 displays not only fingering images, the performer’s back view, and part of the body change image (effect image), but also a video image of the performer, the musical score of the song being played, and a graph showing changes in body change information over time.”, see also [0025], [0043] and “[0069]... [T]he fingering Image is not just an image of the fingers and arms, but also an image of the entire performer....”); and
superimposing the first reference portion on a keyboard portion of a second keyboard instrument, thereby generating a composite video (Fujii: FIG. 16, “[0021] The display unit 1 is, for example, a display unit of a head mounted display (HMD). Head-mounted displays include optically transparent head-mounted displays in which the image of the field of view is optically transmitted and displayed, and video transparent head-mounted displays in which the image of the field of view is captured and displayed, but either type is acceptable. The field of view image is a concept that includes not only an image of an instrument such as a piano that the user is looking at, but also an external scene, etc.”, and “[0071]... [A]s shown in FIG. 16, when the field of view image (actual image) seen while wearing the HMD is field of view image A (multiple people are wearing the HMD and watching), a virtual image B is generated which includes a fingering image seen from that position, the back view of the performer, and part of a body change image (effect image), and the field of view image A and virtual image B are superimposed and image C is displayed on the display unit. In addition, virtual image B shown in Figure 16 displays not only fingering images, the performer’s back view, and part of the body change image (effect image), but also a video image of the performer, the musical score of the song being played, and a graph showing changes in body change information over time.”, see also FIGs. 1, 3-4, 11-12, and 17, [0022], [0025], [0030]-[0031], [0043]-[0046], [0052], [0069], [0078], and [0098]).
However, it is noted that Fujii does not teach:
said computer-readable storage medium is a non-transitory computer-readable storage medium,
but which would have been obvious to include, such that Fujii as modified teaches: the claimed features, since it is well-known in the art, would have been obvious to try as one of a predictable, ascertainable, and finite group of features of transitory and non-transitory with a reasonable expectation of success for computer-readable storage medium, and would have been within the general skill of one of ordinary skill in the art to select features on the basis of their suitability for the intended use for computer-readable storage medium.
Regarding claim 16, this claim is rejected under similar rationale as claim 3 above.
Regarding claim 17, this claim is rejected under similar rationale as claim 5 above.
Regarding claim 18, Fujii as modified teaches:
The non-transitory computer-readable storage medium according to claim 15, further comprising
acquiring recorded video including the second keyboard instrument in real space from an imaging device (6 in FIGs. 3-4 of Fujii) (Fujii: FIGs. 1, 3-4, and 16, “[0021] The display unit 1 [in FIG. 1] is… a display unit of a head mounted display (HMD). Head-mounted displays include optically transparent head-mounted displays in which the image of the field of view is optically transmitted and displayed, and video transparent head-mounted displays in which the image of the field of view is captured and displayed, but either type is acceptable. The field of view image… includes not only an image of an instrument such as a piano that the user is looking at, but also an external scene, etc.”, “[0030]… [A] space recognition sensor 6 (including a visible light camera or an infrared camera, etc.). A typical example of the performance information display device 3 is a head-mounted display (HMD) that... allows the user to view the real world (field of view image) as well as computer graphics and the like.”, “[0045]… [T]he space recognition sensor 6 recognizes an object connected to the floor and estimates its shape and distance. It estimates whether the recognized object matches a stored piano model….”, and [0071], see also FIGs. 2 and 17, [0022], ”[0025]… [A] field of view image including a piano….”, and [0046]),
wherein the first reference portion is superimposed on the keyboard portion of the second keyboard instrument in the recorded video (Fujii: FIG. 16 and [0071], see also FIG. 2, “[0021] The display unit 1 [in FIG. 1] is… a display unit of a head mounted display (HMD). Head-mounted displays include optically transparent head-mounted displays in which the image of the field of view is optically transmitted and displayed, and video transparent head-mounted displays in which the image of the field of view is captured and displayed, but either type is acceptable. The field of view image… includes… an image of an instrument such as a piano that the user is looking at….”, “[0025]… FIG. 2 shows an example of a performance in which a piano performance is taken as an example, and an example of an image in which the joint movement image is a fingering image showing the fingering pattern of the plano player. Specifically, a fingering image… [is] superimposed on a field of view image including a piano….”, see also FIG. 17 and [0031]).
Regarding claim 19, Fujii as modified teaches:
The non-transitory computer-readable storage medium according to claim 18, further comprising
displaying the composite video on a display device (4 in FIGs. 3-4 of Fujii) (Fujii: FIGs. 2-5 and 16, “[0025]… [In FIG. 2,] a fingering image… [is] superimposed on a field of view image including a piano….”, “[0031] FIG. 4 is a diagram showing an example of an HMD which is the performance information display device 3…. The user wears the HMD and practices playing… while viewing images of the musical instruments and performances displayed thereon.”, “[0044]… [T]he image processing unit 13 analyzes, based on the signal from the spatial recognition sensor 6, what kind of Image is displayed on the display unit 4 of the performance information display device 3. specifically, which part of the piano the user is looking at (for example, which key on the plano).”, and [0071], see also FIGs. 1 and 12, “[0021] The display unit 1 is, for example, a display unit of a head mounted display (HMD). Head-mounted displays include optically transparent head-mounted displays in which the image of the field of view is optically transmitted and displayed, and video transparent head-mounted displays in which the image of the field of view is captured and displayed, but either type is acceptable. The field of view image is a concept that includes not only an image of an instrument such as a piano that the user is looking at, but also an external scene, etc.”, and “[0022] The display control unit 2 is… built into a head-mounted display. The display control unit 2… a virtual image including… a joint movement image obtained by motion capturing the performer's performance… on the field of view video and displays it on the display unit 1.”),
wherein the display device and the imaging device are mounted on a head of a user (Fujii: FIGs. 3-4 and 16, “[0030] The performance information display device 3 includes a display unit 4… and a space recognition sensor 6 (including a visible light camera or an infrared camera, etc.). A typical example of the performance information display device 3 is a head-mounted display (HMD) that is worn on the head….”, and [0071]).
Regarding claim 20, this claim is rejected under similar rationale as claim 8 above.
Conclusion
Applicants’ amendments necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicants are reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to K. Kiyabu whose telephone number is (571) 270-7836. The examiner can normally be reached Monday to Thursday 9:00 A.M. - 5:00 P.M. ET.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Temesghen Ghebretinsae, can be reached at (571) 272-3017. The fax number for the organization where this application or proceeding is assigned is (571) 273-8300.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, Applicants are encouraged to use the USPTO Automated Interview Request (AIR) at https://www.uspto.gov/patents/uspto-automated-interview-request-air-form.
Information regarding the status of an application may be obtained from Patent Center. Status information for published applications may be obtained from Patent Center. Status information for unpublished applications is available through Patent Center for authorized users only. Should you have questions about access to Patent Center, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free).
/K. K./
Examiner, Art Unit 2626
/TEMESGHEN GHEBRETINSAE/Supervisory Patent Examiner, Art Unit 2626 8/27/26
Prosecution Insights