Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
Examiner acknowledges that applicant’s priority claims to application no. PCT/JP2022/009828 filed on March 7, 2022, which claims priority from Japanese Patent Application No. 2021-051179 filed on March 25, 2021. Receipt is acknowledged of certified copies of papers required by 37 C.F.R. 1.55.
Status of Claims
The office has received claims 1–21 originally filed on 9/22/2023.
Claims 1, 3, 4, 10, 11, 14, 20, and 21 are currently amended.
No claims have been added
Claims 9 and 19 have been canceled.
Claims 1–8, 10–18, and 20–21 are currently pending and have been examined on the merits.
Claim Rejections - 35 U.S.C. § 101
35 U.S.C. § 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1–8, 10–18, and 20–21 are rejected under 35 U.S.C. § 101 because the claimed invention is directed to an abstract idea without significantly more.
Claim 11—which is representative of claims 1 and 21—recites an abstract idea identified in the bold language below:
A performance analysis system comprising:
an image capturing device for capturing a performance image of performer’s fingers playing a musical instrument;
a display device;
a memory storing instructions; and
a control device including at least one processor that implements the instructions to:
obtain performance data representing performance of a performer playing the musical instrument, wherein the performance data specifies a pitch to be played by the performer;
obtain the performance image;
determine whether the musical instrument is played by the performer in accordance with the performance data;
generate finger position data representing a position of each of the performer’ s fingers from the performance image, in a state where the musical instrument is determined to be played by the performer in accordance with the performance data;
generate fingering data representing fingering in a performance by the performer based on the performance data and the finger position data, wherein the control device, in generating the fingering data:
calculates, for each of the performer’s fingers, a probability that a position of a finger, among the performer’s fingers, when the pitch specified by the performance data is played by the finger, is a position of a finger represented by the finger position data, using the following expression:
p
C
k
η
k
=
n
=
v
0
,
σ
2
E
*
I
(
C
[
k
]
∈
R
n
)
R
n
where p represents the probability, n represents the specified pitch, k represents a finger number, C[k] represents a position of the finger number k, Rn represents a unit area associated with the pitch n, p(C[k]ηk = n) represents a probability that the finger with the finger number k plays the pitch n, symbol I(C[k]
∈
Rn) represents an indicator function that is set to a numerical value of 1 in a state where the position C[k] exists within the unit area Rn and that is set to a numerical value of 0 in a state where the position C[k] exists outside of the unit area Rn, symbol |Rn | represents an area of the unit area Rn, and symbol v(0, σ2E) represents observation noise expressed by a normal distribution of a mean 0 and a variance σ2, symbol E representing a unit matrix of 2 rows and 2 columns and symbol * representing a convolution the observation noise v(0, σ2E); and
estimates, based on the probability calculated for each of the performer’s fingers, which finger played the pitch specified by the performance data; and
display a performance image based on the generated fingering data.
Limitations A, B, and F through J above recite an approach to analyzing the performance of a musician by obtaining information about the performance and images of the performance, analyzing the finger position data in the images, and then generating and displaying fingering data based on the analysis. This recites an abstract idea under the “mental process” grouping because it uses observation, evaluation, and judgment that can be practically performed in the human mind. See MPEP 2106.04(a)(2)(III)(A). Here, the captured images are considered a physical aide such as pen and paper and does not negate the mental nature of the observation, evaluation, and judgment about the performer’s playing of the instrument. The bolded portions of these limitations might be carried out by a teacher or instructor who is analyzing the performance of a student with the goal of identifying deficiencies in that performance and giving feedback to the student to improve future performance. Moreover, the actions of obtaining performance image information, analyzing it, and then outputting the results parallels a claim to “collecting information, analyzing it, and displaying certain results of the collection and analysis,” where the data analysis steps are recited at a high level of generality such that they could practically be performed in the human mind, Electric Power Group v. Alstom, S.A., 830 F.3d 1350, 1353-54, 119 USPQ2d 1739, 1741-42 (Fed. Cir. 2016).
Limitations K through N have been newly added to the claim and define the parameters and a formula used to calculate probability of the performer’s fingers having played a particular pitch to estimate which finger played a pitch specified in the performance data. These limitations recite an abstract idea under the grouping mathematical concepts because, as explained in MPEP 2106.04(a)(2)(I), a claim that recites a numerical formula or equation will be considered as falling within the “mathematical concepts” grouping. Not only does the claim recite a formula, it also explains that the formula is used to calculate a probability and estimate a value, which are themselves mathematical concepts.
This judicial exceptions are not integrated into a practical application because the additional elements in limitations B through E (an imaging capturing device, a display, a memory, and a control device) are recited at a high level of generality. In combination these additional elements recite a general purpose computer. Collectively, these additional elements do not provide more than a general link to a technological environment and amount to an instruction to implement the mental process on a computer. As noted in MPEP 2106.04(a)(2)(III)(C), a claim that requires a computer may still recite a mental process.
MPEP 2106.04(a) explains that the groupings in MPEP § 2106.04(a)(2) are not mutually exclusive, i.e., some claims recite limitations that fall within more than one grouping or sub-grouping. For example, a claim reciting performing mathematical calculations using a formula that could be practically performed in the human mind may be considered to fall within the mathematical concepts grouping and the mental process grouping. Accordingly, examiners should identify at least one abstract idea grouping, but preferably identify all groupings to the extent possible, if a claim limitation(s) is determined to fall within multiple groupings and proceed with the analysis in Step 2A, prong 2.
As shown above, claim 11 recites additional elements in limitations B, C, D, E, and J comprising an image capturing device, display device, memory, and control device. These devices are recited at a high level of generality. Both individually and in combination these additional elements provide only a general link between the recited abstract ideas and a technological environment, i.e., recite the abstract idea with instructions to “apply it” in a generic technological environment. The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the judicial exception because upon reconsideration of the additional elements, and the claim as a whole, there is no more than a generic computer and the relationship between that computer and the abstract idea is generalized.
Therefore, claims 1, 11, and 21 are ineligible.
Regarding claims 2 and 12. These claims further recite that finger position data represents a performer’s right- and left-hand finger positions. This elaborates on what is being observed as part of the mental process and provide input to the mathematical concept. These claims do not recite any new additional elements for consideration under Step 2A, prong 2, or Step 2B. Therefore, claims 2 and 12 are ineligible.
Regarding claims 3 and 13. These claims further recite that finger position is estimated by analyzing the image and determining a left and right hand based on relative location of fingers and thumbs. This elaborates on what is being observed as part of the mental process. These claims do not recite any new additional elements for consideration under Step 2A, prong 2, or Step 2B. Therefore claims 3 and 13 are ineligible.
Regarding claims 4 and 14. These claims elaborate on the mental process by reciting not generating data if the performer is not playing the instrument. These claims do not recite any new additional elements for consideration under Step 2A, prong 2, or Step 2B. Therefore claims 4 and 14 are ineligible.
Regarding claims 5 and 15. These claims further recite “the musical instrument is a keyboard instrument including a keyboard;” “the performance image includes an image of the performer’s fingers and a keyboard image of the keyboard;” and “estimating the position of each of the performer’s fingers by analyzing the performance image.” These limitations simply elaborate on the mental process identified above.
Claims 5 and 15 also further recite “generating a transformation matrix for applying a projective transformation to the performance image so that the keyboard image in the performance image approximates a reference image indicating a reference instrument” and “performing a projective transformation using the transformation matrix on the estimated position of each of the performer’s fingers, and then generating the finger position data representing the position of each of the performer’s fingers after the transformation.” These limitations both elaborate on the mental process above, but also further recite mathematical concepts like mathematical calculations in the form of matrix mathematics like projective transformation. This interpretation of these portions of claims 5 and 15 is supported by applicant’s disclosure at 0041–0050 which describes the equations used to perform the “projective transformation” and explains this transformation is converting the finger positions from one coordinate system to another coordinate system.
Claims 5 and 15 do not recite any new additional elements for consideration under Step 2A, prong 2, or Step 2B. Therefore claims 5 and 15 are ineligible.
Regarding claims 6 and 16. These claims depend from claims 5 and 15 and further elaborate on the mathematical concepts identified in claims 1, 5, 11, and 15 above by reciting the matrix initialization and iterative updating of the matrix using mathematical calculations. Claims 6 and 16 do not recite any new additional elements for consideration under Step 2A, prong 2, or Step 2B. Therefore claims 6 and 16 are ineligible.
Regarding claims 7 and 17. These claims depend from claims 6 and 16 and further elaborate on the mathematical concepts identified in claims 1, 5, 6, 11, 15, and 16 above by reciting an intended use where the initial matrix is set based on an area of the keyboard image. This also leans into the mental process by reciting that the target area corresponding to an instruction by the performer in the image approximately corresponding to a specific pitch, i.e., focus the mathematical analysis on the part of the keyboard being played. This can be practically performed as mental observation without a computer. Claims 7 and 17 do not recite any new additional elements for consideration under Step 2A, prong 2, or Step 2B. Therefore claims 6 and 16 are ineligible.
Regarding claims 8 and 18. These claims depend from claims 5 and 15 but recite elements similar to claims 7 and 17 regarding what target area of the image the finger position data should be extracted from (i.e., observed in) and generating or setting the aforementioned transformation matrix based on that same target area. Again, these limitations elaborate on the mental process and mathematical concept identified in claims 1, 5–7, 11, and 15–17 above but do not recite any new additional elements for consideration under Step 2A, prong 2, or Step 2B. Therefore claims 7 and 17 are ineligible.
Regarding claims 10 and 20. Claims 10 and 20 echo the limitations of canceled claims 9 and 19 (now incorporated into claims 1 and 11) while further reciting mean and variance data of finger position included along with corresponding pitch in control data that specifies the pitch to be played by the performer. These claims elaborate on the mental process and mathematical concepts of claims 1 and 11. The mean and variance data reads more like guidance for the mental observation of whether the performer’s finger position corresponds to the pitch of the note performed at a given moment in the performance. Accordingly, claims 10 and 20 elaborate on the abstract ideas identified in claims 1 and 11.
The machine-learning estimation model in claims 10 and 20 is an additional element beyond the abstract ideas. However, the machine-learning in the claim simply receives and input to calculate a probability estimate of the position of the performer’s finger(s). Thus, the machine-learning estimation model is used passively to carry out the mathematical concept, and could simply be a static model as no actual machine learning is recited in the claims. Accordingly, the presence of the machine-learning model as an additional element is like an instruction to “apply it” to perform math, and only generally links the abstract idea to the generic computing environment. The analysis of this additional element is the same under Step 2A, prong 2 and Step 2B. Therefore, claims 10 and 20 are ineligible.
Novel/Non-Obivious Subject Matter
Claims 1–8, 10–18, 20, and 21 are found to be allowable over the prior art. As discussed in the previous office action, Soejima teaches the musical instrument is a keyboard instrument including a keyboard, the performance image includes an image of the performer’s fingers and a keyboard image of the keyboard, and the control device, in generating the finger position data: estimates the position of each of the performer’ s fingers by analyzing the performance image. For instance, Soejima FIG. 2 is a diagram illustrating an image captured by the image sensor of the camera 5. In the captured image, the x-coordinate and y-coordinate positions corresponding to each pixel of the image sensor are determined. In FIG. 2, the upper left coordinate (O, O), the upper right coordinate (SX, O), the lower left coordinate (O, SY), and the angle of view designated by the coordinates (SX, SY) The horizontal position of the camera 5 is adjusted so that the image 2i and the hand image 7i can be taken. Thus, Soejima teaches the use of a coordinate system that locates the keyboard and the performers hands in the image, which could be used in the projective transformation as that feature is described at 0041–0050 of applicant’s specification. However, the examiner finds that the feature of applying coordinates to generate a transformation matrix that is used to map a reference instrument onto the instrument in the image is novel and non-obvious over the prior art.
Regarding claims 9–10 and 19–20. Soejima teaches the performance data specifies a pitch to be played by the performer but the prior art identified by the examiner does not teach calculating a probability for each of the performer’s fingers and using that data to estimate or indicate the likelihood of which finger is playing the specified pitch. These features are recited in claims 9–10 and 19–20. Claims 10 and 20 additionally nominally incorporate a machine-learning estimation model to help estimate which finger might be playing the pitch. This combination of features is novel and non-obvious over the prior art.
Because claims 1, 11, and 21 now incorporate the subject matter previously recited in claims 9 and 19, claims 1, 11, and 21 and their dependent claims 2–8, 10, 12–18, and 20 are novel and non-obvious over the prior art. The examiner notes that while these claims are allowable over the prior art, these claims stand rejected under 35 U.S.C. § 101 and as such are not allowable at this time.
Response to Amendment and Remarks
Objections
On page 11 the applicant highlights the amendments to the claims and specification have overcome the objections. These amendments address the objections, which are withdrawn.
Section 101
On pages 11–12 of the remarks, applicant lists aspects of the amended independent claims and argues “the above claimed features integrate the examiner’s noted judicial exception into a practical application.”
On pages 12–13 applicant specifically argues:
Here, tracking a player’s fingering performance, namely determining whether a player correctly played the pitch using a correct finger would be too complex for a person to practically monitor. That is, while a person may be able to determine whether the notes have been correctly played based on the performance data, it would not be practical for a person to mentally execute at least the claimed features (D)-(E) in particular, even with using pen and paper.
The examiner respectfully disagrees. Under the broadest reasonable interpretation of the amended claims, “performance data representing performance by a performer on a musical instrument, wherein the performance data specifies a pitch to be played” encompasses a wide variety of musical instruments and the many ways they are played. This could be an instrument as simple as a recorder or as complex as a piano or contrabassoon. The claims imply performance of a pitch determined by finger location, but there are also pitched percussion instruments such as tympani or marimba. Additionally, there is nothing in the claim requiring a tempo, time value, or sequence of the pitched notes to be played, which could include monotone whole notes played at a slow tempo, or even a major scale at one quarter note per second. Under this broadest reasonable interpretation, it would be practical for a person to monitor a performance and assess whether the correct notes are being played using the expected finger positions.
Although the specific limitations noted in applicant’s remarks include calculating a probability that a position of a finger is the same as a position represented by finger position data, this aspect of the claim is not considered a mental process under the Alice framework. Rather, consistent with MPEP 2106.04(a)(2)(I)(A), these newly claimed limitations are considered a mathematical concept because the recite a relationship between variables or numbers and is expressed in words and using mathematical symbols. For example, pressure (p) can be described as the ratio between the magnitude of the normal force (F) and area of the surface on contact (A), or it can be set forth in the form of an equation such as p = F/A. Accordingly, the remarks are not persuasive with respect to Step 2A, prong 1 and whether the claim recites an abstract idea.
On page 13, applicant further argues:
Second, in a PTAB decision in Ex parte Okura, Appeal 2020-002341, independent claim 3 called for a method for liquid chromatography that recited two equations for calculating elution time and mobility of a component in the calculated elution time. The examiner took the position that claim 3 was directed to an abstract idea because of using mathematical equations for calculating mobility. The PTAB held that the recited method steps were sufficient to transform the abstract idea into an inventive concept under Step 2A, Prong 2 of the USPTO’s 2019 Revised 101 Guidance. The PTAB also held that claim 3 is directed to a liquid chromatography process that uses more accurate estimates to control liquid chromatography process to yield more accurate result over prior art methods, and reflects improvement in technology, rooted in computer technology, over the existing technology.
The examiner has considered these remarks but does not find them persuasive. The PTAB decision referenced appears to support the examiner’s position under Step 2A, prong 1 that the claim recites an abstract idea because the decision turns on eligibility at Step 2A, prong 2. If the claim did not recite an abstract idea, it would have been eligible at Step 2A, prong 1 and the analysis would not have reached prong 2. Additionally, there is a wide difference between application of a mathematical concept that improves technological control over a physical or chemical process, such as in Diamond v. Diehr or the claim in the PTAB decision referenced by applicant, and one that simply quantifies the human activity of playing a musical instrument. This difference is more acute where “improvement” of musical performance might be subjective in the mind of the listener as compared to the objective improvement to the quality of physical, chemical, or technological processes. Accordingly, the remarks are not persuasive.
On pages 13–14, the applicant further argues:
Moreover, as recently emphasized by Deputy Commission for Patents in a memorandum dated 4 August 2025, specifically directed to Technology Centers 2100, 2600, and 3600, in computer-related technologies, the examiner must consider the improvements in technology for integrating a judicial exception into a practical application:
[quoting from the memorandum]
Here, by generating the finger position data as set forth in the claimed feature (D) and then generating fingering data as set forth in the claimed feature (E), the present development can more accurately estimate which finger played the pitch/note specified in the performance data in displaying a performance image. This improves the performance analyzing technology, which is rooted in computer technology, and integrates the judicial exception into a practical application under Step 2A, Prong 2, in particular by using the claimed expression set forth in the claimed feature (E)(1).
Indeed, as noted in present ¶ 4, it is difficult to determine the actual fingering with high accuracy using only the note sequence (performance data). In the present development, as noted in present ¶ 72, both finger position data F generated by analyzing the performance image G1 and the performance data P representing the performance by the user are used to generate the fingering data Q to estimate the fingering with high accuracy in comparison with a configuration in which the fingering is estimated only from the performance data P. The claimed features (D)-(E) in particular provide a particular solution for improving accuracy in the finger analyzing technology for a musical instrument.
In view of the foregoing, applicant submits that the pending claims are in condition for allowance. Should the examiner have any issues concerning this reply or any other outstanding issues remaining in this application, applicant urges the examiner to contact the undersigned to expedite prosecution.
The examiner respectfully disagrees. In addition to the response above noting the differences between the present claims and those in the PTAB decision and Diamond v. Diehr, the sentiment that “This improves the performance analyzing technology, which is rooted in computer technology” is not persuasive. Analysis of musical performance is not rooted in technology but in human activity such as the work of musical instrument instructors and tutors which has existed likely since the invention of musical instruments in ancient cultures and societies around the world. Again, this is inherently subjective, and the application of a computer to quantify that performance is categorically different from an inherently or natively technological process. This difference is heightened when considering the broadest reasonable interpretation of the claim, which the examiner discussed above. Accordingly, even upon reconsidering the amended claim and the applicant’s remarks, the examiner is not persuaded and the rejection is maintained.
Section 103
The examiner has considered the amendments and remarks, and concurs with applicant that the independent claims now recite novel/non-obvious subject matter. Therefore, claims 1–8, 10–18, 20, and 21 are now allowable over the art, notwithstanding subject matter eligibility.
Section 112
Notwithstanding applicant’s remarks, the amendments to claims 1, 11, and 21, in combination with the amendments to claims 4 and 14 have clarified and positively recited the different conditions under which finger position data is generated or not generated and thereby have overcome the 112 rejections.
Conclusion
Applicant’s amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 C.F.R. 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 C.F.R. 1.17(a)) pursuant to 37 C.F.R. 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Patrick McAtee whose telephone number is (571)272-7575. The examiner can normally be reached Weekdays 8:30am - 4:30pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Tariq Hafiz can be reached at (571) 272-5350. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
Patrick McAtee
Supervisory Patent Examiner
Art Unit 3698
/PATRICK MCATEE/Supervisory Patent Examiner, Art Unit 3698