DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
This action is responsive to the claims dated 2/23/2024.
Claims 1-10 are presented for examination.
Priority
Acknowledgement is made of applicant’s claim for foreign priority under 35 U.S.C. 119 (a)-(d). The certified copy has been received for the foreign priority application no. DE 10 2023 202 352.0 filed 3/15/2023.
Information Disclosure Statement
The information disclosure statements (IDS) submitted on 2/23/2024 and
3/26/2024 have been considered by the examiner.
Specification
The title of the invention is not descriptive. A new title is required that is clearly indicative of the invention to which the claims are directed.
The following title is suggested: "Apparatus, Sensor System and Method for Detecting an Activity by Converting Sensor Time Series Into a Graph Processed by a Graph Neural Network".
The disclosure is objected to because of the following informalities:
(a) on page 11, lines 19 and 20, the recitation "to form the next node w2" appears to be in error, because w2 designates a time window in Fig. 3 rather than a node, and the node formed from the sensor values at the points in time t2 to t6 is designated K2 on page 12, lines 9 and 10;
(b) on page 12, line 22 and on page 13, line 20, the term "neural graph network" is inconsistent with the term "graph neural network" used throughout the remainder of the disclosure and in the claims;
(c) on page 13, lines 7 and 8, the recitation "the computational effort required to process 10 the sensor data" contains a stray reference numeral that interrupts the sentence; and
(d) the first sentence of the Abstract, "The evaluation of sensor data in order to detect an activity.", is a sentence fragment that lacks a verb and does not read as a complete sentence.
Appropriate correction is required.
Claim Objections
Claims 2, 4, 9 and 10 are objected to because of the following informalities:
In claims 2 and 4, a comma should be inserted after "claim 1" and before "wherein", consistent with the form used in claims 3, 5 and 7;
In claim 9, the recitation "output sensor data as time series of sensor data of the monitored human activity" repeats the term "sensor data" redundantly within a single recitation and should be “sensor values”, and the singular article in "as a time series" is inconsistent with the plural form used in claims 1 and 10 and should be “as time series”; and
In claim 10, the recitation "the sensor data are received as time series of sensor data at discrete points in time" likewise repeats the term "sensor data" redundantly within a single recitation. Applicant is requested to conform the wording of these recitations to the corresponding recitations of claim 1.
Appropriate correction is required.
Claim Interpretation
The following is a quotation of 35 U.S.C. 112(f):
(f) Element in Claim for a Combination. – An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The following is a quotation of pre-AIA 35 U.S.C. 112, sixth paragraph:
An element in a claim for a combination may be expressed as a means or step for performing a specified function without the recital of structure, material, or acts in support thereof, and such claim shall be construed to cover the corresponding structure, material, or acts described in the specification and equivalents thereof.
The claims in this application are given their broadest reasonable interpretation using the plain meaning of the claim language in light of the specification as it would be understood by one of ordinary skill in the art. The broadest reasonable interpretation of a claim element (also commonly referred to as a claim limitation) is limited by the description in the specification when 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is invoked.
As explained in MPEP § 2181, subsection I, claim limitations that meet the following three-prong test will be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph:
(A) the claim limitation uses the term “means” or “step” or a term used as a substitute for “means” that is a generic placeholder (also called a nonce term or a non-structural term having no specific structural meaning) for performing the claimed function;
(B) the term “means” or “step” or the generic placeholder is modified by functional language, typically, but not always linked by the transition word “for” (e.g., “means for”) or another linking word or phrase, such as “configured to” or “so that”; and
(C) the term “means” or “step” or the generic placeholder is not modified by sufficient structure, material, or acts for performing the claimed function.
Use of the word “means” (or “step”) in a claim with functional language creates a rebuttable presumption that the claim limitation is to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites sufficient structure, material, or acts to entirely perform the recited function.
Absence of the word “means” (or “step”) in a claim creates a rebuttable presumption that the claim limitation is not to be treated in accordance with 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph. The presumption that the claim limitation is not interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, is rebutted when the claim limitation recites function without reciting sufficient structure, material or acts to entirely perform the recited function.
Claim limitations in this application that use the word “means” (or “step”) are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action. Conversely, claim limitations in this application that do not use the word “means” (or “step”) are not being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, except as otherwise indicated in an Office action.
This application includes one or more claim limitations that do not use the word “means,” but are nonetheless being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, because the claim limitation(s) uses a generic placeholder that is coupled with functional language without reciting sufficient structure to perform the recited function and the generic placeholder is not preceded by a structural modifier. Such claim limitations are:
(1) an input device configured to receive sensor data from at least one sensor, recited in claims 1 and 9. The term "device" is a generic placeholder listed in MPEP 2181, subsection I(A), and the modifier "input" states what the device receives rather than a type of structural device, so it does not remove the limitation from 35 U.S.C. 112(f). Compare Greenberg v. Ethicon Endo-Surgery, Inc., 91 F.3d 1580, 1583 (Fed. Cir. 1996) ("detent mechanism"). The claimed function is receiving sensor data from at least one sensor. The corresponding structure is the input device 11 of Figure 2, of which the specification states that "This input device 11 can receive the sensor values from the sensors 20-i" and that "the received sensor values can also be temporarily stored in a memory of the input device 11" (specification, page 9, lines 15-19), the sensor values being provided to the apparatus 10 "conductively or via a wireless communication link" (specification, page 9, lines 11-14; see also page 8, lines 15-21). Receiving data and storing data are functions coextensive with a general purpose microprocessor, so no algorithm is required for this limitation. In re Katz Interactive Call Processing Patent Litigation, 639 F.3d 1303, 1316 (Fed. Cir. 2011); EON Corp. IP Holdings LLC v. AT&T Mobility LLC, 785 F.3d 616, 622 (Fed. Cir. 2015). This limitation is interpreted to cover the input device 11 of Figure 2, together with its memory and the conductive or wireless communication link over which it receives the sensor values, and equivalents thereof.
(2) a graph generator configured to create a graph, wherein nodes of the graph include the received sensor data from the at least one sensor, recited in claims 1 and 9. The term "generator" is used here as a generic placeholder, and the modifier "graph" names the output produced rather than a type of structural device. See Advanced Ground Information Systems, Inc. v. Life360, Inc., 830 F.3d 1341, 1348-49 (Fed. Cir. 2016) ("symbol generator"). The claimed function is creating a graph whose nodes include the received sensor data from the at least one sensor. The corresponding structure is the graph generator 12 of Figure 2 programmed to carry out the graph creation procedure described at page 11, line 3 through page 13, line 16 of the specification and illustrated in Figures 3, 4 and 5: the sensor values falling within a specified time window are assigned to a node, the window w1 spanning the points in time t1 to t5 forming a first node K1 and the window then being shifted by one step so that the points in time t2 to t6 form the next node, or alternatively the sensor values of a single point in time are assigned to each node; sensor values of several sensors present at one point in time are combined into a vector, and the vectors of several points in time into a common matrix; and the nodes so formed are then linked by edges specified either by a first dependency matrix A1 populated from ascertained similarities or correlations between the nodes (page 12, lines 15-22) or by a predefined second dependency matrix A2 in which a one denotes an edge between temporally adjacent nodes and a zero denotes the absence of a connection (page 12, line 23 through page 13, line 8). This limitation is interpreted to cover the graph generator 12 programmed to carry out that procedure, and equivalents thereof.
(3) a processing device configured to process the graph using a graph neural network to detect an activity, recited in claims 1 and 9. The term "device" is a generic placeholder listed in MPEP 2181, subsection I(A), and the modifier "processing" states only what the device does. The recited graph neural network identifies the tool the device is said to use and is not itself structure that performs the claimed function. The claimed function is processing the graph using a graph neural network to detect an activity. The specification describes no corresponding structure for that function beyond the labeled block 13 of Figure 2, stating only that a graph neural network "is implemented in the processing device 13" and that the processing device 13 "can thereby in particular detect an activity" (specification, page 9, lines 25-29). Claims 3 and 5 recite additional functions of this same limitation, namely processing the graph using a predefined first dependency matrix and processing the graph using a predefined second dependency matrix, and the specification describes how each of those matrices is populated (page 12, lines 15-22; page 12, line 23 through page 13, line 8); that description does not, however, supply corresponding structure for detecting an activity by means of the graph neural network. Because the specification fails to describe corresponding structure for the entire claimed function, this limitation is rejected under 35 U.S.C. 112(b) below.
(4) a transformation device configured to carry out a time-frequency transformation of the sensor data for the nodes of the graph, recited in claim 6. The term "device" is a generic placeholder and the modifier "transformation" states only what the device does. The claimed function is carrying out a time-frequency transformation of the sensor data for the nodes of the graph. The corresponding structure is the transformation device 14 of Figure 2, of which the specification states that it "can carry out a time-frequency transformation of the sensor values" and that "The time-frequency transformation can in particular be a so-called wavelet transformation," by which the time-discrete sensor values are converted into frequency components of which only the relevant components, for example only low-frequency components in a range up to 15 Hz, are used to create the graph (specification, page 10, lines 16-30). A named, art-recognized mathematical transformation is an algorithm expressed as a mathematical formula and is sufficient corresponding structure. See MPEP 2181, subsection II(B). This limitation is interpreted to cover the transformation device 14 programmed to carry out a time-frequency transformation, in particular a wavelet transformation, of the sensor values, and equivalents thereof.
(5) a preprocessing device configured to carry out filtering and/or preprocessing of the received sensor data, recited in claim 8. The term "device" is a generic placeholder and the modifier "preprocessing" restates the claimed function itself. The limitation recites two functions in the alternative: filtering the received sensor data, and preprocessing the received sensor data. For the filtering function the corresponding structure is the preprocessing device 15 of Figure 2, in which a "band-pass filter or a notch filter can be implemented in the preprocessing device 15" (specification, page 10, lines 10-11). For the preprocessing function the specification describes no corresponding structure, stating only that "any other approaches for filtering or preprocessing the sensor values are generally possible too" (specification, page 10, lines 12-13). Because the described structure does not perform the entire claimed function, this limitation is rejected under 35 U.S.C. 112(b) below.
Because this/these claim limitations are being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, they are being interpreted to cover the corresponding structure described in the specification as performing the claimed function, and equivalents thereof.
If applicant does not intend to have this/these limitation(s) interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, applicant may: (1) amend the claim limitation(s) to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph (e.g., by reciting sufficient structure to perform the claimed function); or (2) present a sufficient showing that the claim limitation(s) recite(s) sufficient structure to perform the claimed function so as to avoid it/them being interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph.
Claim Rejections - 35 U.S.C. 112(a)
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of the first paragraph of pre-AIA 35 U.S.C. 112:
The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor of carrying out his invention.
Claims 1-10 are rejected under 35 U.S.C. 112(a) as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, at the time the application was filed, had possession of the claimed invention.
Each of independent claims 1, 9 and 10 requires that the graph be processed by a neural network in order to produce a detection of an activity. Claims 1 and 9 recite a processing device configured to process the graph using a graph neural network to detect an activity, and claim 10 recites processing the graph using a graph neural network in order to detect an activity. That language defines the operation by the result it achieves, namely the detection of an activity, and identifies the agent that achieves the result only by the name of a class of computational models.
The specification describes the creation of the graph in detail. It describes forming each node from the sensor values lying within a sliding time window, a first node K1 from the points in time t1 to t5 and a second node K2 from the points in time t2 to t6 (specification as filed, page 12, lines 4 to 14); the alternative construction in which each node carries the sensor values of a single point in time and a first graph is formed from the nodes K1 to Kn (page 13, lines 9 to 16); the ascertainment of similarities between nodes, for example in the form of correlations, and their expression in a first dependency matrix A1 (page 12, lines 15 to 22); and the specification of edges between temporally adjacent nodes by a second dependency matrix A2 that is defined in advance, in which connected nodes are denoted by a one and unconnected nodes by a zero (page 12, line 29 to page 13, line 8). The specification does not, however, describe how the graph so created is processed to yield a detected activity.
The entire disclosure directed to that operation is the statement that a graph neural network is implemented in the processing device 13, that the processing device 13 can thus process the graph provided by the graph generator 12 using the graph neural network, and that the processing device 13 can thereby in particular detect an activity (specification as filed, page 9, lines 25 to 29). The corresponding method disclosure states only that in step S3 the graph is processed using a graph neural network, and that this enables activities to be detected (page 14, lines 8 to 10). Each of those passages restates the function recited in the claims rather than describing how the function is performed.
The specification does not disclose the architecture of the graph neural network, the manner in which values are propagated along the edges or aggregated at the nodes, the manner in which an output identifying a detected activity is produced from the processed graph, or the set of activities the network is able to distinguish. The specification likewise does not disclose how the graph neural network is obtained: it identifies no training data, no labeling of such data, no objective or criterion applied during training, and no result obtained from a trained network. The specification contains no working example and identifies no example as prophetic in which an activity is in fact detected. Because a neural network that has not been trained produces no activity detection, the absence of any description of how the network is arrived at leaves the recited result unexplained.
Original claims are part of the disclosure as filed, but the written description requirement applies to original claims as well, and difficulty in satisfying that requirement often arises where the claim language is generic or functional. Where the claims define the invention in functional language specifying a desired result, the algorithm or the steps taken to perform the function must be described in sufficient detail that one of ordinary skill in the art would understand how the inventor intended the function to be performed, and simply restating the function recited in the claim is not sufficient. It is not enough that one skilled in the art could write a program to achieve the claimed function, because the specification must explain how the inventor intends to achieve it. See MPEP Section 2161.01, subsection I; Vasudevan Software, Inc. v. MicroStrategy, Inc., 782 F.3d 671, 681-83 (Fed. Cir. 2015); Ariad Pharmaceuticals, Inc. v. Eli Lilly & Co., 598 F.3d 1336, 1349-51 (Fed. Cir. 2010) (en banc).
Claims 1-8 and claim 10 are rejected for the further reason that the recitation of detecting an activity is not limited to any particular kind of activity or to any particular kind of sensor, whereas the specification describes only the detection of human activity, and in particular sporting activity, from characteristics of a user acquired by body-worn sensors such as speed sensors, acceleration sensors, magnetic field sensors, a gyroscope and pressure sensors (specification as filed, page 8, lines 22 to 24, and page 9, line 29 to page 10, line 6). The only statement addressed to activities of any other kind is the assertion that the principle can generally also be applied to the evaluation of sensor data for detecting actions in other contexts (page 8, lines 1 and 2). That assertion states the boundary of the claimed subject matter without describing what lies within it, and an adequate written description of a claimed genus requires more than a generic statement of an invention's boundaries. Ariad, 598 F.3d at 1349-50. Claim 9 is not rejected on this further ground, because claim 9 requires the at least one sensor to be configured to monitor human activity and to output sensor data of the monitored human activity.
Claims 2-8 depend from claim 1 and are rejected for the reasons given above, none of them supplying the missing description. Claims 3 and 5 recite that the processing device processes the graph using a predefined first dependency matrix and a predefined second dependency matrix respectively, but those matrices specify the similarity between nodes and the edges between nodes and therefore further define the graph that is supplied to the graph neural network, not the manner in which the graph neural network detects an activity.
Applicant may overcome this rejection by directing the Examiner to a description, in the specification as filed, of the manner in which the graph neural network detects an activity, or by amending the claims to recite subject matter commensurate with the description the specification does provide. No new matter may be added.
Claim Rejections - 35 U.S.C. 112(b)
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claims 1-9 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant) regards as the invention.
Claim 1 recites the limitation a processing device configured to process the graph using a graph neural network to detect an activity. This limitation invokes 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, for the reasons given in the Claim Interpretation set forth above. The claimed function is processing the graph using a graph neural network to detect an activity. The specification describes a processing device 13, shown in Figure 2 as a labeled block, and states that a graph neural network "is implemented in the processing device 13" and that the processing device 13 "can thereby in particular detect an activity" (specification, page 9, lines 25-29). The specification does not, however, describe any algorithm or procedure by which the graph neural network detects an activity from the graph. It describes no network architecture, no operation performed on the nodes or the edges of the graph, no training or determination of network parameters, no mapping of a network output to a detected activity, and no rule by which an activity is determined to have been detected. The remainder of the description recites only the advantages said to follow from using a graph neural network rather than a convolutional neural network (specification, page 13, lines 17-27).
Detecting an activity from a graph by means of a graph neural network is not a function coextensive with a general purpose microprocessor, so an algorithm must be described. See MPEP 2181, subsection II(B); Aristocrat Technologies Australia Pty Ltd. v. International Game Technology, 521 F.3d 1328, 1333 (Fed. Cir. 2008); EON Corp. IP Holdings LLC v. AT&T Mobility LLC, 785 F.3d 616, 622-23 (Fed. Cir. 2015). Simply reciting that a computer performs the function is not adequate, and a labeled block that names the function it performs is a black box designed to perform the recited function rather than an explanation of how the function is performed. Blackboard, Inc. v. Desire2Learn, Inc., 574 F.3d 1371, 1383-85 (Fed. Cir. 2009); Advanced Ground Information Systems, Inc. v. Life360, Inc., 830 F.3d 1341, 1348-49 (Fed. Cir. 2016). A bare statement that a known technique or method can be used does not describe structure. Biomedino, LLC v. Waters Technologies Corp., 490 F.3d 946, 952 (Fed. Cir. 2007). Because the written description fails to describe the corresponding structure, material, or acts for performing the entire claimed function, and to clearly link the structure, material, or acts to the function, claim 1 is indefinite and is rejected under 35 U.S.C. 112(b) or pre-AIA 35 U.S.C. 112, second paragraph. See also Williamson v. Citrix Online, LLC, 792 F.3d 1339, 1351-52 (Fed. Cir. 2015) (en banc); MPEP 2185.
Claim 9 recites the same limitation a processing device configured to process the graph using a graph neural network to detect an activity and is rejected under 35 U.S.C. 112(b) or pre-AIA 35 U.S.C. 112, second paragraph, for the same reason.
Claims 2-8 depend from claim 1 and are rejected under 35 U.S.C. 112(b) or pre-AIA 35 U.S.C. 112, second paragraph, because they incorporate the indefinite limitation of claim 1 and do not cure the deficiency. Regarding claims 3 and 5, the further recitations that the processing device is configured to process the graph using a predefined first dependency matrix and using a predefined second dependency matrix add functions of the same limitation; the specification describes how each matrix is populated (page 12, lines 15-22; page 12, line 23 through page 13, line 8), but that description does not describe any algorithm for the function of detecting an activity by means of the graph neural network, so claims 3 and 5 remain indefinite.
Regarding claim 8, the claim recites the limitation a preprocessing device configured to carry out filtering and/or preprocessing of the received sensor data. This limitation invokes 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph, and recites two functions in the alternative: filtering the received sensor data, and preprocessing the received sensor data. For the filtering function, the specification describes a preprocessing device 15 in which a "band-pass filter or a notch filter can be implemented in the preprocessing device 15" (specification, page 10, lines 10-11). For the preprocessing function, the specification describes no structure at all; it states only that "any other approaches for filtering or preprocessing the sensor values are generally possible too" (specification, page 10, lines 12-13), which is a bare statement that known techniques can be used and does not describe structure. Biomedino, 490 F.3d at 952; MPEP 2181, subsection II(A). Because the claim recites the two functions in the alternative, the described band-pass filter and notch filter do not perform the entire claimed function. Where a limitation that invokes 35 U.S.C. 112(f) recites more than one function, the description of an algorithm for fewer than all of the recited functions is treated as the description of no algorithm at all. Media Rights Technologies, Inc. v. Capital One Financial Corp., 800 F.3d 1366, 1374 (Fed. Cir. 2015); Noah Systems, Inc. v. Intuit Inc., 675 F.3d 1302, 1318-19 (Fed. Cir. 2012). Claim 8 is therefore indefinite for this additional reason.
Applicant may: (a) amend the claims so that the claim limitations identified above will no longer be interpreted under 35 U.S.C. 112(f) or pre-AIA 35 U.S.C. 112, sixth paragraph; (b) amend the written description of the specification such that it expressly recites what structure, material, or acts perform the entire claimed function, without introducing any new matter (35 U.S.C. 132(a)); or (c) amend the written description of the specification such that it clearly links the structure, material, or acts described therein to the function recited in the claim, without introducing any new matter (35 U.S.C. 132(a)). An amendment of the written description must not add matter that the original disclosure would not reasonably convey to a person of ordinary skill in the art. In re Rasmussen, 650 F.2d 1212, 1214 (CCPA 1981); 35 U.S.C. 132.
If applicant is of the opinion that the written description of the specification already implicitly or inherently discloses the corresponding structure, material, or acts and clearly links them to the function so that one of ordinary skill in the art would recognize what structure, material, or acts perform the claimed function, applicant should clarify the record by either: (a) amending the written description of the specification such that it expressly recites the corresponding structure, material, or acts for performing the claimed function and clearly links or associates the structure, material, or acts to the claimed function, without introducing any new matter (35 U.S.C. 132(a)); or (b) stating on the record what the corresponding structure, material, or acts, which are implicitly or inherently set forth in the written description of the specification, perform the claimed function.
Claim Rejections - 35 U.S.C. 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claims 1-3, 9 and 10 are rejected under 35 U.S.C. 102(a)(1) as anticipated by Yan et al. (hereinafter Yan) “Deep Transfer Learning with Graph Neural Network for Sensor-Based Human Activity Recognition” 2022.
Regarding independent claim 1, Yan discloses an apparatus for processing sensor data (Yan: p. 7, section 4.2.1, "we use the Pytorch framework"; Yan's implemented graph neural network software necessarily runs on programmed computing hardware and therefore constitutes an apparatus for processing sensor data), comprising:
an input device configured to receive sensor data from at least one sensor (Yan: p. 7, Algorithm 1, "Input: Multichannel wearable sensor signals, the activity labels corresponding to the signal segments, the number of Chebyshev polynomial order K, the learning rate r"; the implemented model necessarily includes an input mechanism that receives signals from the wearable sensors), wherein the sensor data are received as time series of sensor values at discrete points in time (Yan: p. 7, section 4.1.3, "The sampling rate of the sensors is 50Hz in the TNDA HAR dataset."; values sampled at 50 Hz are a time series of sensor values at discrete instants);
a graph generator configured to create a graph (Yan: p. 4, section 3.1, "data preparation which converts multi-channel signals into graph data"; Yan's data-preparation software creates the claimed graph from the received signals), wherein nodes of the graph include the received sensor data from the at least one sensor (Yan: p. 3, section 2.1.2, "X(0) means the original node feature matrix X", p.4, section 3.2 defines X as the multichannel sensor-signal slice, so the original node-feature matrix includes the received sensor signals); and
a processing device configured to process the graph using a graph neural network to detect an activity (Yan: p. 12, Conclusion, "The ResGCNN structure is a multi-layer neural network composed of GNN with Chebyshev filtering functions and residual structures, which is designed to learn sensor signal representations and recognize human activities"; the programmed ResGCNN processes the graph's sensor-signal representation and produces an activity classification).
Regarding dependent claim 2, Yan further discloses the apparatus according to claim 1, each node of the graph respectively includes the sensor data (Yan: p. 3, section 2.1.2, "X(0) means the original node feature matrix X", p.4, section 3.2 defines X as the multichannel sensor-signal slice; each row or vector of X is a graph-node feature formed from one sensor-channel signal in the temporal slice) from a predetermined number of successive points in time (Yan: p. 7, section 4.1.4, "Thirdly, we use a temporal size of 128 with 50% is used to extract signal segments"; each sensor-channel node feature is a fixed 128-sample segment and therefore includes a predetermined number of successive samples).
Regarding dependent claim 3, Yan further discloses the apparatus according to claim 2, wherein the processing device is configured to process the graph using a predefined first dependency matrix (Yan: p. 7, “Algorithm 1: Network Parameter Training of ResGCNN Model”, step 2, "Initialize the adjacency matrix A based on 10 and 11"; ResGCNN Model is configured wherein A is initialized (using a predefined first dependency matrix) before the degree, Laplacian, and GNN filtering computations and is therefore predefined for graph processing), wherein the first dependency matrix specifies a similarity between two respective nodes in the graph (Yan: p. 4, section 3.2, equations 10-11, "We build the HAR non-directed graph adjacency matrix with the Pearson's correlation coefficients and a threshold value ψ"; thus each adjacency matrix element A(i,j) in the graph is determined by the Pearson correlation between the respective sensor-signal node vectors, a pairwise similarity measure).
Regarding independent claim 9, Yan discloses a sensor system, comprising:
at least one sensor configured to monitor human activity (Yan: p. 7, section 4.1.3, "We use the IMU sensors to capture the physical activity information for HAR and topological nonlinear dynamics analysis"; the disclosed body worn IMUs monitor subjects' physical activities for human-activity recognition) and output sensor data as time series of [[sensor data]]sensor values (interpreted per the Claim Objections set forth above) of the monitored human activity (Yan: p. 7, section 4.1.3, "The sampling rate of the sensors is 50Hz in the TNDA HAR dataset"; thus the IMUs output sampled time-series data while the listed human activities are performed); and
an apparatus for processing the sensor data (Yan: p. 7, section 4.2.1, "we use the Pytorch framework"; Yan's implemented graph neural network software necessarily runs on programmed computing hardware and therefore constitutes an apparatus for processing the sensor data), including:
an input device configured to receive the sensor data from the at least one sensor, wherein the sensor data are received as [[a]] time series (interpreted per the Claim Objections set forth above) of sensor values at discrete points in time (Yan: p. 7, Algorithm 1, "Input: Multichannel wearable sensor signals, the activity labels corresponding to the signal segments, the number of Chebyshev polynomial order K, the learning rate r" and section 4.1.3, “sampling rate of the sensors is 50 Hz in the TNDA HAR dataset”; the implemented system receives the IMUs' discretely sampled multichannel time-series signals),
a graph generator configured to create a graph (Yan: p. 4, section 3.1, "data preparation which converts multi-channel signals into graph data"; the data-preparation module creates the sensor-data graph), wherein nodes of the graph include the received sensor data from the at least one sensor (Yan: p. 3, section 2.1.2, "X(0) means the original node feature matrix X", p.4, section 3.2 defines X as the multichannel sensor-signal slice; the received sensor-signal slice X is used as the graph's original node-feature matrix), and
a processing device configured to process the graph using a graph neural network to detect an activity (Yan: p. 12, Conclusion, "The ResGCNN structure is a multi-layer neural network composed of GNN with Chebyshev filtering functions and residual structures, which is designed to learn sensor signal representations and recognize human activities"; the ResGCNN processes the graph representation and detects the monitored human activity).
Regarding independent claim 10, Yan discloses a method for processing sensor data, comprising the following steps: receiving sensor data from at least one sensor, wherein the sensor data are received as time series of [[sensor data]]sensor values (interpreted per the Claim Objections set forth above) at discrete points in time (Yan: p. 7, Algorithm 1, "Input: Multichannel wearable sensor signals, the activity labels corresponding to the signal segments, the number of Chebyshev polynomial order K, the learning rate r" and section 4.1.3, “sampling rate of the sensors is 50 Hz in the TNDA HAR dataset”; the method receives the wearable IMU sensor's discretely sampled multichannel time-series signals);
creating a graph, wherein nodes of the graph include sensor data from the at least one sensor (Yan: p. 4, section 3.1, "data preparation which converts multi-channel signals into graph data" and section 3.2 defines X as the multichannel sensor-signal slice; Yan converts the multichannel signals into graph data and uses the sensor-signal X as the node-feature matrix); and
processing the graph using a graph neural network in order to detect an activity (Yan: p. 12, Conclusion, " The ResGCNN structure is a multi-layer neural network composed of GNN with Chebyshev filtering functions and residual structures, which is designed to learn sensor signal representations and recognize human activities"; the method processes the graph with ResGCNN and outputs the recognized activity).
Claim Rejections - 35 U.S.C. 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 4-5 are rejected under 35 U.S.C. 103 as unpatentable over Yan, as applied in the rejection of claim 1 above, in view of Zhai et al. (hereinafter Zhai) “Spatial Temporal Network for Image and Skeleton Based Group Activity Recognition” 2022.
Regarding dependent claim 4, Yan teaches all the elements of claim 1.
Yan does not expressly teach wherein each node respectively includes the sensor data of a point in time and the graph includes a predetermined number of nodes with the sensor data from successive points in time.
However, Zhai teaches wherein each node respectively includes the sensor data of a point in time (Zhai: p. 8, Spatial GCN, "the attribute of vti is the coordinates and estimate confidence" and further node v(t,i) carries the joint coordinate and confidence derived from image-sensor frame t and therefore contains sensor-derived data for one time point) and the graph includes a predetermined number of nodes with the sensor data from successive points in time (Zhai: p. 8, Spatial GCN, "We construct the temporal graph Gi = (Vi, Ei) on the ith joint node in the different frames, where Vi = {vti|t = 1, 2, . . . , T}" and p. 9, 4.2 Implementation details, “We select T = 10”; the temporal graph includes one node for each of the successive T frames, and Zhai's implementation fixes T at ten).
Because Yan and Zhai are analogous art, with both concerning graph-neural-network activity recognition using temporally ordered, sensor-derived human-motion data, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to use Zhai's fixed length temporal node organization to modify Yan's sensor GNN, with a reasonable expectation of success, to capture dynamics between successive samples. This modification would have been motivated by the desire to provide more robust group activity recognition in video analysis and improved scene understanding (Zhai Abstract and Keywords).
Regarding dependent claim 5, Yan, in view of Zhai, further teaches the apparatus according to claim 4, wherein the processing device is configured to process the graph (Yan: p. 12, Conclusion, "The ResGCNN structure is a multi-layer neural network composed of GNN with Chebyshev filtering functions and residual structures, which is designed to learn sensor signal representations”; ResGCNN structure is configured to process the graph) using a predefined second dependency matrix (Zhai: p. 8, section 3.2, equation 5, "A is the adjacent matrix of the input graph"; the Temporal GCN uses input adjacency A, which is fixed before convolution by Zhai's stated M-frame rule; wherein the modification of Yan with Zhai includes utilizing Zhai’s Temporal GCN with input adjacency matrix A (using a predefined second dependency matrix)), wherein the second dependency matrix specifies edges between the nodes in the graph (Zhai: p. 8, section 3.2, equation 7, "We only consider M frames around the tth frame"; equation 7 sets A(T)(t,t') to one within the fixed temporal neighborhood and zero otherwise, thereby specifying edges between the time-point nodes).
Claim 6 is rejected under 35 U.S.C. 103 as unpatentable over Yan, as applied in the rejection of claim 1 above, in view of Nedorubova et al. (hereinafter Nedorubova) “Human Activity Recognition using Continuous Wavelet Transform and Convolutional Neural Networks” 2021.
Regarding dependent claim 6, Yan, further teaches the apparatus according to claim 1, further comprising: the sensor data for the nodes of the graph (Yan: p. 3, section 2.1.2, "X(0) means the original node feature matrix X"; and section 3.2 defines X as the multichannel sensor-signal slice, so each node of Yan's graph includes the received sensor data of its channel, carried in sampled time-domain form).
Yan does not expressly teach further comprising: a transformation device configured to carry out a time-frequency transformation of the sensor data for the nodes of the graph, and wherein the nodes respectively include the transformed sensor data.
However, Nedorubova teaches a transformation device configured to carry out a time-frequency transformation of the sensor data for nodes (Nedorubova: p. 3, Abstract, "The model we suggest is based on continuous wavelet transform (CWT)…Wavelet transform localizes signal features both in time and frequency domains and after that a CNN extracts these features and recognizes activity", p. 13 and Figure 7; programmed CWT stage (a transformation device) performs a time-frequency transformation of accelerometer data and supplies that representation to an activity classifier implemented with 2D CNN with fully-connected layers of neurons (configured to carry out a time-frequency transformation of the sensor data for nodes)), and wherein the nodes respectively include the transformed sensor data (Nedorubova: p. 13, Figure 7 and section 2.3, "we convert the 1D accelerometer signal into the 2D images via applying CWT in order to extract signal features"; the 2D CWT images are the transformed sensor data, and Nedorubova supplies these transformed data, rather than the raw 1D signal, as shown in Figure 7, as data content presented to its activity classifier neurons (and wherein the nodes respectively include)).
Because Yan and Nedorubova are analogous art, with both from the same field of endeavor as the claimed invention, using neural networks to recognize human activity from sampled wearable accelerometer signals, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Yan's apparatus, with a reasonable expectation of success, by adding a transformation stage that applies Nedorubova's continuous wavelet transformation to each multichannel sensor signal slice that Yan uses as a node feature, and to use the resulting time-frequency representation in place of the raw slice as the node content, so that the modified apparatus teaches further comprising: a transformation device configured to carry out a time-frequency transformation of the sensor data for the nodes of the graph, and wherein the nodes respectively include the transformed sensor data. The motivation is Nedorubova's teaching that wavelet transform localizes signal features both in time and frequency domains, so applying that known transformation to each of Yan's node signals predictably exposes time-localized spectral characteristics to Yan's GNN classifier (KSR: use of a known technique to improve similar devices in the same way with predictable results).
Claim 7 is rejected under 35 U.S.C. 103 as unpatentable over Yan, in view of Nedorubova, as applied in the rejection of claim 6 above, and further in view of Anna Ferrari et al. (hereinafter Ferrari) “Trends in human activity recognition using smartphones” 2021.
Regarding dependent claim 7, Yan, in view of Nedorubova, teach wherein the time-frequency transformation includes a wavelet transformation (Nedorubova: p. 9, section 2.2.5, "Thus, CWT is the method we implement in the current work", p. 13, section 2.3, "we convert the 1D accelerometer signal into the 2D images via applying CWT in order to extract signal features"; CWT is the continuous wavelet transformation applied to the activity sensor signal).
Yan and Nedorubova do not expressly teach and wherein the nodes respectively include only frequency components up to a predetermined cutoff frequency.
However, Ferrari teaches wherein the nodes respectively include only frequency components up to a predetermined cutoff frequency (Ferrari: p. 193, section 4, "Filtering is also used to clear raw data from artifacts. It is stated that a cut-off frequency of 15Hz is enough to capture human body motion which energy spectrum lies between 0 Hz and 15 Hz [49,52]"; teaches low-pass filtering the raw accelerometer signal at a predetermined 15-Hz cut-off frequency, so the filtered signal passes onward contains only frequency components up to that predetermined cutoff).
Because Yan, Nedorubova, and Ferrari are analogous art because each is from the same field of endeavor as the claimed invention, human-activity recognition from wearable or smartphone inertial-sensor signals, it would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to further modify the apparatus of Yan and Nedorubova, with a reasonable expectation of success, in which the time-frequency transformation includes Nedorubova's wavelet transformation, by placing Ferrari's 15-Hz low-pass filtering ahead of that transformation to teach and wherein the nodes respectively include only frequency components up to a predetermined cutoff frequency. The motivation is Ferrari's teaching that filtering clears raw data from artifacts and that a 15-Hz cut-off frequency is enough to capture human body motion, so band-limiting each node signal before the transformation predictably removes artifact energy that cannot represent the monitored activity while preserving the body-motion components that do (KSR: use of a known technique to improve similar devices in the same way with predictable results). Additionally, this combination advantageously opens opportunities in a variety of applications contexts such as surveillance, healthcare, and delivering (Ferrari: p. 189, section 1 Introduction).
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Yan, as applied in the rejection of claim 1 above, in view of Ferrari.
Regarding dependent claim 8, Yan teaches all the elements of claim 1.
Yan does not expressly teach further comprising: a preprocessing device configured to carry out filtering and/or preprocessing of the received sensor data.
However, Ferrari teaches a preprocessing device configured to carry out filtering and/or preprocessing of the received sensor data (Ferrari: p. 192-193, section 4 Preprocessing, "Filtering is also used to clear raw data from artifacts. It is stated that a cut-off frequency of 15Hz is enough to capture human body motion which energy spectrum lies between 0 Hz and 15 Hz"; teaches a filter (a preprocessing device) filtering of raw inertial data before recognition processing is filtering of the received sensor data (configured to carry out filtering and/or preprocessing of the received sensor data)).
Yan and Ferrari are analogous art because both are from the same field of endeavor as the claimed invention, human activity recognition from wearable inertial sensor signals. Ferrari states that filtering clears raw data from artifacts and that human body motion is captured below 15 Hz, so filtering the received sensor values removes energy that cannot represent the monitored activity while preserving what does. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention, with a reasonable expectation of success, to provide the apparatus of Yan with a preprocessing device carrying out filtering of the received sensor data, as taught by Ferrari, to remove artifacts from the raw sensor signals to teach further comprising: a preprocessing device configured to carry out filtering and/or preprocessing of the received sensor data. This modification would have been motivated by the desire to improve the reliability of the activity recognition and advantageously opens opportunities in a variety of applications contexts such as surveillance, healthcare, and delivering (Ferrari: p. 189, section 1 Introduction).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
KUMAR et al., US 2022/0405588 A1 (Dec. 22, 2022) (ABSTRACT Systems, methods, and computer-readable media provide a graph processing system that incorporates a graph neural network (GNN) based recommender system (RS), as well as a method for training a GNN based RS to address feature leakage that leads to overfitting of the trained GNN based RS. A message correction algorithm is used to modify a user node embedding and a positive item node embedding generated by the graph neural network when generating mini batches of training triples used to train the GNN based RS. The GNN message passing operations are performed on one graph only, in contrast to existing approaches which typically run GNN message passing operations on multiple adjusted input graphs constructed for multiple training triples).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KUANG FU CHEN whose telephone number is (571)272-1393. The examiner can normally be reached M-F 9:00-5:30pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Jennifer Welch can be reached on (571) 272-7212. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KC CHEN/Primary Patent Examiner, Art Unit 2143
Prosecution Insights