DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
Regarding claim 1,
Step 1: Is the claim to a process, machine, manufacture or composition of matter?
Yes, the claim is directed to a system.
Step 2A Prong One: Does the claim recite an abstract idea, law of nature, or natural phenomenon?
The limitations of:
generating a label record and adding the label record to a label record database, wherein the label record is columnar and comprises the label modification request and the modification timestamp; (mental evaluation, generating or coming up with a label is something a human can do in their head and/or with pen and paper by looking at data and writing out the label)
in response to receiving, from a second device with a user interface on the computer network, a label history request including the dataset identifier, locate, in the label record database, a plurality of label records corresponding to the dataset identifier; (mental evaluation, a human can locate a label by looking at data and seeing what matches)
Step 2A Prong Two: Does the claim recite additional elements that integrate the judicial exception into a practical application?
The limitations of:
one or more processors; and a non-transitory computer-readable medium comprising instructions that when executed by the one or more processors cause operations comprising: (generic computer components to carry out the abstract idea MPEP 2106.05(f))
receiving, at a first time from a first device on a computer network, a label modification request for a dataset, wherein the label modification request comprises (1) a dataset identifier, (2) a model error indicator, (3) a user identifier, and (4) a new label, wherein the dataset identifier comprises a unique pointer to the dataset, wherein the model error indicator comprises an identifier of an artificial neural network and an indication of model performance of the artificial neural network when processing the dataset, wherein the user identifier comprises a name for a requester of label modification, and wherein the new label comprises a dataset name; (data transmitting, insignificant extra-solution activity MPEP 2106.05(g))
receiving a temporal identifier for the first time and recording the temporal identifier as a modification timestamp, wherein the temporal identifier is standardized across the computer network; based on retrieving the plurality of label records from the label record database, (data transmitting, insignificant extra-solution activity MPEP 2106.05(g))
generate, for display on the user interface, a summary of label records corresponding to the dataset identifier (displaying data on a screen, applying it MPEP 2106.05(f))
Step 2B: Does the claim recite additional elements that amount to significantly more than the judicial exception?
The limitations of:
one or more processors; and a non-transitory computer-readable medium comprising instructions that when executed by the one or more processors cause operations comprising: (generic computer components to carry out the abstract idea MPEP 2106.05(f))
receiving, at a first time from a first device on a computer network, a label modification request for a dataset, wherein the label modification request comprises (1) a dataset identifier, (2) a model error indicator, (3) a user identifier, and (4) a new label, wherein the dataset identifier comprises a unique pointer to the dataset, wherein the model error indicator comprises an identifier of an artificial neural network and an indication of model performance of the artificial neural network when processing the dataset, wherein the user identifier comprises a name for a requester of label modification, and wherein the new label comprises a dataset name; (data transmitting, insignificant extra-solution activity MPEP 2106.05(g), 2106.05(d)(II)(i))
receiving a temporal identifier for the first time and recording the temporal identifier as a modification timestamp, wherein the temporal identifier is standardized across the computer network; based on retrieving the plurality of label records from the label record database, (data transmitting, insignificant extra-solution activity MPEP 2106.05(g), 2106.05(d)(II)(i))
generate, for display on the user interface, a summary of label records corresponding to the dataset identifier (displaying data on a screen, applying it MPEP 2106.05(f))
Regarding claim 2,
Step 1: Is the claim to a process, machine, manufacture or composition of matter?
Yes, the claim is directed to a method.
Step 2A Prong One: Does the claim recite an abstract idea, law of nature, or natural phenomenon?
The limitations of:
in response to the label modification request, determining a dataset identifier and a model error indicator; (mental evaluation, a human can mentally make a determination of the dataset and error)
based on receiving the label modification request for the dataset, determining a modification timestamp; (mental evaluation, a human can mentally determine a modification)
generating a label record, wherein the label record comprises the dataset identifier, the model error indicator, and the modification timestamp; (mental evaluation, a human can mentally generate the record and/or write it down)
Step 2A Prong Two: Does the claim recite additional elements that integrate the judicial exception into a practical application?
The limitations of:
receiving, at a device on a computer network, a label modification request for a dataset; (data transmitting, insignificant extra-solution activity MPEP 2106.05(g))
generating the label record in a label record database, wherein the label record database comprises a plurality of label record (data transmitting, insignificant extra-solution activity MPEP 2106.05(g))
Step 2B: Does the claim recite additional elements that amount to significantly more than the judicial exception?
The limitations of:
receiving, at a device on a computer network, a label modification request for a dataset; (data transmitting, insignificant extra-solution activity MPEP 2106.05(g), 2106.05(d)(II)(i))
generating the label record in a label record database, wherein the label record database comprises a plurality of label record (data transmitting, insignificant extra-solution activity MPEP 2106.05(g). 2106.05(d)(II)(i))
Note independent claim 16 recites the same substantial subject matter as independent claim 2, only differing in embodiment. The difference in embodiment, a computer-readable medium as opposed to a method does not meaningfully change the above analysis and therefore the claim is subject to the same rejection.
Dependent claim 3 recites receiving data and displaying the records, MPEP 2106.05(f) and 2106.05(d)(II)(i) and applying the abstract idea.
Dependent claim 4 recites receiving a request, extracting error and displaying data, MPEP 2106.05(d)(II)(i).
Dependent claim 5 recites determining a user identifier (mental process), populating the record, receiving a third request, and displaying data, MPEP 2106.05(d)(II)(i).
Dependent claim 6 recites determining a field (mental evaluation) and populating it, MPEP 2106.05(d)(II)(i).
Dependent claim 7 recites determining an address and creation time and generating the identifier, mental evaluation.
Dependent claim 8 recites locating data (mental evaluation), inputting and generating data, applying it MPEP 2106.05(f).
Dependent claim 9 recites receiving a temporal identifier and recording it, MPEP 2106.05(d)(II)(i).
Dependent claim 10 recites associating a point in time, MPEP 2106.05(f).
Dependent claim 11 recites determining an update rate (mental evaluation) and generating on a display MPEP 2106.05(f).
Dependent claim 12 recites comparing error with a threshold (mental evaluation) and generating a warning MPEP 2106.05(f).
Dependent claim 13 recites determining whether data has been received and assigning a label (mental evaluation).
Dependent claim 14 recites retrieving label sets and determining a label, MPEP 2106.05(f) and mental evaluation.
Dependent claim 15 recites determining similarity metrics and determining a label, mental evaluation of data.
Dependent claims 17-18 correspond to dependent claims 3-4 respectively.
Dependent claim 19 corresponds to dependent claim 5.
Dependent claim 20 corresponds to dependent claim 8.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 2, 6, 9,16 is/are rejected under 35 U.S.C. 103 as being unpatentable over Malach, Eran, and Shai Shalev-Shwartz. "Decoupling" when to update" from" how to update" in view of Zheng, Songzhu, et al. "Error-bounded correction of noisy labels." further in view of Aber at al. US 2017/0287074
Regarding claims 2 and 16, Malach teaches “a method for documenting label versions for machine learning model input data, the method comprising: receiving, at a device on a computer network, a label modification request for a dataset” (pg. 1 “of “how to update”. As mentioned before, in the presence of noisy labels, if we update only when the classifier’s prediction differs from the available label, then at the end of the optimization process, these few updates will probably be mainly due to noisy labels” updating a label is analogous to modifying a label and pg. 2 “we update these two hypotheses using the backpropagation rule, when they disagree on the label” wherein disagreeing on a label triggers the request);
Malach pertains to label updating/modification. More specifically however, Zheng teaches “in response to the label modification request, determining a dataset identifier and a model error indicator” (Zheng pg. 6 §3 “We propose to directly test the confidence level of the noisy classifier to determine whether a label is correct. One additional requirement is that if we decide that a label is incorrect, we also need to decide what is the correct label. Therefore, instead of checking the confidence level, we check the likelihood ratio between f’s confidence on y and its confidence on its own label prediction, i.e., mx.”);
It would have been obvious to one having ordinary skill in the art at the time that the invention was effectively filed to combine the teachings of Malach with that of Zheng since a combination of known methods would yield predictable results. As shown in Zheng, error rate is a useful indicator in deciding when to update a label and therefore this would operate in a known and predictable manner by allowing better machine learning models.
While the references teach modifying labels, Aber more specifically teaches “based on receiving the label modification request for the dataset, determining a modification timestamp” (Aber [0110] “each record of a database table can include […] timestamps indicating particular actions that the user took with respect to the account (e.g., when the user created the account, when the user last updated the account, and so forth).”);
“generating a label record, wherein the label record comprises the dataset identifier, the model error indicator, and the modification timestamp” (Aber [0110] “. The database table can include one or more records (e.g., one or more “columns” or “rows” of data fields), each containing information regarding a particular user's financial portfolio. For instance, each record of a database table can include a unique identifier associated with the record (e.g., an integer or alphanumeric sequence uniquely identifying the record in the database table), a unique identifier associated with a particular user (e.g., an integer or alphanumeric sequence uniquely identifying the user), and timestamps indicating particular actions that the user took with respect to the account (e.g., when the user created the account, when the user last updated the account, and so forth)”); and
“generating the label record in a label record database, wherein the label record database comprises a plurality of label records” (Aber [0110] “For instance, each record of a database table” i.e. the records are generated and stored)
It would have been obvious to one having ordinary skill in the art at the time that the invention was effectively filed to combine the teachings of Malach and Zheng with that of Aber since as shown in Aber, label/record modification is a known concept and would operate in an expected manner with the systems above. Therefore by combining the techniques shown, one would have a way of updating labels using well-known and understood techniques.
Note that independent claim 16 recites the same substantial subject matter as independent claim 2, only differing in embodiment. The difference in embodiment, a computer-readable medium as opposed to a method is an obvious variation of the other.
Regarding claim 6, the Malach, Zheng, and Aber references have been addressed above. Aber further teaches “wherein generating the label record comprises: determining a first field corresponding to dataset identifiers for the label record; and populating the first field with the dataset identifier” ([0007] “at least one of the one or more third data structures can include a data field containing an identity of the particular second user, a data field containing an indication of the particular financial asset of the financial market, and a data field containing an indication of a predicted future price or return of that financial asset.”)
Regarding claim 9, the Malach, Zheng, and Aber references have been addressed above. Aber further teaches “wherein determining the modification timestamp comprises: receiving a temporal identifier for a point in time, wherein the temporal identifier comprises a standardized setting for recording times across the computer network; and recording the temporal identifier as the modification timestamp” ([0110] “a unique identifier associated with a particular user (e.g., an integer or alphanumeric sequence uniquely identifying the user), and timestamps indicating particular actions that the user took with respect to the account (e.g., when the user created the account, when the user last updated the account, and so forth).”)
Allowable Subject Matter
No prior art has been cited for claims 1, 3-5, 7-8, 10-15, and 17-20. The claims however remain rejected under 101.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure:
Chen et al. USPAT 10,990,850
Copper US 2020/0401939
Jiang, Heinrich, and Ofir Nachum. "Identifying and correcting label bias in machine learning." International conference on artificial intelligence and statistics. PMLR, 2020.
Feng, Jean. "Learning to safely approve updates to machine learning algorithms." Proceedings of the Conference on Health, Inference, and Learning. 2021.
Kulesza, Todd, et al. "Structured labeling to facilitate concept evolution in machine learning." U.S. Patent No. 10,318,572. 11 Jun. 2019.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KEVIN W FIGUEROA whose telephone number is (571)272-4623. The examiner can normally be reached Monday-Friday, 10AM-6PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, MIRANDA HUANG can be reached at (571)270-7092. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
KEVIN W FIGUEROA
Primary Examiner
Art Unit 2124
/Kevin W Figueroa/ Primary Examiner, Art Unit 2124
Prosecution Insights