Examiner thanks Attorney Selfaison for the amendment to advance prosecution.
AIA
DETAILED ACTION
Status of claims
Claims 1-20 examined in response to 10/15/25 RCE.
Application Filed 8/18/2023.
Inventors Oron Madmon, Alexander Zlotnik, Rina Leibovits
Amended 1 14 18
New none
Canceled none
Response to Remarks
Applicant amendment remarks fully considered but unfortunately not fully persuasive.
101
The claim only has 1 model and doesn’t provide the multiple models as Applicant may think. The reason for this is the “intended use” language in the claim leads to features that are not positively claimed, with the consequence that later steps -- which assumed they were positively claimed -- never need happen; the claim doesn’t require their execution. So we have “to determine” and “designated to train” etc which are intended use. What we don’t have is multiple models as applicant seems to think we do; the claim requires a model. All applicant maybe needs to do is rephrase. So if you look at Hodos Fig 1b is says 2 or more.
As to applicant argument that
Mental steps
Examiner
Could be done with pencil and paper. Nothing in the claim makes it impractical to do the mental steps with the aid of tools like pen and paper.
Tedious? Maybe. Not impractical. See above reference to Paul Meier’s survival analysis. In 1950, calculations weren’t even considered tedious; calculations were one’s job and a person would be glad to have some tedium given the alternative (no tedium, no job, no income).
Applicant uses a computer as a tool to implement modeling of human behavior; computer not improved.
Applicant uses a computer as a tool to implement an abstract idea and there are benefits in lower resource consumption when automating (use a dishwasher, driving to work instead of walking, doing calculations on a computer rather than mentally, etc).
As to applicant argument that
No abstract idea (remarks p13)
Examiner
Overbreadth can result in preemption. Too, reading the claim in light of the Spec, claim directed to advertising.
As to applicant argument that
Resource consumption (remarks p15)
Examiner
Part of the idea, e.g. ensemble learning.
As to applicant argument that
Berkheimer (remarks p16)
Examiner
Examiner didn’t assert well-understood, conventional, routine
Besides, Examiner had earlier cited Wiley, an encyclopedia thousands of pages of well-understood, conventional, routine
https://archive.org/details/EncyclopediaofComputer/mode/2up
There’s nothing so substantial claimed that the claim can’t be performed mentally. Remember ‘mental steps’ under MPEP 2105-6 includes that with the aid of pencil and paper. “Computer Intensive Methods In Statistics” (Scientific American 1983 pp 116-131) illustrates that even back in 1983, Sir Ron Fisher didn’t use a computer in 1915 when picking the best model using bootstrapping, nor did Hotelling in 1933 (e.g. pp122-124)
103
As to applicant argument that
No protected model art cited by examiner (remarks p19)
Examiner
protected model ≈
Hodos benchmark and the like, suitability ¶ 25 26 35 39 40 41 in reference to base case Fig 1 & text. Protected, obvious from Hodos Fig 1-2, 6 & text; protected is model maintained with respect to a standard or reference on the way to prune, sample, down-size, update results in best model ¶ 78
Also Cohen ¶ 57 base case
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
Since the automatically analyzing step requires analyzing the only model to determine it is a protected model and designating to train it until it is a best model, then the threshold amount in the next terminating step must be either 0% or 100% because there are no other model variations. If it is 0% then the determining step results in no model in the model variation subset. If it is 100%, then the determining step results in the only model, which is a protected model, being in the model variation subset. Either way, the next terminating limitation results in terminating no processes or resources; Thus, neither the determining step nor the terminating step can be said to impose any functional limitation on the claim scope. Next, the initiating step is based on the model variation subset which mean it is based on an empty set (which raises 112(a) issues as I would not know how to create a new model based on an empty set), or it is based on the original model variation you created. If it is based on the original model variation you created it could either be designating the original model you created (which is the protected model and part of the model variation subset) as a new model variation or perhaps creating a new model variation (you can probably determine this from the Spec.), if it is the first interpretation then changing the name from a model variation, a protected model, a protected model with the model variation subset, or a new model variation does not impose a functional limitation on the scope of the claim. Thus, the limitation would merely require receiving a new model evaluation for the first model variation. If it is intended to require creating a new model variation and receiving a new evaluation score for the new model, then of course it does impose a functional limitation on the scope of the claim, so your prior art would need to disclose creating a new model. If the new model variation is just calling the original model variation a new model variation, then it is already the best model because it was trained to be so, thus the next determining step would not impose a new functional limitation on the claim. If you actually needed to create a new model, then the determining step merely require calling it a best model, because it is the only new model. Thus, regardless of the new model evaluation score it is now called a best model too, just like your first model variation. Now to the omitting step. It requires omitting both the protected model and the new model except the best model. If they are the same model, then the omitting step does not do anything because it is the best model. If your interpretation is that there are two models created (the first model variation and the new model variation), then the claim merely requires omitting one of the models. Which one is omitted unclear. Both models are a best model. the first model was trained unit it became a best model. The second model was also determined to be a best model. Thus, there is a 112(b) antecedent basis issue with the phrase “the best model”, as it could refer to either the first model variation or the new model variation.
The following is a quotation of 35 U.S.C. 112(d):
(d) REFERENCE IN DEPENDENT FORMS.—Subject to subsection (e), a claim in dependent form shall contain a reference to a claim previously set forth and then specify a further limitation of the subject matter claimed. A claim in dependent form shall be construed to incorporate by reference all the limitations of the claim to which it refers.
Claims 8/16 seem to fall short of 112d after the RCE amendment of 10/15/25. Those claims add protected which is already claimed in the independent claims.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title
The claimed invention is directed to a judicial exception (i.e., a law of nature, a natural phenomenon, or an abstract idea) without significantly more. The claim(s) is/are directed to one or more abstract idea(s). The claim(s) does/do not include additional elements that are sufficient to amount to significantly more than the abstract idea(s).
Step 1: (MPEP 2106.03)
The claims and dependents are directed to statutory classes (1 process, 14 machine 18 manufacture). The claims herein are directed to subject matter which would be classified under one of the listed statutory classifications (i.e., 2019 Revised Patent Subject Matter Eligibility Guidance (hereinafter “PEG”) “PEG” Step 1=Yes).
Step 2A, Prong One: Evaluating whether the claim(s) recite(s) a judicial exception -- law of nature, natural phenomenon, abstract idea. (MPEP 2106.04).
Claim 1-20: rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The claim recites an abstract idea,
Mental Steps Certain Methods of Organizing Human Activity.
Alice
clearinghouse
by computer
Here
pick good model, prune bad aka the aphorism “pick the best, leave the rest”
by computer
Bilski
hedge
by computer
https://en.wikipedia.org/wiki/Scientific_method and associated figure
CLAIM 1 (1 computer-implemented method, similar 14 system w/memory and processor to execute idea 18 medium with instructions for execution of idea)
PNG
media_image1.png
339
280
media_image1.png
Greyscale
1. A computer-implemented method for model evaluation and performance-based selection, the method comprising:
O receiving, by one or more processors, a model and corresponding configuration information
O creating, by the one or more processors, one or more model variations based on the model and the corresponding configuration information ,
O automatically analyzing, by the one or more processors, the one or more model variations to determine one or more protected models based on the configuration information, wherein the one or more protected models are designated to train until a best model is determined
O determining, by the one or more processors, a model variation subset based on a threshold amount and one or more evaluation scores of each of the one or more model variations , the threshold amount comprising a percentage of the one or more model variations that are to be omitted
O terminating, by the one or more processors, processes and resources for the one or more model variations not included in the model variation subset or not the one or more protected models
O initiating, by the one or more processors, one or more new model variations based on the model variation subset
O determining, by the one or more processors, a best model of the one or more new model variations based on one or more new model evaluation scores for each of the one or more new model variations
O omitting, by the one or more processors, the one or more protected models and the one or more new model variations except the best model
italics = judicial exception no italics = apply it MPEP 2105-2106
Claim(s) is/are directed to CERTAIN METHODS OF ORGANIZING HUMAN BEHAVIOR, MENTAL STEPS
The scientific method involves forming a hypothesis, test it, gather data, adjust hypothesis. It’s also called the scientific method (Francis Bacon 1620), consumer testing, feedback loop, control loop. Any workpiece will suffice.
The claim is the abstract idea of simulation (A/B testing).
Given the Patent Eligibility Guidance (PEG), the claims steps set forth
Mental Processes such as
concepts performed in the human mind (including an observation, evaluation, judgement, opinion)
Certain Methods of Organizing Human Activity such as
fundamental economic principles or practices (including hedging, insurance, mitigating risk)
commercial or legal interactions (including agreements in the form of contracts; legal obligations; advertising, marketing or sales activities or behaviors; business relations)
managing personal behavior or relationships or interactions between people (including social activities, teaching, and following rules or instructions)
Merriam Webster dictionary defines SIMULATION
PNG
media_image2.png
436
759
media_image2.png
Greyscale
The claim is the abstract idea Of ENSEMBLE LEARNING.
https://en.wikipedia.org/wiki/Ensemble_learning
PNG
media_image3.png
805
632
media_image3.png
Greyscale
Where “Mental Process” is the abstract idea, then any steps that cannot be performed in the human mind or by a human using pen and paper cannot be included as part of the identified abstract idea and, as such, must be considered as an “additional element” of the claimed invention.
So for claim 1:
receiving a model and corresponding configuration information (Additional element. Cannot be done in the human mind);
creating one or more model variations based on the model and the corresponding configuration information. (Mental Process. Can either create a model variation in our mind or write one on a piece of paper);
automatically analyzing the one or more model variations to determine one or more protected models based on the configuration information, wherein the one or more protected models are designated to train until a best model is determined; (Mental Process, Can analyze the model to determine it is a protected model and designate the model as a model to train in the human mind or by doing so on a piece of paper. Applicant does not claim actually training the model which could not be done in the human mind);
determining, a model variation subset based on a threshold amount and one or more evaluation scores of each of the one or more model variations, the threshold amount comprising a percentage of the one or more model variations that are to be omitted (Mental Process. Can make such a determination based on said information in the human mind or using pen and paper upon receipt of the threshold amount and the evaluation score. Applicant never claims executing the model to determine evaluation score, receiving the evaluations scores or where the threshold amount came from. If claimed, the receiving could not be done in the human mind. The executing of the model would be questionable. If you have interpreted a model as just an algorithm of a series of mathematical steps then the human mind could plug in numbers into the algorithm or execute the mathematical steps and determine the results of the algorithm which can be done in the human mind. If the model is something more tangible such as a physical model or a machine learning model then executing the model could not be done in the human mind or by pen and paper and, as such would be an additional element);
terminating, by the one or more processors, processes and resources for the one or more model variations not included in the model variation subset or not the one or more protected models; (Additional element. Cannot be performed in the human mind. Based on the applicant’s disclosure resources as used to train a model and processes are training processes (¶ 3, 14, 16,18, 42-43, 49-50, 52). The human mind can neither perform a training process nor exclude a resource being used to execute a training process);
initiating, by the one or more processors, one or more new model variations based on the model variation subset (Probably could include as Mental Process. Initiating could be just deciding hey this model variation will be a new model or conversely initiating could be just creating a new model variation which as indicate in the first creating step is a Mental Process);
determining, by the one or more processors, a best model of the one or more new model variations based on one or more new model evaluation scores for each of the one or more new model variations (Mental Process just like the first determining step and for the same reasons); and
omitting, by the one or more processors, the one or more protected models and the one or more new model variations except the best model (Mental Process. This is omitting the other created model variations, which if you created them using paper and pencil could be just crossing out the other model variations written down).
Thus, if designate the claim as reciting a “Mental Process” the additional elements that must be considered under Step 2a, Prong 2 and Step 2b, in addition to the one or more processes, are receiving a model… and terminating process and resources…. The receiving step is insignificant extra solution activity, so a general purpose computer with generic computer components and terminating processes and resources used by said process are the key considerations for Step 2a, Prong 2 and Step 2b. As terminating process and resources used by said processes occurs by merely closing a program such as a program of a model is not a technological improvement or an inventive step the claimed additional elements are not sufficient to transform the abstract idea into a practical application under Step 2a, Prong 2 and since it is well-known it would not be considered “significantly more” under Step 2b; closing programs inherently ends processes and resources used by such processes and is common knowledge would probably be sufficient.
Step 2A, Prong Two: Identifying whether there are any additional elements recited in the claim beyond the judicial exception(s); and then evaluating those additional elements individually and in combination to determine whether they integrate the exception into a practical application. Prong Two distinguishes claims that are "directed to" the recited judicial exception from claims that are not "directed to" the recited judicial exception. (MPEP 2106.04).
The claim says one is to take the idea and “apply it” with generic elements generally applied.
This judicial exception is not integrated into a practical application. In particular, the claim only recites an additional element to perform data gathering, data processing. The additional element -- recited at a high-level of generality (i.e., as a generic processor performing a generic computer function of ranking information based on a determined amount of use) such that it amounts no more than mere instructions to apply the exception using a generic computer component. Accordingly, this additional element does not integrate the abstract idea into a practical application because it does not impose any meaningful limits on practicing the abstract idea. The additional step computer, memory, medium, processor MPEP 2106.05 is mere applying the idea on a computer. See (MPEP 21056.05
Dependent claims 2-13 15-17 19-20 Collecting information, analyzing it, and displaying certain results of the collection and analysis (Electric Power Group)
Training/processing with machine learning is an idea itself of organizing information through mathematical correlations, using categories to organize, store and transmit information. Machine learning is old and well-known (NPL: “Approaches to Machine Learning, P. Langley at Carnegie-Mellon University (1984) and the references it refers to from more than a half-century ago).
SAP America (CAFC):
“We may assume that the techniques claimed are “[g]roundbreaking, innovative, or even brilliant,” but that is not enough for eligibility. Ass’n for Molecular Pathology v. Myriad Genetics, Inc., 569 U.S. 576, 591 (2013); buySAFE, Inc. v. Google, Inc., 765 F.3d 1350, 1352 (Fed. Cir. 2014). Nor is it enough for subject-matter eligibility that claimed techniques be novel and nonobvious in light of prior art, passing muster under 35 U.S.C. §§ 102 and 103. See Mayo Collaborative Servs. v. Prometheus Labs., Inc., 566 U.S. 66, 89–90 (2012); Synopsys, Inc. v. Mentor Graphics Corp., 839 F.3d 1138, 1151 (Fed. Cir. 2016) (“[A] claim for a new abstract idea is still an abstract idea. The search for a § 101 inventive concept is thus distinct” from demonstrating novelty or nonobviousness.
There’s nothing so substantial claimed that the claim can’t be performed mentally. Remember ‘mental steps’ under MPEP 2105-6 includes that with the aid of pencil and paper. “Computer Intensive Methods In Statistics” (Scientific American 1983 pp 116-131) illustrates that Sir Ron Fisher didn’t use a computer in 1915 when picking the best model using bootstrapping, nor did Hotelling in 1933 use a computer to do PCA to pick the best and leave the rest (e.g. pp122-124)
Applicant wants a patent to pick good model, prune bad. Related to this concept (pick the best, leave the rest) is a branch of statistics originated by University of Chicago’s Paul Meier, "Nonparametric estimation from incomplete observations” J. Amer. Statist. Assoc. 53 (282): 457–481 (1958) (University of Chicago Press), already provided to appellant, back in the 1950s. This is a seminal paper teaching survival analysis, e.g., of patients. At bottom of p.459-460, reader sees survival analysis data started with a small sample size of 100 in 1955, which could be calculated using pen and paper. Choosing to implement the idea via computer does not transform an idea into patent eligible invention. And to use a computer is mere Appellant’s choice of a particular technological environment, a choice of field of use. Page 463 shows sample size is selectable; it wouldn’t even have to be 100. Paul Meier used patient data. It could have just as easily been advertising data, a difference in data type i.e. the workpiece.
Step 2B: Identifying whether there are any additional elements (features/limitations/steps) recited in the claim beyond the judicial exception(s), and then evaluating those additional elements individually and in combination to determine whether they contribute an inventive concept (i.e., amount to significantly more than the judicial exception(s)). (MPEP 2106.05)
The claim recites additional elements. The claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. As discussed above with respect to integration of the abstract idea into a practical application, the additional element of using a computer, memory, processor, medium and amounts to no more than mere instructions to apply the exception using a generic computer component. See (MPEP 21056.05 Mere instructions to apply an exception using a generic computer component cannot provide an inventive concept. The claim is not patent eligible.
Viewed as a whole, the claim elements do not provide meaningful limitations to transform the abstract idea into a patent eligible application of the abstract idea such that the claims amount to significantly more than the abstract idea itself. The claim limitations do not improve upon the technical field that the abstract idea is applied nor do they improve upon any other technical field. The claimed limitations do not improve upon the functioning of the computer itself. Therefore, the claims are rejected under 35 U.S.C. 101 as being directed to non-statutory subject matter.
The additional element(s) or combination of elements in the claim(s) other than the abstract idea amount(s) to a ‘computer’, ‘memory’, ‘processor’ which use generic elements, MPEP 2016.05(d). Applicant specification says additional elements are generic. Any output ¶ 21, any config ¶ 21, any training ¶ 22, any computer ¶ 28, general purpose computer ¶ 63
FYI
From 2008 Wiley Encyclopedia of Comp Sci and Engr 2365pp
https://archive.org/details/EncyclopediaofComputer/mode/2up
The claim limitations alone or in ordered combination do not improve upon the technical field to which the abstract idea is applied nor do they improve upon any other technical field. The claimed limitations do not improve upon the functioning of any device itself. Wiley Encyclopedia of Computer Science and Engineering (2009) is a general technical reference with these generic elements, which was already provided to Applicant. The reference is the kind a person of ordinary skill in the art would have “hanging on their wall“, e.g. as a pdf shortcut or icon on wallpaper of one’s computer. Display is mentioned 427 times includes display (Wiley p.2261), memory at p. 2263 (mentioned 1700+times in Wiley), database, server p.125, server 610 times (at least e.g. p.1982), processor 639 times (e.g. p. 1242-1243), database 1728 times (e.g. p.1253), storage medium (e.g. p.131), computer (3553 times, e.g. p.283), network (at least p.1700-1707), interface (770 times at least p.1700-1707).
The additional elements alone or in combination are not sufficient to amount to significantly more than the judicial exception because the claims do not provide improvements to another technology or technical field, improvements to the functioning of the computer itself, and do not provide meaningful limitations beyond generic linking use of an abstract idea to a particular technological environment. Additionally, the claims are directed to an abstract idea with additional generic computer elements that do not add meaningful limitations to the abstract idea because they require no more than a generic computer to perform generic computer functions that are generic activities previously known to the industry. Moreover, these generic limitations do not lead to an integrated practical application because they are simply an attempt to limit the abstract idea to a particular technological environment, not meaningful limitations beyond generally linking the use of an abstract idea to a particular technological environment. See Alice Corp p 16 of slip op. noting that none of the hardware recited "offers a meaningful limitation beyond generally linking ‘the use of the [method] to a particular technological environment', that is implementation via computers"(citing Bilski 561 US at 610). Viewed as a whole, these additional claim elements do not provide meaningful limitations to transform the abstract idea into a patent eligible application of the abstract idea such that the claims amount to an integrated practical application. The claim limitations do not improve upon the technical field that the abstract idea is applied nor do they improve upon any other technical field. The claimed limitations do not improve upon the functioning of the computer itself.
Moreover, these generic limitations do not constitute significantly more because they are simply an attempt to limit the abstract idea to a particular technological environment, not meaningful limitations beyond generally linking the use of an abstract idea to a particular technological environment. See Alice Corp p 16 of slip op. noting that none of the hardware recited "offers a meaningful limitation beyond generally linking ‘the use of the [method] to a particular technological environment', that is implementation via computers"(citing Bilski 561 US at 610).
Moreover, mere recitation of a machine or medium in the preamble does not make a claim statutory under 35 U.S.C. 101, as seen in the Board of Patent Appeals Informative Opinion Ex Parte Langemyr (Appeal 2008-1495). Moreover, mere mention of a piece of a computer or processing device does not confer patentability. Alice Corporation Pty. Ltd. v CLS Bank International ("Alice Corp") 573 US __ (2014). Incorporating the two-step test espoused in its recent decision in Mayo v. Prometheus 566 U.S. ___ (2012), the Court describes a first inquiry as to whether the claims at issue are directed to a patent-ineligible concept. If so, the Court requires a second inquiry as to whether the elements, individually or in combination, “transform” the nature of the claims into a patent-eligible invention. The Court described this second step as a search for an inventive concept, “i.e., an element or combination sufficient to ensure that the patent in practice amounts to significantly more than a patent upon the [ineligible concept] itself.”
The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception because the additional elements merely detail generic elements that implement the abstract idea. The generically recited computer elements do not add a meaningful limitation to the abstract idea. The additional element merely instruct that the execution of the abreact idea occurs on other generic technology, but does not offer any disclosure of any additional technology beyond the abstract idea itself. Moreover, the claim steps as an ordered combination do not present significantly more. The claims are not directed to an improvement in computer functionality like in Enfish v Microsoft, but rather to an abstract idea. The claims "do nothing more than spell out what it means to 'apply it on a computer'”, Intellectual Ventures I 792 F.3d p1371 (citing Alice). Nowhere in the claims or specification is there any indication that the computer, processor, medium do something to improved hardware functionality.
The further elements of the claims are merely directed to further abstract ideas and in ordered combination pose a list of abstract ideas, and invoke merely as a tool what is generic. There is no improvement in these items, but rather they are invoked as a tool to solve a business problem (targeted marketing), not a technical problem.
The claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception because the additional elements merely detail generic computer processors and software that implement the abstract idea. The generically recited computer elements do not add a meaningful limitation to the abstract idea because they would be generic in any computer implementation. The additional element merely instruct that the execution of the abstract idea occurs on other generic technology, but does not offer any disclosure of any additional technology beyond the abstract idea itself. Moreover, the claim steps as an ordered combination do not present significantly more. The claims are not directed to an improvement in computer functionality like in Enfish v Microsoft, but rather to an abstract idea. The claims "do nothing more than spell out what it means to 'apply it on a computer'”, Intellectual Ventures I 792 F.3d p1371 (citing Alice). Nowhere in the claims or specification is there any indication that the computer, processor, storage do something nongeneric such that Applicant has improved computer functionality. Applicant presents an idea for which computers are invoked as a tool.
Here, the claims neither improve the technological infrastructure nor provide particular solutions to challenges. Rather, in ordered combination the claim limitations spell out the steps of calculating a number using generic technology (storage, computer, storage, processor – stated at a high level of generality Fig 4 & text).
In addition to these indisputably generic features, Applicant did not invent any of those features, and the claims do not recite them in a manner that produces a result that overrides the generic use of these known features. DDR Holdings, LLC v. Hotels.com, L.P., 773 F.3d 1245, 1258 (Fed. Cir. 2014). When viewed as an ordered combination, the proposed claims recite no more than the sort of “perfectly” generic computer components employed in a customary manner that we have held insufficient to transform the abstract idea into a patent-eligible invention. Intellectual Ventures I LLC v. Symantec Corp., 838 F.3d 1307, 1321 (Fed. Cir. 2016). We must thus conclude that the claims fail step two as well.
Applicant’s invention doable manually. See e.g.
Hodos US 20230289619 (¶ 4 56-57 but for tedious, inefficient, high cognitive load doable by person)
Applicant simply computer implements human behavior, avoids tedium, inefficiency, etc (as Hodos too points out ¶ 4).
CLAIM REJECTIONS - 35 USC § 103
35 U.S.C. 103
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
MPEP 2123: “The use of patents as references is not limited to what the patentees describe as their own inventions or to the problems with which they are concerned. They are part of the literature of the art, relevant for all they contain.” In re Heck, 699 F.2d 1331 (Fed. Cir. 1983) A reference may be relied upon for all that it would have reasonably suggested to one having ordinary skill the art, including nonpreferred embodiments. Merck & Co. v. Biocraft Laboratories, 874 F.2d 804, 10 USPQ2d 1843 (Fed. Cir.), cert. denied, 493 U.S. 975 (1989).”
Claims rejected under 35 USC 103 over Hodos US 20230289619 in view of Leibovits US 20220004896 in view of
Cohen US 20210390577 in view of Walczyk US 20220198320
CLAIM 1 14 18
CLAIM 2
CLAIM 3 15
CLAIM 4
CLAIM 5
CLAIM 6 20
CLAIM 7
CLAIM 8 16
CLAIM 9 17
CLAIM 10
CLAIM 12
CLAIM 13 19
PNG
media_image4.png
708
470
media_image4.png
Greyscale
CLAIM 1 14 18
1. A computer-implemented method for model evaluation and performance-based selection, the method comprising:
Hodos US 20230289619 Abstract, title Adaptive Data Models and Selection Fig 1 2 4 & text
Abstract
… selecting a data model configuration for use in training predictive models comprise receiving two or more data model configurations, extracting a data model for each of the two or more data model configurations from a knowledge graph, generating a separate predictive model for each of the extracted data models, scoring the output of each separate predictive model based on a benchmark data set, and selecting at least one data model configuration of the two or more data model configurations based on the output scores.
[0001] The present application relates to a system, apparatus and method(s) for specifying, evaluating, and selecting a data model configuration for use in training one or more machine learning (ML) predictive models and the like configured for receiving knowledge graph information as input and for providing trained ML predictive model(s) based on said selected data model configuration.
BACKGROUND
[0002] Knowledge graphs are increasingly prevalent tools that can be used to infer new relationships between entities. Data in knowledge graphs can be represented in various ways; typically, nodes can be used to represent entities, and relationships between these entities can be represented as edges. In particular, they can be employed in the field of drug development to infer hitherto unknown relationships between, without limitation, for example genes and diseases. This is often performed by trained machine learning (ML) models that accept a knowledge graph as input, and can output newly inferred relationships.
[0003] In practice, the prediction of new inferences is often performed on subsets of large knowledge graphs in order to reduce so-called noise and the inference of false-positive relationships where none exist. Prior to inferring relationships based on an input knowledge graph or subset thereto, an ML predictive model may be trained on similar subsets of the knowledge graph and subsequently, once trained, applied to as hitherto unseen subsets of the knowledge graph for inferring new relationships and the like therefrom. The creation of the subsets of the knowledge graph or extraction of a subset from the knowledge graph (also known and referred to herein, as a ‘data model’) can be performed according to any number of conventional methods.
[0004] Each data model may comprise or represent data representative of a subset of the knowledge graph and may be extracted from the knowledge graph based on a data model configuration. The data model configuration may comprise or represent data representative of one or more conditions, parameters, values, criteria, relationships, entities, confidence scores, or any other data, node, edge or attribute representing the knowledge graph that may be used for defining and extracting the subset knowledge graph from the knowledge graph. For instance, the edges in the knowledge graph may have associated attributes that, for example, indicate confidence scores for the relationship. In this case, a decision process can be used to define a data model configuration that is used to decide the proportion of edges used to generate a data model for use in inferring new relationships; i.e. a percentage of highest confidence scores is selected while the rest of the full knowledge graph is excluded. Another example may be defining a data model configuration based on a selection of a limited number of types of relationship; for example, in a biomedical domain, the data model may consist only of the subset of the total knowledge graph where entities are related by an edge indicating that a gene ‘causes’ a disease. Currently choosing or defining appropriate data model configuration(s) for filtering, extracting, or deciding which portions or a subset of the knowledge graph are to be used is a manual, ad hoc process that is extremely time-consuming and error-prone.
[0005] There is a desire for a more efficient and robust system for generating and selecting a data model from a knowledge graph for optimising the training of one or more ML predictive model(s) that result in the downstream workflow in robust ML predictive model(s) for inferring relationships and the like from an ever-changing and/or updated knowledge graph and the like. There is a further desire for such a system to enable rapid experimentation, optimisation, and selection of different data model configurations for ensuring the best data model configuration, and hence the best data model, is appropriately chosen for improving the predictive accuracy of downstream ML predictive model(s) trained on and/or applied to such selected data model(s) and improved accuracy of predictions output therefrom (e.g. genes for a query disease).
[0006] The embodiments described below are not limited to implementations which solve any or all of the disadvantages of the known approaches described above.
SUMMARY
[0008] The present disclosure describes a system for specifying, testing, evaluating, and selecting data models based on the predictive performance (or other properties) of corresponding predictive ML models that are trained using the information specified by each of the data models. This system can greatly streamline a process that would otherwise be inefficient and especially in scenarios where it is unclear which parts or subsets of a knowledge graph would be optimally suited to support a given ML task, such as prediction of links between genes and diseases. In turn, the overall predictive performance shall be significantly improved such that more accurate predictive ML models can be derived from the selected data models or data model configurations.
[0009] In a first aspect, the present disclosure provides a computer-implemented method of selecting a data model configuration for use in training predictive models comprising: receiving two or more data model configurations; extracting a data model for each of the two or more data model configurations from a knowledge graph; generating a separate predictive model for each of the extracted data models; scoring the output of each separate predictive model based on a benchmark data set; and selecting at least one data model configuration of the two or more data model configurations based on the output scores.
[0010] In a second aspect, the present disclosure provides a computer-implemented method for training a separate predictive model for each of two or more data model configurations comprising: extracting a set of training data for each of the two or more data model configuration from a knowledge graph; and training the separate predictive model using the set of training data.
PNG
media_image5.png
717
488
media_image5.png
Greyscale
O receiving, by one or more processors, a model and corresponding configuration information
Hodos Fig 1a & text
O creating, by the one or more processors, one or more model variations based on the model and corresponding configuration information, the one or more model variations including one or more protected models designated to train until a best model is determined
Protected, obvious from Hodos Fig 1-2, 6 & text; protected is model maintained with respect to a standard or reference on the way to prune, sample, down-size, update results in best model ¶ 78
O determining, by the one or more processors, a model variation subset based on a threshold amount and one or more evaluation scores of each of the one or more model variations variations , the threshold amount comprising a percentage of the one or more model variations that are to be omitted
Hodos Fig 1-2 4 & text
Hodos percentage, proportion ¶ 4 in combination with ¶ 53
PNG
media_image6.png
406
332
media_image6.png
Greyscale
To use a percentage further shown be
Zlotnik US 20210390577 ratio Fig 4 & text,
Zlotnik US 20210390577 percentage at least ¶ 5
PNG
media_image7.png
300
317
media_image7.png
Greyscale
Leibovit US 20220004896 Fig 6 & text, e.g. ¶ 49-61 remove ≈ omit, weak or low visavis a threshold
NOT EXPLICT IN Hodos first embodiment ¶ 25 is all of
O terminating, by the one or more processors, processes and resources for the one or more model variations not included in the model variation subset or not the one or more protected models
Hodos 1st embodiment ¶ 25 doesn’t say omit
Hodos ¶ 53 prune etc
Hodos Fig 1-2 4 & text
It would have been obvious to combine 1st embodiment Hodos ¶ 25 with 2nd embodiment ¶ 52, 53, etc. This is simply Use of Known Technique (¶ 52) to Improve Similar Art (Hodos ¶ 25) in Same Way
O initiating, by the one or more processors, one or more new model variations based on the model variation subset
Hodos Fig 1 & text iterative, re-tune
Hodos Fig 2 & text iterative
Hodos ¶ 26 46 Fig 1 & text
Hodos ¶ 4 56-57 it could be done manually but that’s tedious, inefficient, hi cognitive load
Hodos Fig 1-2 4 & text ¶ 2-4 8-10 25-27 34-111
O determining, by the one or more processors, a best model of the one or more new model variations based on one or more new model evaluation scores for each of the one or more new model variations and
Hodos Fig 1 2 4 & text,
Hodos Fig 1 & text select one of two or more
Hodos Fig 2 optim, ¶ 2-5 best, ¶ 8-10 optim
O omitting, by the one or more processors, the one or more new model variations except the best model the one or more protected models and
Hodos Fig 1 & text iterative, re-tune
Hodos Fig 2 & text iterative
Hodos ¶ 26 46 Fig 1 & text
Hodos ¶ 4 56-57 it could be done manually but that’s tedious, inefficient
Hodos Fig 1-2 4 & text ¶ 2-4 8-10 25-27 34-111
Reading the claim in light of the Spec, best includes limiting loss. It would have been obvious for one looking at Hodos to search the works of colleagues on the same subject of optimizing (Hodos Fig 2) and find Leibovits US 20220004896 teaching optimization (Leibovits ¶ 3, Fig 4-5) includes less loss (Fig 6-11 & text) and optimizing includes omission (Leibovits ¶ 49 remove and ¶ 46-60). This is Combining Prior Art Elements According to Known Methods
PNG
media_image8.png
408
598
media_image8.png
Greyscale
PNG
media_image9.png
483
613
media_image9.png
Greyscale
PNG
media_image10.png
443
572
media_image10.png
Greyscale
Hodos has a prediction model but its example is biomed (¶ 52, Fig 5 & text)
NOT EXPLICT IN Hodos is click prediction model but see
Cohen Fig 4-5 & text and thus we have a simple substitution (engagement for biomed) of a workpiece (data type). Cohen aka Zlotnik US 20210390577 ¶ 4 36
[0004] Because exploration content items are served in a non-deterministic manner without the benefit of the user engagement model (e.g., randomly selected and served), little benefit may be gained from serving exploration traffic with exploration content items other than for training the user engagement model to understand an audience of users that would be interested in a particular exploration content item (e.g., most users may not engage with the exploration content items). Thus, exploration traffic may be “expensive.”
[0036] Unfortunately, the user engagement model will lack an understanding about what audience of users will find content items, newly introduced to the content serving platform, interesting and engaging. Accordingly, these newly introduced content items (e.g., content items on which the user engagement model has not been adequately trained to output accurate predicted likelihoods of user engagement) are treated as exploration content items. Because the exploration content items are provided to users in a non-deterministic manner without the benefit of the user engagement model, the probability of user engaged is very low (e.g., content items may be randomly selected to provide to client devices of users, whom may ultimate have little interest in such content items). Thus, exploration content items are “costly” because a substantial amount of storage, processing resources, and network bandwidth can be wasted in provided the exploration content items to client devices of users that will end up ignoring the exploration content items. Once enough training data of users engaging or not engaging with an exploration content item has been gathered and used to train the user engagement model to more accurately predict likelihoods of users interacting with the exploration content item, the exploration content item is treated as a normal content item that is able to utilize the user engagement model for the bidding process.
[0050] Exploration content items within the exploration bucket 516 are typically selected to serve exploration traffic in a non-deterministic manner without the benefit of the user engagement model 524 that can otherwise provide relatively accurate predicted likelihoods of users engaging with content items for which the user engagement model 524 has been trained. In an example, when the request 528 for a content item is received from the client device 510, the request 528 may be deemed to be exploration traffic and is assigned to the exploration bucket 516 (e.g., selected to be part of the 5% of overall traffic that is assigned to the exploration bucket 516), an exploration content item may be randomly selected from the exploration bucket 516 and returned to the client device 510 for display to the user. Unfortunately, this non-deterministic manner can be exploited by content providers and is INEFFICIENT. Accordingly, as provided herein, an exploration model 526 is generated by an exploration model generator 522, and is used to more efficiently select exploration content items to provide to client devices for serving exploration traffic in a manner that mitigates exploitation and abuse of the content serving platform 512 by content providers.
PNG
media_image11.png
935
652
media_image11.png
Greyscale
CLAIM 2
2. The computer-implemented method of claim 1, the claim further comprising:
O determining, by the one or more processors, the one or more new model evaluation scores for each of the one or more new model variations and
Hodos Fig 1 2 4 & text,
Hodos Fig 1 & text select one of two or more
O analyzing, by the one or more processors, the one or more new model evaluation scores to determine one of the one or more new model variations as the best model.
Hodos Fig 1 2 4 & text,
Hodos Fig 1 & text select one of two or more
Hodos Fig 2 optim, ¶ 2-5 best, ¶ 8-10 optim
NOT EXPLICT IN Hodos is duration, score but see Walczyk US 20220198320 ¶ 16
[0016] Program 150 is a cognitive multi-pipeline control system controlling multiple parallel-operating machine learning pipelines where feature evaluation, model selection, and confidence scoring are performed in reduced time and with reduced computational resources. In an embodiment, program 150 (i.e., analytical brain) is a multi-pipeline controller that utilizes stop criteria to determine whether to activate or deactivate a particular pipeline path. In this embodiment, program 150 generates confidence scores of the ensemble of pipelined models. In an embodiment, program 150 utilizes a publish and subscribe structure architecture pattern to monitor the determined ensemble utilizing an asynchronous messaging service to communicate model states and events in the pipeline lifecycle, such as dead, blocked, or running processes. In various embodiments, program 150 may implement the following steps: determine a plurality of models to incorporate a plurality of determined features from a received dataset; generate an aggregated prediction utilizing each model, in parallel, in the determined plurality of models subject to stop criteria, wherein stop criteria includes a prediction duration threshold; and a confidence value for the aggregated prediction.
It would have been obvious given Hodos to search the works of other similar teachings and find Walcyzk and combine the two (Combining Prior Art Elements According to Known Methods) for terminating by duration, score.
CLAIM 3 15
3. The computer-implemented method of claim 1, the claim further comprising:
O receiving, by the one or more processors, interaction data for each of the one or more model variations and
Hodos ¶ 52 56
Leibovitz ¶ 35 62 33-41
Cohen ¶ 36 users interacting
O analyzing, by the one or more processors, the one or more model variations to determine the one or more evaluation scores for each of the one or more model variations, wherein the one or more evaluation scores are based on the interaction data
Hodos US 20230289619 Abstract, title Adaptive Data Models and Selection Fig 1 2 4 & text
Motivation to combine above with respect to independent claim
CLAIM 4
4. The computer-implemented method of claim 3, wherein the analyzing the one or more model variations
O occurs after a threshold time period, wherein the threshold time period corresponds to a set time interval
Hodos already says iterate ¶ 26 49 20 51 57 59 60 62 63 64 65 70 107-111
Zlotnik US 20210390577 ¶ 66 and iterative ¶ 6 61 same as Hodos
Leibovits US 20220004896 also ¶ 45 and ¶ 49 iterative same as Hodos
Walczyk US2022198320 ¶ 7 21 25 and iterative ¶ 19 21 same as Hodos
Motivation to combine above with respect to independent claim
And apriori, new data, would need new analysis i.e. iterate
CLAIM 5
Hodos has prediction ¶ 8 52 but NOT EXPLICT IN Hodos is click prediction
5. The computer-implemented method of claim 3, wherein the
O interaction data includes an estimated click prediction.
Leibovits ¶ 41
Cohen ¶ 35
Motivation to combine above with respect to independent claim
CLAIM 6 20
6. The computer-implemented method of claim 1/18, wherein the configuration information includes
O one or more hyperparameters.
Hodos at least ¶ 4 45 46 49 53 61-64 parameter hyperparameter
CLAIM 7
Although Hodos has hyperperameters at least ¶ 4 45 46 49 53 61-64 parameter hyperparameter
NOT EXPLICT IN Hodos is
7. The computer-implemented method of claim 6, wherein the one or more hyperparameters include at least one of:
O an initial value, a lower exploration initial value, and an upper exploration initial value.
Cohen Fig 4-5 & text, ¶ 40 41 60
Motivation to combine above with respect to independent claim
CLAIM 8 16
8. The computer-implemented method of claim 1, the method further comprising:
O assigning, by the one or more processors, a protected classification to at least one of the one or more model variations, wherein the protected classification indicates that at least one of the one or more model variations is to be included in the model variation subset
Hodos benchmark and the like, suitability ¶ 25 26 35 39 40 41 in reference to base case Fig 1 & text
Also Cohen ¶ 57 base case
CLAIM 9 17
9. The computer-implemented method of claim 1, wherein the omitting further comprises:
O omitting, by the one or more processors, one or more processes corresponding to the one or more model variations not included in the model variation subset and
Hodos ¶ 53 prune etc
O omitting, by the one or more processors, one or more resources corresponding to the one or more model variations not included in the model variation subset.
Hodos ¶ 53 prune etc
CLAIM 10
10. The computer-implemented method of claim 1, wherein analyzing the one or more model variations includes
O determining the one or more model variations with a highest evaluation score.
Hodos Abstract, title Adaptive Data Models and Selection Fig 1 2 4 & text
CLAIM 12
12. The computer-implemented method of claim 1, wherein the one or more evaluation scores are based on one or more
O aggregated log losses
Reading the claim in light of the Spec, best includes limiting loss. It would have been obvious for one looking at Hodos to search the works of colleagues on the same subject of optimizing (Hodos Fig 2) and find Leibovits US 20220004896 teaching optimization (Leibovits ¶ 3, Fig 4-5) includes less loss (Fig 6-11 & text) and optimizing includes omission (Leibovits ¶ 49 remove and ¶ 46-60). This is Combining Prior Art Elements According to Known Methods
CLAIM 13 19
13. The computer-implemented method of claim 1, wherein the one or more evaluation scores are based on one or more
O user interactions with the one or more model variation
Hodos US 20230289619 Abstract, title Adaptive Data Models and Selection Fig 1 2 4 & text
Hodos ¶ 52 56
Leibovitz ¶ 35 62 33-41
Cohen ¶ 36 users interacting
Motivation to combine above with respect to independent claim
Claims rejected under 35 USC 103 over Hodos/Leibovits/Cohen/Zlotnik in view of Khavronin US 20170364931
CLAIM 11
NOT EXPLICT IN Hodos is
11. The computer-implemented method of claim 1, wherein the
determining the model variation subset based on the one or more evaluation scores
O occurs after a threshold period of time
Khavronin US 20170364931 ¶ 208
It would have been obvious at the time of filing to combine Hodos and Khavronin because -- looking at Hodos discussion of time constraints, inefficiency, waste -- it would have been obvious to consult the works of others in the same field of model optimizing and find Khavronin and combine with Hodos for the predictable result of determining model variation (Hodos) after threshold time (Khavaronin). The prior art included each element claimed, although not necessarily in a single prior art reference, with the only difference between the claimed invention and the prior art being the lack of actual combination of the elements in a single prior art reference. One of ordinary skill in the art could have combined the elements as claimed by known methods and that in combination, each element merely would have performed the same function as it did separately. One of ordinary skill in the art would have recognized that the results of the combination were predictable. Therefore all the claimed elements were known in the prior art and one skilled in the art could have combined the elements as claimed by known methods with no change in their respective functions, and the combination would have yielded predictable results to one of ordinary skill in the art at the time of the invention.
During prosecution, applicant has an opportunity and a duty to amend ambiguous claims to clearly and precisely define the metes and bounds of the claimed invention. The claim places the public on notice of the scope of the patentee’s right to exclude. See, e.g., Johnson & Johnston Assoc. Inc. v. R.E. Serv. Co., 285 F.3d 1046, 1052, 62 USPQ2d 1225, 1228 (Fed. Cir. 2002) (en banc). As stated in Halliburton Energy Servs., Inc. v. M-I LLC, 514 F.3d 1244, 1255, 85 USPQ2d 1654, 1663 (CAFC 2008):
“We note that the patent drafter is in the best position to resolve the ambiguity in the patent claims, and it is highly desirable that patent examiners demand that applicants do so in appropriate circumstances so that the patent can be amended during prosecution rather than attempting to resolve the ambiguity in litigation”
CONCLUSION
Pertinent prior art cited but not relied upon
US 11756072
Any inquiry concerning this communication or earlier communications from the examiner should be directed to BREFFNI X BAGGOT whose telephone number is (571)272-7154. The examiner can normally be reached M-F 8a-10a, 12p-6p.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Waseem Ashraf can be reached at 571-270-3948. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
BREFFNI BAGGOT
Primary Examiner
Art Unit 3621
/BREFFNI BAGGOT/Primary Examiner, Art Unit 3621