DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Status of Application
This office action is in response to the most recent amendments and arguments filed by applicants on 10/09/24.
No claims are amended
No claims are cancelled
No claims are added
Claims 1-18 are pending
Note:
In independent claims 1 and 10, the claim limitations below are ambiguous:
upon receipt of a request, preparing a list of potential assessors and determining if additional potential assessors are required, if so:
automatically posting a request for potential assessors; and
receiving information about potential assessors; and
Regarding the claim limitation above, the claims are recited at a very high level. It is unclear how the determination is made that additional potential assessors are required? Is there a threshold that they are using to compare the prepared list to? The 13 page originally submitted specification is also not providing much detail to show how such a determination is being made.
For instance, in [0019] of the specification recites: “In some cases, the determine if additional potential assessors are required may include: determining if the instructor has requested additional instructors, and if not, determining if there are sufficient assessors on the list of potential assessors”. The term sufficient is very broad in the specification. What is considered sufficient? Again, is there a threshold being used to make this determination? Sufficient and required are really broad terms and may mean different things to different people. What is sufficient and required to one person may not be to another person.
The claim limitations discussed are broad and the specification does not provide enough detailed support to show to one of ordinary skill in the art what certain terms in the claim limitations mean. As such, for the purposes of this office action, the claim is being understood to mean simply providing or preparing a list of potential assessors.
In light of these notes, the amended claims, do not overcome previously presented rejections under 101 and 103. As is discussed below. This note is intended as a conversation starter to help applicants understand the examiner’s perspective. Applicants are welcome to call the examiner to discuss this further.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-18 is/are rejected under 35 U.S.C. 101 because the claimed invention is directed to a judicial exception (i.e., an abstract idea) without significantly more.
Step One - First, pursuant to step 1 in the January 2019 Guidance on 84 Fed. Reg. 53, the claims 1-9 is/are directed to a method which is a statutory category.
Step One - First, pursuant to step 1 in the January 2019 Guidance on 84 Fed. Reg. 53, the claims 10-18 is/are directed to a system which is a statutory category.
Step 2A Prong 1: Identify the Abstract Idea(s)
The Alice framework, steps 2A-Prong One (part 1 of Mayo Test), here, the claims are analyzed to determine if the claims are directed to a judicial exception. MPEP 2106.04(a). In determining, whether the claims are directed to a judicial exception, the claims are analyzed to evaluate whether the claims recite a judicial exception (Prong One of Step 2A), and whether the claims recite additional elements that integrate the judicial exception into a practical application (Prong Two of Step 2A). See 2019 Revised Patent Subject Matter Eligibility Guidance (“PEG” 2019 Revised Patent Subject Matter Eligibility Guidance, 84 Fed. Reg. 50-57 (Jan. 7, 2019)).
Under the 2019 PEG, Step 2A under which a claim is not “directed to” a judicial exception unless the claim satisfies a two-prong inquiry. Further, particular groupings of abstract ideas are consistent with judicial precedent and are based on an extraction and synthesis of the key concepts identified by the courts as being abstract.
Independent Claims 1 and 10, with respect to the Step 2A, Prong One, when “taken as a whole” the claims as drafted, and given their broadest reasonable interpretation, fall within the Abstract idea grouping of “certain methods of organizing human activity” (business relations; relationships or interactions between people). For instance, independent Method Claim 1 is directed to an abstract idea, as evidenced by claim limitations “monitoring for a request to select assessors for a course by an instructor; upon receipt of a request, preparing a list of potential assessors and determining if additional potential assessors are required, if so: automatically posting a request for potential assessors; and receiving information about potential assessors; and updating the list of potential assessors; if not: proceeding; selecting assessors for the course from the list of potential assessors; assigning the select assessors to a set of students in the course, and monitoring the assigned assessors during and after the course.”
These claim limitations belong to the grouping of “certain methods of organizing human activity” because the claims are related to course assessment by multiple assessors in a learning environment. Managing the assessment of courses for one or more human entities involves organizing human activity based on the description of “certain methods of organizing human activity” provided by the courts. The court have used the phrase “Certain methods of organizing human activity” as —fundamental economic principles or practices (including hedging, insurance, mitigating risk); commercial or legal interactions (including agreements in the form of contracts; legal obligations; advertising, marketing or sales activities or behaviors; business relations); managing personal behavior or relationships or interactions between people (including social activities, teaching, and following rules or instructions).
Independent Claims 10 is/are recite substantially similar limitations to independent claim 1 and is/are rejected under 2A for similar reasons to claim 1 above.
Step 2A Prong 2: Additional Elements That Integrate the Judicial Exception into a Practical Application
With respect to the Step 2A, Prong Two - This judicial exception is not integrated into a practical application. In particular, the claim recites additional elements: “A method for course assessment, the method comprising: A system for course assessment, the system comprising: a processor; a memory storing computer-readable instructions, which, when executed on the processor, generate the following: an assessor management module configured to:” at a high level of generality such that it amounts to no more than: Adding the words “apply it” (or an equivalent) with the judicial exception, or mere instructions to implement an abstract idea on a computer, or merely uses a computer as a tool to perform an abstract idea, as discussed in MPEP 2106.05(f). Accordingly, these additional elements do not integrate the abstract idea into a practical application because they do not impose any meaningful limits on practicing the abstract idea. The claims are directed to an abstract idea with no significantly more elements.
Thus, the additional elements do not integrate the abstract idea into practical application because they do not impose any meaningful limitations on practicing the abstract idea. As a result, claims 1 and 10 do not provide any specifics regarding the integration into a practical application when recited in a claim with a judicial exception. See MPEP 2106.05(f).
Similarly dependent claims 2-9 and 11-18 are also directed to an abstract idea under 2A, first and second prong. In the present application, all of the dependent claims have been evaluated and it was found that they all inherit the deficiencies set forth with respect to the independent claims. For instance, dependent claims 2 recite “further comprising, prior to selecting assessors, displaying the list of potential assessors to the instructor” and dependent claims 3 recite “wherein the displaying the list of potential assessors comprises presenting one or more informational elements for each assessor”. Here, these claims offer further descriptive limitations of elements found in the independent claims which are similar to the abstract idea noted in the independent claim above. As a result, Examiner asserts that dependent claims, such as dependent claims 2-9 and 11-18 are also directed to the abstract idea identified above.
Step 2B: Determine Whether Any Element, Or Combination, Amount to “Significantly More” Than the Abstract Idea Itself
With respect to Step 2B, the claim does not include additional elements that are sufficient to amount to significantly more than the judicial exception. First, the invention lacks improvements to another technology or technical field [see Alice at 2351; 2019 IEG at 55], and lacks meaningful limitations beyond generally linking the use of an abstract idea to a particular technological environment [Alice at 2360, 2019 IEG at 55], and fails to effect a transformation or reduction of a particular article to a different state or thing [2019 IEG, 55]. For the reasons articulated above, the claims recite an abstract idea that is limited to a particular field of endeavor (MPEP § 2106.05(h)) and recites insignificant extra-solution activity (MPEP § 2106.05(g)). By the factors and rationale provided above with respect to these MPEP sections, the additional elements of the claims that fail to integrate the abstract idea into a practical application also fail to amount to “significantly more” than the abstract idea.
As discussed above with respect to integration of the abstract idea into a practical application, the additional element(s) of “A method for course assessment, the method comprising: A system for course assessment, the system comprising: a processor; a memory storing computer-readable instructions, which, when executed on the processor, generate the following: an assessor management module configured to:” are insufficient to amount to significantly more. Applicants originally submitted specification describes the computer components above at least in page/ paragraph XX. In light of the specification, it should be noted that the components discussed above did not meaningfully limit the abstract idea because they merely linked the use of the abstract idea to a particular technological environment (i.e., "implementation via computers"). In light of the specification, it should be noted that the claim limitations discussed above are merely instructions to implement the abstract idea on a computer. See MPEP 2106.05(f). (See MPEP 2106.05(f) - Mere Instructions to Apply an Exception - “Thus, for example, claims that amount to nothing more than an instruction to apply the abstract idea using a generic computer do not render an abstract idea eligible.” Alice Corp., 134 S. Ct. at 235). Mere instructions to apply an exception using computer component cannot provide an inventive concept.). The additional elements amount to no more than a recitation of generic computer elements utilized to perform generic computer functions, such as performing repetitive calculations, Bancorp Services v. Sun Life, 687 F.3d 1266, 1278, 103 USPQ2d 1425, 1433 (Fed. Cir. 2012) ("The computer required by some of Bancorp’s claims is employed only for its most basic function, the performance of repetitive calculations, and as such does not impose meaningful limits on the scope of those claims."); and storing and retrieving information in memory, Versata Dev. Group, Inc. v. SAP Am., Inc., 793 F.3d 1306, 1334, 115 USPQ2d 1681, 1701 (Fed. Cir. 2015); OIP Techs., 788 F.3d at 1363, 115 USPQ2d at 1092-93; see MPEP 2106.05(d)(II).
Therefore, the claims at issue do not require any nonconventional computer, network, or display components, or even a “non-conventional and non-generic arrangement of know, conventional pieces,” but merely call for performance of the claimed on a set of generic computer components” and display devices. All of these additional elements are significantly more because these, again, are merely the software and/or hardware components used to implement the abstract idea on a general-purpose computer. Generically recited computer elements do not add a meaningful limitation to the abstract idea because the Alice decision noted that generic structures that merely apply abstract ideas are not significantly more than the abstract ideas.
The computing elements with a computing device is recited at high level of generality (e.g. a generic device performing a generic computer function of processing data). Thus, this step is no more than mere instructions to apply the exception on a generic computer. In addition, using a processor to process data has been well- understood routing, conventional activity in the industry for many years. Generic computer features, such as system or storage, do not amount to significantly more than the abstract idea. These limitations merely describe implementation for the invention using elements of a general-purpose system, which is not sufficient to amount to significantly more. See, e.g., Alice Corp., 134 S. Ct. 2347, 110 USPQ2d 1976; Versata Dev. Group, Inc. v. SAP Am. Inc., 793 F.3d 1306, 1334, 115 USPQ2d 1681, 1791 (Federal Circuit 2015).
The claim fails to recite any improvements to another technology or technical field, improvements to the functioning of the computer itself, use of a particular machine, effecting a transformation or reduction of a particular article to a different state or thing, adding unconventional steps that confine the claim to a particular useful application, and/or meaningful limitations beyond generally linking the use of an abstract idea to a particular environment. See 84 Fed. Reg. 55. Viewed individually or as a whole, these additional claim element(s) do not provide meaningful limitation(s) to transform the abstract idea into a patent eligible application of the abstract idea such that the claim(s) amounts to significantly more than the abstract idea itself.
Independent Claims 10 is/are recite substantially similar limitations to independent claim 1 and is/are rejected under 2B for similar reasons to claim 1 above.
Further, it should be noted that additional elements of the claimed invention such as claim limitations when considered individually or as an ordered combination along with the other limitations discussed above in method claim 1 also do not meaningfully limit the abstract idea because they merely linked the use of the abstract idea to a particular technological environment (i.e., "implementation via computers"). In light of the specification, it should be noted that the claim limitations discussed above are merely instructions to implement the abstract idea on a computer. See MPEP 2106.
Similarly, dependent claims 2-9 and 11-18 also do not include limitations amounting to significantly more than the abstract idea under the second prong or 2B of the Alice framework. In the present application, all of the dependent claims have been evaluated and it was found that they all inherit the deficiencies set forth with respect to the independent claims. Further, it should be noted that the dependent claims do not include limitations that overcome the stated assertions. Here, the dependent claims recite features/limitations that include computer components identified above in part 2B of analysis of independent claims 1 and 10. As a result, Examiner asserts that dependent claims, such as dependent claims 2-9 and 11-18 are also directed to the abstract idea identified above.
Further, Examiner notes that the addition limitations, when considered as an ordered combination, add nothing that is not already present when looking at the additional elements individually.
For more information on 101 rejections, see MPEP 2106, January 2019 Guidance at https://www.govinfo.gov/content/pkg/FR-2019-01 -07/pdf/2018-28282.pdf
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-18 is/are rejected under 35 U.S.C. 103 as being unpatentable over (US 20180130154 A1) Luca, and further in view of (US 20140180771 A1) Reed et al.
As per claims 1 and 10: Regarding the claim limitations below, Reference Luca shows:
A method for course assessment, the method comprising (Luca, abstract: A system for objective assessment of learning outcomes comprising a data repository comprising at least a hierarchical arrangement of a plurality of learning goals, a report generator coupled to the data repository, an analysis engine coupled to the data repository, a rules engine coupled to the data repository, and an application server adapted to receive application-specific requests from a plurality of client applications and coupled to the data repository. The application server is further adapted to provide an administrative interface for viewing, editing, or deleting a plurality of learning goals and relationships between them, learning assessment tools, learning outcome reports, and learning indexes, and the rules engine performs a plurality of consistency checks to ensure alignment between and among learning goals, learning assessment tools, learning outcomes, and learning indexes. The application server receives learning assessment data and the analysis engine performs analyses to generate a plurality of learning indexes.):
Regarding the claim limitations below, Reference Luca shows:
monitoring for a request to select assessors for a course by an instructor (Luca: [0008]: disclosing the rules engine performs a plurality of consistency checks to ensure alignment between and among learning goals, learning assessment tools, learning outcomes, and learning indexes. [0085] carrying out assessments or evaluations of learning output, using assessment forms, records, rubrics, and the like, calculating individual output level learning indexes, etc.; monitoring learning to identify issues as they occur; performing consistency checks to ensure that goals and expectations are in alignment. [0105] Returning to FIG. 2, in response to a user query or remote data request, an access module 302 configures one or more processors of the evaluation server 102 to receive and parse the data from the database 108. As shown in step 202, the one or more processor of the evaluation server 102 receives data from the database. In one implementation, the data received from the database 108a-b is a collection unstructured and structured data from the database 108. However, the data received can represent one or more post-query transformations, such as filtering the data for features, access privileges (e.g. security), content, excerpts or formats.);
Regarding the claim limitations below, Reference Luca in view of Reed shows:
upon receipt of a request, preparing a list of potential assessors and determining if additional potential assessors are required, if so:
automatically posting a request for potential assessors; and receiving information about potential assessors; and
Luca shows “upon receipt of a request, …. and determining if additional potential assessors are required, if so:
automatically posting a request for potential assessors; and receiving information about potential assessors; and”:
[0010]: In another embodiment, learning indexes are generated for a plurality of aggregates of individual learners, assembled based on membership of individual learners in one or more learning units, zones, or levels. In another embodiment, the learning indexes are used to generate grade reports with feedback for learners. In another embodiment, the report generator generates and distributes reports based at least in part on the aggregated learning indexes, the reports identifying areas of achieved and missed learning relative to established learning goals and expectations. In yet another embodiment, the analysis engine performs analysis of a plurality of learning indexes or learning outcome reports, or both, pertaining to a learner and prepares thereby and distributes a learning improvement plan tailored to the learner. In another embodiment, the analysis engine automatically analyzes progress of the learning improvement plan and, based at least on comparing learning outcome assessments from before and from after implementation of the learning improvement plan, adjusts the learning improvement plan or prepares and distributes a new learning improvement plan. [0011]: a method for objective assessment of learning outcomes is disclosed, the method comprising the steps of: (a) providing an administrative interface via an application server to allow users to specify a plurality of learning goals and expectations; (b) decomposing at least a portion of the learning goals and expectations into achievable and measurable analytics units; (c) organizing the learning goals and expectations into a hierarchy; (d) automatically performing consistency checks to ensure alignment of learning goals and expectations along the hierarchy; (e) providing a plurality of learning assessment tools to a learning assessor in one of online, mobile application, or thick client application formats; (f) receiving learning outcome assessment data at the level of individual learning outcomes from the learning assessor; (g) calculating learning outcomes as learning indexes at the level of an individual output; and (h) preparing and distributing a plurality of learning outcome reports for the individual learner. [0054]: Learning outputs or learning outcomes may be reviewed and assessed (what is commonly referred to as “grading”) by learning assessors or agents qualified to do so, including but not limited to educators, faculty, graders, etc. Individual learning outputs represent output of individual learners but also of groups of learners (in case of team projects). Assessments are made first at the level of individual learning outputs. Learning outcomes and performance define consequences of the processes of learning and education. Achieved (acquired) learning shows what learners learned in relation to planned learning goals; missed learning shows gaps or missed learning in relation to planned learning goals. Learning indexes are numeric measures of leaning that quantify learning outcomes (achieved and or missed learning) in all configurations. [0055]: Learning indexes are first calculated at the level individual of the learning output unit. They can be calculated at all configurations afterwards by “rolling up” or aggregating learning index data starting with raw data at the level of learning outputs and then working up one or more hierarchies, using weighting factors or other formulae that define how aggregation is to be carried out. [0084] Additional activities undertaken during organizing 420 may include designing one or more learning processes 422, designing or creating various forms, records, and/or rubrics or other tools for performing assessments 423 of learning, designing one or more data repositories and specifying data fields including identifying fields for various hierarchy levels, organizations, zones, and the like, establishing routines for and carrying out data collection 424 regarding various aspects of the learning environment (for example, organizational structures within a university, course catalogs, learner rosters, faculty rosters, previous learner learning histories at the same or other institutions, regulatory requirements such as required tests and required proficiency demonstrations, and so forth), performing calculations 425 required to implement a consistent, hierarchical objective learning assessment system, building or establishing data repositories 426 that will be available to appropriate users (such as learners, learning agents, administrators, and so forth), and building a plurality of reports 427 or report templates that may be used by administrators, regulators, and others to assess and analyze the performance of learning processes and learning organizations. [0057] As used herein, “assessment records”, or “rubrics”, or “templates”, mean “a standard of performance for a defined population”, particularly as it is applied against learning goals. Rubrics etc. may comprise, for example, one or more items such as required ID information, goal metrics or analytics or criteria dimensions on which performance is rated, definitions and examples that illustrate the attribute(s) being measured, and a rating scale criteria item, numerical achievable values in various formats such as percentages, absolute numbers, etc., areas where assessors can select achieved learning items, make notes. Dimensions are generally referred to as criteria, the rating scale as levels, and definitions as descriptors. Rubrics or templates typically reflect learning goals metrics for their specific level such as for example the learning output level. They may also reflect learning expectations metrics. [0063] As used herein, an “assessor” is a learning stakeholder (for example, a faculty member, a grader, a teaching assistant, a teacher, an instructor, or the like) or an automated system (such as an automated grading system), or a combination of the two, that is responsible for assessing (grading) one or more learning outcomes. Many examples herein use terms such as “faculty assessor”; these are merely exemplary and other examples are possible as well, according to the invention, and in general the term “assessor” should be understood as defined here. [0098] Data repository 640 may be used to store and document data pertaining to learning goals and processes related to learning goals, all the way down a hierarchy to specific units of learning delivery and learning outputs, including assigned values and formats, analytical means, feedback, etc. Identification of units of learning delivery and learning outputs may also be stored in database 640 (examples to include but not limit degrees, courses, classes, modules, teaching units, assignments). Identification could contain, for example, institution/college codes/ID, degree, course, etc. in formats including acronyms, numbers, symbols, etc. [0117] Once goals have been created and optionally assigned to a hierarchy in step 710, in step 720 processing of learning goals at the level of individual output delivery takes place and one or more analytical criteria may be defined that will be used in assessing progress in achieving goals at various levels of a hierarchy. [0118]: In this sense, units of learning may be hierarchical. According to the embodiment, learning agents, agencies, learners, administrators, or other participants determine one or more overall learning expectations in high-level step 810. In general, specific expectations will be tailored to specific units of learning 812 and correspondingly assigned to one or more levels to create a hierarchy of learning expectations 812. In some embodiments, participants may rank expectations 814 based on a desired order of importance or relevance.).
However, Luca does not explicitly show “preparing a list of potential assessors”. Reed shows the above limitation at least in [0026] FIG. 5 shows an educator list for selection prior to an observation. [0028] FIG. 7 shows a visit list for a specific educator to be observed. [0049] FIG. 28 shows a list of educators whose evaluations can be managed. [0050] FIG. 29 shows an evaluation portfolio for a particular educator. [0055] FIG. 34 shows a list of educators under the Observation tab. [0059] FIG. 38 shows the ability to assign specific professional development coursework to specific indicators based on scores. [0061] FIG. 40 shows a list of required professional development courses required under the Professional Development tab, along with applicable deadlines for completion. [0063] FIG. 42 shows the user completed professional development courses with a rating system to score the coursework.
Reference Luca and Reference Reed are analogous prior art to the claimed invention because the references generally relate to field of monitoring learning outcomes. Further, said references are part of the same classification, i.e., G06Q50. Lastly, said references are filed before the effective filing date of the instant application; hence, said references are analogous prior-art references.
It would have been obvious to one of ordinary skill in the art before the effective filing date of this application for AIA to provide the teachings of Reference Reed, particularly the ability to prepare a list of teachers [0026], in the disclosure of Reference Luca, particularly in the learning indexes [0055], building reports section [0084], assessment records [0057], in order to provide for an automated system for observation and evaluation of teachers which provides consistency, transparency and flexibility to administrators, fully accommodates the specific preferences of schools and school districts, and which includes rubric-agnostic tools that allow educators to make decisions about reporting and remedial measures as taught by Reference Reed (see at least in [0010]), where upon the execution of the method and system of Reference Luca for allowing learning indexes and assessment records monitoring to help make the process of managing and monitoring learning outcomes more efficient and effective.
Further, the claimed invention is merely a combination of old elements in a similar monitoring learning outcomes field of endeavor, and in the combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that, given the existing technical ability to combine the elements as evidenced by Reference Luca in view of Reference Reed, the results of the combination were predictable (MPEP 2143 A);
Regarding the claim limitations below, Reference Luca in view of Reed shows:
updating the list of potential assessors;
(Luca: [0092]: [0092] Once all these individual output learning indexes per established learning goals categories are calculated by the system (after one or more assessors selects values and enters them in the system), the system performs calculations based on formulae to compound, aggregate, weight learning indexes at all configurations, showing achieved learning or and missed learning at those configurations (or adds up and weighs learning indexes at other configurations, for example analytical skills for Module x for all learners). Calculations may readily obtain learning indexes of all learning goal categories as well as overall ones per unit (for example, per module learner X achieved 70% of overall goals, out of which percentage per category can be derived; ranges or whole numbers can be used). In step 555, one or more objective learning assessment results may be combined into a plurality of learning indexes. [0055] disclosing learning indexes are first calculated at the level individual of the learning output unit; they can be calculated at all configurations afterwards by “rolling up” or aggregating learning index data; [0092] one or more objective learning assessment results may be combined into a plurality of learning indexes; [0094] disclosing providing and using objective learning assessment criteria, assessing learning outcomes based on learning goals and or learning expectations and aggregating the results; [0128] the data obtained including assessor assessments inputs at the level of individual learning output, and then aggregate learning indexes may be computed and added.);
[0010]: In another embodiment, learning indexes are generated for a plurality of aggregates of individual learners, assembled based on membership of individual learners in one or more learning units, zones, or levels. In another embodiment, the learning indexes are used to generate grade reports with feedback for learners. In another embodiment, the report generator generates and distributes reports based at least in part on the aggregated learning indexes, the reports identifying areas of achieved and missed learning relative to established learning goals and expectations. In yet another embodiment, the analysis engine performs analysis of a plurality of learning indexes or learning outcome reports, or both, pertaining to a learner and prepares thereby and distributes a learning improvement plan tailored to the learner. In another embodiment, the analysis engine automatically analyzes progress of the learning improvement plan and, based at least on comparing learning outcome assessments from before and from after implementation of the learning improvement plan, adjusts the learning improvement plan or prepares and distributes a new learning improvement plan. [0011]: a method for objective assessment of learning outcomes is disclosed, the method comprising the steps of: (a) providing an administrative interface via an application server to allow users to specify a plurality of learning goals and expectations; (b) decomposing at least a portion of the learning goals and expectations into achievable and measurable analytics units; (c) organizing the learning goals and expectations into a hierarchy; (d) automatically performing consistency checks to ensure alignment of learning goals and expectations along the hierarchy; (e) providing a plurality of learning assessment tools to a learning assessor in one of online, mobile application, or thick client application formats; (f) receiving learning outcome assessment data at the level of individual learning outcomes from the learning assessor; (g) calculating learning outcomes as learning indexes at the level of an individual output; and (h) preparing and distributing a plurality of learning outcome reports for the individual learner. [0054]: Learning outputs or learning outcomes may be reviewed and assessed (what is commonly referred to as “grading”) by learning assessors or agents qualified to do so, including but not limited to educators, faculty, graders, etc. Individual learning outputs represent output of individual learners but also of groups of learners (in case of team projects). Assessments are made first at the level of individual learning outputs. Learning outcomes and performance define consequences of the processes of learning and education. Achieved (acquired) learning shows what learners learned in relation to planned learning goals; missed learning shows gaps or missed learning in relation to planned learning goals. Learning indexes are numeric measures of leaning that quantify learning outcomes (achieved and or missed learning) in all configurations. [0055]: Learning indexes are first calculated at the level individual of the learning output unit. They can be calculated at all configurations afterwards by “rolling up” or aggregating learning index data starting with raw data at the level of learning outputs and then working up one or more hierarchies, using weighting factors or other formulae that define how aggregation is to be carried out. [0084] Additional activities undertaken during organizing 420 may include designing one or more learning processes 422, designing or creating various forms, records, and/or rubrics or other tools for performing assessments 423 of learning, designing one or more data repositories and specifying data fields including identifying fields for various hierarchy levels, organizations, zones, and the like, establishing routines for and carrying out data collection 424 regarding various aspects of the learning environment (for example, organizational structures within a university, course catalogs, learner rosters, faculty rosters, previous learner learning histories at the same or other institutions, regulatory requirements such as required tests and required proficiency demonstrations, and so forth), performing calculations 425 required to implement a consistent, hierarchical objective learning assessment system, building or establishing data repositories 426 that will be available to appropriate users (such as learners, learning agents, administrators, and so forth), and building a plurality of reports 427 or report templates that may be used by administrators, regulators, and others to assess and analyze the performance of learning processes and learning organizations. [0057] As used herein, “assessment records”, or “rubrics”, or “templates”, mean “a standard of performance for a defined population”, particularly as it is applied against learning goals. Rubrics etc. may comprise, for example, one or more items such as required ID information, goal metrics or analytics or criteria dimensions on which performance is rated, definitions and examples that illustrate the attribute(s) being measured, and a rating scale criteria item, numerical achievable values in various formats such as percentages, absolute numbers, etc., areas where assessors can select achieved learning items, make notes. Dimensions are generally referred to as criteria, the rating scale as levels, and definitions as descriptors. Rubrics or templates typically reflect learning goals metrics for their specific level such as for example the learning output level. They may also reflect learning expectations metrics. [0063] As used herein, an “assessor” is a learning stakeholder (for example, a faculty member, a grader, a teaching assistant, a teacher, an instructor, or the like) or an automated system (such as an automated grading system), or a combination of the two, that is responsible for assessing (grading) one or more learning outcomes. Many examples herein use terms such as “faculty assessor”; these are merely exemplary and other examples are possible as well, according to the invention, and in general the term “assessor” should be understood as defined here. [0098] Data repository 640 may be used to store and document data pertaining to learning goals and processes related to learning goals, all the way down a hierarchy to specific units of learning delivery and learning outputs, including assigned values and formats, analytical means, feedback, etc. Identification of units of learning delivery and learning outputs may also be stored in database 640 (examples to include but not limit degrees, courses, classes, modules, teaching units, assignments). Identification could contain, for example, institution/college codes/ID, degree, course, etc. in formats including acronyms, numbers, symbols, etc. [0117] Once goals have been created and optionally assigned to a hierarchy in step 710, in step 720 processing of learning goals at the level of individual output delivery takes place and one or more analytical criteria may be defined that will be used in assessing progress in achieving goals at various levels of a hierarchy. [0118]: In this sense, units of learning may be hierarchical. According to the embodiment, learning agents, agencies, learners, administrators, or other participants determine one or more overall learning expectations in high-level step 810. In general, specific expectations will be tailored to specific units of learning 812 and correspondingly assigned to one or more levels to create a hierarchy of learning expectations 812. In some embodiments, participants may rank expectations 814 based on a desired order of importance or relevance.).
However, Luca does not explicitly show “preparing a list of potential assessors”. Reed shows the above limitation at least in [0026] FIG. 5 shows an educator list for selection prior to an observation. [0028] FIG. 7 shows a visit list for a specific educator to be observed. [0049] FIG. 28 shows a list of educators whose evaluations can be managed. [0050] FIG. 29 shows an evaluation portfolio for a particular educator. [0055] FIG. 34 shows a list of educators under the Observation tab. [0059] FIG. 38 shows the ability to assign specific professional development coursework to specific indicators based on scores. [0061] FIG. 40 shows a list of required professional development courses required under the Professional Development tab, along with applicable deadlines for completion. [0063] FIG. 42 shows the user completed professional development courses with a rating system to score the coursework.
Reference Luca and Reference Reed are analogous prior art to the claimed invention because the references generally relate to field of monitoring learning outcomes. Further, said references are part of the same classification, i.e., G06Q50. Lastly, said references are filed before the effective filing date of the instant application; hence, said references are analogous prior-art references.
It would have been obvious to one of ordinary skill in the art before the effective filing date of this application for AIA to provide the teachings of Reference Reed, particularly the ability to prepare a list of teachers [0026], in the disclosure of Reference Luca, particularly in the learning indexes [0055], building reports section [0084], assessment records [0057], in order to provide for an automated system for observation and evaluation of teachers which provides consistency, transparency and flexibility to administrators, fully accommodates the specific preferences of schools and school districts, and which includes rubric-agnostic tools that allow educators to make decisions about reporting and remedial measures as taught by Reference Reed (see at least in [0010]), where upon the execution of the method and system of Reference Luca for allowing learning indexes and assessment records monitoring to help make the process of managing and monitoring learning outcomes more efficient and effective.
Further, the claimed invention is merely a combination of old elements in a similar monitoring learning outcomes field of endeavor, and in the combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that, given the existing technical ability to combine the elements as evidenced by Reference Luca in view of Reference Reed, the results of the combination were predictable (MPEP 2143 A);
Regarding the claim limitations below, Reference Luca in view of Reed shows:
if not: proceeding;
(Luca [0085] carrying out assessments or evaluations of learning output, using assessment forms, records, rubrics, and the like; [0119] also discloses that when consistency checks fail, corrective steps may be taken as in step 760, and the process may loop back to step 810 or another step, depending on the nature and extent of consistency check failure; [0094] providing and using objective learning assessment criteria, assessing learning outcomes based on learning goals and or learning expectations and aggregating the results, and then reporting on and analyzing the results; [0124] input to the process is from learning assessment forms, rubrics, etc.; learning indexes may be aggregated and compounded; consistency checks may be performed in step 1150, and corrective actions may be taken as required by returning to affected prior steps to correct deficiencies in data consistency);
Regarding the claim limitations below, Reference Luca in view of Reed shows:
selecting assessors for the course from the list of potential assessors;
(Luca: [0123] According to the embodiment, assessment (grading) at the learning output level can be done in many ways, including but not limited to checking appropriate boxes, entering or selecting numbers, entering or selecting ranges, entering or selecting grades or any other conventional assessment indicators, selecting or entering percentages, and so forth, assigning numbers, assigning conventional standards, entering numbers, selecting for example achieved scenario, marking achieved items, clicking (marking, noting, or pushing) on scenarios items to document learning goals or expectations either achieved or missed (or both, in some cases), per all learning goal subdivisions (including units/subunits, criteria, scenarios, categories, subunits, items, parts, and so forth).). Any type of input may be related to formulas and calculations. For example, a learning assessor may select a conventional standard that is associated with numerical ranges. Criteria, scenarios, items may have numeric values. When a learning assessor marks an item or scenario (for example), that item or scenario may have numeric values. All assessment data produced in assessing learning outcomes based on goals, identifier information, learning goals metrics and weights, learning expectations metrics, and weights, learners' individual outputs are stored in data repositories.)
[0010]: In another embodiment, learning indexes are generated for a plurality of aggregates of individual learners, assembled based on membership of individual learners in one or more learning units, zones, or levels. In another embodiment, the learning indexes are used to generate grade reports with feedback for learners. In another embodiment, the report generator generates and distributes reports based at least in part on the aggregated learning indexes, the reports identifying areas of achieved and missed learning relative to established learning goals and expectations. In yet another embodiment, the analysis engine performs analysis of a plurality of learning indexes or learning outcome reports, or both, pertaining to a learner and prepares thereby and distributes a learning improvement plan tailored to the learner. In another embodiment, the analysis engine automatically analyzes progress of the learning improvement plan and, based at least on comparing learning outcome assessments from before and from after implementation of the learning improvement plan, adjusts the learning improvement plan or prepares and distributes a new learning improvement plan. [0011]: a method for objective assessment of learning outcomes is disclosed, the method comprising the steps of: (a) providing an administrative interface via an application server to allow users to specify a plurality of learning goals and expectations; (b) decomposing at least a portion of the learning goals and expectations into achievable and measurable analytics units; (c) organizing the learning goals and expectations into a hierarchy; (d) automatically performing consistency checks to ensure alignment of learning goals and expectations along the hierarchy; (e) providing a plurality of learning assessment tools to a learning assessor in one of online, mobile application, or thick client application formats; (f) receiving learning outcome assessment data at the level of individual learning outcomes from the learning assessor; (g) calculating learning outcomes as learning indexes at the level of an individual output; and (h) preparing and distributing a plurality of learning outcome reports for the individual learner. [0054]: Learning outputs or learning outcomes may be reviewed and assessed (what is commonly referred to as “grading”) by learning assessors or agents qualified to do so, including but not limited to educators, faculty, graders, etc. Individual learning outputs represent output of individual learners but also of groups of learners (in case of team projects). Assessments are made first at the level of individual learning outputs. Learning outcomes and performance define consequences of the processes of learning and education. Achieved (acquired) learning shows what learners learned in relation to planned learning goals; missed learning shows gaps or missed learning in relation to planned learning goals. Learning indexes are numeric measures of leaning that quantify learning outcomes (achieved and or missed learning) in all configurations. [0055]: Learning indexes are first calculated at the level individual of the learning output unit. They can be calculated at all configurations afterwards by “rolling up” or aggregating learning index data starting with raw data at the level of learning outputs and then working up one or more hierarchies, using weighting factors or other formulae that define how aggregation is to be carried out. [0084] Additional activities undertaken during organizing 420 may include designing one or more learning processes 422, designing or creating various forms, records, and/or rubrics or other tools for performing assessments 423 of learning, designing one or more data repositories and specifying data fields including identifying fields for various hierarchy levels, organizations, zones, and the like, establishing routines for and carrying out data collection 424 regarding various aspects of the learning environment (for example, organizational structures within a university, course catalogs, learner rosters, faculty rosters, previous learner learning histories at the same or other institutions, regulatory requirements such as required tests and required proficiency demonstrations, and so forth), performing calculations 425 required to implement a consistent, hierarchical objective learning assessment system, building or establishing data repositories 426 that will be available to appropriate users (such as learners, learning agents, administrators, and so forth), and building a plurality of reports 427 or report templates that may be used by administrators, regulators, and others to assess and analyze the performance of learning processes and learning organizations. [0057] As used herein, “assessment records”, or “rubrics”, or “templates”, mean “a standard of performance for a defined population”, particularly as it is applied against learning goals. Rubrics etc. may comprise, for example, one or more items such as required ID information, goal metrics or analytics or criteria dimensions on which performance is rated, definitions and examples that illustrate the attribute(s) being measured, and a rating scale criteria item, numerical achievable values in various formats such as percentages, absolute numbers, etc., areas where assessors can select achieved learning items, make notes. Dimensions are generally referred to as criteria, the rating scale as levels, and definitions as descriptors. Rubrics or templates typically reflect learning goals metrics for their specific level such as for example the learning output level. They may also reflect learning expectations metrics. [0063] As used herein, an “assessor” is a learning stakeholder (for example, a faculty member, a grader, a teaching assistant, a teacher, an instructor, or the like) or an automated system (such as an automated grading system), or a combination of the two, that is responsible for assessing (grading) one or more learning outcomes. Many examples herein use terms such as “faculty assessor”; these are merely exemplary and other examples are possible as well, according to the invention, and in general the term “assessor” should be understood as defined here. [0098] Data repository 640 may be used to store and document data pertaining to learning goals and processes related to learning goals, all the way down a hierarchy to specific units of learning delivery and learning outputs, including assigned values and formats, analytical means, feedback, etc. Identification of units of learning delivery and learning outputs may also be stored in database 640 (examples to include but not limit degrees, courses, classes, modules, teaching units, assignments). Identification could contain, for example, institution/college codes/ID, degree, course, etc. in formats including acronyms, numbers, symbols, etc. [0117] Once goals have been created and optionally assigned to a hierarchy in step 710, in step 720 processing of learning goals at the level of individual output delivery takes place and one or more analytical criteria may be defined that will be used in assessing progress in achieving goals at various levels of a hierarchy. [0118]: In this sense, units of learning may be hierarchical. According to the embodiment, learning agents, agencies, learners, administrators, or other participants determine one or more overall learning expectations in high-level step 810. In general, specific expectations will be tailored to specific units of learning 812 and correspondingly assigned to one or more levels to create a hierarchy of learning expectations 812. In some embodiments, participants may rank expectations 814 based on a desired order of importance or relevance.).
However, Luca does not explicitly show “preparing a list of potential assessors”. Reed shows the above limitation at least in [0026] FIG. 5 shows an educator list for selection prior to an observation. [0028] FIG. 7 shows a visit list for a specific educator to be observed. [0049] FIG. 28 shows a list of educators whose evaluations can be managed. [0050] FIG. 29 shows an evaluation portfolio for a particular educator. [0055] FIG. 34 shows a list of educators under the Observation tab. [0059] FIG. 38 shows the ability to assign specific professional development coursework to specific indicators based on scores. [0061] FIG. 40 shows a list of required professional development courses required under the Professional Development tab, along with applicable deadlines for completion. [0063] FIG. 42 shows the user completed professional development courses with a rating system to score the coursework.
Reference Luca and Reference Reed are analogous prior art to the claimed invention because the references generally relate to field of monitoring learning outcomes. Further, said references are part of the same classification, i.e., G06Q50. Lastly, said references are filed before the effective filing date of the instant application; hence, said references are analogous prior-art references.
It would have been obvious to one of ordinary skill in the art before the effective filing date of this application for AIA to provide the teachings of Reference Reed, particularly the ability to prepare a list of teachers [0026], in the disclosure of Reference Luca, particularly in the learning indexes [0055], building reports section [0084], assessment records [0057], in order to provide for an automated system for observation and evaluation of teachers which provides consistency, transparency and flexibility to administrators, fully accommodates the specific preferences of schools and school districts, and which includes rubric-agnostic tools that allow educators to make decisions about reporting and remedial measures as taught by Reference Reed (see at least in [0010]), where upon the execution of the method and system of Reference Luca for allowing learning indexes and assessment records monitoring to help make the process of managing and monitoring learning outcomes more efficient and effective.
Further, the claimed invention is merely a combination of old elements in a similar monitoring learning outcomes field of endeavor, and in the combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that, given the existing technical ability to combine the elements as evidenced by Reference Luca in view of Reference Reed, the results of the combination were predictable (MPEP 2143 A).
Regarding the claim limitations below, Reference Luca in view of Reed shows:
assigning the select assessors to a set of students in the course, and monitoring the assigned assessors during and after the course.
(Luca: [0135]: in general learning goals are arranged in tables 1710, 1720, 1730, 1740 according to category (i.e., learning goal type), and individual subcategories may be arranged on individual rows within goal category tables; each row typically will have a subcategory label in a first column 1711, absolute (or percentile, as desired) values of maximum scores for a given subcategory (that is, column 1711 lists maximum scores for each subcategory), actual scores achieved in a second column 1712, percentage of maximum achieved in a third column 1713, and explanatory text for each subcategory in a fourth column 1715. Other columns may of course be added as desired, for example to show class assignments, prior scores, r to provide a text entry field within which a learning assessor make comments. Typically, for each goal category, a first row 1716 presents header information and may comprise a “SUBMIT” button to allow a user to commit a set of category-specific marks to data repository 640 (overall “SUBMIT” button 1750 performs the same function, but commits all learning goal grades entered to data repository 640. A second row 1717 may be provided that presents totals for each column within a given learning goal category; fields in this row are typically populated automatically by programmatically adding the corresponding values from rows 1718-1719 that comprise actual goal-specific grades data.)
As per claims 2 and 11: Regarding the claim limitations below, Reference Luca in view of Reed shows:
further comprising, prior to selecting assessors, displaying the list of potential assessors to the instructor.
Luca shows [0010]: In another embodiment, learning indexes are generated for a plurality of aggregates of individual learners, assembled based on membership of individual learners in one or more learning units, zones, or levels. In another embodiment, the learning indexes are used to generate grade reports with feedback for learners. In another embodiment, the report generator generates and distributes reports based at least in part on the aggregated learning indexes, the reports identifying areas of achieved and missed learning relative to established learning goals and expectations. In yet another embodiment, the analysis engine performs analysis of a plurality of learning indexes or learning outcome reports, or both, pertaining to a learner and prepares thereby and distributes a learning improvement plan tailored to the learner. In another embodiment, the analysis engine automatically analyzes progress of the learning improvement plan and, based at least on comparing learning outcome assessments from before and from after implementation of the learning improvement plan, adjusts the learning improvement plan or prepares and distributes a new learning improvement plan. [0011]: a method for objective assessment of learning outcomes is disclosed, the method comprising the steps of: (a) providing an administrative interface via an application server to allow users to specify a plurality of learning goals and expectations; (b) decomposing at least a portion of the learning goals and expectations into achievable and measurable analytics units; (c) organizing the learning goals and expectations into a hierarchy; (d) automatically performing consistency checks to ensure alignment of learning goals and expectations along the hierarchy; (e) providing a plurality of learning assessment tools to a learning assessor in one of online, mobile application, or thick client application formats; (f) receiving learning outcome assessment data at the level of individual learning outcomes from the learning assessor; (g) calculating learning outcomes as learning indexes at the level of an individual output; and (h) preparing and distributing a plurality of learning outcome reports for the individual learner. [0054]: Learning outputs or learning outcomes may be reviewed and assessed (what is commonly referred to as “grading”) by learning assessors or agents qualified to do so, including but not limited to educators, faculty, graders, etc. Individual learning outputs represent output of individual learners but also of groups of learners (in case of team projects). Assessments are made first at the level of individual learning outputs. Learning outcomes and performance define consequences of the processes of learning and education. Achieved (acquired) learning shows what learners learned in relation to planned learning goals; missed learning shows gaps or missed learning in relation to planned learning goals. Learning indexes are numeric measures of leaning that quantify learning outcomes (achieved and or missed learning) in all configurations. [0055]: Learning indexes are first calculated at the level individual of the learning output unit. They can be calculated at all configurations afterwards by “rolling up” or aggregating learning index data starting with raw data at the level of learning outputs and then working up one or more hierarchies, using weighting factors or other formulae that define how aggregation is to be carried out. [0084] Additional activities undertaken during organizing 420 may include designing one or more learning processes 422, designing or creating various forms, records, and/or rubrics or other tools for performing assessments 423 of learning, designing one or more data repositories and specifying data fields including identifying fields for various hierarchy levels, organizations, zones, and the like, establishing routines for and carrying out data collection 424 regarding various aspects of the learning environment (for example, organizational structures within a university, course catalogs, learner rosters, faculty rosters, previous learner learning histories at the same or other institutions, regulatory requirements such as required tests and required proficiency demonstrations, and so forth), performing calculations 425 required to implement a consistent, hierarchical objective learning assessment system, building or establishing data repositories 426 that will be available to appropriate users (such as learners, learning agents, administrators, and so forth), and building a plurality of reports 427 or report templates that may be used by administrators, regulators, and others to assess and analyze the performance of learning processes and learning organizations. [0057] As used herein, “assessment records”, or “rubrics”, or “templates”, mean “a standard of performance for a defined population”, particularly as it is applied against learning goals. Rubrics etc. may comprise, for example, one or more items such as required ID information, goal metrics or analytics or criteria dimensions on which performance is rated, definitions and examples that illustrate the attribute(s) being measured, and a rating scale criteria item, numerical achievable values in various formats such as percentages, absolute numbers, etc., areas where assessors can select achieved learning items, make notes. Dimensions are generally referred to as criteria, the rating scale as levels, and definitions as descriptors. Rubrics or templates typically reflect learning goals metrics for their specific level such as for example the learning output level. They may also reflect learning expectations metrics. [0063] As used herein, an “assessor” is a learning stakeholder (for example, a faculty member, a grader, a teaching assistant, a teacher, an instructor, or the like) or an automated system (such as an automated grading system), or a combination of the two, that is responsible for assessing (grading) one or more learning outcomes. Many examples herein use terms such as “faculty assessor”; these are merely exemplary and other examples are possible as well, according to the invention, and in general the term “assessor” should be understood as defined here. [0098] Data repository 640 may be used to store and document data pertaining to learning goals and processes related to learning goals, all the way down a hierarchy to specific units of learning delivery and learning outputs, including assigned values and formats, analytical means, feedback, etc. Identification of units of learning delivery and learning outputs may also be stored in database 640 (examples to include but not limit degrees, courses, classes, modules, teaching units, assignments). Identification could contain, for example, institution/college codes/ID, degree, course, etc. in formats including acronyms, numbers, symbols, etc. [0117] Once goals have been created and optionally assigned to a hierarchy in step 710, in step 720 processing of learning goals at the level of individual output delivery takes place and one or more analytical criteria may be defined that will be used in assessing progress in achieving goals at various levels of a hierarchy. [0118]: In this sense, units of learning may be hierarchical. According to the embodiment, learning agents, agencies, learners, administrators, or other participants determine one or more overall learning expectations in high-level step 810. In general, specific expectations will be tailored to specific units of learning 812 and correspondingly assigned to one or more levels to create a hierarchy of learning expectations 812. In some embodiments, participants may rank expectations 814 based on a desired order of importance or relevance.).
However, Luca does not explicitly show “preparing a list of potential assessors”. Reed shows the above limitation at least in [0026] FIG. 5 shows an educator list for selection prior to an observation. [0028] FIG. 7 shows a visit list for a specific educator to be observed. [0049] FIG. 28 shows a list of educators whose evaluations can be managed. [0050] FIG. 29 shows an evaluation portfolio for a particular educator. [0055] FIG. 34 shows a list of educators under the Observation tab. [0059] FIG. 38 shows the ability to assign specific professional development coursework to specific indicators based on scores. [0061] FIG. 40 shows a list of required professional development courses required under the Professional Development tab, along with applicable deadlines for completion. [0063] FIG. 42 shows the user completed professional development courses with a rating system to score the coursework.
Reference Luca and Reference Reed are analogous prior art to the claimed invention because the references generally relate to field of monitoring learning outcomes. Further, said references are part of the same classification, i.e., G06Q50. Lastly, said references are filed before the effective filing date of the instant application; hence, said references are analogous prior-art references.
It would have been obvious to one of ordinary skill in the art before the effective filing date of this application for AIA to provide the teachings of Reference Reed, particularly the ability to prepare a list of teachers [0026], in the disclosure of Reference Luca, particularly in the learning indexes [0055], building reports section [0084], assessment records [0057], in order to provide for an automated system for observation and evaluation of teachers which provides consistency, transparency and flexibility to administrators, fully accommodates the specific preferences of schools and school districts, and which includes rubric-agnostic tools that allow educators to make decisions about reporting and remedial measures as taught by Reference Reed (see at least in [0010]), where upon the execution of the method and system of Reference Luca for allowing learning indexes and assessment records monitoring to help make the process of managing and monitoring learning outcomes more efficient and effective.
Further, the claimed invention is merely a combination of old elements in a similar monitoring learning outcomes field of endeavor, and in the combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that, given the existing technical ability to combine the elements as evidenced by Reference Luca in view of Reference Reed, the results of the combination were predictable (MPEP 2143 A);
As per claims 3 and 12: Regarding the claim limitations below, Reference Luca in view of Reed shows:
wherein the displaying the list of potential assessors comprises presenting one or more informational elements for each assessor.
Luca [0010]: In another embodiment, learning indexes are generated for a plurality of aggregates of individual learners, assembled based on membership of individual learners in one or more learning units, zones, or levels. In another embodiment, the learning indexes are used to generate grade reports with feedback for learners. In another embodiment, the report generator generates and distributes reports based at least in part on the aggregated learning indexes, the reports identifying areas of achieved and missed learning relative to established learning goals and expectations. In yet another embodiment, the analysis engine performs analysis of a plurality of learning indexes or learning outcome reports, or both, pertaining to a learner and prepares thereby and distributes a learning improvement plan tailored to the learner. In another embodiment, the analysis engine automatically analyzes progress of the learning improvement plan and, based at least on comparing learning outcome assessments from before and from after implementation of the learning improvement plan, adjusts the learning improvement plan or prepares and distributes a new learning improvement plan. [0011]: a method for objective assessment of learning outcomes is disclosed, the method comprising the steps of: (a) providing an administrative interface via an application server to allow users to specify a plurality of learning goals and expectations; (b) decomposing at least a portion of the learning goals and expectations into achievable and measurable analytics units; (c) organizing the learning goals and expectations into a hierarchy; (d) automatically performing consistency checks to ensure alignment of learning goals and expectations along the hierarchy; (e) providing a plurality of learning assessment tools to a learning assessor in one of online, mobile application, or thick client application formats; (f) receiving learning outcome assessment data at the level of individual learning outcomes from the learning assessor; (g) calculating learning outcomes as learning indexes at the level of an individual output; and (h) preparing and distributing a plurality of learning outcome reports for the individual learner. [0054]: Learning outputs or learning outcomes may be reviewed and assessed (what is commonly referred to as “grading”) by learning assessors or agents qualified to do so, including but not limited to educators, faculty, graders, etc. Individual learning outputs represent output of individual learners but also of groups of learners (in case of team projects). Assessments are made first at the level of individual learning outputs. Learning outcomes and performance define consequences of the processes of learning and education. Achieved (acquired) learning shows what learners learned in relation to planned learning goals; missed learning shows gaps or missed learning in relation to planned learning goals. Learning indexes are numeric measures of leaning that quantify learning outcomes (achieved and or missed learning) in all configurations. [0055]: Learning indexes are first calculated at the level individual of the learning output unit. They can be calculated at all configurations afterwards by “rolling up” or aggregating learning index data starting with raw data at the level of learning outputs and then working up one or more hierarchies, using weighting factors or other formulae that define how aggregation is to be carried out. [0084] Additional activities undertaken during organizing 420 may include designing one or more learning processes 422, designing or creating various forms, records, and/or rubrics or other tools for performing assessments 423 of learning, designing one or more data repositories and specifying data fields including identifying fields for various hierarchy levels, organizations, zones, and the like, establishing routines for and carrying out data collection 424 regarding various aspects of the learning environment (for example, organizational structures within a university, course catalogs, learner rosters, faculty rosters, previous learner learning histories at the same or other institutions, regulatory requirements such as required tests and required proficiency demonstrations, and so forth), performing calculations 425 required to implement a consistent, hierarchical objective learning assessment system, building or establishing data repositories 426 that will be available to appropriate users (such as learners, learning agents, administrators, and so forth), and building a plurality of reports 427 or report templates that may be used by administrators, regulators, and others to assess and analyze the performance of learning processes and learning organizations. [0057] As used herein, “assessment records”, or “rubrics”, or “templates”, mean “a standard of performance for a defined population”, particularly as it is applied against learning goals. Rubrics etc. may comprise, for example, one or more items such as required ID information, goal metrics or analytics or criteria dimensions on which performance is rated, definitions and examples that illustrate the attribute(s) being measured, and a rating scale criteria item, numerical achievable values in various formats such as percentages, absolute numbers, etc., areas where assessors can select achieved learning items, make notes. Dimensions are generally referred to as criteria, the rating scale as levels, and definitions as descriptors. Rubrics or templates typically reflect learning goals metrics for their specific level such as for example the learning output level. They may also reflect learning expectations metrics. [0063] As used herein, an “assessor” is a learning stakeholder (for example, a faculty member, a grader, a teaching assistant, a teacher, an instructor, or the like) or an automated system (such as an automated grading system), or a combination of the two, that is responsible for assessing (grading) one or more learning outcomes. Many examples herein use terms such as “faculty assessor”; these are merely exemplary and other examples are possible as well, according to the invention, and in general the term “assessor” should be understood as defined here. [0098] Data repository 640 may be used to store and document data pertaining to learning goals and processes related to learning goals, all the way down a hierarchy to specific units of learning delivery and learning outputs, including assigned values and formats, analytical means, feedback, etc. Identification of units of learning delivery and learning outputs may also be stored in database 640 (examples to include but not limit degrees, courses, classes, modules, teaching units, assignments). Identification could contain, for example, institution/college codes/ID, degree, course, etc. in formats including acronyms, numbers, symbols, etc. [0117] Once goals have been created and optionally assigned to a hierarchy in step 710, in step 720 processing of learning goals at the level of individual output delivery takes place and one or more analytical criteria may be defined that will be used in assessing progress in achieving goals at various levels of a hierarchy. [0118]: In this sense, units of learning may be hierarchical. According to the embodiment, learning agents, agencies, learners, administrators, or other participants determine one or more overall learning expectations in high-level step 810. In general, specific expectations will be tailored to specific units of learning 812 and correspondingly assigned to one or more levels to create a hierarchy of learning expectations 812. In some embodiments, participants may rank expectations 814 based on a desired order of importance or relevance.).
However, Luca does not explicitly show “preparing a list of potential assessors”. Reed shows the above limitation at least in [0026] FIG. 5 shows an educator list for selection prior to an observation. [0028] FIG. 7 shows a visit list for a specific educator to be observed. [0049] FIG. 28 shows a list of educators whose evaluations can be managed. [0050] FIG. 29 shows an evaluation portfolio for a particular educator. [0055] FIG. 34 shows a list of educators under the Observation tab. [0059] FIG. 38 shows the ability to assign specific professional development coursework to specific indicators based on scores. [0061] FIG. 40 shows a list of required professional development courses required under the Professional Development tab, along with applicable deadlines for completion. [0063] FIG. 42 shows the user completed professional development courses with a rating system to score the coursework.
Reference Luca and Reference Reed are analogous prior art to the claimed invention because the references generally relate to field of monitoring learning outcomes. Further, said references are part of the same classification, i.e., G06Q50. Lastly, said references are filed before the effective filing date of the instant application; hence, said references are analogous prior-art references.
It would have been obvious to one of ordinary skill in the art before the effective filing date of this application for AIA to provide the teachings of Reference Reed, particularly the ability to prepare a list of teachers [0026], in the disclosure of Reference Luca, particularly in the learning indexes [0055], building reports section [0084], assessment records [0057], in order to provide for an automated system for observation and evaluation of teachers which provides consistency, transparency and flexibility to administrators, fully accommodates the specific preferences of schools and school districts, and which includes rubric-agnostic tools that allow educators to make decisions about reporting and remedial measures as taught by Reference Reed (see at least in [0010]), where upon the execution of the method and system of Reference Luca for allowing learning indexes and assessment records monitoring to help make the process of managing and monitoring learning outcomes more efficient and effective.
Further, the claimed invention is merely a combination of old elements in a similar monitoring learning outcomes field of endeavor, and in the combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that, given the existing technical ability to combine the elements as evidenced by Reference Luca in view of Reference Reed, the results of the combination were predictable (MPEP 2143 A).
As per claims 4 and 13: Regarding the claim limitations below, Reference Luca in view of Reed shows:
wherein the one or more informational elements for each assessor comprises, if available, past evaluations for each assessor.
Luca shows in [0127]: data gathered in step 1210 may comprise identifying information 1211, data pertaining to a plurality of learning goals and learning goal metrics 1212 at various hierarchical levels and at individual learning output level, data pertaining to a plurality of learning expectations and expectations metrics at the level of individual learning output 1213 also at various hierarchical levels, conventional standards (such as numeric or literal grades for example) 1214, faculty or other learning agent learning assessments inputs at the output level 1215 such as previous learning assessments pertaining to a specific learner or group of learners, learning indexes at output level 1216 from learning indexes computation process 1100, and other calculated items (such as, for example, totals, final grades, etc.) 1217 such as assigned grades for previous learning outputs. Grade and grade and feedback reports may comprise final grades, explanatory text regarding one or more meanings of the final grades, reports of achievement of learning goals and/or expectations, such as learning indexes achieved and missed (provided as totals and percentages per scenarios, categories, units, or levels of learning), commentaries, explanations, charts to illustrate achieved, missed, comparisons of learner learning indexes to group learning indexes, and so forth. Reports may provide recommended solutions for learning problems as well as assessment data.
As per claims 5 and 14: Regarding the claim limitations below, Reference Luca in view of Reed shows:
wherein the displaying the list of potential assessors comprises:
presenting a first informational element for assessors that are already approved
Luca [0010]: In another embodiment, learning indexes are generated for a plurality of aggregates of individual learners, assembled based on membership of individual learners in one or more learning units, zones, or levels. In another embodiment, the learning indexes are used to generate grade reports with feedback for learners. In another embodiment, the report generator generates and distributes reports based at least in part on the aggregated learning indexes, the reports identifying areas of achieved and missed learning relative to established learning goals and expectations. In yet another embodiment, the analysis engine performs analysis of a plurality of learning indexes or learning outcome reports, or both, pertaining to a learner and prepares thereby and distributes a learning improvement plan tailored to the learner. In another embodiment, the analysis engine automatically analyzes progress of the learning improvement plan and, based at least on comparing learning outcome assessments from before and from after implementation of the learning improvement plan, adjusts the learning improvement plan or prepares and distributes a new learning improvement plan. [0011]: a method for objective assessment of learning outcomes is disclosed, the method comprising the steps of: (a) providing an administrative interface via an application server to allow users to specify a plurality of learning goals and expectations; (b) decomposing at least a portion of the learning goals and expectations into achievable and measurable analytics units; (c) organizing the learning goals and expectations into a hierarchy; (d) automatically performing consistency checks to ensure alignment of learning goals and expectations along the hierarchy; (e) providing a plurality of learning assessment tools to a learning assessor in one of online, mobile application, or thick client application formats; (f) receiving learning outcome assessment data at the level of individual learning outcomes from the learning assessor; (g) calculating learning outcomes as learning indexes at the level of an individual output; and (h) preparing and distributing a plurality of learning outcome reports for the individual learner. [0054]: Learning outputs or learning outcomes may be reviewed and assessed (what is commonly referred to as “grading”) by learning assessors or agents qualified to do so, including but not limited to educators, faculty, graders, etc. Individual learning outputs represent output of individual learners but also of groups of learners (in case of team projects). Assessments are made first at the level of individual learning outputs. Learning outcomes and performance define consequences of the processes of learning and education. Achieved (acquired) learning shows what learners learned in relation to planned learning goals; missed learning shows gaps or missed learning in relation to planned learning goals. Learning indexes are numeric measures of leaning that quantify learning outcomes (achieved and or missed learning) in all configurations. [0055]: Learning indexes are first calculated at the level individual of the learning output unit. They can be calculated at all configurations afterwards by “rolling up” or aggregating learning index data starting with raw data at the level of learning outputs and then working up one or more hierarchies, using weighting factors or other formulae that define how aggregation is to be carried out. [0084] Additional activities undertaken during organizing 420 may include designing one or more learning processes 422, designing or creating various forms, records, and/or rubrics or other tools for performing assessments 423 of learning, designing one or more data repositories and specifying data fields including identifying fields for various hierarchy levels, organizations, zones, and the like, establishing routines for and carrying out data collection 424 regarding various aspects of the learning environment (for example, organizational structures within a university, course catalogs, learner rosters, faculty rosters, previous learner learning histories at the same or other institutions, regulatory requirements such as required tests and required proficiency demonstrations, and so forth), performing calculations 425 required to implement a consistent, hierarchical objective learning assessment system, building or establishing data repositories 426 that will be available to appropriate users (such as learners, learning agents, administrators, and so forth), and building a plurality of reports 427 or report templates that may be used by administrators, regulators, and others to assess and analyze the performance of learning processes and learning organizations. [0057] As used herein, “assessment records”, or “rubrics”, or “templates”, mean “a standard of performance for a defined population”, particularly as it is applied against learning goals. Rubrics etc. may comprise, for example, one or more items such as required ID information, goal metrics or analytics or criteria dimensions on which performance is rated, definitions and examples that illustrate the attribute(s) being measured, and a rating scale criteria item, numerical achievable values in various formats such as percentages, absolute numbers, etc., areas where assessors can select achieved learning items, make notes. Dimensions are generally referred to as criteria, the rating scale as levels, and definitions as descriptors. Rubrics or templates typically reflect learning goals metrics for their specific level such as for example the learning output level. They may also reflect learning expectations metrics. [0063] As used herein, an “assessor” is a learning stakeholder (for example, a faculty member, a grader, a teaching assistant, a teacher, an instructor, or the like) or an automated system (such as an automated grading system), or a combination of the two, that is responsible for assessing (grading) one or more learning outcomes. Many examples herein use terms such as “faculty assessor”; these are merely exemplary and other examples are possible as well, according to the invention, and in general the term “assessor” should be understood as defined here. [0098] Data repository 640 may be used to store and document data pertaining to learning goals and processes related to learning goals, all the way down a hierarchy to specific units of learning delivery and learning outputs, including assigned values and formats, analytical means, feedback, etc. Identification of units of learning delivery and learning outputs may also be stored in database 640 (examples to include but not limit degrees, courses, classes, modules, teaching units, assignments). Identification could contain, for example, institution/college codes/ID, degree, course, etc. in formats including acronyms, numbers, symbols, etc. [0117] Once goals have been created and optionally assigned to a hierarchy in step 710, in step 720 processing of learning goals at the level of individual output delivery takes place and one or more analytical criteria may be defined that will be used in assessing progress in achieving goals at various levels of a hierarchy. [0118]: In this sense, units of learning may be hierarchical. According to the embodiment, learning agents, agencies, learners, administrators, or other participants determine one or more overall learning expectations in high-level step 810. In general, specific expectations will be tailored to specific units of learning 812 and correspondingly assigned to one or more levels to create a hierarchy of learning expectations 812. In some embodiments, participants may rank expectations 814 based on a desired order of importance or relevance.).
However, Luca does not explicitly show “preparing a list of potential assessors”. Reed shows the above limitation at least in [0026] FIG. 5 shows an educator list for selection prior to an observation. [0028] FIG. 7 shows a visit list for a specific educator to be observed. [0049] FIG. 28 shows a list of educators whose evaluations can be managed. [0050] FIG. 29 shows an evaluation portfolio for a particular educator. [0055] FIG. 34 shows a list of educators under the Observation tab. [0059] FIG. 38 shows the ability to assign specific professional development coursework to specific indicators based on scores. [0061] FIG. 40 shows a list of required professional development courses required under the Professional Development tab, along with applicable deadlines for completion. [0063] FIG. 42 shows the user completed professional development courses with a rating system to score the coursework.
Reference Luca and Reference Reed are analogous prior art to the claimed invention because the references generally relate to field of monitoring learning outcomes. Further, said references are part of the same classification, i.e., G06Q50. Lastly, said references are filed before the effective filing date of the instant application; hence, said references are analogous prior-art references.
It would have been obvious to one of ordinary skill in the art before the effective filing date of this application for AIA to provide the teachings of Reference Reed, particularly the ability to prepare a list of teachers [0026], in the disclosure of Reference Luca, particularly in the learning indexes [0055], building reports section [0084], assessment records [0057], in order to provide for an automated system for observation and evaluation of teachers which provides consistency, transparency and flexibility to administrators, fully accommodates the specific preferences of schools and school districts, and which includes rubric-agnostic tools that allow educators to make decisions about reporting and remedial measures as taught by Reference Reed (see at least in [0010]), where upon the execution of the method and system of Reference Luca for allowing learning indexes and assessment records monitoring to help make the process of managing and monitoring learning outcomes more efficient and effective.
Further, the claimed invention is merely a combination of old elements in a similar monitoring learning outcomes field of endeavor, and in the combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that, given the existing technical ability to combine the elements as evidenced by Reference Luca in view of Reference Reed, the results of the combination were predictable (MPEP 2143 A); and
Regarding the claim limitations below, Reference Luca in view of Reed shows:
presenting a second informational element for assessors that require additional information before registration.
Luca [0010]: In another embodiment, learning indexes are generated for a plurality of aggregates of individual learners, assembled based on membership of individual learners in one or more learning units, zones, or levels. In another embodiment, the learning indexes are used to generate grade reports with feedback for learners. In another embodiment, the report generator generates and distributes reports based at least in part on the aggregated learning indexes, the reports identifying areas of achieved and missed learning relative to established learning goals and expectations. In yet another embodiment, the analysis engine performs analysis of a plurality of learning indexes or learning outcome reports, or both, pertaining to a learner and prepares thereby and distributes a learning improvement plan tailored to the learner. In another embodiment, the analysis engine automatically analyzes progress of the learning improvement plan and, based at least on comparing learning outcome assessments from before and from after implementation of the learning improvement plan, adjusts the learning improvement plan or prepares and distributes a new learning improvement plan. [0011]: a method for objective assessment of learning outcomes is disclosed, the method comprising the steps of: (a) providing an administrative interface via an application server to allow users to specify a plurality of learning goals and expectations; (b) decomposing at least a portion of the learning goals and expectations into achievable and measurable analytics units; (c) organizing the learning goals and expectations into a hierarchy; (d) automatically performing consistency checks to ensure alignment of learning goals and expectations along the hierarchy; (e) providing a plurality of learning assessment tools to a learning assessor in one of online, mobile application, or thick client application formats; (f) receiving learning outcome assessment data at the level of individual learning outcomes from the learning assessor; (g) calculating learning outcomes as learning indexes at the level of an individual output; and (h) preparing and distributing a plurality of learning outcome reports for the individual learner. [0054]: Learning outputs or learning outcomes may be reviewed and assessed (what is commonly referred to as “grading”) by learning assessors or agents qualified to do so, including but not limited to educators, faculty, graders, etc. Individual learning outputs represent output of individual learners but also of groups of learners (in case of team projects). Assessments are made first at the level of individual learning outputs. Learning outcomes and performance define consequences of the processes of learning and education. Achieved (acquired) learning shows what learners learned in relation to planned learning goals; missed learning shows gaps or missed learning in relation to planned learning goals. Learning indexes are numeric measures of leaning that quantify learning outcomes (achieved and or missed learning) in all configurations. [0055]: Learning indexes are first calculated at the level individual of the learning output unit. They can be calculated at all configurations afterwards by “rolling up” or aggregating learning index data starting with raw data at the level of learning outputs and then working up one or more hierarchies, using weighting factors or other formulae that define how aggregation is to be carried out. [0084] Additional activities undertaken during organizing 420 may include designing one or more learning processes 422, designing or creating various forms, records, and/or rubrics or other tools for performing assessments 423 of learning, designing one or more data repositories and specifying data fields including identifying fields for various hierarchy levels, organizations, zones, and the like, establishing routines for and carrying out data collection 424 regarding various aspects of the learning environment (for example, organizational structures within a university, course catalogs, learner rosters, faculty rosters, previous learner learning histories at the same or other institutions, regulatory requirements such as required tests and required proficiency demonstrations, and so forth), performing calculations 425 required to implement a consistent, hierarchical objective learning assessment system, building or establishing data repositories 426 that will be available to appropriate users (such as learners, learning agents, administrators, and so forth), and building a plurality of reports 427 or report templates that may be used by administrators, regulators, and others to assess and analyze the performance of learning processes and learning organizations. [0057] As used herein, “assessment records”, or “rubrics”, or “templates”, mean “a standard of performance for a defined population”, particularly as it is applied against learning goals. Rubrics etc. may comprise, for example, one or more items such as required ID information, goal metrics or analytics or criteria dimensions on which performance is rated, definitions and examples that illustrate the attribute(s) being measured, and a rating scale criteria item, numerical achievable values in various formats such as percentages, absolute numbers, etc., areas where assessors can select achieved learning items, make notes. Dimensions are generally referred to as criteria, the rating scale as levels, and definitions as descriptors. Rubrics or templates typically reflect learning goals metrics for their specific level such as for example the learning output level. They may also reflect learning expectations metrics. [0063] As used herein, an “assessor” is a learning stakeholder (for example, a faculty member, a grader, a teaching assistant, a teacher, an instructor, or the like) or an automated system (such as an automated grading system), or a combination of the two, that is responsible for assessing (grading) one or more learning outcomes. Many examples herein use terms such as “faculty assessor”; these are merely exemplary and other examples are possible as well, according to the invention, and in general the term “assessor” should be understood as defined here. [0098] Data repository 640 may be used to store and document data pertaining to learning goals and processes related to learning goals, all the way down a hierarchy to specific units of learning delivery and learning outputs, including assigned values and formats, analytical means, feedback, etc. Identification of units of learning delivery and learning outputs may also be stored in database 640 (examples to include but not limit degrees, courses, classes, modules, teaching units, assignments). Identification could contain, for example, institution/college codes/ID, degree, course, etc. in formats including acronyms, numbers, symbols, etc. [0117] Once goals have been created and optionally assigned to a hierarchy in step 710, in step 720 processing of learning goals at the level of individual output delivery takes place and one or more analytical criteria may be defined that will be used in assessing progress in achieving goals at various levels of a hierarchy. [0118]: In this sense, units of learning may be hierarchical. According to the embodiment, learning agents, agencies, learners, administrators, or other participants determine one or more overall learning expectations in high-level step 810. In general, specific expectations will be tailored to specific units of learning 812 and correspondingly assigned to one or more levels to create a hierarchy of learning expectations 812. In some embodiments, participants may rank expectations 814 based on a desired order of importance or relevance.).
As per claims 6 and 15: Regarding the claim limitations below, Reference Luca in view of Reed shows:
wherein the assigning the select assessors to the course comprises at least one of:
sending a communication to the accessor notifying of the assignment; and
Regarding the claim limitations below, Reference Luca in view of Reed shows:
providing login information to the assessor.
[0010]: In another embodiment, learning indexes are generated for a plurality of aggregates of individual learners, assembled based on membership of individual learners in one or more learning units, zones, or levels. In another embodiment, the learning indexes are used to generate grade reports with feedback for learners. In another embodiment, the report generator generates and distributes reports based at least in part on the aggregated learning indexes, the reports identifying areas of achieved and missed learning relative to established learning goals and expectations. In yet another embodiment, the analysis engine performs analysis of a plurality of learning indexes or learning outcome reports, or both, pertaining to a learner and prepares thereby and distributes a learning improvement plan tailored to the learner. In another embodiment, the analysis engine automatically analyzes progress of the learning improvement plan and, based at least on comparing learning outcome assessments from before and from after implementation of the learning improvement plan, adjusts the learning improvement plan or prepares and distributes a new learning improvement plan. [0011]: a method for objective assessment of learning outcomes is disclosed, the method comprising the steps of: (a) providing an administrative interface via an application server to allow users to specify a plurality of learning goals and expectations; (b) decomposing at least a portion of the learning goals and expectations into achievable and measurable analytics units; (c) organizing the learning goals and expectations into a hierarchy; (d) automatically performing consistency checks to ensure alignment of learning goals and expectations along the hierarchy; (e) providing a plurality of learning assessment tools to a learning assessor in one of online, mobile application, or thick client application formats; (f) receiving learning outcome assessment data at the level of individual learning outcomes from the learning assessor; (g) calculating learning outcomes as learning indexes at the level of an individual output; and (h) preparing and distributing a plurality of learning outcome reports for the individual learner. [0054]: Learning outputs or learning outcomes may be reviewed and assessed (what is commonly referred to as “grading”) by learning assessors or agents qualified to do so, including but not limited to educators, faculty, graders, etc. Individual learning outputs represent output of individual learners but also of groups of learners (in case of team projects). Assessments are made first at the level of individual learning outputs. Learning outcomes and performance define consequences of the processes of learning and education. Achieved (acquired) learning shows what learners learned in relation to planned learning goals; missed learning shows gaps or missed learning in relation to planned learning goals. Learning indexes are numeric measures of leaning that quantify learning outcomes (achieved and or missed learning) in all configurations. [0055]: Learning indexes are first calculated at the level individual of the learning output unit. They can be calculated at all configurations afterwards by “rolling up” or aggregating learning index data starting with raw data at the level of learning outputs and then working up one or more hierarchies, using weighting factors or other formulae that define how aggregation is to be carried out. [0084] Additional activities undertaken during organizing 420 may include designing one or more learning processes 422, designing or creating various forms, records, and/or rubrics or other tools for performing assessments 423 of learning, designing one or more data repositories and specifying data fields including identifying fields for various hierarchy levels, organizations, zones, and the like, establishing routines for and carrying out data collection 424 regarding various aspects of the learning environment (for example, organizational structures within a university, course catalogs, learner rosters, faculty rosters, previous learner learning histories at the same or other institutions, regulatory requirements such as required tests and required proficiency demonstrations, and so forth), performing calculations 425 required to implement a consistent, hierarchical objective learning assessment system, building or establishing data repositories 426 that will be available to appropriate users (such as learners, learning agents, administrators, and so forth), and building a plurality of reports 427 or report templates that may be used by administrators, regulators, and others to assess and analyze the performance of learning processes and learning organizations. [0057] As used herein, “assessment records”, or “rubrics”, or “templates”, mean “a standard of performance for a defined population”, particularly as it is applied against learning goals. Rubrics etc. may comprise, for example, one or more items such as required ID information, goal metrics or analytics or criteria dimensions on which performance is rated, definitions and examples that illustrate the attribute(s) being measured, and a rating scale criteria item, numerical achievable values in various formats such as percentages, absolute numbers, etc., areas where assessors can select achieved learning items, make notes. Dimensions are generally referred to as criteria, the rating scale as levels, and definitions as descriptors. Rubrics or templates typically reflect learning goals metrics for their specific level such as for example the learning output level. They may also reflect learning expectations metrics. [0063] As used herein, an “assessor” is a learning stakeholder (for example, a faculty member, a grader, a teaching assistant, a teacher, an instructor, or the like) or an automated system (such as an automated grading system), or a combination of the two, that is responsible for assessing (grading) one or more learning outcomes. Many examples herein use terms such as “faculty assessor”; these are merely exemplary and other examples are possible as well, according to the invention, and in general the term “assessor” should be understood as defined here. [0098] Data repository 640 may be used to store and document data pertaining to learning goals and processes related to learning goals, all the way down a hierarchy to specific units of learning delivery and learning outputs, including assigned values and formats, analytical means, feedback, etc. Identification of units of learning delivery and learning outputs may also be stored in database 640 (examples to include but not limit degrees, courses, classes, modules, teaching units, assignments). Identification could contain, for example, institution/college codes/ID, degree, course, etc. in formats including acronyms, numbers, symbols, etc. [0117] Once goals have been created and optionally assigned to a hierarchy in step 710, in step 720 processing of learning goals at the level of individual output delivery takes place and one or more analytical criteria may be defined that will be used in assessing progress in achieving goals at various levels of a hierarchy. [0118]: In this sense, units of learning may be hierarchical. According to the embodiment, learning agents, agencies, learners, administrators, or other participants determine one or more overall learning expectations in high-level step 810. In general, specific expectations will be tailored to specific units of learning 812 and correspondingly assigned to one or more levels to create a hierarchy of learning expectations 812. In some embodiments, participants may rank expectations 814 based on a desired order of importance or relevance.).
However, Luca does not explicitly show “login”. Reed shows the above limitation at least in [0067], [0088]-[0089], [0118].
Reference Luca and Reference Reed are analogous prior art to the claimed invention because the references generally relate to field of monitoring learning outcomes. Further, said references are part of the same classification, i.e., G06Q50. Lastly, said references are filed before the effective filing date of the instant application; hence, said references are analogous prior-art references.
It would have been obvious to one of ordinary skill in the art before the effective filing date of this application for AIA to provide the teachings of Reference Reed, particularly the ability to prepare a list of teachers [0026], in the disclosure of Reference Luca, particularly in the learning indexes [0055], building reports section [0084], assessment records [0057], in order to provide for an automated system for observation and evaluation of teachers which provides consistency, transparency and flexibility to administrators, fully accommodates the specific preferences of schools and school districts, and which includes rubric-agnostic tools that allow educators to make decisions about reporting and remedial measures as taught by Reference Reed (see at least in [0010]), where upon the execution of the method and system of Reference Luca for allowing learning indexes and assessment records monitoring to help make the process of managing and monitoring learning outcomes more efficient and effective.
Further, the claimed invention is merely a combination of old elements in a similar monitoring learning outcomes field of endeavor, and in the combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that, given the existing technical ability to combine the elements as evidenced by Reference Luca in view of Reference Reed, the results of the combination were predictable (MPEP 2143 A).
As per claims 7 and 16: Regarding the claim limitations below, Reference Luca in view of Reed shows:
wherein the assigning the select assessors to a set of students in the course further comprises:
updating the assignment of the assessors to a different set of students during the course.
Luca [0088]: Then, learning expectations metrics or analytics, to include numeric values of learning expectations per levels, units, categories, scenarios of learning are assigned. Then, learning expectations criteria to meet expectations, requirements per levels, units of learning, categories, scenarios of learning are created or specified. Then, learning expectations may be enhanced to clearly explain ranges of achievement of learning goals and what various sub ranges signify in terms of learning achievement, and explanations per ranges and per ratings (such as grades) may be provided. Finally, in some cases additional directions pertaining to how to improve learning based on achieving or not achieving one or more defined learning expectations may be provided. As in the case of learning goals, learning expectations are typically (but not necessarily) assigned one or more weights to facilitate their combination into higher-level aggregates, and to account for varying relative importance of different learning expectations. [0117]: a formula might combine various assignment completion data points, exam and quiz scores, and class participation scores to arrive at a quantitative level that characterizes whether a certain goal is met or not (or to what degree it is met). The method further analyzes each assignment into goal categories units achieved and missed learning.
As per claims 8 and 17: Regarding the claim limitations below, Reference Luca in view of Reed shows:
wherein the monitoring the assigned assessors comprises:
collecting evaluations for the assigned assessors
Luca does not explicitly show the above limitation. However, Reed shows in [0015] Users will be able to save observations or evaluations in progress and return to them later. Users can also start an observation or collection of data for one educator, exit that observation, and the system will automatically save the information collected. Users can start another observation for a different or the same educator without losing data from prior observations. Also, multiple rubrics may be applied to multiple educators, and those multiple observations can be open at the same time and completed any time the observer chooses.
It would have been obvious to one of ordinary skill in the art before the effective filing date of this application for AIA to provide the teachings of Reference Reed, particularly the ability to prepare a list of teachers [0026], in the disclosure of Reference Luca, particularly in the learning indexes [0055], building reports section [0084], assessment records [0057], in order to provide for an automated system for observation and evaluation of teachers which provides consistency, transparency and flexibility to administrators, fully accommodates the specific preferences of schools and school districts, and which includes rubric-agnostic tools that allow educators to make decisions about reporting and remedial measures as taught by Reference Reed (see at least in [0010]), where upon the execution of the method and system of Reference Luca for allowing learning indexes and assessment records monitoring to help make the process of managing and monitoring learning outcomes more efficient and effective.
Further, the claimed invention is merely a combination of old elements in a similar monitoring learning outcomes field of endeavor, and in the combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that, given the existing technical ability to combine the elements as evidenced by Reference Luca in view of Reference Reed, the results of the combination were predictable (MPEP 2143 A); and
Regarding the claim limitations below, Reference Luca in view of Reed shows:
storing the collected evaluations in association with the assigned assessors.
Luca does not explicitly show the above limitation. However, Reed shows in [0015] Users will be able to save observations or evaluations in progress and return to them later. Users can also start an observation or collection of data for one educator, exit that observation, and the system will automatically save the information collected. Users can start another observation for a different or the same educator without losing data from prior observations. Also, multiple rubrics may be applied to multiple educators, and those multiple observations can be open at the same time and completed any time the observer chooses.
It would have been obvious to one of ordinary skill in the art before the effective filing date of this application for AIA to provide the teachings of Reference Reed, particularly the ability to prepare a list of teachers [0026], in the disclosure of Reference Luca, particularly in the learning indexes [0055], building reports section [0084], assessment records [0057], in order to provide for an automated system for observation and evaluation of teachers which provides consistency, transparency and flexibility to administrators, fully accommodates the specific preferences of schools and school districts, and which includes rubric-agnostic tools that allow educators to make decisions about reporting and remedial measures as taught by Reference Reed (see at least in [0010]), where upon the execution of the method and system of Reference Luca for allowing learning indexes and assessment records monitoring to help make the process of managing and monitoring learning outcomes more efficient and effective.
Further, the claimed invention is merely a combination of old elements in a similar monitoring learning outcomes field of endeavor, and in the combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that, given the existing technical ability to combine the elements as evidenced by Reference Luca in view of Reference Reed, the results of the combination were predictable (MPEP 2143 A).
As per claims 9 and 18: Regarding the claim limitations below, Reference Luca in view of Reed shows:
wherein the determining if additional potential assessors are required comprises:
determining if the instructor has requested additional instructors
Luca does not explicitly show the above limitation. Reed shows the above limitation: [0087]: Observers can manage their schedule, and perform many operations, including but not limited to: (1) view all of their assigned evaluations or select a view of only their near-term work schedule; (2) make scheduling changes from their mobile device for flexible and efficient schedule changes; (3) save evaluations on their mobile device, regardless of the availability of a wireless connection, which means evaluations can progress on schedule and are safely stored on the mobile device until the observer syncs the device to the TOWER admin server. Following a synching operation, the uploaded evaluations are archived on the mobile device to avoid interfering with current and future evaluation schedules.
It would have been obvious to one of ordinary skill in the art before the effective filing date of this application for AIA to provide the teachings of Reference Reed, particularly the ability to prepare a list of teachers [0026], in the disclosure of Reference Luca, particularly in the learning indexes [0055], building reports section [0084], assessment records [0057], in order to provide for an automated system for observation and evaluation of teachers which provides consistency, transparency and flexibility to administrators, fully accommodates the specific preferences of schools and school districts, and which includes rubric-agnostic tools that allow educators to make decisions about reporting and remedial measures as taught by Reference Reed (see at least in [0010]), where upon the execution of the method and system of Reference Luca for allowing learning indexes and assessment records monitoring to help make the process of managing and monitoring learning outcomes more efficient and effective.
Further, the claimed invention is merely a combination of old elements in a similar monitoring learning outcomes field of endeavor, and in the combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that, given the existing technical ability to combine the elements as evidenced by Reference Luca in view of Reference Reed, the results of the combination were predictable (MPEP 2143 A), and
Regarding the claim limitations below, Reference Luca in view of Reed shows:
if not, determining if there are sufficient assessors on the list of potential assessors
Luca [0010]: In another embodiment, learning indexes are generated for a plurality of aggregates of individual learners, assembled based on membership of individual learners in one or more learning units, zones, or levels. In another embodiment, the learning indexes are used to generate grade reports with feedback for learners. In another embodiment, the report generator generates and distributes reports based at least in part on the aggregated learning indexes, the reports identifying areas of achieved and missed learning relative to established learning goals and expectations. In yet another embodiment, the analysis engine performs analysis of a plurality of learning indexes or learning outcome reports, or both, pertaining to a learner and prepares thereby and distributes a learning improvement plan tailored to the learner. In another embodiment, the analysis engine automatically analyzes progress of the learning improvement plan and, based at least on comparing learning outcome assessments from before and from after implementation of the learning improvement plan, adjusts the learning improvement plan or prepares and distributes a new learning improvement plan. [0011]: a method for objective assessment of learning outcomes is disclosed, the method comprising the steps of: (a) providing an administrative interface via an application server to allow users to specify a plurality of learning goals and expectations; (b) decomposing at least a portion of the learning goals and expectations into achievable and measurable analytics units; (c) organizing the learning goals and expectations into a hierarchy; (d) automatically performing consistency checks to ensure alignment of learning goals and expectations along the hierarchy; (e) providing a plurality of learning assessment tools to a learning assessor in one of online, mobile application, or thick client application formats; (f) receiving learning outcome assessment data at the level of individual learning outcomes from the learning assessor; (g) calculating learning outcomes as learning indexes at the level of an individual output; and (h) preparing and distributing a plurality of learning outcome reports for the individual learner. [0054]: Learning outputs or learning outcomes may be reviewed and assessed (what is commonly referred to as “grading”) by learning assessors or agents qualified to do so, including but not limited to educators, faculty, graders, etc. Individual learning outputs represent output of individual learners but also of groups of learners (in case of team projects). Assessments are made first at the level of individual learning outputs. Learning outcomes and performance define consequences of the processes of learning and education. Achieved (acquired) learning shows what learners learned in relation to planned learning goals; missed learning shows gaps or missed learning in relation to planned learning goals. Learning indexes are numeric measures of leaning that quantify learning outcomes (achieved and or missed learning) in all configurations. [0055]: Learning indexes are first calculated at the level individual of the learning output unit. They can be calculated at all configurations afterwards by “rolling up” or aggregating learning index data starting with raw data at the level of learning outputs and then working up one or more hierarchies, using weighting factors or other formulae that define how aggregation is to be carried out. [0084] Additional activities undertaken during organizing 420 may include designing one or more learning processes 422, designing or creating various forms, records, and/or rubrics or other tools for performing assessments 423 of learning, designing one or more data repositories and specifying data fields including identifying fields for various hierarchy levels, organizations, zones, and the like, establishing routines for and carrying out data collection 424 regarding various aspects of the learning environment (for example, organizational structures within a university, course catalogs, learner rosters, faculty rosters, previous learner learning histories at the same or other institutions, regulatory requirements such as required tests and required proficiency demonstrations, and so forth), performing calculations 425 required to implement a consistent, hierarchical objective learning assessment system, building or establishing data repositories 426 that will be available to appropriate users (such as learners, learning agents, administrators, and so forth), and building a plurality of reports 427 or report templates that may be used by administrators, regulators, and others to assess and analyze the performance of learning processes and learning organizations. [0057] As used herein, “assessment records”, or “rubrics”, or “templates”, mean “a standard of performance for a defined population”, particularly as it is applied against learning goals. Rubrics etc. may comprise, for example, one or more items such as required ID information, goal metrics or analytics or criteria dimensions on which performance is rated, definitions and examples that illustrate the attribute(s) being measured, and a rating scale criteria item, numerical achievable values in various formats such as percentages, absolute numbers, etc., areas where assessors can select achieved learning items, make notes. Dimensions are generally referred to as criteria, the rating scale as levels, and definitions as descriptors. Rubrics or templates typically reflect learning goals metrics for their specific level such as for example the learning output level. They may also reflect learning expectations metrics. [0063] As used herein, an “assessor” is a learning stakeholder (for example, a faculty member, a grader, a teaching assistant, a teacher, an instructor, or the like) or an automated system (such as an automated grading system), or a combination of the two, that is responsible for assessing (grading) one or more learning outcomes. Many examples herein use terms such as “faculty assessor”; these are merely exemplary and other examples are possible as well, according to the invention, and in general the term “assessor” should be understood as defined here. [0098] Data repository 640 may be used to store and document data pertaining to learning goals and processes related to learning goals, all the way down a hierarchy to specific units of learning delivery and learning outputs, including assigned values and formats, analytical means, feedback, etc. Identification of units of learning delivery and learning outputs may also be stored in database 640 (examples to include but not limit degrees, courses, classes, modules, teaching units, assignments). Identification could contain, for example, institution/college codes/ID, degree, course, etc. in formats including acronyms, numbers, symbols, etc. [0117] Once goals have been created and optionally assigned to a hierarchy in step 710, in step 720 processing of learning goals at the level of individual output delivery takes place and one or more analytical criteria may be defined that will be used in assessing progress in achieving goals at various levels of a hierarchy. [0118]: In this sense, units of learning may be hierarchical. According to the embodiment, learning agents, agencies, learners, administrators, or other participants determine one or more overall learning expectations in high-level step 810. In general, specific expectations will be tailored to specific units of learning 812 and correspondingly assigned to one or more levels to create a hierarchy of learning expectations 812. In some embodiments, participants may rank expectations 814 based on a desired order of importance or relevance.).
However, Luca does not explicitly show “preparing a list of potential assessors”. Reed shows the above limitation at least in [0026] FIG. 5 shows an educator list for selection prior to an observation. [0028] FIG. 7 shows a visit list for a specific educator to be observed. [0049] FIG. 28 shows a list of educators whose evaluations can be managed. [0050] FIG. 29 shows an evaluation portfolio for a particular educator. [0055] FIG. 34 shows a list of educators under the Observation tab. [0059] FIG. 38 shows the ability to assign specific professional development coursework to specific indicators based on scores. [0061] FIG. 40 shows a list of required professional development courses required under the Professional Development tab, along with applicable deadlines for completion. [0063] FIG. 42 shows the user completed professional development courses with a rating system to score the coursework.
Reference Luca and Reference Reed are analogous prior art to the claimed invention because the references generally relate to field of monitoring learning outcomes. Further, said references are part of the same classification, i.e., G06Q50. Lastly, said references are filed before the effective filing date of the instant application; hence, said references are analogous prior-art references.
It would have been obvious to one of ordinary skill in the art before the effective filing date of this application for AIA to provide the teachings of Reference Reed, particularly the ability to prepare a list of teachers [0026], in the disclosure of Reference Luca, particularly in the learning indexes [0055], building reports section [0084], assessment records [0057], in order to provide for an automated system for observation and evaluation of teachers which provides consistency, transparency and flexibility to administrators, fully accommodates the specific preferences of schools and school districts, and which includes rubric-agnostic tools that allow educators to make decisions about reporting and remedial measures as taught by Reference Reed (see at least in [0010]), where upon the execution of the method and system of Reference Luca for allowing learning indexes and assessment records monitoring to help make the process of managing and monitoring learning outcomes more efficient and effective.
Further, the claimed invention is merely a combination of old elements in a similar monitoring learning outcomes field of endeavor, and in the combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that, given the existing technical ability to combine the elements as evidenced by Reference Luca in view of Reference Reed, the results of the combination were predictable (MPEP 2143 A).
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
NPL Reference:
M. S. A. El-Seoud, A. Karkar, I. A. T. F. Taj-Eddin, H. F. El-Sofany, A. Dandashi and J. M. Al-Ja'am, "Semantic-Web automated course management and evaluation system using mobile applications," 2015 International Conference on Interactive Collaborative Learning (ICL), Firenze, Italy, 2015, pp. 271-282, doi: 10.1109/ICL.2015.7318037.
Different types of e-assessment systems that are recognized at universities and based on the campus wireless have been developed. These systems help the students to use their Mobile Phones as learning media to access the information more easily from anywhere and at anytime. Seppala and Alamaki developed a mobile learning project for teacher training. Their study compared the effectiveness of internet, face-to-face and mobile based instructions. Al Masri has proposed a study to compare the effective strategy in paper-based assessment with mobile-based assessment for assessing university students in English literature. It has been found that students gained better scores in mobile phone-based test than in paper-based test. This paper aims to determine and measure the effects of mobile-based assessments on the perception, achievement levels and performance of the students in internet-assisted courses. The main functionalities and features of this paper are: Knowledge evaluation, automatic generation of exams, exam grading, communication, course management, and questions-bank database.Foreign Reference:
(KR 20060117828 A) Shim et al. A system for management learning estimation based on the network is provided to enable an educatee to receive effective employment and higher-level education guide by making an educator smoothly advance schoolwork management in an automated computing environment. A learning estimation management server (101) is connected to a educator client (2), and generally manages an estimation variable setting information storing procedure and a schoolwork calculation procedure for each estimation item of each subject according to a computing item output from the educator client. An estimation variable setting information database block (113) is controlled by the server and stores/manages estimation variable setting information for each estimation item. A schoolwork result calculating module (120) is controlled by the server and generates/stores a schoolwork result corresponding to estimation points of each estimation item by converting/computing the estimation points based on the estimation variable setting information.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NANCY PRASAD whose telephone number is (571)270-3265. The examiner can normally be reached M-F: 8:00 AM - 4:30 PM EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Patricia Munson can be reached at (571)270-5396. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/N.N.P/Examiner, Art Unit 3624 /PATRICIA H MUNSON/Supervisory Patent Examiner, Art Unit 3624
Prosecution Insights