DETAILED ACTION
Notice to Applicant
1. The following is a FINAL Office action upon examination of application number 18/824,151, filed on 09/04/2024. Claims 1-20 are pending in this application, and have been examined on the merits discussed below.
2. The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
3. In the response filed July 23, 2026, Applicant amended claims 1, 10, and 18, and did not cancel any claims. No new claims were presented for examination.
4. Applicant's amendments to claims 1, 10, and 18 are hereby acknowledged. The amendments are not sufficient to overcome the previously issued claim rejection under 35 U.S.C. 101; accordingly, this rejection has been maintained.
Response to Arguments
5. Applicant's arguments filed July 23, 2026, have been fully considered.
6. Applicant submits “Even assuming, arguendo, that the currently amended claims are directed to an abstract idea, the currently amended claims integrate any asserted abstract idea into a practical application under Alice Prong Two of Step 2A.” [Applicant’s Remarks, 07/23/2026, page 12]
The Examiner respectfully disagrees. Under Step 2A, Prong Two of the eligibility inquiry, Applicant argues “even assuming, arguendo, that the currently amended claims are directed to an abstract idea, the currently amended claims integrate any asserted abstract idea into a practical application under Alice Prong Two of Step 2A.” The additional elements in exemplary claim 1 are: at least one processor, at least one non-transitory computer-readable storage medium storing instructions, the system, a respondent client device, and a multimodal model, which merely serves to tie the abstract idea to a particular technological environment (computer-based operating environment) via generic computing hardware, software/instructions, which is not sufficient to amount to a practical application, as noted in MPEP 2106.05. Applicant has provided no facts/evidence, cited any portion of the Specification, nor provided a persuasive line of reasoning showing how the additional elements are integrated with the abstract idea to integrate the abstract idea into a practical application.
Furthermore, it is noted that the claims are devoid of any discernible change, transformation, or improvement to a computer (software or hardware) or any existing technology. Applicant has not shown that any specific technological improvement is achieved within the scope of the claims. It bears emphasis that no processor, system, device, or technological elements are modified or improved upon in any discernible manner. Instead, the result produced by the claims is simply information relating to a customizes follow-up survey inquiry, which is not a technical result or improvement thereof.
Moreover, the additional elements fail to integrate the abstract idea into a practical application because they fail to provide an improvement to the functioning of a computer or to any other technology or technical field, fail to apply the exception with a particular machine, fail to apply the judicial exception to effect a particular treatment or prophylaxis for a disease or medical condition, fail to effect a transformation of a particular article to a different state or thing, and fail to apply/use the abstract idea in a meaningful way beyond generally linking the use of the judicial exception to a particular technological environment. Accordingly, this argument is found unpersuasive.
7. Applicant submits “The Office Action asserts the claims are directed to methods of organizing human activity. However, when the currently amended claims are considered as a whole, the recited limitations extend beyond a generalized description of sales activity and instead recite specific requirements governing how a computing system dynamically generates customized follow-up survey inquiries in real-time utilizing a multimodal model. In light of Ex Parte Desjardins and the technical advantages recited in the Specification (and the claims), Applicant respectfully disagrees, as the currently amended independent claims recite a complex series of steps that improve the functioning of a computer system for generating predictive outputs.” [Applicant’s Remarks, 07/23/2026, page 13]
The Examiner respectfully disagrees. In response to Applicant’s argument that “The Office Action asserts the claims are directed to methods of organizing human activity. However, when the currently amended claims are considered as a whole, the recited limitations extend beyond a generalized description of sales activity and instead recite specific requirements governing how a computing system dynamically generates customized follow-up survey inquiries in real-time utilizing a multimodal model,” it is noted that the claim remains directed to the fundamental business practice of conducting and managing surveys by collecting responses, evaluating those responses, and tailoring subsequent question based on the evaluation. These steps constitute managing indications with respondents to obtain information, which is a form of commercial or business interacting and therefore falls withing the “Certain Methods of Organizing Human Activity” grouping of abstract ideas.
Furthermore, while Applicant cites Ex Parte Desjardins, the rejection fully considers whether the additional elements of claim 1, including the at least one processor, at least one non-transitory computer-readable storage medium storing instructions, the system, a respondent client device, and a multimodal model integrate the abstract idea into a practical application. The claims recites generic computing components performing conventional data processing tasks without specifying a particular improvement in the functioning of the computer. Consistent with MPEP 2106.04(d)(1), 2106.05(a), and 2106.05(f), merely linking conventional computer elements to an abstract idea does not transform the claim into a patent eligible invention. Accordingly, when considered as a whole the claim does not recite a technological improvement under Ex Parte Desjardins and remains directed to an abstract idea. Applicant should amend the claim to clearly recite how the claim elements improve the functioning of the computer or solve a specific technological problem. In Ex Parte Desjardins, the Board found eligibility where the claims and the Specification together describes a particularized technological improvement and the claims reflect how the improvement was achieved. In contrast, the present claim recites only results oriented steps (i.e., receive, generate, generate, provide) without specifying any concrete technological mechanism or change in how a computer operates.
Moreover, although Applicant asserts that the Specification describes improvements, these alleged improvements are not meaningfully reflected in the claim language. Claim 1 does not recite any specific model or algorithm, that would demonstrate an improvement to a technical field, but instead broadly invokes generic processing to achieve the desired outcomes. Therefore, the claim does not integrate the abstract idea into a practical application, nor does it reflect a technological improvement comparable to that in Ex Parte Desjardins. For the reasons above, this argument is found unpersuasive.
8. Applicant submits “The currently amended independent claims recite a technological solution to a technical problem, and in light of McRo, implement any alleged abstract idea into a practical application. As stated in the stated in the MPEP at 2106.04(II)(A)(2) for Step 2A2, a "claim is not 'directed to' a judicial exception, and thus is patent-eligible, if the claim as a whole integrates the recited judicial exception into a practical application of that exception." Specifically, a "claim that integrates a judicial exception into a practical application will apply, rely on, or use the judicial exception in a manner that imposes a meaningful limit on the judicial exception, such that the claim is more than a drafting effort designed to monopolize the judicial exception." MPEP 2106.04(d). Thus, under the second prong of revised Step 2A, the inquiry requires determining whether a recited judicial exception is integrated into a practical application.” [Applicant’s Remarks, 07/23/2026, page 16]
In response to the Applicant’s argument, the Examiner respectfully disagrees. With respect to Applicant's comparison to McRO, Examiner points out that the claims in McRO involved a method for automatically animating lip synchronization and facial expression of three-dimensional characters comprising: obtaining a first set of rules that defines a morph weight set stream as a function of phoneme sequence and times associated with said phoneme sequence; obtaining a plurality of sub-sequences of timed phonemes corresponding to a desired audio sequence for said three-dimensional characters; generating an output morph weight set stream by applying said first set of rules to each sub-sequence of said plurality of sub-sequences of timed phonemes. The claims at issue are far different from the claims in McRO. The claims of the present case involve a method and system for generating a customized follow-up survey inquiry to a respondent. The claims of the instant application do not recite techniques for automatically generating three-dimensional facial expressions matching a prerecorded track of speech. Second, it is noted that the claims in McRO recited a specific asserted improvement in computer animation. In contrast, the claim here is not directed to any improvement in computer functionalities/capabilities - only to provide a customized follow-up survey inquiry to a respondent. The focus of the invention is on the algorithms that have been identified as abstract ideas (as opposed to an improvement to operations of the additional elements, improvement to another technology or technical field). The claims of the instant application thus cannot be characterized as an improvement in computer technology.
Further, contrary to the claims in the McRO decision, the additional elements of the instant application are not reliant on the programmed rules to improve intrinsic operations of the additional elements themselves. In McRO, the rules were deemed to allow the additional elements to accomplish a technical feature presumably not accomplished before. The Examiner points out that the claims in McRO did not simply provide a particular solution to a problem, but the claimed invention in McRO was deemed to provide an improvement in the technology. Again, Applicant’s claimed invention does not provide an improvement in the technology.
It is further noted that the Court in McRO said an improvement in computer-related technology is not limited in the operation of a computer or a computer network per se, but may also be claimed as a set of “rules” basically mathematical relationships that improve computer-related technology by allowing the computer performance of a function not previously performed by a computer. The instant Specification and claims are devoid of any indication that the claimed invention is to a “set of rules (basically mathematical relationships) that improve computer-related technology". Therefore, the Office finds that the concept present in the McRO is not analogous to the instant claimed invention. Based on the foregoing explanation the Examiner finds that the claims are not like those of McRO Applicant contends, because there is no expressed or implied improvement to a computer-related technology. The claims are not “directed to a specific improvement to the way computers operate.” For the reasons above, this argument is found unpersuasive.
Moreover, in response to Applicant’s argument that “claim that integrates a judicial exception into a practical application will apply, rely on, or use the judicial exception in a manner that imposes a meaningful limit on the judicial exception, such that the claim is more than a drafting effort designed to monopolize the judicial exception,” it is noted that preemption is not a standalone test for patent eligibility. Preemption concerns have been addressed by the Examiner through the application of the two-step framework. Applicant’s attempt to show that the recited abstract idea is a specific one is not persuasive. A specific abstract idea is still an abstract idea and is not eligible for patent protection without significantly more recited in the claim. See the July 2015 Update: Subject Matter Eligibility that explains that questions of preemption are inherent in the two-part framework from Alice Corp and Mayo and are resolved by using this framework to distinguish between preemptive claims, and “those that integrate the building blocks into something more…the latter pose no comparable risk of preemption, and therefore remain eligible.” The absence of complete preemption does not guarantee the claim is eligible. Therefore, “[w]here a patent’s claims are deemed only to disclose patent ineligible subject matter under the Mayo framework, as they are in this case, preemption concerns are fully addressed and made moot.” Ariosa Diagnostics, Inc. v. Sequenom, Inc., 788 F.3d 1371, 1379 (Fed. Cir. 2015). See also OIP Tech., Inc. v. Amazon.com, Inc., 788 F.3d 1359, 1362-63 (Fed Cir. 2015). Accordingly, this arguments is found unpersuasive.
For the reasons above, in addition to the reasons provided in the updated §101 rejection below, Applicant’s amendment and supporting arguments are not sufficient to overcome the §101 rejection.
9. Applicant submits “Jain, whether considered singly or in combination with the other cited references, fails to describe, teach, or suggest each limitation recited by independent claims 1, 10, and 18. For example, Jain, whether considered singly or in combination with the other cited references, fails to describe, teach, or suggest generating, by providing a prompt comprising the survey response data to a multimodal model, a response quality classification of the survey response data reflecting a sentiment classification and a topic classification of the survey response data" and "generating a customized follow-up survey inquiry based on the response quality classification reflecting the sentiment classification and the topic classification, the survey inquiry, and the survey response data." as recited by currently amended independent claim 18 and as similarly recited by currently amended independent claims 1 and 10.” [Applicant’s Remarks, 07/23/2026, page 19]
In response to Applicant’s argument that “Jain, whether considered singly or in combination with the other cited references, fails to describe, teach, or suggest each limitation recited by independent claims 1, 10, and 18. For example, Jain, whether considered singly or in combination with the other cited references, fails to describe, teach, or suggest generating, by providing a prompt comprising the survey response data to a multimodal model, a response quality classification of the survey response data reflecting a sentiment classification and a topic classification of the survey response data" and "generating a customized follow-up survey inquiry based on the response quality classification reflecting the sentiment classification and the topic classification, the survey inquiry, and the survey response data." as recited by currently amended independent claim 18 and as similarly recited by currently amended independent claims 1 and 10,” it is noted that this argument is a mere allegation of patentability by the Applicant with no supporting rationale or explanation. Merely stating that the claims do not teach a feature does not offer any insight as to why the specific sections of the prior art relied upon by the Examiner fail to disclose the claimed features. Applicant's arguments amount to a general allegation that the claims define a patentable invention without specifically pointing out how the language of the claims patentably distinguishes them from the references. Moreover, the Examiner notes the limitations being argued by Applicant as being newly amended to the claims in the response filed 07/23/2026, which has been addressed in the updated rejection below. Applicant’s argument has been considered, but it pertains to amendments to independent claims 1/10/18 that are believed to be addressed via the new ground of rejection under §103 set forth in the instant Office action, which incorporates a new reference and citations to address the amended limitations in claims 1/10/18 and supports a conclusion of obviousness of the amended claims.
10. Applicant submits “identifying poor response rates among population segments is not the same as generating "a response quality classification of the survey response data reflecting a sentiment classification and a topic classification of the survey response data" and "a customized follow-up survey inquiry based on the survey response quality classification reflecting the sentiment classification and the topic classification, the survey inquiry, and the survey response data," as recited by currently amended independent claim 18 and as similarly recited by currently amended independent claims 1 and 10.” [Applicant’s Remarks, 07/23/2026, pages 19-20]
In response to Applicant’s argument that “identifying poor response rates among population segments is not the same as generating "a response quality classification of the survey response data reflecting a sentiment classification and a topic classification of the survey response data" and "a customized follow-up survey inquiry based on the survey response quality classification reflecting the sentiment classification and the topic classification, the survey inquiry, and the survey response data," as recited by currently amended independent claim 18 and as similarly recited by currently amended independent claims 1 and 10,” the Examiner notes the limitations being argued by Applicant as being newly amended to the claims in the response filed 07/23/2026, which has been addressed in the updated rejection below. Applicant’s argument has been considered, but it pertains to amendments to independent claims 1/10/18 that are believed to be addressed via the new ground of rejection under §103 set forth in the instant Office action, which incorporates new reference and citations to address the amended limitations in claims 1/10/18 and supports a conclusion of obviousness of the amended claims.
11. Applicant’s remaining arguments either logically depend from the above-rejected arguments, in which case they too are unpersuasive for the reasons set forth above, or they are directed to features which have been newly added via amendment. Therefore, this is now the Examiner's first opportunity to consider these limitations and as such any arguments regarding these limitations would be inappropriate since they have not yet been examined. A full rejection of these limitations will be presented later in this Office Action.
Claim Rejections - 35 USC § 101
12. 35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
13. Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more.
14. Claims 1-20 are rejected under 35 U.S.C. 101 because the claimed invention is directed to an abstract idea without significantly more. The eligibility analysis in support of these findings is provided below, in accordance with the MPEP 2106.
With respect to Step 1 of the eligibility inquiry (as explained in MPEP 2106), it is first noted that the system (claims 1-9), non-transitory computer-readable medium (claims 10-17), and method (claims 18-20), are directed to at least one potentially eligible category of subject matter (machine, article of manufacture, and process, respectively), and therefore claims 1-20 satisfy Step 1 of the eligibility inquiry.
With respect to Step 2A Prong One, it is next noted that the claims recite an abstract idea that falls under the “Certain methods of organizing human activity” group within the enumerated groupings of abstract ideas set forth in MPEP 2106 since the claims set forth steps for managing interactions between people including by following rules or instructions, and also by setting forth steps for managing commercial interactions (e.g., marketing or sales activities or behaviors; business relations). With respect to independent claim 1, the limitations reciting the abstract idea are indicated in bold below: at least one processor; and at least one non-transitory computer-readable storage medium storing instructions that, when executed by the at least one processor, cause the system to: receive, from a respondent client device during administration of a digital survey, survey response data comprising a natural language response to a survey inquiry of the digital survey; generate, by providing a prompt comprising the survey response data to a multimodal model, a response quality classification of the survey response data reflecting a sentiment classification and a topic classification of the survey response data; generate a customized follow-up survey inquiry based on the response quality classification reflecting the sentiment classification and the topic classification, the survey inquiry and the survey response data; and provide, during the administration of the digital survey, the customized follow-up survey inquiry via the respondent client device. These steps describe concepts related to organizing human activity because they involve structuring how survey responses are processed and used to generate follow-up questions, which is a method of managing human interactions with data. Because the above-noted limitations recite steps falling within the Certain methods of organizing human activity abstract idea grouping of MPEP 2106, they have been determined to recite at least one abstract idea when evaluated under Step 2A Prong One of the eligibility inquiry. Independent claims 10 and 18 recite similar limitations as the above-noted limitations recited in claim 1 and are therefore found to recite the same abstract idea.
Therefore, because the limitations above set forth activities falling within the “Certain methods of organizing human activity” abstract idea grouping described in MPEP 2106, the additional elements recited in the claims are further evaluated, individually and in combination, under Step 2A Prong Two and Step 2B below.
With respect to Step 2A Prong Two, the judicial exception is not integrated into a practical application. Independent claims 1, 10, and 18 recite the additional elements of at least one processor, at least one non-transitory computer-readable storage medium storing instructions, the system, a respondent client device, and a multimodal model (claim 1), a non-transitory computer-readable medium storing instructions, at least one processor, a computer device, a respondent client device, and a multimodal model (claim 10) a respondent client device and a multimodal model (claim 18). These additional elements have been evaluated, but fail to integrate the abstract idea into a practical application because they amount to using generic computing elements or instructions (software) to perform the abstract idea, similar to adding the words “apply it” (or an equivalent), which merely serves to link the use of the judicial exception to a particular technological environment (network computing environment). See MPEP 2106.05(f) and 2106.05(h). In addition, these limitations fail to provide an improvement to the functioning of a computer or to any other technology or technical field, fail to apply the exception with a particular machine, fail to apply the judicial exception to effect a particular treatment or prophylaxis for a disease or medical condition, fail to effect a transformation of a particular article to a different state or thing, and fail to apply/use the abstract idea in a meaningful way beyond generally linking the use of the judicial exception to a particular technological environment.
Accordingly, because the Step 2A Prong One and Prong Two analysis resulted in the conclusion that the claims are directed to an abstract idea, additional analysis under Step 2B of the eligibility inquiry must be conducted in order to determine whether any claim element or combination of elements amount to significantly more than the judicial exception.
With respect to Step 2B of the eligibility inquiry, it has been determined that the claims do not include additional elements that are sufficient to amount to significantly more than the judicial exception. Independent claims 1, 10, and 18 recite the additional elements of at least one processor, at least one non-transitory computer-readable storage medium storing instructions, the system, a respondent client device, and a multimodal model (claim 1), a non-transitory computer-readable medium storing instructions, at least one processor, a computer device, a respondent client device, and a multimodal model (claim 10) a respondent client device and a multimodal model (claim 18). These additional elements have been evaluated, but fail to add significantly more to the claims because they amount to using generic computing elements or instructions (software) to perform the abstract idea, similar to adding the words “apply it” (or an equivalent), which merely serves to link the use of the judicial exception to a particular technological environment (network computing environment) and does not amount to significantly more than the abstract idea itself. Notably, Applicant’s Specification suggests that virtually any computing device(s) under the sun may be used to implement the invention, including generic computers (Specification at paragraphs 0041, 0132). Therefore, the additional elements merely describe generic computing elements or computer-executable instructions (software) merely serve to tie the abstract idea to a particular operating environment, which does not add significantly more to the abstract idea. See, e.g., Alice Corp., 134 S. Ct. 2347, 110 USPQ2d 1976; Versata Dev. Group, Inc. v. SAP Am., Inc., 793 F.3d 1306, 1334, 115 USPQ2d 1681, 1701 (Fed. Cir. 2015).
In addition, even if the multimodal model was evaluated as an element beyond software/code for a generic computer to execute, it is noted that that the claimed multimodal model is recited at a high level of generality these elements amount to well-understood, routine, and conventional activity in the art, which fails to add significantly more to the claims. See, e.g., Panda et al., US 2022/0292285 A1 (paragraph 0002: “Most conventional deep multimodal models focus on how to fuse information from multiple data modalities”).
In addition, when taken as an ordered combination, the ordered combination adds nothing that is not already present as when the elements are taken individually. There is no indication that the combination of elements integrate the abstract idea into a practical application. Their collective functions merely provide generic computer implementation. Therefore, when viewed as a whole, these additional claim elements do not provide meaningful limitations to transform the abstract idea into a practical application of the abstract idea or that, as an ordered combination, amount to significantly more than the abstract idea itself.
Dependent claims 2-9, 11-17, and 19-20 recite the same abstract ideas as recited in the independent claims, and have been found to either recite additional details that are part of the abstract idea itself (when analyzed under Step 2A Prong One) along with, at most, additional elements that fail to integrate the abstract idea into a practical application or add significantly more. In particular, dependent claims 2-9, 11-17, and 19-20 further narrow the abstract ideas recited in independent claims 1, 10, and 18 by reciting limitations that fall under the “Certain methods of organizing human activity” abstract idea grouping. For example, dependent claims 2-9, 11-17, and 19-20 recite “generate the customized follow-up survey inquiry in response to determining that the response quality classification of the survey response data indicates a trigger to generate the customized follow-up survey inquiry,” “determine, during creation of the digital survey, one or more natural language prompts defining one or more triggers for customized follow-up survey inquiries based on response quality classifications; and provide the one or more natural language prompts defining the one or more triggers based on the response quality classifications to the multimodal model during creation of the digital survey and survey respondent data,” “generate the response quality classification based on response quality classifications comprising a categorization of completeness, actionability, sentiment, or subject matter according to the one or more triggers for customized follow-up survey inquiries,” “generate the response quality classification based on determining that the survey response data does not include an excluded category according to the one or more triggers for customized follow-up survey inquiries,” “generate the response quality classification of the survey response data by: providing a prompt comprising the survey response data to the multimodal model; receiving, from the multimodal model, a follow-up score; and utilizing the follow-up score to generate the response quality classification of the survey response data,” “wherein utilizing the follow-up score to generate the response quality classification comprises determining, from the follow-up score, a number of triggers indicated by the survey response data and a degree to which the survey response data indicates the number of triggers,” “receive a follow-up response to the customized follow-up survey inquiry; determine a summary for the follow-up response and one or more follow-up responses to one or more additional follow-up survey inquiries; and generate a survey report comprising the summary for the follow-up response and one or more follow-up responses to the one or more additional follow-up survey inquiries,” “predicting customized follow-up survey inquiries for a training set of survey response data; and modifying the multimodal model based on comparing the predicted customized follow-up survey inquiries with ground truth customized follow-up survey inquiries to reduce or minimize a loss of a loss function,”; however, these steps recite limitations that fall within the “certain methods of organizing human activity.” Claim 11-17 and 19-20 have been evaluated as well, however these claims, similarly to dependent claims 2-9, recite details that fall within the scope of the abstract idea itself by providing limitations that narrow the same abstract idea recited in the independent claims accompanied by the same generic computing elements or software as those addressed above in the discussion of the independent claims. The additional elements recited in the dependent claims include an administrator device (claims 3, 12, and 20) and iteratively train the multimodal model (claims 9 and 17). However, each of these elements is recited at a high level of generality and fails to yield any discernible improvement to the computer or to any technology, nor set forth any additional function or result that provided meaningful limitation beyond linking the abstract idea to a particular technological environment (i.e., automated/computing environment), and thus fail to integrate the abstract idea into a practical application.
In addition, even if the training was evaluated as elements beyond software/code for a generic computer to execute, it is noted that that the claimed training is recited at a high level of generality these elements amount to well-understood, routine, and conventional activity in the art, which fails to add significantly more to the claims. See, e.g., Pulis et al., US 12,182,678 (col. 3, lines 63-67 & col. 4, lines 1-19: “systems and methods for aligning large multimodal models (LMMs) or large language models (LLMs) with domain-specific principles using a well-known transformer architecture wherein the LMM or LLM may be post-trained and/or fine-tuned during training using instructions and training data using various alignment and instruction generation elements and processes”). See also, Tanjim, US 2025/0225683 (paragraph 0054: “conventional image generation models are trained on a diverse set of image and text pairs”).
The ordered combination of elements in the dependent claims (including the limitations inherited from the parent claim(s)) add nothing that is not already present as when the elements are taken individually. There is no indication that the combination of elements improves the functioning of a computer or improves any other technology. Their collective functions merely provide generic computer implementation. Accordingly, the subject matter encompassed by the dependent claims fails to amount to a practical application or significantly more than the abstract idea itself.
For more information, see MPEP 2106.
Claim Rejections - 35 USC § 103
15. In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
16. The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
17. The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
18. This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
19. Claims 1-7, 10-15, and 18-20 are rejected under 35 U.S.C. 103 as being unpatentable over Jain et al., Patent No.: US 11,789,837 B1, [hereinafter Jain], in view of Williams et al., Pub. No.: US 2014/0316856 A1, [hereinafter Williams], in further view of Jeffs et al., Pub. No.: US 11,568,420 B2, [hereinafter Jeffs].
As per claim 1, Jain teaches a system (col. 14, lines 63-67 & col. 15, lines 1-7) comprising:
at least one processor (col. 286, lines 11-54: “Embodiments of the invention and all of the functional operations described in this specification can be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the invention can be implemented as one or more computer program products, e.g., one or more modules of computer program instructions encoded on a computer readable medium for execution by, or to control the operation of, data processing apparatus. The computer readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a composition of matter effecting a machine-readable propagated signal, or a combination of one or more of them. The term “data processing apparatus” encompasses all apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them…”); and
at least one non-transitory computer-readable storage medium storing instructions that, when executed by the at least one processor, cause the system (col. 286, lines 11-54: “Embodiments of the invention and all of the functional operations described in this specification can be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the invention can be implemented as one or more computer program products, e.g., one or more modules of computer program instructions encoded on a computer readable medium for execution by, or to control the operation of, data processing apparatus. The computer readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a composition of matter effecting a machine-readable propagated signal, or a combination of one or more of them. The term “data processing apparatus” encompasses all apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them…”) to:
receive, from a respondent client device during administration of a digital survey, survey response data comprising a natural language response to a survey inquiry of the digital survey (col. 8, lines 66-67 & col. 9, lines 1-3, discussing distributing, to the one or more devices, a software module or configuration data that cause the one or more devices to initiate collection of the second types of data; col. 9, lines 10-13, discussing that the second types of data comprise user inputs as responses to surveys [i.e., This shows receiving survey response data] provided as part of the monitoring program; col. 190, lines 35-60, discussing that the process includes accessing data describing a monitoring program, for example, one of many that is supported or managed by a server system. Often, a server system can manage and monitor many different monitoring programs with different monitoring groups, with different procedures and data collection being done for each…The monitoring program can be a research study such as a clinical trial. For example, digital clinical trials typically involve participants that perform measurements and provide survey responses with devices… Through user interactions with an application on a device, the data needed for clinical trials can be effectively and efficiently collected; col. 191, lines 12-42, discussing that the devices provide user inputs received (such as survey responses from users)...The data collection requirements for a monitoring program are stored in a profile or data set for monitoring program and can include more than just the types of data to be collected. For example, the requirements can specify the data quality needed (e.g., accuracy, precision, reliability, etc.), the rate or frequency of data collection needed, the total amount of data needed; col. 195, lines 35-48);
generate, by providing a prompt comprising the survey response data to a multimodal model, a response quality classification of the survey response data (col. 5, lines 16-45, discussing that the system may detect that the accuracy of data received from the monitoring group fails to meet a minimum data quality threshold or is trending below the data quality threshold. An analysis of the collected data and subgroups of devices can reveal that the data collected from the particular subgroup of devices is consistently low quality and, in response, prioritize the particular subgroup of devices above the other subgroups; col. 13, lines ML model to determine quality; col. 13, lines 42-63, discussing that the prediction for the group is generated based on output of one or more machine learning models trained to predict likelihoods of at least one of compliance, retention, or data quality levels, the one or more machine learning models being trained using training data that includes (i) attributes of different individuals enrolled in monitoring programs that have different requirements for participants, (ii) the requirements for participant actions of the monitoring programs, and (iii) observed compliance results for the individuals, and the one or more machine learning models are trained to provide output indicating, for a group or individual, a likelihood or rate of enrollment, compliance, retention, or a minimum data quality level in response to receiving input feature data indicating (i) attributes of individuals or of groups and (ii) data indicating requirements of a monitoring program; col 14, lines 52-62- The various models discussed herein can be machine learning models, for example, a neural networks or classifiers…; col. 86, lines 43-67 & col. 87, lines 1-5, discussing that the computer system may use various factors to determine data quality or which may be used to set minimum data quality requirements. These factors can include response times (e.g., where relatively quick response times and/or relatively long response times may correspond to low data quality),…, response content (e.g., text input that is relatively short or that is below a threshold word count may correspond to low data quality; text input that is relatively long or that is above a threshold word count may correspond to high data quality; etc.), etc. The computer system may use one or more algorithms to determine a data quality score or a data quality rate; col. 195, lines 35-48, discussing that the process incudes monitoring performance of the different devices with respect
to the data collection requirements for the monitoring program. The monitoring can include receiving and assessing data from each of the different devices and users in a monitoring group. The data collected from a device can include the values of measured items (e.g., survey responses, user interaction data, etc.), as well as metadata, context data, and other related information. The system can analyze the received information to characterize the monitoring that has been performed; col. 198, lines 37-67, discussing that based on the training data examples, the system may train a classifier, neural network, decision tree, or other machine learning model to generate values indicating predicted outcomes such as quality of data, rate of data collection, consistency of data collection, likelihood of completing a specific data collection requirement or combination of requirements, and so on. These can be predicted generally, for different types of data, for different data sources or collection procedures (e.g., surveys), and so on. As a result, based on the training, the machine learning model is configured to (i) receive input feature values, and (ii) output in response a predicted value indicating the predicted amount of data collection, quality of collected data,…, and so on...; col. 145, lines 12-20);
generate a customized follow-up survey action based on the survey inquiry and the survey response data (col. 148, lines 8-22, discussing that each time a researcher loads an interface, or when additional data is collected, the system can perform the process with updated data. The system can have certain thresholds or criteria for initiating notifications and corrective actions. For example, the system 210 may have a threshold to notify researchers when the probability that a study will meet its diversity goals falls below 80%. This can trigger notifications on a user interface when a user logs in, notifications pushed to a device or sent through email or text message, or other forms; col. 149, lines 20-31, discussing that the changes that the system identifies to improve compliance can be made by the system automatically, e.g., to preemptively increase survey reminders to members of G3 when the system determines that the predicted survey response compliance rate is low. In some implementations, changes can be recommended to a researcher or other administrator and performed in response to receiving confirmation; col. 201, lines 7-23, discussing that the system may also provide recommendations of interactions to individuals...The system may require authorization or approval of some adjustments, and may provide alerts or user interface data to inform the researchers and enable them to indicate approval to trigger the system to carry out the interventions. Similarly, some changes to interaction with participants in a monitoring program, including changes to devices of the users, may be recommended or requested of the users; col. 202, lines 46-67 & col. 203, lines 1-3, discussing that the system can store a database or other repository of different interventions, with different ways that a monitoring program can be adjusted to correct for different types of poor performance. For example, the system can store a library of different interventions and corresponding conditions that can correct for the different indictors or causes of poor monitoring performance. For failure to complete a survey, the library can indicate sending reminders to the user, increasing the frequency that the survey is taken, changing the font size and layout of the survey, breaking the survey into multiple shorter interactions spaced apart over time (e.g., in the case that user has started a survey but does not complete it), and so on; col. 203, lines 4-36, discussing that the repository can store data indicating how effective different interventions have to improve data collection for different types of data, for users of different attributes,…, and so on. With this effectiveness data, the system can personalize the adjustments it makes by selecting, for each high-priority device or user, the intervention predicted to be most likely to improve monitoring performance and/or predicted to provide the greatest amount of improvement in monitoring performance. In some implementations, the system can train, store, and use a machine learning model trained to predict which interventions are best suited for different user types and/or situations. For example, the model can receive input feature values indicating attributes of a device and its user, as well as potentially values characterizing the past monitoring performance or the problem with monitoring performance that has occurred (e.g., the level of data quality that was achieved, etc.). The model can then output one or more scores, for each of various different interventions, indicating the likelihood of improvement and/or magnitude of improvement that is expected in monitoring performance if the intervention is implemented. From these scores, the system can select one or more interventions that are predicted to best improve monitoring performance for an individual user, for the areas of monitoring that the device or user is most in need of improvement; col. 203, lines 50-67, discussing that the system has a range of interventions...These interventions include communications to cause devices to change their configurations, settings,…, and so on to improve the rate and quality of survey responses. The interventions also include changes to user interfaces as well as sending reminders, notifications, alerts, and other communications to users); and
provide, during the administration of the digital survey, the customized follow-up action inquiry via the respondent client device (col. 202, lines 46-67 & col. 203, lines 1-3, discussing that the system can store a database or other repository of different interventions, with different ways that a monitoring program can be adjusted to correct for different types of poor performance. For example, the system can store a library of different interventions and corresponding conditions that can correct for the different indictors or causes of poor monitoring performance. For failure to complete a survey, the library can indicate sending reminders to the user, increasing the frequency that the survey is taken, changing the font size and layout of the survey, breaking the survey into multiple shorter interactions spaced apart over time (e.g., in the case that user has started a survey but does not complete it), and so on; col. 203, lines 50-67, discussing that the system has a range of interventions...These interventions include communications to cause devices to change their configurations, settings,…, and so on to improve the rate and quality of survey responses. The interventions also include changes to user interfaces as well as sending reminders, notifications, alerts, and other communications to users).
While Jain teaches a response quality classification of the survey response data and generate a customized follow-up action and reminder based on the survey inquiry and the survey response data, Jain does not explicitly teach a response quality classification of the survey response data reflecting a sentiment classification and a topic classification of the survey response data; generate a customized follow-up survey inquiry based on the response quality classification reflecting the sentiment classification and the topic classification, the survey inquiry and the survey response data; and provide, during the administration of the digital survey, the customized follow-up survey inquiry via the respondent client device. Williams in the analogous art of questionnaire generation systems teaches:
generate a customized follow-up survey inquiry based on the survey inquiry and the survey response data (paragraph 0002, discussing an analysis tool that assesses customer responses to an open-ended survey question and tailors additional questions based on deductive processing tools; paragraph 0018, discussing that it is desirable to ask follow-up questions that further explain a particular answer or, as noted further below, ask initial open-ended questions that drive the organization of the remainder of the survey. For example, if a customer initially rates product quality as low, it is useful to ask the customer what they did not like about quality; paragraph 0024, discussing that the facts extracted from customer comments through NLP technology are used to restructure a survey in real time. That is, the facts are used to dynamically create follow-up questions while the survey is being conducted. One example of an open-ended question and follow-up questions is presented below. In this example, a hotel manager desires to understand how customers rate their experience and what was most important to them. Rather than provide a long list of questions that may or may not be of concern to the consumer, an open-ended question is posed to provide the consumer with the opportunity to identify those areas that were most important to the consumer. Follow-up questions are generated in response to the facts and topics of interest generated by the consumer as well as topics of interest generated by the user of the model; paragraph 0035, discussing that an analysis of the topics identified in consumer responses is performed to assess the confidence in the results of identified topics. Closed-ended follow-up questions are asked regarding topics that were identified by the user as important, but which did not appear in the comment or for which the confidence interval is low. Specific rating questions directed towards those topics identified, narrow and refine the rating process to those areas specifically addressed by the consumer and which matter to the consumer. A set of business rules provided by the user is used to ask additional questions based on the specific ratings and/or additional goods/services for which the user desires specific information; paragraph 0020); and
provide, during the administration of the digital survey, the customized follow-up survey inquiry via the respondent client device (paragraph 0027, discussing that based on the consumer response, the following facts might be extracted from the comment using NLP or computational linguistics: The consumer had a poor experience; the bell service was slow; the employees were rude; room service was poor; price was mentioned; and expectations were not met. Based on the facts extracted from the comment, the system generates a set of closed-ended follow-up questions in order to obtain ratings with respect to the specific items referenced. Additionally, specific questions from the system user wishes asked as part of the survey are included to rate user-generated topics in addition to the consumer-generated topics of interest. The following is one example set of follow-up questions: [0028] 1. Computer: "I'm sorry you had a poor experience. We'd like to do better. Can you rate your experience on a scale of 1 to 5?"; paragraph 0038, discussing that if the confidence interval regarding consumer sentiment of the cleanliness of the parking lot was within an acceptable range, follow up questions ranking the parking lot, for example, would not need to be asked. A rating could be assigned based on the textual analytics of the response. For example, if the consumer said "the parking lot was filthy" a value of 1 may be assigned to the consumer rating without the need of a specific follow-up question; paragraph 0039).
Jain is directed towards systems and methods for adaptive data collection. Williams is directed towards a method for conducting real-time dynamic consumer surveys. Therefore, they are deemed to be analogous as they both are directed towards survey generation and analysis. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Jain with Williams because the references are analogous art because they are both directed to solutions for survey analysis, which falls within applicant’s field of endeavor (survey systems), and because modifying Jain to include Williams’ features for including generating a customized follow-up survey inquiry based on the survey inquiry and the survey response data, and provide, during the administration of the digital survey, the customized follow-up survey inquiry via the respondent client device, in the manner claimed, would serve the motivation of improving the value of information harvested from consumer responses (Williams at paragraph 0018); and further obvious because the claimed invention is merely a combination of old elements, and in the combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that the results of the combination were predictable.
The Jain-Williams combination does not explicitly teach a response quality classification of the survey response data reflecting a sentiment classification and a topic classification of the survey response data; and generate a customized follow-up survey inquiry based on the response quality classification reflecting the sentiment classification and the topic classification. However, Jeffs in the analogous art of systems for design, delivery, and analysis of customer feedback surveys teaches these concepts. Jeffs teaches:
a response quality classification of the survey response data reflecting a sentiment classification and a topic classification of the survey response data (col. 5, lines 48-59, discussing a system and a method for designing and analyzing customer feedback surveys. In embodiments of the system, survey content, structure, and delivery is dynamically adapted based upon identified nature or characteristics of an on-going or recently completed customer service interaction or a series of multiple customer service interactions across communication channels. By tailoring customer feedback surveys to content, topics, or sentiments found in one or more interactions with a customer, more responses and more meaningful responses can be obtained from customers; col. 6, lines 20-41, discussing that analytics can be used to identify topics, trends, content, or sentiment of the conversation. In embodiments, the analytics may apply an ontology or predefined set of rules or structures for interpreting the customer interaction content. The results of the analytics can be gleaned for all transcripts, metadata pulled from open responses of the customer, categorized spoken content, and indexing/frequency of words used in the conversation; col. 9, lines 4-16, discussing that the dynamic survey delivery may take into account metadata or identified sentiments from the conversation in order to identify if such a cooling off period is required and an appropriate survey format and delay may be selected for the customer feedback survey; col. 7, lines 24-52, discussing that the survey decision engine decides from the received customer interaction content that it is desirable that the customer be asked survey questions that are not represented in a standard survey. In such an embodiment, based upon specifically identified information or events in the conversation, more specific or detailed survey questions are required. In this event, the survey decision engine can select questions from the stored survey questions in order to create a customized survey…The conversation information provided to the survey decision engine may identify that the customer was switching to a service of a known competitor and questions specifically about that competitor can be selected from survey questions to specialize the survey. These more specific questions may be able to draw more specific information based upon the customer statements during the recorded conversation. In a further non-limiting embodiment, the speech analytics of the conversation may identify that the customer calling in for service was highly frustrated with the product/service/organization. In this event, a customized survey may be created by the decision engine that has additional questions related to customer service/satisfaction. These additional questions may be in the form of a standardized customer satisfaction survey stored or individualized customer survey questions that may be specifically tailored to a particular product or service selected from); and
generate a customized follow-up survey inquiry based on the response quality classification reflecting the sentiment classification and the topic classification (col. 5, lines 48-59, discussing a system and a method for designing and analyzing customer feedback surveys. In embodiments of the system, survey content, structure, and delivery is dynamically adapted based upon identified nature or characteristics of an on-going or recently completed customer service interaction or a series of multiple customer service interactions across communication channels. By tailoring customer feedback surveys to content, topics, or sentiments found in one or more interactions with a customer, more responses and more meaningful responses can be obtained from customers; col. 7, lines 3-23, discussing that speech or textual recognition is used to identify specific topics, events, sentiment and metadata from a conversation between a customer and a customer service agent…; col. 7, lines 24-52, discussing that the survey decision engine decides from the received customer interaction content that it is desirable that the customer be asked survey questions that are not represented in a standard survey. In such an embodiment, based upon specifically identified information or events in the conversation, more specific or detailed survey questions are required. In this event, the survey decision engine can select questions from the stored survey questions in order to create a customized survey…The conversation information provided to the survey decision engine may identify that the customer was switching to a service of a known competitor and questions specifically about that competitor can be selected from survey questions to specialize the survey. These more specific questions may be able to draw more specific information based upon the customer statements during the recorded conversation. In a further non-limiting embodiment, the speech analytics of the conversation may identify that the customer calling in for service was highly frustrated with the product/service/organization. In this event, a customized survey may be created by the decision engine that has additional questions related to customer service/satisfaction. These additional questions may be in the form of a standardized customer satisfaction survey stored or individualized customer survey questions that may be specifically tailored to a particular product or service selected from).
The Jain-Williams combination describes features related to data collection and questionnaire generation. Jeffs is directed towards systems for design, delivery, and analysis of customer feedback surveys. Therefore, they are deemed to be analogous as they both are directed towards survey analysis. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the Jain-Williams combination with Jeffs because the references are analogous art because they are both directed to solutions for survey analysis, which falls within applicant’s field of endeavor (survey systems), and because modifying the Jain-Williams combination to include Jeffs’ features for including a response quality classification of the survey response data reflecting a sentiment classification and a topic classification of the survey response data, and generate a customized follow-up survey inquiry based on the response quality classification reflecting the sentiment classification and the topic classification, in the manner claimed, would serve the motivation of providing a better picture or understanding of the customer feedback (Jeffs at col. 9, lines 49-51); and further obvious because the claimed invention is merely a combination of old elements, and in the combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that the results of the combination were predictable.
As per claim 2, the Jain-Williams-Jeffs combination teaches the system of claim 1. Jain further describes further comprising instructions that, when executed by the at least one processor, cause the system to generate the customized follow-up survey action in response to determining that the response quality classification of the survey response data indicates a trigger to generate the customized follow-up survey action (col. 14, lines 19-29, discussing that the monitoring program is a proposed adapted monitoring program that the one or more computers generated from records for a primary monitoring program that is in progress; the group of candidates includes participants in the primary research study; the method comprises determining whether to recommend or implement the adaptations of the proposed adapted monitoring program based on evaluation of the one or more scores indicative of whether the primary monitoring program will satisfy one or more predetermined conditions; col. 14, lines 30-32, discussing that the evaluation of the one or more scores comprises comparing the one or more scores to predetermined thresholds; col. 148, lines 8-22, discussing that each time a researcher loads an interface, or when additional data is collected, the system can perform the process with updated data. The system can have certain thresholds or criteria for initiating notifications and corrective actions. For example, the system 210 may have a threshold to notify researchers when the probability that a study will meet its diversity goals falls below 80%. This can trigger notifications on a user interface when a user logs in, notifications pushed to a device or sent through email or text message, or other forms; col. 149, lines 20-31, discussing that the changes that the system identifies to improve compliance can be made by the system automatically, e.g., to preemptively increase survey reminders to members of G3 when the system determines that the predicted survey response compliance rate is low. In some implementations, changes can be recommended to a researcher or other administrator and performed in response to receiving confirmation; col. 201, lines 7-23, discussing that the system may also provide recommendations of interactions to individuals...The system may require authorization or approval of some adjustments, and may provide alerts or user interface data to inform the researchers and enable them to indicate approval to trigger the system to carry out the interventions. Similarly, some changes to interaction with participants in a monitoring program, including changes to devices of the users, may be recommended or requested of the users; col. 201, lines 51-67 & col. 202, lines 1-3, discussing that the prioritization of individual users can be used to adjust the order and timing in which interventions are provided. For example, individuals in higher-priority groups can have interventions initiated before those of other lower-priority groups...When the system determines that the need for performance improvement is low or would not significantly change the ability of the monitoring program to meet its diversity objectives, the system may determine not to make adjustments for those devices or user; col. 202, lines 46-67 & col. 203, lines 1-3, discussing that the system can store a database or other repository of different interventions, with different ways that a monitoring program can be adjusted to correct for different types of poor performance. For example, the system can store a library of different interventions and corresponding conditions that can correct for the different indictors or causes of poor monitoring performance. For failure to complete a survey, the library can indicate sending reminders to the user, increasing the frequency that the survey is taken, changing the font size and layout of the survey, breaking the survey into multiple shorter interactions spaced apart over time (e.g., in the case that user has started a survey but does not complete it), and so on; col. 203, lines 4-36, discussing that the repository can store data indicating how effective different interventions have to improve data collection for different types of data, for users of different attributes,…, and so on. With this effectiveness data, the system can personalize the adjustments it makes by selecting, for each high-priority device or user, the intervention predicted to be most likely to improve monitoring performance and/or predicted to provide the greatest amount of improvement in monitoring performance. In some implementations, the system can train, store, and use a machine learning model trained to predict which interventions are best suited for different user types and/or situations. For example, the model can receive input feature values indicating attributes of a device and its user, as well as potentially values characterizing the past monitoring performance or the problem with monitoring performance that has occurred (e.g., the level of data quality that was achieved, etc.). The model can then output one or more scores, for each of various different interventions, indicating the likelihood of improvement and/or magnitude of improvement that is expected in monitoring performance if the intervention is implemented. From these scores, the system can select one or more interventions that are predicted to best improve monitoring performance for an individual user, for the areas of monitoring that the device or user is most in need of improvement; col. 203, lines 50-67, discussing that the system has a range of interventions...These interventions include communications to cause devices to change their configurations, settings,…, and so on to improve the rate and quality of survey responses. The interventions also include changes to user interfaces as well as sending reminders, notifications, alerts, and other communications to users).
Jain does not explicitly teach the customized follow-up survey inquiry. However, Williams in the analogous art of questionnaire generation systems teaches this concept. Williams teaches:
further comprising instructions that, when executed by the at least one processor, cause the system to generate the customized follow-up survey inquiry in response to determining that the response quality classification of the survey response data indicates a trigger to generate the customized follow-up survey inquiry (paragraph 0002, discussing an analysis tool that assesses customer responses to an open-ended survey question and tailors additional questions based on deductive processing tools; paragraph 0018, discussing that it is desirable to ask follow-up questions that further explain a particular answer or, as noted further below, ask initial open-ended questions that drive the organization of the remainder of the survey. For example, if a customer initially rates product quality as low, it is useful to ask the customer what they did not like about quality; paragraph 0024, discussing that the facts extracted from customer comments through NLP technology are used to restructure a survey in real time. That is, the facts are used to dynamically create follow-up questions while the survey is being conducted. One example of an open-ended question and follow-up questions is presented below. In this example, a hotel manager desires to understand how customers rate their experience and what was most important to them. Rather than provide a long list of questions that may or may not be of concern to the consumer, an open-ended question is posed to provide the consumer with the opportunity to identify those areas that were most important to the consumer. Follow-up questions are generated in response to the facts and topics of interest generated by the consumer as well as topics of interest generated by the user of the model; paragraph 0035, discussing that an analysis of the topics identified in consumer responses is performed to assess the confidence in the results of identified topics. Closed-ended follow-up questions are asked regarding topics that were identified by the user as important, but which did not appear in the comment or for which the confidence interval is low. Specific rating questions directed towards those topics identified, narrow and refine the rating process to those areas specifically addressed by the consumer and which matter to the consumer. A set of business rules provided by the user is used to ask additional questions based on the specific ratings and/or additional goods/services for which the user desires specific information; paragraph 0037, discussing that if the calculated statistical confidence interval does not meet or exceed a user-defined threshold value on any of the topics identified, the system prompts the customer to confirm their perceived sentiment with a rating question. For example, if the textual analysis of the consumer comment resulted in a possible negative consumer sentiment with respect to a service, but the confidence interval was below that set by the user, a specific rating question would be generated and posed; paragraph 0020).
Jain is directed towards systems and methods for adaptive data collection. Williams is directed towards a method for conducting real-time dynamic consumer surveys. Therefore, they are deemed to be analogous as they both are directed towards survey generation and analysis. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Jain with Williams because the references are analogous art because they are both directed to solutions for survey analysis, which falls within applicant’s field of endeavor (survey systems), and because modifying Jain to include Williams’ feature for generating the customized follow-up survey inquiry in response to determining that the response quality classification of the survey response data indicates a trigger to generate the customized follow-up survey inquiry, in the manner claimed, would serve the motivation of improving the value of information harvested from consumer responses (Williams at paragraph 0018); and further obvious because the claimed invention is merely a combination of old elements, and in the combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that the results of the combination were predictable.
As per claim 3, the Jain-Williams-Jeffs combination teaches the system of claim 2. Although not explicitly taught by Jain, Williams in the analogous art of questionnaire generation systems teaches further comprising instructions that, when executed by the at least one processor, cause the system to: determine, from an administrator device during creation of the digital survey, one or more natural language prompts defining one or more triggers for customized follow-up survey inquiries based on response quality classifications (paragraph 0037, discussing that if the calculated statistical confidence interval does not meet or exceed a user-defined threshold value on any of the topics identified, the system prompts the customer to confirm their perceived sentiment with a rating question. For example, if the textual analysis of the consumer comment resulted in a possible negative consumer sentiment with respect to a service, but the confidence interval was below that set by the user, a specific rating question would be generated and posed; paragraph 0045, discussing that the quality of the prompts is improved by the use of industry/domain specific text analytics including user-defined business rules, taxonomies, dictionaries, ontologies, hierarchies, and the like. Moreover, the prompts can be driven by text analytics facts acquired from prior elements of the survey, including, but not limited to sentiment analysis, and previously collected structured data (ratings, context data, etc.); paragraph 0046, discussing that a "comment strength" indicator visually cues the consumer as to how useful the comment is. As many consumer surveys are linked to incentives for completing a survey, in one aspect, incentives may be increased for stronger consumer comments. In one aspect of the technology, the "submit" button enabling the consumer to complete the survey is not activated, until the "comment strength" indicator reaches an acceptable level); and
provide the one or more natural language prompts defining the one or more triggers based on the response quality classifications to the multimodal model during creation of the digital survey and survey respondent data (paragraph 0037, discussing that if the calculated statistical confidence interval does not meet or exceed a user-defined threshold value on any of the topics identified, the system prompts the customer to confirm their perceived sentiment with a rating question. For example, if the textual analysis of the consumer comment resulted in a possible negative consumer sentiment with respect to a service, but the confidence interval was below that set by the user, a specific rating question would be generated and posed; paragraph 0041, discussing that the method for conducting real-time dynamic consumer experience surveys comprises beginning the survey by asking the consumer to provide an overall rating of the experience and prompting the consumer with an open-ended question to explain why he or she provided the specific rating. The text of the consumer response is analyzed using natural processing language, for example, to ascertain consumer-identified topics that correlate to the experience score. Contextually sensitive questions are dynamically generated based on user-defined business rules. In one aspect of the technology, those rules are industry specific and are suited to specific business needs. A closed-ended question 68 related to specific topic identified by the consumer in his or response is proffered. In one aspect, the closed-ended question is derived from a set of user-defined rules that relate to the topic identified by the consumer. For example, if the consumer indicates they waited too long for a table, a closed-ended question 68 is posed that provides the consumer with an opportunity to provide a structured response; paragraph 0042, discussing that deductive survey logic is employed to improve the quality of the comments themselves, in addition to direct open-ended questions to the respondent, the system prompts the user to discuss additional points in their comment as they are typing. This can shorten a survey experience and improve the length and quality of the comment itself. Advantageously, this results in an improved survey experience for the respondent while simultaneously yielding better analytic data for analysis. In many situations, the respondent does not have much incentive to elaborate deeply. For example, if a respondent had a great overall experience, it is not uncommon for them to simply reply in their comment something similar to "everything was great." In aggregate, this scenario is so common that phrases like "everything was great" introduce noise into text analysis which presents a large problem to text analytics teams; paragraph 0043, discussing that if a respondent is asked, "Rate your experience on a scale of 1 to 5," and the respondent provides a rating of 5, they are asked in an open-ended question to, "Explain in your own words why you feel that way," They then reply that, "Everything was great." The system uses custom dictionaries and taxonomies to identify numerous ways of expressing that sentiment. Identifying that the respondent has replied with an overly simplistic phrase, the system prompts the respondent to enrich their comment).
Jain is directed towards systems and methods for adaptive data collection. Williams is directed towards a method for conducting real-time dynamic consumer surveys. Therefore, they are deemed to be analogous as they both are directed towards survey generation and analysis. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Jain with Williams because the references are analogous art because they are both directed to solutions for survey analysis, which falls within applicant’s field of endeavor (survey systems), and because modifying Jain to include Williams’ features for determining, from an administrator device during creation of the digital survey, one or more natural language prompts defining one or more triggers for customized follow-up survey inquiries based on response quality classifications, and providing the one or more natural language prompts defining the one or more triggers based on the response quality classifications to the multimodal model during creation of the digital survey and survey respondent data, in the manner claimed, would serve the motivation of improving the value of information harvested from consumer responses (Williams at paragraph 0018); and further obvious because the claimed invention is merely a combination of old elements, and in the combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that the results of the combination were predictable.
As per claim 4, the Jain-Williams-Jeffs combination teaches the system of claim 3. Jain further teaches further comprising instructions that, when executed by the at least one processor, cause the system to generate the response quality classification based on response quality classifications comprising a categorization of completeness, actionability, sentiment, or subject matter (col. 195, lines 49-61, discussing that performance can refer to various characteristics of data collection, including whether needed data collection occurred, the quality and quantity of data obtained, the conditions and context under which data collection occurred, the source of the data, completeness of the data obtained, the reliability and consistency of data collection and transmission to the system over time, and so on. Many aspects of performance can be measured with respect to the data collection requirements for the monitoring program, so that performance is indicated at least in part on the degree to which the data collection requirements are met by a user; col. 197, lines 4-25, discussing that the server can also assess the quality of data received over the course of monitoring. For example, the server can evaluate and generate scores indicating the accuracy, precision, completeness, reliability, consistency, and other characteristics of the data obtained. To do this, the system can examine collected data for inconsistencies, for example, detecting excessive variation from prior measurements or patterns, detecting conflicts between different measurements or data sources, determining if received values are within a predetermined range of what would be considered a reasonable or valid result, and so on).
Jain does not explicitly teach categorization of completeness, actionability, sentiment, or subject matter according to the one or more triggers for customized follow-up survey inquiries. However, Williams in the analogous art of questionnaire generation systems teaches this concept. Williams teaches:
categorization of completeness, actionability, sentiment, or subject matter according to the one or more triggers for customized follow-up survey inquiries (paragraph 0010, discussing that in real-time, the system analyzes the comment with voice and/or text analytics to determine what the customer said about their experience. Each topic mentioned by the customer is tagged and analyzed for sentiment. The next question or set of questions would then be determined based on the customer response to the first and each successive question and so on; paragraph 0037, discussing that if the calculated statistical confidence interval does not meet or exceed a user-defined threshold value on any of the topics identified, the system prompts the customer to confirm their perceived sentiment with a rating question. For example, if the textual analysis of the consumer comment resulted in a possible negative consumer sentiment with respect to a service, but the confidence interval was below that set by the user, a specific rating question would be generated and posed; paragraph 0038, discussing that if the confidence interval regarding consumer sentiment of the cleanliness of the parking lot was within an acceptable range, follow up questions ranking the parking lot, for example, would not need to be asked; paragraph 0045).
Jain is directed towards systems and methods for adaptive data collection. Williams is directed towards a method for conducting real-time dynamic consumer surveys. Therefore, they are deemed to be analogous as they both are directed towards survey generation and analysis. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Jain with Williams because the references are analogous art because they are both directed to solutions for survey analysis, which falls within applicant’s field of endeavor (survey systems), and because modifying Jain to include Williams’ feature for including categorization of completeness, actionability, sentiment, or subject matter according to the one or more triggers for customized follow-up survey inquiries, in the manner claimed, would serve the motivation of improving the value of information harvested from consumer responses (Williams at paragraph 0018); and further obvious because the claimed invention is merely a combination of old elements, and in the combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that the results of the combination were predictable.
As per claim 5, the Jain-Williams-Jeffs combination teaches the system of claim 3. While Jain describes exclusion criteria (col. 31, lines 53-65), Jain does not explicitly teach further comprising instructions that, when executed by the at least one processor, cause the system to generate the response quality classification based on determining that the survey response data does not include an excluded category according to the one or more triggers for customized follow-up survey inquiries. However, Williams in the analogous art of questionnaire generation systems teaches this concept. Williams teaches:
further comprising instructions that, when executed by the at least one processor, cause the system to generate the response quality classification based on determining that the survey response data does not include an excluded category according to the one or more triggers for customized follow-up survey inquiries hence less precise estimates of the parameter (paragraph 0037, discussing that if the calculated statistical confidence interval does not meet or exceed a user-defined threshold value on any of the topics identified, the system prompts the customer to confirm their perceived sentiment with a rating question. For example, if the textual analysis of the consumer comment resulted in a possible negative consumer sentiment with respect to a service, but the confidence interval was below that set by the user, a specific rating question would be generated and posed. Computer: "You mentioned the cleanliness of the parking lot in your response. Can you rate the cleanliness of the parking lot on a scale of 1 to 5?"; paragraph 0038, discussing that if the confidence interval regarding consumer sentiment of the cleanliness of the parking lot was within an acceptable range, follow-up questions ranking the parking lot, for example, would not need to be asked. A rating could be assigned based on the textual analytics of the response. For example, if the consumer said "the parking lot was filthy" a value of 1 may be assigned to the consumer rating without the need of a specific follow-up question. As noted above, if the specific topic was not mentioned by the customer (e.g., the cleanliness of the parking lot), and the user desired to collect information about the parking lot, the survey would generate follow-up questions: "You didn't mention the parking lot. Did you use the parking lot?" After an affirmative answer, a closed-ended question such as "Rate the cleanliness of the parking lot from 1 to 5" may be asked or another open-ended question may be asked such as "What did you think of the parking lot?" In one aspect of the technology, the choice between open or closed-ended questions is a function of user-defined rules setting a value on the topic of interest and the current length of the survey).
Jain is directed towards systems and methods for adaptive data collection. Williams is directed towards a method for conducting real-time dynamic consumer surveys. Therefore, they are deemed to be analogous as they both are directed towards survey generation and analysis. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Jain with Williams because the references are analogous art because they are both directed to solutions for survey analysis, which falls within applicant’s field of endeavor (survey systems), and because modifying Jain to include Williams’ feature for generating the response quality classification based on determining that the survey response data does not include an excluded category according to the one or more triggers for customized follow-up survey inquiries hence less precise estimates of the parameter, in the manner claimed, would serve the motivation of improving the value of information harvested from consumer responses (Williams at paragraph 0018); and further obvious because the claimed invention is merely a combination of old elements, and in the combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that the results of the combination were predictable.
As per claim 6, the Jain-Williams-Jeffs combination teaches the system of claim 1. Jain further teaches further comprising instructions that, when executed by the at least one processor, cause the system to generate the response quality classification of the survey response data by: providing a prompt comprising the survey response data to the multimodal model (col. 5, lines 16-45, discussing that the system may detect that the accuracy of data received from the monitoring group fails to meet a minimum data quality threshold or is trending below the data quality threshold. An analysis of the collected data and subgroups of devices can reveal that the data collected from the particular subgroup of devices is consistently low quality and, in response, prioritize the particular subgroup of devices above the other subgroups; col. 13, lines ML model to determine quality; col. 13, lines 42-63, discussing that the prediction for the group is generated based on output of one or more machine learning models trained to predict likelihoods of at least one of compliance, retention, or data quality levels, the one or more machine learning models being trained using training data that includes (i) attributes of different individuals enrolled in monitoring programs that have different requirements for participants, (ii) the requirements for participant actions of the monitoring programs, and (iii) observed compliance results for the individuals, and the one or more machine learning models are trained to provide output indicating, for a group or individual, a likelihood or rate of enrollment, compliance, retention, or a minimum data quality level in response to receiving input feature data indicating (i) attributes of individuals or of groups and (ii) data indicating requirements of a monitoring program; col 14, lines 52-62- The various models discussed can be machine learning models, for example, a neural networks or classifiers…; col. 86, lines 43-67 & col. 87, lines 1-5, discussing that the computer system may use various factors to determine data quality or which may be used to set minimum data quality requirements. These factors can include response times (e.g., where relatively quick response times and/or relatively long response times may correspond to low data quality),…, response content (e.g., text input that is relatively short or that is below a threshold word count may correspond to low data quality; text input that is relatively long or that is above a threshold word count may correspond to high data quality; etc.), etc. The computer system may use one or more algorithms to determine a data quality score or a data quality rate; col. 195, lines 35-48, discussing that the process incudes monitoring performance of the different devices with respect to the data collection requirements for the monitoring program. The monitoring can include receiving and assessing data from each of the different devices and users in a monitoring group. The data collected from a device can include the values of measured items (e.g., survey responses, user interaction data, etc.), as well as metadata, context data, and other related information. The system can analyze the received information to characterize the monitoring that has been performed; col. 198, lines 37-67, discussing that based on the training data examples, the system may train a classifier, neural network, decision tree, or other machine learning model to generate values indicating predicted outcomes such as quality of data, rate of data collection, consistency of data collection, likelihood of completing a specific data collection requirement or combination of requirements, and so on. These can be predicted generally, for different types of data, for different data sources or collection procedures (e.g., surveys), and so on. As a result, based on the training, the machine learning model is configured to (i) receive input feature values, and (ii) output in response a predicted value indicating the predicted amount of data collection, quality of collected data,…, and so on...; col. 145, lines 12-20);
receiving, from the multimodal model, a follow-up score (col. 189, lines 43-61, discussing that based on the compliance score for the participant P1 being lower than the compliance score for the participant P5, the computer system may determine that it should send four reminders to the P1 device for every upcoming survey and one reminder to the P5 device for every upcoming survey; col. 198, lines 37-67, discussing that based on the training data examples, the system may train a classifier, neural network, decision tree, or other machine learning model to generate values indicating predicted outcomes such as quality of data, rate of data collection, consistency of data collection, likelihood of completing a specific data collection requirement or combination of requirements, and so on… As a result, based on the training, the machine learning model is configured to…output in response a predicted value indicating the predicted amount of data collection, quality of collected data, likelihood of meeting one or more data collection requirements, and so on. Of course, other techniques for prediction can be used instead of or in addition to machine learning models, including statistical analysis, rule-based models, and so on. The various techniques discussed for FIGS. 1-19D for predicting future compliance and other outcomes can be used in the process to generate scores, including group performance scores or individual performance scores, for prioritizing groups and individuals for intervention to improve monitoring; col. 203, lines 4-36, discussing that the system can train, store, and use a machine learning model trained to predict which interventions are best suited for different device types, user types, and/or situations. For example, the model can receive input feature values indicating attributes of a user, as well as potentially values characterizing the past monitoring performance or the problem with monitoring performance that has occurred (e.g., the level of data quality that was achieved, etc.). The model can then output one or more scores, for each of various different interventions, indicating the likelihood of improvement and/or magnitude of improvement that is expected in monitoring performance if the intervention is implemented. From these scores, the system can select one or more interventions that are predicted to best improve monitoring performance for an individual user, for the areas of monitoring that the user is most in need of improvement; col. 87, line 1); and
utilizing the follow-up score to generate the response quality classification of the survey response data (col. 197, lines 40-61, discussing that the performance score can characterize the performance of the group and can be expressed in various different ways. In some cases, measures of performance for individual users in a group are combined and used to generate the performance score for the group. For example, the performance score may be an average percentage of data collection performed, such as an average of the percentages of data collection successfully performed by individual users; col. 198, lines 37-67, discussing that based on the training data examples, the system may train a classifier, neural network, decision tree, or other machine learning model to generate values indicating predicted outcomes such as quality of data, rate of data collection, consistency of data collection, likelihood of completing a specific data collection requirement or combination of requirements, and so on…As a result, based on the training, the machine learning model is configured to…output in response a predicted value indicating the predicted amount of data collection, quality of collected data, likelihood of meeting one or more data collection requirements, and so on. Of course, other techniques for prediction can be used instead of or in addition to machine learning models, including statistical analysis, rule-based models, and so on. The various techniques discussed for FIGS. 1-19D for predicting future compliance and other outcomes can be used in the process to generate scores, including group performance scores or individual performance scores, for prioritizing groups and individuals for intervention to improve monitoring; col. 199, lines 51-67 & col. 200, lines 1-2, discussing that with the performance scores and rankings of the groups, the system can then use those scores and rankings to determine scores and rankings to prioritize individuals. For example, the system can generate an individual priority score for a user…As a result, low group performance and low individual performance can combine to indicate a low combined performance that indicates that a user has a high need of intervention. The combined score can be adjusted based on the other factors, such as a penalty (e.g., adding to the score to show less importance or urgency in intervening) if the data collection is so poor that intervention has a low likelihood of bringing the user into compliance; col. 202, lines 4-16, discussing that the prioritization allows for various types of differential treatment of members of one group compared to another. As an example, the system may make adjustments such as applying a rule that increases communication with users if the group's performance is less than a certain threshold, but not if the performance for the group is above the threshold. Similarly, the system can apply various rule-based changes, where a condition or trigger for taking an action is based on the value of the performance score for the group or for the combined score for an individual; col. 199, lines 1-23; col. 203, lines 4-36).
As per claim 7, the Jain-Williams-Jeffs combination teaches the system of claim 6. Jain further teaches wherein utilizing the follow-up score to generate the response quality classification comprises determining, from the follow-up score, a number of triggers indicated by the survey response data and a degree to which the survey response data indicates the number of triggers (col. 3, lines 1-20, discussing that poor performance in one of a multiple performance categories may trigger a performance analysis by the system; col. 86, lines 43-67 & col. 87, lines 1-5, discussing that the data quality data may include, for example, an indication of data quality for each of the subjects or for different categories of subjects. Additionally, or alternatively, the data quality data may include data quality rates that indicate, for each of the subjects or for each category of subjects, the percent of data that meets minimum data quality requirements. The computer system may use various factors to determine data quality, or which may be used to set minimum data quality requirements. These factors can include response times,…, response content (e.g., text input that is relatively short or that is below a threshold word count may correspond to low data quality; text input that is relatively long or that is above a threshold word count may correspond to high data quality; etc.), etc. The computer system may use one or more algorithms to determine a data quality score; col. 148, lines 15-17, discussing that the system can have certain thresholds or criteria for initiating notifications and corrective actions; col. 189, lines 43-61, discussing that based on the compliance score for the participant P1 being lower than the compliance score for the participant P5, the computer system may determine that it should send four reminders to the P1 device for every upcoming survey and one reminder to the P5 device for every upcoming survey; Col. 191, lines 63-67, discussing that the data collection requirements can also specify the standards or thresholds that define a minimum acceptable data collection performance...; col. 201, lines 7-23, discussing that the system may also provide recommendations of interactions to individuals...The system may require authorization or approval of some adjustments, and may provide alerts or user interface data to inform the researchers and enable them to indicate approval to trigger the system to carry out the interventions; col. 202, lines 4-16, discussing that the prioritization allows for various types of differential treatment of members of one group compared to another. As an example, the system may make adjustments such as applying a rule that increases communication with users through their devices if the group's performance is less than a certain threshold, but not if the performance for the group is above the threshold. Similarly, the system can apply various rule-based changes, where a condition or trigger for taking an action is based on the value of the performance score for the group or for the combined score for an individual; col. 202, lines 46-67 & col. 203, lines 1-3, discussing that the system can store a database or other repository of different interventions, with different ways that a monitoring program can be adjusted to correct for different types of poor performance. For example, the system can store a library of different interventions and corresponding conditions that can correct for the different indictors or causes of poor monitoring performance. For failure to complete a survey, the library can indicate sending reminders to the user, increasing the frequency that the survey is taken, changing the font size and layout of the survey, breaking the survey into multiple shorter interactions spaced apart over time, and so on. For failure to obtain a sensor measurement, examples of interventions include increasing the frequency that the measurement is attempted, attempting to obtain data from another device, sending instructions to change settings of the sensors or of the application that handles the measurements, scheduling the measurements for different times, setting triggers to perform the measurement when certain conditions are reached, and so on; col. 203, lines 4-36, discussing that the system can train, store, and use a machine learning model trained to predict which interventions are best suited for different device types, user types, and/or situations. For example, the model can receive input feature values indicating attributes of a user, as well as potentially values characterizing the past monitoring performance or the problem with monitoring performance that has occurred (e.g., the level of data quality that was achieved, etc.). The model can then output one or more scores, for each of various different interventions, indicating the likelihood of improvement and/or magnitude of improvement that is expected in monitoring performance if the intervention is implemented. From these scores, the system can select one or more interventions that are predicted to best improve monitoring performance for an individual user, for the areas of monitoring that the user is most in need of improvement; col. 215, lines 64-67 & col. 216, lines 1-32, discussing that the computer system detects a condition that provides an opportunity for an adaptation. based on the analysis, the computer system can evaluate the various patterns, correlations, health outcomes, anomalies, outliers, or events, and can use various thresholds and references to detect a condition for providing adaptation; col. 226, lines 27-53, discussing that the rules engine is shown having a set of adaptation rules and adaptation template…the adaptation rules can specify the conditions, triggers, thresholds, and other parameters that the rules engine checks to determine if there is sufficient reason for an adaptation; col. 232, lines 55-63, discussing that when evaluating the collected data, the computer system can monitor the data with respect to objective references (e.g., predetermined thresholds, predetermined classifications, normal or expected ranges, etc.); col. 89, lines 15-29; col. 128, lines 16-27; col. 199, lines 51-67 & col. 200, lines 1-2, col. 203, lines 37-49).
Claim 10 recites substantially similar limitations that stand rejected via the art citations and rationale applied to claim 1, as discussed above. Further, as per claim 10 the Jain-Williams-Jeffs combination teaches a non-transitory computer-readable medium storing instructions that, when executed by at least one processor, cause a computer device (Jain, col. 286, lines 11-37: “Embodiments of the invention and all of the functional operations described in this specification can be implemented in digital electronic circuitry, or in computer software, firmware, or hardware, including the structures disclosed in this specification and their structural equivalents, or in combinations of one or more of them. Embodiments of the invention can be implemented as one or more computer program products, e.g., one or more modules of computer program instructions encoded on a computer readable medium for execution by, or to control the operation of, data processing apparatus. The computer readable medium can be a machine-readable storage device, a machine-readable storage substrate, a memory device, a composition of matter effecting a machine-readable propagated signal, or a combination of one or more of them. The term “data processing apparatus” encompasses all apparatus, devices, and machines for processing data, including by way of example a programmable processor, a computer, or multiple processors or computers. The apparatus can include, in addition to hardware, code that creates an execution environment for the computer program in question, e.g., code that constitutes processor firmware, a protocol stack, a database management system, an operating system, or a combination of one or more of them.”).
Claim 11 recites substantially similar limitations that stand rejected via the art citations and rationale applied to claim 2, as discussed above.
Claim 12 recites substantially similar limitations that stand rejected via the art citations and rationale applied to claim 3, as discussed above.
Claim 13 recites substantially similar limitations that stand rejected via the art citations and rationale applied to claim 4, as discussed above.
Claim 14 recites substantially similar limitations that stand rejected via the art citations and rationale applied to claim 5, as discussed above.
Claim 15 recites substantially similar limitations that stand rejected via the art citations and rationale applied to claims 6 and 7, as discussed above.
Claim 18 recites substantially similar limitations that stand rejected via the art citations and rationale applied to claim 1, as discussed above. Further, as per claim 18 the Jain-Williams-Jeffs combination teaches a computerized method (Jain, col. 11, lines 39-41: “a method performed by one or more computers”).
Claim 19 recites substantially similar limitations that stand rejected via the art citations and rationale applied to claim 2, as discussed above.
Claim 20 recites substantially similar limitations that stand rejected via the art citations and rationale applied to claim 3, as discussed above.
20. Claims 8 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Jain in view of Williams, in view of Jeffs, in view of Poteet et al., Patent No.: US 8,577,884 B2, [hereinafter Poteet], in further view of Barbosa et al., Pub. No.: US 2017/0337477 A1, [hereinafter Barbosa].
As per claim 8, the Jain-Williams combination teaches the system of claim 1. Although not explicitly taught by Jain, Williams in the analogous art of questionnaire generation systems teaches further comprising instructions that, when executed by the at least one processor, cause the system to: receive a follow-up response to the customized follow-up survey inquiry via the respondent client device (paragraph 0018, discussing that it is desirable to ask follow-up questions that further explain a particular answer...For example, if a customer initially rates product quality as low, it is useful to ask the customer what they did not like about quality. In accordance with one aspect of the technology, the model includes one or more descriptive questions that further explain a previous question and that can be used as secondary predictors of the primary value. These questions are also called "drill-in," "attributes," or "features." For example, if the consumer gave a low score to service, a closed-ended follow-up question might be "Select what was bad about the service you received: ("slow | rude | incorrect"). The data model is used for data analysis including trending, comparative studies and statistical/predictive modeling. This data is structured and can be modeled inside a computer database. Even though surveys can be very detailed with many questions, they often cannot capture every case. Here, the consumer score regarding service may not be poor because of the three items provided in the closed-ended question. Advantageously, an unstructured text or "open-ended question" may be used for the respondent to use to fill in the gaps or for a user to gage what is most important to the consumer's experience with his or her goods. For example, an open-ended question following a low score on service might be "Tell us in your own words why you rated us poorly on our service." In this manner, the business concern is not limited by the specific set of potential problems the consumer may have encountered. As a result, survey length may be shortened and the value of information harvested from consumer responses is improved; paragraphs 0024-0027, discussing that the facts extracted from customer comments through NLP technology are used to restructure a survey in real time…. One example of an open-ended question and follow-up questions is presented below. In this example, a hotel manager desires to understand how customers rate their experience and what was most important to them. Rather than provide a long list of questions that may or may not be of concern to the consumer, an open-ended question is posed to provide the consumer with the opportunity to identify those areas that were most important to the consumer. Follow-up questions are generated in response to the facts and topics of interest generated by the consumer as well as topics of interest generated by the user of the model. In this example, information that the hotel manager identifies as important to the business includes overall experience rating, service rating, room rating, fair price, and concierge service rating. All other data is optional but still useful to the hotel manager. The system first poses an open-ended question as noted below with an example response from the consumer. Computer: "Tell us in your own words about your experience." Consumer: "I really didn't have a good experience with your hotel. The bell service was slow and a bit rude, plus I didn't get room service at all. At the prices you charge, I expect more." Based on the consumer response, the following facts might be extracted from the comment using NLP or computational linguistics: The consumer had a poor experience; the bell service was slow; the employees were rude; room service was poor; price was mentioned; and expectations were not met. Based on the facts extracted from the comment, the system generates a set of closed-ended follow-up questions in order to obtain ratings with respect to the specific items referenced. Additionally, specific questions from the system user (i.e., the hotel manager) wishes asked as part of the survey are included to rate user-generated topics in addition to the consumer-generated topics of interest. The following is one example set of follow-up questions: 1. Computer: "I'm sorry you had a poor experience. We'd like to do better. Can you rate your experience on a scale of 1 to 5?"; paragraph 0040, discussing receiving the consumers response to said open-ended question; paragraph 0035).
Jain is directed towards systems and methods for adaptive data collection. Williams is directed towards a method for conducting real-time dynamic consumer surveys. Therefore, they are deemed to be analogous as they both are directed towards survey generation and analysis. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine Jain with Williams because the references are analogous art because they are both directed to solutions for survey analysis, which falls within applicant’s field of endeavor (survey systems), and because modifying Jain to include Williams’ feature for receiving a follow-up response to the customized follow-up survey inquiry via the respondent client device, in the manner claimed, would serve the motivation of improving the value of information harvested from consumer responses (Williams at paragraph 0018); and further obvious because the claimed invention is merely a combination of old elements, and in the combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that the results of the combination were predictable.
The Jain-Williams combination does not explicitly teach determine a summary for the follow-up response and one or more follow-up responses to one or more additional follow-up survey inquiries; and generate a survey report comprising the summary for the follow-up response and one or more follow-up responses to the one or more additional follow-up survey inquiries. Poteet in the analogous art of survey systems teaches:
determine a summary for the response and one or more responses to one or more additional survey inquiries (col. 2, lines 19-30, discussing technologies for providing automated analysis and summarization of free-form comments in survey response data. Through the concepts and technologies presented herein, free-form comments can be analyzed and summarized programmatically, without the need to pre-develop a context or lexicon to describe the scope of responses. The text of the comments in the survey response data is utilized to develop the semantic relationships between words and terms contained therein, and to extract the salient topics represented by the comments. The topics, along with the number of comments relevant to each, are summarized in reports and charts that provide the survey results; col. 8, lines 34-46, discussing that the text mining application generates reports and charts which provide the results of the analysis and summarization of the survey response comments); and
generate a survey report comprising the summary for the response and one or more responses to the one or more additional survey inquiries (col. 2, lines 31-43, discussing that a report is generated which summarizes the topics and their relative importance in the survey response comments based upon the number of comments relevant to each; col. 4, lines 13-21, discussing that the text mining application provides the functionality for collecting, analyzing, and reporting free-form survey response comments; col. 4, lines 54-60, discussing that once the analysis is complete, the text mining application generates reports and charts containing the details of the analysis of the survey response comments).
The Jain-Williams-Jeffs combination describes features related to data collection and questionnaire generation. Poteet is directed towards system and methods for analysis and summarization of survey responses. Therefore, they are deemed to be analogous as they both are directed towards survey analysis. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the Jain-Williams-Jeffs combination with Poteet because the references are analogous art because they are both directed to solutions for data analysis, which falls within applicant’s field of endeavor (survey systems), and because modifying the Jain-Williams-Jeffs combination to include Poteet’s features for determining a summary for the response and one or more responses to one or more additional survey inquiries and generating a survey report comprising the summary for the response and one or more responses to the one or more additional survey inquiries, in the manner claimed, would serve the motivation of making apparent to the analyst what issues or topics are important to the respondents (Poteet at col. 1, lines 15-16); and further obvious because the claimed invention is merely a combination of old elements, and in the combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that the results of the combination were predictable.
The Jain-Williams-Jeffs-Poteet combination does not explicitly teach that the summary is for the follow-up response and one or more follow-up responses to one or more additional follow-up survey inquiries and that the report comprises the summary for the follow-up response and one or more follow-up responses to the one or more additional follow-up survey inquiries. However, Barbosa in the analogous art of question and answer computer systems teaches these concepts. Barbosa teaches:
determine a summary for the follow-up response and one or more follow-up responses to one or more additional follow-up survey inquiries (paragraph 0067, discussing that the response follow-up system modifies an aspect of a user interface based on the one or more follow-up questions. The user interface can be modified by displaying the one or more follow-up questions or links to the one or more follow-up question. The response follow-up system may determine one or more follow-up responses to the one or more follow-up questions. The user interface can be modified by displaying the one or more follow-up responses in combination with the response or by displaying a link to the one or more follow-up responses with the response, such as response or a link to response 204B with response 204A. One or more additional responses to the question may also be received at the response follow-up system. The response follow-up system can determine one or more additional follow-up questions from the list of common follow-up questions based on the one or more additional responses. The one or more additional follow-up questions can be output in combination with the one or more follow-up questions to the user interface; paragraph 0070, discussing that the response follow-up system may analyze a corpus of documents to produce an concept frequency-inverse document frequency for concepts in the documents and identify concepts in the one or more response terms having a lower concept frequency-inverse document frequency in the documents as a basis for generating the one or more follow-up questions. The response follow-up system can also or alternatively search for concepts in the one or more response terms across the corpus to determine a usage context of the one or more response terms and apply the usage context in generating the one or more follow-up questions); and
comprising the summary for the follow-up response and one or more follow-up responses to the one or more additional follow-up survey inquiries (paragraph 0020, discussing that the question-answering system converts the questions received in a natural language format into one or more queries for the content servers to determine responses. Content 114A-N may be returned directly to one or more of the browser applications or can be reformatted as responses and sent to one or more of the browser applications. When the content is sent directly to one or more of the browser applications, the response follow-up system may also track aspects of the content in the responses; paragraph 0025, discussing that the initial question and response can be marked in questions and responses as a question-response pair and the subsequent question can be associated to the question-response pair in follow-up questions; paragraph 0071, discussing that the response follow-up system modifies an aspect of a user interface, such as user interface, based on the one or more follow-up questions. The user interface can be modified by displaying the one or more follow-up questions or links to the one or more follow-up questions. The response follow-up system may determine one or more follow-up responses to the one or more follow-up questions. The user interface can be modified by displaying the one or more follow-up responses in combination with the response or by displaying a link to the one or more follow-up responses with the response, such as response 204B or a link to response 204B with response 204A. One or more additional responses to the question may also be received at the response follow-up system. The response follow-up system can determine one or more additional follow-up questions based on the one or more additional responses. The one or more additional follow-up questions can be output in combination with the one or more follow-up questions to the user interface).
The Jain-Williams-Jeffs-Poteet combination describes features related to data collection and questionnaire generation. Barbosa is directed towards a system for determination of automated response follow-up. Therefore, they are deemed to be analogous as they both are directed towards data collection and analysis. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the Jain-Williams-Jeffs-Poteet combination with Poteet because the references are analogous art because they are both directed to solutions for data analysis, which falls within applicant’s field of endeavor (survey systems), and because modifying the Jain-Williams-Jeffs-Poteet combination to include Barbosa’s features for determining a summary for the follow-up response and one or more follow-up responses to one or more additional follow-up survey inquiries, and comprising the summary for the follow-up response and one or more follow-up responses to the one or more additional follow-up survey inquiries, in the manner claimed, would serve the motivation of automating the creation of follow-up questions based on a response to a question in a question-answering computer system (Barbosa at paragraph 0072); and further obvious because the claimed invention is merely a combination of old elements, and in the combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that the results of the combination were predictable.
Claim 16 recites substantially similar limitations that stand rejected via the art citations and rationale applied to claim 8, as discussed above.
21. Claims 9 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over Jain in view of Williams, in view of Jeffs, in further view of Miller et al., Pub. No.: US 2018/0357240 A1, [hereinafter Miller].
As per claim 9, the Jain-Williams-Jeffs combination teaches the system of claim 1. Jain further teaches further comprising instructions that, when executed by the at least one processor, cause the system to iteratively train the multimodal model by: predicting survey inquiries for a training set of survey response data (col. 13, lines 42-63, discussing that the prediction for the group is generated based on output of one or more machine learning models trained to predict likelihoods of at least one of enrollment, compliance, retention, or data quality levels, the one or more machine learning models being trained using training data that includes (i) attributes of different individuals enrolled in monitoring programs that have different requirements for participants, (ii) the requirements for participant actions of the monitoring programs, and (iii) observed compliance results for the individuals, and the one or more machine learning models are trained to provide output indicating, for a group or individual, a likelihood or rate of enrollment, compliance, retention, or a minimum data quality level in response to receiving input feature data indicating (i) attributes of individuals or of groups and (ii) data indicating requirements of a monitoring program; col. 143, lines 44-67 & col. 144, lines 1-18, discussing that the system may use any of various techniques to make the predictions about future outcomes for the monitoring group and the future characteristics of the monitoring group. One technique is to use statistical techniques to take historical data, identify examples of individuals that would be included the different diversity groups, and then determine this historical success rate…The system may optionally can determine a different prediction for success with respect to different requirements, e.g., 95% expected to provide daily survey responses, 86% expected to provide daily step count data, 85% expected to provide the appropriate level of data quality (potentially making different estimates for different aspects of data quality), and so on; col. 145, lines 12-20, discussing that the system can also use trained machine learning models to predict the future outcomes. One example is a model that predicts a success rate (e.g., either overall or for specific factors such as compliance, data quality, etc.) for a group based on the group's characteristics and the characteristics of monitoring program; col. 198, lines 37-67, discussing that based on the training data examples, the system may train a classifier, neural network, decision tree, or other machine learning model to generate values indicating predicted outcomes such as quality of data, rate of data collection, consistency of data collection, likelihood of completing a specific data collection requirement or combination of requirements, and so on. These can be predicted generally (e.g., for all data types and data collection), for different types of data, for different data sources or collection procedures (e.g., surveys), and so on. As a result, based on the training, the machine learning model is configured to (i) receive input feature values that indicate attributes of a device user, as well as potentially data about historical outcomes, and (ii) output in response a predicted value indicating the predicted amount of data collection, quality of collected data, likelihood of meeting one or more data collection requirements, and so on. Of course, other techniques for prediction can be used instead of or in addition to machine learning models, including statistical analysis, rule-based models, and so on. The various techniques discussed above for FIGS. 1-19D for predicting future compliance and other outcomes can be used in the process to generate scores, including group performance scores or individual performance scores, for prioritizing groups and individuals for intervention to improve monitoring; col. 209, lines 54-67 & col. 210, lines 1-21, discussing that the computer system also benefits from the results and monitoring done for many different research studies, enabling the computer system to learn over time which adaptations are most successful and the conditions in which different changes succeed...); and
ground truth (col. 186, lines 25-29, discussing that the model may be trained to reduce the difference between calculated priority scores and/or the resulting group rank and ground-truth data that indicates an ideal priority score and/or group rank; col. 198, lines 18-36, discussing that the system uses other techniques to improve the accuracy of predictions. For example, the system can use machine learning models trained based on examples of attributes of users and devices (e.g., values for attributes that may include or be different from those used to define the categories) and the corresponding histories and data collection outcomes. In the training, the attributes and current and previous outcomes are used to generate input feature values, and the outcomes actually achieved are used to define training labels or “ground truth” indications as training targets for a model).
Jain does not explicitly teach iteratively train the multimodal model by: predicting customized follow-up survey inquiries for a training set of survey response data; and modifying the multimodal model based on comparing the predicted customized follow-up survey inquiries with ground truth customized follow-up survey inquiries to reduce or minimize a loss of a loss function. However, Miller in the analogous art of information retrieval systems teaches these concepts. Miller teaches:
iteratively train the multimodal model by: predicting customized follow-up survey inquiries for a training set of survey response data (paragraph 0002, discussing information retrieval systems designed for answering questions using machine learning; paragraph 0031, discussing that the system may generate an aggregated result using the second relevance measures generated in the current iteration...The iterative process then repeats until the designated number of iterations have been performed. For example, after the initial iteration, each subsequent iteration of the iterative process may similarly involve generating a current-iteration query vector representation based on an immediately-preceding-iteration query vector representation that is generated in an immediately-preceding iteration, an immediately-preceding-iteration aggregated result that is generated in the immediately-preceding iteration, and a current-iteration machine-learning model Rj; paragraph 0034, discussing that training an embodiment of a Key-Value Memory Network model. In particular embodiments, the whole network may be trained end-to-end, and the model learns to perform the iterative accesses to output the desired target a by minimizing a standard cross-entropy loss between a and the correct answer a. For example, the machine-learning architecture may include any number of models, including the aforementioned matrices A, B and R1,…, RH. The machine-learning models may be trained using a sufficiently large number of samples of training input. Each training input may include an input (e.g., a question or textual task)…Each training sample may also include a target output (also referred to as the ground truth output), which is the known, correct output for the associated input. The machine-learning models may be trained iteratively using the set of training samples. During each training iteration, the models may process the training input of a training sample to generate a training output, which is selected in response to the training input (e.g., an answer to the question). A loss function may then be used to compare the generated training output to the target output(or ground truth), and the result of the comparison may be used to update (e.g., through backpropagation) the models in the machine-learning architecture. For example, back propagation and stochastic gradient descent may thus be used to learn the matrices A, B and R1,…, RH. Once the models have been trained, they may be distributed to and used by any computing system (to automatically answer questions, for example); and
modifying the multimodal model based on comparing the predicted customized follow-up survey inquiries with ground truth customized follow-up survey inquiries to reduce or minimize a loss of a loss function (paragraph 0024, discussing that after receiving the result, it may be used to generate a new query q for subsequent addressing. In particular embodiments, an iterative process may be used to iteratively access the memories. During each iteration, the query may be updated based on the immediately-preceding iteration's query and associated aggregated result. This may be formulated as: qj+1=Rj(qj+oj), where Rj is a machine-learning model..; paragraph 0034, discussing that training an embodiment of a Key-Value Memory Network model. In particular embodiments, the whole network may be trained end-to-end, and the model learns to perform the iterative accesses to output the desired target a by minimizing a standard cross-entropy loss between a and the correct answer a. For example, the machine-learning architecture may include any number of models, including the aforementioned matrices A, B and R1,…, RH. The machine-learning models may be trained using a sufficiently large number of samples of training input. Each training input may include an input (e.g., a question or textual task)…Each training sample may also include a target output (also referred to as the ground truth output), which is the known, correct output for the associated input. The machine-learning models may be trained iteratively using the set of training samples. During each training iteration, the models may process the training input of a training sample to generate a training output, which is selected in response to the training input (e.g., an answer to the question). A loss function may then be used to compare the generated training output to the target output(or ground truth), and the result of the comparison may be used to update (e.g., through backpropagation) the models in the machine-learning architecture. For example, back propagation and stochastic gradient descent may thus be used to learn the matrices A, B and R1,…, RH. Once the models have been trained, they may be distributed to and used by any computing system (to automatically answer questions, for example).
The Jain-Williams-Jeffs combination describes features related to data collection and questionnaire generation. Miller is directed towards an information retrieval system. Therefore they are deemed to be analogous as they both are directed towards data collection and analysis. It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to combine the Jain-Williams-Jeffs combination with Miller because the references are analogous art because they are both directed to solutions for data analysis, which falls within applicant’s field of endeavor (survey systems), and because modifying the Jain-Williams-Jeffs combination to include Miller’s features for iteratively training the multimodal model by: predicting customized follow-up survey inquiries for a training set of survey response data, and modifying the multimodal model based on comparing the predicted customized follow-up survey inquiries with ground truth customized follow-up survey inquiries to reduce or minimize a loss of a loss function, in the manner claimed, would serve the motivation of developing efficient information extraction methods (Miller at paragraph 0005); and further obvious because the claimed invention is merely a combination of old elements, and in the combination each element merely would have performed the same function as it did separately, and one of ordinary skill in the art would have recognized that the results of the combination were predictable.
Claim 17 recites substantially similar limitations that stand rejected via the art citations and rationale applied to claim 9, as discussed above.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Khaled, Pub. No.: US 2021/0035155 A1 – describes generating and distributing digital surveys based on predicting survey responses to digital survey questions.
Jorasch et al., Pub. No.: US 2021/0373676 A1 – describes validating survey responses and embedded survey experiments.
Kopikare, Pub. No.: US 2019/0318370 A1 – describes that the question content assists the survey system in determining whether to include a given question as a subsequent question or a follow-up question to another question based on the themes of the questions.
Ge, Yubin, et al. "What should i ask: A knowledge-driven approach for follow-up questions generation in conversational surveys." Proceedings of the 37th Pacific Asia Conference on Language, Information and Computation. 2023 – describes that generating follow-up questions on the fly could significantly improve conversational survey quality and user experiences by enabling a more dynamic and personalized survey structure.
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any extension fee pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to DARLENE GARCIA-GUERRA whose telephone number is (571) 270-3339. The examiner can normally be reached M-F 7:30a.m.-5:00p.m. EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Brian M. Epstein can be reached on (571) 270-5389. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Darlene Garcia-Guerra/
Primary Examiner, Art Unit 3625
Prosecution Insights