DETAILED ACTION
This communication is in response to the Application filed on 01/15/2025. Claims 1-20 are pending and have been examined. Claims 1, 13 and 17 are independent. This Application was published as US Pub 2025/0156474.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Priority
This application is a Con of 18/674,750 submitted on 05/24/2024. Applicant’s claims for benefit of a provisional application 63/538,497 submitted on 05/24/2024 is acknowledged.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-4, 7-8, 12-15 and 17-18 are rejected under 35 U.S.C. 103 as being unpatentable over Setlur et al. (US Pub 2020/0110779) in view of Hullman et al. ("Hypothetical outcome plots outperform error bars and violin plots for inferences about reliability of variable ordering." PloS one 10.11 (2015).) further in view Erickson et al. (US Pub 2017/0116185) further in view of Shen et al. ("Data player: Automatic generation of data videos with narration-animation interplay." IEEE Transactions on Visualization and Computer Graphics 30.1 (2023): 109-119).
Regarding Claim 1,
Setlur discloses a method for generating multi-modal data representations, comprising: at a computing device having a display, one or more processors, and memory storing one or more programs configured for execution by the one or more processors (Fig.2A: computing device 200; Fig.9, par [159], method steps 904 and 906):
in response to a user query regarding a dataset that includes variability (Fig.9A, par [161], "…The computing device 200 receives (910) user selection of a data source..."; par [145], "…the vague concept "expensive" to range from [avg+SD, max], where avg, SD and max are the average, standard deviation and maximum values for the numerical field "Price"..."):
obtaining the dataset that includes one or more data fields and data corresponding to the one or more data fields (Fig.3, par [089], "…an example data source 310...the data source 310 is a data structure (e.g., spreadsheet) that includes a plurality of data values stored in data columns….");
determining data uncertainty corresponding to the data (Fig.9A, par [181], "…the range of predefined values includes (940) one or more of an average value, a standard deviation, and a maximum value associated with the data field…");
causing the multi-modal data representation to be presented at a user interface of an electronic device (par [029], "…methods, systems, and graphical user interfaces that enable users to easily interact with data visualizations and analyze data using natural language expressions..."; Fig.2A, the user interface 210, par [128], "…the data visualization application 230 assigns different ranks to different data visualization types. A higher rank is assigned to a data visualization that presents views that encode data graphically. Text tables are assigned the lowest rank..."; Fig.4F (e.g., textual output presented with visualization) Fig.9E, par [177], "…The computing device 200 generates (968) and displays a data visualization of the retrieved data sets...").
Setlur discloses a data visualization system that receives the user query directed to a dataset, compute data statistics, and displays data visualization at the user interface. But Seltur renders data visualization as static charts with no animation.
However, Hullman, in the analogous filed of visualization of quantitative data, discloses generating a multi-modal data representation of the data and the data uncertainty, including: rendering a data visualization that represents the data and the data uncertainty (Hullman, Fi.2, 1 Introduction, "…Hypothetical Outcome Plots (HOPs). In its most simple variant, the HOPs approach is to: (1) draw a sample of hypothetical outcomes (draws) from the distribution; (2) for each, make a plot that becomes one frame in an animated presentation..."; 7 Conclusion, "…We present and study Hypothetical Outcome Plots as an alternative to static depictions of probability distributions. To create HOPs, we generate draws from a probability distribution and visualize each draw as an outcome plot…");
Therefore, it would have been obvious to a person of ordinary skill in the art to substitute Hullman's known animated-draw technique for Seltur's static chart-rendering step would have yielded the predictable result of a query-responding visualization that also conveys the queried data's uncertainty through animation. This is a substitution of a known, more distribution-aware rendering technique for Seltur's static chart.
Erickson, in the analogous field of natural language processing system, discloses generating, according to statistics of the dataset, text content describing the data and the data uncertainty (Erickson, Abstract, "…The NLP system is configured to generate at least one confidence level based at least in part on at least one portion of the analysis operation...integrate at least one disfluency into the NL output based at least in part on the at least one confidence level..."; Fig.4: a NLT input/output module system 200, paras[059, 070, 073-075], "…Prosodic parameters are essentially the various vocal patterns and rhythms of speech, including pitch, intonation, nasalization and stress...Disfluency analyzer circuit 210 stores data identifying multiple LOC ranges, which each represent a particular degree of uncertainty or confidence in the initial NL output...Disfluency selection analysis circuit 210 may add prosodic parameters of the target language the integration of disfluent speech or text..."; Fig.7, par [093], "…inserting a linguistic hedge (e.g., "I think that's right") following a low confidence NLP output...");
translating the text content into a speech synthesis markup language to generate an audio narrative of the text content (Erickson, par [064], "…a pause may be inserted immediately before the disfluency to add emphasis..."; par [093], "…adding a rising intonation to the concluding phonemes of an embedded disfluency, adding an expression or gesture indicating uncertainty just before or during the production of a disfluency..."; par [093], "…one or more embodiments provide systems and methods for alerting speakers and hearers that NLP outputs may contain errors by inserting natural disfluencies and other cues in the NLP output (i.e., it is construed that prosodically modified audio/speech NPL output is generated and presented to the users.)...");
Therefore, it would have been obvious to a person of ordinary skill in the art to apply Erickson's confidence-to-hedge/prosody technique to a query-responsive data visualization system displaying the uncertainty of Seltur in view of Hullman with a reasonable expectation that the combination would have yielded the predictable results of spoken and textual description further reflecting data uncertainty with hedge words or prosody.
Seltur in view of Hullman further in view of Erickson does not explicitly disclose how data visualization, text, and audio components are synchronized.
However, Shen, in the analogous field of endeavor, discloses synchronizing the data visualization, the text content, and the audio narrative according to a timestamp of the audio narrative. (Shen, 4.1.2 Narration Entities, "…Static narration text will be converted into audio speech, and each entity will be an audio unit with time...narration entity := (audio, time)...time := (start,duration)..."; 4.4.2 Constraint Encoding, "…We further use Microsoft Azure Text-to-Speech services to automatically generate audio narration and obtain the timestamps of each word in the audio, which also acts as the timeline to arrange animation effects applied to the visual elements...")
Therefore, it would have been obvious to one of ordinary skill in the art, before effective filing date of the claimed invention, to have modified a data visualization and narration generation platform of Seltur in view of Hullman further in view of Erickson with Shen's timestamp-driven synchronization of visualization and TTS-acquired audio voiceover of Data Player with a reasonable expectation of success to lowers the technical barriers associated with creating data videos rich in narration and provide an intuitive interpretation of data charts while vividly articulating the underlying data insights (Shen, Abstract).
Regarding Claim 2,
Seltur in view of Hullman further in view of Erickson further in view of Shen discloses the method of claim 1, wherein determining the data uncertainty corresponding to the data includes determining one or more of: a standard deviation of the data, percentile ranges of the data, confidence intervals of the data, and an entropy of the data (Seltur, Fig.9A, par [161], "…The computing device 200 receives (910) user selection of a data source..."; par [145], "…the vague concept "expensive" to range from [avg+SD, max], where avg, SD and max are the average, standard deviation and maximum values for the numerical field "Price"...").
Regarding Claim 3,
Seltur in view of Hullman further in view of Erickson further in view of Shen discloses the method of claim 1, wherein: the data comprises discrete data points; and generating the data visualization that represents the data and the data uncertainty includes determining a continuous probability curve from discrete data points (Hullman, 3 Study: Methods, Fig.3, "…For each distribution in each sequence, we simulate 5000 draws from a normal distribution. This same set of 5000 draws is used to generate all three visualizations for that distribution...All violin plots are generated in D3 using the histogram function with fine-grained bins...")
Regarding Claim 4,
Seltur in view of Hullman further in view of Erickson further in view of Shen discloses the method of claim 1, wherein the data visualization comprises an animated data visualization with animations that are time-synchronized according to the timestamp of the audio narrative (Hullman, Fi.2, 1 Introduction, "…Hypothetical Outcome Plots (HOPs). In its most simple variant, the HOPs approach is to: (1) draw a sample of hypothetical outcomes (draws) from the distribution; (2) for each, make a plot that becomes one frame in an animated presentation..."; Shen, 4.4.2 Constraint Encoding, "…We further use Microsoft Azure Text-to-Speech services to automatically generate audio narration and obtain the timestamps of each word in the audio, which also acts as the timeline to arrange animation effects applied to the visual elements...").
Regarding Claim 7,
Seltur in view of Hullman further in view of Erickson further in view of Shen discloses the method of claim 1, wherein generating the text content includes: applying one or more natural language templates; and populating the one or more natural language templates with hedge words and summary statistics from the dataset (Erickson, Fig.4: a NLT input/output module system 200, paras[059, 075], "…Prosodic parameters are essentially the various vocal patterns and rhythms of speech, including pitch, intonation, nasalization and stress...Disfluency selection analysis circuit 210 may optionally electronically associates the LOC ranges with various prosodic parameters, which are stored in lookup tables of target language prosodic parameters circuit 214.");.
Regarding Claim 8,
Seltur in view of Hullman further in view of Erickson further in view of Shen discloses the method of claim 1, wherein generating the text content includes inserting one or more hedge words into one or more sentences of the text content to communicate the data uncertainty (Erickson, Fig.4: a NLT input/output module system 200, paras[0070, 073-075], "…Disfluency analyzer circuit 210 stores data identifying multiple LOC ranges, which each represent a particular degree of uncertainty or confidence in the initial NL output...Disfluency selection analysis circuit 210 may add prosodic parameters of the target language the integration of disfluent speech or text..."; Fig.7, par [093], "…inserting a linguistic hedge (e.g., "I think that's right") following a low confidence NLP output...").
Regarding Claim 12,
Seltur in view of Hullman further in view of Erickson further in view of Shen discloses the method of claim 1, wherein generating the audio narrative includes inserting one or more pauses in segments of the audio narrative describing the data uncertainty (Erickson, Fig.4: a NLT input/output module system 200, paras[059, 070, 073-075], "...Disfluency analyzer circuit 210 stores data identifying multiple LOC ranges, which each represent a particular degree of uncertainty or confidence in the initial NL output...Disfluency selection analysis circuit 210 may add prosodic parameters of the target language the integration of disfluent speech or text...For example, a pause may be inserted immediately before the disfluency to add emphasis…";
Claim 13 is a computing device claim with limitations similar to the limitations of Claim 1 and is rejected under similar rationale. Rationale for combination is similar to that provided for Claim 1.
Claim 14 is a computing device claim with limitations similar to the limitations of Claim 2 and is rejected under similar rationale.
Claim 15 is a computing device claim with limitations similar to the limitations of Claim 3 and is rejected under similar rationale.
Claim 17 is a non-transitory computer-readable medium claim with limitations similar to the limitations of Claim 1 and is rejected under similar rationale. Rationale for combination is similar to that provided for Claim 1.
Claim 18 is a non-transitory computer-readable medium claim with limitations similar to the limitations of Claim 8 and is rejected under similar rationale.
Claims 5, 6 and 16 are rejected under 35 U.S.C. 103 as being unpatentable over Setlur in view of Hullman further in view Erickson further in view of Shen further in view of Fernandes et al. ("Uncertainty displays using quantile dotplots or cdfs improve transit decision-making." Proceedings of the 2018 CHI conference on human factors in computing systems. 2018.).
Regarding Claim 5,
Seltur in view of Hullman further in view of Erickson further in view of Shen discloses the method of claim 1 but does not explicitly disclose the rendering of the plurality of quantiles as data visualization.
Fernandes, in analogous field of data visualization, discloses wherein rendering the data visualization includes: dividing the data into a plurality of quantiles (Quantile Dotplots, "…Kay et al. [20] introduced quantile dotplots, which are discrete analogs to the common probability density plot based on Wilkinson dotplots [34]. They found that quantile dotplots allowed people to more precisely extract probability intervals than other common uncertainty visualizations..."); and
rendering each of the quantiles with a respective distinct visual encoding indicating the respective data uncertainty for the respective quantile (Probability Density and Interval Plot, "…Our experiment tested a PDF-interval density function hybrid taking the shape of the density function and marking the central 50% interval within the shape...").
Therefore, it would have been obvious to a person of ordinary skill in the art to apply Fernandes' known data visualization technique using a quantile dotplots to a query-responsive data visualization system displaying the uncertainty of Seltur, Hullman, Erickson, and Shen with a reasonable expectation that the combination would have yielded the predictable results of providing better uncertainty representation to viewers because quantile dotplots extract more precise probability intervals than other common uncertainty visualization (Fernandes, Quantile Dotplots).
Regarding Claim 6,
Seltur in view of Hullman further in view of Erickson further in view of Shen discloses the method of claim 1, wherein the statistics from the dataset include an average of a distribution of the data (Seltur, par [181], "…the range of predefined values includes (940) one or more of an average value, a standard deviation, and a maximum value associated with the data field…"),
Fernandes, in analogous field of data visualization, discloses a range of a middle 50% of data, a full range of the data, and a verbal representation of distribution skew (Fernandes, Interval Plot, "…Our representations plotted the 50% and 95% quantile (equi-tailed) predictive intervals from the most probable time of arrival for a bus..."). Rationale for combination is similar to that provided for Claim 5.
Claim 16 is a computing device claim with limitations similar to the limitations of Claim 5 and is rejected under similar rationale. Rationale for combination is similar to that provided for Claim 5.
Allowable Subject Matter
Claims 9-11 and 19-20 are objected to as being dependent upon rejected base claims 1 and 17 but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims. Cited prior art references, individually or in combination with other cited references, do not explicitly teach a different visual encoding and playback speed of hedge words from the remaining words.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Hammond et al. (US Pat 9,576,009) discloses techniques for automatically generating narratives about data based on communication goal data structures that are associated with configurable content blocks. Using the communication goal data structures as a guide, a computer may generate meaningful narratives by determining the content blocks and narrative analytics associated with a given communication goal data structure (Hammond, Abstract).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JANGWOEN LEE whose telephone number is (703)756-5597. The examiner can normally be reached Monday-Friday 8:00 am - 5:00 pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, BHAVESH MEHTA can be reached at (571)272-7453. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JANGWOEN LEE/ Examiner, Art Unit 2656
/BHAVESH M MEHTA/ Supervisory Patent Examiner, Art Unit 2656
Prosecution Insights