DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
The Response filed on 04 May, 2026 has been correspondingly accepted and considered in the office action. Claims 1-20 are pending. Claims 1, 19, and 20 are independent and amended.
The objection to Claims 10, 12 and 17 have been withdrawn in view of Applicant's amendments to Claim 1, which necessitated the new ground(s) of rejection.
Response to Arguments
Claims 1-9, 11, 13-16 and 18-20 stand rejected under 35 U.S.C. § 103. Applicant’s arguments with respect to 1-9, 11, 13-16 and 18-20 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
In order to expedite prosecution, and as to the material from the Specifications that are not in the Claim and are argued by the Applicant, please note Birnbaum et al. (US Pat US 10,747,823), Shen et al. ("Data player: Automatic generation of data videos with narration-animation interplay." IEEE Transactions on Visualization and Computer Graphics 30.1 (2023): 109-119), Faisman et al. (US Pub 2008/0177786), and Hullman et al. ("Hypothetical outcome plots outperform error bars and violin plots for inferences about reliability of variable ordering." PloS one 10.11 (2015))
For at least the supra provided reasons, Applicant's arguments have been fully considered but they are not persuasive.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-20 are rejected under 35 U.S.C. 103 as being unpatentable over Birnbaum et al. (US Pat US 10,747,823) in view of Hullman et al. ("Hypothetical outcome plots outperform error bars and violin plots for inferences about reliability of variable ordering." PloS one 10.11 (2015)) further in view of Shen et al. ("Data player: Automatic generation of data videos with narration-animation interplay." IEEE Transactions on Visualization and Computer Graphics 30.1 (2023): 109-119.) further in view of Faisman et al. (US Pub 2008/0177786).
Regarding Claim 1,
Birnbaum discloses a method of communicating data uncertainty (Birnbaum, Fig.1, col.5:61-col.6:3, "…a system 100 where a narrative generation artificial intelligence platform 102 is in communication with both a visualization platform 104 and a conversational interface 106 using either text or speech (such as a Chatbot)..."),
performed at a computing device having one or more processors and memory storing one or more programs configured for execution by the one or more processors (col.15:23-46, Fig.5, computer system, memory, and processors), the method comprising:
in response to a user query regarding a dataset (col.5:66-6:3, "…the narrative generation artificial intelligence platform 102 can receive instructions and queries derived from natural language input, e.g., spoken words, for controlling a visualization and/or a narrative story about a visualization..."),
wherein data in the dataset comprises a statistical distribution that includes variability (col.2:18-22, "…automatically generating narratives based on structured data, in many cases primarily structured numerical data...";Fig.7D: Metrics (738), col.18:21-47, "…The variable <METRIC> 738 can be used to refer to or denote a metric by which the measure values will be evaluated as part of the focusing effort… {Starting Value, Ending Value, Average Value, Median Value, Percent Change, Absolute Change, Volatility, ... }):
obtaining a multimodal data representation of the dataset (Fig.3D, col.13:25-29, "…FIG. 3D shows the visualization of FIG. 3A paired with a narrative...";"…This narrative 310 serves to summarize and explain important or interesting aspects of the visualization..."; col.8:50-52, "…if desired, the resulting narrative can be read to the user via a text to speech application (i.e., audio narration)...");
presenting an audio narrative that interprets the variability of the data according to the statistical distribution of the data (Fig.3D, col.13:16-24], "…captions can be automatically generated to accompany a visualization that summarize and explain the important aspects of that visualization in a natural language format..."),
detecting a user interaction with the interactive media playback element (col.8:20-29, "…the user can specify a data set to visualize; the visualization type to utilize; how the data and meta-data map to parameters of the visualization...These systems also typically provide users with the capability of adjusting various parameters of the visualization..."; col.9:39-42, "… In a graphical user interface, these are presented to the user via such interface conventions as ( clickable) labels on the chart or in drop-down menus..."); and
in response to detecting the user interaction, modifying (i) a playback portion of the visual content on the user interface (col., "…a narrative generation system, when linked to a visualization system, can respond dynamically as the user interactively controls the visualization system. as the user changes his or her selection of the data, the nature or details of the visualization, the range of values over which the visualization will be presented, etc., the narrative generation system can automatically generate a new narrative appropriate to the new visualization created in response to these directives...") and (ii) the audio narrative that is time- synchronized with the visual content.
displaying an interactive media playback element in a first region of a user interface of the computing device (Fig.8A: focus filter menu 800, col.19:14-20, "…the menu can be presented via narrative text via GUI..."):
Birnbaum discloses the method of data visualization and narrative generation based on structured numerical data of variability (e.g., percent change, absolute change, or volatility) and focused visualization and focused narratives (Fig.11, col.20:44 - col.22:30) , but does not explicitly discloses the visualization of a statistical distribution of variability. However, Hullman, in the analogous filed of visualization of quantitative data, discloses data representation of the dataset, wherein data in the dataset comprises a statistical distribution that includes variability (Hullman, 1 Introduction, "…we present and study an alternative approach for depicting distributions, which we call Hypothetical Outcome Plots (HOPs). In its most simple variant, the HOPs approach is to: (1) Draw a sample of hypothetical outcomes (draws) from the distribution..."; Fig.2, "…depicts several frames for a HOPs visualization for a single random variable. Each frame contains a horizontal bar that depicts a specific outcome, one draw for that random variable. An interactive controller allows the viewer to start/pause and step through frames.").
Therefore, it would have been obvious to a person of ordinary skill in the art to substitute Hullman's HOPs, a distribution-specific visualization technique for Birnbaum's generic chart type or focused visualization to expand the data visualization method of communicating uncertainty with a reasonable expectation to enables viewers to infer properties of the distribution using mental processes and allow visual presentation of uncertainty to shift from its current emphasis on static, abstract representation of probability distributions to dynamic, concrete presentation of hypothetical outcomes from those distributions (Hullman, Abstract, Conclusion).
Birnbaum in view of Hullman teaches the data visualization and text or audio narration of the structured numerical data of variability, but does not explicitly teaches the coordination among different modes of presentation.
Shen, in the analogous field of endeavor, discloses while presenting the audio narrative that interprets the variability of the data, simultaneously presenting visual content via a visualization in a second region of the user interface that is different from the first region, wherein the visual content is time-synchronized with the audio narrative (Shen, Fig.1: A data video example with narration-animation interplay automatically generated by Data Player from a static visualization and descriptive text.; Fig.2, 4.2 Overview, "…First, a static visualization and corresponding narration text are inputted in the form of Scalable Vector Graphics (SVGs) and plain text (a)...Text-To-Speech (TTS) techniques are used to generate audio voiceovers and return timestamps of each word, which also act as the timeline of the data video (c)... the Large Language Model (LLM) is adopted to establish the semantic links between visual components (one or a group of graphic elements) and narrative entities (one or more words) based on the data facts to be told (d)...").
Therefore, it would have been obvious to one of ordinary skill in the art, before effective filing date of the claimed invention, to have modified a data visualization and narration generation platform of Birnbaum in view of Hullman with Shen's timestamp-driven synchronization of visualization and TTS-acquired audio voiceover of Data Player with a reasonable expectation of success to lowers the technical barriers associated with creating data videos rich in narration and provide an intuitive interpretation of data charts while vividly articulating the underlying data insights (Shen, Abstract).
While Birnbaum teaches the interactive conversational or graphical user interface, where the details of the visualization could be modified, the data representation platform taught by Birnbaum in view of Hullman further in view of Shen does not explicitly teach interactive media playback element, which a user can control the navigation within the time-synchronized visualization-narration interplay.
Faisman, in the analogous field of audio and visual multimedia processing systems, discloses in response to receiving a user input via the interactive media playback element (par [019], "… the acquisition of a multimedia data stream file 100, wherein thereafter the multimedia data stream file is separated into its respective media data stream files 105 (i.e., an audio data stream and a video data stream..."; par [020], "…At 110, a transcription of the audio data stream file is created from the audio data stream...The transcription comprises synchronization information that relates the textual elements of the transcription with the original multimedia data stream file from which the transcription was derived..."),
causing playback of the multimodal data representation on the user interface (Fig.2: GUI 200, par [026], "…During a multimedia data playback operation, transcription text that is time associated with the multimedia data file is highlighted 230 at the left-side display…The timing information that is embedded into the transcription allows for the navigation from the edited text of the transcription to a relational playback position of the media file, and from the playback position of the media file to the text of the transcription...A system user has only to select a character, word, or phrase in the text, and the multimedia file will travel to the corresponding synchronized point within the multimedia playback." ).
Therefore, it would have been obvious to one of ordinary skill in the art, before effective filing date of the claimed invention, to have modified the combined system generating a synchronized narration and animation (e.g., visualization) of Birnbaum in view of Hullman further in view of Shen with the user-selected playback control of transcribed text highlighting and synchronization of the transcript with multimedia of Faisman with a reasonable expectation of success to establish a single playback position input for the timestamp-linked multi-modal (text, audio, animation, video) data representation system without requiring new synchronization mechanism (Faisman, paras [005]).
Regarding Claim 2,
Birnbaum in view of Hullman further in view of Shen further in view of Faisman discloses the method of claim 1, further comprising: while simultaneously presenting the audio narrative and the visual content, concurrently presenting text content in a third region of the user interface that is different from the first region and the second region, wherein the text content is time-synchronized with both the audio narrative and the visual content (Faisman, Fig.3, par [026], "…The highlighted text 230 is associated with the current playback time position 220 of the multimedia data. The playback speed of the multimedia data and the playback length of the multimedia data file are also respectively shown at 225 and 215...").
Regarding Claim 3,
Birnbaum in view of Hullman further in view of Shen further in view of Faisman discloses the method of claim 2, wherein the visual content and the text content are timed-synchronized with the audio narrative according to a timestamp of the audio narrative (Faisman, Fig2: GUI 200, par [026], "…The timing information that is embedded into the transcription allows for the navigation from the edited text of the transcription to a relational playback position of the media file, and from the playback position of the media file to the text of the transcription...").
Regarding Claim 4,
Birnbaum in view of Hullman further in view of Shen further in view of Faisman discloses the method of claim 2, wherein the text content is a text transcript of the audio narrative (Faisman, par [020], "…At 110, a transcription of the audio data stream file is created from the audio data stream...")
Regarding Claim 5,
Birnbaum in view of Hullman further in view of Shen further in view of Faisman discloses the method of claim 4, wherein presenting the text content includes presenting the text transcript sentence-by-sentence in the third region of the user interface (see Faisman, Fig.3; Shen, Fig.4).
Regarding Claim 6,
Birnbaum in view of Hullman further in view of Shen further in view of Faisman discloses the method of claim 2, wherein: the text content includes hedge words; and presenting the text content in the third region of the user interface includes presenting the hedge words with a different visual characteristic than other text in the text content (Birnbaum, Fig.33, col.27:47-60, Additional Analytics Configurations-Formatting, "…the text describing positive changes is highlighted in green, while the text describing negative changes is highlighted in red."; A person of ordinary skill in the field of data visualization would understand that similar formatting rules could be applied to one or more numerical hedge words, which express the approximation or uncertainty.).
Regarding Claim 7,
Birnbaum in view of Hullman further in view of Shen further in view of Faisman discloses the method of claim 2, wherein the text content includes data values of a data field; and presenting the text content in the third region of the user interface includes presenting the data values with a different visual characteristic from other text in the text content (Birnbaum, Fig.33, col.27:47-60, Additional Analytics Configurations-Formatting, "…the text describing positive changes (i.e., data values) is highlighted in green, while the text describing negative changes is highlighted in red.").
Regarding Claim 8,
Birnbaum in view of Hullman further in view of Shen further in view of Faisman discloses the method of claim 7, further comprising: when the audio narrative corresponds to a respective data value of the data field, visually emphasizing the respective data value in the text content (both Shen (e.g., timestamps of each word) and Faisman (e.g., timing information embedded into the transcription) teaches time-synchronization with audio narrative and text.).
Regarding Claim 9,
Birnbaum in view of Hullman further in view of Shen further in view of Faisman discloses the method of claim 1, wherein simultaneously presenting the visual content while presenting the audio narrative includes presenting the visual content as an animated dot plot that is time-synchronized with the audio narrative (Shen, Fig.1 shows the time-synchronized animation of the bar graph with audio narration and text.; Fig.4: Automatic-generated data videos by Data Player. The snapshot images show the effect after the animation has been triggered; A person of ordinary skill in the field of data visualization would understand as common general knowledge that other types of static data charts (e.g., dot plot) could be animated.).
Regarding Claim 10,
Birnbaum in view of Hullman further in view of Shen further in view of Faisman discloses the method of claim 9, wherein the animated dot plot includes: movement of data points of the dot plot from a virtual source point in the user interface; and arrangement of the data points on predefined positions of the dot plot (Shen, Fig.4: Automatic-generated data videos by Data Player. The snapshot images show the effect after the animation has been triggered, 4.3 Tex-Visual Linking, "…To generate data videos with narration-animation interplay, it is crucial to understand the narration text and its relations with the visual elements...").
Regarding Claim 11,
Birnbaum in view of Hullman further in view of Shen further in view of Faisman discloses the method of claim 2, wherein simultaneously presenting the visual content while presenting the audio narrative includes presenting the visual content as an animated density plot that is time-synchronized with the audio narrative (Shen, Fig.1 shows the time-synchronized animation of the data chart with audio narration and text. A person of ordinary skill in the field of data visualization would understand as common general knowledge that other types of static data charts (e.g., density plot) could be animated.).
Regarding Claim 12,
Birnbaum in view of Hullman further in view of Shen further in view of Faisman discloses method of claim 11, wherein the animated density plot includes movement of the density plot relative to a centroid position of the density plot (Shen, 4.4 Animation Sequence Generation, "…To align the semantic intents between the linked narration segments and animated visual elements, we encode a variety of constraints to assign appropriate animations from the pre-defined library to visual elements based on the data facts being presented, the visual structure, and the desired audience engagement..."; 4.4.3 Animation Presets, "…depending on different chart types and element orientations, the configurations of one animation effect are adjusted...").
Regarding Claim 13,
Birnbaum in view of Hullman further in view of Shen further in view of Faisman discloses method of claim 2, wherein: the dataset includes data values of a first data field; and simultaneously presenting the visual content and the text content while presenting the audio narrative includes:
when the audio narrative corresponds to a respective data value of the first data field, simultaneously visually emphasizing: one or more portions of the visual content corresponding to the respective data value; and a portion of the text content that matches the respective data value (Shen, Fig.4: Automatic-generated data videos by Data Player. The snapshot images show the effect after the animation has been triggered, 4.3 Tex-Visual Linking, "…To generate data videos with narration-animation interplay, it is crucial to understand the narration text and its relations with the visual elements...").
Regarding Claim 14
Birnbaum in view of Hullman further in view of Shen further in view of Faisman discloses the method of claim 1, further comprising: prior to causing playback of the multimodal data representation on the user interface, displaying, in the user interface, a plurality of affordances, each affordance of the plurality of affordances corresponding to a respective visualization type for visualizing the visual content (Birnbaum, col.8:20-29, "…the user can specify a data set to visualize; the visualization type to utilize; how the data and meta-data map to parameters of the visualization...These systems also typically provide users with the capability of adjusting various parameters of the visualization..."; col.9:39-42, "… In a graphical user interface, these are presented to the user via such interface conventions as ( clickable) labels on the chart or in drop-down menus..."; Fig.8A: focus field menu 800); and
the method further includes: in response to user selection of a first affordance of the plurality of affordances, corresponding to a first visualization type, presenting the visual content in the first visualization type (Birnbaum, Fig.9A-B: an example process flow for the logic used to generate a focus configuration for use with a visualization (A) and focus configuration data structure (B), Fig.10: an example screenshot showing a visualization that has been focused based on input received through a focus filter menu).
Regarding Claim 15,
Birnbaum in view of Hullman further in view of Shen further in view of Faisman discloses the method of claim 2, wherein: the audio narrative describing the data includes a first data field and one or more hedge words; and concurrently presenting the text content while simultaneously presenting the audio narrative and the visual content includes displaying the text content so that one or more data values of the first data field have a first visual characteristic and the one or more hedge words have a second visual characteristic that is different from the first visual characteristic (Birnbaum, Fig.33, col.27:47-60, Additional Analytics Configurations-Formatting, "…the text describing positive changes (i.e., data values) is highlighted in green, while the text describing negative changes is highlighted in red...a user first applies the formatting rules for 'good changes' and 'bad changes'. These choices are then used when generating the narrative once a user chooses to enable formatting..."; A person of ordinary skill in the field of data visualization would understand that similar formatting rules could be applied to numerical hedge words.).
Regarding Claim 16,
Birnbaum in view of Hullman further in view of Shen further in view of Faisman discloses the method of claim 15, further comprising: in response to receiving a user interaction with a first hedge word of the one or more hedge words, causing the first hedge word to be displayed with a third visual characteristic that is distinct from the first and second visual characteristics (Birnbaum, Fig.33, col.27:47-60, Additional Analytics Configurations-Formatting, "…the text describing positive changes (i.e., data values) is highlighted in green, while the text describing negative changes is highlighted in red...a user first applies the formatting rules for 'good changes' and 'bad changes'. These choices are then used when generating the narrative once a user chooses to enable formatting..."; A person of ordinary skill in the field of data visualization would understand that similar formatting rules could be applied to one or more numerical hedge words.).
Regarding Claim 17,
Birnbaum in view of Hullman further in view of Shen further in view of Faisman discloses the method of claim 16, wherein the third visual characteristic comprises movement of the first hedge word with respect to a centroid position of the first hedge word (Shen, 4.4 Animation Sequence Generation, "…To align the semantic intents between the linked narration segments and animated visual elements, we encode a variety of constraints to assign appropriate animations from the pre-defined library to visual elements based on the data facts being presented, the visual structure, and the desired audience engagement..."; 4.4.3 Animation Presets, "…depending on different chart types and element orientations, the configurations of one animation effect are adjusted..."; it is construed that the hedge words are visual element, not text, and treated as such.).
Regarding Claim 18,
Birnbaum in view of Hullman further in view of Shen further in view of Faisman discloses the method of claim 1, further comprising: after playback of the multimodal data representation, displaying a replay icon on the user interface; and in response to receiving user selection of the replay icon, replaying the multimodal data representation on the user interface (Faisman, Fig.2, par [025], "…GUI 200 (i.e., user interface)...The right-side display 210 further comprises multimedia controls, thus allowing for the control of the listening/viewing aspects of a multimedia data…").
Claim 19 is a computer device claim with limitations similar to the limitations of Claim 1 and is rejected under similar rationale. Rationale for combination is similar to that provided for Claim 1.
Claim 20 is a non-transitory computer-readable storage medium claim with limitations similar to the limitations of Claim 1 and is rejected under similar rationale. Rationale for combination is similar to that provided for Claim 1.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Kurzweil et al. (US Pub 2015/0088505) discloses techniques and systems to provide a narration of a text. In some aspects, the techniques and systems described herein include generating a timing file that includes elapsed time information for expected portions of text that provides an elapsed time period from a reference time in an audio recording to each portion of text in recognized portions of text (Kurzweil, section).
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to JANGWOEN LEE whose telephone number is (703)756-5597. The examiner can normally be reached Monday-Friday 8:00 am - 5:00 pm ET.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, BHAVESH MEHTA can be reached at (571)272-7453. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JANGWOEN LEE/Examiner, Art Unit 2656
/BHAVESH M MEHTA/Supervisory Patent Examiner, Art Unit 2656
Prosecution Insights