DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or nonobviousness.
Claim(s) 1-6, 8-13 and 15-20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Doggett et al. (US Patent Application Publication No. 2020/0019370) in view of Knipp et al. (US Patent Application Publication No. 2016/0225187).
Regarding claim 1, Doggett discloses a method for generating a narrative with corresponding visuals from a user prompt, the method comprising [see Abstract; AI systems that offer an improvisational story telling AI agent that may interact collaboratively with a user. A story telling device may be implemented using i) a natural language understanding (NLU) component to process human language input (e.g., digitized speech or text input); ii) a natural language processing (NLP) component to parse the human language input into a story segment or sequence; iii) a component for storing/recording the story as it is created by collaboration; iv) a component for generating AI-suggested story elements; and v) a natural language generation (NLG) component to transform the AI-generated story segment into natural language that may be presented to the user]:
receiving, by a virtual assistant bot, a request to create a story, wherein the request includes a user-provided prompt that includes a narrative seed from which to create the story [see para. 0030, -0035 and figures 1-2; a natural language understanding (NLU) component to process human language input (e.g., digitized speech or text input); ii) a natural language processing (NLP) component to parse the human language input into a story segment or sequence; iii) a component for storing/recording the story as it is created by collaboration; iv) a component for generating AI-suggested story elements; and v) a natural language generation (NLG) component to transform the AI-generated story segment into natural language that may be presented to the user. In implementations involving vocal interaction between the user and storytelling device, the device may additionally implement a speech synthesis component for transforming the textual natural language generated by the NLG component into auditory speech; which corresponds to receive human language input from a user as storytelling];
generating, by the virtual assistant bot, a narrative generation prompt from a combination of a system prompt and the user-provided prompt [see para. 0046-0056 and figures 3-6; Story generator component configured to generate AI-suggested story segments. The generated suggestion for continuing the story, whether that involves writing a narrative or plot point, or expanding upon character, settings, etc. During operation, there may be full cross-reference between story record component and story generator component to allow referencing of characters and previous story steps; which corresponds to generate storytelling using AI techniques];
sending, by the virtual assistant bot, the narrative generation prompt to a generative content generation service, wherein the content generation service generates narrative text corresponding to the story based on the narrative generation prompt, and further generates visual media prompts corresponding to the narrative text [see para. 0069-0072; the story-writing may be accompanied by automated audio and visual representations of the story as it is being developed. In a VR or AR system, as each agent—human, and AI—suggest a story segment, the story segment may be represented in an audiovisual VR or AR representation around the human participant (e.g., during operation). For example, if a story segment is “and then the princess galloped off to save the prince,” there may appear a representation of a young woman with a crown on horseback, galloping across the visual field of the user. Text-to-video and text-to-animation components may be utilized at this phase for visual story rendering. In AR/VR implementations, any presented VR/AR objects (e.g., characters) may adapt to the environment of the user collaborating with the AI for storytelling; which corresponds to display AR/VR to the narrative text];
generating, by the content generation service, visuals in response to the visual media prompts [see para. 0073; the generated AI story segments may be based, at least in part, on detected environmental conditions. For example, temperature (e.g., as measured in the user's vicinity), time of day (e.g., daytime or nighttime), time of year (e.g., season), the date (e.g., current day of the week, current month, and/or current year), weather conditions (e.g., outside temperature, whether it is rainy or sunny, humidity, cloudiness, fogginess, etc.), location (e.g., location of user collaborating with the AI storytelling agent, whether the location is inside or outside a building, etc.), or other conditions may be sensed or otherwise retrieved (e.g., via geolocation), and incorporated into generated AI story segments; which corresponds to adapting visuals to detected conditions like location and time]; however, Doggett fails to explicitly teach receiving, by a story application and from the content generation service, the narrative text and the visuals in the style defined by the narrative generation prompt; synchronizing, by the story application, the visuals with the narrative text to create the story; and presenting, by the story application, the story to the user, wherein the story is presented as a series of segments.
Knipp discloses receiving, by a story application and from the content generation service, the narrative text and the visuals in the style defined by the narrative generation prompt [see para, 0024-0025 and figures 1C-2B; the Narratarium are designed to augment storytelling and creative play of the user, which include a child or a parent. The story experience provided of Narratarium include story elements (such as plotlines, characters, settings, themes, duration, or other aspects of a story) based on information provided by the user and/or the environment, including information provided in real time, as well as information derived from printed stories, audio recordings, toys, or other sources. Thus, the Narratarium considered to be context aware. For example, as a child tells a story about a jungle, the child's room is filled with greens and browns and foliage comes into view. Animals that live in the jungle introduced or suggested as characters to the story; which corresponds to context environmental data to form inputs for a generative service and visual media with images and video]; synchronizing, by the story application, the visuals with the narrative text to create the story; and presenting, by the story application, the story to the user, wherein the story is presented as a series of segments [see para. 0060; one possible embodiment of production component generates such a structure, which a coded story. It also illustrates how a dynamic story instantiated from a structure by storytelling engine, wherein the block templates or placeholders are filled with specific story content for the story being presented, placeholders may be used for story elements including not only characters, settings, sound effects, visual images, animations, videos, etc., but also story logic; prompts or guidance provided by story guide; story blocks or threads; story element positions and/or motion paths (which may be predetermined from among a number of positions or motion paths and wherein the beginning or ending of the path is the placeholder to be determined from near storytelling time or from an environmental model); or any other aspect of storytelling content that may desired to be modified near storytelling time by storytelling engine; which corresponds to a series of adaptive segements and a presentation story application that receive narrative text and visuals].
It would have been obvious to one of an ordinary skill in the art, having the teachings of Doggett and Knipp before the affective filing date of the claimed invention to modify, collaborative AI narrative generation system of Doggett to incorporate immersive visual generation and synchronization techniques, as taught by Knipp.
One would have been motivated to make such a combination in order to enhance user engagement by making the collaborative stories more visually immersive and contextually integrated onto the real world and applying techniques to achieve in storytelling interactivity.
Regarding claim 2, Doggett discloses wherein the presenting the story to the user comprises: generating, by a text to speech engine of the story application, voice audio corresponding to the narrative text to result in a narrated presentation of the story to the user [see para. 0030; 0033 and ; an improvisational storytelling device implemented using i) a natural language understanding (NLU) component to process human language input (e.g., digitized speech or text input); ii) a natural language processing (NLP) component to parse the human language input into a story segment or sequence; iii) a component for storing/recording the story as it is created by collaboration; iv) a component for generating AI-suggested story elements; and v) a natural language generation (NLG) component to transform the AI-generated story segment into natural language that presented to the user. Involving vocal interaction between the user and storytelling device, the device additionally implement a speech synthesis component for transforming the textual natural language generated by the NLG component into auditory speech; which corresponds to a component for storing/recording the story as it is created by collaboration and a component for generating AI-suggested story elements].
Regarding claim 3, Doggett discloses wherein the system prompt includes characteristics derived from personal information of the user to establish a style for the story [see para. 0044-0045; a suitable speech API such as a GOOGLE speech to text API or AMAZON speech to text API used. A local speech-to-text/NLU model may be run without using an internet connection, which may increase security and allow the user to have full control over their private language data. NLP story parser component configured to parse the human natural language input into a story segment. The human natural language input parsed into suitable or appropriate word or token segments to identify/classify keywords such as character names and/or actions corresponding to a story, and to extract additional language information such as part-of-speech category, syntactic relational category, content versus function word identification, conversion into semantic vectors, among others; which characteristics derived from personal information of the user].
Regarding claim 4, Knipp discloses wherein the content generation service is a multi-modal content generation service capable of generating at least two or more of text, images, and video [see para. 0086; determining content placement based on presentation environment information, such as provided from an environmental model created by generator. When projecting video and images in a presentation environment, it is difficult to anticipate the placement of physical objects like windows, doors, furniture, and even human occupants. Accordingly, include functionality for allowing visual story content (e.g., videos and image) to be placed or projected in a more optimal location within the presentation environment].
Regarding claim 5, Doggett discloses wherein the system prompt includes an output format instruction; and the content generation service provides the story in an output format defined in the output format instruction [see para. 0039; storage stores story generation software, that when executed by a processing component (e.g., a digital signal processor), causes device to perform collaborative AI storytelling functions such as collaboratively generating a story with a user, storing a record of the generated story, and causing speaker to output generated story languages in natural language. In implementations where story generation software is used in an AR/VR environment where device is a HMD, execution of story generation software also cause the HMD to display AR/VR visual elements corresponding to a storytelling experience].
Regarding claims 6 and 20, Knipp discloses further comprising: prior to generating the narrative generation prompt, responding to the request to create the story with a conversational cue directed to a user that provided the request to create the story, wherein the conversational cue encourages the user to respond with additional details for inclusion in the narrative generation prompt [see para. 0068, 0072; a narrative development environment (NDE) that provides the story producer with a human-readable story experience editor and then compiles that narrative to a form that runs on or is usable by a storytelling platform of Narratarium. Such embodiments of production component therefore provide a way for writers, designers, and other producers to more easily describe and create a story and its characters in detail, in a way that best suits their most comfortable way of thinking and working. For example, one embodiment of production component enables a producer (which may also be a Narratarium user) to employ a variety of possible ways to use graphical user interface (GUI) tools to define story elements including, for example: story characters, the general narrative flow of the end user experience, and the narrative structure of that experience. The coded story representation exported or generated by production component takes the form of a markup language like XML, wherein production component uses typical PC GUI elements to tag and describe specific story elements].
Regarding claim 8, Doggett discloses a system for generating a narrative with corresponding visuals from a user prompt, the system comprising: one or more processors; and at least one computer-readable storage medium having stored therein instructions which, when executed by the one or more processors, cause the one or more processors to perform operations comprising [see para. 0039; storage stores story generation software, that when executed by a processing component (e.g., a digital signal processor), causes device to perform collaborative AI storytelling functions such as collaboratively generating a story with a user, storing a record of the generated story]:
receiving, by a virtual assistant bot, a request to create a story, wherein the request includes a user-provided prompt that includes a narrative seed from which to create the story [see para. 0030, -0035 and figures 1-2; a natural language understanding (NLU) component to process human language input (e.g., digitized speech or text input); ii) a natural language processing (NLP) component to parse the human language input into a story segment or sequence; iii) a component for storing/recording the story as it is created by collaboration; iv) a component for generating AI-suggested story elements; and v) a natural language generation (NLG) component to transform the AI-generated story segment into natural language that may be presented to the user. In implementations involving vocal interaction between the user and storytelling device, the device may additionally implement a speech synthesis component for transforming the textual natural language generated by the NLG component into auditory speech; which corresponds to receive human language input from a user as storytelling];
generating, by the virtual assistant bot, a narrative generation prompt from a combination of a system prompt and the user-provided prompt [see para. 0046-0056 and figures 3-6; Story generator component configured to generate AI-suggested story segments. The generated suggestion for continuing the story, whether that involves writing a narrative or plot point, or expanding upon character, settings, etc. During operation, there may be full cross-reference between story record component and story generator component to allow referencing of characters and previous story steps; which corresponds to generate storytelling using AI techniques];
sending, by the virtual assistant bot, the narrative generation prompt to a generative content generation service, wherein the content generation service generates narrative text corresponding to the story based on the narrative generation prompt, and further generates visual media prompts corresponding to the narrative text [see para. 0069-0072; the story-writing may be accompanied by automated audio and visual representations of the story as it is being developed. In a VR or AR system, as each agent—human, and AI—suggest a story segment, the story segment may be represented in an audiovisual VR or AR representation around the human participant (e.g., during operation). For example, if a story segment is “and then the princess galloped off to save the prince,” there may appear a representation of a young woman with a crown on horseback, galloping across the visual field of the user. Text-to-video and text-to-animation components may be utilized at this phase for visual story rendering. In AR/VR implementations, any presented VR/AR objects (e.g., characters) may adapt to the environment of the user collaborating with the AI for storytelling; which corresponds to display AR/VR to the narrative text];
generating, by the content generation service, visuals in response to the visual media prompts [see para. 0073; the generated AI story segments may be based, at least in part, on detected environmental conditions. For example, temperature (e.g., as measured in the user's vicinity), time of day (e.g., daytime or nighttime), time of year (e.g., season), the date (e.g., current day of the week, current month, and/or current year), weather conditions (e.g., outside temperature, whether it is rainy or sunny, humidity, cloudiness, fogginess, etc.), location (e.g., location of user collaborating with the AI storytelling agent, whether the location is inside or outside a building, etc.), or other conditions may be sensed or otherwise retrieved (e.g., via geolocation), and incorporated into generated AI story segments; which corresponds to adapting visuals to detected conditions like location and time]; however, Doggett fails to explicitly teach receiving, by a story application and from the content generation service, the narrative text and the visuals in the style defined by the narrative generation prompt; synchronizing, by the story application, the visuals with the narrative text to create the story; and presenting, by the story application, the story to the user, wherein the story is presented as a series of segments.
Knipp discloses receiving, by a story application and from the content generation service, the narrative text and the visuals in the style defined by the narrative generation prompt [see para, 0024-0025 and figures 1C-2B; the Narratarium are designed to augment storytelling and creative play of the user, which include a child or a parent. The story experience provided of Narratarium include story elements (such as plotlines, characters, settings, themes, duration, or other aspects of a story) based on information provided by the user and/or the environment, including information provided in real time, as well as information derived from printed stories, audio recordings, toys, or other sources. Thus, the Narratarium considered to be context aware. For example, as a child tells a story about a jungle, the child's room is filled with greens and browns and foliage comes into view. Animals that live in the jungle introduced or suggested as characters to the story; which corresponds to context environmental data to form inputs for a generative service and visual media with images and video]; synchronizing, by the story application, the visuals with the narrative text to create the story; and presenting, by the story application, the story to the user, wherein the story is presented as a series of segments [see para. 0060; one possible embodiment of production component generates such a structure, which a coded story. It also illustrates how a dynamic story instantiated from a structure by storytelling engine, wherein the block templates or placeholders are filled with specific story content for the story being presented, placeholders may be used for story elements including not only characters, settings, sound effects, visual images, animations, videos, etc., but also story logic; prompts or guidance provided by story guide; story blocks or threads; story element positions and/or motion paths (which may be predetermined from among a number of positions or motion paths and wherein the beginning or ending of the path is the placeholder to be determined from near storytelling time or from an environmental model); or any other aspect of storytelling content that may desired to be modified near storytelling time by storytelling engine; which corresponds to a series of adaptive segements and a presentation story application that receive narrative text and visuals].
It would have been obvious to one of an ordinary skill in the art, having the teachings of Doggett and Knipp before the affective filing date of the claimed invention to modify, collaborative AI narrative generation system of Doggett to incorporate immersive visual generation and synchronization techniques, as taught by Knipp.
One would have been motivated to make such a combination in order to enhance user engagement by making the collaborative stories more visually immersive and contextually integrated onto the real world and applying techniques to achieve in storytelling interactivity.
Regarding claims 9-13, directly or indirectly dependent on claim 8, essentially correspond to those of claims 9-13 respectively. Accordingly, the same reasoning as in claims 2-6 applies to claims 9-13.
Regarding claim 15 is an independent claim and relates to a non-transitory computer-readable storage medium having stored therein instructions which, when executed by one or more processors, cause the one or more processors to perform operations comprising. Since the features of claim 15 are substantially the same as those of claim 8 except for the category of invention, the same reasoning as in claim 8 applies to claim 15.
Regarding claims 16-20, directly or indirectly dependent on claim 15, essentially correspond to those of claims 2-6 respectively. Accordingly, the same reasoning as in claims 2-6 applies to claims 16-20.
Allowable Subject Matter
Claims 7 and 14 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure (See PTO-892).
Platt et al. (US Patent No. 11, 170,038) discloses Narrative generation techniques can be used in connection with data visualization tools to automatically generate narratives that explain the information conveyed by a visualization of a data set. In example embodiments, new data structures and artificial intelligence (AI) logic can be used by narrative generation software to map different types of visualizations to different types of story configurations that will drive how narrative text is generated by the narrative generation software.
Mohanty et al. (US 2022/0310083) discloses a method comprises receiving at least one natural language input, converting the at least one natural language input to a graphical input, retrieving relationship data from a graph database based at least in part on the graphical input, and generating at least one natural language response of a virtual assistant to the at least one natural language input based at least in part on the relationship data from the graph database. At least the generating is performed using one or more machine learning models.
A reference to specific paragraphs, columns, pages, or figures in a cited prior art reference is not limited to preferred embodiments or any specific examples. It is well settled that a prior art reference, in its entirety, must be considered for all that it expressly teaches and fairly suggests to one having ordinary skill in the art. Stated differently, a prior art disclosure reading on a limitation of Applicant's claim cannot be ignored on the ground that other embodiments disclosed were instead cited. Therefore, the Examiner's citation to a specific portion of a single prior art reference is not intended to exclusively dictate, but rather, to demonstrate an exemplary disclosure commensurate with the specific limitations being addressed. In re Heck, 699 F.2d 1331, 1332-33,216 USPQ 1038, 1039 (Fed. Cir. 1983) (quoting In re Lemelson, 397 F.2d 1006,1009, 158 USPQ 275, 277 (CCPA 1968)). In re: Upsher-Smith Labs. v. Pamlab, LLC, 412 F.3d 1319, 1323, 75 USPQ2d 1213, 1215 (Fed. Cir. 2005); In re Fritch, 972 F.2d 1260, 1264, 23 USPQ2d 1780, 1782 (Fed. Cir. 1992); Merck & Co. v. Biocraft Labs., Inc., 874 F.2d 804, 807, 10 USPQ2d 1843, 1846 (Fed. Cir. 1989); In re Fracalossi, 681 F.2d 792,794 n.1,215 USPQ 569, 570 n.1 (CCPA 1982); In re Lamberti, 545 F.2d 747, 750, 192 USPQ 278, 280 (CCPA 1976); In re Bozek, 416 F.2d 1385, 1390, 163 USPQ 545, 549 (CCPA 1969).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CAO H NGUYEN whose telephone number is (571)272-4053. The examiner can normally be reached on Mon-Fri 9am-5pm.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kieu Vu can be reached on 571-272-4057. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of an application may be obtained from the Patent Application Information Retrieval (PAIR) system. Status information for published applications may be obtained from either Private PAIR or Public PAIR. Status information for unpublished applications is available through Private PAIR only. For more information about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative or access to the automated information system, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CAO H NGUYEN/ Primary Examiner, Art Unit 2171