DETAILED ACTION
This examination is in response to the communication filed on 07/29/2025. Claims 1, 3, 5, 7-10, 12, 13 and 15-24 are currently pending, where claims 2, 4, 6, 11 and 14 have been canceled, claims 1, 3, 7, 9-10, 12-13, 15-17 and 19 have been amended and new claims 20-25.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment/Arguments
Applicant’s amendments to independent claims 1, 10 and 13 to specifically recite the structural connections between the detection network and the generative neural network overcome the rejection under §101. Thus, the rejection has been withdrawn.
Applicant’s arguments with respect to claims 1, 3, 5, 7-10, 12, 13, and 15-19 have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Claim Rejections - 35 USC § 112
The following is a quotation of the first paragraph of 35 U.S.C. 112(a):
(a) IN GENERAL.—The specification shall contain a written description of the invention, and of the manner and process of making and using it, in such full, clear, concise, and exact terms as to enable any person skilled in the art to which it pertains, or with which it is most nearly connected, to make and use the same, and shall set forth the best mode contemplated by the inventor or joint inventor of carrying out the invention.
Claims 23-25 are rejected under 35 U.S.C. 112(a) as failing to comply with the written description requirement. The claim(s) contains subject matter which was not described in the specification in such a way as to reasonably convey to one skilled in the relevant art that the inventor or a joint inventor, or for applications subject to pre-AIA 35 U.S.C. 112, the inventor(s), at the time the application was filed, had possession of the claimed invention.
Although the paragraph of page 13 of the Specification as originally filed describes “Not all parts of a story may qualify as chunks to base images on. For example…sentences containing linking verbs may not quality as chunks to be input to the diffusion model to generate an image”, the specification fails to provide any written description for the specific limitations of added claims 23-25. For example, there is no description of “determining for each word in the segment, a classification of the word as a part-of-speech” as recited in claims 23 and 25 nor is there any description of “the one or more criteria comprises a threshold number of words in a segment having a particular part-of-speech classification” as recited in claim 24.
Claim Rejections - 35 USC § 103
The text of those sections of Title 35, U.S. Code not included in this action can be found in a prior Office action.
Claims 1, 3, 5, 7-10, 12, 13, and 15-25 are rejected under 35 U.S.C. 103 as being unpatentable over Liu et al. (US 20230118966 A1; herein “Liu”) in view of Wang et al. (US 2022/0309597 A1; herein “Wang”), further in view of Zhu et al. (“A text-to-picture synthesis system for augmenting communication”. In AAAI , Vol. 7, July 2007, pp. 1590-1595; herein “Zhu”).
Regarding claims 1, 10, and 13 Liu teaches a system comprising:
one or more processors (Fig. 1, processor 104; Fig. 5, Logic Machine 502 and ¶[0033] teaches “the logic machine 502 may include one or more processors configured to execute software instructions…” ); and
one or more tangible, non-transitory media (¶[0010] teaches “the non-volatile memory 110 stores a story video generation program 112, which contains instructions for the various software modules described herein for execution by the processor 104”) operably connectable to the one or more processors and storing instructions that, when executed, cause the one or more processors to perform operations; an apparatus and a method comprising:
providing, as input to at least one generative neural network, the classification of the object, wherein the at least one generative neural network is trained to generate stories about objects based on classifications of the objects (¶[0011] teaches “the user input 114 may be provided in various formats…Other forms of user input 114, such as audio input, image input, etc., can also be utilized”) and ¶[0012] teaches “In other implementations, the user input 114 is an audio input, image input, etc., and the story video generation module 116 can use the audio input or image input, etc., to generate the story text…the story text generation module 116 includes a sequence-to-sequence transformer model 120 that can be used to generate a new sentence or phrase from a word, a phrase, a sentence, or multiple sentences such that new sentence or phrase has contextual coherence with the user input from which it is generated” );
receiving, as output from the at least one generative neural network, a text story about the object (Fig. 4, step 404 and ¶[0024] teaches at step 404, a story text is generated; ¶[0012] teaches “the user input 114 is an audio input, image input, etc., and the story video generation module 116 can use the audio input or image input, etc., to generate the story text 118”);
providing, as input to an image generator, the subset of segments of the text story (¶[0021] teaches “FIG. 3 shows a diagram schematically illustrating an example framework 300 for generating story images 302 based on a user input 304...The process can be repeated for each story image 302 that is to be generated... In other implementations, a story image 302 is generated for every two or more sentences and/or phrases.” );
receiving, as output from the image generator, a plurality of images, wherein each image is related to a respective segment of the subset of segments of the text story (¶[0013] teaches “The story video generation program 112 includes a story image generation module 122 that receives the story text 118. The story image generation module 122 uses the story text 118 to generate a plurality of story images 124” ); and
presenting the plurality of images on at least one display (¶[0028] teaches “At step 412, a story is outputted. The story includes the story text and a story video with content corresponding to the story text” and ¶[0031] teaches “Computing system 500 may optionally include a display subsystem 506” ).
Liu fails to explicitly disclose obtaining, from a camera, image data representing an image of an object in a real world environment; providing, as input to a detection network the image of the object; and receiving, as output from the detection network, a classification of the object.
Wang teaches an image processing apparatuses and systems implementing deep learning architectures that can learn high-quality representations of images (e.g., of real estate images of properties). (Wand, Abstract.) Mores specifically, Wang teaches instructions executable by at least one processor assembly to:
obtaining, from a camera, image data representing an image of an object in real world environment (¶[0054] teaches “image processing apparatus 300 may include a camera 350…The camera 350 may be operable for recording or capturing images…” and ¶[0105] teaches “At operation 1020, the system obtains (e.g., or receive) a query image…in some case, the operations of this step refer to, or may be performed by, a camera as described with reference to FIG. 3”);
providing, as input to a detection network, the image of the object (Fig. 6, Image Processing Apparatus 605; and ¶[0071] teaches “Fig. 5 shows an example image processing system 500 implementing a combination of transformers…to obtain an image representation from input images (e.g., embedded representation of one or more images of a real estate property)…”);
receiving, as output from the detection network, a classification of the object (Fig. 6, Image Classification Output; Fig. 9, Object Detection Network 925; and ¶[0022] teaches “image processing apparatus 120 may annotate an input image with grammatical information such a real estate labels (e.g., bedroom), detected objects (e.g., bed, dresser, sofa, table, refrigerator, etc.)…” In addition, ¶[0035] teaches “image processing apparatus 205 may implement one or more aspects of object detection techniques to generate an image representation of an image…In some examples, computer vision applications 210 perform object detection to analyze image data, e.g., to identify people or objects in an image”);
Liu differs from the claimed invention, as defined by claims 1, 10 and 13, in that Liu fails to specifically disclose that the input image obtained from a camera capturing at least one object in a real world environment and the objects are classified using a detection network. Classifying real world objects from input images using a detection network is known in the art as evidenced by Wang. Therefore, it would have been obvious to one having ordinary skill in the art, before the effective filing date of the invention, to have modified the story generation system of Liu to include classifying objects in the input image using a detection network as taught by Wang as it merely constitutes the combination of know processes to achieve the predictable result of classifying the objects represented in the input image.
The combination of Liu and Wang fails to explicitly disclose dividing the test story into a plurality of segments and selecting, from the plurality of segments, a subset of segments for generating images according to one or more criteria.
Zhu teaches a text-to-picture synthesis system that includes, inter alia, dividing the test story into a plurality of segments (p. 1591, 1st column teaches “Let the input text be a word sequence W1:n of length n…we first use natural language processing techniques to select k keyphrases (important words or phrases found within W1:n) to ‘draw’…”); selecting, from the plurality of segments, a subset of segments for generating images according to one or more criteria (p. 1591, 1st column teaches “Given a piece of text, e.g., a sentence or a whole book, the first question is, which keyphrases should be selected to the picture?”).
The combination of Liu and Wang differs from the claimed invention, as defined in claims 1, 10 and 13, in that the combination fails to explicitly disclose selecting a subset of segments of the text story for image generation. Selecting key segments for a larger text story to image generation is known in the art as evidenced by Zhu. Therefore, it would have been obvious to one having ordinary skill in the before the effective filing date of the invention to have modified the text-to-image generating system taught by the combination of Liu and Wang to include selecting keyphrases/segments of the text story for image generation as taught by Zhu as it merely constitutes the combination of known processes to the achieve the predictable result to generating images for those segments/sentences of text having the most informative regarding the context of the story.
Regarding claims 3 and 22, the combination of Liu, Wang and Zhu teaches all of the elements of claim 1 and 10 (see detailed element listing above). In addition, Liu further teaches the operations comprise for an image of the plurality of images: presenting, on the at least one display, the image and the respective segment of the text story (¶[0016] teaches “The plurality of animated story images 130 may be concatenated to form a story 134, which includes the story text 118 and a story video including content corresponding to the story text 118 generated from the user input 114.”; concatenating the story images and story text is interpreted as presenting successive the images with the corresponding story text ).
Regarding claim 5, the combination of Liu, Wang and Zhu teaches all of the elements of claim 1 (see detailed element listing above). In addition, Wang further teaches the detection network comprises at least one neural network (¶[0042] teaches “ML model 315 includes knowledge transformer network 320, image encoder 325, caption network 330, search component 335, property classification head 340, and object detection head 345.” And ¶[0047] teaches “ML model 315 may include (e.g., or implement) an artificial neural network…” ).
Liu differs from the claimed invention, as defined by claim 5, in that Liu fails to specifically disclose that the input image includes at least one object in a real world environment and the objects are classified using a detection network includes a neural network. Classifying real world objects from input images using a detection neural network is known in the art as evidenced by Wang. Therefore, it would have been obvious to one having ordinary skill in the art, before the effective filing date of the invention, to have modified the story generation system of Liu to include classifying objects in the input image using a detection neural network as taught by Wang as it merely constitutes the combination of know processes to achieve the predictable result of classifying the objects represented in the input image.
Regarding claims 7 and 16, the combination of Liu, Wang and Zhu teaches all of the elements of claims 6 and 13 (see detailed element listing above). In addition, Liu further teaches the image generator comprises at least one application programming interface (API) (¶[0029] teaches “In some embodiments, the methods and processes described herein may be tied to a computing system of one or more computing devices. In particular, such methods and processes may be implemented as a computer-application program or service, an application-programming interface (API), a library, and/or other computer-program product.”).
Regarding claim 8, the combination of Liu, Wang and Zhu teaches all of the elements of claim 7 (see detailed element listing above). In addition, Liu further teaches the API is associated with at least one machine learning (ML) model (¶[0026] teaches “the animation process includes the use of machine learning models” and ¶[0029] teaches “In some embodiments, the methods and processes described herein may be tied to a computing system of one or more computing devices. In particular, such methods and processes may be implemented as a computer-application program or service, an application-programming interface (API), a library, and/or other computer-program product.” Therefore the API process is associated with at least one machine learning (ML) model).
Regarding claims 12 and 19, the combination of Liu, Wang and Zhu teaches all of the elements of claim 11 and 13 (see detailed element listing above). In addition, Liu further teaches presenting successive images of the plurality of images on the at least one display along with the respective segments to which each image relates (¶[0016] teaches “The plurality of animated story images 130 may be concatenated to form a story 134, which includes the story text 118 and a story video including content corresponding to the story text 118 generated from the user input 114.”; concatenating the story images and story text is interpreted as presenting successive the images with the corresponding story text).
Regarding claims 15 and 20, the combination of Liu, Wang and Zhu teaches all of the elements of claim 13 and 10 (see detailed element listing above). In addition, Wang further teaches the detection network comprises at least one neural network (¶[0042] teaches “ML model 315 includes knowledge transformer network 320, image encoder 325, caption network 330, search component 335, property classification head 340, and object detection head 345.” And ¶[0047] teaches “ML model 315 may include (e.g., or implement) an artificial neural network…”).
Liu differs from the claimed invention, as defined by claim 15, in that Liu fails to specifically disclose that executing machine learning using at least one neural network. Executing machine learning using at least one neural network is known in the art as evidenced by Wang. Therefore, it would have been obvious to one having ordinary skill in the art, before the effective filing date of the invention, to have modified the story generation system of Liu to include classifying objects in the input image using a detection neural network as taught by Wang as it merely constitutes the combination of know processes to achieve the predictable result of classifying the objects represented in the input image.
Regarding claims 9, 17 and 21, the combination of Liu, Wang and Zhu teaches all of the elements of claim 1, 15 and 10 (see detailed element listing above). In addition, Liu further teaches the at least one generative neural network comprises a generative pre-trained transformer (GPTT) (¶[0017] teaches “The text-to-image generation process can also be implemented using various generative models. As described above, generative diffusion models, which are a class of probabilistic generative models that can generate data similar to the data on which they are trained, are viable for such implementations”).
Regarding claim 18, the combination of Liu, Wang and Zhu teaches all of the elements of claim 17 (see detailed element listing above). In addition, Liu further teaches the GPTT is trained on a corpus of documents comprising object identifications (¶[0001] teaches “Generative machine learning algorithms can be implemented in probabilistic models that are able to generate new data through the analysis of regularities and patterns in training datasets”).
Regarding claims 23 and 25, the combination of Liu, Wang and Zhu teaches all of the elements of claim 10 and 1 (see detailed element listing above). In addition, Zhu further teaches selecting, from the plurality of segments, the subset of segments for generating images according to one or more criteria comprises, for a segment of the plurality of segments:
determining, for each word in the segment, a classification of the word as a part-of-speech (p. 1591, 2nd column teaches “…All nouns, proper nouns, and adjectives (except those in a stop list) are selected as candidate words using a part-of-speech tagger…”); and
determining whether the classifications of the words in the segment satisfy the one more criteria for generating images (p. 1592, 1st column teaches “The stationary distribution indicates the centrality or relative importance of each word in the graph, taking into account picturality. We select the 20 words with the highest stationary probabilities, and form keyphrases by merging adjacent instances of the selected words (as long as the resulting phrase has a picturality probability greater than 0.5). Next, we discard phrases lacking nouns…and phrases that are subsumed by other longer phrases”).
The combination of Liu and Wang differs from the claimed invention, as defined in claims 1, 10 and 13, in that the combination fails to explicitly disclose selecting a subset of segments of the text story for image generation. Selecting key segments for a larger text story to image generation using POS tagging is known in the art as evidenced by Zhu. Therefore, it would have been obvious to one having ordinary skill in the before the effective filing date of the invention to have modified the text-to-image generating system taught by the combination of Liu and Wang to include selecting keyphrases/segments of the text story for image generation using POS tagging as taught by Zhu as it merely constitutes the combination of known processes to the achieve the predictable result to generating images for those segments/sentences of text having the most informative regarding the context of the story.
Regarding claim 24, the combination of Liu, Wang, and Zhu teaches all of the elements of claim 23 (see detailed element mapping above). In addition, Zhu further teaches wherein the one or more criteria comprises a threshold number of words in a segment having a particular part-of-speech classification (p. 1592, 1st column teaches “The stationary distribution indicates the centrality or relative importance of each word in the graph, taking into account picturality. We select the 20 words with the highest stationary probabilities, and form keyphrases by merging adjacent instances of the selected words (as long as the resulting phrase has a picturality probability greater than 0.5). Next, we discard phrases lacking nouns…and phrases that are subsumed by other longer phrases” discarding phrases lacking nouns is interpreted as the selection process requiring a threshold number of nouns).
The combination of Liu and Wang differs from the claimed invention, as defined in claim 24, in that the combination fails to explicitly disclose selecting a subset of segments of the text story for image generation. Selecting key segments for a larger text story to image generation using POS tagging is known in the art as evidenced by Zhu. Therefore, it would have been obvious to one having ordinary skill in the before the effective filing date of the invention to have modified the text-to-image generating system taught by the combination of Liu and Wang to include selecting keyphrases/segments of the text story for image generation using POS tagging as taught by Zhu as it merely constitutes the combination of known processes to the achieve the predictable result to generating images for those segments/sentences of text having the most informative regarding the context of the story.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PENNY L CAUDLE whose telephone number is (703)756-1432. The examiner can normally be reached M-Th 8:00 am to 5:00 pm eastern.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Daniel Washburn can be reached at 571-272-5551. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/PENNY L CAUDLE/Examiner, Art Unit 2657
/DANIEL C WASHBURN/Supervisory Patent Examiner, Art Unit 2657