DETAILED ACTION
This action is in response to communication filed on 07 November 2023. Claims 1-20 are pending in the application and have been considered below.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 3-4, 8, 10-11, 15 and 17 are rejected under 35 U.S.C. 103 as being unpatentable over HARTRELL et al. (US20170365083A1) in view of ZAVESKY et al. (US20220165024A1).
As to claim 1, HARTRELL teaches a method comprising: receiving a frame image including one or more panels (see figs. 3-10, e.g. fig. 7, par. 0077, wherein the example image 700 shows a single page 770 of a graphic novel having three panels 702, 704, 706 and five speech bubble objects 722, 724, 726, 728, 732. The scenes in each of the panels 702, 704, 706 show two characters of the graphic novel conversing with each other; as taught by HARTRELL);
executing a first machine-learning model using the frame image, wherein the first machine-learning model segments the frame image into one or more image regions (see par. 0078, wherein in speech bubble object 728, the locations of text regions 746, 750 are detected by the object identification module 414 in the graphic novel analysis system 120 by applying the numerical map of the image 700 to machine-learned models built by the training module 414. The location of speech bubble object 728 is determined based on the identified text regions 746, 750 by identifying distinctions around the text regions 746, 750 indicating the outer boundary 754 of the speech bubble object 728; as taught by HARTRELL);
generating a frame configuration using the frame image, the one or more image regions, and the context corresponding to the one or more panels, wherein the frame configuration identifies a sequence of views to present the frame image, wherein each view of the sequence of views corresponds to a panel of the one or more panels or a portion thereof (see par. 0079, wherein in example image 700, the intended reading order of the panels are in the order of (724, 726, 728), assuming the graphic novel is read top-to-bottom, left-to-right. The anchor points of detected speech bubble objects 722, 724, 726, 728 are used as reference points by the ordering module 422 to determine the presentation orders for the speech bubbles. Specifically, speech bubble objects 722, 724 are determined to be associated with panel 702, speech bubble object 726 associated with panel 704, and speech bubble object 728 associated with panel 706. Speech bubble object 722 is assigned the first presentation order among the four detected speech bubble objects since it is in the left-most panel 702, and its anchor point is positioned above speech bubble object 724. Similarly, speech bubble object 724 is assigned the second order, speech bubble object 726 is assigned the third order, and speech bubble object 728 is assigned the fourth order as it is in the last panel 706. Thus, the presentation metadata of the corresponding graphic novel contains the locations and presentation order of the speech bubble objects (722, 724, 726, 728) for example image 700; as taught by HARTRELL);
and facilitating execution of the frame configuration causing a presentation of a first view of the sequence of views (see par. 0053, wherein the packaging module 426 creates a packaged digital graphic novel that includes the corresponding graphic novel content and presentation metadata indicating how the graphic novel should be presented by the reader device 180. In one embodiment, the packaging module 426 creates a packaged digital graphic novel (e.g., a PDF or fixed layout EPUB file, such as one conforming to the EPUB Region-Based Navigation 1.0 standard) that includes a series of ordered images (e.g., one image per page of the graphic novel, one image per two-page spread of the graphic novel) and presentation metadata corresponding to the digital graphic novel; as taught by HARTRELL).
HARTRELL does not expressly teach executing a second machine-learning model using the frame image, wherein the second machine-learning model identifies a context associated with alphanumeric text positioned within the one or more panels.
In similar field of endeavor, ZAVESKY teaches executing a second machine-learning model using the frame image, wherein the second machine-learning model identifies a context associated with alphanumeric text positioned within the one or more panels (See fig. 2, par. 0024, wherein the AS 104 may extract narrative elements from the plurality of static, two-dimensional images. For instance, a narrative element such as dialogue, recurring bits or jokes, exposition, or the like could be extracted from text on the page of an illustrated book, a thought or speech bubble associated with a character in a comic strip, or the like, where natural language processing techniques could be used to extract meaning from the text. A narrative element could also be inferred from images (e.g., an image of a character shivering may imply that it is cold out, an image of a Christmas tree or a jack-o-lantern may imply that a narrative takes place during a holiday season, etc.), where different image analysis techniques may be used to recognize objects and other elements in the plurality of two-dimensional images; see also par. 0025, wherein in further examples, the AS 104 may build a hierarchy of a narrative, or a narrative arc, from the extracted narrative elements. For instance, machine learning techniques may be used to identify relationships between narrative elements (e.g., a character stating, “I am hungry,” may be related to a later scene in which the character is depicted eating a slice of pizza); see also par. 0046; as taught by ZAVESKY).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the HARTRELL apparatus to include the teachings of ZAVESKY for executing a second machine-learning model using the frame image, wherein the second machine-learning model identifies a context associated with alphanumeric text positioned within the one or more panels. Such a person would have been motivated to make this combination as it is advantageous for the AS 104 to learn recurring narrative elements (e.g., such as recurring jokes, character interactions, and the like) and use these recurring narrative elements to construct an entirely new narrative arc (ZAVESKY, par. 0025).
As to claim 3, HARTRELL and ZAVESKY teach the limitations of claim 1. HARTRELL further teaches wherein the first machine-learning model uses the one or more image regions to determine a presentation order of the one or more panels (see fig. 7, par. 0079, wherein the anchor points of detected speech bubble objects 722, 724, 726, 728 are used as reference points by the ordering module 422 to determine the presentation orders for the speech bubbles. Specifically, speech bubble objects 722, 724 are determined to be associated with panel 702, speech bubble object 726 associated with panel 704, and speech bubble object 728 associated with panel 706. Speech bubble object 722 is assigned the first presentation order among the four detected speech bubble objects since it is in the left-most panel 702, and its anchor point is positioned above speech bubble object 724; as taught by HARTRELL).
As to claim 4, HARTRELL and ZAVESKY teach the limitations of claim 1. ZAVESKY further teaches wherein the second machine-learning model uses the context corresponding to the alphanumeric text to define a presentation order of the one or more panels (see fig. 2, par. 0051, wherein in step 210, the processing system may build a hierarchy of a narrative for the media asset, based on at least a subset of the plurality of narrative elements extracted in step 208. In one example, data models may be used to help to identify narrative elements that may be part of the same narrative arc, as well as an order in which the narrative elements may occur. For instance, a character in a comic strip stating, “I am hungry” may be related to a loose narrative about eating lunch, going hunting, cooking a meal, or the like. A villain stating that he will get revenge on a superhero may be related to a later narrative involving a battle between the villain and the superhero; as taught by ZAVESKY).
Claims 8 and 15 amount to the system, and the non-transitory computer-readable medium storing instructions for executing the method of claim 1, respectively. Accordingly, claims 8 and 15 are rejected for substantially the same reasons as presented above for claim 1 and based on the references’ disclosure of the necessary supporting hardware and software.
Claims 10 and 17 amount to the system, and the non-transitory computer-readable medium storing instructions for executing the method of claim 3, respectively. Accordingly, claims 10 and 17 are rejected for substantially the same reasons as presented above for claim 3 and based on the references’ disclosure of the necessary supporting hardware and software.
Claim 11 amount to the system for executing the method of claim 4. Accordingly, claims 11 is rejected for substantially the same reasons as presented above for claim 4 and based on the references’ disclosure of the necessary supporting hardware and software.
Claims 2, 5-7, 9, 12-14, 16 and 18-20 are rejected under 35 U.S.C. 103 as being unpatentable over HARTRELL et al. (US20170365083A1) in view of ZAVESKY et al. (US20220165024A1) and further NONAKA et al. (US20130282376A1).
As to claim 2, HARTRELL and ZAVESKY teach the limitations of claim 1. HARTRELL and ZAVESKY do not expressly teach wherein the first machine-learning model performs edge detection to identify each panel of the one or more panels.
In similar field of endeavor, NONAKA teaches wherein the first machine-learning model performs edge detection to identify each panel of the one or more panels (see figs. 1-14, par. 0090, wherein a known image analysis technique or text analysis technique is used to analyze and acquire the page information by the page information analysis section 10. For example, the position, size, and type of a content element such as a face, an animal, a building, an automobile, and other objects may be automatically detected based on a feature amount regarding image information thereof. The content element may be automatically detected based on machine learning. For example, the accuracy of detecting an outer edge of a panel or a speech balloon, and a determination threshold value for the validity of a region other than a rectangular region as a panel or a speech balloon is empirically set based on a sample comic for learning; as taught by NONAKA).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the HARTRELL and ZAVESKY apparatus to include the teachings of NONAKA wherein the first machine-learning model performs edge detection to identify each panel of the one or more panels. Such a person would have been motivated to make this combination as the advantageous Effects of Invention are that the viewer device can display the first text information indicating the dialogue within each speech balloon based on the information of the speech balloon region of the file format in both the scroll view and the panel view, and can properly arrange the letter string of the dialogue in an original language, and the letter string of the dialogue converted into any language from the dialogue in the original language within the speech balloon region (NONAKA, par. 0040).
As to claim 5, HARTRELL and ZAVESKY teach the limitations of claim 1. HARTRELL and ZAVESKY do not expressly teach detecting a navigation command associated with the presentation of a first view of the sequence of views; and presenting a subsequent view of the sequence of views based on the navigation command.
In similar field of endeavor, NONAKA teaches detecting a navigation command associated with the presentation of a first view of the sequence of views; and presenting a subsequent view of the sequence of views based on the navigation command (see figs. 1-14, par. 0074, wherein display control information for a scroll view, which enables an image of an entire page to be viewed by moving (scrolling) the image from a present anchor point to a next anchor point and stopping the image at each anchor point for a desired length of time (a preset stay time or a time until scrolling to the next anchor point is manually instructed), includes positional information (coordinates) of each anchor point on the image of the entire page, a transition order of respective anchor points, a stay time, or the like; see also pars. 0075, 0077 and 0079, as taught by NONAKA).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified the HARTRELL and ZAVESKY apparatus to include the teachings of NONAKA for detecting a navigation command associated with the presentation of a first view of the sequence of views; and presenting a subsequent view of the sequence of views based on the navigation command. Such a person would have been motivated to make this combination as it is advantageous for the user to be able to navigate the comic book by different means of navigation such as scrolling after a set time, voice command or gaze (see also NONAKA, par. 0040).
As to claim 6, HARTRELL, ZAVESKY and NONAKA teach the limitations of claim 5. NONAKA further teaches wherein the navigation command is defined based on one of: a voice command, a gesture, device motion, eye movement, device input, or time (see par. 0077, wherein the display control information also includes screen scrolling and/or screen switching. The screen scrolling information may include a scrolling speed, a scrolling direction, a scrolling order, and a method for starting, ending, suspending, and repeating scrolling as detailed information. The screen switching information may include a unit of switching (panel, etc.), a method for determining a switching timing (manual, automatic, semiautomatic), and a display effect (wiping, fading in/fading out, dissolving) accompanying switching as detailed information, as taught by NONAKA).
As to claim 7, HARTRELL, ZAVESKY and NONAKA teach the limitations of claim 5. NONAKA further teaches modifying the frame configuration based on the navigation command, wherein modifying the frame configuration includes adjusting a transition between views of the sequence of views that are remaining (see par. 0082, wherein the display switching speed between the anchor points is set in the reproduction scenario. The speed may be determined based on a past viewing speed acquired as reproduction state information corresponding to a user of the digital book viewer 2 accessing the server 1, or may be determined by applying past viewing speeds acquired as reproduction state information from a plurality of digital book viewers 2 reproducing the same reproduction content to a predetermined arithmetic expression (average, etc.); as taught by NONAKA).
Claims 9 and 16 amount to the system, and the non-transitory computer-readable medium storing instructions for executing the method of claim 2, respectively. Accordingly, claims 9 and 16 are rejected for substantially the same reasons as presented above for claim 2 and based on the references’ disclosure of the necessary supporting hardware and software.
Claims 12 and 18 amount to the system, and the non-transitory computer-readable medium storing instructions for executing the method of claim 5, respectively. Accordingly, claims 12 and 18 are rejected for substantially the same reasons as presented above for claim 5 and based on the references’ disclosure of the necessary supporting hardware and software.
Claims 13 and 19 amount to the system, and the non-transitory computer-readable medium storing instructions for executing the method of claim 6, respectively. Accordingly, claims 13 and 19 are rejected for substantially the same reasons as presented above for claim 6 and based on the references’ disclosure of the necessary supporting hardware and software.
Claims 14 and 20 amount to the system, and the non-transitory computer-readable medium storing instructions for executing the method of claim 7, respectively. Accordingly, claims 14 and 20 are rejected for substantially the same reasons as presented above for claim 7 and based on the references’ disclosure of the necessary supporting hardware and software.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure.
Publication Number
Filing Date
Title
US10452920B2
2017-10-31
Systems and methods for generating a summary storyboard from a plurality of image frames
US8819545B2
2012-10-19
Digital comic editor, method and non-transitory computer-readable medium
US8719029B2
2013-06-20
File format, server, viewer device for digital comic, digital comic generation device
US20130283157A1
2013-06-19
Digital comic viewer device, digital comic viewing system, non-transitory recording medium having viewer program recorded thereon, and digital comic display method
US8930814B2
2012-10-19
Digital comic editor, method and non-transitory computer-readable medium
US20170083196A1
2015-09-23
Computer-Aided Navigation of Digital Graphic Novels
US8952985B2
2012-10-19
Digital comic editor, method and non-transitory computer-readable medium
US20210073458A1
2017-12-27
Comic data display system, method, and program
US20130073952A1
2012-08-24
Methods and Apparatus for Comic Creation
US20110310104A1
2011-06-17
Digital comic book frame transition method
Any inquiry concerning this communication or earlier communications from the examiner should be directed to KOOROSH NEHCHIRI whose telephone number is (408) 918-7643. The examiner can normally be reached M-F, 11-7 PST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, William L. Bashore can be reached on 571-272-4088. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/KOOROSH NEHCHIRI/Examiner, Art Unit 2174
/WILLIAM L BASHORE/ Supervisory Patent Examiner, Art Unit 2174