Ent DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Response to Amendment
The objection to the drawings and the specifications have been withdrawn in view of the amendments filed on 09/01/2026.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-23 is/are rejected under 35 U.S.C. 103 as being unpatentable over Bylinskii et al. (U.S. Doc. No. 11189066) in view of Jetley et al. (U.S. Pub. No. 20170308770) and Gupta et al. (U.S. Pub. No. 20190147288).
Regarding claim 1, Bylinskii discloses a method for predicting an attention level of a viewer corresponding to one or more portions of a media item, comprising (col 11, “FIG. 8 is a flow diagram 800 of an illustrative method of providing a dynamic feedback for an input graphic object, according to an embodiment. “; also, col 3-4, “For instance, an importance map may be a graphical representation showing each pixel of the graphic object within a range from least important to most important where a higher importance indicates a higher probability that the pixel is viewed by a viewer.”; also, col 6, “the dynamic importance map 118a, through the color codes, may show predictions generated by the feedback engine 108 as to where viewers are expected to look at the input graphic object 116a.”): receiving the media item (col 11, “The method may begin at step 802, where the computer may receive an input graphic object. The input graphic object may be a graphic design or a data visualization. In some instances, the computer may display GUI with an upload tool for a user to upload an image, e.g., in a bitmap format, containing the input graphic object”); segmenting the media item into a plurality of segments (col 5, “The notion of importance may also depend upon higher level factors such as semantic categories of design elements (e.g., title text, axis text, data points) within a graphic object.”; also, col 7, “For example, the smart thumbnail image 118c for a graphic object that is a data visualization may include title and other main supporting text, as well as data extremes—in case of a data table, top and bottom of the table, and in case of a data plot, left and right sides of the data plot.”), one or more characteristics associated with each segment of the plurality of segments (col 9, “An instruction to edit/update the input graphic object may include, for example, an instruction to resize a text or an image in the graphic object, an instruction to change font of a text in the graphic object, an instruction to change a color of a text or an image in the graphic object, and an instruction to move a text or an image in the graphic object.”; also, col 12, “the user may change font, size, or color of the text or rearrange the text or images. The user may also resize, crop, and/or change one or more colors of the picture.”); providing the segmented media item and identified characteristics to a machine-learning model (col 11, “At step 804, the computer may execute or deploy a neural network to generate an importance map for the input graphic object. The neural network may have been trained to optimize a loss function containing a pixel-level comparison of the importance output generated by the neural network and the ground truth dataset. In other words, the neural network may generate a pixel-level importance map for the input graphic object.”; also, col 7, “In some embodiments, the neural network 106 may be a convolutional neural network.”; also, col 5, “A computer may train one or more neural networks using bubble view and explicit importance map datasets as ground truth datasets. Ground truth datasets, as described herein, may include pre-classified data used to train the one or more neural networks by comparing the outputs of the one or more neural networks with the corresponding ground truth data. Conventional eye gaze measurements that are used to generate ground truth datasets of human attention, which are generally expensive and time consuming, and may not be feasible for practical purposes. The use of bubble view and explicit importance datasets are generally inexpensive, and provide close approximation of an eye gaze measurement.”) heat map that predicts (col 11-12, “For example, the importance map may be analogous to a heat map with different colors or shades of colors to represent the relative importance of various pixels within the input graphic object.”; also, col 3, “Importance maps may indicate the relative importance of various regions of a graphic design or data visualization.”), respective attention level of the viewer (col 6, “The dynamic importance map 118a, through the color codes, may show predictions generated by the feedback engine 108 as to where viewers are expected to look at the input graphic object 116a.”) segmented media item, the identified characteristics, and the empirical data (col 11, “At step 804, the computer may execute or deploy a neural network to generate an importance map for the input graphic object. The neural network may have been trained to optimize a loss function containing a pixel-level comparison of the importance output generated by the neural network and the ground truth dataset. In other words, the neural network may generate a pixel-level importance map for the input graphic object.”; also, col 11, “At step 806, the computer may display the importance map. The computer may display the importance map in the GUI side-by-side with the input graphic object.”; also, col 4, “An importance map may, for example, show a color coded graphic object with different shades of colors overlaid on the graphic object showing the importance of the respective regions of the graphic object, analogous to a heat map.”). Bylinskii does not disclose wherein each segment of the plurality of segments bounds one or more text items or image items; that has been trained using empirical data from identified user gaze locations on training media items, for each segment of the plurality of segments bounding one or more text items or image items, associated with the entirety of the respective segment corresponding to a respective relative amount of time that a viewer is predicted to look at the respective segment.
However, in a similar field of endeavor, Jetley discloses that has been trained using empirical data from identified user gaze locations on training media items (paragraph 57, “As illustrated in FIG. 3, in the exemplary embodiment, ground-truth saliency maps, referred to herein as attention maps 56, are constructed from the aggregated fixations of multiple observers, ignoring any temporal fixation information. Areas with a high fixation density are interpreted as receiving more attention.”; also, paragraph 58, “For example, at S202, eye gaze data is obtained for a set of observers for each training image (such as at least two observers). Eye gaze data may be obtained by observing the focus of the person's eye on an image, or simulated by receiving mouse clicks on the image made by observers. The collected eye gaze data is combined to generate a set of fixation coordinates 71.”; also, paragraph 40, “At S104, the neural network 64 is trained on the set of training images 62 and their associated attention maps 56 to predict saliency maps 12 for new images 14. The training includes learning weights for layers of the neural network through backward passes of the neural network”)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Bylinskii's invention of receiving a media item, segmenting the media item into a plurality of segments, identifying a characteristic associated with each segment, providing the segmented media item and the identified characteristics to a neural network, and generating a heat map that predicts a respective attention level of the viewer with the features of Jetley's invention of training that neural network using empirical eye-gaze data collected from observers viewing training media items. The combination would have been obvious because Bylinskii trains its neural network by a pixel-level comparison between the network output and a ground truth dataset without fixing what that ground truth is measured from, and Jetley supplies a ground truth of exactly the kind Bylinskii's loss function consumes, an attention map constructed from the aggregated fixations of multiple observers of each training image. A person of ordinary skill would have looked to Jetley because both references train a convolutional network to output a per-location prediction of where a viewer will look on a displayed item, so supplying the training stage with measured fixation data rather than an unspecified ground truth requires no change to the network and yields the predictable result of an attention prediction learned from empirical user gaze locations. Bylinskii's preference for its own crowd-sourced datasets does not discourage the combination, because Bylinskii sets conventional eye gaze measurements aside as expensive and time consuming and as possibly not feasible for practical purposes rather than as technically unsuited, and Bylinskii measures the quality of its own datasets against eye gaze when it states that they provide close approximation of an eye gaze measurement, so for a person of ordinary skill who already has eye-tracking data in hand, as Jetley does, Bylinskii's own benchmark is the reason to use it.
Gupta discloses wherein each segment of the plurality of segments bounds one or more text items or image items (para 27, “Each content item in the set of training content items 132 may include a plurality of elements. An element is any object on which the content creator can take action. Examples of elements include a title, a heading, an image, a paragraph, a table, a text box, a button, a link, a span, etc.”; also, para 27, “For example, a GUI (graphical user interface) may enable a content creator to designate an element by drawing a box or other shape around the element.”; also, para 57, “Process 600 begins by determining the elements of the content item (602). In some implementations, the content elements may be identified or tagged as such by a user. For example, a user, such as the content creator or the system administrator setting up the training content items, may use a graphical user interface to draw a bounding box around each element in the content item.”; also, para 23, “The lowest zoom level may correspond to a bounding box. A bounding box is the smallest shape, typically but not always a rectangular box, that encloses the entire element.”); for each segment of the plurality of segments bounding one or more text items or image items (para 40, “A saliency map shows the elements in a content item represented by their respective saliency scores. In some implementations, the saliency map may be a heat map, which uses differing colorations representing different ranges of saliency scores.”; also, para 63, “In the example of FIG. 8 the content item 700 includes seven different elements. Accordingly, the saliency map 800 includes seven element areas, each area corresponding to one of the elements. The element areas of the saliency map 800 are laid out based on the location of the corresponding element in the original content item. This layout provides a quick visual correspondence between the areas and the original content item.”), associated with the entirety of the respective segment (para 63, “Each element area in the saliency map has a coloration that corresponds to the saliency code determined by the content saliency neural network using the techniques described herein.”; also, para 23, “The lowest zoom level may correspond to a bounding box. A bounding box is the smallest shape, typically but not always a rectangular box, that encloses the entire element.”) corresponding to a respective relative amount of time that a viewer is predicted to look at the respective segment (para 31, “The training module 130 may generate a saliency score for each element in each training content item. The saliency score represents how often and for how long the workers gazed at the element during the tasks and is based on a proportion of the total possible gaze points across all workers for the content item that correspond with the element. Put another way, the saliency score is a measure of how much the element stands out relative to its neighbors. For example, a content item with four elements viewed by four workers over five seconds generates a total of 20 possible gaze point locations. If two of the workers gaze at element 1 for all 5 seconds, and two of the workers gaze at element 1 for two seconds and element 3 for three seconds, the training module 130 may determine that elements 2 and 4 have a saliency score of zero, element 3 has a saliency score of 0.3, and element 1 has a saliency score of 0.7.”; also, para 55, “The system may determine the saliency score for an element by computing the proportion of gaze points in the set of gaze points (e.g., over all workers) that fall on any pixel belonging to the element.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Bylinskii in view of Jetley, in which a neural network trained on measured observer fixations generates a heat map predicting where a viewer will look on a segmented media item, with the features of Gupta's invention of bounding each element of a content item in a box that encloses the entire element, scoring each bounded element by the proportion of the observers' viewing time that fell on it, and rendering one coloration per element area as a heat map. The combination would have been obvious because the base combination already identifies the portions of a graphic object by semantic category, naming title text, axis text and data points, yet reports its prediction as a value per pixel, which leaves the content creator to read an element-level judgment out of a field of pixels. Gupta solves that problem in the same art by attaching a single score to a whole bounded element and coloring the element area with it, and Gupta states the benefit in its own terms when it explains that laying the element areas out at the location of the corresponding element provides a quick visual correspondence between the areas and the original content item. A person of ordinary skill working on Bylinskii's design-feedback tool would have looked to Gupta because both references predict, from a neural network trained on human viewing data, where a viewer will look on a laid-out page of text and images, and the predictable result of scoring and coloring whole bounded elements instead of individual pixels is a heat map whose value at each bounded segment is the share of viewing time that segment is predicted to attract.
Regarding claim 2, Bylinskii as modified by Jetley and Gupta discloses the method of claim 1, further comprising displaying the heat map (Bylinskii: col 11, “At step 806, the computer may display the importance map. The computer may display the importance map in the GUI side-by-side with the input graphic object.”; col 11-12, “the importance map may be analogous to a heat map with different colors or shades of colors to represent the relative importance of various pixels within the input graphic object.”).
Regarding claim 3, Bylinskii as modified by Jetley and Gupta discloses the method of claim 1, wherein the heat map indicates a gaze of the viewer will be focused on each of the plurality of segments of the media item (Bylinskii: col 6, “the dynamic importance map 118a, through the color codes, may show predictions generated by the feedback engine 108 as to where viewers are expected to look at the input graphic object 116a.”; also, col 3-4, “an importance map may be a graphical representation showing each pixel of the graphic object within a range from least important to most important where a higher importance indicates a higher probability that the pixel is viewed by a viewer.”).
Regarding claim 4, Bylinskii as modified by Jetley and Gupta discloses the method of claims 1, wherein the one or more characteristics include at least one selected from size, location, text size, size rate between text and segment, center coordinates of segment, color, capital letters, non-alpha characters, number of characters, and a token term-frequency inverse document frequency (Bylinskii: col 9, “An instruction to edit/update the input graphic object may include, for example, an instruction to resize a text or an image in the graphic object, an instruction to change font of a text in the graphic object, an instruction to change a color of a text or an image in the graphic object, and an instruction to move a text or an image in the graphic object.”; also, col 12, “the user may change font, size, or color of the text or rearrange the text or images. The user may also resize, crop, and/or change one or more colors of the picture.”).
Regarding claim 5, Bylinskii as modified by Jetley and Gupta discloses the method of claim 1, wherein the media item corresponds to at least one selected from a presentation, a website, a poster, and a document (Bylinskii: col 5, “a graphic object may be a graphic design, such as a poster or a data visualization (e.g., chart or a graph).”; also, col 7-8, “the first graphic object 202 may be a data visualization (e.g., containing a bar-graph), the second graphic object 204 may be a sales graphic (e.g., containing a sales graphic in a geographic region), and the third graphic object 206 may be graphic designs (e.g., containing a poster).”; also, col 12, “Within a webpage, for instance, the graphic object may have to be displayed within the main content, in the margins, or as banner advertisements at the top.”).
Regarding claim 6, Bylinskii as modified by Jetley and Gupta discloses the method of claim 1, wherein each segment comprises at least one selected from a text item and an image item (Bylinskii: col 5, “The notion of importance may also depend upon higher level factors such as semantic categories of design elements (e.g., title text, axis text, data points) within a graphic object.”; also, col 12, “the user may change font, size, or color of the text or rearrange the text or images. The user may also resize, crop, and/or change one or more colors of the picture.”).
Regarding claim 7, Bylinskii as modified by Jetley and Gupta discloses the method of claim 1, wherein the media item is received via a user interface, wherein the user interface includes a user interface control for uploading the media item (Bylinskii: col 11, “The method may begin at step 802, where the computer may receive an input graphic object. The input graphic object may be a graphic design or a data visualization. In some instances, the computer may display GUI with an upload tool for a user to upload an image, e.g., in a bitmap format, containing the input graphic object”).
Regarding claim 8, Bylinskii as modified by Jetley and Gupta discloses the method of claim 7, wherein the user interface further comprises a first field configured to receive an indication of a number of pages included in the media item (Bylinskii: col 11, “The method may begin at step 802, where the computer may receive an input graphic object. The input graphic object may be a graphic design or a data visualization. In some instances, the computer may display GUI with an upload tool for a user to upload an image, e.g., in a bitmap format, containing the input graphic object”; also, col 6, “Retargeting is a common task for modern design tools because the input graphic object 116b may have to be targeted for different screen sizes, ranging from televisions to mobile devices with a plurality of screen dimensions. Furthermore, the input graphic object 116b may have to retargeted for other space/aspect ratio constraints, e.g., for different websites and different parts of webpages such as the main content portion or the margins.”).
Regarding claim 9, Bylinskii as modified by Jetley and Gupta discloses the method of claim 1, wherein training the machine-learning model comprises (Bylinskii: col 11, “At step 804, the computer may execute or deploy a neural network to generate an importance map for the input graphic object. The neural network may have been trained to optimize a loss function containing a pixel-level comparison of the importance output generated by the neural network and the ground truth dataset.”; also, col 5, “A computer may train one or more neural networks using bubble view and explicit importance map datasets as ground truth datasets. Ground truth datasets, as described herein, may include pre-classified data used to train the one or more neural networks by comparing the outputs of the one or more neural networks with the corresponding ground truth data. Conventional eye gaze measurements that are used to generate ground truth datasets of human attention, which are generally expensive and time consuming, and may not be feasible for practical purposes. The use of bubble view and explicit importance datasets are generally inexpensive, and provide close approximation of an eye gaze measurement.”): training media item includes one or more segments corresponding to a portion of the displayed training media item and each segment comprising one or more identifying characteristics (Bylinskii: col 5, “The notion of importance may also depend upon higher level factors such as semantic categories of design elements (e.g., title text, axis text, data points) within a graphic object.”, also, col 7, “the smart thumbnail image 118c for a graphic object that is a data visualization may include title and other main supporting text, as well as data extremes—in case of a data table, top and bottom of the table, and in case of a data plot, left and right sides of the data plot.”); training, based on the identified one or more identifying characteristics associated with each segment of the one or more segments (Bylinskii: col 14, “In the explicit importance maps methodology, crowd workers may be provided with images with graphic designs and asked to label important regions of designs using binary masks. By averaging the responses, ground truth importance maps of the graphic design may be generated to train and/or test a neural network.”) machine-learning model configured to receive an input media item and predict the attention level of a viewer corresponding to the one or more segments of the input media item (Bylinskii: col 11, “The method may begin at step 802, where the computer may receive an input graphic object. The input graphic object may be a graphic design or a data visualization”; also, col 6, “the dynamic importance map 118a, through the color codes, may show predictions generated by the feedback engine 108 as to where viewers are expected to look at the input graphic object 116a.”). Bylinskii does not disclose displaying, to a subject, a training media item on a display, receiving, from one or more sensors, eye-tracking information corresponding to a gaze location of the subject on the displayed training media item; determining an eye-tracking score for each of the one or more segments of the training media item based on the eye-tracking data; and the determined eye-tracking score for each of the one or more segments.
However, in a similar field of endeavor, Jetley discloses displaying, to a subject, a training media item on a display (paragraph 99, “The eye tracking data is collected using a head-mounted eye tracking device for 15 different viewers. The 1003 images of this dataset cover natural indoor and outdoor scenes.”; also, paragraph 56, “In an analysis of 300 images viewed by 39 observers, it was found that the fixations for a set of n observers match those from a different set of n observers with an AUC score that increases with the increase in the value of n.”, receiving, from one or more sensors, eye-tracking information corresponding to a gaze location of the subject on the displayed training media item (paragraph 58, “For example, at S202, eye gaze data is obtained for a set of observers for each training image (such as at least two observers). Eye gaze data may be obtained by observing the focus of the person's eye on an image, or simulated by receiving mouse clicks on the image made by observers. The collected eye gaze data is combined to generate a set of fixation coordinates 71.”; also, paragraph 99, “The eye tracking data is collected using a head-mounted eye tracking device for 15 different viewers”); determining an eye-tracking score for each of the one or more segments of the training media item based on the eye-tracking information (paragraph 59-60, “At S204, a binary fixation map 72, denoted b, is generated from the ground-truth eye-fixations 71. For each pixel i of the fixation map: At S206, the binary map b is then convolved (blurred) to approximate the general region fixated on. This can be done with a Gaussian kernel to produce a smoothed map 74, denoted y”); and the determined eye-tracking score for each of the one or more segments (paragraph 40, “At S104, the neural network 64 is trained on the set of training images 62 and their associated attention maps 56 to predict saliency maps 12 for new images 14. The training includes learning weights for layers of the neural network through backward passes of the neural network” also, paragraph 72, “An end-to-end learning method is adopted in which the fully-convolutional network 64 is trained on pairs of training images 62 and ground-truth saliency maps modeled as distributions p and g. The network 64 outputs predicted probability distributions p, while the ground-truth maps g are computed as described above for S102.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Bylinskii's invention of training a neural network on segmented graphic objects whose segments carry identifying characteristics, using ground truth importances averaged over crowd responses, with the features of Jetley's invention of displaying each training image to observers, capturing their gaze with an eye tracking device, converting the captured fixations into a per-region score, and learning the network end-to-end on the resulting maps. The combination would have been obvious because Bylinskii's training stage already consumes a per-location ground truth map averaged over many respondents and Jetley produces exactly such a map from a measured signal, so the substitution is of one ground-truth source for another inside an unchanged pipeline. A person of ordinary skill would have looked to Jetley because Jetley addresses the same problem of obtaining reliable ground truth for a saliency network and reports that agreement between observer sets rises as the number of observers grows, and the predictable result is a model trained from the gaze of subjects actually shown the training media items. Bylinskii's preference for its own crowd-sourced datasets does not discourage the combination, because Bylinskii sets conventional eye gaze measurements aside as expensive and time consuming and as possibly not feasible for practical purposes rather than as technically unsuited, and Bylinskii measures the quality of its own datasets against eye gaze when it states that they provide close approximation of an eye gaze measurement, so for a person of ordinary skill who already has eye-tracking data in hand, as Jetley does, Bylinskii's own benchmark is the reason to use it.
Regarding claim 10, Bylinskii discloses a method for training a system for predicting attention level of a viewer corresponding to one or more portions of a digital media item, comprising (col 11, “FIG. 8 is a flow diagram 800 of an illustrative method of providing a dynamic feedback for an input graphic object, according to an embodiment.”; also, col 11, “At step 804, the computer may execute or deploy a neural network to generate an importance map for the input graphic object. The neural network may have been trained to optimize a loss function containing a pixel-level comparison of the importance output generated by the neural network and the ground truth dataset. In other words, the neural network may generate a pixel-level importance map for the input graphic object.”; also, col 5, “A computer may train one or more neural networks using bubble view and explicit importance map datasets as ground truth datasets. Ground truth datasets, as described herein, may include pre-classified data used to train the one or more neural networks by comparing the outputs of the one or more neural networks with the corresponding ground truth data. Conventional eye gaze measurements that are used to generate ground truth datasets of human attention, which are generally expensive and time consuming, and may not be feasible for practical purposes. The use of bubble view and explicit importance datasets are generally inexpensive, and provide close approximation of an eye gaze measurement.”): training media item includes a plurality of segments each corresponding to a portion of the displayed training media item (col 5, “The notion of importance may also depend upon higher level factors such as semantic categories of design elements (e.g., title text, axis text, data points) within a graphic object.”; also, col 7, “For example, the smart thumbnail image 118c for a graphic object that is a data visualization may include title and other main supporting text, as well as data extremes—in case of a data table, top and bottom of the table, and in case of a data plot, left and right sides of the data plot.”), one or more identifying characteristics (col 9, “An instruction to edit/update the input graphic object may include, for example, an instruction to resize a text or an image in the graphic object, an instruction to change font of a text in the graphic object, an instruction to change a color of a text or an image in the graphic object, and an instruction to move a text or an image in the graphic object."; also, col 12, “More particularly, the user may change font, size, or color of the text or rearrange the text or images. The user may also resize, crop, and/or change one or more colors of the picture.”) ; training, based on the identified one or more characteristics associated with each segment of the plurality of segments (col 14, “In the explicit importance maps methodology, crowd workers may be provided with images with graphic designs and asked to label important regions of designs using binary masks. By averaging the responses, ground truth importance maps of the graphic design may be generated to train and/or test a neural network.”) machine-learning model configured to receive an input media item and predict a respective attention level of a viewer corresponding to each segment of a plurality of segments of the input media item (col 11, “The method may begin at step 802, where the computer may receive an input graphic object. The input graphic object may be a graphic design or a data visualization.”; also, col 6, “the dynamic importance map 118a, through the color codes, may show predictions generated by the feedback engine 108 as to where viewers are expected to look at the input graphic object 116a”). Bylinskii does not disclose displaying, to a subject, a training media item on a display, wherein each segment bounds one or more text items or image items; receiving, from one or more sensors, eye-tracking information corresponding to a gaze location of the subject on the displayed training media item; determining an eye-tracking score for each segment of the plurality of segments bounding one or more text items or image items based on the eye-tracking information, wherein each respective eye-tracking score corresponds to a respective relative amount of time that a viewer looks at the respective segment and the determined eye-tracking score for each of the plurality of segments.
However, in a similar field of endeavor, Jetley discloses displaying, to a subject, a training media item on a display (para 9, “The eye tracking data is collected using a head-mounted eye tracking device for 15 different viewers. The 1003 images of this dataset cover natural indoor and outdoor scenes.”; also, para 56, “In an analysis of 300 images viewed by 39 observers, it was found that the fixations for a set of n observers match those from a different set of n observers with an AUC score that increases with the increase in the value of n.”); receiving, from one or more sensors, eye-tracking information corresponding to a gaze location of the subject on the displayed training media item (para 58, “For example, at S202, eye gaze data is obtained for a set of observers for each training image (such as at least two observers). Eye gaze data may be obtained by observing the focus of the person’s eye on an image, or simulated by receiving mouse clicks on the image made by observers. The collected eye gaze data is combined to generate a set of fixation coordinates 71.”; also, para 99, “The eye tracking data is collected using a head-mounted eye tracking device for 15 different viewers. The 1003 images of this dataset cover natural indoor and outdoor scenes.”); determining an eye-tracking score for each segment of the plurality of segments (para 59, “At S204, a binary fixation map 72, denoted b, is generated from the ground-truth eye-fixations 71.”; also, para 60, “At S206, the binary map b is then convolved (blurred) to approximate the general region fixated on. This can be done with a Gaussian kernel to produce a smoothed map 74, denoted y.”), based on the eye-tracking information (para 58, “For example, at S202, eye gaze data is obtained for a set of observers for each training image (such as at least two observers). Eye gaze data may be obtained by observing the focus of the person’s eye on an image, or simulated by receiving mouse clicks on the image made by observers. The collected eye gaze data is combined to generate a set of fixation coordinates 71.”; also, para 61, “At S208, the smoothed map y is normalized to generate a normalized attention map 76 in which the probabilities range between 0 and 1.”), and the determined eye-tracking score for each of the plurality of segments (para 72, “An end-to-end learning method is adopted in which the fully-convolutional network 64 is trained on pairs of training images 62 and ground-truth saliency maps modeled as distributions p and g. The network 64 outputs predicted probability distributions p, while the ground-truth maps g are computed as described above for S102.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Bylinskii's invention of training a convolutional neural network on graphic objects divided into design elements carrying identifying characteristics, against a per-location ground truth averaged over many respondents, with the features of Jetley's invention of displaying each training image to subjects, receiving their gaze from an eye tracking device, reducing the captured fixations to a per-region score, and learning the network end-to-end on pairs of training images and those scores. The combination would have been obvious because Bylinskii's ground truth is assembled by averaging the responses of many people asked where the important content is, and Jetley obtains the same quantity by measurement rather than by report, from observers who are actually shown the image. A person of ordinary skill would have looked to Jetley because it addresses the same problem of producing reliable ground truth for a saliency network, and reports that the agreement between one set of observers and another rises as the number of observers grows, so the predictable result of the substitution is a training pipeline whose per-segment supervision comes from measured subject gaze. Bylinskii's preference for its own crowd-sourced datasets does not discourage the combination, because Bylinskii sets conventional eye gaze measurements aside as expensive and time consuming and as possibly not feasible for practical purposes rather than as technically unsuited, and Bylinskii measures the quality of its own datasets against eye gaze when it states that they provide close approximation of an eye gaze measurement, so for a person of ordinary skill who already has eye-tracking data in hand, as Jetley does, Bylinskii's own benchmark is the reason to use it.
Gupta discloses wherein each segment bounds one or more text items or image items (para 27, “Each content item in the set of training content items 132 may include a plurality of elements. An element is any object on which the content creator can take action. Examples of elements include a title, a heading, an image, a paragraph, a table, a text box, a button, a link, a span, etc.”; also, para 27, “For example, a GUI (graphical user interface) may enable a content creator to designate an element by drawing a box or other shape around the element.”; also, para 23, “The lowest zoom level may correspond to a bounding box. A bounding box is the smallest shape, typically but not always a rectangular box, that encloses the entire element.”) bounding one or more text items or image items (para 27, “Each content item in the set of training content items 132 may include a plurality of elements. An element is any object on which the content creator can take action. Examples of elements include a title, a heading, an image, a paragraph, a table, a text box, a button, a link, a span, etc.”; also, para 56, “Process 600 begins by determining the elements of the content item (602). In some implementations, the content elements may be identified or tagged as such by a user. For example, a user, such as the content creator or the system administrator setting up the training content items, may use a graphical user interface to draw a bounding box around each element in the content item.”) wherein each respective eye-tracking score corresponds to a respective relative amount of time that a viewer looks at the respective segment (para 31, ‘The training module 130 may generate a saliency score for each element in each training content item. The saliency score represents how often and for how long the workers gazed at the element during the tasks and is based on a proportion of the total possible gaze points across all workers for the content item that correspond with the element. Put another way, the saliency score is a measure of how much the element stands out relative to its neighbors. For example, a content item with four elements viewed by four workers over five seconds generates a total of 20 possible gaze point locations. If two of the workers gaze at element 1 for all 5 seconds, and two of the workers gaze at element 1 for two seconds and element 3 for three seconds, the training module 130 may determine that elements 2 and 4 have a saliency score of zero, element 3 has a saliency score of 0.3, and element 1 has a saliency score of 0.7.”; also, para 55, “The system may determine the saliency score for an element by computing the proportion of gaze points in the set of gaze points (e.g., over all workers) that fall on any pixel belonging to the element.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Bylinskii in view of Jetley, in which a network is trained on training media items divided into segments and on gaze measured from subjects viewing them, with the features of Gupta's invention of bounding each element of the training content item in a box that encloses the entire element and computing that element's score as the proportion of the total viewing time the observers spent on it. The combination would have been obvious because the base combination already divides the training item into segments carrying characteristics and already measures gaze on the displayed item, but reduces the measurement to a per-location value, so a score for a given segment must still be derived; Gupta supplies that step directly by defining the element as a bounded region and computing one score per region as a proportion of the gaze points available for the whole item. A person of ordinary skill would have looked to Gupta because Gupta trains the same kind of network on the same kind of measured viewing data for the same purpose of telling a content creator which parts of a layout will be looked at, and the predictable result is a per-segment training score expressing how long a viewer looks at that segment relative to the others.
Regarding claim 11, Bylinskii as modified by Jetley and Gupta discloses the method of claim 10, repeating the displaying, receiving, determining, and training with a second training media item.
However, in a similar field of endeavor, Jetley discloses further comprising repeating the displaying, receiving, determining, and training with a second training media item (paragraph 58, “For example, at S202, eye gaze data is obtained for a set of observers for each training image (such as at least two observers). Eye gaze data may be obtained by observing the focus of the person's eye on an image, or simulated by receiving mouse clicks on the image made by observers. The collected eye gaze data is combined to generate a set of fixation coordinates 71.”; also, paragraph 72, “An end-to-end learning method is adopted in which the fully-convolutional network 64 is trained on pairs of training images 62 and ground-truth saliency maps modeled as distributions p and g.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Bylinskii's invention of training a convolutional neural network to predict an importance heat map for a segmented graphic object with the features of Jetley's invention of running the gaze-collection and network-training cycle over each image of a set of training images. The combination would have been obvious because Jetley obtains gaze data for a set of observers for each training image and trains its network on pairs of training images and ground-truth maps, so the cycle Jetley describes operates over a set of images one image at a time rather than over a single image. A person of ordinary skill training the network of the base combination would have repeated the cycle for a further training media item because a single image cannot constrain the weights of a convolutional network, and the predictable result is a pipeline in which the displaying, receiving, determining and training operations are repeated with a second training media item.
Regarding claim 12, Bylinskii as modified by Jetley and Gupta discloses the method of claim 10, further comprising repeating the displaying, receiving, determining, and training with a second subject (Bylinskii: col 14, “In the bubble-view methodology, crowd workers may be shown a blurry image of a data visualization, and may be instructed to type a text caption describing the image. By clicking on different parts of the image, the crowd workers may reveal small regions—or bubbles—of the image at full resolution.”; also, col 14, “In the explicit importance maps methodology, crowd workers may be provided with images with graphic designs and asked to label important regions of designs using binary masks. By averaging the responses, ground truth importance maps of the graphic design may be generated to train and/or test a neural network.”).
Regarding claim 13, Bylinskii as modified by Jetley and Gupta discloses the method of claim 10, further comprising segmenting the training media item into the plurality of segments. (Bylinskii: col 5, “The notion of importance may also depend upon higher level factors such as semantic categories of design elements (e.g., title text, axis text, data points) within a graphic object”; also, col 7, “the smart thumbnail image 118c for a graphic object that is a data visualization may include title and other main supporting text, as well as data extremes—in case of a data table, top and bottom of the table, and in case of a data plot, left and right sides of the data plot.”).
Regarding claim 14, Bylinskii as modified by Jetley and Gupta discloses the method of claim 10, further comprising identifying one or more characteristics associated with each segment of the plurality of segments (Bylinskii: col 9, “An instruction to edit/update the input graphic object may include, for example, an instruction to resize a text or an image in the graphic object, an instruction to change font of a text in the graphic object, an instruction to change a color of a text or an image in the graphic object, and an instruction to move a text or an image in the graphic object.”; also, col 12, “the user may change font, size, or color of the text or rearrange the text or images. The user may also resize, crop, and/or change one or more colors of the picture.”).
Regarding claim 15, Bylinskii as modified by Jetley and Gupta discloses the method of claim 10, wherein Jetley further discloses the eye-tracking score is based on an amount of time the gaze location of the subject is focused on the respective segment (paragraph 59-60, “At S204, a binary fixation map 72, denoted b, is generated from the ground-truth eye-fixations 71. For each pixel i of the fixation map: At S206, the binary map b is then convolved (blurred) to approximate the general region fixated on. This can be done with a Gaussian kernel to produce a smoothed map 74, denoted y.”; also, paragraph 64, “where g.sub.i is the probability of pixel i being fixated on”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Bylinskii's invention of training a neural network to predict an importance heat map for a segmented graphic object with the features of Jetley's invention of building the per-region score from the fixations recorded at each location of the training image and blurring that record to capture the density of fixation over the surrounding region. The combination would have been obvious because each fixation Jetley records is an interval during which the subject's gaze rested at that location, so the accumulated fixation density Jetley computes for a region rises with the time the subject's gaze was held there, and normalizing that density as Jetley does places the regions of one image on a common scale. A person of ordinary skill would have adopted Jetley's scoring because the base combination needs a numerical supervision value per segment and Jetley supplies one that already reflects how long the subject looked, with the predictable result that the eye-tracking score for a segment is based on the amount of time the subject's gaze is focused on that segment.
Regarding claim 16, Bylinskii as modified by Jetley and Gupta discloses the method of claim 10, wherein Bylinskii further discloses the system is configured to output a heat map that predicts a location of a gaze of the viewer's gaze on the displayed media item (col 11-12, “For example, the importance map may be analogous to a heat map with different colors or shades of colors to represent the relative importance of various pixels within the input graphic object. “; also, col 6, “The dynamic importance map 118a, through the color codes, may show predictions generated by the feedback engine 108 as to where viewers are expected to look at the input graphic object 116a. “; also, col 11, “At step 806, the computer may display the importance map. The computer may display the importance map in the GUI side-by-side with the input graphic object.”) based on the eye-tracking score for each of the one or more segments and the one or more identifying characteristics for each of the plurality of segments (Bylinskii: col 11, “At step 804, the computer may execute or deploy a neural network to generate an importance map for the input graphic object. The neural network may have been trained to optimize a loss function containing a pixel-level comparison of the importance output generated by the neural network and the ground truth dataset. In other words, the neural network may generate a pixel-level importance map for the input graphic object.”; also, col 5, “The notion of importance may also depend upon higher level factors such as semantic categories of design elements (e.g., title text, axis text, data points) within a graphic object.”).
Regarding claim 17, Bylinskii as modified by Jetley and Gupta discloses the method of claim 16, wherein Jetley further discloses the heat map indicates a length of time that a gaze of the viewer's gaze will be focused on each segment of the one or more segments (paragraph 59, “At S204, a binary fixation map 72, denoted b, is generated from the ground-truth eye-fixations 71.”; also, para 60, ‘At S206, the binary map b is then convolved (blurred) to approximate the general region fixated on. This can be done with a Gaussian kernel to produce a smoothed map 74, denoted y.”; also, para 61, “At S208, the smoothed map y is normalized to generate a normalized attention map 76 in which the probabilities range between 0 and 1”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Bylinskii's invention of generating an importance heat map whose shading varies with the predicted importance of a segmented graphic object with the features of Jetley's invention of setting the value at each location of the map from the fixations recorded there and smoothing that value over the surrounding region. The combination would have been obvious because each recorded fixation is an interval of gaze held at that location, so the smoothed and normalized value Jetley produces for a region grows with the length of time the viewer's gaze rested on it. A person of ordinary skill would have encoded gaze duration into the map because a map that reports how long a viewer will dwell on a segment tells a content creator more than one that reports only whether the segment is attended, and the predictable result is a heat map indicating a length of time that a gaze of the viewer will be focused on each segment.
Regarding claim 18, Bylinskii as modified by Jetley and Gupta discloses the method of claim 10, wherein the eye-tracking information includes an eye-tracking bubble, wherein the eye-tracking bubble indicates a portion of the media item corresponding to the subject's gaze location (Bylinskii: col 14, “In the bubble-view methodology, crowd workers may be shown a blurry image of a data visualization, and may be instructed to type a text caption describing the image. By clicking on different parts of the image, the crowd workers may reveal small regions—or bubbles—of the image at full resolution. These clicks may be referred to as bubble clicks. The regions with most amount of bubble clicks may be the more important regions within the image of data visualization.”; also, Abstract, “The one or more neural networks are trained to optimize a loss function comprising a pixel-level comparison between the outputs generated by the neural networks and the ground truth dataset generated from a bubble view methodology or an explicit importance maps methodology.”).
Regarding claim 19, Bylinskii as modified by Jetley and Gupta discloses the method of claim 10, wherein each segment comprises at least one selected from a word and an image (Bylinskii: col 5, “The notion of importance may also depend upon higher level factors such as semantic categories of design elements (e.g., title text, axis text, data points) within a graphic object.”; also, col 12, “the user may change font, size, or color of the text or rearrange the text or images. The user may also resize, crop, and/or change one or more colors of the picture.”).
Regarding claim 20, Bylinskii as modified by Jetley and Gupta discloses the method of claim 10, wherein the one or more characteristics include at least one selected from size, location, text size, size rate between text and segment, center coordinates of segment, color, capital letters, non-alpha characters, number of characters, and a token term-frequency inverse document frequency (Bylinskii: col 9, “An instruction to edit/update the input graphic object may include, for example, an instruction to resize a text or an image in the graphic object, an instruction to change font of a text in the graphic object, an instruction to change a color of a text or an image in the graphic object, and an instruction to move a text or an image in the graphic object.”; also, col 12, “the user may change font, size, or color of the text or rearrange the text or images.”).
Regarding claim 21, Bylinskii as modified by Jetley and Gupta discloses the method of claim 10, wherein the media item corresponds to at least one selected from a presentation, a website, a poster, and a document (Bylinskii: col 5, “a graphic object may be a graphic design, such as a poster or a data visualization (e.g., chart or a graph).”; also, col 7-8, “the first graphic object 202 may be a data visualization (e.g., containing a bar-graph), the second graphic object 204 may be a sales graphic (e.g., containing a sales graphic in a geographic region), and the third graphic object 206 may be graphic designs (e.g., containing a poster).”; also, col 12, “Within a webpage, for instance, the graphic object may have to be displayed within the main content, in the margins, or as banner advertisements at the top”).
Regarding claim 22, Bylinskii as modified by Jetley and Gupta discloses the method of claim 10, further comprising identifying one or more characteristics of the training media item by applying natural language processing to the training media item (Bylinskii: col 5, “The notion of importance may also depend upon higher level factors such as semantic categories of design elements (e.g., title text, axis text, data points) within a graphic object.”, also, col 7, “the smart thumbnail image 118c for a graphic object that is a data visualization may include title and other main supporting text, as well as data extremes—in case of a data table, top and bottom of the table, and in case of a data plot, left and right sides of the data plot”).
Regarding claim 23, Bylinskii discloses a system for predicting an attention level of a viewer corresponding to one or more portions of a media item, comprising (col 5, “As shown, the system 100 may comprise a processor 102, a non-transitory memory 104, and an electronic display 105.”; also, col 3-4, “For instance, an importance map may be a graphical representation showing each pixel of the graphic object within a range from least important to most important where a higher importance indicates a higher probability that the pixel is viewed by a viewer.”): a display (col 5, “As shown, the system 100 may comprise a processor 102, a non-transitory memory 104, and an electronic display 105.”; also, col 11, “At step 806, the computer may display the importance map. The computer may display the importance map in the GUI side-by-side with the input graphic object.”); one or more processors (col 6, “The processor 102 may execute multiple engines (e.g., a real-time feedback engine 108, a retargeting engine 110, a smart thumbnails engine 112, a color palette (or theme) extraction engine 114) that may, in turn, execute a neural network 106 stored in the non-transitory memory 104.”); a memory (col 6, “The processor 102 may execute multiple engines (e.g., a real-time feedback engine 108, a retargeting engine 110, a smart thumbnails engine 112, a color palette (or theme) extraction engine 114) that may, in turn, execute a neural network 106 stored in the non-transitory memory 104.”); and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, the one or more programs including instructions for (col 6, “The processor 102 may execute multiple engines (e.g., a real-time feedback engine 108, a retargeting engine 110, a smart thumbnails engine 112, a color palette (or theme) extraction engine 114) that may, in turn, execute a neural network 106 stored in the non-transitory memory 104.”; also, col 11, “At step 806, the computer may display the importance map. The computer may display the importance map in the GUI side-by-side with the input graphic object”): receiving the media item (col 11, “The method may begin at step 802, where the computer may receive an input graphic object. The input graphic object may be a graphic design or a data visualization. In some instances, the computer may display GUI with an upload tool for a user to upload an image, e.g., in a bitmap format, containing the input graphic object.”); segmenting the media item into a plurality of segments (col 5, “The notion of importance may also depend upon higher level factors such as semantic categories of design elements (e.g., title text, axis text, data points) within a graphic object.”; also, col 7, “For example, the smart thumbnail image 118c for a graphic object that is a data visualization may include title and other main supporting text, as well as data extremes—in case of a data table, top and bottom of the table, and in case of a data plot, left and right sides of the data plot.”), one or more characteristics associated with each segment of the plurality of segments (col 9, “An instruction to edit/update the input graphic object may include, for example, an instruction to resize a text or an image in the graphic object, an instruction to change font of a text in the graphic object, an instruction to change a color of a text or an image in the graphic object, and an instruction to move a text or an image in the graphic object.”; also, col 12, “More particularly, the user may change font, size, or color of the text or rearrange the text or images. The user may also resize, crop, and/or change one or more colors of the picture.”); providing the segmented media item and identified characteristics to a machine- learning model (col 11, “At step 804, the computer may execute or deploy a neural network to generate an importance map for the input graphic object. The neural network may have been trained to optimize a loss function containing a pixel-level comparison of the importance output generated by the neural network and the ground truth dataset.”; also, col 7, “In some embodiments, the neural network 106 may be a convolutional neural network.”; also, col 5, “A computer may train one or more neural networks using bubble view and explicit importance map datasets as ground truth datasets. Ground truth datasets, as described herein, may include pre-classified data used to train the one or more neural networks by comparing the outputs of the one or more neural networks with the corresponding ground truth data. Conventional eye gaze measurements that are used to generate ground truth datasets of human attention, which are generally expensive and time consuming, and may not be feasible for practical purposes. The use of bubble view and explicit importance datasets are generally inexpensive, and provide close approximation of an eye gaze measurement.”) generating a heat map that predicts (col 11-12, “For example, the importance map may be analogous to a heat map with different colors or shades of colors to represent the relative importance of various pixels within the input graphic object.”; also, col 3, “Importance maps may indicate the relative importance of various regions of a graphic design or data visualization.”), respective attention level of the viewer (col 6, “The dynamic importance map 118a, through the color codes, may show predictions generated by the feedback engine 108 as to where viewers are expected to look at the input graphic object 116a.”) respective attention level for the respective segment is based on the segmented media item, the identified characteristics, and the empirical data (col 11, “At step 804, the computer may execute or deploy a neural network to generate an importance map for the input graphic object. The neural network may have been trained to optimize a loss function containing a pixel-level comparison of the importance output generated by the neural network and the ground truth dataset. In other words, the neural network may generate a pixel-level importance map for the input graphic object.”; also, col 11, “At step 806, the computer may display the importance map. The computer may display the importance map in the GUI side-by-side with the input graphic object.”; also, col 4, “An importance map may, for example, show a color coded graphic object with different shades of colors overlaid on the graphic object showing the importance of the respective regions of the graphic object, analogous to a heat map.”). Bylinskii does not disclose wherein each segment of the plurality of segments bounds one or more text items or image items; that has been trained using empirical data from identified user gaze locations on training media items; for each segment of the plurality of segments bounding one or more text items or image items; associated with the entirety of the respective segment corresponding to a respective relative amount of time that a viewer is predicted to look at the respective segments.
However, in a similar field of endeavor, Jetley discloses that has been trained using empirical data from identified user gaze locations on training media items (para 57, “As illustrated in FIG. 3, in the exemplary embodiment, ground-truth saliency maps, referred to herein as attention maps 56, are constructed from the aggregated fixations of multiple observers, ignoring any temporal fixation information. Areas with a high fixation density are interpreted as receiving more attention.”; also, para 58, “For example, at S202, eye gaze data is obtained for a set of observers for each training image (such as at least two observers). Eye gaze data may be obtained by observing the focus of the person’s eye on an image, or simulated by receiving mouse clicks on the image made by observers. The collected eye gaze data is combined to generate a set of fixation coordinates 71.”; also, para 40, “At S104, the neural network 64 is trained on the set of training images 62 and their associated attention maps 56 to predict saliency maps 12 for new images 14. The training includes learning weights for layers of the neural network through backward passes of the neural network, as described in more detail with respect to FIG. 4.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have modified Bylinskii's invention of a system having a processor, a memory and an electronic display that receives a media item, segmenting the media item into a plurality of segments, identifying a characteristic associated with each segment, providing the segmented media item and the identified characteristics to a neural network, and generating a heat map that predicts a respective attention level of the viewer with the features of Jetley's invention of training that neural network using empirical eye-gaze data collected from observers viewing training media items. The combination would have been obvious because Bylinskii trains its neural network by a pixel-level comparison between the network output and a ground truth dataset without fixing what that ground truth is measured from, and Jetley supplies a ground truth of exactly the kind Bylinskii's loss function consumes, an attention map constructed from the aggregated fixations of multiple observers of each training image. A person of ordinary skill would have looked to Jetley because both references train a convolutional network to output a per-location prediction of where a viewer will look on a displayed item, so supplying the training stage with measured fixation data rather than an unspecified ground truth requires no change to the network and yields the predictable result of an attention prediction learned from empirical user gaze locations. Bylinskii's preference for its own crowd-sourced datasets does not discourage the combination, because Bylinskii sets conventional eye gaze measurements aside as expensive and time consuming and as possibly not feasible for practical purposes rather than as technically unsuited, and Bylinskii measures the quality of its own datasets against eye gaze when it states that they provide close approximation of an eye gaze measurement, so for a person of ordinary skill who already has eye-tracking data in hand, as Jetley does, Bylinskii's own benchmark is the reason to use it.
Gupta discloses each segment of the plurality of segments bounds one or more text items or image items (para 27, “Each content item in the set of training content items 132 may include a plurality of elements. An element is any object on which the content creator can take action. Examples of elements include a title, a heading, an image, a paragraph, a table, a text box, a button, a link, a span, etc.”; also, para 27, “For example, a GUI (graphical user interface) may enable a content creator to designate an element by drawing a box or other shape around the element.”; also, para 57, “Process 600 begins by determining the elements of the content item (602). In some implementations, the content elements may be identified or tagged as such by a user. For example, a user, such as the content creator or the system administrator setting up the training content items, may use a graphical user interface to draw a bounding box around each element in the content item.”; also, para 23, “The lowest zoom level may correspond to a bounding box. A bounding box is the smallest shape, typically but not always a rectangular box, that encloses the entire element.”); for each segment of the plurality of segments bounding one or more text items or image items (para 40, “A saliency map shows the elements in a content item represented by their respective saliency scores. In some implementations, the saliency map may be a heat map, which uses differing colorations representing different ranges of saliency scores.”; also, para 63, “In the example of FIG. 8 the content item 700 includes seven different elements. Accordingly, the saliency map 800 includes seven element areas, each area corresponding to one of the elements. The element areas of the saliency map 800 are laid out based on the location of the corresponding element in the original content item. This layout provides a quick visual correspondence between the areas and the original content item.”); associated with the entirety of the respective segment (para 63, “Each element area in the saliency map has a coloration that corresponds to the saliency code determined by the content saliency neural network using the techniques described herein.”; also, para 23, “The lowest zoom level may correspond to a bounding box. A bounding box is the smallest shape, typically but not always a rectangular box, that encloses the entire element.”) corresponding to a respective relative amount of time that a viewer is predicted to look at the respective segments (para 31, “The training module 130 may generate a saliency score for each element in each training content item. The saliency score represents how often and for how long the workers gazed at the element during the tasks and is based on a proportion of the total possible gaze points across all workers for the content item that correspond with the element. Put another way, the saliency score is a measure of how much the element stands out relative to its neighbors. For example, a content item with four elements viewed by four workers over five seconds generates a total of 20 possible gaze point locations. If two of the workers gaze at element 1 for all 5 seconds, and two of the workers gaze at element 1 for two seconds and element 3 for three seconds, the training module 130 may determine that elements 2 and 4 have a saliency score of zero, element 3 has a saliency score of 0.3, and element 1 has a saliency score of 0.7.”; also, para 55, “The system may determine the saliency score for an element by computing the proportion of gaze points in the set of gaze points (e.g., over all workers) that fall on any pixel belonging to the element.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Bylinskii in view of Jetley, in which a neural network trained on measured observer fixations generates a heat map predicting where a viewer will look on a segmented media item, with the features of Gupta's invention of bounding each element of a content item in a box that encloses the entire element, scoring each bounded element by the proportion of the observers' viewing time that fell on it, and rendering one coloration per element area as a heat map. The combination would have been obvious because the base combination already identifies the portions of a graphic object by semantic category, naming title text, axis text and data points, yet reports its prediction as a value per pixel, which leaves the content creator to read an element-level judgment out of a field of pixels. Gupta solves that problem in the same art by attaching a single score to a whole bounded element and coloring the element area with it, and Gupta states the benefit in its own terms when it explains that laying the element areas out at the location of the corresponding element provides a quick visual correspondence between the areas and the original content item. A person of ordinary skill working on Bylinskii's design-feedback tool would have looked to Gupta because both references predict, from a neural network trained on human viewing data, where a viewer will look on a laid-out page of text and images, and the predictable result of scoring and coloring whole bounded elements instead of individual pixels is a heat map whose value at each bounded segment is the share of viewing time that segment is predicted to attract.
Claim(s) 24-26 is/are rejected under 35 U.S.C. 103 as being unpatentable over Bylinskii et al. (U.S. Doc. No. 11189066) as modified by Jetley et al. (U.S. Pub. No. 20170308770) and Gupta et al. (U.S. Pub. No. 20190147288), further in view of Nguyen et al. (U.S. Pub. No. 20160034441).
Regarding claim 24, Bylinskii as modified by Jetley and Gupta discloses the method of claim 1, wherein Bylinskii further discloses segmenting the media item into the plurality of segments comprises (col 5, “The notion of importance may also depend upon higher level factors such as semantic categories of design elements (e.g., title text, axis text, data points) within a graphic object.”; also, col 7, “For example, the smart thumbnail image 118c for a graphic object that is a data visualization may include title and other main supporting text, as well as data extremes—in case of a data table, top and bottom of the table, and in case of a data plot, left and right sides of the data plot.”): not disclose applying optical character recognition to the media item to identify text; and segmenting the media item into portions based on the identified text.
However, in a similar field of endeavor, Nguyen discloses applying optical character recognition to the media item to identify text (para 65, “At block 140, an OCR process is performed to identify areas of text in input image 205. This may include, e.g., determining orientation and proximity of text candidates. There are two different outputs of OCR process 140. One output (element 215) shows where individual words are identified and placed within their own bounding boxes ("one box per word"). The other output (element 220) shows where lines of text are identified without regard to individual word boundaries, and bounding boxes are identified for each line ("one box per line").”); and segmenting the media item into portions based on the identified text (para 97, “In this operation, OCR words (in image 510) and OCR text lines (in image 220) are merged into text blocks (put another way, OCR text lines are split into text boxes).”; also, para 97, “Accordingly, a text line is split if the word-level OCR indicates that the distance between two words exceeds a heuristic threshold (e.g., the height of the words).”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Bylinskii as modified by Jetley and Gupta , in which a media item is divided into segments that bound its text and image items and a trained network predicts a per-segment attention level, with the features of Nguyen's invention of running optical character recognition over the image of the item to locate its text and placing a bounding box around each word and each line that the recognition returns. The combination would have been obvious because the base combination requires segments that bound text items but obtains its element boundaries from a content creator's manual tagging or from markup already present in the source, neither of which is available when the media item is received as a rendered image, which is the form Bylinskii expressly contemplates when it receives a bitmap uploaded through the interface. Nguyen answers that gap for exactly that input by deriving the boundaries from the rendered pixels themselves. A person of ordinary skill would have looked to Nguyen because Nguyen decomposes a rendered two-dimensional layout into bounded text and non-text regions, which is the same operation the base combination performs by other means, and the predictable result is a segmentation of the media item into portions delimited by the text that the optical character recognition identifies.
Regarding claim 25, Bylinskii as modified by Jetley and Gupta discloses the method of claim 1, wherein Bylinskii further discloses segmenting the media item into the plurality of segments comprises (col 5, “The notion of importance may also depend upon higher level factors such as semantic categories of design elements (e.g., title text, axis text, data points) within a graphic object.”; also, col 7, “For example, the smart thumbnail image 118c for a graphic object that is a data visualization may include title and other main supporting text, as well as data extremes—in case of a data table, top and bottom of the table, and in case of a data plot, left and right sides of the data plot.”): applying computer vision techniques to the media item to identify one or more image items; and segmenting the media item into portions based on the identified one or more image items.
However, in a similar field of endeavor Nguyen discloses applying computer vision techniques to the media item to identify one or more image items (para 65, “At block 130, computer vision techniques are applied to the input graphical representation (element 205) to identify attributes (e.g., candidate atomic elements) of the UI screen. The output of block 130 is a view hierarchy representation of vision boxes, i.e., nested bounding boxes of screen elements identified using computer vision (element 210).”; also, para 73, “First, input image (graphical representation) 205 is processed using a Canny edge detection algorithm to identify or detect edges of items and objects (image elements) included in input image 205 to produce a modified Canny image 310.”; also, para 65, “Alternatively, if a vision box and underlying words are not aligned then the vision box likely represents an icon or pure graphical (i.e., non-text) element.”); and segmenting the media item into portions based on the identified one or more image items (para 65, “At block 130, computer vision techniques are applied to the input graphical representation (element 205) to identify attributes (e.g., candidate atomic elements) of the UI screen. The output of block 130 is a view hierarchy representation of vision boxes, i.e., nested bounding boxes of screen elements identified using computer vision (element 210).”; also, para 73, “First, input image (graphical representation) 205 is processed using a Canny edge detection algorithm to identify or detect edges of items and objects (image elements) included in input image 205 to produce a modified Canny image 310.”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Bylinskii as modified by Jetley and Gupta , in which a media item is divided into segments that bound its text and image items and a trained network predicts a per-segment attention level, with the features of Nguyen's invention of applying computer vision to the rendered image to detect the edges of the items and objects it contains, returning nested bounding boxes for them, and treating a box that does not align with recognized words as a graphical rather than a textual element. The combination would have been obvious because the base combination must bound image items as well as text items, and a text-only recognition pass leaves the pictures, icons and charts in a graphic design unbounded. A person of ordinary skill would have looked to Nguyen because Nguyen solves that same residual problem on the same kind of input, a rendered page of mixed text and graphics, and the predictable result is a segmentation of the media item into portions delimited by the image items that the computer vision pass identifies.
Regarding claim 26, Bylinskii as modified by Jetley and Gupta discloses the method of claim 1, segmenting the media item into the plurality of segments is based on distances between the identified one or more text items or image items.
However, in a similar field of endeavor, Nguyen discloses wherein segmenting the media item into the plurality of segments is based on distances between the identified one or more text items or image items (para 97, “Accordingly, a text line is split if the word-level OCR indicates that the distance between two words exceeds a heuristic threshold (e.g., the height of the words).”; also, para 97, “In this operation, OCR words (in image 510) and OCR text lines (in image 220) are merged into text blocks (put another way, OCR text lines are split into text boxes).”).
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to have further modified Bylinskii as modified by Jetley and Gupta , in which a media item is divided into segments that bound its text and image items, with the features of Nguyen's invention of forming those bounded regions by comparing the gap between neighboring recognized items against a threshold and splitting or merging them on that comparison. The combination would have been obvious because the base combination's segments must each bound one or more items, which requires a rule for deciding which neighboring items belong to the same segment, and spacing is the signal a rendered layout actually carries for that decision. Nguyen states the problem in its own terms when it observes that a recognized line of text often blends unrelated content that merely happens to be printed on the same line, and resolves it by the distance between the items. A person of ordinary skill would have adopted that rule because it needs nothing beyond the geometry already produced by the recognition pass, with the predictable result that the segmentation of the media item is based on distances between the identified text items or image items.
Response to Arguments
Applicants arguments filed 09/01/2026 has been fully considered.
On page 11 of the Applicant's Remarks, with respect to the rejections of claims 8, 9, 10, 13 and 22 under 35 U.S.C. 112(b), the Applicant argues that claim 8 has been amended to depend on claim 7 so that the term "user interface" has proper antecedent basis, that claims 9 and 10 have been amended to recite "eye-tracking information" rather than "eye-tracking data" so that the recited term has proper antecedent basis, that claim 13 has been amended to remove the term "the media item" so that claim 13 is in proper antecedent basis form, and that claim 22 has been amended to recite "the training media item" rather than "the media item." These arguments are persuasive with respect to each of claims 8, 9, 10, 13 and 22. Those rejections have therefore been withdrawn.
On page 13 of the Applicant's Remarks, with respect to the rejection of claims 1, 10 and 23, the Applicant argues that "the language relied on by the Examiner merely discloses the existence of different design elements in a media item, rather than segmenting a media item into segments bounding design elements in a media item," and that Bylinskii therefore fails to disclose or suggest segmenting the media item into a plurality of segments wherein each segment of the plurality of segments bounds one or more text items or image items. This argument is persuasive only as to the language the amendment added. The act of segmenting the media item into segments, each segment corresponding to a portion of the media item, was recited in claim 1 as originally examined and was found in Bylinskii at columns 5 and 7 in the Office action of June 3, 2026; the amendment replaced "one or more segments" with "a plurality of segments" and did not introduce that act, so the finding on it is maintained. As shown in the citations, Bylinskii discloses that "For example, the smart thumbnail image 118c for a graphic object that is a data visualization may include title and other main supporting text, as well as data extremes—in case of a data table, top and bottom of the table, and in case of a data plot, left and right sides of the data plot." (Bylinskii, column 7), which identifies the separate portions into which one graphic object is divided, and that "The notion of importance may also depend upon higher level factors such as semantic categories of design elements (e.g., title text, axis text, data points) within a graphic object." (Bylinskii, column 5). To that extent the Applicant's argument is persuasive, and upon further consideration and as necessitated by Applicant's amendment a new ground of rejection under 35 U.S.C. 103 over Bylinskii as modified by Jetley and Gupta , is made as set forth above. The newly added requirement that each segment bound one or more text items or image items is addressed above under Gupta, which discloses that "The lowest zoom level may correspond to a bounding box. A bounding box is the smallest shape, typically but not always a rectangular box, that encloses the entire element." (Gupta, paragraph [0023]) and that "Each content item in the set of training content items 132 may include a plurality of elements. An element is any object on which the content creator can take action. Examples of elements include a title, a heading, an image, a paragraph, a table, a text box, a button, a link, a span, etc." (Gupta, paragraph [0027]).
On pages 13 and 14 of the Applicant's Remarks, with respect to the rejection of claims 1, 10 and 23, the Applicant argues that "the heat map of Bylinskii is generated based on an importance level associated with each pixel of a graphic object, rather than segments of the graphic object bounding one or more text items or image items," resting that argument on Bylinskii's statement that "The importance map may be a graphical representation showing each pixel of the graphic object within a range from least important to most important." This argument is not persuasive as to the generating of a heat map that predicts an attention level of the viewer, and is persuasive only as to the language the amendment added. As shown in the citations, Bylinskii discloses that "Importance maps may indicate the relative importance of various regions of a graphic design or data visualization." (Bylinskii, column 3) and that "An importance map may, for example, show a color coded graphic object with different shades of colors overlaid on the graphic object showing the importance of the respective regions of the graphic object, analogous to a heat map." (Bylinskii, column 4), so the sentence the Applicant quotes states the resolution at which Bylinskii computes importance and not the unit at which Bylinskii reports it, and the generating of a heat map predicting the attention level of the viewer corresponding to the segments of the media item was recited in claim 1 as originally examined and found in Bylinskii at columns 11 and 12. To that extent the Applicant's argument is persuasive, and upon further consideration and as necessitated by Applicant's amendment a new ground of rejection under 35 U.S.C. 103 over Bylinskii as modified by Jetley and Gupta , is made as set forth above. The added requirements that the predicted value be a respective value associated with the entirety of the respective segment and that it correspond to a respective relative amount of time the viewer is predicted to look at that segment are addressed above under Gupta, which discloses that "Each element area in the saliency map has a coloration that corresponds to the saliency code determined by the content saliency neural network using the techniques described herein." (Gupta, paragraph [0063]) and that "The saliency score represents how often and for how long the workers gazed at the element during the tasks and is based on a proportion of the total possible gaze points across all workers for the content item that correspond with the element." (Gupta, paragraph [0031]).
On page 14 of the Applicant's Remarks, with respect to the rejection of claims 1, 10 and 23, the Applicant argues that "Jetley fails to cure the deficiencies of Bylinskii and is not relied on by the Examiner for such purpose." This argument has been considered but is moot because it does not apply to the new combination of references being used in the current rejection. Jetley is relied upon for the training of the machine-learning model using empirical data from identified user gaze locations on training media items, and the limitations the Applicant identifies as added by the amendment are supplied by Gupta.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to Jai Li whose telephone number is (571)272-1170. The examiner can normally be reached Mon-Thu between 06:00-16:00 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Xiao Wu can be reached at (571)272-7761. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/JAI W LI/Junior Patent Examiner, Art Unit 2613
/XIAO M WU/Supervisory Patent Examiner, Art Unit 2613
