DETAILED ACTION
This office action is responsive to communication(s) filed on 12/17/2025.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Continued Examination Under 37 CFR 1.114
A request for continued examination under 37 CFR 1.114, including the fee set forth in 37 CFR 1.17(e), was filed in this application after final rejection. Since this application is eligible for continued examination under 37 CFR 1.114, and the fee set forth in 37 CFR 1.17(e) has been timely paid, the finality of the previous Office action has been withdrawn pursuant to 37 CFR 1.114. Applicant's submission filed on 12/17/2025 has been entered.
Claims Status
Claims 1-9 and 17-27 are pending and are currently being examined.
Claims 1, 17 and 21 are independent.
Claims 10-16 are previously canceled.
Claims 1, 6, 17, 21, 22, 23 and 24 are newly amended.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1-2, 17 and 19 is/are rejected under 35 U.S.C. 103 as being unpatentable over Biswas; Mainak et al. (hereinafter Biswas – US 20190332861 A1) in view of Jovanovic; Katarina et al. (hereinafter Jovanovic – US 20210200942 A1).
Independent Claim 1:
Biswas teaches:
A method comprising: (e.g., see fig. 4)
generating a design representation (determine layout [design representation] of items of a digital content with respect to each other, fig. 4:404 and ¶ 58),
comprising design properties of a digital design document having a plurality of digital design elements; (the layout defining design properties for the items [plurality of digital design elements] of the digital content [digital design document], such as positioning and dimensions [e.g., width height], ¶¶ 59 and 92, and arrangement of layout)
constructing a digital design graph from the design representation (using the layout [from the design representation], generating a layout tree, fig. 4:406 and ¶ 63, which is used to generate a decision tree [digital design graph], figs. 4:408 and 19 and ¶ 102)
by: […] generating nodes based on the plurality of digital design elements, wherein the nodes comprise an image node that represents the image design element and a text node that represents the text design element of the plurality of digital design elements; (the decision tree generated has nodes representing one or more digital items [generating nodes based on the plurality of digital design elements], ¶¶ 109-110 and fig. 19. The items include text [text design element of the plurality of digital design elements] and images [image design element], such as icons or three dimensional images, col 3:39-57)
and generating edges between the nodes (the generated decision tree includes lines representing connections/paths [generating edges between the nodes] between the decision nodes, as illustrated in fig. 19. Also see ¶¶ 113-114)
based on [design properties]; (the generated decision tree includes certain arrangement rules that are based on design properties, such as height of the items, ¶¶ 104 and 122)
generating a structural representation for the digital design document based on the digital design graph; (generating a digital content design, including arrangement of items, fig. 4:412, the content design is based on the a layout definition [structural representation] that is derived from the decision tree [based on the digital design graph], fig. 4:408-412, ¶¶ 102, 115, and 123 and fig. 3.)
and based on the structural representation, generating a recommended revision to the digital design document, generating a modified digital design document, or identifying an additional digital design document corresponding to the digital design document. (generating a digital content design [generating a recommended revision to the digital design document], including arrangement of items, fig. 4:412, the content design is based on a layout definition [structural representation] that is derived from the decision tree [based on the digital design graph], fig. 4:408-412, ¶¶ 102, 115, and 123 and fig. 3. The user, or creative professional uses e.g., Adobe® InDesign® as interacting with Adobe® Experience Manager®, Abstract and ¶ 56. As such, the digital content design is herein broadly interpreted as being a recommended revision to the digital design document, because it can serve as a recommended starting point that users/creative professionals may choose to leave as-is or revise.)
Biswas does not appear to expressly teach, but Jovanovic teaches:
generating semantic weights reflecting semantic relationships between the design properties of the plurality of digital design elements by generating a first semantic weight reflecting a measure of common semantic meaning in a semantic feature space between an image design element and a text design element in the digital design document (in determining alternative layouts, weighted scores for an alternative layouts can be determined based on a degree to which the content elements [semantic feature space] are symmetrical (e.g., left and right side symmetry, top and bottom symmetry, etc.) and/or amount of space between elements, where alternative layouts with better symmetry and/or less space between elements are scored higher, ¶¶ 7, 57 and 61. These elements include image and textual elements, such as an image and its caption, ¶ 11 [semantic feature space between an image design element and a text design element in the digital design document].)
that the design properties are “distances between the plurality of digital design elements and the semantic weights by generating at least one edge between the image node and the text node based on a distance between the image design element and the text design element within the digital design document and the first semantic weight reflecting the measure of common semantic meaning in the semantic feature space between the image design element and the text design element” (in determining alternative layouts, weighted scores for an alternative layouts can be determined based on a degree to which the content elements [semantic feature space] are symmetrical (e.g., left and right side symmetry, top and bottom symmetry, etc.) and/or amount of space between elements [distances between the plurality of digital design elements and the semantic weights], where alternative layouts with better symmetry and/or less space between elements [common semantic meaning] are scored higher, ¶¶ 7, 57 and 61. This ensures that semantic relationships between elements [reflecting the measure of common semantic meaning in the semantic feature space] are preserved in the alternative layouts, ¶ 10. These elements include image and textual elements, such as an image and its caption, ¶ 11.)
Accordingly, it would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to modify the method of Biswas to include generating semantic weights reflecting semantic relationships between the design properties of the plurality of digital design elements, and that the design properties are “distances between the plurality of digital design elements and the semantic weights”, as taught by Jovanovic.
One would have been motivated to make such a combination in order to improve the usability of the method by improving the visual quality of the representation of the design document, by ensuring it is more visually appealing and aligned with original user’s intent, Jovanovic ¶¶ 17 and 62.
Claims 2:
The rejection of claim 1 is incorporated. Biswas further teaches:
wherein generating the design representation comprising the design properties further comprises:
extracting element properties by extracting types of elements or source URLs for the plurality of digital design elements; (information of original content is received [extracting] and is updated to have a consistent look and feel [typo in reference states “look-and-field”] with the original design, ¶¶ 56 and 114. A look and feel similarity is considered when determining an appropriate layout based on “adjacency” [element properties] between items of the digital content, ¶¶ 33 and 57. The content takes on a variety of forms, ¶ 46, e.g., a single node may have two type of digital content, ¶ 118. As such, the exacting of content is herein interpreted as “extracting types of elements…for the plurality of digital design elements”)
and extracting geometric properties from the digital design document by extracting element sizes, or element locations of the plurality of digital design elements. (information of original content is received [extracting] and is updated to have a consistent look and feel [typo in reference states “look-and-field”] with the original design, ¶¶ 56 and 114. A look and feel similarity is considered when determining an appropriate layout based on “adjacency” [element properties] between items of the digital content, ¶¶ 33 and 57. This similarity is also achieved by considering original height and width [extracting geometric properties…extracting element sizes] ensure a width-height ratio [element properties] of the original items is maintained, ¶¶ 118 and 122. Because these original properties are considered and maintained when layout is updated, it is interpreted that these properties are received/extracted as part of receiving the original content. For purposes of compact prosecution only, the examiner interprets the limitation(s) as being directed to “element”, “geometric”, and “style properties” that are part of the design properties, and that “element properties” includes geometric and non-geometric features other than size and position/location, such as name of the height-width ratio of the elements, and that “geometric properties” include features such as length and width of the elements)
Independent Claim 17:
Claim(s) 17 is directed to a computer-readable medium for accomplishing the steps of the method in claim 1, and is rejected using similar rationale(s).
Claim 19:
The rejection of claim 17 is incorporated. Biswas-Jovanovic further teaches
generating the edges between the nodes [based on the design properties] (the generated decision tree includes a chain of arrangement rules that are based on the design properties, such as height of the items, Biswas ¶¶ 104 and 12)
wherein constructing the digital design graph further comprises: generating continuity weights, symmetry weights, or closure weights, between the plurality of digital design elements and that the generating of the edges are “based on the continuity weights, the symmetry weights, or the closure weights between the plurality of digital design elements” (in determining alternative layouts, a weighted score for an alternative layout can be determined based on a degree to which the content elements are symmetrical (e.g., left and right side symmetry, top and bottom symmetry, etc.) [symmetry weights], where alternative layouts with better symmetry are scored higher, Jovanovic ¶¶ 19, 57 and 61).
Claim(s) 3 is/are rejected under 35 U.S.C. 103 as being unpatentable over Biswas (US 20190332861 A1) and Jovanovic ( US 20210200942 A1), as applied to claim 1 above, and further in view of Kol; Tal et al. (hereinafter Kol – US 20170032050 A1).
Claim 3:
The rejection of claim 1 and is incorporated. Biswas further teaches that information of original content is received [extracting, from the digital design document,] and is updated to have a consistent look and feel [typo in Biswas states “look-and-field”] with the original design, ¶¶ 56 and 114. A look and feel similarity is considered when determining an appropriate layout based on “adjacency” [style properties] between items of the digital content, ¶¶ 33 and 57. This similarity is also achieved by considering original height and width [style properties] ensure a width-height ratio of the original items is maintained, ¶¶ 118 and 122. Because these original properties are considered and maintained when layout is updated, it is interpreted that these properties are received/extracted as part of receiving the original content.
Biswas-Jovanovic does not appear to expressly teach, but Kol teaches:
wherein generating the design representation comprising the design properties further comprises extracting, from the digital design document, style properties by extracting colors, opacities of the plurality of digital design elements, backgrounds, or blend modes or extracting, from the digital design document, inferred tags by extracting meta data from the digital design document. (the converting of a website to an application, by extracting a set of brand colors from a website and setting an associated color theme for the brand colors [extracting, from the digital design document, style properties by extracting colors, opacities of the plurality of digital design elements, backgrounds, or blend modes], and assigning multiple colors within the theme to a plurality of user interface elements of the application version of the website, in order to preserve the look and feel of the website, Abstract and ¶ 21)
Accordingly, it would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to further modify the method of Biswas to include wherein generating the design representation comprising the design properties further comprises extracting, from the digital design document, style properties by extracting colors, opacities of the plurality of digital design elements, backgrounds, or blend modes or extracting, from the digital design document, inferred tags by extracting meta data from the digital design document, as taught by Kol.
One would have been motivated to make such a combination in order to achieve the preservation of look and feel of documents in a known and effective manner, Kol ¶ 21.
Claim(s) 5, 7, 9 and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Biswas (US 20190332861 A1) and Jovanovic ( US 20210200942 A1), as applied to claims 1 and 17 above, and further in view of Li; Weiyu (hereinafter Weiyu_L – US 20240256951 A1, filed on 1/27/2023, before instant application).
Claim 5:
The rejection of claim 1 is incorporated. Biswas further teaches layout analysis that includes combining items into item groups, e.g., see ¶ 95, which are used to generate layout tree, and defining relationships between the representations of items [rectangles] can be done using machine learning, ¶ 99.
Biswas-Jovanovic does not appear to expressly teach, but Weiyu_L teaches:
wherein generating the semantic weights constructing the digital design graph further comprises:
generating, utilizing a trained embedding model, embeddings of the plurality of digital design elements in a semantic feature space; (a neural network model that is trained [utilizing a trained embedding model] to build models that obtain object embeddings [generating…embeddings of the plurality of digital design elements], ¶ 141, wherein the embeddings are obtained by obtaining a numerical representation of the object, and the embeddings/vectors include properties that are usable for computing things like groupings and rankings, ¶ 179 and fig. 10. The objects are related, e.g., semantically [digital design elements in a semantic feature space], ¶ 141)
comparing the embeddings of the plurality of digital design elements to determine measures of semantic similarity; (comparing the embeddings of different objects [of the plurality of digital design elements] can result in outputting similarity scores [to determine measures of semantic similarity] for the objects that are used to group the different objects together, ¶ 198. Such models are trained based on one or more graphs representing the objects [based on the digital design graph], e.g., see ¶ 197 and fig. 1)
and generating the semantic weights based on the measures of semantic similarity. (outputting similarity scores [generating the semantic weights based on the measures of semantic similarity] for the objects that are used to group the different objects together, ¶ 198)
Accordingly, it would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to further modify the method of Biswas to include generating, utilizing a trained embedding model, embeddings of the plurality of digital design elements in a semantic feature space; comparing the embeddings of the plurality of digital design elements to determine measures of semantic similarity; and generating the semantic weights based on the measures of semantic similarity, as taught by Weiyu_L.
One would have been motivated to make such a combination in order to improve the performance of the method by increasing recall of related objects for grouping, Weiyu_L ¶ 143 and Biswas ¶¶ 95 and 99.
Claim 7:
The rejection of claim 1 is incorporated. Biswas further teaches layout analysis that includes combining items into item groups, e.g., see ¶ 95, which are used to generate layout tree, and defining relationships between the representations of items [rectangles] can be done using machine learning, ¶ 99.
However, Biswas-Jovanovic does not appear to expressly teach, but Weiyu_L teaches:
wherein generating the structural representation further comprises determining element groupings by: generating, utilizing a grouping machine-learning model, a grouping score from a first digital design element and a second digital design element within the digital design document based on the digital design graph; (a neural network model [machine-learning model] that is trained to build models that obtain object embeddings [e.g., vectors of numbers], ¶ 141, wherein the embeddings are obtained by obtaining a numerical representation of the object [representation of structural representation], and the embeddings/vectors include properties that are usable for computing things like groupings and rankings, ¶ 179 and fig. 10. Comparing embeddings of different objects [e.g., alerts] can result in outputting similarity scores [grouping score] for the objects that are used to group the different objects together, ¶ 198. Such models are trained based on one or more graphs representing the objects [based on the digital design graph], e.g., see ¶ 197 and fig. 10)
and determining that the first digital design element and the second digital design element are part of a first group based on the grouping score (Comparing embeddings of different objects [e.g., alerts] can result in outputting similarity scores [grouping score] for the objects that are used to group the different objects together, ¶ 198.).
Accordingly, it would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to further modify the method of Biswas to include wherein generating the structural representation further comprises determining element groupings by: generating, utilizing a grouping machine-learning model, a grouping score from a first digital design element and a second digital design element within the digital design document based on the digital design graph; and determining that the first digital design element and the second digital design element are part of a first group based on the grouping score, as taught by Weiyu_L.
One would have been motivated to make such a combination in order to improve the performance of the method by increasing recall of related objects for grouping, Weiyu_L ¶ 143 and Biswas ¶¶ 95 and 99.
Claim 9:
The rejection of claim 1 is incorporated. Biswas-Jovanovic does not appear to expressly teach, but Weiyu_L teaches:
comprises generating an embedding vector or an adjacency matrix as the structural representation that represents the digital design graph. (a neural network model that is trained to build models that obtain object embeddings, e.g., vectors of number [generating an embedding vector…as the structural representation that represents the digital design graph], ¶ 141)
Accordingly, it would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to further modify the method of Biswas to include comprises generating an embedding vector or an adjacency matrix as the structural representation that represents the digital design graph, as taught by Weiyu_L.
One would have been motivated to make such a combination in order to improve the performance of the method by increasing recall of related objects for grouping, Weiyu_L ¶ 143 and Biswas ¶¶ 95 and 99.
Claim 20:
The rejection of claim 17 is incorporated. Biswas further teaches:
and based on the structural representation, generating a recommended revision to the digital design document, generate a modified digital design document, or identify an additional digital design document corresponding to the digital design document. (generating a digital content design [generating a recommended revision to the digital design document], including arrangement of items, fig. 4:412, the content design is based on a layout definition [structural representation] that is derived from the decision tree [based on the digital design graph], fig. 4:408-412, ¶¶ 102, 115, and 123 and fig. 3. The user, or creative professional uses e.g., Adobe® InDesign® as interacting with Adobe® Experience Manager®, Abstract and ¶ 56. As such, the digital content design is herein broadly interpreted as being a recommended revision to the digital design document, because it can serve as a recommended starting point that users/creative professionals may choose to leave as-is or revise)
and a layout analysis that includes combining items into item groups, e.g., see ¶ 95, which are used to generate layout tree, and defining relationships between the representations of items [rectangles] can be done using machine learning, ¶ 99.
However, Biswas does not appear to expressly teach, but Weiyu_L teaches:
wherein the structural representation further comprises utilizing a machine learning model to generate an adjacency matrix or an embedding vector as the structural representation from the digital design graph (a neural network model [machine-learning model] that is trained to build models that obtain object embeddings, e.g., vectors of numbers [embedding vector as the structural representation], ¶ 141, wherein the embeddings are obtained by obtaining a numerical representation of the object [representation of structural representation], and the embeddings/vectors include properties [features] that are usable for computing things like groupings and rankings, ¶ 179 and fig. 10. Comparing embeddings of different objects [e.g., alerts] can result in outputting similarity scores [grouping score] for the objects that are used to group the different objects together, ¶ 198. Such models are trained based on one or more graphs representing the objects [from the digital design graph], e.g., see ¶ 197 and fig. 10. The vectors include object feature vectors [feature representation], ¶ 138).
Accordingly, it would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to further modify the method of Biswas to include wherein the structural representation further comprises utilizing a machine learning model to generate a feature representation from the digital design graph, as taught by Weiyu_L.
One would have been motivated to make such a combination in order to improve the performance of the method by increasing recall of related objects for grouping, Weiyu_L ¶ 143 and Biswas ¶¶ 95 and 99.
Claim(s) 4 and 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Biswas (US 20190332861 A1) and Jovanovic (US 20210200942 A1), as applied to claims 1 and 17 above, and further in view of Roberts; Tara U. et al. (hereinafter Roberts – US 20190102681 A1).
Claim 4:
The rejection of claim 1 is incorporated. Biswas-Jovanovic does not appear to expressly teach, but Roberts teaches:
wherein generating the design representation further comprises anonymizing the digital design document by encoding user-identifiable information from text strings within the digital design document (a notification [digital design document] layout template, generated by a machine-learning model, ¶ 56, includes dynamic text components, which include anonymized data or partially anonymized data, ¶ 58, wherein the data is stripped from personally identifiable information PII [user-identifiable information], ¶ 41, according to privacy policies and laws, ¶ 28).
Accordingly, it would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to further modify the method of Biswas to include wherein generating the design representation further comprises anonymizing the digital design document by encoding user-identifiable information from text strings within the digital design document, as taught by Roberts.
One would have been motivated to make such a combination in order to improve the method by allowing the generating of design layouts that comply with privacy policies and laws, Roberts ¶ 28.
Claim 18:
The rejection of claim 17 is incorporated. Biswas further teaches:
wherein generating the design representation comprising the design properties further comprises:
extracting from the digital design document at least one of style properties or inferred tags; (information of original content is received [extracting from the digital design document] and is updated to have a consistent look and feel [typo in Biswas states “look-and-field”] with the original design, ¶¶ 56 and 114. A look and feel similarity is considered when determining an appropriate layout based on “adjacency” [style properties] between items of the digital content, ¶¶ 33 and 57. This similarity is also achieved by considering original height and width [style properties] ensure a width-height ratio of the original items is maintained, ¶¶ 118 and 122. Because these original properties are considered and maintained when layout is updated, it is interpreted that these properties are received/extracted as part of receiving the original content. For purposes of compact prosecution only, the examiner interprets the limitation(s) as being directed to “element”, “geometric”, and “style properties” that are part of the design properties, and that “element properties” includes geometric and non-geometric features other than size and position/location, such as name of the height-width ratio of the elements, and that “geometric properties” include features such as length and width of the elements, and that style properties include geometric and element properties.)
However, Biswas-Jovanovic does not appear to expressly teach, but Roberts teaches:
and anonymizing the digital design document by encoding user-identifiable information from text strings within the digital design document (a notification [digital design document] layout template, generated by a machine-learning model, ¶ 56, includes dynamic text components, which include anonymized data or partially anonymized data [encoding], ¶ 58, wherein the data is stripped from personally identifiable information PII [user-identifiable information], ¶ 41, according to privacy policies and laws, ¶ 28).
Accordingly, it would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to further modify the medium of Biswas to include and anonymizing the digital design document by encoding user-identifiable information from text strings within the digital design document, as taught by Roberts.
One would have been motivated to make such a combination in order to improve the medium by allowing the generating of design layouts that comply with privacy policies and laws, Roberts ¶ 28.
Claim(s) 6 is/are rejected under 35 U.S.C. 103 as being unpatentable over Biswas (US 20190332861 A1) in view of Jovanovic (US 20210200942 A1) and Weiyu_L (US 20240256951 A1), as applied to claim 5 above, and further in view of Sharma; Natasha (hereinafter Sharma, Non-Patent Literature, “K-means Clustering Explained, 11/4/2021).
Claim 6:
The rejection of claim 5 is incorporated. Biswas-Jovanovic-Weiyu further teaches:
generating a first edge between the image node and the text node based on the distance between the image design element and the text design element within the digital design document; (in determining alternative layouts, weighted scores for an alternative layouts can be determined based on a degree [“edges”, or relationships] to which the content elements are symmetrical (e.g., left and right side symmetry, top and bottom symmetry, etc.) and/or amount of space between elements [distance], where alternative layouts with better symmetry and/or less space between elements are scored higher, Jovanovic ¶¶ 7, 57 and 61. This ensures that semantic relationships between elements are preserved in the alternative layouts, Jovanovic ¶ 10. These elements include image and textual elements, such as an image and its caption [distance between the image design element and the text design element], Jovanovic ¶ 11.)
and generating a second edge between the image node and the text node based on the first semantic weight reflecting the measure of common semantic meaning in the semantic feature space between the image design element and the text design element (in determining alternative layouts, weighted scores for an alternative layouts can be determined based on a degree [“edges”, or relationships] to which the content elements are symmetrical (e.g., left and right side symmetry, top and bottom symmetry, etc.) and/or amount of space between elements [distance], where alternative layouts with better symmetry and/or less space between elements are scored higher, Jovanovic ¶¶ 7, 57 and 61. This ensures that semantic relationships [reflecting the measure of common semantic meaning in the semantic feature space between the image design element and the text design element] between elements are preserved in the alternative layouts, Jovanovic ¶ 10. These elements include image and textual elements, such as an image and its caption [distance between the image design element and the text design element], Jovanovic ¶ 11.)
Biswas-Jovanovic-Weiyu_L does not appear to expressly teach, but Sharma teaches:
wherein constructing the digital design graph further comprises determining the distances between the plurality of digital design elements by determining distances between centroids of the plurality of digital design elements. (Datasets can contain millions of records and not all algorithms scale efficiently. K-Means is one of the most popular algorithms and it is also scale-efficient, Page 1, in which the objective is to minimize the sum of distances between the data points and the cluster centroid, to identify the correct group each data point should belong to, Page 2. It is can be seen that K-means clustering must determine the distances between the plurality of digital design elements by determining distances between centroids of the plurality of digital design elements. The distance calculation is the central mechanism that drives both the cluster assignment and centroid update steps of the algorithm.)
Accordingly, it would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to further modify the method of Biswas to include wherein constructing the digital design graph further comprises determining the distances between the plurality of digital design elements by determining distances between centroids of the plurality of digital design elements, as taught by Sharma .
One would have been motivated to make such a combination in order improve the efficiency in the method by being able to find unusual data points/outliers in the data or unknown properties to find a suitable grouping in the dataset, Sharma Page 1.
Claim(s) 8 is/are rejected under 35 U.S.C. 103 as being unpatentable over Biswas (US 20190332861 A1) in view of Jovanovic ( US 20210200942 A1), as applied to claim 1 above, and further in view of Li; Jia et al. (hereinafter Jia_L – US 20200372660 A1).
Claim 8:
The rejection of claim 1 is incorporated. Biswas further teaches that the visual structure is represented and generated based on one or more graphs that include leaf nodes, e.g., end nodes at the deepest level of the trees, figs. 15-17 and 19, and defining relationships between the representations of items [rectangles] can be done using machine learning, ¶ 99.
However, Biswas-Jovanovic does not appear to expressly teach, but Jia_L teaches:
wherein generating the structural representation further comprises determining a visual structure inference by:
generating foreground leaf nodes and background leaf nodes from the plurality of digital design elements within the digital design document; (content, e.g., image, is segmented/grouped into foreground and background portions, Abstract and ¶ 11)
determining, utilizing a grouping machine-learning model, a first element grouping from the foreground leaf nodes and a second element grouping from the background leaf nodes; (the neural network segments foreground and background portions [generating and determining…first…and a second element grouping], Abstract and ¶ 11)
and generating a visual structure inference comprising the first element grouping and the second element grouping (the neural network can generate foreground and background prediction results for the image [visual structure inference], Abstract and ¶ 11, which are fused together to product a full prediction [comprising the first element grouping and the second element grouping], ¶ 114 and fig. 4 ).
Accordingly, it would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to further modify the method of Biswas to include wherein generating the structural representation further comprises determining a visual structure inference by: generating foreground leaf nodes and background leaf nodes from the plurality of digital design elements within the digital design document; determining, utilizing a grouping machine-learning model, a first element grouping from the foreground leaf nodes and a second element grouping from the background leaf nodes; and generating a visual structure inference comprising the first element grouping and the second element grouping, as taught by Jia_L.
One would have been motivated to make such a combination in order to improve the capabilities of the method to include detecting the most important object, ¶ 3, using a powerful neural network [machine-learning] model, Jia_L ¶ 4.
Claim(s) 21, 22 and 25 is/are rejected under 35 U.S.C. 103 as being unpatentable over Biswas (US 20190332861 A1) in view of Jovanovic (US 20210200942 A1) and Roberts; Tara U. et al. (hereinafter Roberts – US 20190102681 A1).
Claim 21:
Claim 21 is includes similar limitations to those of claim 1 and is rejected over the prior art, as applied above to claim 1.
However, unlike claim 1, claim 21 also includes the following limitations, which Biswas-Jovanovic does not appear to expressly teach, but Roberts teaches:
generating an anonymized design representation from the design representation by encoding user-identifiable information from text strings within the digital design document and that the constructing of the graph is from the “anonymized” representation (a notification [digital design document] layout template, generated by a machine-learning model, ¶ 56, includes dynamic text components, which include anonymized data or partially anonymized data, ¶ 58, wherein the data is stripped from personally identifiable information PII [user-identifiable information], ¶ 41, according to privacy policies and laws, ¶ 28)
Accordingly, it would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to further modify the method of Biswas to include generating an anonymized design representation from the design representation by encoding user-identifiable information from text strings within the digital design document and that the constructing of the graph is from the “anonymized” representation, as taught by Roberts.
One would have been motivated to make such a combination in order to improve the method by allowing the generating of design layouts that comply with privacy policies and laws, Roberts ¶ 28.
Claim 22:
The rejection of claim 21 is incorporated. Biswas further teaches:
wherein generating the design representation comprising the design properties further comprises:
receiving an indication from a client device to generate the recommended revision to the digital design document, generate the modified digital design document, or to identify the additional digital design document corresponding to the digital design document. (as mentioned above, Biswas teaches generating a recommended revision to the digital design document, generating a modified digital design document, or identifying an additional digital design document corresponding to the digital design document. See claim 1’s mapped referenced in claim 21’s mapping. In Biswas, the system accomplishing the steps using a client device and/or server, ¶ 127 and fig 24, and the recommended designs are generated based on interactions from a creative professional using [receiving an indication from a client device] the design system, ¶ 4.)
Claim 25:
The rejection of claim 21 is incorporated. Biswas-Jovanovic further teaches:
wherein generating the semantic weights comprises:
generating the nodes, by encoding digital design element types for the plurality of digital design elements; (the elements/items include a plurality of different types/forms of content items, e.g., “such as digital images 124, text 126, digital media 128 [e.g., vector graphics, illustrations, digital videos, digital audio], and so forth”, Biswas ¶ 46 and fig. 1, transforming the items into a digital design graph [i.e., generating a decision tree], Biswas figs. 4:408 and 19 and ¶ 102, is herein, broadly interpreted as “encoding digital design element types for the plurality of digital design elements”)
and generating the edges between the nodes, by encoding relationships between nodes based on the design properties. (the generated decision tree includes a chain of [generating the edges between the nodes, by encoding relationships between nodes] arrangement rules that are based on the design properties, such as height of the items, Biswas ¶¶ 104 and 12)
Claim(s) 23 and 24 is/are rejected under 35 U.S.C. 103 as being unpatentable over Biswas (US 20190332861 A1) in view of Jovanovic (US 20210200942 A1) and Roberts (US 20190102681 A1), as applied to claim 21 above, and further in view of Finsterwald; Joseph Charles et al. (hereinafter Finsterwald – US 20160041957 A1).
Claim 23:
The rejection of claim 21 is incorporated. Biswas further teaches:
wherein generating the design representation comprising the design properties further comprises:
generating a first design representation comprising the design properties of the digital design document having the plurality of digital design elements; (determine layout [design representation] of items of a digital content with respect to each other, fig. 4:404 and ¶ 58 and the layout defining design properties for the items [plurality of digital design elements] of the digital content [digital design document], such as positioning and dimensions [e.g., width height], ¶¶ 59 and 92, and arrangement of layout)
Biswas-Jovanovic-Roberts does not appear to expressly teach, but Finsterwald teaches:
and transmitting the first design representation to a first service application of a chainable design representation pipeline, (an original design is submitted to various suggestion providers to generate a chain of design outputs, Abstract and ¶ 67 and fig. 1, wherein the later suggestion providers build upon the suggestion(s) of one or more previous providers, ¶ 114)
wherein the first service application performs at least one of determining tags for the digital design document based on the first design representation, digital document resizing of the digital design document based on the first design representation, generating recommendations based on the first design representation, generating design variations for the digital design document based on the first design representation, or similarity searches for the digital design document based on the first design representation (the suggestions are generated by the providers [generating recommendations…generating design variations], e.g., by detecting keyword tags [determining tags] and conducting an analysis based on the tags, ¶ 131).
Accordingly, it would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to further modify the method of Biswas to include and transmitting the first design representation to a first service application of a chainable design representation pipeline, wherein the first service application performs at least one of determining tags for the digital design document based on the first design representation, digital document resizing of the digital design document based on the first design representation, generating recommendations based on the first design representation, generating design variations for the digital design document based on the first design representation, or similarity searches for the digital design document based on the first design representation, as taught by Finsterwald.
One would have been motivated to make such a combination in order to improve the quality of the method by ensuring that the design maximizes each of the universally-accepted good design principles without too much back and forth, Finsterwald ¶¶ 113 and 114.
Claim 24:
The rejection of claim 23 is incorporated. Finsterwald further teaches:
wherein generating the design representation further comprises:
generating, utilizing a second service application of the chainable design representation pipeline, a second design representation from the first design representation; (the later suggestion providers build upon the suggestion(s) of one or more previous providers, ¶ 114)
and generating, utilizing the first service application of the chainable design representation pipeline, the design representation that accumulates structural information from the first design representation and the second design representation. (the suggestions generated by each of the suggestion providers are be taken into account, and combined [accumulates] where it best makes sense, ¶ 113)
Claim(s) 26 is/are rejected under 35 U.S.C. 103 as being unpatentable over Biswas (US 20190332861 A1) in view of Jovanovic ( US 20210200942 A1) and Roberts (US 20190102681 A1), as applied to claim 21 above, and further in view of Sharma; Natasha (hereinafter Sharma, Non-Patent Literature, “K-means Clustering Explained, 11/4/2021).
Claim 26:
The rejection of claim 21 is incorporated. Biswas does not appear to expressly teach, but Sharma teaches:
wherein constructing the digital design graph further comprises determining the distances between the plurality of digital design elements by determining distances between centroids of the plurality of digital design elements. (Datasets can contain millions of records and not all algorithms scale efficiently. K-Means is one of the most popular algorithms and it is also scale-efficient, Page 1, in which the objective is to minimize the sum of distances between the data points and the cluster centroid, to identify the correct group each data point should belong to, Page 2. It is can be seen that K-means clustering must determine the distances between the plurality of digital design elements by determining distances between centroids of the plurality of digital design elements. The distance calculation is the central mechanism that drives both the cluster assignment and centroid update steps of the algorithm.)
Accordingly, it would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to further modify the method of Biswas to include wherein constructing the digital design graph further comprises determining the distances between the plurality of digital design elements by determining distances between centroids of the plurality of digital design elements, as taught by Sharma .
One would have been motivated to make such a combination in order improve the efficiency in the method by being able to find unusual data points/outliers in the data or unknown properties to find a suitable grouping in the dataset, Sharma Page 1.
Claim(s) 27 is/are rejected under 35 U.S.C. 103 as being unpatentable over Biswas (US 20190332861 A1) and Jovanovic ( US 20210200942 A1) and Roberts (US 20190102681 A1), as applied to claim 21 above, and further in view of Li; Weiyu (hereinafter Weiyu_L – US 20240256951 A1, filed on 1/27/2023, before instant application).
Claim 27:
The rejection of claim 21 is incorporated. Biswas further teaches layout analysis that includes combining items into item groups, e.g., see ¶ 95, which are used to generate layout tree, and defining relationships between the representations of items [rectangles] can be done using machine learning, ¶ 99.
However, Biswas does not appear to expressly teach, but Weiyu_L teaches:
wherein generating the structural representation further comprises determining element groupings by: generating, utilizing a grouping machine-learning model, a grouping score from a first digital design element and a second digital design element within the digital design document based on the digital design graph; (a neural network model [machine-learning model] that is trained to build models that obtain object embeddings [e.g., vectors of numbers], ¶ 141, wherein the embeddings are obtained by obtaining a numerical representation of the object [representation of structural representation], and the embeddings/vectors include properties that are usable for computing things like groupings and rankings, ¶ 179 and fig. 10. Comparing embeddings of different objects [e.g., alerts] can result in outputting similarity scores [grouping score] for the objects that are used to group the different objects together, ¶ 198. Such models are trained based on one or more graphs representing the objects [based on the digital design graph], e.g., see ¶ 197 and fig. 10)
and determining that the first digital design element and the second digital design element are part of a first group based on the grouping score (Comparing embeddings of different objects [e.g., alerts] can result in outputting similarity scores [grouping score] for the objects that are used to group the different objects together, ¶ 198.).
Accordingly, it would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to further modify the method of Biswas to include wherein generating the structural representation further comprises determining element groupings by: generating, utilizing a grouping machine-learning model, a grouping score from a first digital design element and a second digital design element within the digital design document based on the digital design graph; and determining that the first digital design element and the second digital design element are part of a first group based on the grouping score, as taught by Weiyu_L.
One would have been motivated to make such a combination in order to improve the performance of the method by increasing recall of related objects for grouping, Weiyu_L ¶ 143 and Biswas ¶¶ 95 and 99.
Response to Arguments
Applicant's 103 arguments have been fully considered but are moot in view of the new grounds of rejection above.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Below is a list of these references, including why they are pertinent:
Li, Jianan (“Attribute-conditioned Layout GAN for Automatic Graphic Design”, Non-patent document) is pertinent to claim 1 for disclosing, the addressing of problem of specifying the locations and sizes of design elements with respect to element attributes, such as area, aspect ratio and reading-order, by introducing an attribute-conditioned Layout GAN to incorporate the attributes of design elements for graphic layout generation by forcing both the generator and the discriminator to meet attribute conditions, various loss designs following different design principles for layout optimization, and demonstrating the synthesizing of graphic layouts conditioned on different element attributes, adjusting well-designed layouts to new sizes while retaining elements' original reading-orders, see Abstract. A copy of this document was provided by the applicant with IDS 8/31/2023.
Chernov; Mykola et al. US 10839147 B1, is pertinent to claim 1 for disclosing a for classifying input fields and groups of input fields of a webpage, forming a hierarchy with grouped fields and a report file, and identifying elements of the HTML-documents on the webpage, Abstract, wherein the object recognition is performed using machine learning models, col 8:8-16, and involves generating a graph of the objects, cols 10:8-25.
Graves; Catherine et al. US 20220138204 A1, is pertinent to claim 1 for disclosing representing logical rules using a multigraph or decision tree, ¶ 74 and fig. 5.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to GABRIEL S MERCADO whose telephone number is (408)918-7537. The examiner can normally be reached Mon-Fri 8am-5pm (Eastern Time).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kieu Vu can be reached at (571) 272-4057. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Gabriel Mercado/Primary Examiner, Art Unit 2171
Prosecution Insights