DETAILED ACTION
This office action is responsive to communication(s) filed on 3/30/2026.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claims Status
Claims 1-9 and 17-27 are pending and are currently being examined.
Claims 1, 17 and 21 are independent.
Claims 10-16 are previously canceled.
Claims 1, 6, 7, 8, 9, 17, 19, 20, 21, 26, 27 are newly amended.
Claim Rejections - 35 USC § 112(b)
The following is a quotation of 35 U.S.C. 112(b):
(b) CONCLUSION.—The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the inventor or a joint inventor regards as the invention.
The following is a quotation of 35 U.S.C. 112 (pre-AIA ), second paragraph:
The specification shall conclude with one or more claims particularly pointing out and distinctly claiming the subject matter which the applicant regards as his invention.
Claim 9 and 20 are rejected under 35 U.S.C. 112(b) or 35 U.S.C. 112 (pre-AIA ), second paragraph, as being indefinite for failing to particularly point out and distinctly claim the subject matter which the inventor or a joint inventor (or for applications subject to pre-AIA 35 U.S.C. 112, the applicant), regards as the invention.
Claim 9 is ambiguous because it refers to a broader “embedding vector”, but depends on Claim 1, which specifies the narrower requirement of “machine learning embedding vectors”.
Claim 20, depending on claim 17, also has similar issue.
Claim Rejections - 35 USC § 112(d)
The following is a quotation of 35 U.S.C. 112(d):
(d) REFERENCE IN DEPENDENT FORMS.—Subject to subsection (e), a claim in dependent form shall contain a reference to a claim previously set forth and then specify a further limitation of the subject matter claimed. A claim in dependent form shall be construed to incorporate by reference all the limitations of the claim to which it refers.
The following is a quotation of pre-AIA 35 U.S.C. 112, fourth paragraph:
Subject to the following paragraph [i.e., the fifth paragraph of pre-AIA 35 U.S.C. 112], a claim in dependent form shall contain a reference to a claim previously set forth and then specify a further limitation of the subject matter claimed. A claim in dependent form shall be construed to incorporate by reference all the limitations of the claim to which it refers.
Claim 9 is rejected under 35 U.S.C. 112(d) or pre-AIA 35 U.S.C. 112, 4th paragraph, as being of improper dependent form for failing to further limit the subject matter of the claim upon which it depends, or for failing to include all the limitations of the claim upon which it depends.
Claim 9 recites “wherein generating the multigraph structural representation comprises generating an embedding vector or an adjacency matrix as the multigraph structural representation that represents the digital design multigraph”. However, in slightly different wording, claim 1 on which claim 9 depends, already provides the same limitation. Namely, claim 1 recited “generating a multigraph structural representation for the digital design document based on the digital design multigraph, wherein the multigraph structural representation comprises at least one of machine learning embedding vectors or an adjacency matrix”. Therefore, claim 9 fails to further limit the subject matter of the claim upon which it depends.
Applicant may cancel the claim(s), amend the claim(s) to place the claim(s) in proper dependent form, rewrite the claim(s) in independent form, or present a sufficient showing that the dependent claim(s) complies with the statutory requirements.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102 of this title, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 1, 2, 9, 17, 19 and 20 is/are rejected under 35 U.S.C. 103 as being unpatentable over Biswas; Mainak et al. (hereinafter Biswas – US 20190332861 A1) in view of Jovanovic; Katarina et al. (hereinafter Jovanovic – US 20210200942 A1) and Chaari; Nada et al. (hereinafter Chaari – Non-Patent Literature, "Multigraph classification using learnable integration network with application to gender fingerprinting"), Pgs 250-263, 2022).
Independent Claim 1:
Biswas teaches:
A method comprising: (e.g., see fig. 4)
generating a design representation (determine layout [design representation] of items of a digital content with respect to each other, fig. 4:404 and ¶ 58),
comprising design properties of a digital design document having a plurality of digital design elements; (the layout defining design properties for the items [plurality of digital design elements] of the digital content [digital design document], such as positioning and dimensions [e.g., width height], ¶¶ 59 and 92, and arrangement of layout)constructing a digital design […]graph from the design representation (using the layout [from the design representation], generating a layout tree, fig. 4:406 and ¶ 63, which is used to generate a decision tree [digital design graph], figs. 4:408 and 19 and ¶ 102)
wherein the digital design […]graph comprising multiple different edges (the generated decision tree includes lines representing connections/paths [edges] between the decision nodes, which represent the relationships between the nodes/objects, as illustrated in fig. 19, see ¶¶ 63, 99 and 113-114)
between image node and a text node (the edges/relationships are between the decision tree generated has nodes/items, ¶¶ 109-110 and fig. 19, e.g., text and images, ¶ 34.)
and the digital design […]graph is constructed by:
[…] generating nodes based on the plurality of digital design elements, wherein the nodes comprise the image node that represents the image design element and the text node that represents the text design element of the plurality of digital design elements; (the decision tree generated has nodes representing one or more digital items [generating nodes based on the plurality of digital design elements], ¶¶ 109-110 and fig. 19.)
and generating edges between the nodes (the generated decision tree includes lines representing connections/paths [generating edges between the nodes] between the decision nodes, as illustrated in fig. 19. Also see ¶¶ 113-114)
based on [design properties]; (the generated decision tree includes certain arrangement rules that are based on design properties, such as height of the items, ¶¶ 104 and 122)
generating a […]graph structural representation for the digital design document based on the digital design […]graph ,[…]; (generating a digital content design, including arrangement of items, fig. 4:412, the content design is based on the layout definition [structural representation] that is derived from the decision tree [based on the digital design graph], fig. 4:408-412, ¶¶ 102, 115, and 123 and fig. 3.)
and based on the […]graph structural representation and utilizing a machine learning model, performing at least one of: generating a recommended revision to the digital design document, generating a modified digital design document, or identifying an additional digital design document corresponding to the digital design document. (generating a digital content design [generating a recommended revision to the digital design document], including arrangement of items, fig. 4:412, the content design is based on a layout definition [structural representation] that is derived from the decision tree [based on the digital design graph], fig. 4:408-412, ¶¶ 102, 115, and 123 and fig. 3. The user, or creative professional uses e.g., Adobe® InDesign® as interacting with Adobe® Experience Manager®, Abstract and ¶ 56. As such, the digital content design is herein broadly interpreted as being a recommended revision to the digital design document, because it can serve as a recommended starting point that users/creative professionals may choose to leave as-is or revise. Biswas further teaches layout analysis that includes combining items into item groups, e.g., see ¶ 95, which are used to generate layout tree, and defining relationships between the representations of items [rectangles] can be done using machine learning, ¶ 99, because this is interpreted as utilizing a machine learning model to generate the document(s))
Biswas does not appear to expressly teach, but Jovanovic teaches:
generating semantic weights reflecting semantic relationships between the design properties of the plurality of digital design elements by generating a first semantic weight reflecting a measure of common semantic meaning in a semantic feature space between an image design element and a text design element in the digital design document (in determining alternative layouts, weighted scores for an alternative layouts can be determined based on a degree to which the content elements [semantic feature space] are symmetrical (e.g., left and right side symmetry, top and bottom symmetry, etc.) and/or amount of space between elements, where alternative layouts with better symmetry and/or less space between elements are scored higher, ¶¶ 7, 57 and 61. These elements include image and textual elements, such as an image and its caption, ¶ 11 [semantic feature space between an image design element and a text design element in the digital design document].)
that the design properties are “distances between the plurality of digital design elements and the semantic weights by generating a first edge between the image node and the text node based on a distance between the image design element and the text design element within the digital design document; and generating [an] edge between the image node and the text node from the first semantic weight reflecting the measure of common semantic meaning in the semantic feature space between the image design element and the text design element” (in determining alternative layouts, weighted scores for an alternative layouts can be determined based on a degree to which the content elements [semantic feature space] are symmetrical (e.g., left and right side symmetry, top and bottom symmetry, etc.) and/or amount of space between elements [distances between the plurality of digital design elements and the semantic weights], where alternative layouts with better symmetry and/or less space between elements [common semantic meaning] are scored higher, ¶¶ 7, 57 and 61. This ensures that semantic relationships between elements [reflecting the measure of common semantic meaning in the semantic feature space] are preserved in the alternative layouts, ¶ 10. These elements include image and textual elements, such as an image and its caption, ¶ 11.)
Accordingly, it would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to modify the method of Biswas to include generating semantic weights reflecting semantic relationships between the design properties of the plurality of digital design elements by generating a first semantic weight reflecting a measure of common semantic meaning in a semantic feature space between an image design element and a text design element in the digital design document, that the design properties are “distances between the plurality of digital design elements and the semantic weights by generating a first edge between the image node and the text node based on a distance between the image design element and the text design element within the digital design document; and generating [an] edge between the image node and the text node from the first semantic weight reflecting the measure of common semantic meaning in the semantic feature space between the image design element and the text design element”, as taught by Jovanovic.
One would have been motivated to make such a combination in order to improve the usability of the method by improving the visual quality of the representation of the design document, by ensuring it is more visually appealing and aligned with original user’s intent, Jovanovic ¶¶ 17 and 62.
Biswas does not appear to expressly teach, but Chaari teaches:
that the graph is a “multi”-graph, wherein the digital design multigraph comprises multiple different edges and a “second” edge between the text and image nodes (multigraphs allow more than one relation between a pair of nodes, we can represent rich data more succinctly which in turn helps in capturing complementary information between multi-source data that cannot be discovered in the otherwise simple graphs, Pg 250)
wherein the multigraph structural representation comprises at least one of machine learning embedding vectors or an adjacency matrix (the disclosed machine learning framework learns “how to optimally construct the meta-path adjacency matrix across heterogeneous data views” and utilizes a classification block with “consecutive GNN-based node embedding layers”, Pg 262)
Accordingly, it would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to further modify the method of Biswas to include that the graph is a “multi”-graph, wherein the digital design multigraph comprises multiple different edges and a “second” edge between the text and image node, wherein the multigraph structural representation comprises at least one of machine learning embedding vectors or an adjacency matrix, as taught by Chaari.
One would have been motivated to make such a combination in order to enrich the method by allowing to capture more complementary information from the data, Chaari Pg 250.
Claims 2:
The rejection of claim 1 is incorporated. Biswas further teaches:
wherein generating the design representation comprising the design properties further comprises:
extracting element properties by extracting types of elements or source URLs for the plurality of digital design elements; (information of original content is received [extracting] and is updated to have a consistent look and feel [typo in reference states “look-and-field”] with the original design, ¶¶ 56 and 114. A look and feel similarity is considered when determining an appropriate layout based on “adjacency” [element properties] between items of the digital content, ¶¶ 33 and 57. The content takes on a variety of forms, ¶ 46, e.g., a single node may have two type of digital content, ¶ 118. As such, the exacting of content is herein interpreted as “extracting types of elements…for the plurality of digital design elements”)
and extracting geometric properties from the digital design document by extracting element sizes, or element locations of the plurality of digital design elements. (information of original content is received [extracting] and is updated to have a consistent look and feel [typo in reference states “look-and-field”] with the original design, ¶¶ 56 and 114. A look and feel similarity is considered when determining an appropriate layout based on “adjacency” [element properties] between items of the digital content, ¶¶ 33 and 57. This similarity is also achieved by considering original height and width [extracting geometric properties…extracting element sizes] ensure a width-height ratio [element properties] of the original items is maintained, ¶¶ 118 and 122. Because these original properties are considered and maintained when layout is updated, it is interpreted that these properties are received/extracted as part of receiving the original content. For purposes of compact prosecution only, the examiner interprets the limitation(s) as being directed to “element”, “geometric”, and “style properties” that are part of the design properties, and that “element properties” includes geometric and non-geometric features other than size and position/location, such as name of the height-width ratio of the elements, and that “geometric properties” include features such as length and width of the elements)
Claim 9:
The rejection of claim 1 is incorporated. Biswas-Jovanovic-Chaari further teaches:
wherein generating the multigraph structural representation comprises generating an embedding vector or an adjacency matrix as the multigraph structural representation that represents the digital design multigraph. (this limitation is covered by mapping of claim 1’s limitation of generating a multigraph structural representation for the digital design document based on the digital design multigraph, wherein the multigraph structural representation comprises at least one of machine learning embedding vectors or an adjacency matrix)
Independent Claim 17:
Claim(s) 17 is directed to a computer-readable medium for accomplishing the steps of the method in claim 1, and is rejected using similar rationale(s).
Claim 19:
The rejection of claim 17 is incorporated. Biswas-Jovanovic-Chaari further teaches
wherein constructing the digital design multigraph further comprises: generating continuity weights, symmetry weights, or closure weights, between the plurality of digital design elements and that the generating of the edges are “based on the continuity weights, the symmetry weights, or the closure weights between the plurality of digital design elements” (in determining alternative layouts, a weighted score for an alternative layout can be determined based on a degree to which the content elements are symmetrical (e.g., left and right side symmetry, top and bottom symmetry, etc.) [symmetry weights], where alternative layouts with better symmetry are scored higher, Jovanovic ¶¶ 19, 57 and 61. See Chaari for multigraph).
generating the edges between the nodes [based on the design properties] (the generated decision tree includes a chain of arrangement rules that are based on the design properties, such as height of the items, Biswas ¶¶ 104 and 12)
Claim 20:
The rejection of claim 17 is incorporated. Biswas, as modified, further teaches:
wherein the multigraph structural representation further comprises: utilizing a machine learning model to generate an adjacency matrix or an embedding vector as the multigraph structural representation from the digital design multigraph, and based on the multigraph structural representation, generating a recommended revision to the digital design document, generate a modified digital design document, or identify an additional digital design document corresponding to the digital design document. (generating a digital content design [generating a recommended revision to the digital design document], including arrangement of items, Biswas fig. 4:412, the content design is based on a layout definition [structural representation] that is derived from the decision tree [based on the digital design graph], Biswas fig. 4:408-412, ¶¶ 102, 115, and 123 and fig. 3. The user, or creative professional uses e.g., Adobe® InDesign® as interacting with Adobe® Experience Manager®, Biswas Abstract and ¶ 56. As such, the digital content design is herein broadly interpreted as being a recommended revision to the digital design document, because it can serve as a recommended starting point that users/creative professionals may choose to leave as-is or revise. Biswas further teaches layout analysis that includes combining items into item groups, e.g., see ¶ 95, which are used to generate layout tree, and defining relationships between the representations of items [rectangles] can be done using machine learning, ¶ 99, because this is interpreted as utilizing a machine learning model to generate the document(s)). See Chaari for multigraph. See mapping for claim 1. )
Claim(s) 3 is/are rejected under 35 U.S.C. 103 as being unpatentable over Biswas (US 20190332861 A1), Jovanovic ( US 20210200942 A1) and Chaari (Non-Patent Literature), as applied to claim 1 above, and further in view of Kol; Tal et al. (hereinafter Kol – US 20170032050 A1).
Claim 3:
The rejection of claim 1 and is incorporated. Biswas further teaches that information of original content is received [extracting, from the digital design document,] and is updated to have a consistent look and feel [typo in Biswas states “look-and-field”] with the original design, ¶¶ 56 and 114. A look and feel similarity is considered when determining an appropriate layout based on “adjacency” [style properties] between items of the digital content, ¶¶ 33 and 57. This similarity is also achieved by considering original height and width [style properties] ensure a width-height ratio of the original items is maintained, ¶¶ 118 and 122. Because these original properties are considered and maintained when layout is updated, it is interpreted that these properties are received/extracted as part of receiving the original content.
Biswas-Jovanovic-Chaari does not appear to expressly teach, but Kol teaches:
wherein generating the design representation comprising the design properties further comprises extracting, from the digital design document, style properties by extracting colors, opacities of the plurality of digital design elements, backgrounds, or blend modes or extracting, from the digital design document, inferred tags by extracting meta data from the digital design document. (the converting of a website to an application, by extracting a set of brand colors from a website and setting an associated color theme for the brand colors [extracting, from the digital design document, style properties by extracting colors, opacities of the plurality of digital design elements, backgrounds, or blend modes], and assigning multiple colors within the theme to a plurality of user interface elements of the application version of the website, in order to preserve the look and feel of the website, Abstract and ¶ 21)
Accordingly, it would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to further modify the method of Biswas to include wherein generating the design representation comprising the design properties further comprises extracting, from the digital design document, style properties by extracting colors, opacities of the plurality of digital design elements, backgrounds, or blend modes or extracting, from the digital design document, inferred tags by extracting meta data from the digital design document, as taught by Kol.
One would have been motivated to make such a combination in order to achieve the preservation of look and feel of documents in a known and effective manner, Kol ¶ 21.
Claim(s) 4 and 18 is/are rejected under 35 U.S.C. 103 as being unpatentable over Biswas (US 20190332861 A1), Jovanovic (US 20210200942 A1) and Chaari (Non-Patent Literature), as applied to claims 1 and 17 above, and further in view of Roberts; Tara U. et al. (hereinafter Roberts – US 20190102681 A1).
Claim 4:
The rejection of claim 1 is incorporated. Biswas-Jovanovic does not appear to expressly teach, but Roberts teaches:
wherein generating the design representation further comprises anonymizing the digital design document by encoding user-identifiable information from text strings within the digital design document (a notification [digital design document] layout template, generated by a machine-learning model, ¶ 56, includes dynamic text components, which include anonymized data or partially anonymized data, ¶ 58, wherein the data is stripped from personally identifiable information PII [user-identifiable information], ¶ 41, according to privacy policies and laws, ¶ 28).
Accordingly, it would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to further modify the method of Biswas to include wherein generating the design representation further comprises anonymizing the digital design document by encoding user-identifiable information from text strings within the digital design document, as taught by Roberts.
One would have been motivated to make such a combination in order to improve the method by allowing the generating of design layouts that comply with privacy policies and laws, Roberts ¶ 28.
Claim 18:
The rejection of claim 17 is incorporated. Biswas further teaches:
wherein generating the design representation comprising the design properties further comprises:
extracting from the digital design document at least one of style properties or inferred tags; (information of original content is received [extracting from the digital design document] and is updated to have a consistent look and feel [typo in Biswas states “look-and-field”] with the original design, ¶¶ 56 and 114. A look and feel similarity is considered when determining an appropriate layout based on “adjacency” [style properties] between items of the digital content, ¶¶ 33 and 57. This similarity is also achieved by considering original height and width [style properties] ensure a width-height ratio of the original items is maintained, ¶¶ 118 and 122. Because these original properties are considered and maintained when layout is updated, it is interpreted that these properties are received/extracted as part of receiving the original content. For purposes of compact prosecution only, the examiner interprets the limitation(s) as being directed to “element”, “geometric”, and “style properties” that are part of the design properties, and that “element properties” includes geometric and non-geometric features other than size and position/location, such as name of the height-width ratio of the elements, and that “geometric properties” include features such as length and width of the elements, and that style properties include geometric and element properties.)
However, Biswas-Jovanovic does not appear to expressly teach, but Roberts teaches:
and anonymizing the digital design document by encoding user-identifiable information from text strings within the digital design document (a notification [digital design document] layout template, generated by a machine-learning model, ¶ 56, includes dynamic text components, which include anonymized data or partially anonymized data [encoding], ¶ 58, wherein the data is stripped from personally identifiable information PII [user-identifiable information], ¶ 41, according to privacy policies and laws, ¶ 28).
Accordingly, it would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to further modify the medium of Biswas to include and anonymizing the digital design document by encoding user-identifiable information from text strings within the digital design document, as taught by Roberts.
One would have been motivated to make such a combination in order to improve the medium by allowing the generating of design layouts that comply with privacy policies and laws, Roberts ¶ 28.
Claim(s) 5-7 is/are rejected under 35 U.S.C. 103 as being unpatentable over Biswas (US 20190332861 A1), Jovanovic ( US 20210200942 A1) and Chaari (Non-Patent Literature), as applied to claims 1 and 17 above, and further in view of Li; Weiyu (hereinafter Weiyu_L – US 20240256951 A1, filed on 1/27/2023, before instant application).
Claim 5:
The rejection of claim 1 is incorporated. Biswas-Jovanovic-Chaari does not appear to expressly teach, but Weiyu_L teaches:
wherein generating the semantic weights comprises:
generating, utilizing a trained embedding model, embeddings of the plurality of digital design elements in a semantic feature space; (a neural network model that is trained [utilizing a trained embedding model] to build models that obtain object embeddings [generating…embeddings of the plurality of digital design elements], ¶ 141, wherein the embeddings are obtained by obtaining a numerical representation of the object, and the embeddings/vectors include properties that are usable for computing things like groupings and rankings, ¶ 179 and fig. 10. The objects are related, e.g., semantically [digital design elements in a semantic feature space], ¶ 141)
comparing the embeddings of the plurality of digital design elements to determine measures of semantic similarity; (comparing the embeddings of different objects [of the plurality of digital design elements] can result in outputting similarity scores [to determine measures of semantic similarity] for the objects that are used to group the different objects together, ¶ 198. Such models are trained based on one or more graphs representing the objects [based on the digital design graph], e.g., see ¶ 197 and fig. 1)
and generating the semantic weights based on the measures of semantic similarity. (outputting similarity scores [generating the semantic weights based on the measures of semantic similarity] for the objects that are used to group the different objects together, ¶ 198)
Accordingly, it would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to further modify the method of Biswas to include generating, utilizing a trained embedding model, embeddings of the plurality of digital design elements in a semantic feature space; comparing the embeddings of the plurality of digital design elements to determine measures of semantic similarity; and generating the semantic weights based on the measures of semantic similarity, as taught by Weiyu_L.
One would have been motivated to make such a combination in order to improve the performance of the method by increasing recall of related objects for grouping, Weiyu_L ¶ 143 and Biswas ¶¶ 95 and 99.
Claim 6:
The rejection of claim 5 is incorporated. Biswas-Jovanovic-Chaari-Weiyu_L further teaches:
wherein constructing the digital design multigraph further comprises: generating a third edge between the image node and the text node based on an alignment between the image design element and the text design element within the digital design document. (Chaari teaches that multigraphs allow more than one relation between a pair of nodes [e.g., a third edge], we can represent rich data more succinctly which in turn helps in capturing complementary information between multi-source data that cannot be discovered in the otherwise simple graphs, Pg 250. Biswas teaches that the edges/relationships are between the decision tree generated has nodes/items, ¶¶ 109-110 and fig. 19, e.g., text and images, ¶ 34. Jovanovic teaches that the edges include alignment between content elements and that the elements are text and images, e.g., see ¶¶ 8, 61 and 84).
Claim 7:
The rejection of claim 1 is incorporated. Biswas further teaches layout analysis that includes combining items into item groups, e.g., see ¶ 95, which are used to generate layout tree, and defining relationships between the representations of items [rectangles] can be done using machine learning, ¶ 99.
However, Biswas-Jovanovic-Chaari does not appear to expressly teach, but Weiyu_L teaches:
wherein generating the multigraph structural representation further comprises determining element groupings by: generating, utilizing a grouping machine-learning model, a grouping score from a first digital design element and a second digital design element within the digital design document based on the digital design multigraph; (a neural network model [machine-learning model] that is trained to build models that obtain object embeddings [e.g., vectors of numbers], ¶ 141, wherein the embeddings are obtained by obtaining a numerical representation of the object [representation of structural representation], and the embeddings/vectors include properties that are usable for computing things like groupings and rankings, ¶ 179 and fig. 10. Comparing embeddings of different objects [e.g., alerts] can result in outputting similarity scores [grouping score] for the objects that are used to group the different objects together, ¶ 198. Such models are trained based on one or more graphs representing the objects [based on the digital design graph], e.g., see ¶ 197 and fig. 10. For multigraph see Chaari.)
and determining that the first digital design element and the second digital design element are part of a first group based on the grouping score (Comparing embeddings of different objects [e.g., alerts] can result in outputting similarity scores [grouping score] for the objects that are used to group the different objects together, ¶ 198.).
Accordingly, it would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to further modify the method of Biswas to include wherein generating the structural representation further comprises determining element groupings by: generating, utilizing a grouping machine-learning model, a grouping score from a first digital design element and a second digital design element within the digital design document based on the digital design graph; and determining that the first digital design element and the second digital design element are part of a first group based on the grouping score, as taught by Weiyu_L.
One would have been motivated to make such a combination in order to improve the performance of the method by increasing recall of related objects for grouping, Weiyu_L ¶ 143 and Biswas ¶¶ 95 and 99.
Claim(s) 8 is/are rejected under 35 U.S.C. 103 as being unpatentable over Biswas (US 20190332861 A1) in view of Jovanovic (US 20210200942 A1) and Chaari (Non-Patent Literature), as applied to claim 1 above, and further in view of Li; Jia et al. (hereinafter Jia_L – US 20200372660 A1).
Claim 8:
The rejection of claim 1 is incorporated. Biswas further teaches that the visual structure is represented and generated based on one or more graphs that include leaf nodes, e.g., end nodes at the deepest level of the trees, figs. 15-17 and 19, and defining relationships between the representations of items [rectangles] can be done using machine learning, ¶ 99.
However, Biswas-Jovanovic does not appear to expressly teach, but Jia_L teaches:
wherein generating the multigraph structural representation further comprises determining a visual structure inference by:
generating foreground leaf nodes and background leaf nodes from the plurality of digital design elements within the digital design document; (content, e.g., image, is segmented/grouped into foreground and background portions, Abstract and ¶ 11. See Chaari for multigraph)
determining, utilizing a grouping machine-learning model, a first element grouping from the foreground leaf nodes and a second element grouping from the background leaf nodes; (the neural network segments foreground and background portions [generating and determining…first…and a second element grouping], Abstract and ¶ 11)
and generating a visual structure inference comprising the first element grouping and the second element grouping (the neural network can generate foreground and background prediction results for the image [visual structure inference], Abstract and ¶ 11, which are fused together to product a full prediction [comprising the first element grouping and the second element grouping], ¶ 114 and fig. 4).
Accordingly, it would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to further modify the method of Biswas to include wherein generating the multigraph structural representation further comprises determining a visual structure inference by: generating foreground leaf nodes and background leaf nodes from the plurality of digital design elements within the digital design document determining, utilizing a grouping machine-learning model, a first element grouping from the foreground leaf nodes and a second element grouping from the background leaf nodes; and generating a visual structure inference comprising the first element grouping and the second element grouping, as taught by Jia_L.
One would have been motivated to make such a combination in order to improve the capabilities of the method to include detecting the most important object, ¶ 3, using a powerful neural network [machine-learning] model, Jia_L ¶ 4.
Claim(s) 21, 22, 25 and 27 is/are rejected under 35 U.S.C. 103 as being unpatentable over Biswas (US 20190332861 A1) in view of Jovanovic (US 20210200942 A1) and Chaari (Non-Patent Literature).
Claim 21:
Claim 21 is includes similar limitations to those of claim 1 and is rejected over the prior art, as applied above to claim 1.
However, unlike claim 1, claim 21 also has the following limitation that is also taught by the combination of Biswas-Jovanovic-Chaari:
and generating a third edge between the image node and the text node based on an alignment between the image design element and the text design element within the digital design document; (Chaari teaches that multigraphs allow more than one relation between a pair of nodes [e.g., a third edge], we can represent rich data more succinctly which in turn helps in capturing complementary information between multi-source data that cannot be discovered in the otherwise simple graphs, Pg 250. Biswas teaches that the edges/relationships are between the decision tree generated has nodes/items, ¶¶ 109-110 and fig. 19, e.g., text and images, ¶ 34. Jovanovic teaches that the edges include alignment between content elements and that the elements are text and images, e.g., see ¶¶ 8, 61 and 84)
Claim 22:
The rejection of claim 21 is incorporated. Biswas further teaches:
wherein generating the design representation comprising the design properties further comprises:
receiving an indication from a client device to generate the recommended revision to the digital design document, generate the modified digital design document, or to identify the additional digital design document corresponding to the digital design document. (as mentioned above, Biswas teaches generating a recommended revision to the digital design document, generating a modified digital design document, or identifying an additional digital design document corresponding to the digital design document. See claim 1’s mapped referenced in claim 21’s mapping. In Biswas, the system accomplishing the steps using a client device and/or server, ¶ 127 and fig 24, and the recommended designs are generated based on interactions from a creative professional using [receiving an indication from a client device] the design system, ¶ 4.)
Claim 25:
The rejection of claim 21 is incorporated. Biswas, as modified, further teaches:
wherein generating the semantic weights comprises:
generating the nodes, by encoding digital design element types for the plurality of digital design elements; (the elements/items include a plurality of different types/forms of content items, e.g., “such as digital images 124, text 126, digital media 128 [e.g., vector graphics, illustrations, digital videos, digital audio], and so forth”, Biswas ¶ 46 and fig. 1, transforming the items into a digital design graph [i.e., generating a decision tree], Biswas figs. 4:408 and 19 and ¶ 102, is herein, broadly interpreted as “encoding digital design element types for the plurality of digital design elements”)
and generating the edges between the nodes, by encoding relationships between nodes based on the design properties. (the generated decision tree includes a chain of [generating the edges between the nodes, by encoding relationships between nodes] arrangement rules that are based on the design properties, such as height of the items, Biswas ¶¶ 104 and 12)
Claim 27:
The rejection of claim 21 is incorporated. Biswas, as modified, also teaches:
wherein utilizing the machine learning model comprises: generating a plurality of entity embeddings for the nodes of the digital design multigraph, wherein the plurality of entity embeddings comprises an image node entity embedding corresponding to the image node and a text node entity embedding corresponding to the text node; (Chaari teaches that the disclosed machine learning framework learns “how to optimally construct the meta-path adjacency matrix across heterogeneous data views” and utilizes a classification block with “consecutive GNN-based node embedding layers”, Pg 262. Biswas teaches text and image nodes, see ¶ 34)
and utilizing the machine learning model to process the plurality of entity embeddings for the nodes and perform at least one of:
generating the recommended revision to the digital design document, generating the modified digital design document, or identifying the additional digital design document corresponding to the digital design document. (generating a digital content design [generating a recommended revision to the digital design document], including arrangement of items, fig. 4:412, the content design is based on a layout definition [structural representation] that is derived from the decision tree [based on the digital design graph], fig. 4:408-412, ¶¶ 102, 115, and 123 and fig. 3. The user, or creative professional uses e.g., Adobe® InDesign® as interacting with Adobe® Experience Manager®, Abstract and ¶ 56. As such, the digital content design is herein broadly interpreted as being a recommended revision to the digital design document, because it can serve as a recommended starting point that users/creative professionals may choose to leave as-is or revise. Biswas further teaches layout analysis that includes combining items into item groups, e.g., see ¶ 95, which are used to generate layout tree, and defining relationships between the representations of items [rectangles] can be done using machine learning, ¶ 99, because this is interpreted as utilizing a machine learning model to generate the document(s))
Claim(s) 23 and 24 is/are rejected under 35 U.S.C. 103 as being unpatentable over Biswas (US 20190332861 A1) in view of Jovanovic (US 20210200942 A1) and Chaari (Non-Patent Literature), as applied to claim 21 above, and further in view of Finsterwald; Joseph Charles et al. (hereinafter Finsterwald – US 20160041957 A1).
Claim 23:
The rejection of claim 21 is incorporated. Biswas further teaches:
wherein generating the design representation comprising the design properties further comprises:
generating a first design representation comprising the design properties of the digital design document having the plurality of digital design elements; (determine layout [design representation] of items of a digital content with respect to each other, fig. 4:404 and ¶ 58 and the layout defining design properties for the items [plurality of digital design elements] of the digital content [digital design document], such as positioning and dimensions [e.g., width height], ¶¶ 59 and 92, and arrangement of layout)
Biswas-Jovanovic-Roberts does not appear to expressly teach, but Finsterwald teaches:
and transmitting the first design representation to a first service application of a chainable design representation pipeline, (an original design is submitted to various suggestion providers to generate a chain of design outputs, Abstract and ¶ 67 and fig. 1, wherein the later suggestion providers build upon the suggestion(s) of one or more previous providers, ¶ 114)
wherein the first service application performs at least one of determining tags for the digital design document based on the first design representation, digital document resizing of the digital design document based on the first design representation, generating recommendations based on the first design representation, generating design variations for the digital design document based on the first design representation, or similarity searches for the digital design document based on the first design representation (the suggestions are generated by the providers [generating recommendations…generating design variations], e.g., by detecting keyword tags [determining tags] and conducting an analysis based on the tags, ¶ 131).
Accordingly, it would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to further modify the method of Biswas to include and transmitting the first design representation to a first service application of a chainable design representation pipeline, wherein the first service application performs at least one of determining tags for the digital design document based on the first design representation, digital document resizing of the digital design document based on the first design representation, generating recommendations based on the first design representation, generating design variations for the digital design document based on the first design representation, or similarity searches for the digital design document based on the first design representation, as taught by Finsterwald.
One would have been motivated to make such a combination in order to improve the quality of the method by ensuring that the design maximizes each of the universally-accepted good design principles without too much back and forth, Finsterwald ¶¶ 113 and 114.
Claim 24:
The rejection of claim 23 is incorporated. Finsterwald further teaches:
wherein generating the design representation further comprises:
generating, utilizing a second service application of the chainable design representation pipeline, a second design representation from the first design representation; (the later suggestion providers build upon the suggestion(s) of one or more previous providers, ¶ 114)
and generating, utilizing the first service application of the chainable design representation pipeline, the design representation that accumulates structural information from the first design representation and the second design representation. (the suggestions generated by each of the suggestion providers are be taken into account, and combined [accumulates] where it best makes sense, ¶ 113)
Claim(s) 26 is/are rejected under 35 U.S.C. 103 as being unpatentable over Biswas (US 20190332861 A1) in view of Jovanovic ( US 20210200942 A1) and Chaari (Non-Patent Literature), as applied to claim 21 above, and further in view of Sharma; Natasha (hereinafter Sharma, Non-Patent Literature, “K-means Clustering Explained, 11/4/2021).
Claim 26:
The rejection of claim 21 is incorporated. Biswas, as modified, does not appear to expressly teach, but Sharma teaches:
wherein constructing the digital design multigraph further comprises determining the distances between the plurality of digital design elements by determining distances between centroids of the plurality of digital design elements. (Datasets can contain millions of records and not all algorithms scale efficiently. K-Means is one of the most popular algorithms and it is also scale-efficient, Page 1, in which the objective is to minimize the sum of distances between the data points and the cluster centroid, to identify the correct group each data point should belong to, Page 2. It is can be seen that K-means clustering must determine the distances between the plurality of digital design elements by determining distances between centroids of the plurality of digital design elements. The distance calculation is the central mechanism that drives both the cluster assignment and centroid update steps of the algorithm. For multigraph see Chaari)
Accordingly, it would have been obvious to a person having ordinary skill in the art, before the effective filing date of the claimed invention, to further modify the method of Biswas to include wherein constructing the digital design graph further comprises determining the distances between the plurality of digital design elements by determining distances between centroids of the plurality of digital design elements, as taught by Sharma.
One would have been motivated to make such a combination in order improve the efficiency in the method by being able to find unusual data points/outliers in the data or unknown properties to find a suitable grouping in the dataset, Sharma Page 1.
Response to Arguments
Applicant's 103 arguments have been fully considered but are moot in view of the new grounds of rejection above.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure. Below is a list of these references, including why they are pertinent:
Li, Jianan (“Attribute-conditioned Layout GAN for Automatic Graphic Design”, Non-patent document) is pertinent to claim 1 for disclosing, the addressing of problem of specifying the locations and sizes of design elements with respect to element attributes, such as area, aspect ratio and reading-order, by introducing an attribute-conditioned Layout GAN to incorporate the attributes of design elements for graphic layout generation by forcing both the generator and the discriminator to meet attribute conditions, various loss designs following different design principles for layout optimization, and demonstrating the synthesizing of graphic layouts conditioned on different element attributes, adjusting well-designed layouts to new sizes while retaining elements' original reading-orders, see Abstract. A copy of this document was provided by the applicant with IDS 8/31/2023.
Chernov; Mykola et al. US 10839147 B1, is pertinent to claim 1 for disclosing a for classifying input fields and groups of input fields of a webpage, forming a hierarchy with grouped fields and a report file, and identifying elements of the HTML-documents on the webpage, Abstract, wherein the object recognition is performed using machine learning models, col 8:8-16, and involves generating a graph of the objects, cols 10:8-25.
Graves; Catherine et al. US 20220138204 A1, is pertinent to claim 1 for disclosing representing logical rules using a multigraph or decision tree, ¶ 74 and fig. 5.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to GABRIEL S MERCADO whose telephone number is (408)918-7537. The examiner can normally be reached Mon-Fri 8am-5pm (Eastern Time).
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kieu Vu can be reached at (571) 272-4057. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Gabriel Mercado/Primary Examiner, Art Unit 2171
Prosecution Insights