DETAILED ACTION
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Objections
Claims 1, 11, and 12 are objected to because of the following informalities:
Claim 1 in Lines 6-8 reciting the limitation, “Sequentially selecting artworks to be placed in respective outlines in the layout based on the sequence of tokens in the layout the artwork embeddings of the previously selected artworks,” has a minor grammatical error that makes the limitation unclear. Based on the prior limitations, the sequence of tokens in the layout is different from the artwork embeddings. There should be a conjunction or comma, or some way to distinguish between the two. For example, “Sequentially selecting artworks to be placed in respective outlines in the layout based on the sequence of tokens in the layout and the artwork embeddings of the previously selected artworks,”.
Claims 11 and 12 recites similar limitations as to Claim 1 and are therefore also objected in a similar manner.
Appropriate correction is required.
Claim Rejections - 35 USC § 103
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1, 3, 5, 7, 11-12, 15-16, 18 are rejected under 35 U.S.C. 103 as being unpatentable over Verma et al. (US 20240303881 A1) (Hereinafter referred to as Verma) in view of Pearce et al. (US 20180060933 A1) (Hereinafter referred to as Pearce) and in further view of Gong et al. (US 20250077596 A1) (Hereinafter referred to as Gong).
Regarding Claim 1, Verma discloses A method for generating walls of artworks, comprising: (See Abstract, “Embodiments are disclosed for machine learning-based generation of recommended layouts. The method includes . . .”)
generating a sequence of tokens that corresponds to a layout, wherein the layout includes outlines allocated for and corresponding to a number of requested artworks; (See [0002], “To generate the recommended layouts, embeddings are created to represent each design input of the set of design inputs. These design inputs may include text, images, shapes, etc. The design elements are tokenized, . . . one or more new sets of tokens that represent the same design elements but with new positions, new sizes, etc. Also see [0033], “The layout template 212 includes bounding boxes that represent each of the design inputs 202.
In this case, Verma teaches to tokenize (generate a sequence of tokens) design elements which include images (artworks), the tokens represent the new position (layout) of the design elements. Note that the bounding boxes would correspond to “outlines” and the number of inputted design elements would correspond to “a number of request artworks”.)
generating artwork embeddings of a plurality of artworks; (See [0025], “. . . the embedding generator 108 accesses an image embedding (e.g., CLIP) model to produce an image embedding 114 from an image of the existing layout.”)
recommending layouts based on the sequence of tokens in the layout the artwork embeddings; and (See [0002], “The embeddings are input into a layout generator which one or more new sets of tokens that represent the same design elements but with new positions, new sizes, etc. These new tokens can then be used to generate recommended layouts.” Also see [0027], "At numeral 4, the layout generator 116 receives the design element embeddings 112 and/or the image embedding 114. The layout generator 116 is trained to generate a set of recommended layouts from the received embeddings.”)
displaying the recommended layouts. (See [0028], “The set of recommended layouts 118 can be presented to the user via a user interface or stored in a cache or non-volatile memory device.” Also see [0046], “The layout recommendation system 604 outputs the recommended layouts 624 to the user interface of the user interface manager 602.”)
However, Verma fails to explicitly disclose
sequentially selecting artworks to be placed in respective outlines in the layout based on the sequence of tokens in the layout the artwork embeddings of the previously selected artworks, wherein the selected artworks meet a pre-determined threshold of artistic quality; and
displaying the requested wall of artworks, wherein the wall of artworks is a visual display of pieces of art arranged on a surface.
Pearce teaches sequentially selecting artworks to be placed in respective outlines in the layout based on the sequence of tokens in the layout the artwork embeddings; and (See [0006], “In one embodiment, the wall art system generates the uniquely customized wall art template for the one or more items of wall art by generating outlines of the actual dimensions of each of the one or more items in the layout as they appear in the electronic display of the user device.” Also see [0025], “in one embodiment, the user is prompted to select, purchase and/or customize the curated layout of items in the context of the wall art environment.” Lastly, see Fig. 5c showing a layout on the wall with art items.
In this case, Pearce teaches for a user to select artworks to be placed in the layout. This corresponds to the limitation of “sequentially selecting artworks to be placed in respective outlines in the layout” as a user would typically be considered as selecting one artwork at a time, and thus “sequentially selecting”. In combination with Verma [0002] and [0027] already teaching to generate a recommended layout based on the tokens and artwork embeddings, the above limitation of “sequentially selecting artworks to be placed in respective outlines in the layout based on the sequence of tokens in the layout the artwork embeddings” is taught, as the layout used by Pearce would be a recommended layout from Verma and that layout is based on the tokens and artwork embeddings.)
displaying the requested wall of artworks, wherein the wall of artworks is a visual display of pieces of art arranged on a surface. (See Fig. 5c showing a display of a wall of artworks arranged on a surface of a wall.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Verma with Pearce to include sequentially selecting artworks to be placed and displaying the requested wall of artworks.
The motivation to combine Verma with Pearce would have been obvious as both are within the same field of arranging design elements/artworks into a layout (See Pearce Abstract). The benefit of allowing a user to sequentially selecting artworks is that it can allow for better coherency and theming for artworks as the user would be going one by one. Lastly, note that the act of displaying the wall of artworks would be obvious as showing the final product allows for easier evaluations by the user on the quality of the artworks and layout.
However, Verma in view of Pearce still fails to explicitly disclose sequentially selecting artworks to be placed in respective outlines in the layout based on the sequence of tokens in the layout the artwork embeddings of the previously selected artworks, wherein the selected artworks meet a pre-determined threshold of artistic quality; and
Gong teaches sequentially selecting artworks to be placed in respective outlines in the layout based on the sequence of tokens in the layout the artwork embeddings of the previously selected artworks, wherein the selected artworks meet a pre-determined threshold of artistic quality; and (See [0005], “The method may include identifying candidate content items for recommendation to a user.” Also see [0008], “In some implementations, identifying the candidate content items may include obtaining a prior content item embedding for a prior content item associated with the user; and selecting content items that are associated with respective content item embeddings that are within a threshold distance of the prior content item embedding.”
In summary, Gong teaches the idea of recommending content items (artworks) to the user. The recommended content items are based on the previous content item embedding associated with the user (previously selected artworks), and have to be within some threshold distance of the prior content item embedding (pre-determined threshold of artistic quality). In combination with Pearce [0025] already teaching for the user to select the artworks, in this case, the artworks would be the recommended content items taught by Gong and thus, the above limitation are taught.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Verma in view of Pearce with Gong to include using the embedding of the previously selected content and have a threshold distance for the recommended content.
The motivation to combine Verma in view of Pearce with Gong would have been obvious as Verma and Gong are all within the same field of recommending content and extracting embeddings with machine learning models (See Gong Abstract and [0006]). The benefit of using the prior content embeddings and thresholding is that it makes for a better recommendation system, which in turn returns more relevant material to the user (See Gon [0003]).
Regarding Claim 3, Verma in view of Pearce and Gong disclose The method of claim 1, wherein artwork embeddings are generated through the use of Contrastive Language-Image Pre-training (CLIP) encodings. (See Verma [0025], “In some embodiments, if the design input selections 106 include an existing layout. The existing layout defines a position or perceptual features of the design inputs (e.g., a relative size between a text element and a shape element) the embedding generator 108 accesses an image embedding (e.g., CLIP) model to produce an image embedding 114 from an image of the existing layout.”)
Regarding Claim 5, Verma in view of Pearce and Gong disclose The method of claim 1, wherein the sequence of tokens include a token corresponding to frame border and a token corresponding to frame type. (See Verma [0033], “The layout generator 116 receives the token embedding 122 and the position embedding 120 and generates a layout template 212. The layout template 212 includes bounding boxes that represent each of the design inputs 202.”
Also see Pearce [0064], “For example, template 600 includes outlines 602 for four items of art from a user curated layout, 602a/602b/602c/602d. Each outline 602 reflects the product style of the item. For example, 602a outline is greater in width to reflect a wood frame mounted item while 602c outline is narrower in width to reflect a metal mounted item. As another example, 602b and 602d outlines are projected to reflect the thickness associated with a canvas mounted item 602b and a wood panel mounted item 602d.” Here, Pearce teaches the idea of different frame type such as wood or metal, which are associated with the thickness of the outlines. In combination with Verma, there would instead be tokens that correspond to a frame border and a token corresponding to frame type associated with the thickness of the outlines. The motivation to combine would have been similar to that of Claim 1 rejection motivation.)
Regarding Claim 7, Verma in view of Pearce and Gong disclose The method of claim 1, further comprising: training a first machine learning model to generate layouts; and (See Verma [0051], “The training manager 612 can teach, guide, tune, and/or train one or more neural networks. In particular, the training manager 612 can train a neural network based on a plurality of training data. For example, the machine learning models 616 can include some models trained to generate embedding spaces to represent various types of design inputs and other models, such as the layout generator, trained to generate recommended layouts using the embedding spaces.”)
training a second machine learning model to generate walls of artworks based on the generated layouts. (See Pearce [0027], “In one embodiment, the novel wall art system generates the wall art template from the user's unique selections and customization of wall art in the context of the wall art environment, including generating outlines of the actual dimensions of each of the one or more items in the curated layout as they appear in the electronic display of the user device.” Here, Pearce teaches the function of generating walls of artwork based on the selected artworks and generated layouts.
See Verma [0028], “The set of recommended layouts 118 can be presented to the user via a user interface or stored in a cache or non-volatile memory device.” Lastly, see Verma [0049], “Neural network manager 610 may host a multitude of neural networks or other machine learning models, such as machine learning models 616.” Here, Verma teaches the possibility of using multiple neural networks to achieve the functions of recommending a layout, which is presented to the UI. Being presentable means that the recommended layout with design elements (wall of artworks) has to have been generated in order for it to be shown in a UI. When combined with Pearce already teaching the function of generating a wall of artwork, then in one common scenario, a second machine learning model can be trained to perform that function and thus, the above limitations are taught. The motivation to combine would have been similar to that of Claim 1 rejection motivation.)
Regarding Claim 11, Verma in view of Pearce and Gong disclose A non-transitory computer readable medium having stored thereon instructions for causing a processing circuitry to execute a process, the process comprising: (See Verma [0074], “In general, a processor (e.g., a microprocessor) receives instructions, from a non-transitory computer-readable medium, (e.g., a memory, etc.), and executes those instructions, thereby performing one or more processes, including one or more of the processes described herein.”)
generating a sequence of tokens that corresponds to a layout, wherein the layout includes outlines allocated for and corresponding to a number of requested artworks; generating artwork embeddings of a plurality of artworks; sequentially selecting artworks to be placed in respective outlines in the layout based on the sequence of tokens in the layout the artwork embeddings of the previously selected artworks, wherein the selected artworks meet a pre-determined threshold of artistic quality; and displaying the requested wall of artworks, wherein the wall of artworks is a visual display of pieces of art arranged on a surface. (The above limitations are similar to those of Claim 1, and are therefore rejected under a similar rationale as that of Claim 1.)
Regarding Claim 12, Verma in view of Pearce and Gong disclose A system for generating walls of artworks using at least one machine learning model, comprising: a processing circuitry; and a memory, the memory containing instructions that, when executed by the processing circuitry, configure the system to: (See Verma [0015], “One or more embodiments of the present disclosure include a vector graphics system including a layout generation system that generates recommended layouts from a set of design inputs.” Also see Verma [0074], “Embodiments of the present disclosure may comprise or utilize a special purpose or general-purpose computer including computer hardware, such as, for example, one or more processors and system memory, as discussed in greater detail below.”)
generate a sequence of tokens that corresponds to a layout, wherein the layout includes outlines allocated for and corresponding to a number of requested artworks; generate artwork embeddings of a plurality of artworks; sequentially select artworks to be placed in respective outlines in the layout based on the sequence of tokens in the layout the artwork embeddings of the previously selected artworks, wherein the selected artworks meet a pre-determined threshold of artistic quality; and display the requested wall of artworks, wherein the wall of artworks is a visual display of pieces of art arranged on a surface. (The above limitations are similar to those of Claim 1, and are therefore rejected under a similar rationale as that of Claim 1.)
Regarding Claim 15, Claim 15 recites similar limitations as to Claim 3 and is therefore rejected under a similar rationale.
Regarding Claim 16, Claim 16 recites similar limitations as to Claim 5 and is therefore rejected under a similar rationale.
Regarding Claim 18, Claim 18 recites similar limitations as to Claim 7 and is therefore rejected under a similar rationale.
Claims 2 and 13 are rejected under 35 U.S.C. 103 as being unpatentable over Verma in view of Pearce and Gong and in further view of Shariantnia (“Easy Object Detection with Transformers: Simple Implementation of Pix2Seq Model in PyTorch”).
Regarding Claim 2, Verma in view of Pearce and Gong disclose The method of claim 1, wherein generating the sequence of tokens further comprises: token includes the coordinates of at least one of the corners of each outline and the aspect ratio of each outline; (See Verma [0032], “As described above, the embedding generator 108 accesses the machine learning models hosted by neural network manager 124 to generate the token embedding 122 and the position embedding 120 from the set of tokens 216. The token embedding 122 includes the type of design input (e.g., text, image, shape) and the position or size. The position embedding 120 includes an origin point (e.g., an x-y coordinate) and a size is represented by pixel dimensions of the height and width.”
Also see Verma [0033], “The layout generator 116 receives the token embedding 122 and the position embedding 120 and generates a layout template 212.”)
However, Verma in view of Pearce and Gong fails to explicitly disclose generating a start-of-sequence token corresponding to a number of outlines in a requested layout;
sequentially generating tokens in a sequence based on the previously generated tokens, wherein the start-of-sequence token is a previously generated token, and wherein each sequentially generated token includes the coordinates of at least one of the corners of each outline and the aspect ratio of each outline; and
generating an end-of-sequence token.
Shariantnia teaches generating a start-of-sequence token corresponding to a number of outlines in a requested layout; (See Page 4 showing beginning of sentence (BOS) token. Note that Shariantnia states the “it generates a sequence of tokens telling where each object is” and thus since it’s for “each object”, that means the token would correspond “to a number of outlines in a requested layout” where we consider the objects as artworks.)
sequentially generating tokens in a sequence based on the previously generated tokens, wherein the start-of-sequence token is a previously generated token, and wherein each sequentially generated token includes the coordinates of at least one of the corners of each outline and the aspect ratio of each outline; and (See Page 4 showing a generated sequence of tokens. These tokens define the bounding boxes for the each of objects. All the tokens for the red bounding box is defined in a continuous sequence and assumedly, the same can be said for the green and the unshown blue bounding boxes. This can thus be considered as “sequentially generating tokens in a sequence based on the previously generated tokens” as each token would be sequential to the previous one. In combination with Verma [0032] teaching to use x-y coordinates for an origin point and pixel dimensions of the height and width (coordinates of at least one of the corners of each outline and the aspect ratio of each outline), then the above limitations are taught.)
generating an end-of-sequence token. (See Page 4 showing end of sentence (EOS) token.)
It would have been obvious to one of ordinary skill in the art before the effective filing date of the claimed invention to modify Verma in view of Pearce and Gong with Shariantnia to include having a BOS and EOS tokens as well as generating the tokens sequentially.
The motivation to combine Verma in view of Pearce and Gong with Shariantnia would have been obvious as both Verma and Shariantnia both use tokens and embeddings to capture design elements/objects (See Shariantnia Page 4). Note that Shariantnia is simply showing that it’s really common to have a BOS and EOS token and sequentially generating a sequence of tokens, as shown by the Pix2Seq model example on Page 4.
Regarding Claim 13, Claim 13 recites similar limitations as to Claim 2 and is therefore rejected under a similar rationale.
Allowable Subject Matter
Claims 4, 6, 8-10, 14, 17, and 19-21 is objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
The following is a statement of reasons for the indication of allowable subject matter:
Regarding Claim 4, the cited prior art does not disclose or render obvious the combination of elements cited in the claims as a whole. Specifically, the cited prior art fails to disclose or render obvious the limitations: initializing a matrix of probabilities of overlaps, wherein the matrix of probabilities of overlaps is initialized to assign probabilities to coordinates of the outlines of the layout; and assigning the coordinates of the generated tokens with a reduced probability, wherein the probability is reduced below a pre-determined threshold value. Thus, Claim 4 contains allowable subject matter.
Regarding Claim 6, the cited prior art does not disclose or render obvious the combination of elements cited in the claims as a whole. Specifically, the cited prior art fails to disclose or render obvious the limitations: initializing a matrix of probabilities of duplicates, wherein the matrix of probabilities of duplicates is initialized to assign probabilities to artworks; and assigning selected artworks with a reduced probability, wherein the probability is reduced below a pre-determined threshold value. Thus, Claim 6 contains allowable subject matter.
Regarding Claim 8, the cited prior art does not disclose or render obvious the combination of elements cited in the claims as a whole. Specifically, the cited prior art fails to disclose or render obvious the limitations: collecting a set of layouts, wherein each layout in the collected set of layouts is represented by a respective sequence of tokens, and wherein the collected set of layouts includes a plurality of layouts with varying numbers of outlines and configurations of the outlines; receiving a plurality of requests to create layouts for a range of artworks, wherein the range of artworks includes artworks of different number of outlines and configurations of the outlines; generating each requested layouts; and applying a loss function to each token in a sequence representing the layouts until the loss, computed by the loss function, is below a pre-determined threshold, wherein weights of the at least one machine learning model are updated with respect to each token in each sequence. Thus, Claim 8 contains allowable subject matter.
Claim 9 is dependent upon Claim 8 and therefore also contains allowable subject matter.
Regarding Claim 10, the cited prior art does not disclose or render obvious the combination of elements cited in the claims as a whole. Specifically, the cited prior art fails to disclose or render obvious the limitations: receiving a layout with a specific number of outlines for artworks, wherein the layouts are represented as a sequence of tokens; sequentially selecting artworks based on the received layout and artwork embeddings of the previously generated artworks, wherein the artwork embeddings are generated using CLIP encodings; generating a wall of artworks; and applying a loss function to each token in a sequence representing the selected artworks until the average loss, computed by the loss function, is below a pre-determined threshold value, wherein weights of the at least one machine learning model are updated with respect to each token. Thus, Claim 10 contains allowable subject matter.
Regarding Claim 14, Claim 14 recites similar limitations as to Claim 4 and therefore contains similar allowable subject matter.
Regarding Claim 17, Claim 17 recites similar limitations as to Claim 6 and therefore contains similar allowable subject matter.
Regarding Claim 19, Claim 19 recites similar limitations as to Claim 8 and therefore contains similar allowable subject matter.
Regarding Claim 20, Claim 20 recites similar limitations as to Claim 9 and therefore contains similar allowable subject matter.
Regarding Claim 21, Claim 21 recites similar limitations as to Claim 10 and therefore contains similar allowable subject matter.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to THANG G HUYNH whose telephone number is (571)272-5432. The examiner can normally be reached Mon-Thu 7:30am-4:30pm EST | Fri 7:30am-11:30am EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Kee Tung can be reached at (571)272-7794. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/T.G.H./Examiner, Art Unit 2611
/KEE M TUNG/Supervisory Patent Examiner, Art Unit 2611
Prosecution Insights