Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-16 and 19-30 are rejected under 35 U.S.C. 101 because the claimed invention is directed to a mental process without significantly more.
Representative claim 1 recites:
“1. A computer system comprising:
one or more processors; and
a memory to store a set of instructions,
wherein the one or more processors execute instructions stored in the memory to perform operations comprising:
dividing, into a plurality of design fragments, one or more hierarchical structures representing one or more design interfaces configured to be displayed as components of a user interface;
storing, in an index, at least one mapping between the plurality of design fragments and a plurality of design embeddings generated by an image embedding model for the plurality of design fragments;
receiving, via the user interface, an image query specifying a search image;
generating, using the image embedding model, a search embedding corresponding to the image query;
processing the image query, including respectively determining a vector similarity between each of the plurality of design embeddings in the index and the search embedding; and
displaying, at the user interface based on the vector similarity, at least one output design fragment corresponding to at least one output design embedding from the plurality of design embeddings.”
Claim 28 contains similar subject matter.
The claims contain mental process steps of “dividing, into a plurality of design fragments, one or more hierarchical structures…,” “generating, using the image embedding model, a search embedding corresponding to the image query,” “the image query….”
The claims contain additional elements to the mental process in the form of “one or more processors,” “a memory,” “storing, in an index, at least one mapping…”, “receiving, via the user interface, an image query specifying a search image,” and “displaying, at the user interface based on the vector similarity, at least one output design fragment…”
This judicial exception is not integrated into a practical application because the claimed additional elements do not appear to improve the processing of a computer, require the use of a specific machine, effect a transformation or reduction of a particular article to a different state or thing, or provide a technological solution to a technological problem.
The “one or more processors” and “memory” are recited at a high level of generality. They appear to be generic computing hardware elements. The recitation of generic hardware is little more than using a computer to perform an abstract idea, see MPEP 2106.05(f)(2). “Storing, in an index, at least one mapping…” in a memory is similarly insignificant extra-solution activity and is well-known (see MPEP 2106.05(d)(II) and MPEP 2106.05(g)). “Receiving, via the user interface, an image query specifying a search image…” appears to be a data gathering step, and is thus mere pre-solution insignificant activity (see MPEP 2106.05(g). “Displaying, at the user interface based on the vector similarity, at least one output design fragment…” appears to be merely displaying a search result, or an output of a data analysis, is insignificant post-solution activity (see MPEP 2106.05(g)(3)).
The “image embedding model” appears to be merely applying a generic machine learning process to a new information analysis context. As noted in Recentive Analytics v. Fox. Corp., No. 23-2437 (Fed. Cir. 2025), merely applying a generic machine learning system to a different context does not provide a practical application.
It is noted that none of the additional elements appear to improve the processing of a computer, require the use of a specific machine, effect a transformation or reduction of a particular article to a different state or thing, or provide a technological solution to a technological problem. As such, none of the additional elements appear to integrate the judicial exception into a practical application.
None of the additional elements are sufficient to amount to significantly more than the judicial exception, in part or in whole.
The recitation of generic hardware of the “one or more processors” and “memory” is little more than using a computer to perform an abstract idea, see MPEP 2106.05(f)(2). “Storing, in an index, at least one mapping…” the source data and the integrated extracted source data in a memory is nothing more than storing data in a memory, which is recognized as well-understood, routine, and conventional (see MPEP 2106.05(d)(II)). The additional element of “Receiving, via the user interface, an image query specifying a search image…” is merely extra-solution activity data gathering and is well understood, routine, and conventional (see MPEP 2106.05(g)). “Displaying, at the user interface based on the vector similarity, at least one output design fragment…” is insignificant extra-solution activity and is well known (see MPEP 2106.05(g)((3).
As noted in Recentive Analytics v. Fox. Corp., No. 23-2437 (Fed. Cir. 2025), merely applying a generic machine learning system to a different context does not provide significantly more than an abstract idea.
None of the additional elements, in part or in whole, appear to improve the processing of a computer, require the use of a particular machine, effect a transformation or reduction of a particular article to a different state or thing, or add a specific limitation other than what is well understood, routine, or conventional. As such, none of the additional elements appears to be, in part or in whole, significantly more than the judicial exception.
Dependent claims 2-15 and 29-30 are merely directed towards additional limitations that further define data types or further describe data analyses that will occur. The “models” of claims 2, 5, 8 are recited at a high level of abstraction and appear to simply be data analyses. None of dependent claims 2-15 or 29-30 appear to include additional elements that incorporate the claimed subject matter into a practical application. The dependent claims also do not include additional elements that, in part or in whole, appear to be significantly more than the abstract idea.
Claim 16 recites:
“16. A non-transitory computer-readable medium that stores instructions, executable by one or more processors, to cause the one or more processors to perform operations comprising:
receiving a search query specifying a representation of one or more design elements configured to be displayed as components of a user interface;
generating one or more first image embeddings associated with the representation using one or more machine learning models comprising an image embedding model;
matching the one or more first image embeddings to a set of second image embeddings for a set of design fragments included in an index, wherein each design fragment in the index corresponds to a subset of one or more hierarchical structures representing one or more design interfaces;
generating a set of search results using the second image set of embeddings; and
outputting the set of search results in a response to the search query.”
The claims contain mental process steps of “generating one or more first image embeddings…,” “matching one or more embeddings…” and “generating a set of search results…”.
The claims contain additional elements to the mental process in the form of “a non-transitory computer readable medium,” “receiving a search query,” and “outputting the set of search results.”
This judicial exception is not integrated into a practical application because the claimed additional elements do not appear to improve the processing of a computer, require the use of a specific machine, effect a transformation or reduction of a particular article to a different state or thing, or provide a technological solution to a technological problem.
The “non-transitory computer readable medium” is recited at a high level of generality. It appears to be generic computing hardware elements. The recitation of generic hardware is little more than using a computer to perform an abstract idea, see MPEP 2106.05(f)(2). Receiving a search query appears to be a data gathering step, and is thus mere pre-solution insignificant activity (see MPEP 2106.05(g). Outputting the result of a data analysis is insignificant post-solution activity (see MPEP 2106.05(g)(3)).
The “one or more machine learning models comprising an image embedding model” appears to be merely applying a generic machine learning process to a new information analysis context. As noted in Recentive Analytics v. Fox. Corp., No. 23-2437 (Fed. Cir. 2025), merely applying a generic machine learning system to a different context does not provide a practical application.
It is noted that none of the additional elements appear to improve the processing of a computer, require the use of a specific machine, effect a transformation or reduction of a particular article to a different state or thing, or provide a technological solution to a technological problem. As such, none of the additional elements appear to integrate the judicial exception into a practical application.
None of the additional elements are sufficient to amount to significantly more than the judicial exception, in part or in whole.
The recitation of generic hardware of the “non-transitory computer readable medium” is little more than using a computer to perform an abstract idea, see MPEP 2106.05(f)(2). The additional element of “receiving a search query” is merely extra-solution activity data gathering and is well understood, routine, and conventional (see MPEP 2106.05(g)). Outputting results of a data analysis is insignificant extra-solution activity and is well known (see MPEP 2106.05(g)((3).
As noted in Recentive Analytics v. Fox. Corp., No. 23-2437 (Fed. Cir. 2025), merely applying a generic machine learning system, such as the claimed “one or more machine learning models comprising an image embedding model,” to a different context does not provide significantly more than an abstract idea.
None of the additional elements, in part or in whole, appear to improve the processing of a computer, require the use of a particular machine, effect a transformation or reduction of a particular article to a different state or thing, or add a specific limitation other than what is well understood, routine, or conventional. As such, none of the additional elements appears to be, in part or in whole, significantly more than the judicial exception.
Dependent claims 19-27 are merely directed towards additional limitations that further define data types or further describe data analyses that will occur. None of dependent claims 19-27 appear to include additional elements that incorporate the claimed subject matter into a practical application. The dependent claims also do not include additional elements that, in part or in whole, appear to be significantly more than the abstract idea.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claims 1-2, 6-7, 9, 14, and 28-29 are rejected under 35 U.S.C. 103 as being unpatentable over Shi et al. (US Pre-Grant Publication 2022/0413997) in view of Ben Shaul et al. (US Pre-Grant Publication 2024/0086460).
As to claim 1, Shi teaches a computer system comprising:
one or more processors (see Shi paragraphs [0022]-[0023]); and
a memory to store a set of instructions (see Shi paragraphs [0022]-[0023]),
wherein the one or more processors execute instructions stored in the memory to perform operations comprising:
dividing, into a plurality of design fragments, one or more hierarchical structures representing one or more design interfaces configured to be displayed as components of a user interface (see Shi paragraphs [0043]-[0045]. A group of GUI elements may be analyzed and divided. Hierarchies amongst the GUI elements are recognized and maintained in an index);
storing, in an index, at least one mapping between the plurality of design fragments and a plurality of design [records] generated by an image [analysis] model for the plurality of design fragments (see Shi paragraphs [0043]-[0045]. An index is created that stores a mapping between each design fragment and a location. It is noted that a machine learning image analysis model is used to perform this analysis);
Shi does not explicitly teach:
storing, in an index, at least one mapping between the plurality of design fragments and a plurality of design embeddings generated by an image embedding model for the plurality of design fragments;
receiving, via the user interface, an image query specifying a search image;
generating, using the image embedding model, a search embedding corresponding to the image query;
processing the image query, including respectively determining a vector similarity between each of the plurality of design embeddings in the index and the search embedding; and
displaying, at the user interface based on the vector similarity, at least one output design fragment corresponding to at least one output design embedding from the plurality of design embeddings.”
Ben Shaul teaches:
storing, in an index, at least one mapping between the plurality of design fragments and a plurality of design embeddings generated by an image embedding model for the plurality of design fragments (see Ben Shaul paragraphs [0031]-[0034]. Ben Shaul teaches that images may be segmented into tiles. These are image fragments (“design fragments”). These fragments are indexed. As noted in paragraph [0034], Ben Shaul generates an embedding for each tile. These tiles are indexed, see paragraph [0033]);
receiving, via the user interface, an image query specifying a search image (see Ben Shaul paragraph [0041]. Ben Shaul teaches wherein a user may submit a query containing an image);
generating, using the image embedding model, a search embedding corresponding to the image query (see Ben Shaul paragraph [0043]. The image query undergoes machine learning analysis to produce an image embedding);
processing the image query, including respectively determining a vector similarity between each of the plurality of design embeddings in the index and the search embedding (see Ben Shaul paragraph [0045]. The embedding from the query image is compared to embeddings from the dataset. This comparison includes determining a distance between the query image embedding and the embeddings in the dataset. As noted in paragraph [0034], proximity in the embedding space represents similarity. As noted in paragraph [0045], the embeddings are feature vectors. Also see paragraph [0130]); and
displaying, at the user interface based on the vector similarity, at least one output design fragment corresponding to at least one output design embedding from the plurality of design embeddings (see Ben Shaul paragraphs [0048]-[0049]. Ben Shaul teaches to identify tiles corresponding to the nearest embeddings and present results. It is noted that Ben Shaul does not explicitly teach design elements. However, Shi teaches to create an index of user interface design elements. Ben Shaul teaches to create an index of fragments of images. Combined, the references will allow Ben Shaul to index and search fragments of design elements of Shi).
It would have been obvious to one of ordinary skill in the art before the earliest filing date of the invention to have modified Shi by the teachings of Ben Shaul because Ben Shaul provides to Shi the benefit of a efficiently generating embeddings of images and fragments (see Ben Shaul paragraph [0030]). This will improve the indexing aspects of Shi.
As to claim 2, Shi as modified by Ben Shaul teaches the computer system of claim 1, wherein the operations further comprise:
generating the plurality of design embeddings using the image embedding model (see Ben Shaul paragraphs [0031]-[0034]).
As to claim 6, Shi as modified by Ben Shaul teaches the computer system of claim 1, wherein dividing the one or more hierarchical structures into the plurality of design fragments comprises generating a design fragment from a node within the one or more hierarchical structures based on one or more attributes associated with the node (see (see Shi paragraphs [0044]-[0045]).
As to claim 7, Shi as modified by Ben Shaul teaches the computer system of claim 6, wherein the one or more attributes comprise at least one of a node type or a dimension (see Shi paragraphs [0043]-[0045]).
As to claim 9, Shi as modified by Ben Shaul teaches the computer system of claim 1, wherein processing the image query comprises:
adding the at least one output design fragment corresponding to the at least one output design embedding to a set of search results for the image query (see Ben Shaul paragraphs [0048]-[0049]).
As to claim 14, Shi as modified by Ben Shaul teaches the computer system of claim 1, wherein the operations further comprise detecting a change to the one or more hierarchical structures prior to dividing the one or more hierarchical structures into the plurality of design fragments (see Shi paragraphs [0043]-[0045]).
As to claim 28, see the rejection of claim 1.
As to claim 29, Shi as modified teaches wherein the operations comprise:
generating a rendering of the one or more design interfaces and identifying the plurality of design fragments from rendered properties of the rendering (see Shi paragraphs [0043]-[0045]).
Claim 3-5 and 15 are rejected under 35 U.S.C. 103 as being unpatentable over Shi et al. (US Pre-Grant Publication 2022/0413997) in view of Ben Shaul et al. (US Pre-Grant Publication 2024/0086460), in view of Forsyth et al. (US Pre-Grant Publication 2020/0311798).
As to claim 3, Shi as modified teaches the computer system of claim 2.
Shi does not teach wherein generating the plurality of design embeddings comprises:
generating a text-based representation of a design fragment from the plurality of design fragments; and
converting the text-based representation into a design embedding for the design fragment.
Forsyth teaches wherein generating the plurality of design embeddings comprises:
generating a text-based representation of a design fragment from the plurality of design fragments (see Forsyth paragraphs [0057]-[0059] for the embedding process. Paragraph [0059] goes into greater detail about forming a text-based representation of an image and creating an embedding based on that image); and
converting the text-based representation into a design embedding for the design fragment (see Forsyth paragraphs [0057]-[0059]).
It would have been obvious to one of ordinary skill in the art before the earliest filing date of the invention to have modified Shi by the teachings of Forsyth because Forsyth provides to Shi the benefit of improved type-specific embeddings, which will improve performance (see Forsyth paragraph [0152]). Forsyth also provides the benefits of multi-modal embedding, which will improve search and indexing operations (see Forsyth paragraphs [0029]-[0031]).
As to claim 4, Shi as modified by Forsyth teaches the computer system of claim 3, wherein the text-based representation comprises at least one of a text-based description of the design fragment or a text-based definition of the design fragment (see Forsyth paragraphs [0057]-[0059]).
As to claim 5, Shi as modified teaches the computer system of claim 2.
Shi does not teach wherein the operations further comprise:
Generating the plurality of design embeddings using at least one of a text embedding model or a multimodal embedding model.
Forsyth teaches wherein the operations further comprise:
Generating the plurality of design embeddings using at least one of a text embedding model or a multimodal embedding model (see Forsyth paragraphs [0057]-[0059]).
It would have been obvious to one of ordinary skill in the art before the earliest filing date of the invention to have modified Shi by the teachings of Forsyth because Forsyth provides to Shi the benefit of improved type-specific embeddings, which will improve performance (see Forsyth paragraph [0152]). Forsyth also provides the benefits of multi-modal embedding, which will improve search and indexing operations (see Forsyth paragraphs [0029]-[0031]).
As to claim 15, Shi as modified teaches the computer system of claim 1.
Shi does not teach wherein the image query further specifies one or more design elements using at least one a text-based description, a sketch, or at least a portion of a design interface.
Forsyth teaches wherein the image query further specifies one or more design elements using at least one a text-based description, a sketch, or at least a portion of a design interface (see Forsyth paragraphs [0054]-[0055]).
It would have been obvious to one of ordinary skill in the art before the earliest filing date of the invention to have modified Shi by the teachings of Forsyth because Forsyth provides to Shi the benefit of improved type-specific embeddings, which will improve performance (see Forsyth paragraph [0152]). Forsyth also provides the benefits of multi-modal embedding, which will improve search and indexing operations (see Forsyth paragraphs [0029]-[0031]).
Claim 8 is rejected under 35 U.S.C. 103 as being unpatentable over Shi et al. (US Pre-Grant Publication 2022/0413997) in view of Ben Shaul et al. (US Pre-Grant Publication 2024/0086460) in view of Rogynskyy et al. (US Pre-Grant Publication 2019/0361934).
As to claim 8, Shi as modified teaches the computer system of claim 6.
Shi does not teach wherein the design fragment is generated based on a classification score outputted by a machine learning model from a representation of the node.
Rogynskyy teaches wherein the design fragment is generated based on a classification score outputted by a machine learning model from a representation of the node (see Rogynskyy paragraphs [0709]-[0710]. Rogynskyy shows that nodes me be categorized based on connections between the nodes. A relevancy score may be used to categorize nodes and the relationships between nodes in a hierarchical data structure. As noted in paragraph [0726]-[0727], such analysis may rely on machine learning).
It would have been obvious to one of ordinary skill in the art before the earliest filing date of the invention to have modified Shi by the teachings of Rogynskyy because both references are directed towards analysis of hierarchical data structures. Rogynskyy merely provides additional analysis to Shi that will help to better parse and understand the relationships between two nodes in a hierarchical data structure, which will improve the analysis of Shi.
Claim 10 is rejected under 35 U.S.C. 103 as being unpatentable over Shi et al. (US Pre-Grant Publication 2022/0413997) in view of Ben Shaul et al. (US Pre-Grant Publication 2024/0086460), and further in view of Abdel-Mottaleb et al. (US Patent 6,285,995).
As to claim 10, Shi as modified teaches the computer system of claim 9.
Shi as modified does not teach wherein the operations further comprise:
matching the search embedding to a set of clusters of embeddings in the index;
determining a set of representative design fragments associated with the set of clusters of embeddings; and
adding the set of representative design fragments to the set of search results.
Abdel-Mottaleb teaches wherein the operations further comprise:
matching the search embedding to a set of clusters of embeddings in the index (see Abdel-Mottaleb 5:12-34. Vectors representing images are stored as clusters. Also see Abdel-Mottaleb 5:35-65. Clusters may be searched);
determining a set of representative design fragments associated with the set of clusters of embeddings (see Abdel-Mottaleb 5:12-34. Clusters may be represented by particular images); and
adding the set of representative design fragments to the set of search results (see Abdel-Mottaleb 5:35-6:6. Clusters may be searched and matching images may be added to the set of search results).
It would have been obvious to one of ordinary skill in the art before the earliest filing date of the invention to have modified Shi by the teachings of Abdel-Mottaleb because Abdel-Mottaleb provides to Shi the benefit of being able to conduct searches for matching objects to a query in a reduced period of time (see Abdel-Mottaleb 1:66-2:2).
Claim 11 is rejected under 35 U.S.C. 103 as being unpatentable over Shi et al. (US Pre-Grant Publication 2022/0413997) in view of Ben Shaul et al. (US Pre-Grant Publication 2024/0086460), in view of Abdel-Mottaleb et al. (US Patent 6,285,995), and further in view of Capper et al. (US Pre-Grant Publication 2005/0144158).
As to claim 11, Shi as modified by Ben Shaul teaches the computer system of claim 10.
Shi as modified does not teach wherein processing the image query further comprises:
matching a user selection of a design fragment in the set of representative design fragments to a cluster in the set of clusters of embeddings; and
updating the set of search results with one or more additional design fragments in the cluster.
Capper teaches wherein processing the image query further comprises:
matching a user selection of a design fragment in the set of representative design fragments to a cluster in the set of clusters of embeddings (see paragraphs [0072]-[0075]. A user may select results in a cluster); and
updating the set of search results with one or more additional design fragments in the cluster (see paragraphs [0072]-[0075]. The user selection of a result from the cluster updates the search results rankings in the cluster with additional results).
It would have been obvious to one of ordinary skill in the art before the earliest filing date of the invention to have modified Shi by the teachings of Capper because both references are directed towards data analysis to identify matching results. Capper merely provides additional analysis to Shi that will help to better analyze results for a user and aid a user in additional discovery of results.
Claim 12 is rejected under 35 U.S.C. 103 as being unpatentable over Shi et al. (US Pre-Grant Publication 2022/0413997) in view of Ben Shaul et al. (US Pre-Grant Publication 2024/0086460), in view of Abdel-Mottaleb et al. (US Patent 6,285,995), and further in view of Bouhnik et al. (US Pre-Grant Publication 2019/0361934).
As to claim 12, Shi as modified teaches the computer system of claim 10.
Shi does not teach wherein matching the search embedding to the set of clusters of embeddings comprises determining that a similarity between the search embedding and a set of representative embeddings for the set of clusters of embeddings meets or exceeds a threshold.
Bouhnik teaches wherein matching the search embedding to the set of clusters of embeddings comprises determining that a similarity between the search embedding and a set of representative embeddings for the set of clusters of embeddings meets or exceeds a threshold (see Bouhnik paragraph [0045]).
It would have been obvious to one of ordinary skill in the art before the earliest filing date of the invention to have modified Shi by the teachings of Bouhnik because both references are directed towards data analysis to identify matching results. Bouhnik merely provides additional analysis to Shi that will help to better analyze and group results for a user, which will improve the usability of the search system of Shi.
Claim 13 is rejected under 35 U.S.C. 103 as being unpatentable over Shi et al. (US Pre-Grant Publication 2022/0413997) in view of Ben Shaul et al. (US Pre-Grant Publication 2024/0086460) in view of Amacker (US Patent 8,631,029).
As to claim 13, Shi teaches the computer system of claim 9.
Shi does not teach wherein processing the image query further comprises:
determining a second embedding for a design fragment that is selected by a user from the set of search results; and
generating an additional set of search results based on the second embedding.
Amacker teaches wherein processing the search query further comprises:
determining a second embedding for a design fragment that is selected by a user from the set of search results (see Amacker 5:22-6:15 and Abstract. A user may select a result. Subsequent results shown to a user will take into account the user selection. See Ben Shaul paragraph [0045] for searching using image embedding); and
generating an additional set of search results based on the second embedding (see Amacker 5:22-6:15 and Abstract. See Ben Shaul paragraph [0045] for searching using image embedding).
It would have been obvious to one of ordinary skill in the art before the earliest filing date of the invention to have modified Shi by the teachings of Amacker because both references are directed towards data analysis to identify matching results. Amacker merely provides additional analysis to Shi that will help to better analyze results for a user and aid a user in additional discovery of results.
Claims 16, 19, and 24 are rejected under 35 U.S.C. 103 as being unpatentable over Ben Shaul et al. (US Pre-Grant Publication 2024/0086460) in view of Shi et al. (US Pre-Grant Publication 2022/0413997).
As to claim 16, Ben Shaul teaches a non-transitory computer-readable medium that stores instructions, executable by one or more processors (see Ben Shaul paragraph [0119]), to cause the one or more processors to perform operations comprising:
receiving a search query specifying a representation of one or more design elements … (see Ben Shaul paragraph [0041] and the rejection of claim 1);
generating one or more first image embeddings associated with the representation using one or more machine learning models comprising an image embedding model (see Ben Shaul paragraph [0043] and the rejection of claim 1);
matching the one or more first image embeddings to a set of second image embeddings for a set of design fragments included in an index (see Ben Shaul paragraph [0045] and the rejection of claim 1);
generating a set of search results using the second image set of embeddings (see Ben Shaul paragraphs [0048]-[0049] and the rejection of claim 1); and
outputting the set of search results in a response to the search query (see Ben Shaul paragraphs [0048]-[0049] and the rejection of claim 1).
Ben Shaul does not explicitly teach:
… a representation of one or more design elements configured to be displayed as components of a user interface;
a set of design fragments included in an index, wherein each design fragment in the index corresponds to a subset of one or more hierarchical structures representing one or more design interfaces
Shi teaches:
… a representation of one or more design elements configured to be displayed as components of a user interface (see Shi paragraphs [0043]-[0045]);
a set of design fragments included in an index, wherein each design fragment in the index corresponds to a subset of one or more hierarchical structures representing one or more design interfaces (see Shi paragraphs [0043]-[0045]).
It would have been obvious to one of ordinary skill in the art before the earliest filing date of the invention to have modified Ben Shaul by the teachings of Shi because Shi provides to Ben Shaul the benefit of additional analyses that may be used to identify elements using machine learning analysis of screenshots and images. This will expand the capabilities of Ben Shaul in indexing images and responding to user requests.
As to claim 19, Ben Shaul as modified teaches the non-transitory computer-readable medium of claim 16, wherein the one or more first image embeddings are matched to the set of second image embeddings based on one or more distances computed between the one or more first image embeddings and the set of second image embeddings (see Ben Shaul paragraph [0045]).
As to claim 24, Ben Shaul as modified by Shi teaches the non-transitory computer-readable medium of claim 16, wherein generating the set of search results comprises:
retrieving, from the index, the set of design fragments from a set of mappings that include the set of second image embeddings (see Ben Shaul paragraphs [0048]-[0049]); and
adding the set of design fragments to the set of search results (see Ben Shaul paragraphs [0048]-[0049]).
Claims 20-22 are rejected under 35 U.S.C. 103 as being unpatentable over Ben Shaul et al. (US Pre-Grant Publication 2024/0086460) in view of Shi et al. (US Pre-Grant Publication 2022/0413997) in view of Bouhnik et al. (US Pre-Grant Publication 2019/0361934).
As to claim 20, Ben Shaul as modified teaches the non-transitory computer-readable medium of claim 16.
Ben Shaul does not teach wherein generating the set of search results comprises:
determining that the set of second image embeddings corresponds to representative embeddings for a set of clusters of embeddings;
determining a set of representative design fragments associated with the set of clusters of embeddings; and
adding the set of representative design fragments to the set of search results.
Bouhnik teaches wherein generating the set of search results comprises:
determining that the set of embeddings corresponds to representative embeddings for a set of clusters of embeddings (see Bouhnik paragraphs [0045] and [0066]);
determining a set of representative design fragments associated with the set of clusters of embeddings (see Bouhnik paragraphs [0045] and [0066]; and
adding the set of representative design fragments to the set of search results (see Bouhnik paragraphs [0045] and [0066]).
It would have been obvious to one of ordinary skill in the art before the earliest filing date of the invention to have modified Ben Shaul by the teachings of Bouhnik because both references are directed towards data analysis to identify matching results. Bouhnik merely provides additional analysis to Ben Shaul that will help to better analyze and group results for a user, which will improve the usability of the search system of Ben Shaul.
As to claim 21, Ben Shaul as modified by Bouhnik teaches the non-transitory computer-readable medium of claim 20, wherein determining the set of representative design fragments comprises retrieving, from the index, the set of representative design fragments from a set of mappings that include the representative embeddings (see Bouhnik paragraphs [0045] and [0066]).
As to claim 22, Ben Shaul as modified by Bouhnik teaches the non-transitory computer-readable medium of claim 21, wherein each of the representative embeddings comprises a centroid of a corresponding cluster in the set of clusters of embeddings (see Bouhnik paragraph [0066]).
Claim 23 is rejected under 35 U.S.C. 103 as being unpatentable over Ben Shaul et al. (US Pre-Grant Publication 2024/0086460) in view of Shi et al. (US Pre-Grant Publication 2022/0413997), in view of Bouhnik et al. (US Pre-Grant Publication 2019/0361934), and further in view of Capper et al. (US Pre-Grant Publication 2005/0144158).
As to claim 23, Ben Shaul as modified as modified teaches the non-transitory computer-readable medium of claim 20.
Ben Shaul as modified does not teach wherein generating the set of search results further comprises:
matching a user selection of a design fragment in the set of representative design fragments to a cluster in the set of clusters of embeddings; and
updating the set of search results with one or more additional design fragments from the cluster.
Capper wherein generating the set of search results further comprises:
matching a user selection of a design fragment in the set of representative design fragments to a cluster in the set of clusters of embeddings (see Capper paragraphs [0072]-[0075]); and
updating the set of search results with one or more additional design fragments from the cluster (see Capper paragraphs [0072]-[0075]).
It would have been obvious to one of ordinary skill in the art before the earliest filing date of the invention to have modified Ben Shaul by the teachings of Capper because both references are directed towards data analysis to identify matching results. Capper merely provides additional analysis to Ben Shaul that will help to better analyze results for a user and aid a user in additional discovery of results.
Claims 25-26 are rejected under 35 U.S.C. 103 as being unpatentable over Ben Shaul et al. (US Pre-Grant Publication 2024/0086460) in view of Shi et al. (US Pre-Grant Publication 2022/0413997) in view of Amacker (US Patent 8,631,029).
As to claim 25, Ben Shaul as modified teaches the non-transitory computer-readable medium of claim 16.
Ben Shaul as modified does not teach wherein generating the set of search results comprises:
determining a third image embedding for a design fragment that is selected by a user from the set of search results; and
generating an additional set of search results based on the third image embedding.
Amacker teaches wherein generating the set of search results comprises:
determining a third image embedding for a design fragment that is selected by a user from the set of search results (see Amacker 5:22-6:15 and Abstract. Also see Ben Shaul paragraph [0045] for searching for image embeddings from a query image); and
generating an additional set of search results based on the third image embedding (see Amacker 5:22-6:15 and Abstract. See Ben Shaul paragraph [0045] for searching using image embedding).
It would have been obvious to one of ordinary skill in the art before the earliest filing date of the invention to have modified Ben Shaul by the teachings of Amacker because both references are directed towards data analysis to identify matching results. Amacker merely provides additional analysis to Ben Shaul that will help to better analyze results for a user and aid a user in additional discovery of results.
.
As to claim 26, Ben Shaul as modified by Amacker teaches the non-transitory computer-readable medium of claim 25, wherein generating the additional set of search results comprises:
matching the third image embedding to an additional set of embeddings in the index (see Amacker 5:22-6:15 and Abstract. See Ben Shaul paragraph [0045] for searching using image embedding); and
adding, to the additional set of search results, an additional set of design fragments mapped to the additional set of embeddings within the index (see Amacker 5:22-6:15 and Abstract. See Ben Shaul paragraph [0045] for searching using image embedding).
Claim 27 is rejected under 35 U.S.C. 103 as being unpatentable over Ben Shaul et al. (US Pre-Grant Publication 2024/0086460) in view of Shi et al. (US Pre-Grant Publication 2022/0413997) in view of Amacker (US Patent 8,631,029), and further in view of Armstrong et al. (US Pre-Grant Publication 2023/0161809).
As to claim 27, Ben Shaul as modified teaches the non-transitory computer-readable medium of claim 25.
Ben Shaul as modified does not teach wherein generating the additional set of search results comprises:
matching an aggregation of the one or more first image embeddings with the third image embedding to an additional set of embeddings in the index; and
adding, to the additional set of search results, an additional set of design fragments mapped to the additional set of embeddings within the index.
Armstrong teaches wherein generating the additional set of search results comprises:
matching an aggregation of the one or more first image embeddings with the third image embedding to an additional set of embeddings in the index (see Armstrong paragraphs [0125]-[0127]); and
adding, to the additional set of search results, an additional set of design fragments mapped to the additional set of embeddings within the index (see Armstrong paragraphs [0125]-[0127]).
It would have been obvious to one of ordinary skill in the art before the earliest filing date of the invention to have modified Ben Shaul by the teachings of Armstrong because both references are directed towards data analysis to identify matching results. Armstrong merely provides additional analysis to Ben Shaul that will help to better analyze and group results for a user, which will improve the usability of the search system of Ben Shaul.
Claim 30 is rejected under 35 U.S.C. 103 as being unpatentable over Shi et al. (US Pre-Grant Publication 2022/0413997) in view of Ben Shaul et al. (US Pre-Grant Publication 2024/0086460), and further in view of Hepper et al. (US Pre-Grant Publication 2015/0269128).
As to claim 30, Shi as modified teaches the computer system of claim 29.
Shi as modified does not teach wherein the operations comprise:
generating the rendering using a headless browser.
Hepper teaches wherein the operations comprise:
generating the rendering using a headless browser (see Hepper paragraph [0022]).
It would have been obvious to one of ordinary skill in the art before the earliest filing date of the invention to have modified Shi by the teachings of Hepper because Hepper provides a graphical user interface output that will show any results in a mode that may completely fill up the active area of the display. This will make displaying results of any searching more clear for a user.
Response to Arguments
Applicant's arguments filed 17 June 2026 have been fully considered but they are not persuasive.
Response to 35 USC 101 Arguments
Applicant cites to the specification as filed, paragraphs [0020]-[0021], [0059], and [0068].
Applicant then recites the claim language, without differentiating between mental process elements and elements beyond the mental process.
Applicant argues that “amended claim 1 recites a specific functional structure that embodies the technical improvement, rather than reciting the improvement merely as an outcome.”
In response to this argument, it is noted that the improvement, as described in paragraph [0021] of the specification, appears to be an improvement in efficiency by automating a manual task. It is directed towards increasing the efficiency of a user’s ability to find similar designs. Searching – particularly a user’s ability to manually search – is a mental process. Any alleged improvement to searching alone is an improvement to a mental process. An improved mental process remains a mental process, albeit improved, and is patent ineligible.
Cited paragraph [0059] includes features which are not claimed, such as “a Contrastive Language Pretraining neural network and/or another type of multimodal embedding model.” Applicant is reminded that unclaimed features from the specification receive no patentable weight until claimed.
Cited paragraph [0068] is merely a list of mathematical algorithms for calculating similarity, most of which are not claimed.
It is unclear from Applicant’s citations how any alleged improvement is tied to an additional element in the claim beyond the mental process. As noted above, an improved mental process remains a mental process, albeit improved. Applicant is requested to identify the elements in the claims that are not data identification, analysis, or judgment steps and explain how at least one of those additional elements integrate the mental process into either an improvement to computing technology in view of the prior art when filed as supported by the specification or provide an improvement to computing technology as supported by the specification.
Applicant argues that “Second, the recited operations in amended claim 1 do not end in an abstract result, but instead recite the specific operations (e.g., "generating, using the image embedding model, a search embedding," "respectively determining a vector similarity between each of the plurality of design embeddings in the index and the search embedding") requiring a technique by which the result is produced. The Federal Circuit has repeatedly held that claims reciting a specific technique that achieves a result, rather than the result itself, are not abstract. See Thales Visionix Inc. v. United States, 850 F.3d 1343 (Fed. Cir. 2017) ("[T]he claims are directed to systems and methods that use inertial sensors in a non- conventional manner...."). Amended claim 1 recites how the result is produced by reciting specific operations and specific results produced via the image embedding model and the user interface using the computer system.”
In response to this argument, it is noted that generating a search embedding and determining a vector similarity are both data analysis steps. A human being equipped with a generic computer is capable of performing such steps. Because a human being equipped with a computer is capable of performing these data analysis steps of generating a search embedding and determining a vector similarity, these are mental process steps and alone cannot integrate the claimed invention into a practical application.
Applicant argues that “Third, the recited operations in amended claim 1 do not end in an abstract determination, but instead produce a concrete technical output that changes the operation of the recited index and user interface. The image embedding model generates the plurality of design embeddings mapped to the plurality of design fragments in the index and generates the search embedding from the image query, the vector similarity is respectively determined between each of the plurality of design embeddings in the index and the search embedding, and the output design fragment corresponding to an output design embedding from the plurality of design embeddings is displayed at the user interface based on the vector similarity. As a result, amended claim 1 directly corresponds to the standard of eligibility recognized in Diamond v. Diehr, 450 U.S. 175 (1981) ("When a claim...considered as a whole, is performing a function which the patent laws were designed to protect (e.g., transforming or reducing an article to a different state or thing), then the claim satisfies § 101's requirements."). Id. at 192.”
In response to this argument, the claims conclude in displaying a result of an analysis. It is noted that nothing is done with this display. An analysis is performed and a display of the result is output to a user. While displaying a result of an analysis is an additional element beyond a mental process, it appears to be merely post-solution activity that is well-understood and conventional (see MPEP 2106.05(g)(3)). Applicant has not identified, with citations to the specification as filed, how simply displaying a result and not using the result in any fashion integrates the mental process steps of the claims into a practical application.
Applicant argues that “Amended claim 1 also recites features consistent with what the M.P.E.P. identifies as weighing toward eligibility, namely, improving a technology or technical field (M.P.E.P. § 2106.05(a)), effecting a particular transformation (§ 2106.05(c)), and applying any exception in a manner that imposes meaningful limits (§ 2106.05(e)).”
Applicant adds “Fourth, the language recited in amended claim 1 corresponds directly to "claims...directed to an improvement in the functioning of a computer" and "teaches that the patent is not facing a situation where a general-purpose computer is merely invoked to automate a known practice." See Enfish, 822 F.3d at 1336; see also DDR Holdings, LLC v. Hotels.com, L.P., 773 F.3d 1245 (Fed. Cir. 2014) ("not merely invoking a computer to automate a known practice"); Uniloc USA, Inc. v. LG Elecs. USA, Inc., 957 F.3d 1303 (Fed. Cir. 2020) ("directed to improvements to the functionality of a computer or network platform itself').”
Applicant is requested to identify the additional elements of the claims beyond the mental process that result in these outcomes, as supported by a citation to the specification as filed explaining how these outcomes are achieved using those particular elements.
Applicant argues that “The Office Action characterizes original claim 1 as reciting "mental process without significantly more," including the steps of "dividing one or more hierarchical structures", "updating an index", and "processing a search query." Office Action, pg 2. Although Applicant respectfully disagrees, amended claim 1 now recites "storing, in an index, at least one mapping between the plurality of design fragments and a plurality of design embeddings generated by an image embedding model for the plurality of design fragments," "generating, using the image embedding model, a search embedding corresponding to the image query," and "processing the image query, including respectively determining a vector similarity between each of the plurality of design embeddings in the index and the search embedding." The human mind practically cannot perform these features recited in amended claim 1, whether manually or with pen and paper, because such image embedding operations, even for relatively small images, are outside the scope of practical mental activity. For example, using a Vision Transformer (ViT), base variant with 16x16 patches having 86M parameters, generating 196 embeddings for a 224x224 pixel image can be estimated to consume about 231 MFLOPs. Dosovitskiy et al., "An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale," ICLR 2021 (arXiv:2010.11929, October 2020); Kaplan et al., "Scaling Laws for Neural Language Models," arXiv:2001.08361 (2020), cited for general FLOP-counting convention for transformer matrix multiplications. See October 2019 Patent Eligibility Guidance Update, pg 7 ("Claims do not recite a mental process when they do not contain limitations that can practically be performed in the human mind."). Amended claim 1 therefore recites features that cannot practically be performed in the human mind and do not fall within the mental-processes grouping.”
In response to this argument, it is noted that such operations may be performed by a human being with a generic machine while still being patent ineligible. As noted in MPEP 2106.04(a)(2) III C, “claims can recite a mental process even if they are claimed as being performed on a computer.
The Supreme Court recognized this in Benson, determining that a mathematical algorithm for converting binary coded decimal to pure binary within a computer’s shift register was an abstract idea. The Court concluded that the algorithm could be performed purely mentally even though the claimed procedures "can be carried out in existing computers long in use, no new machinery being necessary." 409 U.S at 67, 175 USPQ at 675. See also Mortgage Grader, 811 F.3d at 1324, 117 USPQ2d at 1699 (concluding that concept of "anonymous loan shopping" recited in a computer system claim is an abstract idea because it could be "performed by humans without a computer").”
MPEP 2106.04(a)(2) III C 1-3 further elaborate on the idea that a claim may still be directed towards an abstract idea despite the use of a generic machine. Thus, though the claims may not be performed solely in a human mind, the storing, generating, and processing steps may be performed by a human with a generic computer.
It is noted that the current claims do not require using a “Vision Transformer (ViT), base variant with 16x16 patches having 86M parameters, generating 196 embeddings for a 224x224 pixel image.” Applicant is reminded that unclaimed features from the specification receive no patentable weight until claimed.
As noted in Recentive Analytics v. Fox. Corp., No. 23-2437 (Fed. Cir. 2025), merely applying a generic machine learning system, such as the claimed “one or more machine learning models comprising an image embedding model,” to a different context does not provide significantly more than an abstract idea. Thus, though an image embedding model is claimed, because no details of the learning process of the model are provided, the embedding model appears to be a generic machine learning model. In view of Recentive and is thus insufficient to overcome 35 USC 101.
Applicant argues that “Amended claim 1 recites a specific computer system as an ordered combination of operations that produces a concrete technical result,” then lists out the claimed steps, including a plurality of data analysis (mental process) steps.
Applicant cites the specification, paragraphs [0020] and [0021].
In response to this argument, it is noted that merely automating a manual task does not integrate a mental process into a practical application. See MPEP 2106.05(a)(I)(iii).
iii. Mere automation of manual processes, such as using a generic computer to process an application for financing a purchase, Credit Acceptance Corp. v. Westlake Services, 859 F.3d 1044, 1055, 123 USPQ2d 1100, 1108-09 (Fed. Cir. 2017) or speeding up a loan-application process by enabling borrowers to avoid physically going to or calling each lender and filling out a loan application, LendingTree, LLC v. Zillow, Inc., 656 Fed. App'x 991, 996-97 (Fed. Cir. 2016) (non-precedential);
Applicant argues that “The Office Action characterizes original claim 1 as reciting "'one or more processors' and 'memory'" that "are recited at a high level of generality" and "appear to be generic computing hardware elements," such that "[t]he recitation of generic hardware is little more than using a computer to perform an abstract idea." Office Action, pg 3. Although Applicant respectfully disagrees, as noted above, amended claim 1 does not recite generic computing functions. For example, amended claim 1's "image embedding model," "design embeddings...for the plurality of design fragments," "vector similarity between each of the plurality of design embeddings in the index and the search embedding," and "displaying, at the user interface based on the vector similarity, at least one output design fragment" represent a specific technical implementation that is a practical application under § 101.”
As noted above, merely invoking a generic machine learning model (“image embedding model”) that generates design embeddings does not provide a practical application in view of Recentive Analytics v. Fox. Corp., No. 23-2437 (Fed. Cir. 2025).
Calculating vector similarity is a data analysis and thus a mental process.
Displaying the output of a data analysis, without more, does not represent a practical application because merely outputting a result of an analysis is well-understood, routine, and conventional (see MPEP 2106.05(g)(3)).
Applicant’s remaining arguments with respect to claims have been considered but are moot because the new ground of rejection does not rely on any reference applied in the prior rejection of record for any teaching or matter specifically challenged in the argument.
Conclusion
Applicant's amendment necessitated the new ground(s) of rejection presented in this Office action. Accordingly, THIS ACTION IS MADE FINAL. See MPEP § 706.07(a). Applicant is reminded of the extension of time policy as set forth in 37 CFR 1.136(a).
A shortened statutory period for reply to this final action is set to expire THREE MONTHS from the mailing date of this action. In the event a first reply is filed within TWO MONTHS of the mailing date of this final action and the advisory action is not mailed until after the end of the THREE-MONTH shortened statutory period, then the shortened statutory period will expire on the date the advisory action is mailed, and any nonprovisional extension fee (37 CFR 1.17(a)) pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. In no event, however, will the statutory period for reply expire later than SIX MONTHS from the mailing date of this final action.
Any inquiry concerning this communication or earlier communications from the examiner should be directed to CHARLES D ADAMS whose telephone number is (571)272-3938. The examiner can normally be reached M-F, 9-5:30 EST.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Aleksandr Kerzhner can be reached at 5712701760. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/CHARLES D ADAMS/Primary Examiner, Art Unit 2165
