Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
Acknowledgment is made of the information disclosure statements filed April 25, 2024, which comply with 37 CFR 1.97. As such, the information disclosure statements have been placed in the application file and the information referred to therein has been considered by the examiner.
Claim Rejections - 35 USC § 101
35 U.S.C. 101 reads as follows:
Whoever invents or discovers any new and useful process, machine, manufacture, or composition of matter, or any new and useful improvement thereof, may obtain a patent therefor, subject to the conditions and requirements of this title.
Claims 1-27 are rejected under 35 U.S.C. 101 because the claimed invention is directed to non-statutory subject matter.
According to the USPTO guidelines, a claim is directed to non-statutory subject matter if:
Step 1: The claim does not fall within one of the four statutory categories of invention (process, machine, manufacture, or composition of matter), or,
Step 2: The claim recites a judicial exception, e.g. an abstract idea, without reciting additional elements that amount to significantly more than the judicial exception, as determined using the following analysis:
Step 2A, Prong 1: Does the claim recite an abstract idea, law of nature, or natural phenomenon?
Step 2A, Prong 2: Does the claim recite additional elements that integrate the judicial exception into a practical application?
Step 2B: Does the claim recite additional elements that amount to significantly more than the judicial exception?
MPEP 2106.04(a)(2)(I) states: "The mathematical concepts grouping is defined as
mathematical relationships, mathematical formulas or equations, and mathematical
calculations."
MPEP 2106.04(a)(2)(III) states: "Accordingly, the "mental processes" abstract idea grouping is defined as concepts performed in the human mind, and examples of mental
processes include observations, evaluations, judgements, and opinions.
Further, the MPEP states: "The courts do not distinguish between mental processes that
are performed entirely in the human mind and mental processes that require a human to use a
physical aid (e.g. pen and paper or a slide run) to perform the claim limitation.
Using the two-step inquiry, it is clear that Claims 1-19 are each directed to non-statutory
subject matter as shown below:
Please note the following:
The following groups of claims are expressed in different statutory categories:
Claims 1-20 are directed to a method for providing hierarchical machine learning model visualization.
Claims 21-27 are directed to a system comprising of one or more processing devices and one or more memory units configured to carry out a process for providing hierarchical machine learning model visualization.
With respect to Claims 1 and 21, which are independent claims with identical claim limitations:
Step 1: Claim 1 is directed to a computer-implemented method, also known as a process, which is one of the four statutory categories of patentable subject matter. Claim 21 is directed to a system comprising one or more processing devices and one or more memory units storing instructions, corresponding to a machine, which is one of the four statutory categories of patentable subject matter.
Step 2A, Prong 1: A judicial exception is recited in the claims as they recite mental processes, which are abstract ideas:
"identifying a hierarchical structure for the computational graph, the hierarchical structure defining a hierarchical aggregation of nodes from the plurality of nodes of the computational graph;"; Identifying a hierarchical structure that groups the nodes of a computational graph into aggregations is an abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion), e.g., a person can look at a graph of operations and decide which nodes belong together as a layer or block - See MPEP § 2106.04(a)(2)(III).
"generating a visual layout of the computational graph, the visual layout including respective visual elements for each of one or more node aggregation layers defined by the hierarchical structure, wherein the respective visual elements for each of the one or more node aggregation layers are provided in place of respective visual elements for individual nodes of the plurality of nodes assigned to the one or more node aggregation layers; and"; Generating a visual layout in which a single visual element represents a group of aggregated nodes in place of the individual nodes is an abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion), e.g., a person can draw a diagram of the graph in which one box represents each group of nodes rather than drawing each node individually - See MPEP § 2106.04(a)(2)(III).
Step 2A, Prong 2: The claims do not recite additional elements that integrate the judicial exception into a practical application:
"receiving a computational graph for a machine learning model, the computational graph including a plurality of nodes and a plurality of edges connecting different ones of the plurality of nodes, each node representing a respective operation of the machine learning model and each edge representing a data flow between a respective pair of nodes connected by the edge;"; Receiving a computational graph for a machine learning model is considered insignificant extra-solution activity (mere data gathering) - see MPEP 2106.05(g) in addition to generally links the use of the abstract idea to a particular technological environment or field of use - See MPEP § 2106.05(h).
"providing the visual layout of the computational graph for display in a user interface at a user device."; Providing a visual layout for display in a user interface at a user device only amounts to "apply it" and the mere instructions to apply the abstract idea using a generic computer component - see MPEP 2106.05(f)(1) and is insignificant extra-solution activity (outputting the result) - see MPEP 2106.05(g) in addition to generally links the use of the abstract idea to a particular technological environment or field of use - See MPEP § 2106.05(h).
Step 2B: The claims do not recite additional elements that amount to significantly more than the judicial exception. Receiving a computational graph and providing a visual layout for display at a user device constitute receiving or transmitting data over a network and presenting data, which have been recognized as well-understood, routine, and conventional when they are claimed in a generic manner - See MPEP 2106.05(d)(II). The recited computer, processing devices, memory units, user interface, and user device are generic computer components used in their ordinary capacity as tools to perform the abstract idea and amount to "apply it" - see MPEP 2106.05(f)(1)-(2). The recitation that the computational graph is for a machine learning model is generally linked to a particular technological environment or field of use (AI/ML) - see MPEP 2106.05(h). The claims fail to recite details of how the hierarchical structure is identified or how the visual layout is generated beyond the result itself - see MPEP 2106.05(f)(1). Considered as an ordered combination, the additional elements add nothing that is not already present when they are considered individually.
Therefore, Claims 1 and 21 are directed to non-statutory subject matter and rejected.
With respect to Claims 2 and 22, which are dependent claims with identical claim limitations:
Step 2A, Prong 1: The claims recite the abstract ideas of Claims 1 and 21, and further recite the following mental processes:
"in response to receiving the user input, assigning the individual nodes or node aggregation layers targeted for aggregation to a new node aggregation layer in the hierarchical structure; and"; Assigning selected nodes or node aggregation layers to a new group is an abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion) - See MPEP § 2106.04(a)(2)(III).
"modifying the visual layout of the computational graph by replacing respective visual elements for the individual nodes or node aggregation layers in the group with a new visual element for the new node aggregation layer."; Modifying a layout by replacing several visual elements with a single new visual element is an abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion), e.g., a person can redraw a diagram with one box in place of several - See MPEP § 2106.04(a)(2)(III).
Step 2A, Prong 2: The claims do not recite additional elements that integrate the judicial exception into a practical application:
"receiving, from the user interface, a user input that identifies a selection of visual elements from the visual layout of the computational graph for a group of individual nodes or node aggregation layers targeted for aggregation;"; Receiving a user selection through a user interface is considered insignificant extra-solution activity (mere data gathering) - see MPEP 2106.05(g) in addition to generally links the use of the abstract idea to a particular technological environment or field of use - See MPEP § 2106.05(h).
Step 2B: The claims do not recite additional elements that amount to significantly more than the judicial exception. Receiving user input through a user interface is well-understood, routine, and conventional receiving of data when claimed in a generic manner - See MPEP 2106.05(d)(II). The usage of a user interface is generally linked to a particular technological environment or field of use (GUI) - see MPEP 2106.05(h).
Therefore, Claims 2 and 22 are directed to non-statutory subject matter and rejected.
With respect to Claim 3:
Step 2A, Prong 1: The claim recites the abstract ideas of Claims 1 and 2, and further recites the following mental process:
"....to assign the individual node or node aggregation layers targeted for aggregation to the new node aggregation layer in the hierarchical structure."; Assigning nodes or node aggregation layers to a new node aggregation layer is an abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion) - See MPEP § 2106.04(a)(2)(III).
Step 2A, Prong 2: The claim does not recite additional elements that integrate the judicial exception into a practical application:
"conditioning a one-shot machine learning model on the group of individual nodes or node aggregation layers targeted for aggregation;"; Conditioning a one-shot machine learning model, recited at a high level of generality without any detail of how the conditioning is performed, only amounts to "apply it" and the mere instructions to implement an abstract idea on a computer - see MPEP 2106.05(f)(1) in addition to generally links the use of the abstract idea to a particular technological environment or field of use - See MPEP § 2106.05(h).
"and processing the computational graph with the one-shot machine learning model...."; Utilizing a one-shot machine learning model to perform the assignment only amounts to "apply it" and the mere instructions to implement an abstract idea on a computer - see MPEP 2106.05(f)(1) in addition to generally links the use of the abstract idea to a particular technological environment or field of use - See MPEP § 2106.05(h).
Step 2B: The claim does not recite additional elements that amount to significantly more than the judicial exception. The claim fails to recite how the one-shot model is conditioned or how it performs the assignment and uses the model merely as a tool to perform the abstract idea - see MPEP 2106.05(f)(1). The usage of a machine learning model is generally linked to a particular technological environment or field of use (AI/ML) see MPEP 2106.05(h).
Therefore, Claim 3 is directed to non-statutory subject matter and rejected.
With respect to Claim 4:
Step 2A, Prong 1: The claim incorporates the abstract ideas of Claims 1 and 2 and does not recite any additional abstract idea.
Step 2A, Prong 2: The claim does not recite additional elements that integrate the judicial exception into a practical application:
"providing the modified visual layout for display in the user interface at the user device."; Providing a modified visual layout for display in a user interface only amounts to "apply it" and the mere instructions to apply the abstract idea using a generic computer component - see MPEP 2106.05(f)(1) and is insignificant extra-solution activity (outputting the result) - see MPEP 2106.05(g) in addition to generally links the use of the abstract idea to a particular technological environment or field of use - See MPEP § 2106.05(h).
Step 2B: The claim does not recite additional elements that amount to significantly more than the judicial exception. Presenting data on a display is well-understood, routine, and conventional when claimed in a generic manner - See MPEP 2106.05(d)(II). The usage of a user interface to display the layout is generally linked to a particular technological environment or field of use (GUI) - see MPEP 2106.05(h).
Therefore, Claim 4 is directed to non-statutory subject matter and rejected.
With respect to Claim 5:
Step 2A, Prong 1: The claim recites the abstract ideas of Claim 1, and further recites the following mental process:
"wherein identifying the hierarchical structure for the computational graph includes performing a search of the computational graph for subgraphs with similar architectures."; Identifying a hierarchical structure by searching a graph for subgraphs with similar architectures is an abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion), e.g., a person can visually compare portions of a graph for repeated patterns - See MPEP § 2106.04(a)(2)(III).
Step 2A, Prong 2: The claim does not recite any additional elements that integrate the judicial exception into a practical application:
Step 2B: The claim does not recite additional elements that amount to significantly more than the judicial exception for the reasons set forth above.
Therefore, Claim 5 is directed to non-statutory subject matter and rejected.
With respect to Claim 6:
Step 2A, Prong 1: The claim recites the abstract ideas of Claim 1, and further recites the following mental process:
"....to identify subgraphs with similar architectures."; Identifying subgraphs with similar architectures is an abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion) - See MPEP § 2106.04(a)(2)(III).
Step 2A, Prong 2: The claim does not recite additional elements that integrate the judicial exception into a practical application:
"wherein identifying the hierarchical structure for the computational graph includes processing the computational graph with a one-shot machine learning model...."; Utilizing a one-shot machine learning model to identify the subgraphs only amounts to "apply it" and the mere instructions to implement an abstract idea on a computer - see MPEP 2106.05(f)(1) in addition to generally links the use of the abstract idea to a particular technological environment or field of use - See MPEP § 2106.05(h).
Step 2B: The claim does not recite additional elements that amount to significantly more than the judicial exception. The one-shot model is recited at a high level of generality without detail of how it identifies the subgraphs and amounts to "apply it" - see MPEP 2106.05(f)(1). The usage of a machine learning model is generally linked to a particular technological environment or field of use (AI/ML) - see MPEP 2106.05(h).
Therefore, Claim 6 is directed to non-statutory subject matter and rejected.
With respect to Claim 7:
Step 2A, Prong 1: The claim recites the abstract ideas of Claim 1, and further recites the following mental process:
"wherein the hierarchical structure includes a main body of the computational graph, layers of the computational graph, and connections between the layers of the computational graph."; Identifying a hierarchical structure that includes a main body, layers, and connections between the layers of a graph is an abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion) - See MPEP § 2106.04(a)(2)(III).
Step 2A, Prong 2: The claim does not recite any additional elements beyond those of Claim 1, which do not integrate the judicial exception into a practical application for the reasons set forth above.
Step 2B: The claim does not recite any additional elements beyond those of Claim 1, which do not amount to significantly more than the judicial exception for the reasons set forth above.
Therefore, Claim 7 is directed to non-statutory subject matter and rejected.
With respect to Claim 8:
Step 2A, Prong 1: The claim recites the abstract ideas of Claims 1 and 7, and further recites the following mental process:
"wherein processing the computational graph further includes determining a number of nodes in each layer of the computational graph to identify a main body of the computational graph."; Determining the number of nodes in each layer to identify a main body of a graph is an abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion), e.g., a person can count the nodes in each layer - See MPEP § 2106.04(a)(2)(III).
Step 2A, Prong 2: The claim does not recite any additional elements beyond those of Claim 1, which do not integrate the judicial exception into a practical application for the reasons set forth above.
Step 2B: The claim does not recite any additional elements beyond those of Claim 1, which do not amount to significantly more than the judicial exception for the reasons set forth above.
Therefore, Claim 8 is directed to non-statutory subject matter and rejected.
With respect to Claim 9:
Step 2A, Prong 1: The claim recites the abstract ideas of Claims 1 and 7, and further recites the following mental process:
"wherein processing the computational graph further includes analyzing types of nodes in the computational graph to identify different layers in the machine learning model."; Analyzing the types of nodes in a graph to identify different layers is an abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion) - See MPEP § 2106.04(a)(2)(III).
Step 2A, Prong 2: The claim does not recite any additional elements beyond those of Claim 1, which do not integrate the judicial exception into a practical application for the reasons set forth above.
Step 2B: The claim does not recite any additional elements beyond those of Claim 1, which do not amount to significantly more than the judicial exception for the reasons set forth above.
Therefore, Claim 9 is directed to non-statutory subject matter and rejected.
With respect to Claim 10:
Step 2A, Prong 1: The claim recites the abstract ideas of Claims 1 and 7, and further recites the following mental process:
"wherein processing the computational graph further includes analyzing a plurality of edges between the plurality of nodes in the computational graph to identify dependencies between the layers."; Analyzing the edges between nodes to identify dependencies between layers is an abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion), e.g., a person can trace connections on a drawn graph - See MPEP § 2106.04(a)(2)(III).
Step 2A, Prong 2: The claim does not recite any additional elements beyond those of Claim 1, which do not integrate the judicial exception into a practical application for the reasons set forth above.
Step 2B: The claim does not recite any additional elements beyond those of Claim 1, which do not amount to significantly more than the judicial exception for the reasons set forth above.
Therefore, Claim 10 is directed to non-statutory subject matter and rejected.
With respect to Claim 11:
Step 2A, Prong 1: The claim recites the abstract ideas of Claims 1 and 2, and further recites the following mental process:
"wherein the user input further includes a tag labeling the group of individual nodes or node aggregation layers targeted for aggregation and the new visual element for the new node aggregation layer is labeled with the tag."; Labeling a group of nodes or node aggregation layers with a tag and labeling the new visual element for the new node aggregation layer with that tag is an abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion), e.g., a person can choose a name for a group of nodes and write that name on the box drawn for the group - See MPEP § 2106.04(a)(2)(III).
Step 2A, Prong 2: The claim does not recite any additional elements beyond those of Claims 1 and 2, which do not integrate the judicial exception into a practical application for the reasons set forth above.
Step 2B: The claim does not recite any additional elements beyond those of Claims 1 and 2, which do not amount to significantly more than the judicial exception for the reasons set forth above.
Therefore, Claim 11 is directed to non-statutory subject matter and rejected.
With respect to Claim 12:
Step 2A, Prong 1: The claim recites the abstract ideas of Claim 1, and further recites the following mental process:
"wherein the visual layout of the computational graph is generated in a hierarchical layout."; Generating the visual layout in a hierarchical layout is an abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion), e.g., a person can draw a tree style diagram - See MPEP § 2106.04(a)(2)(III).
Step 2A, Prong 2: The claim does not recite any additional elements beyond those of Claim 1, which do not integrate the judicial exception into a practical application for the reasons set forth above.
Step 2B: The claim does not recite any additional elements beyond those of Claim 1, which do not amount to significantly more than the judicial exception for the reasons set forth above.
Therefore, Claim 12 is directed to non-statutory subject matter and rejected.
With respect to Claim 13:
Step 2A, Prong 1: The claim recites the abstract ideas of Claim 1 and does not recite any additional abstract idea.
Step 2A, Prong 2: The claim does not recite additional elements that integrate the judicial exception into a practical application:
"wherein the visual layout of the computational graph is presented within a debugging tool for the machine learning model."; Presenting the visual layout within a debugging tool only amounts to "apply it" and the mere instructions to apply the abstract idea using a generic computer component - see MPEP 2106.05(f)(1) in addition to generally links the use of the abstract idea to a particular technological environment or field of use - See MPEP § 2106.05(h).
Step 2B: The claim does not recite additional elements that amount to significantly more than the judicial exception. The debugging tool is recited generically and merely confines the abstract idea to a particular technological environment or field of use (software debugging) - see MPEP 2106.05(f)(1) and 2106.05(h).
Therefore, Claim 13 is directed to non-statutory subject matter and rejected.
With respect to Claim 14:
Step 2A, Prong 1: The claim recites the abstract ideas of Claim 1, and further recites the following mental process:
"....wherein the visual layout of the computational graph includes an input visual element of the input and an output visual element of the output at corresponding portions of the visual layout of the computational graph."; Including an input visual element and an output visual element at corresponding portions of the visual layout is an abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion), e.g., a person can draw labeled input and output boxes on a diagram - See MPEP § 2106.04(a)(2)(III).
Step 2A, Prong 2: The claim does not recite additional elements that integrate the judicial exception into a practical application:
"providing an input to the machine learning model to generate an output,...."; Providing an input to a machine learning model to generate an output, recited at a high level of generality, only amounts to "apply it" and the mere instructions to implement an abstract idea on a computer - see MPEP 2106.05(f)(1) and is insignificant extra-solution activity (mere data gathering) - see MPEP 2106.05(g) in addition to generally links the use of the abstract idea to a particular technological environment or field of use - See MPEP § 2106.05(h).
Step 2B: The claim does not recite additional elements that amount to significantly more than the judicial exception. Executing a model on an input using a generic computer is a computer functioning in its ordinary capacity - see MPEP 2106.05(f)(2). The usage of a machine learning model is generally linked to a particular technological environment or field of use (AI/ML) - see MPEP 2106.05(h).
Therefore, Claim 14 is directed to non-statutory subject matter and rejected.
With respect to Claim 15:
Step 2A, Prong 1: The claim incorporates the abstract ideas of Claim 1 and does not recite any additional abstract idea.
Step 2A, Prong 2: The claim does not recite additional elements that integrate the judicial exception into a practical application:
"wherein the machine learning model is a neural network."; Specifying that the machine learning model is a neural network generally links the use of the abstract idea to a particular technological environment or field of use - See MPEP § 2106.05(h).
Step 2B: The claim does not recite additional elements that amount to significantly more than the judicial exception. Limiting the type of model whose computational graph is visualized does not change how the abstract idea is performed and is generally linked to a particular technological environment or field of use (AI/ML) - see MPEP 2106.05(h).
Therefore, Claim 15 is directed to non-statutory subject matter and rejected.
With respect to Claim 16:
Step 2A, Prong 1: The claim incorporates the abstract ideas of Claim 1 and does not recite any additional abstract idea.
Step 2A, Prong 2: The claim does not recite additional elements that integrate the judicial exception into a practical application:
"wherein the neural network includes at least one selected from the following group: (a) a recurrent neural network; (b) a feedforward neural network; (c) a convolutional neural network; or (d) a transformer neural network."; Specifying a recurrent neural network, feedforward neural network, convolutional neural network, or a transformer neural network generally links the use of the abstract idea to a particular technological environment or field of use - See MPEP § 2106.05(h).
Step 2B: The claim does not recite additional elements that amount to significantly more than the judicial exception. Limiting the neural network to a particular known type does not change how the abstract idea is performed and is generally linked to a particular technological environment or field of use (AI/ML) - see MPEP 2106.05(h).
Therefore, Claim 16 is directed to non-statutory subject matter and rejected.
With respect to Claim 17:
Step 2A, Prong 1: The claim incorporates the abstract ideas of Claim 1 and does not recite any additional abstract idea.
Step 2A, Prong 2: The claim does not recite additional elements that integrate the judicial exception into a practical application:
"wherein the user interface is included within a web application."; Including the user interface within a web application only amounts to "apply it" and the mere instructions to implement an abstract idea using a generic computer component - see MPEP 2106.05(f)(2) in addition to generally links the use of the abstract idea to a particular technological environment or field of use - See MPEP § 2106.05(h).
Step 2B: The claim does not recite additional elements that amount to significantly more than the judicial exception. The web application is recited generically and is generally linked to a particular technological environment or field of use (Internet/web) see MPEP 2106.05(f)(2) and 2106.05(h).
Therefore, Claim 17 is directed to non-statutory subject matter and rejected.
With respect to Claim 18:
Step 2A, Prong 1: The claim recites the abstract ideas of Claim 1, and further recites the following mental process:
"wherein identifying a hierarchical structure for the computational graph is further based on a domain of the machine learning model."; Identifying a hierarchical structure for the computational graph based on a domain of the machine learning model is an abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion), e.g., a person can consider what the model is used for when deciding how to group its nodes - See MPEP § 2106.04(a)(2)(III).
Step 2A, Prong 2: The claim does not recite any additional elements beyond those of Claim 1, which do not integrate the judicial exception into a practical application for the reasons set forth above.
Step 2B: The claim does not recite any additional elements beyond those of Claim 1, which do not amount to significantly more than the judicial exception for the reasons set forth above.
Therefore, Claim 18 is directed to non-statutory subject matter and rejected.
With respect to Claim 19:
Step 2A, Prong 1: The claim recites the abstract ideas of Claim 1, and further recites the following mental process:
"....to identify a hierarchical structure for the computational graph,...."; Identifying a hierarchical structure for the computational graph is an abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion), as set forth with respect to Claim 1 - See MPEP § 2106.04(a)(2)(III).
Step 2A, Prong 2: The claim does not recite additional elements that integrate the judicial exception into a practical application:
"wherein a neural network is used...."; Utilizing a neural network to identify the hierarchical structure only amounts to "apply it" and the mere instructions to implement an abstract idea on a computer - see MPEP 2106.05(f)(1) in addition to generally links the use of the abstract idea to a particular technological environment or field of use - See MPEP § 2106.05(h).
"wherein the neural network is trained on training example machine learning models with labeled hierarchical structures."; Reciting that the neural network is trained on labeled training examples, without any detail of the training or of how the trained network identifies the structure, only amounts to "apply it" and the mere instructions to implement an abstract idea on a computer - see MPEP 2106.05(f)(1) in addition to generally links the use of the abstract idea to a particular technological environment or field of use - See MPEP § 2106.05(h).
Step 2B: The claim does not recite additional elements that amount to significantly more than the judicial exception. The trained neural network is recited as a generic tool for performing the abstract idea, and the claim fails to recite details of how a solution or outcome is accomplished - see MPEP 2106.05(f)(1). The usage of a neural network is generally linked to a particular technological environment or field of use (AI/ML) - see MPEP 2106.05(h).
Therefore, Claim 19 is directed to non-statutory subject matter and rejected.
With respect to Claim 20:
Step 2A, Prong 1: The claim recites the abstract ideas of Claim 1, and further recites the following mental process:
"wherein identifying a hierarchical structure for the computational graph includes iteratively aggregating the plurality of nodes from a high level of granularity to a low level of granularity."; Identifying a hierarchical structure by iteratively grouping nodes from a high level of granularity to a low level of granularity is an abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion) - See MPEP § 2106.04(a)(2)(III).
Step 2A, Prong 2: The claim does not recite any additional elements beyond those of Claim 1, which do not integrate the judicial exception into a practical application for the reasons set forth above.
Step 2B: The claim does not recite any additional elements beyond those of Claim 1, which do not amount to significantly more than the judicial exception for the reasons set forth above.
Therefore, Claim 20 is directed to non-statutory subject matter and rejected.
With respect to Claim 23:
Step 2A, Prong 1: The claim recites the abstract ideas of Claims 21 and 22, and further recites the following mental process:
"wherein the group of individual nodes or node aggregation layers targeted for aggregation include at least one individual node from the plurality of nodes and at least one aggregation layer for at least one sub-group of nodes from the plurality of nodes."; Grouping at least one individual node together with at least one existing aggregation layer is an abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion) - See MPEP § 2106.04(a)(2)(III).
Step 2A, Prong 2: The claim does not recite any additional elements beyond those of Claims 21 and 22, which do not integrate the judicial exception into a practical application for the reasons set forth above.
Step 2B: The claim does not recite any additional elements beyond those of Claims 21 and 22, which do not amount to significantly more than the judicial exception for the reasons set forth above.
Therefore, Claim 23 is directed to non-statutory subject matter and rejected.
With respect to Claim 24:
Step 2A, Prong 1: The claim recites the abstract ideas of Claim 21, and further recites the following mental process:
"wherein the hierarchical structure includes a first node aggregation layer that defines a collection of nodes aggregated without user input based on a zero-shot machine learning model."; Defining a node aggregation layer that collects nodes aggregated without user input based on a zero-shot machine learning model is an abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion), e.g., a person can group nodes by recognizing common structures in a graph without having been shown a prior example - See MPEP § 2106.04(a)(2)(III).
Step 2A, Prong 2: The claim does not recite any additional elements beyond those of Claim 21, which do not integrate the judicial exception into a practical application for the reasons set forth above.
Step 2B: The claim does not recite any additional elements beyond those of Claim 21, which do not amount to significantly more than the judicial exception for the reasons set forth above.
Therefore, Claim 24 is directed to non-statutory subject matter and rejected.
With respect to Claim 25:
Step 2A, Prong 1: The claim recites the abstract ideas of Claims 21 and 22, and further recites the following mental processes:
"wherein to identify a hierarchical structure for the computational graph includes to: identify a pattern associated with the group of individual nodes or node aggregation layers targeted for aggregation;"; Identifying a pattern associated with a selected group of nodes or node aggregation layers is an abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion) - See MPEP § 2106.04(a)(2)(III).
"and search the computational graph for other occurrences of the pattern, and automatically create additional aggregation layers for nodes or layers that match the pattern."; Searching a graph for other occurrences of a pattern and automatically creating additional aggregation layers for the nodes or layers that match is an abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion), e.g., a person can scan a graph for repeats of a pattern and circle each match as its own group - See MPEP § 2106.04(a)(2)(III).
Step 2A, Prong 2: The claim does not recite any additional elements beyond those of Claims 21 and 22, which do not integrate the judicial exception into a practical application for the reasons set forth above.
Step 2B: The claim does not recite any additional elements beyond those of Claims 21 and 22, which do not amount to significantly more than the judicial exception for the reasons set forth above.
Therefore, Claim 25 is directed to non-statutory subject matter and rejected.
With respect to Claim 26:
Step 2A, Prong 1: The claim incorporates the abstract ideas of Claim 21, and further recites the following mental process:
"wherein the visual layout further includes respective visual elements for each of one or more individual nodes from the plurality of nodes of the computational graph."; Including in the visual layout respective visual elements (boxes) for each of one or more individual nodes of the computational graph, in addition to the visual elements for the node aggregation layers, is an abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion), e.g., a person can draw a separate box for each individual node that is not part of a group - See MPEP § 2106.04(a)(2)(III).
Step 2A, Prong 2: The claim does not recite any additional elements that integrate the judicial exception into practical application:
Step 2B: The claim does not recite additional elements that amount to significantly more than the judicial exception.
Therefore, Claim 26 is directed to non-statutory subject matter and rejected.
With respect to Claim 27:
Step 2A, Prong 1: The claim incorporates the abstract ideas of Claims 21 and 26, and further recites the following mental processes:
"wherein the visual layout further includes visual elements that represent connections between individual nodes or groups of nodes;"; Including visual elements that represent connections between individual nodes or groups of nodes is an abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion), e.g., a person can draw lines between boxes - See MPEP § 2106.04(a)(2)(III).
"wherein the visual layout of the computational graph is generated by aggregating at least a portion of the plurality of nodes at different levels of granularity based on architectural features identified in the computational graph."; Generating the visual layout by aggregating nodes at different levels of granularity based on architectural features identified in the graph is an abstract idea of a mental process that can practically be performed in the human mind, with or without the use of a physical aid such as pen and paper (including an observation, evaluation, judgment, opinion) - See MPEP § 2106.04(a)(2)(III).
Step 2A, Prong 2: The claim does not recite any additional elements beyond those of Claim 21, which do not integrate the judicial exception into a practical application for the reasons set forth above.
Step 2B: The claim does not recite any additional elements beyond those of Claim 21, which do not amount to significantly more than the judicial exception for the reasons set forth above.
Therefore, Claim 27 is directed to non-statutory subject matter and rejected.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless – (a)(1) the claimed invention was patented, described in a printed publication, or in public use, on sale, or otherwise available to the public before the effective filing date of the claimed invention.
Claims 1, 5, 7, 8, 9, 10, 12, 13, 15, 16, 17, 18, 21, 26, and 27 are rejected under 35 U.S.C. 102(a)(1) as being anticipated by Pan et. AL, (Towards Efficient Visual Simplification of Computational Graphs in Deep Neural Networks, published December 21st, 2022, hereinafter “Pan”). Pan was published on December 21st , 2022 and this date is before the effective filing date of the instant application, i.e., April 25th , 2024. Therefore, Pan constitutes prior art under 35 U.S.C. 102(a)(1).
With respect to Claims 1 and 21:
Pan teaches:
"receiving a computational graph for a machine learning model, the computational graph including a plurality of nodes and a plurality of edges connecting different ones of the plurality of nodes, each node representing a respective operation of the machine learning model and each edge representing a data flow between a respective pair of nodes connected by the edge;" ([Page 4, Section 5.2] details receiving (parsing) a computational graph (raw graph) for a machine learning model (input DNN model), “1) Raw Graph denotes the computational graph structure data generated from the input DNN model by parsing the summary file. The raw graph contains nodes of different hierarchies and edges between nodes.” [Page 3, Section 3] discloses that the nodes of the computational graph represent operations (operators) and the edges represent data flow between them, “A computational graph is a DAG whose nodes represent tensors and operators, and the edges denote the data flow between them.” [Page 4, Section 5.1] further discloses that each node of the displayed graph is an operation node (OperationNode) and that tensors are not separate graph nodes but are attached to the operation nodes, “We separate nodes into OperationNode, DataNode, and MetaNode. OperationNode refers to nodes of operators…For the general visual design of the layout, we attach a DataNode to an OperationNode to represent their connections because edges between a DataNode and an OperationNode are unidirectional.” [Page 6, Section 5.2.2] confirms that each edge connects a respective pair of operation nodes, “Specifically, an edge from OperationNode s to OperationNode t can be sliced into at most three line segments…”)
"identifying a hierarchical structure for the computational graph, the hierarchical structure defining a hierarchical aggregation of nodes from the plurality of nodes of the computational graph;" ([Page 3, Section 3] details identifying a hierarchical structure (hierarchy of a hierarchical clustered graph) defining a hierarchical aggregation of nodes (recursive grouping into metanodes), “The hierarchy of a graph is defined as a recursive grouping placed on the nodes of the initial graph [58]. The nodes of a hierarchical graph contain metanodes and leaves. Metanodes denote subgraphs that contain a subset of nodes and a subset of the edges between these nodes (nodes of level 1 and 2 in Fig. 2b); leaves are the nodes of the input graph (nodes of level 3 in Fig. 2b). We represent a hierarchical computational graph as H0 = (H1 , E1 ), where H i = (H i+1 , E i+1 )… H i represents a set of nodes at the i-th level of the hierarchy.” [Page 4, Section 5.2] discloses that Pan’s system identifies and organizes this hierarchy from the received graph, “2) Processed Graph is created from the raw graph data. We organize the hierarchical structure information of nodes in a tree and identify the leaf nodes’ edges.” [Page 6, Section 5.2.2] further discloses identifying aggregation levels (modules) of the hierarchy, “In the first stage, all the MetaNodes are traversed to yield the module information. A MetaNode would be identified as a ModuleNode or “module” (R1.1), if its number of descendants is greater than the user-defined threshold. The module level of a ModuleNode is equivalent to its level of the hierarchy.” [Page 5, Section 5.2.1] additionally discloses that the system itself restructures the aggregation, “we propose to remove them by ungrouping one of the MetaNodes in the cycle and rebuild two new MetaNodes.”) Examiner’s Note: Pan does not just display a pre-existing grouping. Pan organizes the hierarchy into a tree, identifies modules and their levels, and regroups nodes into new MetaNodes.
"generating a visual layout of the computational graph, the visual layout including respective visual elements for each of one or more node aggregation layers defined by the hierarchical structure, wherein the respective visual elements for each of the one or more node aggregation layers are provided in place of respective visual elements for individual nodes of the plurality of nodes assigned to the one or more node aggregation layers;" ([Page 4, Fig. 3] details the respective visual elements (MetaNode glyphs) for each node aggregation layer (MetaNode) each rendered in either a “collapsed” or an “expanded” state, “Fig. 3. The hierarchy and visual encoding of various types of nodes.” [Page 5, Section 5.2] discloses generating a visual layout (visible graph) in which only expanded aggregations reveal their member nodes, such that a collapsed MetaNode is displayed in place of the individual nodes assigned to it, “4) Visible Graph. Only expanded nodes of the processed graph are visible in the expanded structures. A visible graph is generated by means of the module-based edge-pruning algorithm (R1, R3.1)…” [Page 7, Section 5.4] further discloses generating the layout for each aggregation level, “A computational graph is generated by recursively calculating the port–constraint orthogonal layout for each expanded MetaNode’s subgraph (visible graph).” [Page 7, Fig. 10] further confirms that the aggregated visual elements replace the individual node elements, “(c) The main view illustrates visual simplifications of the computational graph. Here, a BERT model is employed. The node and edge numbers are reduced from 4052 and 6353 to 55 and 74, respectively.”)
"and providing the visual layout of the computational graph for display in a user interface at a user device." ([Page 7, Section 5.4] details providing the visual layout for display in a user interface (web-based visual interface), “We implement a web-based visual interface (as shown in Fig. 10), powered by Django and ELK.” [Page 10, Section 6.3.1] discloses that the layout is displayed at a user device (participant’s screen via a browser), “Participants are required to finish the tasks online remotely using a Chrome browser on a screen with a resolution of 1,920 × 1,080.” [Page 7, Section 5.3] further discloses the user interface in which the layout is provided, “The computational graph visualization tool is shown in Fig. 10. The model configuration view (a) allows the user to select a model and displays the hierarchical structure of the model’s computational graph…Otherwise, the simplified computational graph layout (c) is given.”)
Therefore, Claims 1 and 21 are rejected.
With respect to Claim 5:
Pan teaches:
"wherein identifying the hierarchical structure for the computational graph includes performing a search of the computational graph for subgraphs with similar architectures." ([Page 6, Section 5.2.3] details performing a search of the computational graph for subgraphs with similar architectures (isomorphic subgraphs), “An efficient strategy is required to recognize the regularity of the isomorphic subgraphs (R2.1)…Our solution is to search isomorphic subgraphs from the entire computational graph and stack the subgraphs node by node, significantly reducing repetitive elements without sacrificing the intuitiveness of topological structures.” [Page 2, Section 1] discloses that the searched subgraphs include similar as well as identical structures, “an isomorphic subgraph stacking technique that can automatically merge similar or identical graph substructures.” [Page 6, Section 5.2.3] further discloses the result of the search forming a new aggregation of nodes, “Two subgraphs in purple and three subgraphs in yellow are stacked into “piles”. The number on the “pile” node denotes the repeat times of the subgraph it belongs to.”) Examiner’s Note: Each “pile” (PiledNode, Fig. 3 and Fig. 10 legend) aggregates the nodes of multiple similar subgraphs into a single visual element and therefore forms part of the hierarchical aggregation of nodes identified for the graph.
Therefore, Claim 5 is rejected.
With respect to Claim 7:
Pan teaches:
"wherein the hierarchical structure includes a main body of the computational graph, layers of the computational graph, and connections between the layers of the computational graph." ([Page 9, Section 6.2.1] details the hierarchical structure including a main body (top-level MetaNode “Main”) of the computational graph, “The original computational graph is composed of two top-level MetaNodes, namely, Gradients and Main, which are automatically defined by MindSpore. Main denotes the user-defined network structures including optimizers, and Gradient represents the gradient calculation of the parameters from Main.” [Page 4, Section 5.1] discloses the layers of the computational graph, “The CGM employs the cycle-removing algorithm (see Section 5.2.1) and abstracts MetaNodes into various DNN layers. In the CGM, MetaNodes are divided into CNNLayer (Convolutional Neural Network layer), RNNLayer (Recurrent Neural Network layer), FCLayer (Fully Connected layer), and NormalLayer (other layers).” [Page 6, Section 5.2.2] discloses the connections between the layers (modules), “Edges can be categorized into three types (see Fig. 7)… 1) A ModuleEdge starts on the output port of a ModuleNode and ends on the input port of another ModuleNode. 2) A HiddenEdge connects the port of the ModuleNode and the border of its inner NonModuleNode… 3) The other edges are NormalEdges, whose two ends are on the node’s border.”) Examiner’s Note: Main is the forward-propagation body of the network, distinguished from the Gradients (back-propagation) portion.
Therefore, Claim 7 is rejected.
With respect to Claim 8:
Pan teaches:
"wherein processing the computational graph further includes determining a number of nodes in each layer of the computational graph to identify a main body of the computational graph." ([Page 6, Section 5.2.2] details determining a number of nodes in each layer (number of descendants of each MetaNode) to identify modules of the computational graph, “In the first stage, all the MetaNodes are traversed to yield the module information. A MetaNode would be identified as a ModuleNode or “module” (R1.1), if its number of descendants is greater than the user-defined threshold. The module level of a ModuleNode is equivalent to its level of the hierarchy.” [Page 6, Section 5.2.2] further discloses that this count identifies the main body (Main) of the computational graph, “As illustrated in Fig. 7, two top-level nodes named Main and Gradients are recognized as ModuleNodes whose module level equals 1.” [Page 9, Section 6.2.1] confirms that Main is the main body of the network, “Main denotes the user-defined network structures including optimizers.”)
Therefore, Claim 8 is rejected.
With respect to Claim 9:
Pan teaches:
"wherein processing the computational graph further includes analyzing types of nodes in the computational graph to identify different layers in the machine learning model." ([Page 4, Section 5.1] details identifying different layers in the machine learning model (CNNLayer, RNNLayer, FCLayer, NormalLayer) from the nodes of the computational graph, “The CGM employs the cycle-removing algorithm (see Section 5.2.1) and abstracts MetaNodes into various DNN layers. In the CGM, MetaNodes are divided into CNNLayer (Convolutional Neural Network layer), RNNLayer (Recurrent Neural Network layer), FCLayer (Fully Connected layer), and NormalLayer (other layers).” [Page 7, Section 5.2.3] discloses analyzing the types of nodes, “For each node n i N, we establish a string set S ni composed of n i ’s type, neighbors type, parent id, indegree, outdegree, and #(auxiliary nodes).” [Page 7, Fig. 10(b)] further discloses the resulting layer identification, “The concept graph of BERT shows its forward propagation graph structure with the cycles removed. The FC layers like “dense” are simplified.”) Examiner’s Note: Classifying a MetaNode as a CNNLayer, RNNLayer, or FCLayer necessarily depends on the types of operator nodes it contains (convolution operators for a CNNLayer, matrix-multiplication operators for an FCLayer).
Therefore, Claim 9 is rejected.
With respect to Claim 10:
Pan teaches:
"wherein processing the computational graph further includes analyzing a plurality of edges between the plurality of nodes in the computational graph to identify dependencies between the layers." ([Page 5, Section 5.2.1] details analyzing the edges between nodes (traversing edges from DataNodes through OperationNodes) to identify dependencies between layers (directed cycles between MetaNodes), “To detect all cycles, we start from a DataNode which has no input. We traverse all OperationNodes whose input is a DataNode because a DataNode attaches to an OperationNode. In traversing all nodes, we record the parent node of each visited node. If a node n has no inclusion relationship with the last visited one, and n’s parent node has been recorded, then its parent node must be in a cycle.” [Page 6, Section 5.2.2] further discloses analyzing each edge relative to the modules it crosses, “In the next stage, we update all edges based on the identified modules. Edges across modules of different levels that share common line segments are considered for the update.” [Page 7, Section 5.2.3] further discloses encoding each edge by the dependency it represents, “Each edge e i E is encoded as a string s ei =“source type → target type”, where source type represents the type of the source node and target type represents the type of the target node.”) Examiner’s Note: A directed cycle between MetaNodes G1 and G2 (Page 5, Fig. 6) is an identified dependency between layers, which Pan resolves by regrouping so that the displayed layers reflect the true forward-propagation dependencies.
Therefore, Claim 10 is rejected.
With respect to Claim 12:
Pan teaches:
"wherein the visual layout of the computational graph is generated in a hierarchical layout." ([Page 3, Section 2.2] details generating the visual layout in a layered (hierarchical) layout, “Our work adopts the layered layout algorithm to generate an orthogonal style.” [Page 3, Section 2.3] discloses that the layout is applied level by level through the hierarchy, “Our hierarchical directed graph is drawn by using ELK (Eclipse Layout Kernel, an open-source library of layered layout algorithms) recursively in each hierarchy (details are introduced in Section 5.4).” [Page 7, Section 5.4] further discloses, “ELK provides a layer-based layout algorithm that is suited for node–link diagrams with an inherent direction and ports. A computational graph is generated by recursively calculating the port–constraint orthogonal layout for each expanded MetaNode’s subgraph (visible graph).”)
Therefore, Claim 12 is rejected.
With respect to Claim 13:
Pan teaches:
"wherein the visual layout of the computational graph is presented within a debugging tool for the machine learning model." ([Page 1, Abstract] details that the visual layout is presented within a tool for diagnosing (debugging) the machine learning model, “Experimental results and usage scenarios demonstrate that our tool reduces 60% elements on average and hence enhances the performance for recognizing and diagnosing DNN models.” [Page 8, Section 6.2] discloses using the layout for model diagnosis, “the second one illustrates how our approach assists model diagnosis with the simplified computational graph.” [Page 10, Section 6.3.1] further discloses using the layout to locate defects in the model’s code, “DT1 (R1, R3): Check whether the code of DNN training model matches the computational graph…If not, they need to identify where the stop gradient is used in the code of the given computational graph.”) Examiner’s Note: In Section 6.2.2, a developer uses the layout to locate the operator responsible for a performance bottleneck (Conv2 at FP32 precision), corrects it, and retrains the network (Page 10, “he sets Conv2 from FP32 to FP16 and retrains the network”), which is debugging of the machine learning model.
Therefore, Claim 13 is rejected.
With respect to Claim 15:
Pan teaches:
"wherein the machine learning model is a neural network." ([Page 1, Abstract] details that the machine learning model is a neural network (deep neural network), “A computational graph in a deep neural network (DNN) denotes a specific data flow diagram (DFD) composed of many tensors and operators.” [Page 2, Section 1] further discloses representative neural networks, “We examine the efficiency of our solution with representative models like: BERT [1] and ResNet-50 [25].”)
Therefore, Claim 15 is rejected.
With respect to Claim 16:
Pan teaches:
"wherein the neural network includes at least one selected from the following group:
(a) a recurrent neural network;" ([Page 4, Section 5.1] discloses recurrent neural network layers identified within the neural network, “In the CGM, MetaNodes are divided into CNNLayer (Convolutional Neural Network layer), RNNLayer (Recurrent Neural Network layer), FCLayer (Fully Connected layer), and NormalLayer (other layers).”)
"(b) a feedforward neural network;" ([Page 4, Section 5.1] further discloses fully connected (feedforward) layers, “FCLayer (Fully Connected layer).” [Page 9, Section 6.2.2] further discloses a feedforward network, “Another male developer is analyzing the operator time-consumption rankings during the training process of a LeNet network.”)
"(c) a convolutional neural network; or" ([Page 2, Section 1] details convolutional neural networks, “We examine the efficiency of our solution with representative models like: BERT [1] and ResNet-50 [25].” [Page 4, Section 5.1] further discloses, “CNNLayer (Convolutional Neural Network layer).”)
"(d) a transformer neural network." ([Page 6, Section 5.2.3] details a transformer neural network (BERT), “For example, as a typical embedding model for NLP, BERT [1] stacks multiple layers of attention, and its computational graph becomes dominated by repeated isomorphic subgraphs of these layers.”)
Therefore, Claim 16 is rejected.
With respect to Claim 17:
Pan teaches:
"wherein the user interface is included within a web application." ([Page 7, Section 5.4] details the user interface being included within a web application (web-based visual interface), “We implement a web-based visual interface (as shown in Fig. 10), powered by Django and ELK.” [Page 7, Section 6.1] further discloses that the web application is accessed online through a browser, “We compare our approach (the online system is accessible) with existing methods…using the Google Chrome browser.”)
Therefore, Claim 17 is rejected.
With respect to Claim 18:
Pan teaches:
"wherein identifying a hierarchical structure for the computational graph is further based on a domain of the machine learning model." ([pg. 3, sec. 3] discloses machine learning models from different domains (fields), “MindSpore is an open-source framework [27] that allows creating new DNNs or leveraging built-in DNNs for fields, such as computer vision, natural language processing, and graph processing.” [pg. 4, sec. 5.1] discloses identifying the hierarchical structure based on the kind of network the aggregations belong to, “In the CGM, MetaNodes are divided into CNNLayer (Convolutional Neural Network layer), RNNLayer (Recurrent Neural Network layer), FCLayer (Fully Connected layer), and NormalLayer (other layers).” [pg. 4, sec. 4] further discloses that the hierarchy is structured to conform to domain knowledge, “R3.2: The graph layout should conform to users’ domain knowledge.”) Examiner’s Note: Under the broadest reasonable interpretation, a domain of the machine learning model includes the field or type of the model (convolutional networks used for computer vision versus recurrent networks used for sequence data).
Therefore, Claim 18 is rejected.
With respect to Claim 26:
Pan teaches:
"wherein the visual layout further includes respective visual elements for each of one or more individual nodes from the plurality of nodes of the computational graph." ([Page 4, Section 5.1 and Fig. 3] detail respective visual elements (OperationNode glyphs) for individual nodes, “OperationNode refers to nodes of operators…The hierarchy and representations of the nodes are illustrated in Fig. 3.” [Page 5, Fig. 5(d)] discloses the visible graph layout containing individual OperationNodes (Reshape, Select, MatMul, BiasAdd) displayed together with MetaNodes (Main, fc, Gradient1). [Page 7, Fig. 10(c)] further discloses individual nodes displayed alongside aggregated MetaNodes, “(c) The main view illustrates visual simplifications of the computational graph…Hovering on a port highlights its connected edges in orange (c1).”) Examiner’s Note: In Fig. 10(c), individual operation nodes such as ReduceSum, Greater, Select, and Sqrt are rendered as their own visual elements in the same layout as the collapsed MetaNodes networkBertCLS and lossScaling.
Therefore, Claim 26 is rejected.
With respect to Claim 27:
Pan teaches:
"wherein the visual layout further includes visual elements that represent connections between individual nodes or groups of nodes;" ([Page 6, Section 5.2.2] details visual elements (ModuleEdge, HiddenEdge, NormalEdge) that represent connections between individual nodes (NonModuleNodes, OperationNodes) or groups of nodes (ModuleNodes), “1) A ModuleEdge starts on the output port of a ModuleNode and ends on the input port of another ModuleNode. 2) A HiddenEdge connects the port of the ModuleNode and the border of its inner NonModuleNode. A HiddenEdge is represented as a Bézier curve and is invisible by default… 3) The other edges are NormalEdges, whose two ends are on the node’s border.”)
"wherein the visual layout of the computational graph is generated by aggregating at least a portion of the plurality of nodes at different levels of granularity based on architectural features identified in the computational graph." ([Page 6, Section 5.2.2] details aggregating nodes at different levels of granularity (module levels), “The module level of a ModuleNode is equivalent to its level of the hierarchy…The node inside Main named network train is also a ModuleNode with a level-2 port.” [Page 6, Section 5.2.3] discloses aggregating nodes based on architectural features identified in the computational graph (isomorphic structures), “A large amount of identical or similar structures duplicate in many computational graphs. For example, as a typical embedding model for NLP, BERT [1] stacks multiple layers of attention…Our solution is to search isomorphic subgraphs from the entire computational graph and stack the subgraphs node by node.” [Page 7, Section 5.2.3] further discloses that those architectural features include each node’s type, neighbors, and degree, “we establish a string set S ni composed of n i ’s type, neighbors type, parent id, indegree, outdegree, and #(auxiliary nodes).”)
Therefore, Claim 27 is rejected.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
This application currently names joint inventors. In considering patentability of the claims the examiner presumes that the subject matter of the various claims was commonly owned as of the effective filing date of the claimed invention(s) absent any evidence to the contrary. Applicant is advised of the obligation under 37 CFR 1.56 to point out the inventor and effective filing dates of each claim that was not commonly owned as of the effective filing date of the later invention in order for the examiner to consider the applicability of 35 U.S.C. 102(b)(2)(C) for any potential 35 U.S.C. 102(a)(2) prior art against the later invention.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
The factual inquiries for establishing a background for determining obviousness under 35 U.S.C. 103 are summarized as follows:
1. Determining the scope and contents of the prior art.
2. Ascertaining the differences between the prior art and the claims at issue.
3. Resolving the level of ordinary skill in the pertinent art.
4. Considering objective evidence present in the application indicating obviousness or non-obviousness.
Claims 2, 4, 22, and 23 are rejected under 35 U.S.C. 103 as being unpatentable over Pan, (Towards Efficient Visual Simplification of Computational Graphs in Deep Neural Networks, published December 21st, 2022, hereinafter “Pan”), in view of Gao et. Al, (US8935129B1, filed June 4th, hereinafter “Gao”). Both dates are before the effective filing date of the instant application, i.e., April 25th, 2024. Therefore, Pan and Gao constitute prior art under 35 U.S.C. 103.
With respect to Claims 2 and 22:
Pan teaches:
"receiving, from the user interface, a user input that identifies a selection of visual elements from the visual layout of the computational graph---------" ([Page 7, Section 5.3] details receiving, from the user interface, a user input that identifies a selection of visual elements from the visual layout, “The user can freely expand, collapse, and ungroup a MetaNode. Dragging an OperationNode modifies the layout automatically… 2) the connected edges can be highlighted by selecting one or two nodes as endpoints in the graph.”)
Pan does not appear to explicitly disclose:
"-------------for a group of individual nodes or node aggregation layers targeted for aggregation;"
"in response to receiving the user input, assigning the individual nodes or node aggregation layers targeted for aggregation to a new node aggregation layer in the hierarchical structure;"
"and modifying the visual layout of the computational graph by replacing respective visual elements for the individual nodes or node aggregation layers in the group with a new visual element for the new node aggregation layer."
However, Gao teaches:
"-------------for a group of individual nodes or node aggregation layers targeted for aggregation;" ([pg. 24, col. 9] discloses a selection of a group of individual nodes (primitive nodes) or node aggregation layers (group nodes) targeted for aggregation (to be subsumed), “At step 610, a group node is created corresponding to a group of two or more selected nodes (e.g., primitive nodes or group nodes) to be subsumed.” [pg. 22, col. 6] discloses that the selection is made through the user interface of the application displaying the graph, “While in a simplified form, a user may readily access the semantics via a query entered in a user interface of the modeling application 145. Further, if a user edits or changes the simplified form of the graph, via the user interface of the modeling application 145, the such changes may be propagated…”)
"in response to receiving the user input, assigning the individual nodes or node aggregation layers targeted for aggregation to a new node aggregation layer in the hierarchical structure;" ([pg. 24, col. 9] details assigning the selected nodes to a new node aggregation layer (new group node) in the hierarchy in response to receiving the user input, “At step 610, a group node is created corresponding to a group of two or more selected nodes (e.g., primitive nodes or group nodes) to be subsumed. At step 620, each of the selected nodes is added to the set of nodes (‘Node Set’) 430 in the group node’s Reference Object 415. At step 630, any links (e.g., primitive links or group links) that extend between ones of the selected nodes are added to the set of links (‘Link Set’) 435 in the group node’s Reference Object 415.”)
"and modifying the visual layout of the computational graph by replacing respective visual elements for the individual nodes or node aggregation layers in the group with a new visual element for the new node aggregation layer." ([pg. 24, col. 9, FIG. 6, step 680] discloses modifying the graph by removing the visual elements of the selected nodes so that the new group node replaces them, “At step 680, any selected nodes, and any links that exists between a selected node and a non-selected node, are removed from the graph, to create a less-granular (simplified) representation.” [pg. 26, col. 14] further discloses the new visual elements replacing the grouped elements in the displayed graph, “Group nodes (‘Transaction 1’) 1410, (‘Transaction 2’) 1420, (‘Transaction 5’) 1430 have replaced various of the elements.”)
Pan and Gao are analogous art and in the same field of invention because both references pertain to simplifying the display of large node-link graphs by grouping nodes into group nodes of a hierarchy while preserving the connections between them. While Pan teaches a hierarchical clustered computational graph in which the user can expand, collapse, and ungroup MetaNodes, Gao teaches grouping user-selected nodes into a new group node that replaces them in the displayed graph. It would have been obvious to a person having ordinary skill in the art (PHOSITA) before the effective filing date of the claimed invention to implement the base reference of Pan (interactive exploration of MetaNodes of a computational graph) with the teachings of Gao (user-directed grouping of selected nodes into a new group node) in order to allow the user to create new MetaNodes in addition to expanding, collapsing, and ungrouping existing ones. One of ordinary skill in the art would be motivated to do so because Gao’s grouping allows the system “to simplify a graph to a less-granular representation while preserving the graph semantics, and fully restoring the graph to a more-granular representation, {Summary of Gao},” which directly serves Pan’s stated goal of simplifying “less critical elements [to] relieve[] visual cluttering and ambiguity, {Page 4, Section 4 of Pan}.”
Therefore, Claims 2 and 22 are rejected.
With respect to Claim 4:
The combination of Pan-Gao teaches:
"providing the modified visual layout for display in the user interface at the user device." (Pan [Page 7, Section 5.3] details providing the modified visual layout (changed layout) for display in the user interface, “Dragging an OperationNode modifies the layout automatically. A stable transition animation is supported when the layout changes.” Pan [Page 7, Section 5.4] further discloses providing each modified layout through the web-based interface, “We implement a stable transition animation when the layout changes on the basis of the “interactive mode” provided by ELK.”)
Therefore, Claim 4 is rejected.
With respect to Claim 23:
Pan teaches:
"wherein the group of individual nodes or node aggregation layers ------------- include at least one individual node from the plurality of nodes and at least one aggregation layer for at least one sub-group of nodes from the plurality of nodes." ([Page 5, Fig. 5(b)] details an aggregation (MetaNode “Main”) that includes both at least one individual node (OperationNodes “Reshape” and “Const1”) and at least one aggregation layer for a sub-group of nodes (MetaNode “fc” containing MatMul and BiasAdd). [Page 6, Section 5.2.2] further discloses aggregations containing both nested aggregations and individual nodes, “The node inside Main named network train is also a ModuleNode with a level-2 port…the child node softmax of the network train, which is a NonModuleNode…”)
Pan does not appear to explicitly disclose:
"------targeted for aggregation-----"
However, Gao teaches:
"------targeted for aggregation-----" ([pg. 22, col. 5] discloses that the group targeted for aggregation (selected nodes to be subsumed) may include both individual nodes (primitive nodes) and aggregation layers (other groups), “Groups may be discrete collections of primitive nodes, or may nest among each other, such that a group may include other groups, as well as individual primitive nodes.” [FIG. 6, step 610] further discloses, “a group node is created corresponding to a group of two or more selected nodes (e.g., primitive nodes or group nodes) to be subsumed.”)
Therefore, Claim 23 is rejected.
Claim 3 is rejected under 35 U.S.C. 103 as being unpatentable over Pan in view of Gao, further in view of Song et. Al, (Interactive Visual Pattern Search on Graph Data via Graph Representation Learning, published February 18th, 2022, hereinafter “Song”).
With respect to Claim 3:
The combination of Pan-Gao does not appear to explicitly disclose:
"conditioning a one-shot machine learning model on the group of individual nodes or node aggregation layers targeted for aggregation;"
"and processing the computational graph with the one-shot machine learning model to assign the individual node or node aggregation layers targeted for aggregation to the new node aggregation layer in the hierarchical structure."
However, Song teaches:
"conditioning a one-shot machine learning model on the group of individual nodes or node aggregation layers targeted for aggregation;" ([pg. 6, Section 4.2.1] details conditioning a machine learning model on a single user-selected group of nodes (query pattern selected from a graph instance), “In this panel, the user can quickly select a group of nodes and the subgraph they induce by brushing a rectangular area on the visualization. They can also construct the pattern in a more precise manner by clicking the + and - button on the top right corner of each node.” [pg. 1] discloses that the model operates from that one example (example-based search), “We propose a visual analytics system GraphQ to support human-in-the-loop, example-based, subgraph pattern search in a database containing many individual graphs.”) Examiner’s Note: The user provides one example by brushing a single group of nodes in the displayed graph (sec. 3.2.1) and that example becomes the query. Song’s GNNs (NeuroMatch and NeuroAlign) are trained before the user makes any selection and they find every subgraph that matches the new query without any further training. The one selected example therefore conditions the model, and is the only information about the pattern the model receives (one-shot machine learning).
"and processing the computational graph with the one-shot machine learning model to assign the individual node or node aggregation layers targeted for aggregation to the new node aggregation layer in the hierarchical structure." ([pg. 3, sec. 3.3] details processing the graph with the machine learning model to locate every occurrence of the selected pattern, “NeuroMatch resolves this issue by decomposing the given query and target graphs into many small regions and learns the subgraph relationship in these small regions first.” [pg. 4, sec. 3.4] further discloses identifying, node by node, which nodes of the graph correspond to the selected pattern, “NeuroAlign directly takes the node embeddings obtained from GNN on the entire graphs and…computes the similarity between each query embedding and every target embeddings through an attention network.” [pg. 6, Section 4.2.1] discloses that the matched nodes are marked in the displayed graph, “The matching nodes are highlighted based on the results returned by the node alignment module.”) Examiner’s Note: Gao teaches assigning the selected nodes to the new group node (steps 610-630), and Pan teaches assigning every matching isomorphic subgraph to a common aggregated element (Page 6, Section 5.2.3, “stack the subgraphs node by node”). Song supplies the learned model that identifies the selected pattern and its matches for that assignment.
Pan, Gao, and Song are analogous art and in the same field of invention because all three references pertain to interactive visual analysis of node-link graphs in which structural patterns of nodes are identified and grouped. While Pan teaches stacking repeated isomorphic subgraphs of a computational graph and Gao teaches grouping user-selected nodes into a new group node, Song teaches a pre-trained graph neural network that is conditioned on a single user-selected pattern and locates its occurrences in graphs such as program control-flow graphs. It would have been obvious to a person having ordinary skill in the art (PHOSITA) before the effective filing date of the claimed invention to implement the base reference of Pan (isomorphic subgraph stacking) with the teachings of Gao (user-directed grouping) and the teachings of Song (example-conditioned neural subgraph matching) in order to assign the user-selected group, and structures matching it, to the new node aggregation layer. One of ordinary skill in the art would be motivated to do so because, “Identifying subgraph patterns in graphs is an important approach to understand their structural properties, {Abstract of Song},” and because Song’s model reports “19%–29% improvement in node-alignment accuracy compared to baseline, {NeuroAlign section of Song}.”
Therefore, Claim 3 is rejected.
Claim 6 is rejected under 35 U.S.C. 103 as being unpatentable over Pan, (Towards Efficient Visual Simplification of Computational Graphs in Deep Neural Networks, published December 21st, 2022, hereinafter “Pan”), in view of Song et. Al, (Interactive Visual Pattern Search on Graph Data via Graph Representation Learning, published February 18th, 2022, hereinafter “Song”).
With respect to Claim 6:
Pan teaches:
"wherein identifying the hierarchical structure for the computational graph includes processing the computational graph ---------------- to identify subgraphs with similar architectures." ([Page 6, Section 5.2.3] details processing the computational graph to identify subgraphs with similar architectures (isomorphic subgraphs), “Our solution is to search isomorphic subgraphs from the entire computational graph and stack the subgraphs node by node.” [Page 7, Section 5.2.3] further discloses, “To effectively recognize and stack isomorphic subgraphs, we design a hash-based algorithm to represent each detected subgraph.”)
Pan does not appear to explicitly disclose:
"----------with a one-shot machine learning model------------"
However, Song teaches:
"----------with a one-shot machine learning model------------" ([Section 4.2.1] and the [Abstract] detail a machine learning model conditioned on a single example pattern (example-based subgraph pattern search), “In this panel, the user can quickly select a group of nodes and the subgraph they induce by brushing a rectangular area on the visualization,” and, “We propose a visual analytics system GraphQ to support human-in-the-loop, example-based, subgraph pattern search.” [pg. 3, sec. 3.3] discloses processing the graph with the model to identify subgraphs matching that example, “NeuroMatch resolves this issue by decomposing the given query and target graphs into many small regions and learns the subgraph relationship in these small regions first.”) See the Examiner’s Note for Claim 3 regarding the one-shot character of Song’s pre-trained, example-conditioned model.
Pan and Song are analogous art and in the same field of invention because both references pertain to identifying repeated structural patterns in graphs for visual analysis. While Pan teaches a hash-based search for isomorphic subgraphs in a computational graph, Song teaches a pre-trained graph neural network that identifies subgraphs matching an example pattern. It would have been obvious to a person having ordinary skill in the art (PHOSITA) before the effective filing date of the claimed invention to implement the base reference of Pan (isomorphic subgraph search) with the teachings of Song (example-conditioned neural subgraph matching) in order to identify structurally similar subgraphs that are not exact hash matches. One of ordinary skill in the art would be motivated to do so because Song’s approach achieves “up to 100x speedup compared to combinatorial algorithms, {Abstract of GraphQ}.”
Therefore, Claim 6 is rejected.
Claims 11 and 25 are rejected under 35 U.S.C. 103 as being unpatentable over Pan, in view of Gao, further in view of Bauerle et. Al, (Net2Vis – A Visual Grammar for Automatically Generating Publication-Tailored CNN Architecture Visualizations, published February 10th, 2021, hereinafter “Bauerle”).
With respect to Claim 11:
The combination of Pan-Gao does not appear to explicitly disclose:
"wherein the user input further includes a tag labeling the group of individual nodes or node aggregation layers targeted for aggregation and the new visual element for the new node aggregation layer is labeled with the tag."
However, Bauerle teaches:
"wherein the user input further includes a tag labeling the group of individual nodes or node aggregation layers targeted for aggregation and the new visual element for the new node aggregation layer is labeled with the tag." ([pg. 5, section 4.3] details labeling each aggregation of selected layers and resolving its identity for the viewer, “we adopt the paradigm of providing ways to aggregate multiple layers and replacing them with a new, single glyph. As this removes direct insight into the content of aggregations, we resolve them in a legend below the network graph.” [pg. 7, section 5.1] further discloses that each glyph is labeled with its name, “This legend contains a glyph for each layer type in the network and displays the name of its layer.”) Examiner’s Note: Gao likewise displays each new group node with a label (“Group nodes (‘Transaction 1’) 1410, (‘Transaction 2’) 1420, (‘Transaction 5’) 1430 have replaced various of the elements”), and Pan’s aggregations carry user-defined namespace names. Accepting the label as part of the same user input that selects the group is a predictable use of these known labeling techniques.
Pan, Gao, and Bauerle are analogous art and in the same field of invention because all three references pertain to simplifying the display of graphs by replacing groups of nodes with single, labeled visual elements. While Pan teaches user-defined namespace labels on MetaNodes and Gao teaches labeled group nodes created from user-selected nodes, Bauerle teaches replacing selected layer sequences with a single glyph whose name is resolved for the viewer. It would have been obvious to a person having ordinary skill in the art (PHOSITA) before the effective filing date of the claimed invention to implement the base reference of Pan with the teachings of Gao and Bauerle in order to label the new node aggregation layer with a tag supplied with the user’s grouping input. One of ordinary skill in the art would be motivated to do so because an aggregation “removes direct insight into the content of aggregations, {section 4.3 of Bauerle},” and a label restores that insight.
Therefore, Claim 11 is rejected.
With respect to Claim 25:
Pan mostly teaches:
"and search the computational graph for other occurrences of the pattern, and automatically create additional aggregation layers for nodes or layers that match the pattern." ([Page 6, section 5.2.3] details searching the entire computational graph for occurrences of a pattern (isomorphic subgraph) and automatically creating additional aggregation layers (piles) for the matching nodes, “Our solution is to search isomorphic subgraphs from the entire computational graph and stack the subgraphs node by node, significantly reducing repetitive elements…Two subgraphs in purple and three subgraphs in yellow are stacked into “piles”.” [page 2, section 1] further discloses that the aggregation is automatic, “an isomorphic subgraph stacking technique that can automatically merge similar or identical graph substructures.”)
The combination of Pan-Gao does not appear to explicitly disclose:
"identify a pattern associated with the group of individual nodes or node aggregation layers targeted for aggregation;"
However, Bauerle teaches:
"identify a pattern associated with the group of individual nodes or node aggregation layers targeted for aggregation;" ([Section 4.3] details identifying the pattern (layer sequence) associated with the group selected for aggregation and applying it across the entire graph, “As aggregations substitute all occurrences of selected layer sequences throughout the graph, only parts of the network graph that can be represented by one sequential layer can be aggregated.” [Section 4.3] further discloses searching for recurring occurrences of such patterns, “To obtain automatic aggregations, we propose to analyze all sequential parts of the network. We then search for recurring sequences of layers.”)
"search the computational graph for other occurrences of the pattern, and automatically create additional aggregation layers for nodes or layers that match the pattern." ([Section 4.3] details automatically substituting every other occurrence of the selected pattern with an aggregation, “As aggregations substitute all occurrences of selected layer sequences throughout the graph…” and, “we adopt the paradigm of providing ways to aggregate multiple layers and replacing them with a new, single glyph.”) Examiner’s Note: In Bauerle, a single selection of a layer sequence causes every matching occurrence in the network to be replaced by the aggregation glyph.
Pan, Gao, and Bauerle are analogous art and in the same field of invention because all three references pertain to simplifying the display of graphs by replacing groups of nodes with single visual elements. While Pan teaches automatically stacking all occurrences of isomorphic subgraphs and Gao teaches grouping user-selected nodes, Bauerle teaches applying an aggregation defined by a selected layer sequence to every occurrence of that sequence in the network. It would have been obvious to a person having ordinary skill in the art (PHOSITA) before the effective filing date of the claimed invention to implement the base reference of Pan with the teachings of Gao and Bauerle in order to seed Pan’s isomorphic search with the pattern of the user’s selected group. One of ordinary skill in the art would be motivated to do so because, “As modern networks become increasingly complex, the aggregation of layers is inevitable, {Section 4.3 of Bauerle}.”
Therefore, Claim 25 is rejected.
Claim 14 is rejected under 35 U.S.C. 103 as being unpatentable over Pan in view of Bauerle et. Al, (Net2Vis – A Visual Grammar for Automatically Generating Publication-Tailored CNN Architecture Visualizations, published February 10th, 2021, hereinafter “Bauerle”).
With respect to Claim 14:
Pan does not appear to explicitly disclose:
"providing an input to the machine learning model to generate an output, wherein the visual layout of the computational graph includes an input visual element of the input and an output visual element of the output at corresponding portions of the visual layout of the computational graph."
However, Bauerle teaches:
"providing an input to the machine learning model to generate an output, wherein the visual layout of the computational graph includes an input visual element of the input and an output visual element of the output at corresponding portions of the visual layout of the computational graph." ([pg. 6 sec. 4.5] discloses including an input visual element (input sample) and an output visual element (output sample) of the model at corresponding portions (left and right ends) of the visual layout, “As described in the property analysis of Section III, input and output samples may, for some networks, be helpful. Directly integrating such samples would, however, require the user to provide training or testing data…Instead, we suggest to optionally provide placeholders for input and output samples which users can replace during post-processing with actual samples.” [Fig. 1] further discloses the placement of those elements, “Dashed boxes on the left and right serve as placeholders to provide input and output samples.”) Examiner’s Note: An “actual” output sample of a network is the output generated by providing the corresponding input sample (“training or testing data”) to the network; displaying the actual input and output samples at the input and output ends of the layout therefore requires providing the input to the model to generate the output.
Pan and Bauerle are analogous art and in the same field of invention because both references pertain to automatically generating visualizations of neural network architectures. While Pan teaches a forward-propagation layout of a computational graph from input layer to output layer, Bauerle teaches displaying input and output samples of the network at the corresponding ends of the architecture visualization. It would have been obvious to a person having ordinary skill in the art (PHOSITA) before the effective filing date of the claimed invention to implement the base reference of Pan with the teachings of Bauerle in order to present the model’s computation in the context of a concrete input and the output it produces. One of ordinary skill in the art would be motivated to do so because, “input and output samples may, for some networks, be helpful, {sec 4.5 of Bauerle}.”
Therefore, Claim 14 is rejected.
Claim 19 is rejected under 35 U.S.C. 103 as being unpatentable over Pan in view of Kunal et. Al, (GANA: Graph Convolutional Network Based Automated Netlist Annotation for Analog Circuits, published March 13th, 2020, hereinafter “Kunal”).
With respect to Claim 19:
Pan teaches:
“-------on training example machine learning models----------" ([pg. 3, sec. 3] discloses machine learning models whose computational graphs carry labeled hierarchical structures (namespaces), “Namespaces are used by TensorFlow [59] and MindSpore [27] to yield a hierarchically structured graph representation. The namespaces are generated automatically or in a user-defined manner.” [Page 5, Fig. 5(a)] discloses the hierarchy labels on each node of such a model (e.g., “Main / Const1,” “Main / fc / MatMul,” “Gradient / Reshape1”).) Examiner’s Note: Each namespace-labeled MindSpore or TensorFlow model graph of Pan is an example machine learning model with a labeled hierarchical structure. Pan also identifies the need to recover hierarchy where it is not available).
Pan does not appear to explicitly disclose:
"wherein a neural network is used to identify a hierarchical structure for the computational graph, wherein the neural network is trained-------on training example machine learning models----------with labeled hierarchical structures."
However, Kunal teaches:
"wherein a neural network is used to identify a hierarchical structure for the computational graph," ([Abstract] discloses using a trained neural network (graph convolutional network) to identify a hierarchical structure (hierarchical representation) of a graph of interconnected computational elements (netlist), “Automated subcircuit identification and annotation enables the creation of hierarchical representations of analog netlists…The novel approach in this paper is based on the use of a trained graph convolutional neural network (GCN).” [pg. 56, section II.B] discloses applying the network to a flattened graph to identify its hierarchy, “GCN-based recognition: We create a graph representation of the flattened netlist and identify sub-blocks using a GCN-based approach… Primitive annotation: We take each sub-block and recursively identify lower-level blocks primitives within it.” [pg. 56, section II.A] further discloses the resulting hierarchy, “Our recognition algorithm processes primitives and sub-blocks hierarchically to recognize an entire circuit of this type, and build its hierarchy tree.”)
"wherein the neural network is trained----------------with labeled hierarchical structures." ([pg. 58, section V.A] details training the neural network on training examples labeled with their hierarchical structure (sub-block labels), “two labeled datasets for OTA and radio frequency (RF) sub-blocks have been created with characteristics as listed in Table I…The OTA bias dataset consists of multiple OTA configuration with appropriate signal and bias subcircuit labels, while the RF data dataset consists of different RF circuits, with labels attached to elements.” [pg. 60, section V.B] further discloses applying the trained network to new graphs not in the training set, “The architectures that combine LNAs, mixers, and oscillators used in this set are similar to some architectures in the RF_data training set, but the precise circuits are distinct, with no repetitions from the training set.”) Examiner’s Note: A circuit netlist, like a computational graph, is a directed graph of computational elements whose edges carry signal (data) flow, and Kunal’s task of recovering the block hierarchy of a flattened netlist is the same task as recovering the module hierarchy of a flattened model graph. Pan supplies the training examples that are machine learning models (namespace-labeled model graphs), and Kunal supplies the neural network trained on hierarchy-labeled examples.
Pan and Kunal are analogous art because both references pertain to recognizing the hierarchical block structure of a large graph of interconnected computational elements in order to present it at multiple levels of abstraction as well as identifying the hierarchy of a graph whose hierarchy is not directly available. While Pan teaches a hierarchical clustered computational graph built from namespace-labeled models, Kunal teaches training a graph neural network on hierarchy-labeled examples and applying it to recover the hierarchy of unseen flattened graphs. It would have been obvious to a person having ordinary skill in the art (PHOSITA) before the effective filing date of the claimed invention to implement the base reference of Pan (hierarchical clustered computational graphs of namespace-labeled models) with the teachings of Kunal (neural network trained on hierarchy-labeled graphs to recover hierarchy) in order to train a neural network on Pan’s namespace-labeled model graphs and use it to identify the hierarchical structure of model graphs lacking namespace information. One of ordinary skill in the art would be motivated to do so because Pan recognizes that for dynamically constructed graphs “The computational graph cannot easily represent the model structures, {Page 3, Section 3 of Pan},” and Kunal teaches that automated identification “enables the creation of hierarchical representations…and can facilitate a variety of design automation tasks, {Abstract of Kunal}.”
Therefore, Claim 19 is rejected.
Claim 20 is rejected under 35 U.S.C. 103 as being unpatentable over Pan in view of Jonyer et. Al (Graph-Based Hierarchical Conceptual Clustering, published May 2000, hereinafter “Jonyer”).
With respect to Claim 20:
Pan teaches:
"wherein identifying a hierarchical structure for the computational graph includes iteratively aggregating the plurality of nodes----------" ([pg. 3, sec. 3] details a recursive (iterative) aggregation of the plurality of nodes into levels, “The hierarchy of a graph is defined as a recursive grouping placed on the nodes of the initial graph [58]…We represent a hierarchical computational graph as H0 = (H1 , E1 ), where H i = (H i+1 , E i+1 )…H i represents a set of nodes at the i-th level of the hierarchy.” [pg. 6, sec. 5.2.3] further discloses iterative aggregation passes, “After several rounds of traversals, a clear majority of isomorphic subgraphs in the graph are reached.”)
Pan does not appear to explicitly disclose:
"---------from a high level of granularity to a low level of granularity."
However, Jonyer teaches:
"---------from a high level of granularity to a low level of granularity." ([pg. 2] discloses iteratively aggregating the nodes starting from the most granular data (instances of substructures in the input graph) and proceeding to progressively coarser levels, “This discovery process continues iteratively until all possible subgraphs have been considered, or the algorithm reaches a user specified limit. After the best substructure is found and the graph is compressed, the process starts all over, finding another best substructure. This search is guided by the Minimum Description Length (MDL) principle, originally developed by Rissanen [Rissanen 1989]. According to the evaluation heuristic, the best substructure is the one that minimizes the description length of the graph when compressed by the substructure. When compressing the graph, all instances of the substructure are replaced by a single vertex, which is a pointer to the substructure’s definition.”) Examiner’s Note: Each iteration of Jonyer operates on the graph as compressed by the previous iteration, so the aggregation proceeds from the finest (highest granularity) level of the input graph to successively coarser (lower granularity) levels.
Pan and Jonyer are analogous art and in the same field of invention because both references pertain to simplifying large graphs by discovering repeated substructures and abstracting them into a hierarchy of aggregated nodes. While Pan teaches a recursive hierarchy of MetaNodes over a computational graph, Jonyer teaches building that hierarchy bottom-up by iteratively replacing discovered substructures with single nodes. It would have been obvious to a person having ordinary skill in the art (PHOSITA) before the effective filing date of the claimed invention to implement the base reference of Pan (recursive hierarchical clustered computational graph) with the teachings of Jonyer (iterative bottom-up substructure compression) in order to construct Pan’s levels H i from the finest level upward and in order to establish a hierarchy that provides varying levels of interpretation that can be accessed based on the specific data analysis goals.
Therefore, Claim 20 is rejected
Claim 24 is rejected under 35 U.S.C. 103 as being unpatentable over Pan in view of Ying et. Al, (Representation Learning for Frequent Subgraph Mining”, published February 22nd, 2024, hereinafter “Ying”).
With respect to Claim 24:
Pan teaches:
"wherein the hierarchical structure includes a first node aggregation layer that defines a collection of nodes aggregated without user input----------" ([pg. 9, Section 6.2.1] details node aggregation layers (top-level MetaNodes) defining collections of nodes aggregated without user input (automatically defined), “The original computational graph is composed of two top-level MetaNodes, namely, Gradients and Main, which are automatically defined by MindSpore.” [pg.2, section 1] further discloses aggregation layers created automatically from repeated structures, “an isomorphic subgraph stacking technique that can automatically merge similar or identical graph substructures.” [Page 6, Section 5.2.3] discloses the resulting aggregation, “Two subgraphs in purple and three subgraphs in yellow are stacked into “piles”.”)
Pan does not appear to explicitly disclose:
"----------based on a zero-shot machine learning model."
However, Ying teaches:
"----------based on a zero-shot machine learning model." ([pg. 7, section 3.6] discloses a zero-shot machine learning model (pretrained graph neural network applied to unseen graphs without fine-tuning), “The SPMiner GNN needs to be trained only once and can be applied to any input graph. In fact, we train the SPMiner GNN on a set of synthetic graphs. We generate millions of synthetic training datasets to learn a general order embedding space agnostic to the dataset domain. This pretrained GNN model can then be immediately applied to real-world datasets not seen in training without further fine-tuning.” [pg. 1] discloses that the model identifies, without user input, collections of nodes forming repeated structures, “SPMiner combines graph neural networks, order embedding space, and an efficient search strategy to identify network subgraph patterns that appear most frequently in the target graph.”) Examiner’s Note: A model trained once on synthetic graphs and then applied to target graphs and without any user-provided example, is a zero-shot machine learning model under the broadest reasonable interpretation.
Pan and Ying are analogous art and in the same field of invention because both references pertain to automatically identifying repeated subgraph structures in large graphs. While Pan teaches automatically stacking repeated isomorphic subgraphs of a computational graph into aggregated piles using a hash-based algorithm, Ying teaches a pretrained graph neural network that identifies the most frequent subgraph patterns in an unseen target graph. It would have been obvious to a person having ordinary skill in the art (PHOSITA) before the effective filing date of the claimed invention to implement the base reference of Pan (automatic aggregation of repeated structures) with the teachings of Ying (zero-shot neural frequent subgraph mining) in order to identify the repeated structures that define Pan’s aggregation layers with a model that generalizes to new graphs. One of ordinary skill in the art would be motivated to do so because exhaustive search is costly, “finding large commonly-occurring motifs remains a challenging problem not only due to its NP-hard subroutine of subgraph counting, but also the exponential growth of the number of possible subgraphs patterns, {Abstract of Ying},” and because the pretrained model “can then be immediately applied to real-world datasets not seen in training without further fine-tuning, {Section 3.6 of Ying}.”
Therefore, Claim 24 is rejected.
Conclusion
The prior art made of record and not relied upon is considered pertinent to applicant's disclosure: Xing et. Al (US11860977B1: Hierarchical graph neural networks for visual clustering, Filed May 04, 2021).
Any inquiry concerning this communication or earlier communications from the examiner should be directed to NOOR F CHEEMA whose telephone number is (571)272-9642. The examiner can normally be reached Monday-Friday 7:30am-5:00pm alternative Fridays off.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Mariela Reyes can be reached at (571) 270-1006. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/N.F.C./ Examiner, Art Unit 2142
/Mariela Reyes/ Supervisory Patent Examiner, Art Unit 2142
Prosecution Insights