DETAILED ACTION
This Office action is responsive to the following communication: Application filed on 24 April 2025.
Claim(s) 1-15 is/are pending and present for examination. Claim(s) 1, 12, and 14 is/are in independent form.
Notice of Pre-AIA or AIA Status
The present application, filed on or after March 16, 2013, is being examined under the first inventor to file provisions of the AIA .
Information Disclosure Statement
The information disclosure statement (IDS) submitted on 24 April 2025 is being considered by the examiner.
Drawings
The drawings were received on 24 April 2025. These drawings are accepted.
Claim Rejections - 35 USC § 102
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of the appropriate paragraphs of 35 U.S.C. 102 that form the basis for the rejections under this section made in this Office action:
A person shall be entitled to a patent unless –
(a)(2) the claimed invention was described in a patent issued under section 151, or in an application for patent published or deemed published under section 122(b), in which the patent or application, as the case may be, names another inventor and was effectively filed before the effective filing date of the claimed invention.
Claim(s) 1, 3, 4, 7, and 12-15 is/are rejected under 35 U.S.C. 102(a)(2) as being anticipated by Ramanasankaran et al (hereinafter referred to as “Ram”), USPGPUB No. 2024/0345551, filed on 22 January 2024, claiming priority to 12 April 2023, and published on 17 October 2024.
As per independent claims 1, 12, and 14, Ram teaches:
A computer system for processing construction data, the computer system comprising one or more processors
receiving{See Ram, [0154], wherein this reads over “By pulling only relevant nodes and/or edges, the fine-tuned graph query completion model allows for the just-in-time graph querying approach shown in FIG. 11 to attach a more specific and smaller set of knowledge to the user prompt before applying the user prompt to the completion model, at step 916. Thus, the workflow 1100 allows for the system to provide accurate responses to user prompts based on relevant nodes and/or edges and associated enrichment data, while requiring less overall data processing and analysis compared to the full ahead-of-time training process discussed above, with respect to FIGS. 9 and 10, thereby reducing computational burden placed on the system, as well as system power consumption and time-to-compute durations.”};
mapping the query to a set of entity relations for the query, using a first generator model{See Ram, [0151], wherein this reads over “The fine-tuned graph query completion model can produce a graph query (e.g., a SPARQL query) configured to pull nodes and/or edges from a knowledge graph that are relevant to the user prompt (e.g., relevant to a question indicated within the user prompt).”};
retrieving from the knowledge graph a subgraph, using the set of entity relations for the query {See Ram, [0151], wherein this reads over “Once the graph query has been generated, the graph query can be applied to a graph query runner, at step 1104, to pull the relevant nodes and/or edges from the knowledge graph obtained at step 902 to generate a relevant knowledge subgraph, at step 1106.”};
mapping{See Ram, [0153], wherein this reads over “Once the knowledge subgraph has been generated, the nodes and edges within the knowledge subgraph can be grouped and enriched, at step 1002, and translated into natural text prompts, at step 904, as discussed above, with respect to FIGS. 9 and 10. However, in the workflow 1100, the natural text prompts generated based on the relevant knowledge subgraph can be used in place of identifying the top related information from the stored embedded vectors within a similarity search database.”}, wherein the second generator model is inverse to the first generator model, and the first generator model and the second generator model are trained for cycle consistency {See Ram, [0072], wherein this reads over “For example, the second model 116 can be used to process unstructured information regarding items of equipment into predefined template formats compatible with various third models, such that outputs of the second model 116 can be provided as inputs to the third models; this can allow more accurate training of the third models, more training data to be generated for the third models, and/or more data available for use by the third models. The second model 116 can receive inputs from one or more third models, which can provide greater data to the second model 116 for processing”},
whereby structured entity relations output by the first generator model are input to the second generator model, and unstructured data output by the second generator model is input to the first generator model {See Ram, [0072], wherein this reads over “For example, the second model 116 can be used to process unstructured information regarding items of equipment into predefined template formats compatible with various third models, such that outputs of the second model 116 can be provided as inputs to the third models; this can allow more accurate training of the third models, more training data to be generated for the third models, and/or more data available for use by the third models. The second model 116 can receive inputs from one or more third models, which can provide greater data to the second model 116 for processing”}; and
providing to the user the unstructured data output for the query {See Ram, [0154], wherein this reads over “Thus, the workflow 1100 allows for the system to provide accurate responses to user prompts based on relevant nodes and/or edges and associated enrichment data, while requiring less overall data processing and analysis compared to the full ahead-of-time training process discussed above, with respect to FIGS. 9 and 10, thereby reducing computational burden placed on the system, as well as system power consumption and time-to-compute durations.”}.
As per dependent claims 3, 13, and 15, Ram teaches:
The computer system of claim 1, wherein the query comprises query input with at least one of: words of natural language, images, floor plans, architectural drawings, technical drawings, time-dependent graphs, measurement data, audio recordings, or video recordings, and the one or more processors are configured to map the query input to a sequence of multimodal tokens, to use the first generator model to map the sequence of multimodal tokens to a set of knowledge graph triples defining the entity relations, and to use the second generator model to map the subgraph to a sequence of multimodal tokens defining the unstructured data output for the query, the unstructured data output for the query comprising at least one of: words of natural language, images, floor plans, architectural drawings, technical drawings, time-dependent graphs, audio files or video files {See Ram, [0041], wherein this reads over “The system 100 can determine relations between data from different sources, such as by using timeseries information and identifiers of the sites or buildings at which items of equipment are present to detect relationships between various different data relating to the items of equipment (e.g., to train the models 104, 116 using both timeseries data (e.g., sensor data; outputs of algorithms or models, etc.) regarding a given item of equipment and freeform natural language reports regarding the given item of equipment)”; and [0043], wherein this reads over “The data can be of any of a plurality of formats (e.g., text, speech, audio, image, video, etc.), including multi-modal formats. For example, the data may be received from service technicians in forms such as text (e.g., laptop/desktop or mobile application text entry), audio, and/or video (e.g., dictating findings while capturing video).”}.
As per dependent claim 4, Ram teaches:
The computer system of claim 1,wherein the one or more processors are configured to denote each of the entities and the relations in the knowledge graph with a unique token sequence {See Ram, [0034], wherein this reads over “For example, the first model 104 can include at least one GPT model. The GPT model can receive an input sequence, and can parse the input sequence to determine a sequence of tokens (e.g., words or other semantic units of the input sequence, such as by using Byte Pair Encoding tokenization). The GPT model can include or be coupled with a vocabulary of tokens, which can be represented as a one-hot encoding vector, where each token of the vocabulary has a corresponding index in the encoding vector; as such, the GPT model can convert the input sequence into a modified input sequence, such as by applying an embedding matrix to the token tokens of the input sequence (e.g., using a neural network embedding function), and/or applying positional encoding (e.g., sin-cosine positional encoding) to the tokens of the input sequence.”}.
As per dependent claim 7, Ram teaches:
The computer system of claim 1,wherein the one or more processors are configured to train the first generator model and the second generator model using positive reference data, including at least one of: truthful entity relations or truthful unstructured reference data, negative reference data, including at least one of: false entity relations or false unstructured reference data, and anchor data including pairs of truthful entity relations matched with corresponding truthful unstructured reference data {See Ram, [0068], wherein this reads over “The feedback repository 124 can include feedback received from users regarding output presented by the applications 120. For example, for at least a subset of outputs presented by the applications 120, the applications 120 can present one or more user input elements for receiving feedback regarding the outputs. The user input elements can include, for example, indications of binary feedback regarding the outputs (e.g., good/bad feedback; feedback indicating the outputs do or do not meet the user's criteria, such as criteria regarding technical accuracy or precision); indications of multiple levels of feedback (e.g., scoring the outputs on a predetermined scale, such as a 1-5 scale or 1-10 scale); freeform feedback (e.g., text or audio feedback); or various combinations thereof.”}.
Claim Rejections - 35 USC § 103
In the event the determination of the status of the application as subject to AIA 35 U.S.C. 102 and 103 (or as subject to pre-AIA 35 U.S.C. 102 and 103) is incorrect, any correction of the statutory basis (i.e., changing from AIA to pre-AIA ) for the rejection will not be considered a new ground of rejection if the prior art relied upon, and the rationale supporting the rejection, would be the same under either status.
The following is a quotation of 35 U.S.C. 103 which forms the basis for all obviousness rejections set forth in this Office action:
A patent for a claimed invention may not be obtained, notwithstanding that the claimed invention is not identically disclosed as set forth in section 102, if the differences between the claimed invention and the prior art are such that the claimed invention as a whole would have been obvious before the effective filing date of the claimed invention to a person having ordinary skill in the art to which the claimed invention pertains. Patentability shall not be negated by the manner in which the invention was made.
Claim(s) 2 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ram, in view of Gan et al, USPGPUB No. 2025/0181937, filed on 3 December 2024, and published on 5 June 2025.
As per dependent claim 2, Ram, in combination with Gan, discloses:
The computer system of claim 1, wherein the query comprises natural language input, and the one or more processors are configured to map the natural language input to a sequence of tokens {See Ram, [0141], wherein this reads over “In some implementations, prior to training the machine learning model, the natural language prompts can be converted into vectors to be indexed and stored within a vector storage and similarity search database. For example, in some instances, the natural language text may be provided to an embedding model to convert the natural language text into indexable vectors that can be easily utilized by the machine learning models described herein. The embedding model can receive the natural language prompts (and/or one or more tokens thereof as generated by a tokenizer, such as a byte pair encoding tokenizer, etc.), and apply the natural language prompts as input to the embedding model to cause the embedding model to generate respective vectors representing the natural language prompts in an n-dimensional space.”}, to use the first generator model to map the sequence of tokens to a set of knowledge graph triples defining the entity relations {See Gan, [0060], wherein this reads over “In this embodiment, the semantic relatedness score is used to measure trustworthiness of semantic relatedness between the candidate entity word and the source entity in the corresponding relation. In an example, a knowledge triple can be formed by using the source entity, the corresponding relation, and the candidate entity word, and then the corresponding semantic relatedness score can be obtained by using various knowledge graph triple trustworthiness measurement methods.”}, and to use the second generator model to map the subgraph to a sequence of tokens defining natural language output for the query {See Ram, [0034], wherein this reads over “For example, the first model 104 can include at least one GPT model. The GPT model can receive an input sequence, and can parse the input sequence to determine a sequence of tokens (e.g., words or other semantic units of the input sequence, such as by using Byte Pair Encoding tokenization). The GPT model can include or be coupled with a vocabulary of tokens, which can be represented as a one-hot encoding vector, where each token of the vocabulary has a corresponding index in the encoding vector; as such, the GPT model can convert the input sequence into a modified input sequence, such as by applying an embedding matrix to the token tokens of the input sequence (e.g., using a neural network embedding function), and/or applying positional encoding (e.g., sin-cosine positional encoding) to the tokens of the input sequence.”}.
Ram fails to expressly disclose the features of “use the first generator model to map the sequence of tokens to a set of knowledge graph triples defining the entity relations.” Gan is directed to the invention of a large language model-based knowledge method. Specifically, Gan discloses that “a knowledge triple can be formed by using the source entity, the corresponding relation, and the candidate entity word, and then the corresponding semantic relatedness score can be obtained by using various knowledge graph triple trustworthiness measurement methods.” See Gan, [0060]. That is, Gan discloses a knowledge graph triple utilized for trustworthiness or confidence. It would have been obvious to one of ordinary skill in the art at the time of the effective filing date of the instant application to improve the prior art of Ram with that of Gan such that models of Ram may further include a set of knowledge graph triples as so disclosed by Gan. One of ordinary skill in the art would have been motivated to make the aforementioned combination such that the quality of LLMs may be improved.
Claim(s) 9 and 10 is/are rejected under 35 U.S.C. 103 as being unpatentable over Ram, in view of Bhatia et al, USPGPUB No. 2020/0104395, filed on 2 October 2018, and published on 2 April 2020.
As per dependent claim 9, Ram, in combination with Bhatia, discloses:
The computer system of claim 1, wherein the first generator model comprises a neural network, the second generator model comprises a neural network, and the one or more processors are configured to determine reliability of output generated by one of the neural networks for a current input to the respective neural network, based on vectorized state information of the respective neural network, the vectorized state information including at least an embedding vector formed by last hidden layer activations of the respective neural network {See Bhatia, [0144], wherein this reads over “In addition, using the error loss feedback vector 654, the user embeddings system 104 can train the interaction-to-vector neural network 600 via back propagation until the overall loss is minimized. Indeed, the user embeddings system 104 can conclude training when the interaction-to-vector neural network 600 converges and/or the total training loss amount is minimized. For example, the user embeddings system 104 utilizes the error loss feedback vector 654 to tune the weights and parameters of the first weighted matrix 614 and the second weighted matrix 624 to iteratively minimize loss. In additional embodiments, the user embeddings system 104 utilizes the error loss feedback vector 654 to tune parameters of the hidden layer 620 (e.g., add, remove, or modify neurons) to further minimize error loss.”}, and to discard the output from the respective neural network if said output is characterized by vectorized state information which has a similarity below a defined similarity threshold with respect to vectorized state information produced by the respective neural network for truthful training data {See Bhatia, [0083], wherein this reads over “In some embodiments, the user embeddings system 104 applies an additional minimum similarity threshold to exclude user segment that are beyond a similarity threshold distance to a base user. Indeed, if the next closest additional user to the base user is too far away (e.g., based on a radial distance or a Euclidean radius distance in vector representation space), then the user embeddings system 104 determines that the additional user is not similar enough to the base user to be grouped to the user and/or included in the expanded user segment.”}.
Ram fails to expressly disclose the features of the instant claim. Bhatia is directed to the invention of automatic segment expansion of user embeddings using multiple user embedding representation types. Specifically, Bhatia discloses that “the user embeddings system 104 utilizes the error loss feedback vector 654 to tune parameters of the hidden layer 620 (e.g., add, remove, or modify neurons) to further minimize error loss.” See Bhatia, [0144]. This disclosure would read upon the claimed feature of “wherein the first generator model comprises a neural network, the second generator model comprises a neural network, and the one or more processors are configured to determine reliability of output generated by one of the neural networks for a current input to the respective neural network, based on vectorized state information of the respective neural network, the vectorized state information including at least an embedding vector formed by last hidden layer activations of the respective neural network.” Additionally, Bhatia discloses that “the user embeddings system 104 applies an additional minimum similarity threshold to exclude user segment that are beyond a similarity threshold distance to a base user” and “if the next closest additional user to the base user is too far away (e.g., based on a radial distance or a Euclidean radius distance in vector representation space), then the user embeddings system 104 determines that the additional user is not similar enough to the base user to be grouped to the user and/or included in the expanded user segment” which would read upon the claimed feature of See Bhatia, [0083]. It would have been obvious to one of ordinary skill in the art at the time of the effective filing date of the instant application to improve the prior art of Ram with that of Bhatia such that the building management system with natural language model-based data structure generation may further include determine the reliability of the neural network via the error loss feedback vector evaluation of Bhatia. One of ordinary skill in the art would have been motivated to make the aforementioned combination such that the reliability of the output may be determined for improved performance.
As per dependent claim 10, Ram, in combination with Bhatia, discloses:
The computer system of claim 9, wherein the one or more processors are configured to determine the reliability of output generated by one of the neural networks for an input sequence to the respective neural network, based on vectorized state information generated from a series of the vectorized state information produced by the respective neural network for the input sequence {See Bhatia, [0143], wherein this reads over “Further, in some embodiments, the loss layer 650 utilizes a loss model to determine an amount of loss (i.e., training loss), which is used to train the interaction-to-vector neural network 600. For example, the loss layer 650 determines training loss by comparing the output probabilities 642a-442n to a ground truth (e.g., training data) to determine the error loss between each of the output probabilities 642a-442n and the ground truth, which is shown as the loss vectors 652a-452n. In particular, the user embeddings system 104 determines the cross-entropy loss between the output probabilities 642a-442n and the training data.”}.
Allowable Subject Matter
Claims 5, 6, 8, and 11 are objected to as being dependent upon a rejected base claim, but would be allowable if rewritten in independent form including all of the limitations of the base claim and any intervening claims.
Conclusion
Any inquiry concerning this communication or earlier communications from the examiner should be directed to PAUL KIM whose telephone number is (571)272-2737. The examiner can normally be reached Monday-Friday, 9AM-5PM.
Examiner interviews are available via telephone, in-person, and video conferencing using a USPTO supplied web-based collaboration tool. To schedule an interview, applicant is encouraged to use the USPTO Automated Interview Request (AIR) at http://www.uspto.gov/interviewpractice.
If attempts to reach the examiner by telephone are unsuccessful, the examiner’s supervisor, Sanjiv Shah can be reached at (571) 272-4098. The fax phone number for the organization where this application or proceeding is assigned is 571-273-8300.
Information regarding the status of published or unpublished applications may be obtained from Patent Center. Unpublished application information in Patent Center is available to registered users. To file and manage patent submissions in Patent Center, visit: https://patentcenter.uspto.gov. Visit https://www.uspto.gov/patents/apply/patent-center for more information about Patent Center and https://www.uspto.gov/patents/docx for information about filing in DOCX format. For additional questions, contact the Electronic Business Center (EBC) at 866-217-9197 (toll-free). If you would like assistance from a USPTO Customer Service Representative, call 800-786-9199 (IN USA OR CANADA) or 571-272-1000.
/Paul Kim/
Primary Examiner
Art Unit 2166
/PK/
Prosecution Insights